2024-03-09T14:14:32.452 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-09T14:14:32.452 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_cbodley@teuthology --name cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi --archive /home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479 --description rgw/verify/{0-install accounts$/{none} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} --verbose -- /home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479/orig.config.yaml 2024-03-09T14:14:32.500 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-09T14:14:32.588 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479 branch: wip-rgw-account-v3 description: rgw/verify/{0-install accounts$/{none} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} email: null first_in_suite: false job_id: '7588479' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-rgw-account-v3 ceph: conf: client: debug rgw: 20 rgw bucket counters cache: true rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw max chunk size: 4194304 rgw obj stripe size: 6291456 rgw s3 auth use sts: true rgw sts key: abcdefghijklmnop rgw torrent flag: true rgw user counters cache: true setgroup: ceph setuser: ceph global: mon client directed command retry: 5 ms inject socket failures: 5000 osd heartbeat grace: 40 osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 rgw override bucket index max shards: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd fast shutdown: false osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled - \(OSD_SLOW_PING_TIME sha1: c00d9c609fb42a2a25f1c196bed064ca8bb02413 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c00d9c609fb42a2a25f1c196bed064ca8bb02413 openssl_keys: rgw.client.0: ca: root client: client.0 embed-key: true root: client: client.0 cn: teuthology install: - client.0 key-type: rsa:4096 rgw: client.0: ssl certificate: rgw.client.0 valgrind: - --tool=memcheck - --max-threads=1024 compression type: random ec-data-pool: false frontend: beast inline data: false storage classes: LUKEWARM, FROZEN s3tests: accounts: iam alt root: RGW99999999999999999 iam root: RGW88888888888888888 force-branch: wip-iam-user-apis git_remote: https://github.com/cbodley/ storage classes: LUKEWARM, FROZEN thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rgw-account-v3 sha1: c00d9c609fb42a2a25f1c196bed064ca8bb02413 owner: scheduled_cbodley@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 3347 sha1: c00d9c609fb42a2a25f1c196bed064ca8bb02413 sleep_before_teardown: 0 suite: rgw suite_branch: wip-rgw-account-v3 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c00d9c609fb42a2a25f1c196bed064ca8bb02413 targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDXuFOquNPh30bZgAJqTouKGG7p0NNxx6xXxJjUuU/VardxqUiKgFx1dYvcLV+s2NboGkt+PBwES7RqxIXJxcpQ= smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKaPOccuN9FVN66Lr7uOPJ7Q4JA1CmmSB1iQvohXt3rGZ6uBS7/6xY3JYjapv/HuJSah4Lhzo+bkGObImlln7ic= tasks: - install: extra_system_packages: deb: - s3cmd - maven rpm: - s3cmd - maven - ceph: null - openssl_keys: null - rgw: client.0: null - tox: - client.0 - workunit: clients: client.0: - rgw/run-bucket-check.sh - workunit: clients: client.0: - cls/test_cls_lock.sh - cls/test_cls_log.sh - cls/test_cls_refcount.sh - cls/test_cls_rgw.sh - cls/test_cls_rgw_gc.sh - cls/test_cls_rgw_stats.sh - cls/test_cls_cmpomap.sh - cls/test_cls_2pc_queue.sh - cls/test_cls_user.sh - rgw/test_rgw_gc_log.sh - rgw/test_rgw_obj.sh - rgw/test_librgw_file.sh - rgw/test_awssdkv4_sig.sh - workunit: clients: client.0: - rgw/test_rgw_s3_mp_reupload.sh - ragweed: client.0: default-branch: ceph-master rgw_server: client.0 stages: prepare,check - workunit: clients: client.0: - rgw/run-reshard.sh - s3tests-java: client.0: force-branch: ceph-master force-repo: https://github.com/ceph/java_s3tests.git - s3tests: client.0: rgw_server: client.0 - workunit: clients: client.0: - rgw/run-versioning.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-09_13:26:09 tube: smithi user: cbodley verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-09T14:14:32.588 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa; will attempt to use it 2024-03-09T14:14:32.589 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks 2024-03-09T14:14:32.589 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-09T14:14:32.590 INFO:teuthology.task.internal:Checking packages... 2024-03-09T14:14:32.616 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c00d9c609fb42a2a25f1c196bed064ca8bb02413' 2024-03-09T14:14:32.616 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-09T14:14:32.616 INFO:teuthology.packaging:ref: None 2024-03-09T14:14:32.616 INFO:teuthology.packaging:tag: None 2024-03-09T14:14:32.617 INFO:teuthology.packaging:branch: wip-rgw-account-v3 2024-03-09T14:14:32.617 INFO:teuthology.packaging:sha1: c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:14:32.617 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-rgw-account-v3 2024-03-09T14:14:32.773 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2079-gc00d9c60-1jammy 2024-03-09T14:14:32.775 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-09T14:14:32.781 INFO:teuthology.task.internal:no buildpackages task found 2024-03-09T14:14:32.781 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-09T14:14:32.817 INFO:teuthology.task.internal:Saving configuration 2024-03-09T14:14:32.834 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-09T14:14:32.841 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T14:14:32.867 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-09 14:09:16.574079', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE8EEPFs16ID0U99dHnapzqR1rBlKZIOVI95CpEtZzhJ'} 2024-03-09T14:14:32.889 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-09 14:09:16.576045', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO4NBSgLb0gYTKl8wYarbKhJ3wL0QUA3Ma+AxPa1q0oY'} 2024-03-09T14:14:32.889 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-09T14:14:32.896 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-09T14:14:32.896 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-09T14:14:32.896 INFO:teuthology.run_tasks:Running task console_log... 2024-03-09T14:14:32.990 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f20751f90d0>, signals=[15]) 2024-03-09T14:14:32.991 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-09T14:14:32.998 INFO:teuthology.task.internal:Opening connections... 2024-03-09T14:14:32.999 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-03-09T14:14:33.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T14:14:33.088 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-03-09T14:14:33.088 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T14:14:33.167 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-09T14:14:33.174 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-03-09T14:14:33.179 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-03-09T14:14:33.179 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-03-09T14:14:33.226 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-09T14:14:33.226 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="Ubuntu" 2024-03-09T14:14:33.226 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="22.04" 2024-03-09T14:14:33.226 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-09T14:14:33.226 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_CODENAME=jammy 2024-03-09T14:14:33.226 INFO:teuthology.orchestra.run.smithi001.stdout:ID=ubuntu 2024-03-09T14:14:33.227 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE=debian 2024-03-09T14:14:33.227 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-09T14:14:33.227 INFO:teuthology.orchestra.run.smithi001.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-09T14:14:33.227 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-09T14:14:33.227 INFO:teuthology.orchestra.run.smithi001.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-09T14:14:33.227 INFO:teuthology.orchestra.run.smithi001.stdout:UBUNTU_CODENAME=jammy 2024-03-09T14:14:33.228 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-03-09T14:14:33.252 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-03-09T14:14:33.257 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-03-09T14:14:33.257 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-03-09T14:14:33.305 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-09T14:14:33.305 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="Ubuntu" 2024-03-09T14:14:33.306 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="22.04" 2024-03-09T14:14:33.306 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-09T14:14:33.306 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_CODENAME=jammy 2024-03-09T14:14:33.306 INFO:teuthology.orchestra.run.smithi053.stdout:ID=ubuntu 2024-03-09T14:14:33.306 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE=debian 2024-03-09T14:14:33.306 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-09T14:14:33.306 INFO:teuthology.orchestra.run.smithi053.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-09T14:14:33.306 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-09T14:14:33.306 INFO:teuthology.orchestra.run.smithi053.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-09T14:14:33.307 INFO:teuthology.orchestra.run.smithi053.stdout:UBUNTU_CODENAME=jammy 2024-03-09T14:14:33.307 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-03-09T14:14:33.327 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-09T14:14:33.338 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-09T14:14:33.367 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-09T14:14:33.367 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-03-09T14:14:33.369 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-03-09T14:14:33.374 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-09T14:14:33.400 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-09T14:14:33.401 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-03-09T14:14:33.415 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-03-09T14:14:33.430 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-09T14:14:33.480 INFO:teuthology.run_tasks:Running task kernel... 2024-03-09T14:14:33.497 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-09T14:14:33.497 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-09T14:14:33.498 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-09T14:14:33.498 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-09T14:14:33.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:14:33.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:14:33.505 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-03-09T14:14:33.506 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-03-09T14:14:33.554 INFO:teuthology.orchestra.run.smithi001.stdout:5.15.0-56-generic 2024-03-09T14:14:33.555 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.15.0-56-generic 2024-03-09T14:14:33.555 DEBUG:teuthology.orchestra.run.smithi001:> sudo apt-get clean 2024-03-09T14:14:33.558 INFO:teuthology.orchestra.run.smithi053.stdout:5.15.0-56-generic 2024-03-09T14:14:33.558 INFO:teuthology.task.kernel:Running kernel on smithi053: 5.15.0-56-generic 2024-03-09T14:14:33.558 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-get clean 2024-03-09T14:14:33.672 DEBUG:teuthology.orchestra.run.smithi001:> sudo apt-get update 2024-03-09T14:14:33.691 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-get update 2024-03-09T14:14:33.812 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-09T14:14:33.823 INFO:teuthology.orchestra.run.smithi001.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-09T14:14:33.843 INFO:teuthology.orchestra.run.smithi001.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-09T14:14:33.865 INFO:teuthology.orchestra.run.smithi053.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-09T14:14:33.946 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-09T14:14:33.951 INFO:teuthology.orchestra.run.smithi001.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-09T14:14:34.028 INFO:teuthology.orchestra.run.smithi053.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-09T14:14:34.066 INFO:teuthology.orchestra.run.smithi001.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [419 kB] 2024-03-09T14:14:34.084 INFO:teuthology.orchestra.run.smithi053.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,239 kB] 2024-03-09T14:14:34.143 INFO:teuthology.orchestra.run.smithi001.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,458 kB] 2024-03-09T14:14:34.151 INFO:teuthology.orchestra.run.smithi001.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,239 kB] 2024-03-09T14:14:34.193 INFO:teuthology.orchestra.run.smithi001.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [222 kB] 2024-03-09T14:14:34.197 INFO:teuthology.orchestra.run.smithi001.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-09T14:14:34.198 INFO:teuthology.orchestra.run.smithi001.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,523 kB] 2024-03-09T14:14:34.199 INFO:teuthology.orchestra.run.smithi053.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [419 kB] 2024-03-09T14:14:34.207 INFO:teuthology.orchestra.run.smithi053.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [222 kB] 2024-03-09T14:14:34.213 INFO:teuthology.orchestra.run.smithi053.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-09T14:14:34.213 INFO:teuthology.orchestra.run.smithi053.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-09T14:14:34.214 INFO:teuthology.orchestra.run.smithi053.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,523 kB] 2024-03-09T14:14:34.227 INFO:teuthology.orchestra.run.smithi001.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-09T14:14:34.228 INFO:teuthology.orchestra.run.smithi001.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [253 kB] 2024-03-09T14:14:34.234 INFO:teuthology.orchestra.run.smithi001.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-09T14:14:34.234 INFO:teuthology.orchestra.run.smithi001.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-03-09T14:14:34.246 INFO:teuthology.orchestra.run.smithi001.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [597 kB] 2024-03-09T14:14:34.249 INFO:teuthology.orchestra.run.smithi053.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [253 kB] 2024-03-09T14:14:34.254 INFO:teuthology.orchestra.run.smithi053.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-09T14:14:34.255 INFO:teuthology.orchestra.run.smithi053.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-03-09T14:14:34.255 INFO:teuthology.orchestra.run.smithi001.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-09T14:14:34.257 INFO:teuthology.orchestra.run.smithi001.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-09T14:14:34.263 INFO:teuthology.orchestra.run.smithi001.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-09T14:14:34.263 INFO:teuthology.orchestra.run.smithi001.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-09T14:14:34.264 INFO:teuthology.orchestra.run.smithi053.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [597 kB] 2024-03-09T14:14:34.269 INFO:teuthology.orchestra.run.smithi053.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-09T14:14:34.271 INFO:teuthology.orchestra.run.smithi053.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-09T14:14:34.275 INFO:teuthology.orchestra.run.smithi053.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-09T14:14:34.275 INFO:teuthology.orchestra.run.smithi053.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-09T14:14:34.276 INFO:teuthology.orchestra.run.smithi053.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-09T14:14:34.276 INFO:teuthology.orchestra.run.smithi053.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-09T14:14:34.287 INFO:teuthology.orchestra.run.smithi001.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [585 kB] 2024-03-09T14:14:34.287 INFO:teuthology.orchestra.run.smithi001.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-09T14:14:34.288 INFO:teuthology.orchestra.run.smithi001.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-09T14:14:34.298 INFO:teuthology.orchestra.run.smithi001.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [283 kB] 2024-03-09T14:14:34.304 INFO:teuthology.orchestra.run.smithi001.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-09T14:14:34.305 INFO:teuthology.orchestra.run.smithi001.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,557 kB] 2024-03-09T14:14:34.333 INFO:teuthology.orchestra.run.smithi001.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-09T14:14:34.333 INFO:teuthology.orchestra.run.smithi001.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [259 kB] 2024-03-09T14:14:34.338 INFO:teuthology.orchestra.run.smithi001.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-09T14:14:34.338 INFO:teuthology.orchestra.run.smithi001.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,054 kB] 2024-03-09T14:14:34.351 INFO:teuthology.orchestra.run.smithi001.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [693 kB] 2024-03-09T14:14:34.358 INFO:teuthology.orchestra.run.smithi001.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [238 kB] 2024-03-09T14:14:34.361 INFO:teuthology.orchestra.run.smithi001.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-09T14:14:34.361 INFO:teuthology.orchestra.run.smithi001.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-09T14:14:34.362 INFO:teuthology.orchestra.run.smithi001.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-09T14:14:34.364 INFO:teuthology.orchestra.run.smithi001.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-09T14:14:34.364 INFO:teuthology.orchestra.run.smithi001.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-09T14:14:34.365 INFO:teuthology.orchestra.run.smithi001.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-03-09T14:14:34.365 INFO:teuthology.orchestra.run.smithi001.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-03-09T14:14:34.379 INFO:teuthology.orchestra.run.smithi053.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-09T14:14:34.383 INFO:teuthology.orchestra.run.smithi001.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-03-09T14:14:34.384 INFO:teuthology.orchestra.run.smithi001.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-09T14:14:34.384 INFO:teuthology.orchestra.run.smithi001.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-03-09T14:14:34.384 INFO:teuthology.orchestra.run.smithi001.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2024-03-09T14:14:34.384 INFO:teuthology.orchestra.run.smithi001.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-03-09T14:14:34.384 INFO:teuthology.orchestra.run.smithi001.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-09T14:14:34.517 INFO:teuthology.orchestra.run.smithi053.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,458 kB] 2024-03-09T14:14:34.817 INFO:teuthology.orchestra.run.smithi053.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [585 kB] 2024-03-09T14:14:34.856 INFO:teuthology.orchestra.run.smithi053.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [283 kB] 2024-03-09T14:14:34.872 INFO:teuthology.orchestra.run.smithi053.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-09T14:14:34.872 INFO:teuthology.orchestra.run.smithi053.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,557 kB] 2024-03-09T14:14:34.932 INFO:teuthology.orchestra.run.smithi053.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-09T14:14:34.933 INFO:teuthology.orchestra.run.smithi053.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [259 kB] 2024-03-09T14:14:34.939 INFO:teuthology.orchestra.run.smithi053.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-09T14:14:35.103 INFO:teuthology.orchestra.run.smithi053.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [693 kB] 2024-03-09T14:14:35.680 INFO:teuthology.orchestra.run.smithi053.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,054 kB] 2024-03-09T14:14:35.790 INFO:teuthology.orchestra.run.smithi053.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [238 kB] 2024-03-09T14:14:35.810 INFO:teuthology.orchestra.run.smithi053.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-09T14:14:35.811 INFO:teuthology.orchestra.run.smithi053.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-09T14:14:35.815 INFO:teuthology.orchestra.run.smithi053.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-09T14:14:35.815 INFO:teuthology.orchestra.run.smithi053.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-09T14:14:35.816 INFO:teuthology.orchestra.run.smithi053.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-09T14:14:35.816 INFO:teuthology.orchestra.run.smithi053.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-03-09T14:14:35.819 INFO:teuthology.orchestra.run.smithi053.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-03-09T14:14:35.821 INFO:teuthology.orchestra.run.smithi053.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-03-09T14:14:35.986 INFO:teuthology.orchestra.run.smithi053.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-09T14:14:36.069 INFO:teuthology.orchestra.run.smithi053.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-03-09T14:14:36.074 INFO:teuthology.orchestra.run.smithi053.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2024-03-09T14:14:36.193 INFO:teuthology.orchestra.run.smithi053.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-03-09T14:14:36.244 INFO:teuthology.orchestra.run.smithi053.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-09T14:14:39.926 INFO:teuthology.orchestra.run.smithi001.stdout:Fetched 12.1 MB in 2s (6,812 kB/s) 2024-03-09T14:14:40.821 INFO:teuthology.orchestra.run.smithi053.stdout:Fetched 12.1 MB in 2s (4,864 kB/s) 2024-03-09T14:14:41.214 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-03-09T14:14:41.232 DEBUG:teuthology.orchestra.run.smithi001:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-09T14:14:41.293 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-03-09T14:14:41.498 INFO:teuthology.orchestra.run.smithi001.stdout:Building dependency tree... 2024-03-09T14:14:41.498 INFO:teuthology.orchestra.run.smithi001.stdout:Reading state information... 2024-03-09T14:14:41.677 INFO:teuthology.orchestra.run.smithi001.stdout:The following packages were automatically installed and are no longer required: 2024-03-09T14:14:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-09T14:14:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-09T14:14:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-09T14:14:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-09T14:14:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: python2.7-minimal 2024-03-09T14:14:41.678 INFO:teuthology.orchestra.run.smithi001.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-09T14:14:41.679 INFO:teuthology.orchestra.run.smithi001.stdout:The following additional packages will be installed: 2024-03-09T14:14:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: linux-generic linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-09T14:14:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: linux-headers-generic linux-image-5.15.0-100-generic 2024-03-09T14:14:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: linux-modules-5.15.0-100-generic linux-modules-extra-5.15.0-100-generic 2024-03-09T14:14:41.681 INFO:teuthology.orchestra.run.smithi001.stdout:Suggested packages: 2024-03-09T14:14:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-09T14:14:41.681 INFO:teuthology.orchestra.run.smithi001.stdout:Recommended packages: 2024-03-09T14:14:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: thermald 2024-03-09T14:14:41.732 INFO:teuthology.orchestra.run.smithi001.stdout:The following NEW packages will be installed: 2024-03-09T14:14:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-09T14:14:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: linux-image-5.15.0-100-generic linux-modules-5.15.0-100-generic 2024-03-09T14:14:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: linux-modules-extra-5.15.0-100-generic 2024-03-09T14:14:41.733 INFO:teuthology.orchestra.run.smithi001.stdout:The following packages will be upgraded: 2024-03-09T14:14:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-09T14:14:41.792 INFO:teuthology.orchestra.run.smithi001.stdout:3 upgraded, 5 newly installed, 0 to remove and 288 not upgraded. 2024-03-09T14:14:41.792 INFO:teuthology.orchestra.run.smithi001.stdout:Need to get 113 MB of archives. 2024-03-09T14:14:41.792 INFO:teuthology.orchestra.run.smithi001.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-09T14:14:41.792 INFO:teuthology.orchestra.run.smithi001.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-100-generic amd64 5.15.0-100.110 [22.6 MB] 2024-03-09T14:14:42.170 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2024-03-09T14:14:42.187 DEBUG:teuthology.orchestra.run.smithi053:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-09T14:14:42.250 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2024-03-09T14:14:42.287 INFO:teuthology.orchestra.run.smithi001.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-100-generic amd64 5.15.0-100.110 [11.5 MB] 2024-03-09T14:14:42.435 INFO:teuthology.orchestra.run.smithi001.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-100-generic amd64 5.15.0-100.110 [63.8 MB] 2024-03-09T14:14:42.447 INFO:teuthology.orchestra.run.smithi053.stdout:Building dependency tree... 2024-03-09T14:14:42.448 INFO:teuthology.orchestra.run.smithi053.stdout:Reading state information... 2024-03-09T14:14:42.623 INFO:teuthology.orchestra.run.smithi053.stdout:The following packages were automatically installed and are no longer required: 2024-03-09T14:14:42.624 INFO:teuthology.orchestra.run.smithi053.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-09T14:14:42.624 INFO:teuthology.orchestra.run.smithi053.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-09T14:14:42.624 INFO:teuthology.orchestra.run.smithi053.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-09T14:14:42.624 INFO:teuthology.orchestra.run.smithi053.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-09T14:14:42.624 INFO:teuthology.orchestra.run.smithi053.stdout: python2.7-minimal 2024-03-09T14:14:42.625 INFO:teuthology.orchestra.run.smithi053.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-09T14:14:42.626 INFO:teuthology.orchestra.run.smithi053.stdout:The following additional packages will be installed: 2024-03-09T14:14:42.626 INFO:teuthology.orchestra.run.smithi053.stdout: linux-generic linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-09T14:14:42.626 INFO:teuthology.orchestra.run.smithi053.stdout: linux-headers-generic linux-image-5.15.0-100-generic 2024-03-09T14:14:42.626 INFO:teuthology.orchestra.run.smithi053.stdout: linux-modules-5.15.0-100-generic linux-modules-extra-5.15.0-100-generic 2024-03-09T14:14:42.627 INFO:teuthology.orchestra.run.smithi053.stdout:Suggested packages: 2024-03-09T14:14:42.627 INFO:teuthology.orchestra.run.smithi053.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-09T14:14:42.628 INFO:teuthology.orchestra.run.smithi053.stdout:Recommended packages: 2024-03-09T14:14:42.628 INFO:teuthology.orchestra.run.smithi053.stdout: thermald 2024-03-09T14:14:42.787 INFO:teuthology.orchestra.run.smithi053.stdout:The following NEW packages will be installed: 2024-03-09T14:14:42.787 INFO:teuthology.orchestra.run.smithi053.stdout: linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-09T14:14:42.787 INFO:teuthology.orchestra.run.smithi053.stdout: linux-image-5.15.0-100-generic linux-modules-5.15.0-100-generic 2024-03-09T14:14:42.788 INFO:teuthology.orchestra.run.smithi053.stdout: linux-modules-extra-5.15.0-100-generic 2024-03-09T14:14:42.789 INFO:teuthology.orchestra.run.smithi053.stdout:The following packages will be upgraded: 2024-03-09T14:14:42.789 INFO:teuthology.orchestra.run.smithi053.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-09T14:14:43.001 INFO:teuthology.orchestra.run.smithi053.stdout:3 upgraded, 5 newly installed, 0 to remove and 288 not upgraded. 2024-03-09T14:14:43.001 INFO:teuthology.orchestra.run.smithi053.stdout:Need to get 113 MB of archives. 2024-03-09T14:14:43.001 INFO:teuthology.orchestra.run.smithi053.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-09T14:14:43.001 INFO:teuthology.orchestra.run.smithi053.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-100-generic amd64 5.15.0-100.110 [22.6 MB] 2024-03-09T14:14:43.384 INFO:teuthology.orchestra.run.smithi001.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.100.97 [1,696 B] 2024-03-09T14:14:43.384 INFO:teuthology.orchestra.run.smithi001.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.100.97 [2,454 B] 2024-03-09T14:14:43.384 INFO:teuthology.orchestra.run.smithi001.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100 all 5.15.0-100.110 [12.3 MB] 2024-03-09T14:14:43.685 INFO:teuthology.orchestra.run.smithi001.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100-generic amd64 5.15.0-100.110 [2,858 kB] 2024-03-09T14:14:43.751 INFO:teuthology.orchestra.run.smithi001.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.100.97 [2,292 B] 2024-03-09T14:14:44.180 INFO:teuthology.orchestra.run.smithi001.stdout:Fetched 113 MB in 2s (56.3 MB/s) 2024-03-09T14:14:44.368 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package linux-modules-5.15.0-100-generic. 2024-03-09T14:14:44.399 INFO:teuthology.orchestra.run.smithi053.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-100-generic amd64 5.15.0-100.110 [11.5 MB] 2024-03-09T14:14:44.733 INFO:teuthology.orchestra.run.smithi053.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-100-generic amd64 5.15.0-100.110 [63.8 MB] 2024-03-09T14:14:46.324 INFO:teuthology.orchestra.run.smithi001.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-09T14:14:46.328 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../0-linux-modules-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-09T14:14:46.391 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:14:46.846 INFO:teuthology.orchestra.run.smithi053.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.100.97 [1,696 B] 2024-03-09T14:14:46.846 INFO:teuthology.orchestra.run.smithi053.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.100.97 [2,454 B] 2024-03-09T14:14:46.847 INFO:teuthology.orchestra.run.smithi053.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100 all 5.15.0-100.110 [12.3 MB] 2024-03-09T14:14:47.332 INFO:teuthology.orchestra.run.smithi053.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100-generic amd64 5.15.0-100.110 [2,858 kB] 2024-03-09T14:14:47.434 INFO:teuthology.orchestra.run.smithi053.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.100.97 [2,292 B] 2024-03-09T14:14:47.856 INFO:teuthology.orchestra.run.smithi053.stdout:Fetched 113 MB in 5s (24.4 MB/s) 2024-03-09T14:14:48.025 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package linux-modules-5.15.0-100-generic. 2024-03-09T14:14:48.416 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package linux-image-5.15.0-100-generic. 2024-03-09T14:14:48.438 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../1-linux-image-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-09T14:14:48.538 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:14:48.912 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-100-generic. 2024-03-09T14:14:48.932 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-09T14:14:48.961 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:14:50.020 INFO:teuthology.orchestra.run.smithi053.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-09T14:14:50.024 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../0-linux-modules-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-09T14:14:50.106 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:14:52.416 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package linux-image-5.15.0-100-generic. 2024-03-09T14:14:52.438 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../1-linux-image-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-09T14:14:52.550 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:14:53.036 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-100-generic. 2024-03-09T14:14:53.057 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-09T14:14:53.094 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:14:54.713 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../3-linux-generic_5.15.0.100.97_amd64.deb ... 2024-03-09T14:14:55.179 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-09T14:14:55.529 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.100.97_amd64.deb ... 2024-03-09T14:14:55.641 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-image-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-09T14:14:55.885 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package linux-headers-5.15.0-100. 2024-03-09T14:14:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../5-linux-headers-5.15.0-100_5.15.0-100.110_all.deb ... 2024-03-09T14:14:55.952 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-09T14:14:59.432 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../3-linux-generic_5.15.0.100.97_amd64.deb ... 2024-03-09T14:14:59.542 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-09T14:14:59.875 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.100.97_amd64.deb ... 2024-03-09T14:15:00.028 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-image-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-09T14:15:00.323 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package linux-headers-5.15.0-100. 2024-03-09T14:15:00.345 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../5-linux-headers-5.15.0-100_5.15.0-100.110_all.deb ... 2024-03-09T14:15:00.389 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-09T14:15:01.251 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package linux-headers-5.15.0-100-generic. 2024-03-09T14:15:01.289 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../6-linux-headers-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-09T14:15:01.326 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:15:03.939 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.100.97_amd64.deb ... 2024-03-09T14:15:04.061 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-headers-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-09T14:15:04.387 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-09T14:15:04.508 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:15:04.634 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-headers-generic (5.15.0.100.97) ... 2024-03-09T14:15:04.762 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:15:06.112 INFO:teuthology.orchestra.run.smithi001.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-09T14:15:06.113 INFO:teuthology.orchestra.run.smithi001.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-09T14:15:06.113 INFO:teuthology.orchestra.run.smithi001.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-100-generic 2024-03-09T14:15:06.113 INFO:teuthology.orchestra.run.smithi001.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-100-generic 2024-03-09T14:15:06.263 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package linux-headers-5.15.0-100-generic. 2024-03-09T14:15:06.280 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:15:06.300 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../6-linux-headers-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-09T14:15:06.338 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:15:07.699 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:15:09.073 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.100.97_amd64.deb ... 2024-03-09T14:15:09.116 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-image-generic (5.15.0.100.97) ... 2024-03-09T14:15:09.197 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-headers-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-09T14:15:09.298 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-generic (5.15.0.100.97) ... 2024-03-09T14:15:10.019 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:15:10.087 INFO:teuthology.orchestra.run.smithi001.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-09T14:15:10.087 INFO:teuthology.orchestra.run.smithi001.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-100-generic 2024-03-09T14:15:10.098 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-09T14:15:10.210 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:15:10.312 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-headers-generic (5.15.0.100.97) ... 2024-03-09T14:15:10.441 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:15:11.786 INFO:teuthology.orchestra.run.smithi053.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-09T14:15:11.786 INFO:teuthology.orchestra.run.smithi053.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-09T14:15:11.786 INFO:teuthology.orchestra.run.smithi053.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-100-generic 2024-03-09T14:15:11.786 INFO:teuthology.orchestra.run.smithi053.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-100-generic 2024-03-09T14:15:11.935 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:15:13.353 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:15:14.746 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-image-generic (5.15.0.100.97) ... 2024-03-09T14:15:14.872 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-generic (5.15.0.100.97) ... 2024-03-09T14:15:15.007 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-09T14:15:15.808 INFO:teuthology.orchestra.run.smithi053.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-09T14:15:15.808 INFO:teuthology.orchestra.run.smithi053.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-100-generic 2024-03-09T14:15:24.884 INFO:teuthology.orchestra.run.smithi001.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-09T14:15:24.885 INFO:teuthology.orchestra.run.smithi001.stdout:Sourcing file `/etc/default/grub' 2024-03-09T14:15:24.895 INFO:teuthology.orchestra.run.smithi001.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-09T14:15:24.905 INFO:teuthology.orchestra.run.smithi001.stdout:Generating grub configuration file ... 2024-03-09T14:15:25.154 INFO:teuthology.orchestra.run.smithi001.stdout:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-09T14:15:25.177 INFO:teuthology.orchestra.run.smithi001.stdout:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-09T14:15:25.490 INFO:teuthology.orchestra.run.smithi001.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-09T14:15:25.496 INFO:teuthology.orchestra.run.smithi001.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-09T14:15:25.574 INFO:teuthology.orchestra.run.smithi001.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-09T14:15:25.579 INFO:teuthology.orchestra.run.smithi001.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-09T14:15:25.729 INFO:teuthology.orchestra.run.smithi001.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-09T14:15:25.729 INFO:teuthology.orchestra.run.smithi001.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-09T14:15:25.729 INFO:teuthology.orchestra.run.smithi001.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-09T14:15:25.775 INFO:teuthology.orchestra.run.smithi001.stdout:done 2024-03-09T14:15:26.259 DEBUG:teuthology.orchestra.run.smithi001:> dpkg -s linux-image-generic 2024-03-09T14:15:26.279 INFO:teuthology.orchestra.run.smithi001.stdout:Package: linux-image-generic 2024-03-09T14:15:26.279 INFO:teuthology.orchestra.run.smithi001.stdout:Status: install ok installed 2024-03-09T14:15:26.279 INFO:teuthology.orchestra.run.smithi001.stdout:Priority: optional 2024-03-09T14:15:26.279 INFO:teuthology.orchestra.run.smithi001.stdout:Section: kernel 2024-03-09T14:15:26.279 INFO:teuthology.orchestra.run.smithi001.stdout:Installed-Size: 21 2024-03-09T14:15:26.279 INFO:teuthology.orchestra.run.smithi001.stdout:Maintainer: Ubuntu Kernel Team 2024-03-09T14:15:26.280 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: amd64 2024-03-09T14:15:26.280 INFO:teuthology.orchestra.run.smithi001.stdout:Source: linux-meta 2024-03-09T14:15:26.280 INFO:teuthology.orchestra.run.smithi001.stdout:Version: 5.15.0.100.97 2024-03-09T14:15:26.280 INFO:teuthology.orchestra.run.smithi001.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-09T14:15:26.280 INFO:teuthology.orchestra.run.smithi001.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-09T14:15:26.280 INFO:teuthology.orchestra.run.smithi001.stdout:Recommends: thermald 2024-03-09T14:15:26.280 INFO:teuthology.orchestra.run.smithi001.stdout:Description: Generic Linux kernel image 2024-03-09T14:15:26.280 INFO:teuthology.orchestra.run.smithi001.stdout: This package will always depend on the latest generic kernel image 2024-03-09T14:15:26.280 INFO:teuthology.orchestra.run.smithi001.stdout: available. 2024-03-09T14:15:26.280 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-100-generic 2024-03-09T14:15:26.281 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-09T14:15:26.281 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-09T14:15:26.281 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-09T14:15:26.281 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=None, version=distro) 2024-03-09T14:15:26.281 DEBUG:teuthology.orchestra.run.smithi001:> sudo apt-get clean 2024-03-09T14:15:26.426 DEBUG:teuthology.orchestra.run.smithi001:> sudo apt-get update 2024-03-09T14:15:26.557 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-09T14:15:26.557 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-09T14:15:26.567 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-09T14:15:26.587 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-09T14:15:27.927 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-03-09T14:15:27.945 DEBUG:teuthology.orchestra.run.smithi001:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-09T14:15:28.007 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-03-09T14:15:28.205 INFO:teuthology.orchestra.run.smithi001.stdout:Building dependency tree... 2024-03-09T14:15:28.205 INFO:teuthology.orchestra.run.smithi001.stdout:Reading state information... 2024-03-09T14:15:28.387 INFO:teuthology.orchestra.run.smithi001.stdout:linux-image-generic is already the newest version (5.15.0.100.97). 2024-03-09T14:15:28.387 INFO:teuthology.orchestra.run.smithi001.stdout:The following packages were automatically installed and are no longer required: 2024-03-09T14:15:28.387 INFO:teuthology.orchestra.run.smithi001.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-09T14:15:28.387 INFO:teuthology.orchestra.run.smithi001.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-09T14:15:28.387 INFO:teuthology.orchestra.run.smithi001.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-09T14:15:28.388 INFO:teuthology.orchestra.run.smithi001.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-09T14:15:28.388 INFO:teuthology.orchestra.run.smithi001.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-09T14:15:28.388 INFO:teuthology.orchestra.run.smithi001.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-09T14:15:28.427 INFO:teuthology.orchestra.run.smithi001.stdout:0 upgraded, 0 newly installed, 0 to remove and 288 not upgraded. 2024-03-09T14:15:28.429 DEBUG:teuthology.orchestra.run.smithi001:> dpkg -s linux-image-generic 2024-03-09T14:15:28.446 INFO:teuthology.orchestra.run.smithi001.stdout:Package: linux-image-generic 2024-03-09T14:15:28.446 INFO:teuthology.orchestra.run.smithi001.stdout:Status: install ok installed 2024-03-09T14:15:28.446 INFO:teuthology.orchestra.run.smithi001.stdout:Priority: optional 2024-03-09T14:15:28.447 INFO:teuthology.orchestra.run.smithi001.stdout:Section: kernel 2024-03-09T14:15:28.447 INFO:teuthology.orchestra.run.smithi001.stdout:Installed-Size: 21 2024-03-09T14:15:28.447 INFO:teuthology.orchestra.run.smithi001.stdout:Maintainer: Ubuntu Kernel Team 2024-03-09T14:15:28.447 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: amd64 2024-03-09T14:15:28.447 INFO:teuthology.orchestra.run.smithi001.stdout:Source: linux-meta 2024-03-09T14:15:28.447 INFO:teuthology.orchestra.run.smithi001.stdout:Version: 5.15.0.100.97 2024-03-09T14:15:28.447 INFO:teuthology.orchestra.run.smithi001.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-09T14:15:28.447 INFO:teuthology.orchestra.run.smithi001.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-09T14:15:28.447 INFO:teuthology.orchestra.run.smithi001.stdout:Recommends: thermald 2024-03-09T14:15:28.448 INFO:teuthology.orchestra.run.smithi001.stdout:Description: Generic Linux kernel image 2024-03-09T14:15:28.448 INFO:teuthology.orchestra.run.smithi001.stdout: This package will always depend on the latest generic kernel image 2024-03-09T14:15:28.448 INFO:teuthology.orchestra.run.smithi001.stdout: available. 2024-03-09T14:15:28.448 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-09T14:15:28.494 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.zHsznAkovk 2024-03-09T14:15:28.495 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub/grub.cfg /tmp/tmp.zHsznAkovk 2024-03-09T14:15:28.550 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.zHsznAkovk 2024-03-09T14:15:28.670 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.zHsznAkovk is 10KB 2024-03-09T14:15:28.683 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.zHsznAkovk 2024-03-09T14:15:28.688 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-09T14:15:28.742 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:15:28.742 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-09T14:15:28.742 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-09T14:15:28.809 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-100-generic 2024-03-09T14:15:28.810 DEBUG:teuthology.orchestra.run.smithi001:> sudo update-grub 2024-03-09T14:15:29.898 INFO:teuthology.orchestra.run.smithi001.stderr:Sourcing file `/etc/default/grub' 2024-03-09T14:15:29.899 INFO:teuthology.orchestra.run.smithi001.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-09T14:15:29.901 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-03-09T14:15:30.092 INFO:teuthology.orchestra.run.smithi001.stderr:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-09T14:15:30.096 INFO:teuthology.orchestra.run.smithi053.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-09T14:15:30.097 INFO:teuthology.orchestra.run.smithi053.stdout:Sourcing file `/etc/default/grub' 2024-03-09T14:15:30.099 INFO:teuthology.orchestra.run.smithi001.stderr:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-09T14:15:30.108 INFO:teuthology.orchestra.run.smithi053.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-09T14:15:30.136 INFO:teuthology.orchestra.run.smithi053.stdout:Generating grub configuration file ... 2024-03-09T14:15:30.327 INFO:teuthology.orchestra.run.smithi001.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-09T14:15:30.333 INFO:teuthology.orchestra.run.smithi001.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-09T14:15:30.376 INFO:teuthology.orchestra.run.smithi001.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-09T14:15:30.381 INFO:teuthology.orchestra.run.smithi001.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-09T14:15:30.418 INFO:teuthology.orchestra.run.smithi053.stdout:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-09T14:15:30.443 INFO:teuthology.orchestra.run.smithi053.stdout:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-09T14:15:30.489 INFO:teuthology.orchestra.run.smithi001.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-09T14:15:30.489 INFO:teuthology.orchestra.run.smithi001.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-09T14:15:30.489 INFO:teuthology.orchestra.run.smithi001.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-09T14:15:30.500 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-03-09T14:15:30.508 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-03-09T14:15:30.750 INFO:teuthology.orchestra.run.smithi053.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-09T14:15:30.756 INFO:teuthology.orchestra.run.smithi053.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-09T14:15:30.823 INFO:teuthology.orchestra.run.smithi053.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-09T14:15:30.828 INFO:teuthology.orchestra.run.smithi053.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-09T14:15:30.999 INFO:teuthology.orchestra.run.smithi053.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-09T14:15:30.999 INFO:teuthology.orchestra.run.smithi053.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-09T14:15:30.999 INFO:teuthology.orchestra.run.smithi053.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-09T14:15:31.045 INFO:teuthology.orchestra.run.smithi053.stdout:done 2024-03-09T14:15:31.485 DEBUG:teuthology.orchestra.run.smithi053:> dpkg -s linux-image-generic 2024-03-09T14:15:31.507 INFO:teuthology.orchestra.run.smithi053.stdout:Package: linux-image-generic 2024-03-09T14:15:31.507 INFO:teuthology.orchestra.run.smithi053.stdout:Status: install ok installed 2024-03-09T14:15:31.507 INFO:teuthology.orchestra.run.smithi053.stdout:Priority: optional 2024-03-09T14:15:31.507 INFO:teuthology.orchestra.run.smithi053.stdout:Section: kernel 2024-03-09T14:15:31.508 INFO:teuthology.orchestra.run.smithi053.stdout:Installed-Size: 21 2024-03-09T14:15:31.508 INFO:teuthology.orchestra.run.smithi053.stdout:Maintainer: Ubuntu Kernel Team 2024-03-09T14:15:31.508 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: amd64 2024-03-09T14:15:31.508 INFO:teuthology.orchestra.run.smithi053.stdout:Source: linux-meta 2024-03-09T14:15:31.508 INFO:teuthology.orchestra.run.smithi053.stdout:Version: 5.15.0.100.97 2024-03-09T14:15:31.508 INFO:teuthology.orchestra.run.smithi053.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-09T14:15:31.508 INFO:teuthology.orchestra.run.smithi053.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-09T14:15:31.508 INFO:teuthology.orchestra.run.smithi053.stdout:Recommends: thermald 2024-03-09T14:15:31.508 INFO:teuthology.orchestra.run.smithi053.stdout:Description: Generic Linux kernel image 2024-03-09T14:15:31.508 INFO:teuthology.orchestra.run.smithi053.stdout: This package will always depend on the latest generic kernel image 2024-03-09T14:15:31.508 INFO:teuthology.orchestra.run.smithi053.stdout: available. 2024-03-09T14:15:31.509 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-100-generic 2024-03-09T14:15:31.509 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-09T14:15:31.509 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-09T14:15:31.509 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-09T14:15:31.509 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi053.front.sepia.ceph.com, path=None, version=distro) 2024-03-09T14:15:31.509 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-get clean 2024-03-09T14:15:31.704 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-get update 2024-03-09T14:15:31.831 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-09T14:15:31.831 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-09T14:15:31.843 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-09T14:15:31.863 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-09T14:15:33.166 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2024-03-09T14:15:33.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-09T14:15:33.246 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2024-03-09T14:15:33.444 INFO:teuthology.orchestra.run.smithi053.stdout:Building dependency tree... 2024-03-09T14:15:33.445 INFO:teuthology.orchestra.run.smithi053.stdout:Reading state information... 2024-03-09T14:15:33.628 INFO:teuthology.orchestra.run.smithi053.stdout:linux-image-generic is already the newest version (5.15.0.100.97). 2024-03-09T14:15:33.628 INFO:teuthology.orchestra.run.smithi053.stdout:The following packages were automatically installed and are no longer required: 2024-03-09T14:15:33.628 INFO:teuthology.orchestra.run.smithi053.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-09T14:15:33.628 INFO:teuthology.orchestra.run.smithi053.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-09T14:15:33.628 INFO:teuthology.orchestra.run.smithi053.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-09T14:15:33.628 INFO:teuthology.orchestra.run.smithi053.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-09T14:15:33.629 INFO:teuthology.orchestra.run.smithi053.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-09T14:15:33.629 INFO:teuthology.orchestra.run.smithi053.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-09T14:15:33.670 INFO:teuthology.orchestra.run.smithi053.stdout:0 upgraded, 0 newly installed, 0 to remove and 288 not upgraded. 2024-03-09T14:15:33.672 DEBUG:teuthology.orchestra.run.smithi053:> dpkg -s linux-image-generic 2024-03-09T14:15:33.688 INFO:teuthology.orchestra.run.smithi053.stdout:Package: linux-image-generic 2024-03-09T14:15:33.688 INFO:teuthology.orchestra.run.smithi053.stdout:Status: install ok installed 2024-03-09T14:15:33.688 INFO:teuthology.orchestra.run.smithi053.stdout:Priority: optional 2024-03-09T14:15:33.688 INFO:teuthology.orchestra.run.smithi053.stdout:Section: kernel 2024-03-09T14:15:33.688 INFO:teuthology.orchestra.run.smithi053.stdout:Installed-Size: 21 2024-03-09T14:15:33.688 INFO:teuthology.orchestra.run.smithi053.stdout:Maintainer: Ubuntu Kernel Team 2024-03-09T14:15:33.688 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: amd64 2024-03-09T14:15:33.689 INFO:teuthology.orchestra.run.smithi053.stdout:Source: linux-meta 2024-03-09T14:15:33.689 INFO:teuthology.orchestra.run.smithi053.stdout:Version: 5.15.0.100.97 2024-03-09T14:15:33.689 INFO:teuthology.orchestra.run.smithi053.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-09T14:15:33.689 INFO:teuthology.orchestra.run.smithi053.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-09T14:15:33.689 INFO:teuthology.orchestra.run.smithi053.stdout:Recommends: thermald 2024-03-09T14:15:33.689 INFO:teuthology.orchestra.run.smithi053.stdout:Description: Generic Linux kernel image 2024-03-09T14:15:33.689 INFO:teuthology.orchestra.run.smithi053.stdout: This package will always depend on the latest generic kernel image 2024-03-09T14:15:33.689 INFO:teuthology.orchestra.run.smithi053.stdout: available. 2024-03-09T14:15:33.689 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-03-09T14:15:33.733 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.1h0FvDqxeK 2024-03-09T14:15:33.733 DEBUG:teuthology.orchestra.run.smithi053:> sudo cp /boot/grub/grub.cfg /tmp/tmp.1h0FvDqxeK 2024-03-09T14:15:33.798 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.1h0FvDqxeK 2024-03-09T14:15:33.911 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.1h0FvDqxeK is 10KB 2024-03-09T14:15:33.927 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.1h0FvDqxeK 2024-03-09T14:15:33.933 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-09T14:15:33.989 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:15:33.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-09T14:15:33.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-09T14:15:34.061 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-100-generic 2024-03-09T14:15:34.061 DEBUG:teuthology.orchestra.run.smithi053:> sudo update-grub 2024-03-09T14:15:35.232 INFO:teuthology.orchestra.run.smithi053.stderr:Sourcing file `/etc/default/grub' 2024-03-09T14:15:35.233 INFO:teuthology.orchestra.run.smithi053.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-09T14:15:35.236 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2024-03-09T14:15:35.417 INFO:teuthology.orchestra.run.smithi053.stderr:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-09T14:15:35.424 INFO:teuthology.orchestra.run.smithi053.stderr:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-09T14:15:35.650 INFO:teuthology.orchestra.run.smithi053.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-09T14:15:35.655 INFO:teuthology.orchestra.run.smithi053.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-09T14:15:35.699 INFO:teuthology.orchestra.run.smithi053.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-09T14:15:35.705 INFO:teuthology.orchestra.run.smithi053.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-09T14:15:35.834 INFO:teuthology.orchestra.run.smithi053.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-09T14:15:35.834 INFO:teuthology.orchestra.run.smithi053.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-09T14:15:35.834 INFO:teuthology.orchestra.run.smithi053.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-09T14:15:35.863 INFO:teuthology.orchestra.run.smithi053.stderr:done 2024-03-09T14:15:35.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo shutdown -r now 2024-03-09T14:16:00.574 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-09T14:16:00.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-09T14:16:00.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T14:16:05.870 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-09T14:16:05.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-09T14:16:05.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T14:16:18.978 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-03-09T14:16:24.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-09T14:16:27.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-09T14:16:27.986 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T14:16:33.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-09T14:16:33.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T14:16:51.743 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-09T14:17:03.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-09T14:17:03.756 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T14:17:06.441 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-09T14:17:21.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-09T14:17:21.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T14:17:21.741 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-03-09T14:17:22.274 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-09T14:17:22.274 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-100-generic"... 2024-03-09T14:17:22.274 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-03-09T14:17:22.323 INFO:teuthology.orchestra.run.smithi053.stdout:5.15.0-100-generic 2024-03-09T14:17:22.324 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-100-generic vs 5.15.0-100-generic 2024-03-09T14:17:22.324 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-09T14:17:22.324 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-09T14:17:23.325 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-09T14:17:23.326 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-09T14:17:23.406 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2024-03-09T14:17:23.438 DEBUG:teuthology.parallel:result is None 2024-03-09T14:17:27.988 DEBUG:teuthology.orchestra.remote:timed out 2024-03-09T14:17:39.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-09T14:17:39.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T14:17:40.244 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-03-09T14:17:40.737 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-09T14:17:40.738 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-100-generic"... 2024-03-09T14:17:40.738 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-03-09T14:17:40.782 INFO:teuthology.orchestra.run.smithi001.stdout:5.15.0-100-generic 2024-03-09T14:17:40.782 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-100-generic vs 5.15.0-100-generic 2024-03-09T14:17:40.783 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-09T14:17:40.783 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-09T14:17:41.784 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-09T14:17:41.784 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-09T14:17:41.852 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-03-09T14:17:41.876 DEBUG:teuthology.parallel:result is None 2024-03-09T14:17:41.876 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-09T14:17:41.884 INFO:teuthology.task.internal:Creating test directory... 2024-03-09T14:17:41.884 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-09T14:17:41.887 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-09T14:17:41.893 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-09T14:17:41.936 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-09T14:17:41.986 INFO:teuthology.task.internal:Creating archive directory... 2024-03-09T14:17:41.986 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-09T14:17:41.989 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-09T14:17:42.010 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-09T14:17:42.038 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-09T14:17:42.039 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-09T14:17:42.041 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-09T14:17:42.062 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:17:42.064 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:17:42.070 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:17:42.073 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:17:42.074 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-09T14:17:42.081 INFO:teuthology.task.internal:Configuring sudo... 2024-03-09T14:17:42.081 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-09T14:17:42.115 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-09T14:17:42.132 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-09T14:17:42.141 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-09T14:17:42.141 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-09T14:17:42.171 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-09T14:17:42.179 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-09T14:17:42.223 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-09T14:17:42.271 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:17:42.271 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-09T14:17:42.337 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-09T14:17:42.343 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-09T14:17:42.392 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:17:42.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-09T14:17:42.469 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-03-09T14:17:42.472 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-03-09T14:17:42.548 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-09T14:17:42.556 INFO:teuthology.task.internal:Starting timer... 2024-03-09T14:17:42.556 INFO:teuthology.run_tasks:Running task pcp... 2024-03-09T14:17:42.615 INFO:teuthology.run_tasks:Running task selinux... 2024-03-09T14:17:42.647 DEBUG:teuthology.task.selinux:Excluding smithi001: OS 'ubuntu' does not support SELinux 2024-03-09T14:17:42.647 DEBUG:teuthology.task.selinux:Excluding smithi053: OS 'ubuntu' does not support SELinux 2024-03-09T14:17:42.647 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-09T14:17:42.648 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-09T14:17:42.648 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-09T14:17:42.648 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-09T14:17:42.690 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-09T14:17:42.706 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-09T14:17:42.708 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi053.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-09T14:24:50.066 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi053.front.sepia.ceph.com')] 2024-03-09T14:24:50.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-09T14:24:50.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T14:24:50.161 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-03-09T14:24:50.238 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-09T14:24:50.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-09T14:24:50.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T14:24:50.320 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-03-09T14:24:50.403 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-09T14:24:50.403 INFO:teuthology.run_tasks:Running task clock... 2024-03-09T14:24:50.415 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-09T14:24:50.416 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-09T14:24:50.416 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-09T14:24:50.419 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-09T14:24:50.419 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-09T14:24:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-09T14:24:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: Command line: ntpd -gq 2024-03-09T14:24:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: ---------------------------------------------------- 2024-03-09T14:24:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: ntp-4 is maintained by Network Time Foundation, 2024-03-09T14:24:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-09T14:24:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: corporation. Support and training for ntp-4 are 2024-03-09T14:24:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: available at https://www.nwtime.org/support 2024-03-09T14:24:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: ---------------------------------------------------- 2024-03-09T14:24:50.447 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: proto: precision = 0.066 usec (-24) 2024-03-09T14:24:50.447 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: basedate set to 2022-02-04 2024-03-09T14:24:50.447 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: gps base set to 2022-02-06 (week 2196) 2024-03-09T14:24:50.447 INFO:teuthology.orchestra.run.smithi001.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-09T14:24:50.448 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-09T14:24:50.448 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: restrict ::: KOD does nothing without LIMITED. 2024-03-09T14:24:50.448 INFO:teuthology.orchestra.run.smithi001.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-09T14:24:50.449 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: Listen and drop on 0 v6wildcard [::]:123 2024-03-09T14:24:50.449 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-09T14:24:50.449 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: Listen normally on 2 lo 127.0.0.1:123 2024-03-09T14:24:50.450 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: Listen normally on 3 ens1f0 172.21.15.1:123 2024-03-09T14:24:50.450 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: Listen normally on 4 lo [::1]:123 2024-03-09T14:24:50.450 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15e8%4]:123 2024-03-09T14:24:50.450 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:50 ntpd[17045]: Listening on routing socket on fd #22 for interface updates 2024-03-09T14:24:50.477 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-09T14:24:50.477 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: Command line: ntpd -gq 2024-03-09T14:24:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: ---------------------------------------------------- 2024-03-09T14:24:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: ntp-4 is maintained by Network Time Foundation, 2024-03-09T14:24:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-09T14:24:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: corporation. Support and training for ntp-4 are 2024-03-09T14:24:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: available at https://www.nwtime.org/support 2024-03-09T14:24:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: ---------------------------------------------------- 2024-03-09T14:24:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: proto: precision = 0.068 usec (-24) 2024-03-09T14:24:50.479 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: basedate set to 2022-02-04 2024-03-09T14:24:50.479 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: gps base set to 2022-02-06 (week 2196) 2024-03-09T14:24:50.479 INFO:teuthology.orchestra.run.smithi053.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-09T14:24:50.480 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-09T14:24:50.480 INFO:teuthology.orchestra.run.smithi053.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-09T14:24:50.481 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: restrict ::: KOD does nothing without LIMITED. 2024-03-09T14:24:50.481 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: Listen and drop on 0 v6wildcard [::]:123 2024-03-09T14:24:50.481 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-09T14:24:50.481 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: Listen normally on 2 lo 127.0.0.1:123 2024-03-09T14:24:50.482 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: Listen normally on 3 ens1f0 172.21.15.53:123 2024-03-09T14:24:50.482 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: Listen normally on 4 lo [::1]:123 2024-03-09T14:24:50.482 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:3718%4]:123 2024-03-09T14:24:50.482 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:50 ntpd[17043]: Listening on routing socket on fd #22 for interface updates 2024-03-09T14:24:51.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:51.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:51.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:51.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:51.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:51.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:51.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:51.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:51.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:51.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:51.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:51.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:51.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:51.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:51 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:51.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:51.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:51 ntpd[17043]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:53.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:53 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:53.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:53 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:53.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:53 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:53.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:53 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:53.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:53 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:53.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:53 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:53.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:53 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:53.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:53 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:53.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:53 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:53.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:53 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:53.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:53 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:53.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:53 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:53.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:53 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:53.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:53 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:53.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:53 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:53.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:53 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:53.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:53 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:53.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:53 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:53.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:53 ntpd[17043]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:53.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:53 ntpd[17043]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:53.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:53 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:53.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:53 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:53.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:53 ntpd[17043]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:53.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:53 ntpd[17043]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:54.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:54 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:54.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:54 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:54.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:54 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:54.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:54 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:54.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:54 ntpd[17043]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:54.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:54 ntpd[17043]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:55.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:55.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:55.449 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:55.449 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:55.449 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:55.449 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:55 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:55.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:55 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:55.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:55 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:55.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:55 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:55.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:55 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:55.480 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:55 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:55.480 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:55 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:55.480 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:55 ntpd[17043]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:55.480 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:55 ntpd[17043]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:55.480 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:55 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:55.480 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:55 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:55.480 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:55 ntpd[17043]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:55.480 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:55 ntpd[17043]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:56.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:56 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:56.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:56 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:56.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:56 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:56.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:56 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:56.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:56 ntpd[17043]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:56.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:56 ntpd[17043]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:57.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:57 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:57.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:57 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:57.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:57 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:57.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:57 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:57.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:57 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:57.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:57 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:57.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:57 ntpd[17045]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:57.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:57 ntpd[17045]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:57.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:57 ntpd[17045]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:57.447 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:57 ntpd[17045]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: 9 Mar 14:24:57 ntpd[17045]: ntpd: time slew -0.001040 s 2024-03-09T14:24:57.448 INFO:teuthology.orchestra.run.smithi001.stdout:ntpd: time slew -0.001040s 2024-03-09T14:24:57.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:57 ntpd[17045]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-09T14:24:57.448 INFO:teuthology.orchestra.run.smithi001.stderr: 9 Mar 14:24:57 ntpd[17045]: can't open /var/log/ntpstats/loopstats.20240309: Permission denied 2024-03-09T14:24:57.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:57 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:57.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:57 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:57.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:57 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:57.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:57 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:57.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:57 ntpd[17043]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-09T14:24:57.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:57 ntpd[17043]: can't open /var/log/ntpstats/rawstats.20240309: Permission denied 2024-03-09T14:24:57.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:57 ntpd[17043]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-09T14:24:57.478 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:57 ntpd[17043]: can't open /var/log/ntpstats/peerstats.20240309: Permission denied 2024-03-09T14:24:57.479 INFO:teuthology.orchestra.run.smithi053.stdout: 9 Mar 14:24:57 ntpd[17043]: ntpd: time slew -0.000940 s 2024-03-09T14:24:57.479 INFO:teuthology.orchestra.run.smithi053.stdout:ntpd: time slew -0.000940s 2024-03-09T14:24:57.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:57 ntpd[17043]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-09T14:24:57.479 INFO:teuthology.orchestra.run.smithi053.stderr: 9 Mar 14:24:57 ntpd[17043]: can't open /var/log/ntpstats/loopstats.20240309: Permission denied 2024-03-09T14:24:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: remote refid st t when poll reach delay offset jitter 2024-03-09T14:24:57.501 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================== 2024-03-09T14:24:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-09T14:24:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-09T14:24:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-09T14:24:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-09T14:24:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: remote refid st t when poll reach delay offset jitter 2024-03-09T14:24:57.528 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================== 2024-03-09T14:24:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-09T14:24:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-09T14:24:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-09T14:24:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-09T14:24:57.530 INFO:teuthology.run_tasks:Running task install... 2024-03-09T14:24:57.541 DEBUG:teuthology.task.install:project ceph 2024-03-09T14:24:57.541 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c00d9c609fb42a2a25f1c196bed064ca8bb02413'}} 2024-03-09T14:24:57.541 DEBUG:teuthology.task.install:config {'extra_system_packages': {'deb': ['s3cmd', 'maven'], 'rpm': ['s3cmd', 'maven']}, 'flavor': 'default', 'sha1': 'c00d9c609fb42a2a25f1c196bed064ca8bb02413'} 2024-03-09T14:24:57.542 INFO:teuthology.task.install:Using flavor: default 2024-03-09T14:24:57.548 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-09T14:24:57.548 INFO:teuthology.task.install:extra packages: [] 2024-03-09T14:24:57.549 DEBUG:teuthology.orchestra.run.smithi001:> sudo apt-key list | grep Ceph 2024-03-09T14:24:57.549 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-key list | grep Ceph 2024-03-09T14:24:57.610 INFO:teuthology.orchestra.run.smithi001.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-09T14:24:57.634 INFO:teuthology.orchestra.run.smithi053.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-09T14:24:57.640 INFO:teuthology.orchestra.run.smithi001.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-09T14:24:57.640 INFO:teuthology.orchestra.run.smithi001.stdout:uid [ unknown] Ceph.com (release key) 2024-03-09T14:24:57.641 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-09T14:24:57.641 INFO:teuthology.task.install.deb:Installing system (non-project) packages: s3cmd, maven on remote deb x86_64 2024-03-09T14:24:57.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:24:57.661 INFO:teuthology.orchestra.run.smithi053.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-09T14:24:57.661 INFO:teuthology.orchestra.run.smithi053.stdout:uid [ unknown] Ceph.com (release key) 2024-03-09T14:24:57.661 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-09T14:24:57.661 INFO:teuthology.task.install.deb:Installing system (non-project) packages: s3cmd, maven on remote deb x86_64 2024-03-09T14:24:57.662 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:24:57.814 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default/ 2024-03-09T14:24:57.814 INFO:teuthology.task.install.deb:Package version is 19.0.0-2079-gc00d9c60-1jammy 2024-03-09T14:24:57.872 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default/ 2024-03-09T14:24:57.873 INFO:teuthology.task.install.deb:Package version is 19.0.0-2079-gc00d9c60-1jammy 2024-03-09T14:24:58.010 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:24:58.010 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-09T14:24:58.026 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-get update 2024-03-09T14:24:58.065 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:24:58.065 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-09T14:24:58.080 DEBUG:teuthology.orchestra.run.smithi001:> sudo apt-get update 2024-03-09T14:24:58.199 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-09T14:24:58.214 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-09T14:24:58.235 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-09T14:24:58.255 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-09T14:24:58.255 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-09T14:24:58.268 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-09T14:24:58.287 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-09T14:24:58.315 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-09T14:24:58.355 INFO:teuthology.orchestra.run.smithi053.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy InRelease 2024-03-09T14:24:58.403 INFO:teuthology.orchestra.run.smithi053.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-09T14:24:58.410 INFO:teuthology.orchestra.run.smithi001.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy InRelease 2024-03-09T14:24:58.451 INFO:teuthology.orchestra.run.smithi053.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-09T14:24:58.458 INFO:teuthology.orchestra.run.smithi001.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-09T14:24:58.501 INFO:teuthology.orchestra.run.smithi053.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-03-09T14:24:58.507 INFO:teuthology.orchestra.run.smithi001.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-09T14:24:58.555 INFO:teuthology.orchestra.run.smithi053.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-09T14:24:58.560 INFO:teuthology.orchestra.run.smithi001.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-03-09T14:24:58.616 INFO:teuthology.orchestra.run.smithi001.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-09T14:24:58.637 INFO:teuthology.orchestra.run.smithi053.stdout:Fetched 28.9 kB in 0s (67.8 kB/s) 2024-03-09T14:24:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:Fetched 28.9 kB in 0s (60.3 kB/s) 2024-03-09T14:24:59.594 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2024-03-09T14:24:59.610 DEBUG:teuthology.orchestra.run.smithi053:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2079-gc00d9c60-1jammy cephadm=19.0.0-2079-gc00d9c60-1jammy ceph-mds=19.0.0-2079-gc00d9c60-1jammy ceph-mgr=19.0.0-2079-gc00d9c60-1jammy ceph-common=19.0.0-2079-gc00d9c60-1jammy ceph-fuse=19.0.0-2079-gc00d9c60-1jammy ceph-test=19.0.0-2079-gc00d9c60-1jammy ceph-volume=19.0.0-2079-gc00d9c60-1jammy radosgw=19.0.0-2079-gc00d9c60-1jammy python3-rados=19.0.0-2079-gc00d9c60-1jammy python3-rgw=19.0.0-2079-gc00d9c60-1jammy python3-cephfs=19.0.0-2079-gc00d9c60-1jammy python3-rbd=19.0.0-2079-gc00d9c60-1jammy libcephfs2=19.0.0-2079-gc00d9c60-1jammy libcephfs-dev=19.0.0-2079-gc00d9c60-1jammy librados2=19.0.0-2079-gc00d9c60-1jammy librbd1=19.0.0-2079-gc00d9c60-1jammy rbd-fuse=19.0.0-2079-gc00d9c60-1jammy 2024-03-09T14:24:59.673 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2024-03-09T14:24:59.737 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-03-09T14:24:59.753 DEBUG:teuthology.orchestra.run.smithi001:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2079-gc00d9c60-1jammy cephadm=19.0.0-2079-gc00d9c60-1jammy ceph-mds=19.0.0-2079-gc00d9c60-1jammy ceph-mgr=19.0.0-2079-gc00d9c60-1jammy ceph-common=19.0.0-2079-gc00d9c60-1jammy ceph-fuse=19.0.0-2079-gc00d9c60-1jammy ceph-test=19.0.0-2079-gc00d9c60-1jammy ceph-volume=19.0.0-2079-gc00d9c60-1jammy radosgw=19.0.0-2079-gc00d9c60-1jammy python3-rados=19.0.0-2079-gc00d9c60-1jammy python3-rgw=19.0.0-2079-gc00d9c60-1jammy python3-cephfs=19.0.0-2079-gc00d9c60-1jammy python3-rbd=19.0.0-2079-gc00d9c60-1jammy libcephfs2=19.0.0-2079-gc00d9c60-1jammy libcephfs-dev=19.0.0-2079-gc00d9c60-1jammy librados2=19.0.0-2079-gc00d9c60-1jammy librbd1=19.0.0-2079-gc00d9c60-1jammy rbd-fuse=19.0.0-2079-gc00d9c60-1jammy 2024-03-09T14:24:59.815 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-03-09T14:24:59.870 INFO:teuthology.orchestra.run.smithi053.stdout:Building dependency tree... 2024-03-09T14:24:59.870 INFO:teuthology.orchestra.run.smithi053.stdout:Reading state information... 2024-03-09T14:25:00.012 INFO:teuthology.orchestra.run.smithi001.stdout:Building dependency tree... 2024-03-09T14:25:00.012 INFO:teuthology.orchestra.run.smithi001.stdout:Reading state information... 2024-03-09T14:25:00.057 INFO:teuthology.orchestra.run.smithi053.stdout:The following packages were automatically installed and are no longer required: 2024-03-09T14:25:00.057 INFO:teuthology.orchestra.run.smithi053.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-09T14:25:00.057 INFO:teuthology.orchestra.run.smithi053.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-09T14:25:00.058 INFO:teuthology.orchestra.run.smithi053.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-09T14:25:00.058 INFO:teuthology.orchestra.run.smithi053.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-09T14:25:00.058 INFO:teuthology.orchestra.run.smithi053.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-09T14:25:00.059 INFO:teuthology.orchestra.run.smithi053.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-09T14:25:00.059 INFO:teuthology.orchestra.run.smithi053.stdout:The following additional packages will be installed: 2024-03-09T14:25:00.059 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-09T14:25:00.061 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-09T14:25:00.061 INFO:teuthology.orchestra.run.smithi053.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-09T14:25:00.061 INFO:teuthology.orchestra.run.smithi053.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-09T14:25:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-09T14:25:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-09T14:25:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-09T14:25:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-09T14:25:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-09T14:25:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-09T14:25:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-09T14:25:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-09T14:25:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-09T14:25:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-09T14:25:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-09T14:25:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-09T14:25:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-09T14:25:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-09T14:25:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-09T14:25:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-09T14:25:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-09T14:25:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-09T14:25:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-09T14:25:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet zip 2024-03-09T14:25:00.063 INFO:teuthology.orchestra.run.smithi053.stdout:Suggested packages: 2024-03-09T14:25:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-09T14:25:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-09T14:25:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-09T14:25:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-09T14:25:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-09T14:25:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-09T14:25:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-09T14:25:00.064 INFO:teuthology.orchestra.run.smithi053.stdout:Recommended packages: 2024-03-09T14:25:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: btrfs-tools 2024-03-09T14:25:00.169 INFO:teuthology.orchestra.run.smithi053.stdout:The following NEW packages will be installed: 2024-03-09T14:25:00.169 INFO:teuthology.orchestra.run.smithi053.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-09T14:25:00.169 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-09T14:25:00.169 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-09T14:25:00.169 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-09T14:25:00.169 INFO:teuthology.orchestra.run.smithi053.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-09T14:25:00.169 INFO:teuthology.orchestra.run.smithi053.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-09T14:25:00.171 INFO:teuthology.orchestra.run.smithi053.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-09T14:25:00.171 INFO:teuthology.orchestra.run.smithi053.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-09T14:25:00.171 INFO:teuthology.orchestra.run.smithi053.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-09T14:25:00.171 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-09T14:25:00.171 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-09T14:25:00.171 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-09T14:25:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-09T14:25:00.198 INFO:teuthology.orchestra.run.smithi001.stdout:The following packages were automatically installed and are no longer required: 2024-03-09T14:25:00.198 INFO:teuthology.orchestra.run.smithi001.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-09T14:25:00.199 INFO:teuthology.orchestra.run.smithi001.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-09T14:25:00.199 INFO:teuthology.orchestra.run.smithi001.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-09T14:25:00.199 INFO:teuthology.orchestra.run.smithi001.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-09T14:25:00.199 INFO:teuthology.orchestra.run.smithi001.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-09T14:25:00.200 INFO:teuthology.orchestra.run.smithi001.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-09T14:25:00.200 INFO:teuthology.orchestra.run.smithi001.stdout:The following additional packages will be installed: 2024-03-09T14:25:00.200 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-09T14:25:00.200 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-09T14:25:00.200 INFO:teuthology.orchestra.run.smithi001.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-09T14:25:00.200 INFO:teuthology.orchestra.run.smithi001.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-09T14:25:00.201 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-09T14:25:00.201 INFO:teuthology.orchestra.run.smithi001.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-09T14:25:00.201 INFO:teuthology.orchestra.run.smithi001.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-09T14:25:00.201 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-09T14:25:00.201 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-09T14:25:00.201 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-09T14:25:00.201 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-09T14:25:00.202 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet zip 2024-03-09T14:25:00.203 INFO:teuthology.orchestra.run.smithi001.stdout:Suggested packages: 2024-03-09T14:25:00.203 INFO:teuthology.orchestra.run.smithi001.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-09T14:25:00.203 INFO:teuthology.orchestra.run.smithi001.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-09T14:25:00.203 INFO:teuthology.orchestra.run.smithi001.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-09T14:25:00.203 INFO:teuthology.orchestra.run.smithi001.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-09T14:25:00.203 INFO:teuthology.orchestra.run.smithi001.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-09T14:25:00.203 INFO:teuthology.orchestra.run.smithi001.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-09T14:25:00.203 INFO:teuthology.orchestra.run.smithi001.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-09T14:25:00.203 INFO:teuthology.orchestra.run.smithi001.stdout:Recommended packages: 2024-03-09T14:25:00.203 INFO:teuthology.orchestra.run.smithi001.stdout: btrfs-tools 2024-03-09T14:25:00.310 INFO:teuthology.orchestra.run.smithi001.stdout:The following NEW packages will be installed: 2024-03-09T14:25:00.310 INFO:teuthology.orchestra.run.smithi001.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-09T14:25:00.310 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-09T14:25:00.310 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-09T14:25:00.310 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-09T14:25:00.310 INFO:teuthology.orchestra.run.smithi001.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-09T14:25:00.310 INFO:teuthology.orchestra.run.smithi001.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-09T14:25:00.311 INFO:teuthology.orchestra.run.smithi001.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-09T14:25:00.312 INFO:teuthology.orchestra.run.smithi001.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-09T14:25:00.313 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-09T14:25:00.313 INFO:teuthology.orchestra.run.smithi001.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-09T14:25:00.313 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-09T14:25:00.313 INFO:teuthology.orchestra.run.smithi001.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-09T14:25:00.359 INFO:teuthology.orchestra.run.smithi053.stdout:0 upgraded, 118 newly installed, 0 to remove and 275 not upgraded. 2024-03-09T14:25:00.359 INFO:teuthology.orchestra.run.smithi053.stdout:Need to get 178 MB of archives. 2024-03-09T14:25:00.359 INFO:teuthology.orchestra.run.smithi053.stdout:After this operation, 736 MB of additional disk space will be used. 2024-03-09T14:25:00.359 INFO:teuthology.orchestra.run.smithi053.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-09T14:25:00.378 INFO:teuthology.orchestra.run.smithi001.stdout:0 upgraded, 118 newly installed, 0 to remove and 275 not upgraded. 2024-03-09T14:25:00.379 INFO:teuthology.orchestra.run.smithi001.stdout:Need to get 178 MB of archives. 2024-03-09T14:25:00.379 INFO:teuthology.orchestra.run.smithi001.stdout:After this operation, 736 MB of additional disk space will be used. 2024-03-09T14:25:00.379 INFO:teuthology.orchestra.run.smithi001.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-09T14:25:00.478 INFO:teuthology.orchestra.run.smithi053.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2079-gc00d9c60-1jammy [3,545 kB] 2024-03-09T14:25:00.486 INFO:teuthology.orchestra.run.smithi001.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-09T14:25:00.488 INFO:teuthology.orchestra.run.smithi001.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-09T14:25:00.510 INFO:teuthology.orchestra.run.smithi001.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-09T14:25:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-09T14:25:00.598 INFO:teuthology.orchestra.run.smithi001.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-09T14:25:00.614 INFO:teuthology.orchestra.run.smithi001.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-09T14:25:00.620 INFO:teuthology.orchestra.run.smithi001.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-09T14:25:00.621 INFO:teuthology.orchestra.run.smithi001.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-09T14:25:00.622 INFO:teuthology.orchestra.run.smithi001.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-09T14:25:00.635 INFO:teuthology.orchestra.run.smithi001.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-09T14:25:00.639 INFO:teuthology.orchestra.run.smithi001.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-09T14:25:00.643 INFO:teuthology.orchestra.run.smithi001.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-09T14:25:00.644 INFO:teuthology.orchestra.run.smithi001.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-09T14:25:00.664 INFO:teuthology.orchestra.run.smithi001.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-09T14:25:00.667 INFO:teuthology.orchestra.run.smithi001.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-09T14:25:00.670 INFO:teuthology.orchestra.run.smithi001.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-09T14:25:00.672 INFO:teuthology.orchestra.run.smithi001.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-09T14:25:00.673 INFO:teuthology.orchestra.run.smithi001.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-09T14:25:00.674 INFO:teuthology.orchestra.run.smithi001.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-09T14:25:00.675 INFO:teuthology.orchestra.run.smithi001.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-09T14:25:00.685 INFO:teuthology.orchestra.run.smithi001.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-09T14:25:00.685 INFO:teuthology.orchestra.run.smithi001.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-09T14:25:00.686 INFO:teuthology.orchestra.run.smithi001.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-09T14:25:00.706 INFO:teuthology.orchestra.run.smithi001.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-09T14:25:00.707 INFO:teuthology.orchestra.run.smithi001.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-09T14:25:00.707 INFO:teuthology.orchestra.run.smithi001.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-09T14:25:00.711 INFO:teuthology.orchestra.run.smithi001.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-09T14:25:00.711 INFO:teuthology.orchestra.run.smithi001.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-09T14:25:00.712 INFO:teuthology.orchestra.run.smithi001.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-09T14:25:00.713 INFO:teuthology.orchestra.run.smithi001.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-09T14:25:00.728 INFO:teuthology.orchestra.run.smithi001.stdout:Get:32 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2079-gc00d9c60-1jammy [3,545 kB] 2024-03-09T14:25:00.732 INFO:teuthology.orchestra.run.smithi001.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-09T14:25:00.732 INFO:teuthology.orchestra.run.smithi001.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-09T14:25:00.733 INFO:teuthology.orchestra.run.smithi001.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-09T14:25:00.760 INFO:teuthology.orchestra.run.smithi001.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-09T14:25:00.761 INFO:teuthology.orchestra.run.smithi001.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-09T14:25:00.761 INFO:teuthology.orchestra.run.smithi001.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-09T14:25:00.769 INFO:teuthology.orchestra.run.smithi001.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-09T14:25:00.769 INFO:teuthology.orchestra.run.smithi001.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-09T14:25:00.770 INFO:teuthology.orchestra.run.smithi001.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-09T14:25:00.771 INFO:teuthology.orchestra.run.smithi001.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-09T14:25:00.787 INFO:teuthology.orchestra.run.smithi053.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-09T14:25:00.788 INFO:teuthology.orchestra.run.smithi001.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-09T14:25:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-09T14:25:00.791 INFO:teuthology.orchestra.run.smithi001.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-09T14:25:00.800 INFO:teuthology.orchestra.run.smithi053.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-09T14:25:00.808 INFO:teuthology.orchestra.run.smithi001.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-09T14:25:00.809 INFO:teuthology.orchestra.run.smithi001.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-09T14:25:00.813 INFO:teuthology.orchestra.run.smithi001.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-09T14:25:00.813 INFO:teuthology.orchestra.run.smithi001.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-09T14:25:00.814 INFO:teuthology.orchestra.run.smithi001.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-09T14:25:00.818 INFO:teuthology.orchestra.run.smithi001.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-09T14:25:00.882 INFO:teuthology.orchestra.run.smithi053.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-09T14:25:00.883 INFO:teuthology.orchestra.run.smithi001.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-09T14:25:00.884 INFO:teuthology.orchestra.run.smithi001.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-09T14:25:00.885 INFO:teuthology.orchestra.run.smithi001.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-09T14:25:00.885 INFO:teuthology.orchestra.run.smithi001.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-09T14:25:00.886 INFO:teuthology.orchestra.run.smithi001.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-09T14:25:00.905 INFO:teuthology.orchestra.run.smithi001.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-09T14:25:00.906 INFO:teuthology.orchestra.run.smithi001.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-09T14:25:00.907 INFO:teuthology.orchestra.run.smithi001.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-09T14:25:00.922 INFO:teuthology.orchestra.run.smithi001.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-09T14:25:00.922 INFO:teuthology.orchestra.run.smithi001.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-09T14:25:00.927 INFO:teuthology.orchestra.run.smithi001.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-09T14:25:00.928 INFO:teuthology.orchestra.run.smithi001.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-09T14:25:00.928 INFO:teuthology.orchestra.run.smithi001.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-09T14:25:00.928 INFO:teuthology.orchestra.run.smithi001.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-09T14:25:00.929 INFO:teuthology.orchestra.run.smithi001.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-09T14:25:00.947 INFO:teuthology.orchestra.run.smithi001.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-09T14:25:00.951 INFO:teuthology.orchestra.run.smithi001.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-09T14:25:00.952 INFO:teuthology.orchestra.run.smithi001.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-09T14:25:00.954 INFO:teuthology.orchestra.run.smithi001.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-09T14:25:00.954 INFO:teuthology.orchestra.run.smithi001.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-09T14:25:00.968 INFO:teuthology.orchestra.run.smithi001.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-09T14:25:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-09T14:25:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-09T14:25:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-09T14:25:00.976 INFO:teuthology.orchestra.run.smithi001.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-09T14:25:00.988 INFO:teuthology.orchestra.run.smithi001.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-09T14:25:00.992 INFO:teuthology.orchestra.run.smithi001.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-09T14:25:00.992 INFO:teuthology.orchestra.run.smithi001.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-09T14:25:00.993 INFO:teuthology.orchestra.run.smithi001.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-09T14:25:00.993 INFO:teuthology.orchestra.run.smithi001.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-09T14:25:01.009 INFO:teuthology.orchestra.run.smithi001.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-09T14:25:01.010 INFO:teuthology.orchestra.run.smithi001.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-09T14:25:01.018 INFO:teuthology.orchestra.run.smithi001.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-09T14:25:01.018 INFO:teuthology.orchestra.run.smithi001.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-09T14:25:01.018 INFO:teuthology.orchestra.run.smithi001.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-09T14:25:01.032 INFO:teuthology.orchestra.run.smithi001.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-09T14:25:01.033 INFO:teuthology.orchestra.run.smithi001.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-09T14:25:01.112 INFO:teuthology.orchestra.run.smithi053.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-09T14:25:01.123 INFO:teuthology.orchestra.run.smithi053.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-09T14:25:01.153 INFO:teuthology.orchestra.run.smithi053.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-09T14:25:01.162 INFO:teuthology.orchestra.run.smithi053.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-09T14:25:01.163 INFO:teuthology.orchestra.run.smithi053.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-09T14:25:01.164 INFO:teuthology.orchestra.run.smithi053.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-09T14:25:01.186 INFO:teuthology.orchestra.run.smithi053.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-09T14:25:01.247 INFO:teuthology.orchestra.run.smithi053.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2079-gc00d9c60-1jammy [3,220 kB] 2024-03-09T14:25:01.354 INFO:teuthology.orchestra.run.smithi053.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-09T14:25:01.636 INFO:teuthology.orchestra.run.smithi053.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2079-gc00d9c60-1jammy [711 kB] 2024-03-09T14:25:01.706 INFO:teuthology.orchestra.run.smithi053.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2079-gc00d9c60-1jammy [356 kB] 2024-03-09T14:25:01.760 INFO:teuthology.orchestra.run.smithi053.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2079-gc00d9c60-1jammy [32.8 kB] 2024-03-09T14:25:01.761 INFO:teuthology.orchestra.run.smithi053.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2079-gc00d9c60-1jammy [180 kB] 2024-03-09T14:25:01.762 INFO:teuthology.orchestra.run.smithi053.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-09T14:25:01.768 INFO:teuthology.orchestra.run.smithi053.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2079-gc00d9c60-1jammy [64.6 kB] 2024-03-09T14:25:01.772 INFO:teuthology.orchestra.run.smithi053.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2079-gc00d9c60-1jammy [330 kB] 2024-03-09T14:25:01.795 INFO:teuthology.orchestra.run.smithi053.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-09T14:25:01.797 INFO:teuthology.orchestra.run.smithi053.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-09T14:25:01.826 INFO:teuthology.orchestra.run.smithi053.stdout:Get:24 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2079-gc00d9c60-1jammy [6,744 kB] 2024-03-09T14:25:01.877 INFO:teuthology.orchestra.run.smithi053.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-09T14:25:01.928 INFO:teuthology.orchestra.run.smithi053.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-09T14:25:01.950 INFO:teuthology.orchestra.run.smithi053.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-09T14:25:01.957 INFO:teuthology.orchestra.run.smithi053.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-09T14:25:01.958 INFO:teuthology.orchestra.run.smithi053.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-09T14:25:01.970 INFO:teuthology.orchestra.run.smithi053.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-09T14:25:01.970 INFO:teuthology.orchestra.run.smithi053.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-09T14:25:02.339 INFO:teuthology.orchestra.run.smithi053.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-09T14:25:02.422 INFO:teuthology.orchestra.run.smithi053.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-09T14:25:02.432 INFO:teuthology.orchestra.run.smithi053.stdout:Get:34 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2079-gc00d9c60-1jammy [112 kB] 2024-03-09T14:25:02.435 INFO:teuthology.orchestra.run.smithi053.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2079-gc00d9c60-1jammy [466 kB] 2024-03-09T14:25:02.450 INFO:teuthology.orchestra.run.smithi053.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-09T14:25:02.468 INFO:teuthology.orchestra.run.smithi053.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-09T14:25:02.483 INFO:teuthology.orchestra.run.smithi053.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2079-gc00d9c60-1jammy [24.5 MB] 2024-03-09T14:25:02.512 INFO:teuthology.orchestra.run.smithi053.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-09T14:25:02.722 INFO:teuthology.orchestra.run.smithi053.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-09T14:25:02.737 INFO:teuthology.orchestra.run.smithi053.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-09T14:25:02.744 INFO:teuthology.orchestra.run.smithi053.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-09T14:25:02.750 INFO:teuthology.orchestra.run.smithi053.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-09T14:25:02.771 INFO:teuthology.orchestra.run.smithi053.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-09T14:25:02.774 INFO:teuthology.orchestra.run.smithi053.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-09T14:25:03.133 INFO:teuthology.orchestra.run.smithi001.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2079-gc00d9c60-1jammy [3,220 kB] 2024-03-09T14:25:03.262 INFO:teuthology.orchestra.run.smithi053.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-09T14:25:03.580 INFO:teuthology.orchestra.run.smithi053.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-09T14:25:03.617 INFO:teuthology.orchestra.run.smithi053.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-09T14:25:03.628 INFO:teuthology.orchestra.run.smithi053.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-09T14:25:03.833 INFO:teuthology.orchestra.run.smithi053.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-09T14:25:03.833 INFO:teuthology.orchestra.run.smithi053.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-09T14:25:03.839 INFO:teuthology.orchestra.run.smithi053.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-09T14:25:03.845 INFO:teuthology.orchestra.run.smithi053.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-09T14:25:03.861 INFO:teuthology.orchestra.run.smithi053.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-09T14:25:03.882 INFO:teuthology.orchestra.run.smithi053.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-09T14:25:03.887 INFO:teuthology.orchestra.run.smithi053.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-09T14:25:04.108 INFO:teuthology.orchestra.run.smithi053.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-09T14:25:04.372 INFO:teuthology.orchestra.run.smithi053.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-09T14:25:04.529 INFO:teuthology.orchestra.run.smithi053.stdout:Get:59 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2079-gc00d9c60-1jammy [5,088 kB] 2024-03-09T14:25:04.550 INFO:teuthology.orchestra.run.smithi053.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-09T14:25:04.564 INFO:teuthology.orchestra.run.smithi053.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-09T14:25:04.583 INFO:teuthology.orchestra.run.smithi053.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-09T14:25:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2079-gc00d9c60-1jammy [711 kB] 2024-03-09T14:25:04.648 INFO:teuthology.orchestra.run.smithi053.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-09T14:25:04.863 INFO:teuthology.orchestra.run.smithi053.stdout:Get:64 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2079-gc00d9c60-1jammy [243 kB] 2024-03-09T14:25:04.872 INFO:teuthology.orchestra.run.smithi053.stdout:Get:65 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2079-gc00d9c60-1jammy [124 kB] 2024-03-09T14:25:04.878 INFO:teuthology.orchestra.run.smithi053.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2079-gc00d9c60-1jammy [1,522 kB] 2024-03-09T14:25:04.939 INFO:teuthology.orchestra.run.smithi053.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-09T14:25:04.942 INFO:teuthology.orchestra.run.smithi053.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-09T14:25:04.944 INFO:teuthology.orchestra.run.smithi053.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-09T14:25:04.945 INFO:teuthology.orchestra.run.smithi053.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-09T14:25:04.947 INFO:teuthology.orchestra.run.smithi053.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-09T14:25:04.973 INFO:teuthology.orchestra.run.smithi001.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2079-gc00d9c60-1jammy [356 kB] 2024-03-09T14:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2079-gc00d9c60-1jammy [6,249 kB] 2024-03-09T14:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2079-gc00d9c60-1jammy [32.8 kB] 2024-03-09T14:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2079-gc00d9c60-1jammy [180 kB] 2024-03-09T14:25:05.168 INFO:teuthology.orchestra.run.smithi053.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-09T14:25:05.234 INFO:teuthology.orchestra.run.smithi001.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2079-gc00d9c60-1jammy [64.6 kB] 2024-03-09T14:25:05.242 INFO:teuthology.orchestra.run.smithi001.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2079-gc00d9c60-1jammy [330 kB] 2024-03-09T14:25:05.411 INFO:teuthology.orchestra.run.smithi001.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2079-gc00d9c60-1jammy [6,744 kB] 2024-03-09T14:25:05.513 INFO:teuthology.orchestra.run.smithi053.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2079-gc00d9c60-1jammy [22.8 MB] 2024-03-09T14:25:05.597 INFO:teuthology.orchestra.run.smithi053.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-09T14:25:05.603 INFO:teuthology.orchestra.run.smithi053.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-09T14:25:05.893 INFO:teuthology.orchestra.run.smithi053.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-09T14:25:05.894 INFO:teuthology.orchestra.run.smithi053.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-09T14:25:05.897 INFO:teuthology.orchestra.run.smithi053.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-09T14:25:05.899 INFO:teuthology.orchestra.run.smithi053.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-09T14:25:05.901 INFO:teuthology.orchestra.run.smithi053.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-09T14:25:05.902 INFO:teuthology.orchestra.run.smithi053.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-09T14:25:05.903 INFO:teuthology.orchestra.run.smithi053.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-09T14:25:05.905 INFO:teuthology.orchestra.run.smithi053.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-09T14:25:06.086 INFO:teuthology.orchestra.run.smithi053.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-09T14:25:06.500 INFO:teuthology.orchestra.run.smithi053.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-09T14:25:06.546 INFO:teuthology.orchestra.run.smithi053.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-09T14:25:06.582 INFO:teuthology.orchestra.run.smithi053.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-09T14:25:06.678 INFO:teuthology.orchestra.run.smithi053.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-09T14:25:06.718 INFO:teuthology.orchestra.run.smithi053.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-09T14:25:06.729 INFO:teuthology.orchestra.run.smithi053.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-09T14:25:06.821 INFO:teuthology.orchestra.run.smithi053.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-09T14:25:06.823 INFO:teuthology.orchestra.run.smithi053.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2079-gc00d9c60-1jammy [14.0 kB] 2024-03-09T14:25:06.825 INFO:teuthology.orchestra.run.smithi053.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2079-gc00d9c60-1jammy [867 kB] 2024-03-09T14:25:06.841 INFO:teuthology.orchestra.run.smithi053.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-09T14:25:06.843 INFO:teuthology.orchestra.run.smithi053.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-09T14:25:06.853 INFO:teuthology.orchestra.run.smithi053.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2079-gc00d9c60-1jammy [2,191 kB] 2024-03-09T14:25:06.857 INFO:teuthology.orchestra.run.smithi053.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-09T14:25:06.897 INFO:teuthology.orchestra.run.smithi053.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2079-gc00d9c60-1jammy [777 kB] 2024-03-09T14:25:06.935 INFO:teuthology.orchestra.run.smithi053.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2079-gc00d9c60-1jammy [144 kB] 2024-03-09T14:25:06.945 INFO:teuthology.orchestra.run.smithi053.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2079-gc00d9c60-1jammy [1,934 kB] 2024-03-09T14:25:07.022 INFO:teuthology.orchestra.run.smithi053.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-09T14:25:07.042 INFO:teuthology.orchestra.run.smithi053.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2079-gc00d9c60-1jammy [8,625 kB] 2024-03-09T14:25:07.296 INFO:teuthology.orchestra.run.smithi053.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-09T14:25:07.314 INFO:teuthology.orchestra.run.smithi053.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-09T14:25:07.468 INFO:teuthology.orchestra.run.smithi053.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-09T14:25:07.488 INFO:teuthology.orchestra.run.smithi053.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2079-gc00d9c60-1jammy [14.2 kB] 2024-03-09T14:25:07.489 INFO:teuthology.orchestra.run.smithi053.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2079-gc00d9c60-1jammy [50.5 MB] 2024-03-09T14:25:07.496 INFO:teuthology.orchestra.run.smithi053.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-09T14:25:07.656 INFO:teuthology.orchestra.run.smithi053.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-09T14:25:07.658 INFO:teuthology.orchestra.run.smithi053.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-09T14:25:07.661 INFO:teuthology.orchestra.run.smithi053.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-09T14:25:07.683 INFO:teuthology.orchestra.run.smithi053.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-09T14:25:07.687 INFO:teuthology.orchestra.run.smithi053.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-09T14:25:07.794 INFO:teuthology.orchestra.run.smithi001.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2079-gc00d9c60-1jammy [112 kB] 2024-03-09T14:25:07.826 INFO:teuthology.orchestra.run.smithi001.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2079-gc00d9c60-1jammy [466 kB] 2024-03-09T14:25:07.985 INFO:teuthology.orchestra.run.smithi001.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2079-gc00d9c60-1jammy [24.5 MB] 2024-03-09T14:25:10.362 INFO:teuthology.orchestra.run.smithi053.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2079-gc00d9c60-1jammy [121 kB] 2024-03-09T14:25:10.364 INFO:teuthology.orchestra.run.smithi053.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2079-gc00d9c60-1jammy [38.4 kB] 2024-03-09T14:25:10.365 INFO:teuthology.orchestra.run.smithi053.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2079-gc00d9c60-1jammy [13.3 MB] 2024-03-09T14:25:10.866 INFO:teuthology.orchestra.run.smithi053.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2079-gc00d9c60-1jammy [91.5 kB] 2024-03-09T14:25:11.449 INFO:teuthology.orchestra.run.smithi053.stdout:Fetched 178 MB in 11s (16.6 MB/s) 2024-03-09T14:25:11.599 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-09T14:25:11.691 INFO:teuthology.orchestra.run.smithi053.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-09T14:25:11.696 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-09T14:25:11.765 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-09T14:25:12.027 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-09T14:25:12.043 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-09T14:25:12.076 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-09T14:25:12.438 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-09T14:25:12.465 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-09T14:25:12.505 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-09T14:25:12.967 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-09T14:25:12.994 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-09T14:25:13.041 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-09T14:25:13.193 INFO:teuthology.orchestra.run.smithi001.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2079-gc00d9c60-1jammy [5,088 kB] 2024-03-09T14:25:13.403 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-09T14:25:13.430 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-09T14:25:13.470 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-09T14:25:13.700 INFO:teuthology.orchestra.run.smithi001.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2079-gc00d9c60-1jammy [243 kB] 2024-03-09T14:25:13.706 INFO:teuthology.orchestra.run.smithi001.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2079-gc00d9c60-1jammy [124 kB] 2024-03-09T14:25:13.736 INFO:teuthology.orchestra.run.smithi001.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2079-gc00d9c60-1jammy [1,522 kB] 2024-03-09T14:25:13.823 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-09T14:25:13.850 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-09T14:25:13.883 INFO:teuthology.orchestra.run.smithi001.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2079-gc00d9c60-1jammy [6,249 kB] 2024-03-09T14:25:13.889 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-09T14:25:14.218 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-09T14:25:14.245 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-09T14:25:14.276 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-09T14:25:14.488 INFO:teuthology.orchestra.run.smithi001.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2079-gc00d9c60-1jammy [22.8 MB] 2024-03-09T14:25:14.546 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package librados2. 2024-03-09T14:25:14.573 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../007-librados2_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:14.612 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking librados2 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:15.057 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package librbd1. 2024-03-09T14:25:15.084 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../008-librbd1_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:15.123 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking librbd1 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:15.602 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libcephfs2. 2024-03-09T14:25:15.629 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:15.668 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libcephfs2 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:15.912 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-rados. 2024-03-09T14:25:15.928 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:15.962 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-rados (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:16.274 INFO:teuthology.orchestra.run.smithi001.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2079-gc00d9c60-1jammy [14.0 kB] 2024-03-09T14:25:16.275 INFO:teuthology.orchestra.run.smithi001.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2079-gc00d9c60-1jammy [867 kB] 2024-03-09T14:25:16.291 INFO:teuthology.orchestra.run.smithi001.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2079-gc00d9c60-1jammy [2,191 kB] 2024-03-09T14:25:16.298 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-09T14:25:16.325 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:25:16.365 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-ceph-argparse (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:16.415 INFO:teuthology.orchestra.run.smithi001.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2079-gc00d9c60-1jammy [777 kB] 2024-03-09T14:25:16.455 INFO:teuthology.orchestra.run.smithi001.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2079-gc00d9c60-1jammy [144 kB] 2024-03-09T14:25:16.458 INFO:teuthology.orchestra.run.smithi001.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2079-gc00d9c60-1jammy [1,934 kB] 2024-03-09T14:25:16.566 INFO:teuthology.orchestra.run.smithi001.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2079-gc00d9c60-1jammy [8,625 kB] 2024-03-09T14:25:16.601 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cephfs. 2024-03-09T14:25:16.628 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:16.668 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cephfs (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:16.987 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-09T14:25:17.014 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:25:17.030 INFO:teuthology.orchestra.run.smithi001.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2079-gc00d9c60-1jammy [14.2 kB] 2024-03-09T14:25:17.030 INFO:teuthology.orchestra.run.smithi001.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2079-gc00d9c60-1jammy [50.5 MB] 2024-03-09T14:25:17.054 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-ceph-common (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:17.357 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-09T14:25:17.384 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-09T14:25:17.423 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-09T14:25:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-prettytable. 2024-03-09T14:25:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-09T14:25:18.035 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-09T14:25:18.296 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-rbd. 2024-03-09T14:25:18.312 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:18.354 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-rbd (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:18.707 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-09T14:25:18.734 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-09T14:25:18.774 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-09T14:25:19.027 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-09T14:25:19.043 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-09T14:25:19.077 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-09T14:25:19.388 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-09T14:25:19.416 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-09T14:25:19.455 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-09T14:25:19.783 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package lua5.1. 2024-03-09T14:25:19.811 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-09T14:25:19.850 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-09T14:25:20.145 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package lua-any. 2024-03-09T14:25:20.172 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-09T14:25:20.203 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-09T14:25:20.431 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package zip. 2024-03-09T14:25:20.446 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-09T14:25:20.489 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking zip (3.0-12build2) ... 2024-03-09T14:25:20.817 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package unzip. 2024-03-09T14:25:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-09T14:25:20.884 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-09T14:25:21.254 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package luarocks. 2024-03-09T14:25:21.270 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-09T14:25:21.304 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-09T14:25:21.621 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package librgw2. 2024-03-09T14:25:21.637 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../025-librgw2_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:21.679 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking librgw2 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:22.232 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-rgw. 2024-03-09T14:25:22.258 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:22.298 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-rgw (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:22.635 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-09T14:25:22.661 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-09T14:25:22.718 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-09T14:25:23.038 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libradosstriper1. 2024-03-09T14:25:23.065 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:23.104 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libradosstriper1 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:23.541 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-common. 2024-03-09T14:25:23.568 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:23.607 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-common (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:24.893 INFO:teuthology.orchestra.run.smithi001.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2079-gc00d9c60-1jammy [121 kB] 2024-03-09T14:25:24.895 INFO:teuthology.orchestra.run.smithi001.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2079-gc00d9c60-1jammy [38.4 kB] 2024-03-09T14:25:24.896 INFO:teuthology.orchestra.run.smithi001.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2079-gc00d9c60-1jammy [13.3 MB] 2024-03-09T14:25:24.955 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-base. 2024-03-09T14:25:24.982 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:25.031 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-base (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:25.685 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-09T14:25:25.711 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-09T14:25:25.751 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-09T14:25:26.029 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cheroot. 2024-03-09T14:25:26.056 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-09T14:25:26.087 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-09T14:25:26.399 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-09T14:25:26.425 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-09T14:25:26.465 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-09T14:25:26.777 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-09T14:25:26.804 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-09T14:25:26.843 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-09T14:25:27.121 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-09T14:25:27.137 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-09T14:25:27.171 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-09T14:25:27.214 INFO:teuthology.orchestra.run.smithi001.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-rgw-account-v3/c00d9c609fb42a2a25f1c196bed064ca8bb02413/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2079-gc00d9c60-1jammy [91.5 kB] 2024-03-09T14:25:27.424 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-tempora. 2024-03-09T14:25:27.451 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-09T14:25:27.491 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-09T14:25:27.779 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-portend. 2024-03-09T14:25:27.802 INFO:teuthology.orchestra.run.smithi001.stdout:Fetched 178 MB in 27s (6,600 kB/s) 2024-03-09T14:25:27.806 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-09T14:25:27.846 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-09T14:25:27.964 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-09T14:25:28.067 INFO:teuthology.orchestra.run.smithi001.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-09T14:25:28.073 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-09T14:25:28.123 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-09T14:25:28.151 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-09T14:25:28.154 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-09T14:25:28.190 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-09T14:25:28.458 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-09T14:25:28.484 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-09T14:25:28.516 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-09T14:25:28.635 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-09T14:25:28.662 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-09T14:25:28.701 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-09T14:25:28.844 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-09T14:25:28.871 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-09T14:25:28.910 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-09T14:25:29.030 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-natsort. 2024-03-09T14:25:29.057 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-09T14:25:29.096 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-09T14:25:29.372 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-09T14:25:29.374 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-logutils. 2024-03-09T14:25:29.399 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-09T14:25:29.402 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-09T14:25:29.441 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-09T14:25:29.447 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-09T14:25:29.727 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-09T14:25:29.754 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-09T14:25:29.793 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-09T14:25:29.817 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-09T14:25:29.844 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-09T14:25:29.875 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-09T14:25:30.105 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-mako. 2024-03-09T14:25:30.132 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-09T14:25:30.171 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-09T14:25:30.228 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-09T14:25:30.256 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-09T14:25:30.295 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-09T14:25:30.475 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-09T14:25:30.502 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-09T14:25:30.533 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-09T14:25:30.631 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-09T14:25:30.658 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-09T14:25:30.698 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-09T14:25:30.827 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-09T14:25:30.855 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-09T14:25:30.902 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-09T14:25:31.034 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package librados2. 2024-03-09T14:25:31.061 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../007-librados2_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:31.100 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking librados2 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:31.197 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-webob. 2024-03-09T14:25:31.224 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-09T14:25:31.263 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-09T14:25:31.578 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package librbd1. 2024-03-09T14:25:31.605 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../008-librbd1_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:31.616 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-waitress. 2024-03-09T14:25:31.644 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-09T14:25:31.662 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking librbd1 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:31.686 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-09T14:25:31.978 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-tempita. 2024-03-09T14:25:32.005 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-09T14:25:32.052 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-09T14:25:32.115 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libcephfs2. 2024-03-09T14:25:32.141 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:32.181 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libcephfs2 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:32.322 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-paste. 2024-03-09T14:25:32.349 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-09T14:25:32.388 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-09T14:25:32.467 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-rados. 2024-03-09T14:25:32.482 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:32.567 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-rados (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:32.725 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-09T14:25:32.741 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-09T14:25:32.783 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-09T14:25:32.945 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-09T14:25:32.971 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:25:33.011 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-ceph-argparse (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:33.044 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-09T14:25:33.060 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-09T14:25:33.111 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-09T14:25:33.337 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-cephfs. 2024-03-09T14:25:33.364 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:33.747 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-cephfs (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:34.233 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-09T14:25:34.260 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:25:34.300 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-ceph-common (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:34.406 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-webtest. 2024-03-09T14:25:34.433 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-09T14:25:34.473 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-09T14:25:34.595 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-09T14:25:34.621 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-09T14:25:34.661 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-09T14:25:34.801 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pecan. 2024-03-09T14:25:34.828 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-09T14:25:34.867 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-09T14:25:34.922 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-prettytable. 2024-03-09T14:25:34.949 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-09T14:25:34.980 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-09T14:25:35.233 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-rbd. 2024-03-09T14:25:35.239 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-certifi. 2024-03-09T14:25:35.249 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:35.266 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-09T14:25:35.291 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-rbd (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:35.305 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-09T14:25:35.616 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-idna. 2024-03-09T14:25:35.643 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-09T14:25:35.653 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-09T14:25:35.679 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-09T14:25:35.682 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-09T14:25:35.719 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-09T14:25:35.988 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-09T14:25:36.004 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-09T14:25:36.019 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-urllib3. 2024-03-09T14:25:36.038 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-09T14:25:36.046 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-09T14:25:36.085 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-09T14:25:36.291 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-09T14:25:36.307 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-09T14:25:36.341 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-09T14:25:36.389 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-requests. 2024-03-09T14:25:36.416 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-09T14:25:36.455 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-09T14:25:36.670 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package lua5.1. 2024-03-09T14:25:36.696 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-09T14:25:36.736 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-09T14:25:36.758 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-09T14:25:36.785 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-09T14:25:36.824 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-09T14:25:37.072 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package lua-any. 2024-03-09T14:25:37.099 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-09T14:25:37.111 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-09T14:25:37.138 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:25:37.139 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-09T14:25:37.178 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:37.358 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package zip. 2024-03-09T14:25:37.374 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-09T14:25:37.408 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking zip (3.0-12build2) ... 2024-03-09T14:25:37.574 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-09T14:25:37.601 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:37.640 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:37.853 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package unzip. 2024-03-09T14:25:37.880 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-09T14:25:37.919 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-09T14:25:37.927 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-09T14:25:37.954 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-09T14:25:37.993 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-09T14:25:38.288 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-openssl. 2024-03-09T14:25:38.315 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package luarocks. 2024-03-09T14:25:38.315 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-09T14:25:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-09T14:25:38.354 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-09T14:25:38.381 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-09T14:25:38.707 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr. 2024-03-09T14:25:38.734 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:38.739 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package librgw2. 2024-03-09T14:25:38.766 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../025-librgw2_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:38.774 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:38.797 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking librgw2 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:39.110 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mon. 2024-03-09T14:25:39.137 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:39.168 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mon (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:39.325 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-rgw. 2024-03-09T14:25:39.351 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:39.391 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-rgw (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:39.727 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-09T14:25:39.754 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-09T14:25:39.797 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-osd. 2024-03-09T14:25:39.802 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-09T14:25:39.824 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:39.864 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-osd (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:40.105 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libradosstriper1. 2024-03-09T14:25:40.132 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:40.172 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libradosstriper1 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:40.517 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-common. 2024-03-09T14:25:40.543 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:40.575 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-common (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:40.925 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph. 2024-03-09T14:25:40.941 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../066-ceph_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:40.983 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:41.303 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-fuse. 2024-03-09T14:25:41.330 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:41.369 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-fuse (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:41.740 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mds. 2024-03-09T14:25:41.767 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:41.805 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mds (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:41.897 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-base. 2024-03-09T14:25:41.924 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:41.973 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-base (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:42.192 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package cephadm. 2024-03-09T14:25:42.217 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../069-cephadm_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:42.258 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking cephadm (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:42.536 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-09T14:25:42.551 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-09T14:25:42.563 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-09T14:25:42.578 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-09T14:25:42.594 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-09T14:25:42.617 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-09T14:25:42.904 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-cheroot. 2024-03-09T14:25:42.930 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-09T14:25:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-09T14:25:42.957 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-09T14:25:42.970 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-09T14:25:42.997 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-09T14:25:43.265 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-09T14:25:43.292 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-09T14:25:43.323 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-09T14:25:43.601 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-09T14:25:43.628 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-09T14:25:43.667 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-09T14:25:43.979 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-09T14:25:44.002 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-babel. 2024-03-09T14:25:44.005 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-09T14:25:44.030 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-09T14:25:44.045 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-09T14:25:44.059 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-09T14:25:44.331 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-tempora. 2024-03-09T14:25:44.354 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jinja2. 2024-03-09T14:25:44.358 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-09T14:25:44.382 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-09T14:25:44.398 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-09T14:25:44.412 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-09T14:25:44.676 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-portend. 2024-03-09T14:25:44.703 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-09T14:25:44.734 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-09T14:25:44.774 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-09T14:25:44.801 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:25:44.840 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:45.020 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-09T14:25:45.047 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-09T14:25:45.087 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-09T14:25:45.202 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-09T14:25:45.229 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-09T14:25:45.268 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-09T14:25:45.331 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-09T14:25:45.347 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-09T14:25:45.389 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-09T14:25:45.563 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-routes. 2024-03-09T14:25:45.591 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-09T14:25:45.629 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-09T14:25:45.851 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-natsort. 2024-03-09T14:25:45.879 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-09T14:25:45.918 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-09T14:25:45.991 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-09T14:25:46.018 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:25:46.057 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:46.229 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-logutils. 2024-03-09T14:25:46.257 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-09T14:25:46.296 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-09T14:25:46.469 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-09T14:25:46.486 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-09T14:25:46.527 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-09T14:25:46.574 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-09T14:25:46.591 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-09T14:25:46.624 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-09T14:25:46.902 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-mako. 2024-03-09T14:25:46.929 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-09T14:25:46.968 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-09T14:25:46.981 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-joblib. 2024-03-09T14:25:47.008 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-09T14:25:47.039 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-09T14:25:47.313 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-09T14:25:47.340 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-09T14:25:47.379 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-09T14:25:47.409 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-09T14:25:47.436 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-09T14:25:47.475 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-09T14:25:47.682 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-09T14:25:47.709 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-09T14:25:47.749 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-09T14:25:47.761 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-sklearn. 2024-03-09T14:25:47.789 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-09T14:25:47.820 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-09T14:25:48.176 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-webob. 2024-03-09T14:25:48.203 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-09T14:25:48.235 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-09T14:25:48.548 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-waitress. 2024-03-09T14:25:48.553 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-09T14:25:48.564 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-09T14:25:48.581 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:25:48.602 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-09T14:25:48.705 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:48.893 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-tempita. 2024-03-09T14:25:48.919 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-09T14:25:48.959 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-09T14:25:49.262 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-paste. 2024-03-09T14:25:49.289 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-09T14:25:49.328 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-09T14:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-09T14:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-09T14:25:49.732 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-09T14:25:49.817 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cachetools. 2024-03-09T14:25:49.845 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-09T14:25:49.875 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-09T14:25:50.009 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-09T14:25:50.026 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-09T14:25:50.059 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-09T14:25:50.178 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-09T14:25:50.206 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-09T14:25:50.245 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-09T14:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-webtest. 2024-03-09T14:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-09T14:25:50.429 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-09T14:25:50.573 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-09T14:25:50.601 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-09T14:25:50.640 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-09T14:25:50.765 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pecan. 2024-03-09T14:25:50.793 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-09T14:25:50.832 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-09T14:25:50.951 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-rsa. 2024-03-09T14:25:50.979 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-09T14:25:51.017 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-09T14:25:51.195 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-certifi. 2024-03-09T14:25:51.222 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-09T14:25:51.261 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-09T14:25:51.338 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-google-auth. 2024-03-09T14:25:51.366 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-09T14:25:51.404 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-09T14:25:51.548 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-idna. 2024-03-09T14:25:51.575 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-09T14:25:51.606 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-09T14:25:51.724 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-09T14:25:51.752 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-09T14:25:51.790 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-09T14:25:51.900 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-urllib3. 2024-03-09T14:25:51.927 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-09T14:25:51.967 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-09T14:25:52.068 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-websocket. 2024-03-09T14:25:52.085 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-09T14:25:52.118 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-09T14:25:52.303 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-requests. 2024-03-09T14:25:52.330 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-09T14:25:52.370 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-09T14:25:52.422 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-09T14:25:52.450 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-09T14:25:52.488 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-09T14:25:52.665 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-09T14:25:52.691 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-09T14:25:52.722 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-09T14:25:53.018 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-09T14:25:53.044 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:25:53.084 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:53.177 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-09T14:25:53.206 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:25:53.243 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-09T14:25:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:53.530 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:53.563 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-09T14:25:53.591 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-09T14:25:53.629 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-09T14:25:53.834 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-09T14:25:53.860 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-09T14:25:53.900 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-09T14:25:53.974 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-09T14:25:54.002 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-09T14:25:54.040 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-09T14:25:54.161 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-openssl. 2024-03-09T14:25:54.188 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-09T14:25:54.219 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-09T14:25:54.268 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package jq. 2024-03-09T14:25:54.285 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-09T14:25:54.318 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-09T14:25:54.547 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mgr. 2024-03-09T14:25:54.554 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package socat. 2024-03-09T14:25:54.574 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:54.582 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-09T14:25:54.613 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mgr (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:54.620 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-09T14:25:54.965 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package xmlstarlet. 2024-03-09T14:25:54.991 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-09T14:25:55.032 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-09T14:25:55.167 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mon. 2024-03-09T14:25:55.194 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:55.233 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mon (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:55.370 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-test. 2024-03-09T14:25:55.398 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:55.436 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-test (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:55.779 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-osd. 2024-03-09T14:25:55.806 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:55.836 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-osd (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:57.007 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph. 2024-03-09T14:25:57.023 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../066-ceph_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:57.065 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:57.384 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-fuse. 2024-03-09T14:25:57.411 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:57.450 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-fuse (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:57.782 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-volume. 2024-03-09T14:25:57.810 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:25:57.820 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mds. 2024-03-09T14:25:57.847 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:57.848 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-volume (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:57.887 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mds (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:58.152 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-09T14:25:58.180 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:58.219 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libcephfs-dev (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:58.382 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package cephadm. 2024-03-09T14:25:58.398 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../069-cephadm_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:25:58.440 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking cephadm (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:25:58.538 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-09T14:25:58.566 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-09T14:25:58.605 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-09T14:25:58.726 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-09T14:25:58.753 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-09T14:25:58.784 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-09T14:25:58.968 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-09T14:25:58.997 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-09T14:25:59.043 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-09T14:25:59.092 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-09T14:25:59.095 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-09T14:25:59.137 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-09T14:25:59.463 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package nvme-cli. 2024-03-09T14:25:59.492 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-09T14:25:59.530 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-09T14:25:59.952 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package pkg-config. 2024-03-09T14:25:59.979 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-09T14:26:00.018 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-09T14:26:00.142 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-babel. 2024-03-09T14:26:00.170 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-09T14:26:00.208 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-09T14:26:00.279 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-09T14:26:00.307 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-09T14:26:00.345 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-09T14:26:00.528 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-jinja2. 2024-03-09T14:26:00.555 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-09T14:26:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-09T14:26:00.782 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-09T14:26:00.810 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-09T14:26:00.848 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-09T14:26:00.922 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-09T14:26:00.950 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:26:00.988 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:01.134 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pastescript. 2024-03-09T14:26:01.162 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-09T14:26:01.201 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-09T14:26:01.308 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-09T14:26:01.336 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-09T14:26:01.375 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-09T14:26:01.504 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pluggy. 2024-03-09T14:26:01.532 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-09T14:26:01.570 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-09T14:26:01.653 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-routes. 2024-03-09T14:26:01.680 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-09T14:26:01.719 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-09T14:26:01.857 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-psutil. 2024-03-09T14:26:01.885 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-09T14:26:01.924 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-09T14:26:01.997 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-09T14:26:02.025 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:26:02.055 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:02.260 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-py. 2024-03-09T14:26:02.288 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-09T14:26:02.326 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-09T14:26:02.476 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-09T14:26:02.492 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-09T14:26:02.534 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-09T14:26:02.680 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pygments. 2024-03-09T14:26:02.708 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-09T14:26:02.746 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-09T14:26:03.046 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-joblib. 2024-03-09T14:26:03.073 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-09T14:26:03.112 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-09T14:26:03.200 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-09T14:26:03.228 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-09T14:26:03.266 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-09T14:26:03.482 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-09T14:26:03.510 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-09T14:26:03.536 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-toml. 2024-03-09T14:26:03.553 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-09T14:26:03.586 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-09T14:26:03.657 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-09T14:26:03.856 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pytest. 2024-03-09T14:26:03.884 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-09T14:26:03.922 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-09T14:26:03.960 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-sklearn. 2024-03-09T14:26:03.987 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-09T14:26:04.026 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-09T14:26:04.225 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-simplejson. 2024-03-09T14:26:04.250 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-09T14:26:04.292 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-09T14:26:04.544 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-09T14:26:04.561 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-09T14:26:04.594 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-09T14:26:04.718 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-09T14:26:04.746 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:26:04.870 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:05.658 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package radosgw. 2024-03-09T14:26:05.687 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../116-radosgw_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:26:05.725 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking radosgw (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:05.782 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-cachetools. 2024-03-09T14:26:05.809 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-09T14:26:05.840 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-09T14:26:06.135 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-09T14:26:06.162 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-09T14:26:06.201 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-09T14:26:06.529 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-09T14:26:06.544 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package rbd-fuse. 2024-03-09T14:26:06.557 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-09T14:26:06.561 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:26:06.596 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-09T14:26:06.602 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking rbd-fuse (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:06.899 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-09T14:26:06.916 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-rsa. 2024-03-09T14:26:06.943 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-09T14:26:06.982 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-09T14:26:07.166 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-09T14:26:07.261 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-google-auth. 2024-03-09T14:26:07.277 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-09T14:26:07.292 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-09T14:26:07.310 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-09T14:26:07.566 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-09T14:26:07.622 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-09T14:26:07.649 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-09T14:26:07.688 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-09T14:26:07.886 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-09T14:26:08.042 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-websocket. 2024-03-09T14:26:08.069 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-09T14:26:08.108 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-09T14:26:08.306 INFO:teuthology.orchestra.run.smithi053.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-09T14:26:08.402 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up cephadm (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:08.420 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-09T14:26:08.447 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-09T14:26:08.486 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-09T14:26:08.986 INFO:teuthology.orchestra.run.smithi053.stdout:Adding system user cephadm....done 2024-03-09T14:26:09.125 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-09T14:26:09.142 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:26:09.159 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-09T14:26:09.183 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:09.503 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-09T14:26:09.530 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-09T14:26:09.569 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-09T14:26:09.611 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-09T14:26:09.870 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-09T14:26:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-09T14:26:09.925 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-09T14:26:09.963 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-09T14:26:09.996 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-09T14:26:10.191 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package jq. 2024-03-09T14:26:10.208 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-09T14:26:10.241 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-09T14:26:10.256 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-09T14:26:10.511 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package socat. 2024-03-09T14:26:10.538 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-09T14:26:10.557 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-09T14:26:10.577 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-09T14:26:10.659 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-09T14:26:10.880 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package xmlstarlet. 2024-03-09T14:26:10.908 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-09T14:26:10.946 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-09T14:26:10.951 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-09T14:26:11.309 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-test. 2024-03-09T14:26:11.320 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-09T14:26:11.326 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:26:11.367 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-test (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:11.580 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-09T14:26:11.864 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-09T14:26:12.108 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-09T14:26:12.342 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-09T14:26:12.635 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-09T14:26:12.894 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-ceph-argparse (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:13.153 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-09T14:26:13.279 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-09T14:26:13.414 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-09T14:26:13.540 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-09T14:26:13.555 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-volume. 2024-03-09T14:26:13.583 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2079-gc00d9c60-1jammy_all.deb ... 2024-03-09T14:26:13.621 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-volume (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:13.791 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-09T14:26:14.101 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-09T14:26:14.129 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:26:14.137 INFO:teuthology.orchestra.run.smithi053.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-09T14:26:14.167 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libcephfs-dev (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:14.173 INFO:teuthology.orchestra.run.smithi053.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-09T14:26:14.236 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-09T14:26:14.362 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-09T14:26:14.403 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-09T14:26:14.428 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-09T14:26:14.469 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-09T14:26:14.729 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-09T14:26:14.791 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-09T14:26:14.819 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-09T14:26:14.855 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-09T14:26:14.865 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-09T14:26:15.131 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-09T14:26:15.228 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package nvme-cli. 2024-03-09T14:26:15.256 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-09T14:26:15.319 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-09T14:26:15.391 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-09T14:26:15.684 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up zip (3.0-12build2) ... 2024-03-09T14:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package pkg-config. 2024-03-09T14:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-09T14:26:15.785 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-09T14:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-09T14:26:16.035 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-09T14:26:16.052 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-09T14:26:16.085 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-09T14:26:16.453 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-09T14:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-09T14:26:16.533 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-09T14:26:16.571 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-09T14:26:16.713 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-09T14:26:16.839 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-09T14:26:16.874 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pastescript. 2024-03-09T14:26:16.902 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-09T14:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-09T14:26:16.965 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-09T14:26:17.258 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-09T14:26:17.269 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pluggy. 2024-03-09T14:26:17.297 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-09T14:26:17.335 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-09T14:26:17.622 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-psutil. 2024-03-09T14:26:17.625 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-09T14:26:17.649 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-09T14:26:17.688 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-09T14:26:17.871 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-09T14:26:18.033 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-py. 2024-03-09T14:26:18.061 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-09T14:26:18.107 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-09T14:26:18.245 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-09T14:26:18.419 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pygments. 2024-03-09T14:26:18.447 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-09T14:26:18.485 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-09T14:26:18.546 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-09T14:26:18.897 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-09T14:26:18.939 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-09T14:26:18.967 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-09T14:26:19.006 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-09T14:26:19.157 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-09T14:26:19.258 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-ceph-common (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:19.301 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-toml. 2024-03-09T14:26:19.328 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-09T14:26:19.367 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-09T14:26:19.567 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-idna (3.3-1) ... 2024-03-09T14:26:19.670 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pytest. 2024-03-09T14:26:19.698 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-09T14:26:19.736 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-09T14:26:19.893 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-09T14:26:20.106 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-simplejson. 2024-03-09T14:26:20.134 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-09T14:26:20.172 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-09T14:26:20.450 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-09T14:26:20.467 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-09T14:26:20.508 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-09T14:26:21.142 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-09T14:26:21.221 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package radosgw. 2024-03-09T14:26:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../116-radosgw_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:26:21.287 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking radosgw (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:21.359 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-09T14:26:21.493 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-09T14:26:21.778 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-09T14:26:21.904 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-09T14:26:22.030 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-09T14:26:22.115 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package rbd-fuse. 2024-03-09T14:26:22.131 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2079-gc00d9c60-1jammy_amd64.deb ... 2024-03-09T14:26:22.173 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking rbd-fuse (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:22.339 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-09T14:26:22.470 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-09T14:26:22.607 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-09T14:26:22.720 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-09T14:26:22.846 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-09T14:26:22.867 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-09T14:26:22.968 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-09T14:26:23.130 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-09T14:26:23.253 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-09T14:26:23.473 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-09T14:26:23.512 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-09T14:26:23.779 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-09T14:26:23.911 INFO:teuthology.orchestra.run.smithi001.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-09T14:26:23.981 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up cephadm (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:24.047 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-09T14:26:24.307 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-09T14:26:24.523 INFO:teuthology.orchestra.run.smithi001.stdout:Adding system user cephadm....done 2024-03-09T14:26:24.558 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-09T14:26:24.667 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-09T14:26:24.678 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-09T14:26:24.943 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-09T14:26:24.954 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-09T14:26:25.069 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-09T14:26:25.223 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-09T14:26:25.348 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-09T14:26:25.387 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-09T14:26:25.617 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-09T14:26:25.696 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-09T14:26:25.867 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-09T14:26:25.956 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-09T14:26:25.993 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-09T14:26:26.065 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-09T14:26:26.192 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-09T14:26:26.263 INFO:teuthology.orchestra.run.smithi053.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-09T14:26:26.345 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-09T14:26:26.501 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-09T14:26:26.619 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-09T14:26:26.722 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-09T14:26:27.007 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-09T14:26:27.012 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-09T14:26:27.271 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-09T14:26:27.393 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-09T14:26:27.397 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-09T14:26:27.653 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-09T14:26:27.665 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-09T14:26:27.766 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-09T14:26:27.913 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-09T14:26:28.059 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-09T14:26:28.188 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-09T14:26:28.335 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-09T14:26:28.448 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-ceph-argparse (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:28.611 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-09T14:26:28.716 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-09T14:26:28.737 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-09T14:26:28.841 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-09T14:26:28.951 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-09T14:26:29.005 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-09T14:26:29.185 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-09T14:26:29.289 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-09T14:26:29.628 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-09T14:26:29.634 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-09T14:26:30.016 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-09T14:26:30.020 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-09T14:26:30.043 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-09T14:26:30.108 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-09T14:26:30.224 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-09T14:26:30.346 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-09T14:26:30.583 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-09T14:26:30.699 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-09T14:26:30.735 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-09T14:26:30.800 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up librados2 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:30.901 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:31.019 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-09T14:26:31.026 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-09T14:26:31.295 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-09T14:26:31.363 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libcephfs2 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:31.488 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libradosstriper1 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:31.588 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up zip (3.0-12build2) ... 2024-03-09T14:26:31.615 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-09T14:26:31.705 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-09T14:26:31.925 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up librbd1 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:32.034 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-modules-core (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:32.160 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-fuse (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:32.379 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-09T14:26:32.379 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-09T14:26:32.432 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-09T14:26:32.700 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-09T14:26:32.826 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-09T14:26:32.861 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libcephfs-dev (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:32.962 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-09T14:26:32.987 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-rados (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:33.113 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up librgw2 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:33.239 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-09T14:26:33.261 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-09T14:26:33.629 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-09T14:26:33.864 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-09T14:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-09T14:26:34.451 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-rbd (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:34.533 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-09T14:26:34.577 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up rbd-fuse (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:34.703 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-rgw (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:34.829 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cephfs (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:34.885 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-09T14:26:34.955 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-common (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:35.144 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-09T14:26:35.262 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-ceph-common (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:35.423 INFO:teuthology.orchestra.run.smithi053.stdout:Adding group ceph....done 2024-03-09T14:26:35.571 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-idna (3.3-1) ... 2024-03-09T14:26:35.791 INFO:teuthology.orchestra.run.smithi053.stdout:Adding system user ceph....done 2024-03-09T14:26:35.960 INFO:teuthology.orchestra.run.smithi053.stdout:Setting system user ceph properties....done 2024-03-09T14:26:35.978 INFO:teuthology.orchestra.run.smithi053.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-09T14:26:36.022 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-09T14:26:36.119 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-09T14:26:36.426 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-09T14:26:36.890 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-test (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:37.041 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up radosgw (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:37.295 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-09T14:26:37.500 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-09T14:26:37.500 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-09T14:26:37.521 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-09T14:26:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-09T14:26:37.951 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-base (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:38.040 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-09T14:26:38.166 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-09T14:26:38.292 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-09T14:26:38.306 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-09T14:26:38.743 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-09T14:26:38.769 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mds (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:39.020 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-09T14:26:39.024 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-09T14:26:39.024 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-09T14:26:39.262 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-09T14:26:39.371 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-09T14:26:39.513 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:39.681 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-09T14:26:39.730 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-09T14:26:39.730 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-09T14:26:39.949 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-09T14:26:40.189 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-osd (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:40.208 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-09T14:26:40.501 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-09T14:26:40.501 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-09T14:26:40.592 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-09T14:26:40.958 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:40.960 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-09T14:26:41.084 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:41.228 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-09T14:26:41.235 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mon (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:41.329 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-09T14:26:41.605 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-09T14:26:41.638 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-09T14:26:41.638 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-09T14:26:41.731 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-09T14:26:42.032 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-09T14:26:42.064 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-cephadm (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:42.242 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:42.342 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-09T14:26:42.367 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-dashboard (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:42.518 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-volume (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:42.601 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-09T14:26:42.727 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-09T14:26:42.837 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-09T14:26:42.871 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-09T14:26:42.910 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-09T14:26:42.980 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-09T14:26:43.107 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-09T14:26:43.180 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-09T14:26:43.306 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-09T14:26:43.699 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-09T14:26:43.967 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-09T14:26:44.093 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-09T14:26:44.233 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-09T14:26:44.360 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-09T14:26:44.470 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-09T14:26:44.779 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-09T14:26:44.860 INFO:teuthology.orchestra.run.smithi053.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-09T14:26:44.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install s3cmd maven 2024-03-09T14:26:44.922 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2024-03-09T14:26:45.072 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-09T14:26:45.119 INFO:teuthology.orchestra.run.smithi053.stdout:Building dependency tree... 2024-03-09T14:26:45.120 INFO:teuthology.orchestra.run.smithi053.stdout:Reading state information... 2024-03-09T14:26:45.288 INFO:teuthology.orchestra.run.smithi053.stdout:The following packages were automatically installed and are no longer required: 2024-03-09T14:26:45.289 INFO:teuthology.orchestra.run.smithi053.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-09T14:26:45.289 INFO:teuthology.orchestra.run.smithi053.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-09T14:26:45.289 INFO:teuthology.orchestra.run.smithi053.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-09T14:26:45.289 INFO:teuthology.orchestra.run.smithi053.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-09T14:26:45.289 INFO:teuthology.orchestra.run.smithi053.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-09T14:26:45.289 INFO:teuthology.orchestra.run.smithi053.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-09T14:26:45.290 INFO:teuthology.orchestra.run.smithi053.stdout:The following additional packages will be installed: 2024-03-09T14:26:45.290 INFO:teuthology.orchestra.run.smithi053.stdout: libaopalliance-java libapache-pom-java libatinject-jsr330-api-java 2024-03-09T14:26:45.290 INFO:teuthology.orchestra.run.smithi053.stdout: libcdi-api-java libcommons-cli-java libcommons-io-java libcommons-lang3-java 2024-03-09T14:26:45.291 INFO:teuthology.orchestra.run.smithi053.stdout: libcommons-parent-java libgeronimo-annotation-1.3-spec-java 2024-03-09T14:26:45.291 INFO:teuthology.orchestra.run.smithi053.stdout: libgeronimo-interceptor-3.0-spec-java libguava-java libguice-java 2024-03-09T14:26:45.291 INFO:teuthology.orchestra.run.smithi053.stdout: libhawtjni-runtime-java libjansi-java libjansi-native-java libjsr305-java 2024-03-09T14:26:45.291 INFO:teuthology.orchestra.run.smithi053.stdout: libmaven-parent-java libmaven-resolver-java libmaven-shared-utils-java 2024-03-09T14:26:45.291 INFO:teuthology.orchestra.run.smithi053.stdout: libmaven3-core-java libplexus-cipher-java libplexus-classworlds-java 2024-03-09T14:26:45.291 INFO:teuthology.orchestra.run.smithi053.stdout: libplexus-component-annotations-java libplexus-interpolation-java 2024-03-09T14:26:45.291 INFO:teuthology.orchestra.run.smithi053.stdout: libplexus-sec-dispatcher-java libplexus-utils2-java libsisu-inject-java 2024-03-09T14:26:45.291 INFO:teuthology.orchestra.run.smithi053.stdout: libsisu-plexus-java libslf4j-java libwagon-file-java 2024-03-09T14:26:45.291 INFO:teuthology.orchestra.run.smithi053.stdout: libwagon-http-shaded-java libwagon-provider-api-java python3-magic 2024-03-09T14:26:45.292 INFO:teuthology.orchestra.run.smithi053.stdout:Suggested packages: 2024-03-09T14:26:45.292 INFO:teuthology.orchestra.run.smithi053.stdout: libaopalliance-java-doc libatinject-jsr330-api-java-doc libel-api-java 2024-03-09T14:26:45.292 INFO:teuthology.orchestra.run.smithi053.stdout: libcommons-io-java-doc libcommons-lang3-java-doc libasm-java libcglib-java 2024-03-09T14:26:45.292 INFO:teuthology.orchestra.run.smithi053.stdout: libjsr305-java-doc libmaven-shared-utils-java-doc liblogback-java 2024-03-09T14:26:45.292 INFO:teuthology.orchestra.run.smithi053.stdout: libplexus-classworlds-java-doc libplexus-sec-dispatcher-java-doc 2024-03-09T14:26:45.292 INFO:teuthology.orchestra.run.smithi053.stdout: libplexus-utils2-java-doc testng libcommons-logging-java liblog4j1.2-java 2024-03-09T14:26:45.329 INFO:teuthology.orchestra.run.smithi053.stdout:The following NEW packages will be installed: 2024-03-09T14:26:45.329 INFO:teuthology.orchestra.run.smithi053.stdout: libaopalliance-java libapache-pom-java libatinject-jsr330-api-java 2024-03-09T14:26:45.329 INFO:teuthology.orchestra.run.smithi053.stdout: libcdi-api-java libcommons-cli-java libcommons-io-java libcommons-lang3-java 2024-03-09T14:26:45.329 INFO:teuthology.orchestra.run.smithi053.stdout: libcommons-parent-java libgeronimo-annotation-1.3-spec-java 2024-03-09T14:26:45.329 INFO:teuthology.orchestra.run.smithi053.stdout: libgeronimo-interceptor-3.0-spec-java libguava-java libguice-java 2024-03-09T14:26:45.330 INFO:teuthology.orchestra.run.smithi053.stdout: libhawtjni-runtime-java libjansi-java libjansi-native-java libjsr305-java 2024-03-09T14:26:45.330 INFO:teuthology.orchestra.run.smithi053.stdout: libmaven-parent-java libmaven-resolver-java libmaven-shared-utils-java 2024-03-09T14:26:45.330 INFO:teuthology.orchestra.run.smithi053.stdout: libmaven3-core-java libplexus-cipher-java libplexus-classworlds-java 2024-03-09T14:26:45.330 INFO:teuthology.orchestra.run.smithi053.stdout: libplexus-component-annotations-java libplexus-interpolation-java 2024-03-09T14:26:45.330 INFO:teuthology.orchestra.run.smithi053.stdout: libplexus-sec-dispatcher-java libplexus-utils2-java libsisu-inject-java 2024-03-09T14:26:45.330 INFO:teuthology.orchestra.run.smithi053.stdout: libsisu-plexus-java libslf4j-java libwagon-file-java 2024-03-09T14:26:45.330 INFO:teuthology.orchestra.run.smithi053.stdout: libwagon-http-shaded-java libwagon-provider-api-java maven python3-magic 2024-03-09T14:26:45.330 INFO:teuthology.orchestra.run.smithi053.stdout: s3cmd 2024-03-09T14:26:45.356 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-09T14:26:45.482 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-09T14:26:45.511 INFO:teuthology.orchestra.run.smithi053.stdout:0 upgraded, 35 newly installed, 0 to remove and 275 not upgraded. 2024-03-09T14:26:45.511 INFO:teuthology.orchestra.run.smithi053.stdout:Need to get 10.4 MB of archives. 2024-03-09T14:26:45.511 INFO:teuthology.orchestra.run.smithi053.stdout:After this operation, 13.9 MB of additional disk space will be used. 2024-03-09T14:26:45.511 INFO:teuthology.orchestra.run.smithi053.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libapache-pom-java all 18-1 [4,720 B] 2024-03-09T14:26:45.593 INFO:teuthology.orchestra.run.smithi053.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libatinject-jsr330-api-java all 1.0+ds1-5 [5,348 B] 2024-03-09T14:26:45.594 INFO:teuthology.orchestra.run.smithi053.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libgeronimo-interceptor-3.0-spec-java all 1.0.1-4fakesync [8,616 B] 2024-03-09T14:26:45.611 INFO:teuthology.orchestra.run.smithi053.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libcdi-api-java all 1.2-3 [54.3 kB] 2024-03-09T14:26:45.767 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-09T14:26:45.775 INFO:teuthology.orchestra.run.smithi053.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libcommons-cli-java all 1.4-2 [55.8 kB] 2024-03-09T14:26:45.850 INFO:teuthology.orchestra.run.smithi053.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libcommons-parent-java all 43-1 [10.8 kB] 2024-03-09T14:26:45.860 INFO:teuthology.orchestra.run.smithi053.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libcommons-io-java all 2.11.0-2 [297 kB] 2024-03-09T14:26:45.997 INFO:teuthology.orchestra.run.smithi053.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libcommons-lang3-java all 3.11-1 [526 kB] 2024-03-09T14:26:46.043 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-09T14:26:46.090 INFO:teuthology.orchestra.run.smithi053.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libgeronimo-annotation-1.3-spec-java all 1.3-1 [11.2 kB] 2024-03-09T14:26:46.091 INFO:teuthology.orchestra.run.smithi053.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libjsr305-java all 0.1~+svn49-11 [27.0 kB] 2024-03-09T14:26:46.094 INFO:teuthology.orchestra.run.smithi053.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libguava-java all 29.0-6 [2,418 kB] 2024-03-09T14:26:46.396 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-09T14:26:46.403 INFO:teuthology.orchestra.run.smithi053.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libaopalliance-java all 20070526-6 [9,084 B] 2024-03-09T14:26:46.486 INFO:teuthology.orchestra.run.smithi053.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libguice-java all 4.2.3-2 [1,434 kB] 2024-03-09T14:26:46.781 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-09T14:26:47.006 INFO:teuthology.orchestra.run.smithi053.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libhawtjni-runtime-java all 1.17-1 [28.8 kB] 2024-03-09T14:26:47.007 INFO:teuthology.orchestra.run.smithi053.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libjansi-native-java all 1.8-1 [23.8 kB] 2024-03-09T14:26:47.009 INFO:teuthology.orchestra.run.smithi053.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libjansi-java all 1.18-1 [56.8 kB] 2024-03-09T14:26:47.014 INFO:teuthology.orchestra.run.smithi053.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libmaven-parent-java all 31-2 [5,140 B] 2024-03-09T14:26:47.014 INFO:teuthology.orchestra.run.smithi053.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libplexus-utils2-java all 3.3.0-1 [250 kB] 2024-03-09T14:26:47.031 INFO:teuthology.orchestra.run.smithi053.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libwagon-provider-api-java all 3.3.4-1 [48.5 kB] 2024-03-09T14:26:47.034 INFO:teuthology.orchestra.run.smithi053.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libmaven-resolver-java all 1.4.2-3build1 [555 kB] 2024-03-09T14:26:47.066 INFO:teuthology.orchestra.run.smithi053.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libmaven-shared-utils-java all 3.3.0-1 [149 kB] 2024-03-09T14:26:47.072 INFO:teuthology.orchestra.run.smithi053.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libplexus-cipher-java all 1.8-2 [15.1 kB] 2024-03-09T14:26:47.074 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-09T14:26:47.237 INFO:teuthology.orchestra.run.smithi053.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libplexus-classworlds-java all 2.6.0-1 [49.4 kB] 2024-03-09T14:26:47.427 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-09T14:26:47.494 INFO:teuthology.orchestra.run.smithi053.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libplexus-component-annotations-java all 2.1.0-1 [6,564 B] 2024-03-09T14:26:47.501 INFO:teuthology.orchestra.run.smithi053.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libplexus-interpolation-java all 1.26-1 [76.8 kB] 2024-03-09T14:26:47.545 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up librados2 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:47.594 INFO:teuthology.orchestra.run.smithi053.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libplexus-sec-dispatcher-java all 1.4-4 [28.1 kB] 2024-03-09T14:26:47.622 INFO:teuthology.orchestra.run.smithi053.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libslf4j-java all 1.7.32-1 [141 kB] 2024-03-09T14:26:47.654 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:47.701 INFO:teuthology.orchestra.run.smithi053.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libsisu-inject-java all 0.3.4-2 [347 kB] 2024-03-09T14:26:47.763 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-09T14:26:47.801 INFO:teuthology.orchestra.run.smithi053.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libsisu-plexus-java all 0.3.4-3 [181 kB] 2024-03-09T14:26:47.830 INFO:teuthology.orchestra.run.smithi053.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libmaven3-core-java all 3.6.3-5 [1,535 kB] 2024-03-09T14:26:47.956 INFO:teuthology.orchestra.run.smithi053.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libwagon-file-java all 3.3.4-1 [7,928 B] 2024-03-09T14:26:47.957 INFO:teuthology.orchestra.run.smithi053.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libwagon-http-shaded-java all 3.3.4-1 [1,855 kB] 2024-03-09T14:26:48.024 INFO:teuthology.orchestra.run.smithi053.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/universe amd64 maven all 3.6.3-5 [17.8 kB] 2024-03-09T14:26:48.100 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libcephfs2 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:48.189 INFO:teuthology.orchestra.run.smithi053.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-magic all 2:0.4.24-2 [12.6 kB] 2024-03-09T14:26:48.225 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libradosstriper1 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:48.272 INFO:teuthology.orchestra.run.smithi053.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/universe amd64 s3cmd all 2.2.0-1 [120 kB] 2024-03-09T14:26:48.360 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-09T14:26:48.645 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up librbd1 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:48.746 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mgr-modules-core (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:48.847 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-fuse (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:48.935 INFO:teuthology.orchestra.run.smithi053.stdout:Fetched 10.4 MB in 3s (3,295 kB/s) 2024-03-09T14:26:49.023 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libapache-pom-java. 2024-03-09T14:26:49.057 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-09T14:26:49.057 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-09T14:26:49.082 INFO:teuthology.orchestra.run.smithi053.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173568 files and directories currently installed.) 2024-03-09T14:26:49.087 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../00-libapache-pom-java_18-1_all.deb ... 2024-03-09T14:26:49.130 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libapache-pom-java (18-1) ... 2024-03-09T14:26:49.392 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libatinject-jsr330-api-java. 2024-03-09T14:26:49.420 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../01-libatinject-jsr330-api-java_1.0+ds1-5_all.deb ... 2024-03-09T14:26:49.483 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libatinject-jsr330-api-java (1.0+ds1-5) ... 2024-03-09T14:26:49.590 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libcephfs-dev (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:49.725 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-rados (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:49.736 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libgeronimo-interceptor-3.0-spec-java. 2024-03-09T14:26:49.753 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../02-libgeronimo-interceptor-3.0-spec-java_1.0.1-4fakesync_all.deb ... 2024-03-09T14:26:49.794 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libgeronimo-interceptor-3.0-spec-java (1.0.1-4fakesync) ... 2024-03-09T14:26:50.021 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libcdi-api-java. 2024-03-09T14:26:50.039 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../03-libcdi-api-java_1.2-3_all.deb ... 2024-03-09T14:26:50.071 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libcdi-api-java (1.2-3) ... 2024-03-09T14:26:50.252 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up librgw2 (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:50.299 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libcommons-cli-java. 2024-03-09T14:26:50.327 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../04-libcommons-cli-java_1.4-2_all.deb ... 2024-03-09T14:26:50.357 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libcommons-cli-java (1.4-2) ... 2024-03-09T14:26:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-09T14:26:50.601 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libcommons-parent-java. 2024-03-09T14:26:50.630 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../05-libcommons-parent-java_43-1_all.deb ... 2024-03-09T14:26:50.668 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libcommons-parent-java (43-1) ... 2024-03-09T14:26:50.903 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libcommons-io-java. 2024-03-09T14:26:50.920 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../06-libcommons-io-java_2.11.0-2_all.deb ... 2024-03-09T14:26:50.962 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libcommons-io-java (2.11.0-2) ... 2024-03-09T14:26:51.231 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libcommons-lang3-java. 2024-03-09T14:26:51.248 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../07-libcommons-lang3-java_3.11-1_all.deb ... 2024-03-09T14:26:51.289 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libcommons-lang3-java (3.11-1) ... 2024-03-09T14:26:51.550 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libgeronimo-annotation-1.3-spec-java. 2024-03-09T14:26:51.567 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../08-libgeronimo-annotation-1.3-spec-java_1.3-1_all.deb ... 2024-03-09T14:26:51.600 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libgeronimo-annotation-1.3-spec-java (1.3-1) ... 2024-03-09T14:26:51.673 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-rbd (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:51.799 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up rbd-fuse (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:51.836 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libjsr305-java. 2024-03-09T14:26:51.852 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../09-libjsr305-java_0.1~+svn49-11_all.deb ... 2024-03-09T14:26:51.894 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libjsr305-java (0.1~+svn49-11) ... 2024-03-09T14:26:51.925 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-rgw (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:52.050 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-cephfs (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:52.147 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libguava-java. 2024-03-09T14:26:52.175 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../10-libguava-java_29.0-6_all.deb ... 2024-03-09T14:26:52.177 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-common (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:52.213 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libguava-java (29.0-6) ... 2024-03-09T14:26:52.658 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libaopalliance-java. 2024-03-09T14:26:52.674 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../11-libaopalliance-java_20070526-6_all.deb ... 2024-03-09T14:26:52.686 INFO:teuthology.orchestra.run.smithi001.stdout:Adding group ceph....done 2024-03-09T14:26:52.716 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libaopalliance-java (20070526-6) ... 2024-03-09T14:26:52.968 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libguice-java. 2024-03-09T14:26:52.985 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../12-libguice-java_4.2.3-2_all.deb ... 2024-03-09T14:26:53.018 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libguice-java (4.2.3-2) ... 2024-03-09T14:26:53.046 INFO:teuthology.orchestra.run.smithi001.stdout:Adding system user ceph....done 2024-03-09T14:26:53.215 INFO:teuthology.orchestra.run.smithi001.stdout:Setting system user ceph properties....done 2024-03-09T14:26:53.233 INFO:teuthology.orchestra.run.smithi001.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-09T14:26:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-09T14:26:53.397 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libhawtjni-runtime-java. 2024-03-09T14:26:53.414 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../13-libhawtjni-runtime-java_1.17-1_all.deb ... 2024-03-09T14:26:53.472 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libhawtjni-runtime-java (1.17-1) ... 2024-03-09T14:26:53.733 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libjansi-native-java. 2024-03-09T14:26:53.744 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-09T14:26:53.758 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../14-libjansi-native-java_1.8-1_all.deb ... 2024-03-09T14:26:53.799 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libjansi-native-java (1.8-1) ... 2024-03-09T14:26:54.035 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libjansi-java. 2024-03-09T14:26:54.052 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../15-libjansi-java_1.18-1_all.deb ... 2024-03-09T14:26:54.110 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libjansi-java (1.18-1) ... 2024-03-09T14:26:54.278 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-test (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:54.354 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libmaven-parent-java. 2024-03-09T14:26:54.372 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../16-libmaven-parent-java_31-2_all.deb ... 2024-03-09T14:26:54.379 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up radosgw (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:54.404 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libmaven-parent-java (31-2) ... 2024-03-09T14:26:54.624 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libplexus-utils2-java. 2024-03-09T14:26:54.651 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../17-libplexus-utils2-java_3.3.0-1_all.deb ... 2024-03-09T14:26:54.690 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libplexus-utils2-java (3.3.0-1) ... 2024-03-09T14:26:54.832 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-09T14:26:54.832 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-09T14:26:54.968 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libwagon-provider-api-java. 2024-03-09T14:26:54.997 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../18-libwagon-provider-api-java_3.3.4-1_all.deb ... 2024-03-09T14:26:55.034 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libwagon-provider-api-java (3.3.4-1) ... 2024-03-09T14:26:55.295 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libmaven-resolver-java. 2024-03-09T14:26:55.323 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-base (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:55.324 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../19-libmaven-resolver-java_1.4.2-3build1_all.deb ... 2024-03-09T14:26:55.362 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libmaven-resolver-java (1.4.2-3build1) ... 2024-03-09T14:26:55.657 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libmaven-shared-utils-java. 2024-03-09T14:26:55.674 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../20-libmaven-shared-utils-java_3.3.0-1_all.deb ... 2024-03-09T14:26:55.689 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-09T14:26:55.707 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libmaven-shared-utils-java (3.3.0-1) ... 2024-03-09T14:26:55.985 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libplexus-cipher-java. 2024-03-09T14:26:56.013 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../21-libplexus-cipher-java_1.8-2_all.deb ... 2024-03-09T14:26:56.051 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libplexus-cipher-java (1.8-2) ... 2024-03-09T14:26:56.175 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mds (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:56.329 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libplexus-classworlds-java. 2024-03-09T14:26:56.346 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../22-libplexus-classworlds-java_2.6.0-1_all.deb ... 2024-03-09T14:26:56.387 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libplexus-classworlds-java (2.6.0-1) ... 2024-03-09T14:26:56.404 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-09T14:26:56.404 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-09T14:26:56.623 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libplexus-component-annotations-java. 2024-03-09T14:26:56.640 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../23-libplexus-component-annotations-java_2.1.0-1_all.deb ... 2024-03-09T14:26:56.681 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libplexus-component-annotations-java (2.1.0-1) ... 2024-03-09T14:26:56.894 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mgr (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:56.919 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libplexus-interpolation-java. 2024-03-09T14:26:56.936 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../24-libplexus-interpolation-java_1.26-1_all.deb ... 2024-03-09T14:26:56.977 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libplexus-interpolation-java (1.26-1) ... 2024-03-09T14:26:57.115 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-09T14:26:57.115 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-09T14:26:57.247 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libplexus-sec-dispatcher-java. 2024-03-09T14:26:57.275 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../25-libplexus-sec-dispatcher-java_1.4-4_all.deb ... 2024-03-09T14:26:57.313 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libplexus-sec-dispatcher-java (1.4-4) ... 2024-03-09T14:26:57.574 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libslf4j-java. 2024-03-09T14:26:57.591 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../26-libslf4j-java_1.7.32-1_all.deb ... 2024-03-09T14:26:57.612 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-osd (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:57.632 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libslf4j-java (1.7.32-1) ... 2024-03-09T14:26:57.870 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-09T14:26:57.870 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-09T14:26:57.886 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libsisu-inject-java. 2024-03-09T14:26:57.903 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../27-libsisu-inject-java_0.3.4-2_all.deb ... 2024-03-09T14:26:57.936 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libsisu-inject-java (0.3.4-2) ... 2024-03-09T14:26:58.231 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libsisu-plexus-java. 2024-03-09T14:26:58.247 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../28-libsisu-plexus-java_0.3.4-3_all.deb ... 2024-03-09T14:26:58.289 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libsisu-plexus-java (0.3.4-3) ... 2024-03-09T14:26:58.383 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:58.575 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libmaven3-core-java. 2024-03-09T14:26:58.592 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../29-libmaven3-core-java_3.6.3-5_all.deb ... 2024-03-09T14:26:58.601 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:58.633 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libmaven3-core-java (3.6.3-5) ... 2024-03-09T14:26:58.816 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mon (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:59.005 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libwagon-file-java. 2024-03-09T14:26:59.021 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../30-libwagon-file-java_3.3.4-1_all.deb ... 2024-03-09T14:26:59.063 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libwagon-file-java (3.3.4-1) ... 2024-03-09T14:26:59.081 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-09T14:26:59.081 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-09T14:26:59.307 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libwagon-http-shaded-java. 2024-03-09T14:26:59.336 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../31-libwagon-http-shaded-java_3.3.4-1_all.deb ... 2024-03-09T14:26:59.374 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libwagon-http-shaded-java (3.3.4-1) ... 2024-03-09T14:26:59.560 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mgr-cephadm (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:59.711 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:59.818 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package maven. 2024-03-09T14:26:59.837 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mgr-dashboard (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:26:59.846 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../32-maven_3.6.3-5_all.deb ... 2024-03-09T14:26:59.885 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking maven (3.6.3-5) ... 2024-03-09T14:26:59.988 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-volume (19.0.0-2079-gc00d9c60-1jammy) ... 2024-03-09T14:27:00.197 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-magic. 2024-03-09T14:27:00.225 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../33-python3-magic_2%3a0.4.24-2_all.deb ... 2024-03-09T14:27:00.263 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-magic (2:0.4.24-2) ... 2024-03-09T14:27:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-09T14:27:00.508 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package s3cmd. 2024-03-09T14:27:00.521 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../34-s3cmd_2.2.0-1_all.deb ... 2024-03-09T14:27:00.566 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking s3cmd (2.2.0-1) ... 2024-03-09T14:27:00.613 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-09T14:27:00.765 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-09T14:27:00.946 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libslf4j-java (1.7.32-1) ... 2024-03-09T14:27:01.097 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libplexus-utils2-java (3.3.0-1) ... 2024-03-09T14:27:01.249 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libplexus-classworlds-java (2.6.0-1) ... 2024-03-09T14:27:01.374 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libjsr305-java (0.1~+svn49-11) ... 2024-03-09T14:27:01.500 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libaopalliance-java (20070526-6) ... 2024-03-09T14:27:01.514 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-09T14:27:01.626 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libcommons-cli-java (1.4-2) ... 2024-03-09T14:27:01.761 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libplexus-component-annotations-java (2.1.0-1) ... 2024-03-09T14:27:01.895 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libplexus-cipher-java (1.8-2) ... 2024-03-09T14:27:02.046 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libgeronimo-annotation-1.3-spec-java (1.3-1) ... 2024-03-09T14:27:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-09T14:27:02.188 DEBUG:teuthology.orchestra.run.smithi001:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install s3cmd maven 2024-03-09T14:27:02.197 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libgeronimo-interceptor-3.0-spec-java (1.0.1-4fakesync) ... 2024-03-09T14:27:02.252 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-03-09T14:27:02.348 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libapache-pom-java (18-1) ... 2024-03-09T14:27:02.450 INFO:teuthology.orchestra.run.smithi001.stdout:Building dependency tree... 2024-03-09T14:27:02.451 INFO:teuthology.orchestra.run.smithi001.stdout:Reading state information... 2024-03-09T14:27:02.474 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libatinject-jsr330-api-java (1.0+ds1-5) ... 2024-03-09T14:27:02.617 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libplexus-interpolation-java (1.26-1) ... 2024-03-09T14:27:02.634 INFO:teuthology.orchestra.run.smithi001.stdout:The following packages were automatically installed and are no longer required: 2024-03-09T14:27:02.634 INFO:teuthology.orchestra.run.smithi001.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-09T14:27:02.634 INFO:teuthology.orchestra.run.smithi001.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-09T14:27:02.635 INFO:teuthology.orchestra.run.smithi001.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-09T14:27:02.635 INFO:teuthology.orchestra.run.smithi001.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-09T14:27:02.635 INFO:teuthology.orchestra.run.smithi001.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-09T14:27:02.635 INFO:teuthology.orchestra.run.smithi001.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-09T14:27:02.636 INFO:teuthology.orchestra.run.smithi001.stdout:The following additional packages will be installed: 2024-03-09T14:27:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: libaopalliance-java libapache-pom-java libatinject-jsr330-api-java 2024-03-09T14:27:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: libcdi-api-java libcommons-cli-java libcommons-io-java libcommons-lang3-java 2024-03-09T14:27:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: libcommons-parent-java libgeronimo-annotation-1.3-spec-java 2024-03-09T14:27:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: libgeronimo-interceptor-3.0-spec-java libguava-java libguice-java 2024-03-09T14:27:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: libhawtjni-runtime-java libjansi-java libjansi-native-java libjsr305-java 2024-03-09T14:27:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: libmaven-parent-java libmaven-resolver-java libmaven-shared-utils-java 2024-03-09T14:27:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: libmaven3-core-java libplexus-cipher-java libplexus-classworlds-java 2024-03-09T14:27:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: libplexus-component-annotations-java libplexus-interpolation-java 2024-03-09T14:27:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: libplexus-sec-dispatcher-java libplexus-utils2-java libsisu-inject-java 2024-03-09T14:27:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: libsisu-plexus-java libslf4j-java libwagon-file-java 2024-03-09T14:27:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: libwagon-http-shaded-java libwagon-provider-api-java python3-magic 2024-03-09T14:27:02.638 INFO:teuthology.orchestra.run.smithi001.stdout:Suggested packages: 2024-03-09T14:27:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: libaopalliance-java-doc libatinject-jsr330-api-java-doc libel-api-java 2024-03-09T14:27:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: libcommons-io-java-doc libcommons-lang3-java-doc libasm-java libcglib-java 2024-03-09T14:27:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: libjsr305-java-doc libmaven-shared-utils-java-doc liblogback-java 2024-03-09T14:27:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: libplexus-classworlds-java-doc libplexus-sec-dispatcher-java-doc 2024-03-09T14:27:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: libplexus-utils2-java-doc testng libcommons-logging-java liblog4j1.2-java 2024-03-09T14:27:02.678 INFO:teuthology.orchestra.run.smithi001.stdout:The following NEW packages will be installed: 2024-03-09T14:27:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: libaopalliance-java libapache-pom-java libatinject-jsr330-api-java 2024-03-09T14:27:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: libcdi-api-java libcommons-cli-java libcommons-io-java libcommons-lang3-java 2024-03-09T14:27:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: libcommons-parent-java libgeronimo-annotation-1.3-spec-java 2024-03-09T14:27:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: libgeronimo-interceptor-3.0-spec-java libguava-java libguice-java 2024-03-09T14:27:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: libhawtjni-runtime-java libjansi-java libjansi-native-java libjsr305-java 2024-03-09T14:27:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: libmaven-parent-java libmaven-resolver-java libmaven-shared-utils-java 2024-03-09T14:27:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: libmaven3-core-java libplexus-cipher-java libplexus-classworlds-java 2024-03-09T14:27:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: libplexus-component-annotations-java libplexus-interpolation-java 2024-03-09T14:27:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: libplexus-sec-dispatcher-java libplexus-utils2-java libsisu-inject-java 2024-03-09T14:27:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: libsisu-plexus-java libslf4j-java libwagon-file-java 2024-03-09T14:27:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: libwagon-http-shaded-java libwagon-provider-api-java maven python3-magic 2024-03-09T14:27:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: s3cmd 2024-03-09T14:27:02.741 INFO:teuthology.orchestra.run.smithi001.stdout:0 upgraded, 35 newly installed, 0 to remove and 275 not upgraded. 2024-03-09T14:27:02.741 INFO:teuthology.orchestra.run.smithi001.stdout:Need to get 10.4 MB of archives. 2024-03-09T14:27:02.741 INFO:teuthology.orchestra.run.smithi001.stdout:After this operation, 13.9 MB of additional disk space will be used. 2024-03-09T14:27:02.741 INFO:teuthology.orchestra.run.smithi001.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libapache-pom-java all 18-1 [4,720 B] 2024-03-09T14:27:02.743 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libplexus-sec-dispatcher-java (1.4-4) ... 2024-03-09T14:27:02.763 INFO:teuthology.orchestra.run.smithi001.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libatinject-jsr330-api-java all 1.0+ds1-5 [5,348 B] 2024-03-09T14:27:02.763 INFO:teuthology.orchestra.run.smithi001.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libgeronimo-interceptor-3.0-spec-java all 1.0.1-4fakesync [8,616 B] 2024-03-09T14:27:02.767 INFO:teuthology.orchestra.run.smithi001.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libcdi-api-java all 1.2-3 [54.3 kB] 2024-03-09T14:27:02.809 INFO:teuthology.orchestra.run.smithi001.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libcommons-cli-java all 1.4-2 [55.8 kB] 2024-03-09T14:27:02.827 INFO:teuthology.orchestra.run.smithi001.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libcommons-parent-java all 43-1 [10.8 kB] 2024-03-09T14:27:02.829 INFO:teuthology.orchestra.run.smithi001.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libcommons-io-java all 2.11.0-2 [297 kB] 2024-03-09T14:27:02.864 INFO:teuthology.orchestra.run.smithi001.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libcommons-lang3-java all 3.11-1 [526 kB] 2024-03-09T14:27:02.891 INFO:teuthology.orchestra.run.smithi001.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libgeronimo-annotation-1.3-spec-java all 1.3-1 [11.2 kB] 2024-03-09T14:27:02.891 INFO:teuthology.orchestra.run.smithi001.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libjsr305-java all 0.1~+svn49-11 [27.0 kB] 2024-03-09T14:27:02.893 INFO:teuthology.orchestra.run.smithi001.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libguava-java all 29.0-6 [2,418 kB] 2024-03-09T14:27:02.903 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libwagon-http-shaded-java (3.3.4-1) ... 2024-03-09T14:27:02.943 INFO:teuthology.orchestra.run.smithi001.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libaopalliance-java all 20070526-6 [9,084 B] 2024-03-09T14:27:02.943 INFO:teuthology.orchestra.run.smithi001.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libguice-java all 4.2.3-2 [1,434 kB] 2024-03-09T14:27:02.962 INFO:teuthology.orchestra.run.smithi001.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libhawtjni-runtime-java all 1.17-1 [28.8 kB] 2024-03-09T14:27:02.962 INFO:teuthology.orchestra.run.smithi001.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libjansi-native-java all 1.8-1 [23.8 kB] 2024-03-09T14:27:02.963 INFO:teuthology.orchestra.run.smithi001.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libjansi-java all 1.18-1 [56.8 kB] 2024-03-09T14:27:02.965 INFO:teuthology.orchestra.run.smithi001.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libmaven-parent-java all 31-2 [5,140 B] 2024-03-09T14:27:02.965 INFO:teuthology.orchestra.run.smithi001.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libplexus-utils2-java all 3.3.0-1 [250 kB] 2024-03-09T14:27:02.967 INFO:teuthology.orchestra.run.smithi001.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libwagon-provider-api-java all 3.3.4-1 [48.5 kB] 2024-03-09T14:27:02.968 INFO:teuthology.orchestra.run.smithi001.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libmaven-resolver-java all 1.4.2-3build1 [555 kB] 2024-03-09T14:27:02.974 INFO:teuthology.orchestra.run.smithi001.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libmaven-shared-utils-java all 3.3.0-1 [149 kB] 2024-03-09T14:27:02.986 INFO:teuthology.orchestra.run.smithi001.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libplexus-cipher-java all 1.8-2 [15.1 kB] 2024-03-09T14:27:02.986 INFO:teuthology.orchestra.run.smithi001.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libplexus-classworlds-java all 2.6.0-1 [49.4 kB] 2024-03-09T14:27:02.987 INFO:teuthology.orchestra.run.smithi001.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libplexus-component-annotations-java all 2.1.0-1 [6,564 B] 2024-03-09T14:27:02.987 INFO:teuthology.orchestra.run.smithi001.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libplexus-interpolation-java all 1.26-1 [76.8 kB] 2024-03-09T14:27:03.006 INFO:teuthology.orchestra.run.smithi001.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libplexus-sec-dispatcher-java all 1.4-4 [28.1 kB] 2024-03-09T14:27:03.007 INFO:teuthology.orchestra.run.smithi001.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libslf4j-java all 1.7.32-1 [141 kB] 2024-03-09T14:27:03.009 INFO:teuthology.orchestra.run.smithi001.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libsisu-inject-java all 0.3.4-2 [347 kB] 2024-03-09T14:27:03.012 INFO:teuthology.orchestra.run.smithi001.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libsisu-plexus-java all 0.3.4-3 [181 kB] 2024-03-09T14:27:03.014 INFO:teuthology.orchestra.run.smithi001.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libmaven3-core-java all 3.6.3-5 [1,535 kB] 2024-03-09T14:27:03.029 INFO:teuthology.orchestra.run.smithi001.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libwagon-file-java all 3.3.4-1 [7,928 B] 2024-03-09T14:27:03.030 INFO:teuthology.orchestra.run.smithi001.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libwagon-http-shaded-java all 3.3.4-1 [1,855 kB] 2024-03-09T14:27:03.045 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-magic (2:0.4.24-2) ... 2024-03-09T14:27:03.049 INFO:teuthology.orchestra.run.smithi001.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/universe amd64 maven all 3.6.3-5 [17.8 kB] 2024-03-09T14:27:03.049 INFO:teuthology.orchestra.run.smithi001.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-magic all 2:0.4.24-2 [12.6 kB] 2024-03-09T14:27:03.049 INFO:teuthology.orchestra.run.smithi001.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/universe amd64 s3cmd all 2.2.0-1 [120 kB] 2024-03-09T14:27:03.330 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libcdi-api-java (1.2-3) ... 2024-03-09T14:27:03.481 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libhawtjni-runtime-java (1.17-1) ... 2024-03-09T14:27:03.505 INFO:teuthology.orchestra.run.smithi001.stdout:Fetched 10.4 MB in 0s (28.7 MB/s) 2024-03-09T14:27:03.594 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libapache-pom-java. 2024-03-09T14:27:03.616 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libwagon-provider-api-java (3.3.4-1) ... 2024-03-09T14:27:03.653 INFO:teuthology.orchestra.run.smithi001.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173568 files and directories currently installed.) 2024-03-09T14:27:03.658 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../00-libapache-pom-java_18-1_all.deb ... 2024-03-09T14:27:03.710 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libapache-pom-java (18-1) ... 2024-03-09T14:27:03.742 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libmaven-parent-java (31-2) ... 2024-03-09T14:27:03.868 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libcommons-parent-java (43-1) ... 2024-03-09T14:27:03.971 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libatinject-jsr330-api-java. 2024-03-09T14:27:03.988 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../01-libatinject-jsr330-api-java_1.0+ds1-5_all.deb ... 2024-03-09T14:27:04.003 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libsisu-inject-java (0.3.4-2) ... 2024-03-09T14:27:04.029 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libatinject-jsr330-api-java (1.0+ds1-5) ... 2024-03-09T14:27:04.129 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up s3cmd (2.2.0-1) ... 2024-03-09T14:27:04.265 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libgeronimo-interceptor-3.0-spec-java. 2024-03-09T14:27:04.283 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../02-libgeronimo-interceptor-3.0-spec-java_1.0.1-4fakesync_all.deb ... 2024-03-09T14:27:04.315 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libgeronimo-interceptor-3.0-spec-java (1.0.1-4fakesync) ... 2024-03-09T14:27:04.455 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libsisu-plexus-java (0.3.4-3) ... 2024-03-09T14:27:04.568 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libcdi-api-java. 2024-03-09T14:27:04.596 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../03-libcdi-api-java_1.2-3_all.deb ... 2024-03-09T14:27:04.623 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libmaven-resolver-java (1.4.2-3build1) ... 2024-03-09T14:27:04.634 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libcdi-api-java (1.2-3) ... 2024-03-09T14:27:04.758 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libguava-java (29.0-6) ... 2024-03-09T14:27:04.883 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libcommons-lang3-java (3.11-1) ... 2024-03-09T14:27:04.987 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libcommons-cli-java. 2024-03-09T14:27:05.004 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../04-libcommons-cli-java_1.4-2_all.deb ... 2024-03-09T14:27:05.010 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libjansi-native-java (1.8-1) ... 2024-03-09T14:27:05.045 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libcommons-cli-java (1.4-2) ... 2024-03-09T14:27:05.144 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libwagon-file-java (3.3.4-1) ... 2024-03-09T14:27:05.270 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libcommons-io-java (2.11.0-2) ... 2024-03-09T14:27:05.281 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libcommons-parent-java. 2024-03-09T14:27:05.297 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../05-libcommons-parent-java_43-1_all.deb ... 2024-03-09T14:27:05.339 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libcommons-parent-java (43-1) ... 2024-03-09T14:27:05.421 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libguice-java (4.2.3-2) ... 2024-03-09T14:27:05.572 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libjansi-java (1.18-1) ... 2024-03-09T14:27:05.591 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libcommons-io-java. 2024-03-09T14:27:05.608 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../06-libcommons-io-java_2.11.0-2_all.deb ... 2024-03-09T14:27:05.641 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libcommons-io-java (2.11.0-2) ... 2024-03-09T14:27:05.723 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libmaven-shared-utils-java (3.3.0-1) ... 2024-03-09T14:27:05.858 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libmaven3-core-java (3.6.3-5) ... 2024-03-09T14:27:05.903 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libcommons-lang3-java. 2024-03-09T14:27:05.931 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../07-libcommons-lang3-java_3.11-1_all.deb ... 2024-03-09T14:27:05.969 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libcommons-lang3-java (3.11-1) ... 2024-03-09T14:27:05.984 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up maven (3.6.3-5) ... 2024-03-09T14:27:06.272 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libgeronimo-annotation-1.3-spec-java. 2024-03-09T14:27:06.282 INFO:teuthology.orchestra.run.smithi053.stdout:update-alternatives: using /usr/share/maven/bin/mvn to provide /usr/bin/mvn (mvn) in auto mode 2024-03-09T14:27:06.300 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../08-libgeronimo-annotation-1.3-spec-java_1.3-1_all.deb ... 2024-03-09T14:27:06.355 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libgeronimo-annotation-1.3-spec-java (1.3-1) ... 2024-03-09T14:27:06.355 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-09T14:27:06.607 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libjsr305-java. 2024-03-09T14:27:06.635 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../09-libjsr305-java_0.1~+svn49-11_all.deb ... 2024-03-09T14:27:06.674 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libjsr305-java (0.1~+svn49-11) ... 2024-03-09T14:27:06.910 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libguava-java. 2024-03-09T14:27:06.935 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../10-libguava-java_29.0-6_all.deb ... 2024-03-09T14:27:06.977 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libguava-java (29.0-6) ... 2024-03-09T14:27:07.413 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libaopalliance-java. 2024-03-09T14:27:07.430 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../11-libaopalliance-java_20070526-6_all.deb ... 2024-03-09T14:27:07.471 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libaopalliance-java (20070526-6) ... 2024-03-09T14:27:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libguice-java. 2024-03-09T14:27:07.727 INFO:teuthology.orchestra.run.smithi053.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-09T14:27:07.733 DEBUG:teuthology.parallel:result is None 2024-03-09T14:27:07.740 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../12-libguice-java_4.2.3-2_all.deb ... 2024-03-09T14:27:07.781 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libguice-java (4.2.3-2) ... 2024-03-09T14:27:08.160 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libhawtjni-runtime-java. 2024-03-09T14:27:08.177 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../13-libhawtjni-runtime-java_1.17-1_all.deb ... 2024-03-09T14:27:08.218 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libhawtjni-runtime-java (1.17-1) ... 2024-03-09T14:27:08.429 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libjansi-native-java. 2024-03-09T14:27:08.443 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../14-libjansi-native-java_1.8-1_all.deb ... 2024-03-09T14:27:08.479 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libjansi-native-java (1.8-1) ... 2024-03-09T14:27:08.732 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libjansi-java. 2024-03-09T14:27:08.760 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../15-libjansi-java_1.18-1_all.deb ... 2024-03-09T14:27:08.798 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libjansi-java (1.18-1) ... 2024-03-09T14:27:09.051 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libmaven-parent-java. 2024-03-09T14:27:09.067 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../16-libmaven-parent-java_31-2_all.deb ... 2024-03-09T14:27:09.109 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libmaven-parent-java (31-2) ... 2024-03-09T14:27:09.395 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libplexus-utils2-java. 2024-03-09T14:27:09.423 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../17-libplexus-utils2-java_3.3.0-1_all.deb ... 2024-03-09T14:27:09.461 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libplexus-utils2-java (3.3.0-1) ... 2024-03-09T14:27:09.714 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libwagon-provider-api-java. 2024-03-09T14:27:09.731 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../18-libwagon-provider-api-java_3.3.4-1_all.deb ... 2024-03-09T14:27:09.772 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libwagon-provider-api-java (3.3.4-1) ... 2024-03-09T14:27:10.017 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libmaven-resolver-java. 2024-03-09T14:27:10.034 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../19-libmaven-resolver-java_1.4.2-3build1_all.deb ... 2024-03-09T14:27:10.075 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libmaven-resolver-java (1.4.2-3build1) ... 2024-03-09T14:27:10.404 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libmaven-shared-utils-java. 2024-03-09T14:27:10.432 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../20-libmaven-shared-utils-java_3.3.0-1_all.deb ... 2024-03-09T14:27:10.471 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libmaven-shared-utils-java (3.3.0-1) ... 2024-03-09T14:27:10.731 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libplexus-cipher-java. 2024-03-09T14:27:10.748 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../21-libplexus-cipher-java_1.8-2_all.deb ... 2024-03-09T14:27:10.789 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libplexus-cipher-java (1.8-2) ... 2024-03-09T14:27:11.000 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libplexus-classworlds-java. 2024-03-09T14:27:11.018 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../22-libplexus-classworlds-java_2.6.0-1_all.deb ... 2024-03-09T14:27:11.059 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libplexus-classworlds-java (2.6.0-1) ... 2024-03-09T14:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libplexus-component-annotations-java. 2024-03-09T14:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../23-libplexus-component-annotations-java_2.1.0-1_all.deb ... 2024-03-09T14:27:11.386 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libplexus-component-annotations-java (2.1.0-1) ... 2024-03-09T14:27:11.639 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libplexus-interpolation-java. 2024-03-09T14:27:11.667 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../24-libplexus-interpolation-java_1.26-1_all.deb ... 2024-03-09T14:27:11.705 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libplexus-interpolation-java (1.26-1) ... 2024-03-09T14:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libplexus-sec-dispatcher-java. 2024-03-09T14:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../25-libplexus-sec-dispatcher-java_1.4-4_all.deb ... 2024-03-09T14:27:12.032 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libplexus-sec-dispatcher-java (1.4-4) ... 2024-03-09T14:27:12.268 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libslf4j-java. 2024-03-09T14:27:12.296 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../26-libslf4j-java_1.7.32-1_all.deb ... 2024-03-09T14:27:12.335 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libslf4j-java (1.7.32-1) ... 2024-03-09T14:27:12.597 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libsisu-inject-java. 2024-03-09T14:27:12.614 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../27-libsisu-inject-java_0.3.4-2_all.deb ... 2024-03-09T14:27:12.655 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libsisu-inject-java (0.3.4-2) ... 2024-03-09T14:27:12.925 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libsisu-plexus-java. 2024-03-09T14:27:12.941 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../28-libsisu-plexus-java_0.3.4-3_all.deb ... 2024-03-09T14:27:12.983 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libsisu-plexus-java (0.3.4-3) ... 2024-03-09T14:27:13.277 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libmaven3-core-java. 2024-03-09T14:27:13.294 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../29-libmaven3-core-java_3.6.3-5_all.deb ... 2024-03-09T14:27:13.335 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libmaven3-core-java (3.6.3-5) ... 2024-03-09T14:27:13.707 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libwagon-file-java. 2024-03-09T14:27:13.734 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../30-libwagon-file-java_3.3.4-1_all.deb ... 2024-03-09T14:27:13.773 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libwagon-file-java (3.3.4-1) ... 2024-03-09T14:27:14.025 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libwagon-http-shaded-java. 2024-03-09T14:27:14.053 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../31-libwagon-http-shaded-java_3.3.4-1_all.deb ... 2024-03-09T14:27:14.092 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libwagon-http-shaded-java (3.3.4-1) ... 2024-03-09T14:27:14.528 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package maven. 2024-03-09T14:27:14.558 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../32-maven_3.6.3-5_all.deb ... 2024-03-09T14:27:14.594 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking maven (3.6.3-5) ... 2024-03-09T14:27:14.900 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-magic. 2024-03-09T14:27:14.928 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../33-python3-magic_2%3a0.4.24-2_all.deb ... 2024-03-09T14:27:14.966 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-magic (2:0.4.24-2) ... 2024-03-09T14:27:15.252 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package s3cmd. 2024-03-09T14:27:15.281 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../34-s3cmd_2.2.0-1_all.deb ... 2024-03-09T14:27:15.319 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking s3cmd (2.2.0-1) ... 2024-03-09T14:27:15.682 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libslf4j-java (1.7.32-1) ... 2024-03-09T14:27:15.824 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libplexus-utils2-java (3.3.0-1) ... 2024-03-09T14:27:15.950 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libplexus-classworlds-java (2.6.0-1) ... 2024-03-09T14:27:16.076 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libjsr305-java (0.1~+svn49-11) ... 2024-03-09T14:27:16.202 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libaopalliance-java (20070526-6) ... 2024-03-09T14:27:16.328 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libcommons-cli-java (1.4-2) ... 2024-03-09T14:27:16.445 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libplexus-component-annotations-java (2.1.0-1) ... 2024-03-09T14:27:16.546 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libplexus-cipher-java (1.8-2) ... 2024-03-09T14:27:16.656 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libgeronimo-annotation-1.3-spec-java (1.3-1) ... 2024-03-09T14:27:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libgeronimo-interceptor-3.0-spec-java (1.0.1-4fakesync) ... 2024-03-09T14:27:16.908 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libapache-pom-java (18-1) ... 2024-03-09T14:27:17.050 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libatinject-jsr330-api-java (1.0+ds1-5) ... 2024-03-09T14:27:17.176 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libplexus-interpolation-java (1.26-1) ... 2024-03-09T14:27:17.302 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libplexus-sec-dispatcher-java (1.4-4) ... 2024-03-09T14:27:17.428 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libwagon-http-shaded-java (3.3.4-1) ... 2024-03-09T14:27:17.546 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-magic (2:0.4.24-2) ... 2024-03-09T14:27:17.789 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libcdi-api-java (1.2-3) ... 2024-03-09T14:27:17.898 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libhawtjni-runtime-java (1.17-1) ... 2024-03-09T14:27:18.024 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libwagon-provider-api-java (3.3.4-1) ... 2024-03-09T14:27:18.150 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libmaven-parent-java (31-2) ... 2024-03-09T14:27:18.276 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libcommons-parent-java (43-1) ... 2024-03-09T14:27:18.402 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libsisu-inject-java (0.3.4-2) ... 2024-03-09T14:27:18.528 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up s3cmd (2.2.0-1) ... 2024-03-09T14:27:18.829 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libsisu-plexus-java (0.3.4-3) ... 2024-03-09T14:27:18.964 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libmaven-resolver-java (1.4.2-3build1) ... 2024-03-09T14:27:19.090 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libguava-java (29.0-6) ... 2024-03-09T14:27:19.216 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libcommons-lang3-java (3.11-1) ... 2024-03-09T14:27:19.342 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libjansi-native-java (1.8-1) ... 2024-03-09T14:27:19.468 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libwagon-file-java (3.3.4-1) ... 2024-03-09T14:27:19.593 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libcommons-io-java (2.11.0-2) ... 2024-03-09T14:27:19.711 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libguice-java (4.2.3-2) ... 2024-03-09T14:27:19.820 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libjansi-java (1.18-1) ... 2024-03-09T14:27:19.930 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libmaven-shared-utils-java (3.3.0-1) ... 2024-03-09T14:27:20.073 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libmaven3-core-java (3.6.3-5) ... 2024-03-09T14:27:20.199 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up maven (3.6.3-5) ... 2024-03-09T14:27:20.455 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/share/maven/bin/mvn to provide /usr/bin/mvn (mvn) in auto mode 2024-03-09T14:27:20.519 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-09T14:27:21.834 INFO:teuthology.orchestra.run.smithi001.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-09T14:27:21.839 DEBUG:teuthology.parallel:result is None 2024-03-09T14:27:21.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:27:22.059 DEBUG:teuthology.orchestra.run.smithi001:> dpkg-query -W -f '${Version}' ceph 2024-03-09T14:27:22.078 INFO:teuthology.orchestra.run.smithi001.stdout:19.0.0-2079-gc00d9c60-1jammy 2024-03-09T14:27:22.079 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2079-gc00d9c60-1jammy 2024-03-09T14:27:22.079 INFO:teuthology.task.install:The correct ceph version 19.0.0-2079-gc00d9c60-1jammy is installed. 2024-03-09T14:27:22.081 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:27:22.303 DEBUG:teuthology.orchestra.run.smithi053:> dpkg-query -W -f '${Version}' ceph 2024-03-09T14:27:22.323 INFO:teuthology.orchestra.run.smithi053.stdout:19.0.0-2079-gc00d9c60-1jammy 2024-03-09T14:27:22.323 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2079-gc00d9c60-1jammy 2024-03-09T14:27:22.323 INFO:teuthology.task.install:The correct ceph version 19.0.0-2079-gc00d9c60-1jammy is installed. 2024-03-09T14:27:22.325 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-09T14:27:22.326 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:22.326 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-09T14:27:22.341 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:22.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-09T14:27:22.380 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-09T14:27:22.380 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:22.381 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-03-09T14:27:22.398 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-09T14:27:22.453 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:22.454 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-03-09T14:27:22.467 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-09T14:27:22.523 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-09T14:27:22.523 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:22.524 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-09T14:27:22.537 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-09T14:27:22.594 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:22.594 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-09T14:27:22.608 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-09T14:27:22.663 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-09T14:27:22.664 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:22.664 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-03-09T14:27:22.677 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-09T14:27:22.733 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:22.733 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-03-09T14:27:22.747 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-09T14:27:22.800 INFO:teuthology.run_tasks:Running task ceph... 2024-03-09T14:27:22.880 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-09T14:27:22.880 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /var/log/ceph 2024-03-09T14:27:22.883 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /var/log/ceph 2024-03-09T14:27:22.895 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-09T14:27:22.896 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-09T14:27:22.939 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-09T14:27:22.951 INFO:tasks.ceph:Creating extra log directories... 2024-03-09T14:27:22.951 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-09T14:27:22.994 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-09T14:27:23.019 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-09T14:27:23.019 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw bucket counters cache': True, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw max chunk size': 4194304, 'rgw obj stripe size': 6291456, 'rgw s3 auth use sts': True, 'rgw sts key': 'abcdefghijklmnop', 'rgw torrent flag': True, 'rgw user counters cache': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd heartbeat grace': 40, 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10, 'rgw override bucket index max shards': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-09T14:27:23.019 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479', 'branch': 'wip-rgw-account-v3', 'description': 'rgw/verify/{0-install accounts$/{none} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7588479', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-rgw-account-v3'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw bucket counters cache': True, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw max chunk size': 4194304, 'rgw obj stripe size': 6291456, 'rgw s3 auth use sts': True, 'rgw sts key': 'abcdefghijklmnop', 'rgw torrent flag': True, 'rgw user counters cache': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd heartbeat grace': 40, 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10, 'rgw override bucket index max shards': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c00d9c609fb42a2a25f1c196bed064ca8bb02413'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c00d9c609fb42a2a25f1c196bed064ca8bb02413'}}, 'openssl_keys': {'rgw.client.0': {'ca': 'root', 'client': 'client.0', 'embed-key': True}, 'root': {'client': 'client.0', 'cn': 'teuthology', 'install': ['client.0'], 'key-type': 'rsa:4096'}}, 'rgw': {'client.0': {'ssl certificate': 'rgw.client.0', 'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'compression type': 'random', 'ec-data-pool': False, 'frontend': 'beast', 'inline data': False, 'storage classes': 'LUKEWARM, FROZEN'}, 's3tests': {'accounts': {'iam alt root': 'RGW99999999999999999', 'iam root': 'RGW88888888888888888'}, 'force-branch': 'wip-iam-user-apis', 'git_remote': 'https://github.com/cbodley/', 'storage classes': 'LUKEWARM, FROZEN'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rgw-account-v3', 'sha1': 'c00d9c609fb42a2a25f1c196bed064ca8bb02413'}}, 'owner': 'scheduled_cbodley@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 3347, 'sha1': 'c00d9c609fb42a2a25f1c196bed064ca8bb02413', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-rgw-account-v3', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c00d9c609fb42a2a25f1c196bed064ca8bb02413', 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDXuFOquNPh30bZgAJqTouKGG7p0NNxx6xXxJjUuU/VardxqUiKgFx1dYvcLV+s2NboGkt+PBwES7RqxIXJxcpQ=', 'smithi053.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKaPOccuN9FVN66Lr7uOPJ7Q4JA1CmmSB1iQvohXt3rGZ6uBS7/6xY3JYjapv/HuJSah4Lhzo+bkGObImlln7ic='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_system_packages': {'deb': ['s3cmd', 'maven'], 'rpm': ['s3cmd', 'maven']}, 'flavor': 'default', 'sha1': 'c00d9c609fb42a2a25f1c196bed064ca8bb02413'}}, {'ceph': None}, {'openssl_keys': None}, {'rgw': {'client.0': None}}, {'tox': ['client.0']}, {'workunit': {'clients': {'client.0': ['rgw/run-bucket-check.sh']}}}, {'workunit': {'clients': {'client.0': ['cls/test_cls_lock.sh', 'cls/test_cls_log.sh', 'cls/test_cls_refcount.sh', 'cls/test_cls_rgw.sh', 'cls/test_cls_rgw_gc.sh', 'cls/test_cls_rgw_stats.sh', 'cls/test_cls_cmpomap.sh', 'cls/test_cls_2pc_queue.sh', 'cls/test_cls_user.sh', 'rgw/test_rgw_gc_log.sh', 'rgw/test_rgw_obj.sh', 'rgw/test_librgw_file.sh', 'rgw/test_awssdkv4_sig.sh']}}}, {'workunit': {'clients': {'client.0': ['rgw/test_rgw_s3_mp_reupload.sh']}}}, {'ragweed': {'client.0': {'default-branch': 'ceph-master', 'rgw_server': 'client.0', 'stages': 'prepare,check'}}}, {'workunit': {'clients': {'client.0': ['rgw/run-reshard.sh']}}}, {'s3tests-java': {'client.0': {'force-branch': 'ceph-master', 'force-repo': 'https://github.com/ceph/java_s3tests.git'}}}, {'s3tests': {'client.0': {'rgw_server': 'client.0'}}}, {'workunit': {'clients': {'client.0': ['rgw/run-versioning.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-09_13:26:09', 'tube': 'smithi', 'user': 'cbodley', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-09T14:27:23.020 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-09T14:27:23.051 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-09T14:27:23.057 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-03-09T14:27:23.099 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/run/ceph 2024-03-09T14:27:23.113 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:23.113 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-03-09T14:27:23.158 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-09T14:27:23.159 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-03-09T14:27:23.206 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-09T14:27:23.206 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-09T14:27:23.206 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 884 Links: 1 2024-03-09T14:27:23.206 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-09T14:27:23.206 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-09 14:21:32.471662577 +0000 2024-03-09T14:27:23.206 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-09 14:21:32.163665325 +0000 2024-03-09T14:27:23.206 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-09 14:21:32.163665325 +0000 2024-03-09T14:27:23.206 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-03-09T14:27:23.207 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-09T14:27:23.260 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-09T14:27:23.260 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-09T14:27:23.260 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000242941 s, 2.1 MB/s 2024-03-09T14:27:23.261 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-09T14:27:23.311 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-03-09T14:27:23.362 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-09T14:27:23.362 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-09T14:27:23.362 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-03-09T14:27:23.363 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-09T14:27:23.363 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-09 14:21:32.727660294 +0000 2024-03-09T14:27:23.363 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-09 14:21:32.723660328 +0000 2024-03-09T14:27:23.363 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-09 14:21:32.723660328 +0000 2024-03-09T14:27:23.363 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-03-09T14:27:23.363 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-09T14:27:23.416 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-09T14:27:23.416 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-09T14:27:23.416 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000256559 s, 2.0 MB/s 2024-03-09T14:27:23.418 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-09T14:27:23.467 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-03-09T14:27:23.514 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-09T14:27:23.514 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-09T14:27:23.514 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 918 Links: 1 2024-03-09T14:27:23.514 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-09T14:27:23.514 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-09 14:21:33.243655690 +0000 2024-03-09T14:27:23.514 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-09 14:21:33.239655726 +0000 2024-03-09T14:27:23.514 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-09 14:21:33.239655726 +0000 2024-03-09T14:27:23.514 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-03-09T14:27:23.515 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-09T14:27:23.569 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-09T14:27:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-09T14:27:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000296684 s, 1.7 MB/s 2024-03-09T14:27:23.570 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-09T14:27:23.619 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-03-09T14:27:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-09T14:27:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-09T14:27:23.666 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 936 Links: 1 2024-03-09T14:27:23.666 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-09T14:27:23.666 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-09 14:21:33.735651301 +0000 2024-03-09T14:27:23.666 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-09 14:21:33.731651336 +0000 2024-03-09T14:27:23.666 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-09 14:21:33.731651336 +0000 2024-03-09T14:27:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-03-09T14:27:23.666 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-09T14:27:23.720 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-09T14:27:23.720 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-09T14:27:23.720 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000283204 s, 1.8 MB/s 2024-03-09T14:27:23.721 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-09T14:27:23.772 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-09T14:27:23.772 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:23.772 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-03-09T14:27:23.778 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-09T14:27:23.778 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-03-09T14:27:23.824 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-09T14:27:23.824 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-09T14:27:23.824 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 878 Links: 1 2024-03-09T14:27:23.824 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-09T14:27:23.824 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-09 14:21:32.410989000 +0000 2024-03-09T14:27:23.825 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-09 14:21:32.150991319 +0000 2024-03-09T14:27:23.825 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-09 14:21:32.150991319 +0000 2024-03-09T14:27:23.825 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: - 2024-03-09T14:27:23.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-09T14:27:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-09T14:27:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-09T14:27:23.879 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000309241 s, 1.7 MB/s 2024-03-09T14:27:23.880 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-09T14:27:23.930 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-03-09T14:27:23.976 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-09T14:27:23.976 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-09T14:27:23.977 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-03-09T14:27:23.977 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-09T14:27:23.977 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-09 14:21:32.886984753 +0000 2024-03-09T14:27:23.977 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-09 14:21:32.634987001 +0000 2024-03-09T14:27:23.977 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-09 14:21:32.634987001 +0000 2024-03-09T14:27:23.977 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: - 2024-03-09T14:27:23.977 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-09T14:27:24.030 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-09T14:27:24.030 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-09T14:27:24.030 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000203632 s, 2.5 MB/s 2024-03-09T14:27:24.032 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-09T14:27:24.082 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-03-09T14:27:24.128 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-09T14:27:24.128 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-09T14:27:24.128 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 916 Links: 1 2024-03-09T14:27:24.128 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-09T14:27:24.128 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-09 14:21:33.398980186 +0000 2024-03-09T14:27:24.128 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-09 14:21:33.130982577 +0000 2024-03-09T14:27:24.129 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-09 14:21:33.130982577 +0000 2024-03-09T14:27:24.129 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: - 2024-03-09T14:27:24.129 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-09T14:27:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-09T14:27:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-09T14:27:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000227908 s, 2.2 MB/s 2024-03-09T14:27:24.184 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-09T14:27:24.233 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-03-09T14:27:24.280 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-09T14:27:24.280 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-09T14:27:24.280 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 928 Links: 1 2024-03-09T14:27:24.280 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-09T14:27:24.280 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-09 14:21:33.622978188 +0000 2024-03-09T14:27:24.280 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-09 14:21:33.618978223 +0000 2024-03-09T14:27:24.281 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-09 14:21:33.618978223 +0000 2024-03-09T14:27:24.281 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: - 2024-03-09T14:27:24.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-09T14:27:24.335 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-09T14:27:24.335 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-09T14:27:24.335 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000302788 s, 1.7 MB/s 2024-03-09T14:27:24.336 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-09T14:27:24.385 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-09T14:27:24.386 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-09T14:27:24.386 INFO:tasks.ceph:Generating config... 2024-03-09T14:27:24.388 INFO:tasks.ceph:[client] debug rgw = 20 2024-03-09T14:27:24.388 INFO:tasks.ceph:[client] rgw bucket counters cache = True 2024-03-09T14:27:24.388 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-03-09T14:27:24.388 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-03-09T14:27:24.388 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-03-09T14:27:24.388 INFO:tasks.ceph:[client] rgw max chunk size = 4194304 2024-03-09T14:27:24.388 INFO:tasks.ceph:[client] rgw obj stripe size = 6291456 2024-03-09T14:27:24.388 INFO:tasks.ceph:[client] rgw s3 auth use sts = True 2024-03-09T14:27:24.388 INFO:tasks.ceph:[client] rgw sts key = abcdefghijklmnop 2024-03-09T14:27:24.388 INFO:tasks.ceph:[client] rgw torrent flag = True 2024-03-09T14:27:24.388 INFO:tasks.ceph:[client] rgw user counters cache = True 2024-03-09T14:27:24.388 INFO:tasks.ceph:[client] setgroup = ceph 2024-03-09T14:27:24.389 INFO:tasks.ceph:[client] setuser = ceph 2024-03-09T14:27:24.389 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-09T14:27:24.389 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-09T14:27:24.389 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2024-03-09T14:27:24.389 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 10 2024-03-09T14:27:24.389 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 10 2024-03-09T14:27:24.389 INFO:tasks.ceph:[global] rgw override bucket index max shards = 1 2024-03-09T14:27:24.389 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-09T14:27:24.389 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-09T14:27:24.389 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-09T14:27:24.389 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-09T14:27:24.389 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-09T14:27:24.389 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-09T14:27:24.389 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-09T14:27:24.390 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-09T14:27:24.390 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-09T14:27:24.390 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-09T14:27:24.390 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-09T14:27:24.390 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-09T14:27:24.390 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-09T14:27:24.390 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-09T14:27:24.390 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-09T14:27:24.390 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-09T14:27:24.390 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-09T14:27:24.390 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-09T14:27:24.390 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-09T14:27:24.391 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-09T14:27:24.391 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-09T14:27:24.391 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-09T14:27:24.391 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-09T14:27:24.391 INFO:tasks.ceph:Setting up mon.a... 2024-03-09T14:27:24.391 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-09T14:27:24.426 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.keyring 2024-03-09T14:27:24.449 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-09T14:27:24.531 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-09T14:27:24.585 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.1'), ('mon.c', '[v2:172.21.15.1:3301,v1:172.21.15.1:6790]'), ('mon.b', '172.21.15.53')] 2024-03-09T14:27:24.585 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.1,[v2:172.21.15.1:3301,v1:172.21.15.1:6790],172.21.15.53', 'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd heartbeat grace': 40, 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10, 'rgw override bucket index max shards': 1}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw bucket counters cache': True, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw max chunk size': 4194304, 'rgw obj stripe size': 6291456, 'rgw s3 auth use sts': True, 'rgw sts key': 'abcdefghijklmnop', 'rgw torrent flag': True, 'rgw user counters cache': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-09T14:27:24.586 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:24.586 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-09T14:27:24.634 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.1 --addv c '[v2:172.21.15.1:3301,v1:172.21.15.1:6790]' --add b 172.21.15.53 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:27:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:27:24.719 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:24.719 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: generated fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:27:24.719 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = quincy 2024-03-09T14:27:24.719 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 0 2024-03-09T14:27:24.720 INFO:teuthology.orchestra.run.smithi001.stdout:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:27:24.720 INFO:teuthology.orchestra.run.smithi001.stdout:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:27:24.720 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:27:24.720 INFO:teuthology.orchestra.run.smithi001.stdout:min_mon_release 17 (quincy) 2024-03-09T14:27:24.720 INFO:teuthology.orchestra.run.smithi001.stdout:election_strategy: 1 2024-03-09T14:27:24.720 INFO:teuthology.orchestra.run.smithi001.stdout:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:27:24.720 INFO:teuthology.orchestra.run.smithi001.stdout:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:27:24.720 INFO:teuthology.orchestra.run.smithi001.stdout:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:27:24.720 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-09T14:27:24.721 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-09T14:27:24.769 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fe441b42-34a9-46d6-87e9-ac75455181ab... 2024-03-09T14:27:24.769 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-09T14:27:24.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-09T14:27:24.842 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-03-09T14:27:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-03-09T14:27:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-09T14:27:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-03-09T14:27:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-03-09T14:27:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-03-09T14:27:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-03-09T14:27:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-03-09T14:27:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-03-09T14:27:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-03-09T14:27:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-03-09T14:27:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-03-09T14:27:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-03-09T14:27:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-03-09T14:27:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-03-09T14:27:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-09T14:27:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-09T14:27:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-03-09T14:27:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-09T14:27:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-03-09T14:27:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-03-09T14:27:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-03-09T14:27:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-03-09T14:27:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-03-09T14:27:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = "172.21.15.1,[v2:172.21.15.1:3301,v1:172.21.15.1:6790],172.21.15.53" 2024-03-09T14:27:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: mon client directed command retry = 5 2024-03-09T14:27:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject socket failures = 5000 2024-03-09T14:27:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: osd heartbeat grace = 40 2024-03-09T14:27:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: osd_max_pg_log_entries = 10 2024-03-09T14:27:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: osd_min_pg_log_entries = 10 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: rgw override bucket index max shards = 1 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 600 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-03-09T14:27:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-03-09T14:27:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-03-09T14:27:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-03-09T14:27:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug trim objects = true 2024-03-09T14:27:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-03-09T14:27:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-03-09T14:27:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-03-09T14:27:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-03-09T14:27:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-03-09T14:27:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-03-09T14:27:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-03-09T14:27:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-03-09T14:27:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-03-09T14:27:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: bdev async discard = True 2024-03-09T14:27:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: bdev enable discard = True 2024-03-09T14:27:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore allocator = bitmap 2024-03-09T14:27:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-03-09T14:27:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-03-09T14:27:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 1/20 2024-03-09T14:27:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 1/20 2024-03-09T14:27:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-09T14:27:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-03-09T14:27:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 4/10 2024-03-09T14:27:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-03-09T14:27:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-03-09T14:27:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-03-09T14:27:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-03-09T14:27:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: osd fast shutdown = False 2024-03-09T14:27:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-03-09T14:27:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: osd shutdown pgref assert = True 2024-03-09T14:27:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.851 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-03-09T14:27:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-09T14:27:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-03-09T14:27:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-03-09T14:27:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-03-09T14:27:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-03-09T14:27:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-03-09T14:27:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-03-09T14:27:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.852 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-03-09T14:27:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-09T14:27:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-03-09T14:27:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-03-09T14:27:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-03-09T14:27:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-03-09T14:27:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-03-09T14:27:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-03-09T14:27:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-03-09T14:27:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-03-09T14:27:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-03-09T14:27:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-09T14:27:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-03-09T14:27:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-03-09T14:27:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-03-09T14:27:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-09T14:27:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-09T14:27:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd crush smoke test = False 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: debug rgw = 20 2024-03-09T14:27:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: rgw bucket counters cache = True 2024-03-09T14:27:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: rgw crypt require ssl = False 2024-03-09T14:27:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: rgw crypt s3 kms backend = testing 2024-03-09T14:27:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-03-09T14:27:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: rgw max chunk size = 4194304 2024-03-09T14:27:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: rgw obj stripe size = 6291456 2024-03-09T14:27:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: rgw s3 auth use sts = True 2024-03-09T14:27:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: rgw sts key = abcdefghijklmnop 2024-03-09T14:27:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: rgw torrent flag = True 2024-03-09T14:27:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: rgw user counters cache = True 2024-03-09T14:27:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: setgroup = ceph 2024-03-09T14:27:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: setuser = ceph 2024-03-09T14:27:24.856 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-03-09T14:27:24.856 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.c] 2024-03-09T14:27:24.856 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.b] 2024-03-09T14:27:24.857 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-03-09T14:27:24.857 INFO:teuthology.orchestra.run.smithi053.stdout: chdir = "" 2024-03-09T14:27:24.857 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-09T14:27:24.858 INFO:teuthology.orchestra.run.smithi053.stdout: auth supported = cephx 2024-03-09T14:27:24.858 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.858 INFO:teuthology.orchestra.run.smithi053.stdout: filestore xattr use omap = true 2024-03-09T14:27:24.858 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.858 INFO:teuthology.orchestra.run.smithi053.stdout: mon clock drift allowed = 1.000 2024-03-09T14:27:24.858 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.858 INFO:teuthology.orchestra.run.smithi053.stdout: osd crush chooseleaf type = 0 2024-03-09T14:27:24.858 INFO:teuthology.orchestra.run.smithi053.stdout: auth debug = true 2024-03-09T14:27:24.858 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.858 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on old message = true 2024-03-09T14:27:24.858 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on bug = true 2024-03-09T14:27:24.858 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: mon pg warn max object skew = 0 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: # disable pg_autoscaler by default for new pools 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default size = 2 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow primary affinity = true 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow pg remap = true 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on legacy crush tunables = false 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on crush straw calc version zero = false 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on no sortbitwise = false 2024-03-09T14:27:24.859 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on osd down out interval zero = false 2024-03-09T14:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on too few osds = false 2024-03-09T14:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-09T14:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-09T14:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: mon_allow_pool_size_one = true 2024-03-09T14:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-09T14:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: osd default data pool replay window = 5 2024-03-09T14:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: mon allow pool delete = true 2024-03-09T14:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: mon cluster log file level = debug 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: debug asserts on shutdown = true 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: mon health detail to clog = false 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: mon host = "172.21.15.1,[v2:172.21.15.1:3301,v1:172.21.15.1:6790],172.21.15.53" 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: mon client directed command retry = 5 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: ms inject socket failures = 5000 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: osd heartbeat grace = 40 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: osd_max_pg_log_entries = 10 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: osd_min_pg_log_entries = 10 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: rgw override bucket index max shards = 1 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout:[osd] 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: osd journal size = 100 2024-03-09T14:27:24.861 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub load threshold = 5.0 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub max interval = 600 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock profile = high_recovery_ops 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: osd recover clone overlap = true 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: osd recovery max chunk = 1048576 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug shutdown = true 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug op order = true 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify stray on activate = true 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug trim objects = true 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.862 INFO:teuthology.orchestra.run.smithi053.stdout: osd open classes on start = true 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug pg log writeout = true 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: osd deep scrub update digest min age = 30 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: osd map max advance = 10 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: journal zero on create = true 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: filestore ondisk finisher threads = 3 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: filestore apply finisher threads = 3 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: bdev debug aio = true 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug misdirected ops = true 2024-03-09T14:27:24.863 INFO:teuthology.orchestra.run.smithi053.stdout: bdev async discard = True 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: bdev enable discard = True 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore allocator = bitmap 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore block size = 96636764160 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore fsck on mount = True 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluefs = 1/20 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluestore = 1/20 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: debug osd = 20 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: debug rocksdb = 4/10 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd backfillfull_ratio = 0.85 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd full ratio = 0.9 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd nearfull ratio = 0.8 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: osd failsafe full ratio = 0.95 2024-03-09T14:27:24.864 INFO:teuthology.orchestra.run.smithi053.stdout: osd fast shutdown = False 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout: osd objectstore = bluestore 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout: osd shutdown pgref assert = True 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr] 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout: debug mgr = 20 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout: mgr/telemetry/nag = false 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout:[mon] 2024-03-09T14:27:24.865 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: debug paxos = 20 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: mon data avail warn = 5 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: mon mgr mkfs grace = 240 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd reporter subtree level = osd 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd prime pg temp = true 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: auth mon ticket ttl = 660 # 11m 2024-03-09T14:27:24.866 INFO:teuthology.orchestra.run.smithi053.stdout: auth service ticket ttl = 240 # 4m 2024-03-09T14:27:24.867 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.867 INFO:teuthology.orchestra.run.smithi053.stdout: # don't complain about insecure global_id in the test suite 2024-03-09T14:27:24.867 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-09T14:27:24.867 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-09T14:27:24.867 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.867 INFO:teuthology.orchestra.run.smithi053.stdout: # 1m isn't quite enough 2024-03-09T14:27:24.867 INFO:teuthology.orchestra.run.smithi053.stdout: mon_down_mkfs_grace = 2m 2024-03-09T14:27:24.867 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.867 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_filestore_osds = false 2024-03-09T14:27:24.867 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd crush smoke test = False 2024-03-09T14:27:24.867 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-09T14:27:24.867 INFO:teuthology.orchestra.run.smithi053.stdout:[client] 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: rgw cache enabled = true 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable ops log = true 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable usage log = true 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: debug rgw = 20 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: rgw bucket counters cache = True 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: rgw crypt require ssl = False 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: rgw crypt s3 kms backend = testing 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: rgw max chunk size = 4194304 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: rgw obj stripe size = 6291456 2024-03-09T14:27:24.868 INFO:teuthology.orchestra.run.smithi053.stdout: rgw s3 auth use sts = True 2024-03-09T14:27:24.869 INFO:teuthology.orchestra.run.smithi053.stdout: rgw sts key = abcdefghijklmnop 2024-03-09T14:27:24.869 INFO:teuthology.orchestra.run.smithi053.stdout: rgw torrent flag = True 2024-03-09T14:27:24.869 INFO:teuthology.orchestra.run.smithi053.stdout: rgw user counters cache = True 2024-03-09T14:27:24.869 INFO:teuthology.orchestra.run.smithi053.stdout: setgroup = ceph 2024-03-09T14:27:24.869 INFO:teuthology.orchestra.run.smithi053.stdout: setuser = ceph 2024-03-09T14:27:24.869 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.a] 2024-03-09T14:27:24.869 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.c] 2024-03-09T14:27:24.869 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.b] 2024-03-09T14:27:24.869 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-09T14:27:24.869 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-09T14:27:24.953 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-09T14:27:24.953 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:24.953 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-09T14:27:24.959 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:24.959 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-09T14:27:25.006 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-03-09T14:27:25.006 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:25.006 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-09T14:27:25.006 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-09T14:27:25.071 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:25.071 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:25.118 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi053.front.sepia.ceph.com 2024-03-09T14:27:25.118 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:25.118 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-09T14:27:25.118 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-09T14:27:25.141 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:25.142 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:25.188 INFO:tasks.ceph:Setting up mon nodes... 2024-03-09T14:27:25.188 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-09T14:27:25.189 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-09T14:27:25.233 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-09T14:27:25.236 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-09T14:27:25.281 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-09T14:27:25.293 INFO:tasks.ceph:Setting up mds nodes... 2024-03-09T14:27:25.293 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-09T14:27:25.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-09T14:27:25.329 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-09T14:27:25.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-09T14:27:25.377 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-09T14:27:25.389 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-09T14:27:25.389 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-09T14:27:25.389 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-09T14:27:25.402 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-09T14:27:25.402 INFO:tasks.ceph:role: osd.0 2024-03-09T14:27:25.402 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-09T14:27:25.402 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-09T14:27:25.458 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-09T14:27:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-09T14:27:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-09T14:27:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-09T14:27:25.458 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-09T14:27:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-09T14:27:25.458 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-09T14:27:25.458 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-09T14:27:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-09T14:27:25.459 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-09T14:27:25.482 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-09T14:27:25.484 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-09T14:27:25.484 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-09T14:27:25.542 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-09T14:27:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: /sbin/restorecon: command not found 2024-03-09T14:27:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:27:25.596 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-09T14:27:25.649 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-09T14:27:25.649 INFO:tasks.ceph:role: osd.1 2024-03-09T14:27:25.649 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-09T14:27:25.650 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-09T14:27:25.703 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-09T14:27:25.703 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-09T14:27:25.703 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-09T14:27:25.703 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-09T14:27:25.703 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-09T14:27:25.703 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-09T14:27:25.703 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-09T14:27:25.704 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-09T14:27:25.704 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-09T14:27:25.704 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-09T14:27:25.727 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-09T14:27:25.728 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-09T14:27:25.729 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-09T14:27:25.791 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-09T14:27:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:27:25.844 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: /sbin/restorecon: command not found 2024-03-09T14:27:25.844 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-09T14:27:25.896 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-09T14:27:25.897 INFO:tasks.ceph:role: osd.2 2024-03-09T14:27:25.897 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-09T14:27:25.897 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-09T14:27:25.954 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-09T14:27:25.954 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-09T14:27:25.954 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-09T14:27:25.954 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-09T14:27:25.954 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-09T14:27:25.954 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-09T14:27:25.954 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-09T14:27:25.954 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-09T14:27:25.955 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-09T14:27:25.955 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-09T14:27:25.978 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-09T14:27:25.979 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-09T14:27:25.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-09T14:27:26.036 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-09T14:27:26.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:27:26.088 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: /sbin/restorecon: command not found 2024-03-09T14:27:26.088 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-09T14:27:26.140 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-09T14:27:26.140 INFO:tasks.ceph:role: osd.3 2024-03-09T14:27:26.140 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-09T14:27:26.140 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-09T14:27:26.194 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-09T14:27:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-09T14:27:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-09T14:27:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-09T14:27:26.194 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-09T14:27:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-09T14:27:26.194 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-09T14:27:26.194 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-09T14:27:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-09T14:27:26.194 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-09T14:27:26.217 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-09T14:27:26.219 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-09T14:27:26.219 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-09T14:27:26.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-09T14:27:26.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:27:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: /sbin/restorecon: command not found 2024-03-09T14:27:26.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:26.447+0000 7f66704dc7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-09T14:27:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:26.447+0000 7f66704dc7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-09T14:27:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:26.447+0000 7f66704dc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-09T14:27:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:26.447+0000 7f66704dc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-09T14:27:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:26.447+0000 7f66704dc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-09T14:27:26.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:26.447+0000 7f66704dc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-09T14:27:29.418 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-09T14:27:29.431 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:29.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:29.507+0000 7f48987b27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-09T14:27:29.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:29.507+0000 7f48987b27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-09T14:27:29.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:29.507+0000 7f48987b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-09T14:27:29.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:29.507+0000 7f48987b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-09T14:27:29.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:29.507+0000 7f48987b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-09T14:27:29.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:29.507+0000 7f48987b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-09T14:27:32.475 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-09T14:27:32.531 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:32.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:32.611+0000 7faa255cd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-09T14:27:32.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:32.611+0000 7faa255cd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-09T14:27:32.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:32.611+0000 7faa255cd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-09T14:27:32.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:32.611+0000 7faa255cd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-09T14:27:32.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:32.611+0000 7faa255cd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-09T14:27:32.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:32.611+0000 7faa255cd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-09T14:27:35.576 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-09T14:27:35.631 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:35.711+0000 7f62a845b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-09T14:27:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:35.711+0000 7f62a845b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-09T14:27:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:35.711+0000 7f62a845b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-09T14:27:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:35.711+0000 7f62a845b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-09T14:27:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:35.711+0000 7f62a845b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-09T14:27:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:27:35.711+0000 7f62a845b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-09T14:27:38.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-09T14:27:38.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-09T14:27:38.753 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-09T14:27:38.753 INFO:tasks.ceph:role: osd.4 2024-03-09T14:27:38.753 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-09T14:27:38.753 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-09T14:27:38.808 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-09T14:27:38.808 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-09T14:27:38.808 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-09T14:27:38.809 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-09T14:27:38.809 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-09T14:27:38.809 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-03-09T14:27:38.809 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-09T14:27:38.809 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-09T14:27:38.809 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-09T14:27:38.809 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-09T14:27:38.832 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-03-09T14:27:38.834 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-03-09T14:27:38.834 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-09T14:27:38.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-09T14:27:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:27:38.940 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: /sbin/restorecon: command not found 2024-03-09T14:27:38.941 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-09T14:27:38.993 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-09T14:27:38.993 INFO:tasks.ceph:role: osd.5 2024-03-09T14:27:38.993 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-09T14:27:38.994 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-09T14:27:39.047 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-09T14:27:39.047 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-09T14:27:39.047 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-09T14:27:39.047 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-09T14:27:39.047 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-09T14:27:39.047 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-03-09T14:27:39.047 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-09T14:27:39.047 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-09T14:27:39.047 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-09T14:27:39.047 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-09T14:27:39.071 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-03-09T14:27:39.072 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-03-09T14:27:39.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-09T14:27:39.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-09T14:27:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:27:39.184 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: /sbin/restorecon: command not found 2024-03-09T14:27:39.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-09T14:27:39.239 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-09T14:27:39.239 INFO:tasks.ceph:role: osd.6 2024-03-09T14:27:39.239 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-09T14:27:39.239 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-09T14:27:39.296 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-09T14:27:39.296 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-09T14:27:39.296 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-09T14:27:39.296 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-09T14:27:39.297 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-09T14:27:39.297 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-03-09T14:27:39.297 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-09T14:27:39.297 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-09T14:27:39.297 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-09T14:27:39.297 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-09T14:27:39.320 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-03-09T14:27:39.322 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-03-09T14:27:39.323 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-09T14:27:39.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-09T14:27:39.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:27:39.428 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: /sbin/restorecon: command not found 2024-03-09T14:27:39.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-09T14:27:39.480 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-09T14:27:39.481 INFO:tasks.ceph:role: osd.7 2024-03-09T14:27:39.481 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-09T14:27:39.481 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-09T14:27:39.536 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-09T14:27:39.536 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-09T14:27:39.536 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-09T14:27:39.536 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-09T14:27:39.536 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-09T14:27:39.536 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-03-09T14:27:39.536 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-09T14:27:39.536 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-09T14:27:39.536 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-09T14:27:39.536 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-09T14:27:39.559 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-03-09T14:27:39.561 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-03-09T14:27:39.561 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-09T14:27:39.625 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-09T14:27:39.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:27:39.677 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: /sbin/restorecon: command not found 2024-03-09T14:27:39.677 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:39.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:39.785+0000 7f9f726397c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-09T14:27:39.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:39.785+0000 7f9f726397c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-09T14:27:39.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:39.785+0000 7f9f726397c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-09T14:27:39.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:39.785+0000 7f9f726397c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-09T14:27:39.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:39.785+0000 7f9f726397c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-09T14:27:39.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:39.785+0000 7f9f726397c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-09T14:27:42.764 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-09T14:27:42.777 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:42.857+0000 7fd6b9b647c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-09T14:27:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:42.857+0000 7fd6b9b647c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-09T14:27:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:42.857+0000 7fd6b9b647c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-09T14:27:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:42.857+0000 7fd6b9b647c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-09T14:27:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:42.857+0000 7fd6b9b647c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-09T14:27:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:42.857+0000 7fd6b9b647c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-09T14:27:45.823 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-09T14:27:45.876 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:45.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:45.956+0000 7fa03a0e17c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-09T14:27:45.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:45.960+0000 7fa03a0e17c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-09T14:27:45.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:45.960+0000 7fa03a0e17c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-09T14:27:45.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:45.960+0000 7fa03a0e17c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-09T14:27:45.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:45.960+0000 7fa03a0e17c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-09T14:27:45.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:45.960+0000 7fa03a0e17c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-09T14:27:48.944 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-09T14:27:49.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:49.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:49.084+0000 7fad903d37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-09T14:27:49.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:49.084+0000 7fad903d37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-09T14:27:49.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:49.084+0000 7fad903d37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-09T14:27:49.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:49.084+0000 7fad903d37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-09T14:27:49.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:49.084+0000 7fad903d37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-09T14:27:49.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-09T14:27:49.084+0000 7fad903d37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-09T14:27:52.062 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-09T14:27:52.117 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-09T14:27:52.117 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:52.117 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-09T14:27:52.131 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:52.131 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-09T14:27:52.185 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:52.185 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-09T14:27:52.245 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:52.245 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-09T14:27:52.302 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:52.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-09T14:27:52.357 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:52.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-09T14:27:52.372 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:52.372 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-09T14:27:52.428 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:52.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-09T14:27:52.484 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:52.484 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-09T14:27:52.540 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:52.540 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-09T14:27:52.596 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:52.596 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-09T14:27:52.602 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:52.602 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-09T14:27:52.644 INFO:tasks.ceph:Adding keys to all mons... 2024-03-09T14:27:52.644 DEBUG:teuthology.orchestra.run.smithi001:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-09T14:27:52.646 DEBUG:teuthology.orchestra.run.smithi053:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-09T14:27:52.690 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.y] 2024-03-09T14:27:52.690 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDNcexll0P2DRAAKcdXxgBWd1grCbQP7pffxw== 2024-03-09T14:27:52.690 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.0] 2024-03-09T14:27:52.691 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDOcexlUnzbGhAAYKlW6Hoc6djGoMM48pgHgQ== 2024-03-09T14:27:52.691 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.1] 2024-03-09T14:27:52.691 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDRcexlkfyWHhAAsl/dJ3a5PN2PU5ReuKNQ1A== 2024-03-09T14:27:52.691 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.2] 2024-03-09T14:27:52.691 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDUcexlL0ClJBAA+gN742+7Gr0bj2uYWcI4ug== 2024-03-09T14:27:52.691 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.3] 2024-03-09T14:27:52.691 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDXcexlsbKUKhAAZI32aHgY7fRlXyPysg04wQ== 2024-03-09T14:27:52.691 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.x] 2024-03-09T14:27:52.691 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDNcexlO1fmEBAAEmDj+NxEQ2dR3iUff4ulLQ== 2024-03-09T14:27:52.692 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.4] 2024-03-09T14:27:52.692 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDbcexl/UXlLhAAPQ+obO/Jt1rKHSC4BK9gIw== 2024-03-09T14:27:52.692 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.5] 2024-03-09T14:27:52.692 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDecexlxG5MMxAA4DN9y3gAiGXyrh2uVSIKww== 2024-03-09T14:27:52.692 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.6] 2024-03-09T14:27:52.692 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDhcexlTtdPORAAorm8iQmccwJ+vDmQDCq3dA== 2024-03-09T14:27:52.692 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.7] 2024-03-09T14:27:52.692 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDlcexldpAvBRAAcQR/oIrzg9DKYGiXirtOZg== 2024-03-09T14:27:52.692 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-09T14:27:52.693 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDNcexlGkWwExAAoygxXwvK3jESwnGVi2uNEw== 2024-03-09T14:27:52.693 INFO:teuthology.orchestra.run.smithi001.stdout:[client.1] 2024-03-09T14:27:52.693 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDNcexlG4uVFhAANm77U4aQzk5pv9RrUlNkOQ== 2024-03-09T14:27:52.698 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.y] 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDNcexll0P2DRAAKcdXxgBWd1grCbQP7pffxw== 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.0] 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDOcexlUnzbGhAAYKlW6Hoc6djGoMM48pgHgQ== 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.1] 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDRcexlkfyWHhAAsl/dJ3a5PN2PU5ReuKNQ1A== 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.2] 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDUcexlL0ClJBAA+gN742+7Gr0bj2uYWcI4ug== 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.3] 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDXcexlsbKUKhAAZI32aHgY7fRlXyPysg04wQ== 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.x] 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDNcexlO1fmEBAAEmDj+NxEQ2dR3iUff4ulLQ== 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.4] 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDbcexl/UXlLhAAPQ+obO/Jt1rKHSC4BK9gIw== 2024-03-09T14:27:52.699 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.5] 2024-03-09T14:27:52.700 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDecexlxG5MMxAA4DN9y3gAiGXyrh2uVSIKww== 2024-03-09T14:27:52.700 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.6] 2024-03-09T14:27:52.700 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDhcexlTtdPORAAorm8iQmccwJ+vDmQDCq3dA== 2024-03-09T14:27:52.700 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.7] 2024-03-09T14:27:52.700 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDlcexldpAvBRAAcQR/oIrzg9DKYGiXirtOZg== 2024-03-09T14:27:52.700 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-03-09T14:27:52.700 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDNcexlGkWwExAAoygxXwvK3jESwnGVi2uNEw== 2024-03-09T14:27:52.700 INFO:teuthology.orchestra.run.smithi053.stdout:[client.1] 2024-03-09T14:27:52.700 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDNcexlG4uVFhAANm77U4aQzk5pv9RrUlNkOQ== 2024-03-09T14:27:52.701 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-09T14:27:52.739 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-09T14:27:52.790 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:52.792 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:52.838 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:52.878 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:52.921 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:52.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:53.005 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:53.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:53.091 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-09T14:27:53.135 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-09T14:27:53.179 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:53.222 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:53.266 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:53.311 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:53.355 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:53.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:53.443 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:53.445 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-09T14:27:53.524 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-09T14:27:53.530 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-09T14:27:53.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-09T14:27:53.615 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-09T14:27:53.663 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-09T14:27:53.663 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-09T14:27:53.674 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-09T14:27:53.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-09T14:27:53.815 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-09T14:27:53.873 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-09T14:27:53.970 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-09T14:27:53.981 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-09T14:27:53.996 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-09T14:27:54.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-09T14:27:54.102 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:54.105 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-09T14:27:54.147 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-09T14:27:54.147 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-09T14:27:54.147 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-09T14:27:54.150 INFO:tasks.ceph.mon.a:Started 2024-03-09T14:27:54.150 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-09T14:27:54.151 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-09T14:27:54.153 INFO:tasks.ceph.mon.c:Started 2024-03-09T14:27:54.153 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-09T14:27:54.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-09T14:27:54.193 INFO:tasks.ceph.mon.b:Started 2024-03-09T14:27:54.193 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-09T14:27:54.193 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-09T14:27:54.193 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-09T14:27:54.198 INFO:tasks.ceph.mgr.y:Started 2024-03-09T14:27:54.198 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-09T14:27:54.198 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-09T14:27:54.200 INFO:tasks.ceph.mgr.x:Started 2024-03-09T14:27:54.200 INFO:tasks.ceph:Setting crush tunables to default 2024-03-09T14:27:54.200 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd crush tunables default 2024-03-09T14:27:59.632 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-03-09T14:27:59.651 INFO:tasks.ceph:check_enable_crimson: False 2024-03-09T14:27:59.651 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-09T14:27:59.651 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:59.651 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-09T14:27:59.662 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:59.662 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-09T14:27:59.714 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:59.714 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-09T14:27:59.769 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:27:59.769 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-09T14:27:59.825 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:59.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-09T14:27:59.839 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:59.840 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-09T14:27:59.899 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:59.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-09T14:27:59.960 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-09T14:27:59.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-09T14:28:00.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 94cb6315-8a11-4b66-8842-2d1925ae9e28 0 2024-03-09T14:28:00.031 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:00.028+0000 7f5cbaacb180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-09T14:28:00.253 INFO:teuthology.orchestra.run.smithi053.stdout:0 2024-03-09T14:28:00.271 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new c5007a43-9e9d-4bc4-bc3f-d8996bf2d160 1 2024-03-09T14:28:00.475 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-03-09T14:28:00.493 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 8ef94bd2-a269-44a0-8870-eecdf5a35a1d 2 2024-03-09T14:28:00.530 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:00.527+0000 7f01c1873180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-09T14:28:00.598 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:00.596+0000 7f5cbaacb180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-09T14:28:00.697 INFO:teuthology.orchestra.run.smithi053.stdout:2 2024-03-09T14:28:00.714 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 878834c0-0f19-4e43-bea6-4e8b4f972c67 3 2024-03-09T14:28:00.811 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:00.812+0000 7f5cbaacb180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-09T14:28:00.907 INFO:teuthology.orchestra.run.smithi053.stdout:3 2024-03-09T14:28:00.922 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 787d3950-2e1f-4714-af31-12d811b3b8c2 4 2024-03-09T14:28:01.030 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:01.027+0000 7f01c1873180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-09T14:28:01.118 INFO:teuthology.orchestra.run.smithi053.stdout:4 2024-03-09T14:28:01.135 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 8cb1f2fa-3eec-4750-a028-b8b3c07846d2 5 2024-03-09T14:28:01.231 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:01.227+0000 7f01c1873180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-09T14:28:01.325 INFO:teuthology.orchestra.run.smithi053.stdout:5 2024-03-09T14:28:01.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 094803a5-1cb6-452c-abe9-890105c65516 6 2024-03-09T14:28:01.394 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:01.392+0000 7f5cbaacb180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-09T14:28:01.452 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:01.452+0000 7f5cbaacb180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-09T14:28:01.509 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:01.508+0000 7f5cbaacb180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-09T14:28:01.543 INFO:teuthology.orchestra.run.smithi053.stdout:6 2024-03-09T14:28:01.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 6bbf6c2e-f1fe-4644-918d-53a25c557cb3 7 2024-03-09T14:28:01.592 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:01.592+0000 7f5cbaacb180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-09T14:28:01.648 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:01.648+0000 7f5cbaacb180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-09T14:28:01.732 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:01.732+0000 7f5cbaacb180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-09T14:28:01.755 INFO:teuthology.orchestra.run.smithi053.stdout:7 2024-03-09T14:28:01.772 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-09T14:28:01.772 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-09T14:28:01.774 INFO:tasks.ceph.osd.0:Started 2024-03-09T14:28:01.775 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-09T14:28:01.775 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-09T14:28:01.777 INFO:tasks.ceph.osd.1:Started 2024-03-09T14:28:01.777 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-09T14:28:01.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-09T14:28:01.779 INFO:tasks.ceph.osd.2:Started 2024-03-09T14:28:01.779 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-09T14:28:01.780 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-09T14:28:01.782 INFO:tasks.ceph.osd.3:Started 2024-03-09T14:28:01.782 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-09T14:28:01.782 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-09T14:28:01.789 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:01.788+0000 7f5cbaacb180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-09T14:28:01.790 INFO:tasks.ceph.osd.4:Started 2024-03-09T14:28:01.790 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-09T14:28:01.791 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-09T14:28:01.793 INFO:tasks.ceph.osd.5:Started 2024-03-09T14:28:01.793 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-09T14:28:01.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-09T14:28:01.795 INFO:tasks.ceph.osd.6:Started 2024-03-09T14:28:01.795 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-09T14:28:01.795 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-09T14:28:01.797 INFO:tasks.ceph.osd.7:Started 2024-03-09T14:28:01.797 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T14:28:01.808 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:01.803+0000 7f01c1873180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-09T14:28:01.889 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:01.887+0000 7f01c1873180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-09T14:28:01.948 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:01.943+0000 7f01c1873180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-09T14:28:01.970 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:01.968+0000 7f5cbaacb180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-09T14:28:01.994 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:28:01.994 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":10,"fsid":"fe441b42-34a9-46d6-87e9-ac75455181ab","created":"2024-03-09T14:27:59.514149+0000","modified":"2024-03-09T14:28:01.753379+0000","last_up_change":"0.000000","last_in_change":"2024-03-09T14:28:01.753379+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"94cb6315-8a11-4b66-8842-2d1925ae9e28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5007a43-9e9d-4bc4-bc3f-d8996bf2d160","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8ef94bd2-a269-44a0-8870-eecdf5a35a1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"878834c0-0f19-4e43-bea6-4e8b4f972c67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"787d3950-2e1f-4714-af31-12d811b3b8c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8cb1f2fa-3eec-4750-a028-b8b3c07846d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"094803a5-1cb6-452c-abe9-890105c65516","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6bbf6c2e-f1fe-4644-918d-53a25c557cb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T14:28:02.014 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-09T14:28:02.014 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-09T14:28:02.032 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:02.027+0000 7f01c1873180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-09T14:28:02.035 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:02.032+0000 7f5cbaacb180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-09T14:28:02.089 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:02.087+0000 7f01c1873180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-09T14:28:02.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:02.167+0000 7f01c1873180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-09T14:28:02.226 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:02.223+0000 7f01c1873180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-09T14:28:02.382 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:02.379+0000 7f01c1873180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-09T14:28:02.413 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-09T14:28:02.412+0000 7f0fdde347c0 -1 Falling back to public interface 2024-03-09T14:28:02.413 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-09T14:28:02.412+0000 7f005889e7c0 -1 Falling back to public interface 2024-03-09T14:28:02.413 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-09T14:28:02.412+0000 7f005785e640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-03-09T14:28:02.415 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-09T14:28:02.411+0000 7f4848c937c0 -1 Falling back to public interface 2024-03-09T14:28:02.416 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-09T14:28:02.411+0000 7ff37db5a7c0 -1 Falling back to public interface 2024-03-09T14:28:02.416 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-09T14:28:02.411+0000 7fb1bf1ce7c0 -1 Falling back to public interface 2024-03-09T14:28:02.428 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-09T14:28:02.428+0000 7fcceeeaa7c0 -1 Falling back to public interface 2024-03-09T14:28:02.430 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-09T14:28:02.427+0000 7fa83904f7c0 -1 Falling back to public interface 2024-03-09T14:28:02.444 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-09T14:28:02.444+0000 7f65e2be07c0 -1 Falling back to public interface 2024-03-09T14:28:02.449 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:02.447+0000 7f01c1873180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-09T14:28:02.705 INFO:tasks.ceph.mgr.x.smithi053.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-09T14:28:02.705 INFO:tasks.ceph.mgr.x.smithi053.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-09T14:28:02.705 INFO:tasks.ceph.mgr.x.smithi053.stderr: from numpy import show_config as show_numpy_config 2024-03-09T14:28:02.751 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:02.748+0000 7f5cbaacb180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-09T14:28:02.821 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-09T14:28:02.892 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:02.892+0000 7f5cbaacb180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-09T14:28:03.006 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-09T14:28:03.006 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"fe441b42-34a9-46d6-87e9-ac75455181ab","created":"2024-03-09T14:27:59.514149+0000","modified":"2024-03-09T14:28:01.753379+0000","last_up_change":"0.000000","last_in_change":"2024-03-09T14:28:01.753379+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"94cb6315-8a11-4b66-8842-2d1925ae9e28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5007a43-9e9d-4bc4-bc3f-d8996bf2d160","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8ef94bd2-a269-44a0-8870-eecdf5a35a1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"878834c0-0f19-4e43-bea6-4e8b4f972c67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"787d3950-2e1f-4714-af31-12d811b3b8c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8cb1f2fa-3eec-4750-a028-b8b3c07846d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"094803a5-1cb6-452c-abe9-890105c65516","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6bbf6c2e-f1fe-4644-918d-53a25c557cb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T14:28:03.021 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-09T14:28:03.060 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:03.060+0000 7f5cbaacb180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-09T14:28:03.070 INFO:tasks.ceph.mgr.y.smithi001.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-09T14:28:03.070 INFO:tasks.ceph.mgr.y.smithi001.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-09T14:28:03.070 INFO:tasks.ceph.mgr.y.smithi001.stderr: from numpy import show_config as show_numpy_config 2024-03-09T14:28:03.117 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:03.115+0000 7f01c1873180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-09T14:28:03.127 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:03.124+0000 7f5cbaacb180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-09T14:28:03.256 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:03.251+0000 7f01c1873180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-09T14:28:03.266 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:03.264+0000 7f5cbaacb180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-09T14:28:03.422 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:03.419+0000 7f01c1873180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-09T14:28:03.488 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:03.483+0000 7f01c1873180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-09T14:28:03.629 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:03.623+0000 7f01c1873180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-09T14:28:03.894 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:03.892+0000 7f5cbaacb180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-09T14:28:03.973 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:03.972+0000 7f5cbaacb180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-09T14:28:04.161 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:04.160+0000 7f5cbaacb180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-09T14:28:04.238 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:04.236+0000 7f5cbaacb180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-09T14:28:04.267 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:04.263+0000 7f01c1873180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-09T14:28:04.346 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:04.343+0000 7f01c1873180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-09T14:28:04.536 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:04.531+0000 7f01c1873180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-09T14:28:04.542 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:04.540+0000 7f5cbaacb180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-09T14:28:04.607 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:04.608+0000 7f5cbaacb180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-09T14:28:04.613 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:04.611+0000 7f01c1873180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-09T14:28:04.938 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:04.935+0000 7f01c1873180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-09T14:28:05.013 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-09T14:28:05.011+0000 7f01c1873180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-09T14:28:05.648 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:05.648+0000 7f5c61f19640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-09T14:28:05.648 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-09T14:28:05.648+0000 7f5c61f19640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-09T14:28:06.062 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-09T14:28:06.059+0000 7fa83904f7c0 -1 osd.2 0 log_to_monitors true 2024-03-09T14:28:06.069 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-09T14:28:06.068+0000 7f005889e7c0 -1 osd.7 0 log_to_monitors true 2024-03-09T14:28:06.093 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-09T14:28:06.092+0000 7f0fdde347c0 -1 osd.6 0 log_to_monitors true 2024-03-09T14:28:06.128 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-09T14:28:06.123+0000 7fb1bf1ce7c0 -1 osd.1 0 log_to_monitors true 2024-03-09T14:28:06.265 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-09T14:28:06.259+0000 7ff37db5a7c0 -1 osd.0 0 log_to_monitors true 2024-03-09T14:28:06.277 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-09T14:28:06.275+0000 7f4848c937c0 -1 osd.3 0 log_to_monitors true 2024-03-09T14:28:06.292 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-09T14:28:06.292+0000 7fcceeeaa7c0 -1 osd.4 0 log_to_monitors true 2024-03-09T14:28:06.377 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-09T14:28:06.376+0000 7f65e2be07c0 -1 osd.5 0 log_to_monitors true 2024-03-09T14:28:09.174 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-09T14:28:09.172+0000 7f005585a640 -1 osd.7 0 waiting for initial osdmap 2024-03-09T14:28:09.204 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-09T14:28:09.204+0000 7f004c66b640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T14:28:09.239 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-09T14:28:09.235+0000 7fa8347d5640 -1 osd.2 0 waiting for initial osdmap 2024-03-09T14:28:09.245 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-09T14:28:09.243+0000 7f4844419640 -1 osd.3 0 waiting for initial osdmap 2024-03-09T14:28:09.259 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-09T14:28:09.260+0000 7fccea630640 -1 osd.4 0 waiting for initial osdmap 2024-03-09T14:28:09.274 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-09T14:28:09.271+0000 7f483ca60640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T14:28:09.276 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-09T14:28:09.271+0000 7fb1ba954640 -1 osd.1 0 waiting for initial osdmap 2024-03-09T14:28:09.284 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-09T14:28:09.284+0000 7fcce2c77640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T14:28:09.289 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-09T14:28:09.287+0000 7fa82ce1c640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T14:28:09.300 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-09T14:28:09.295+0000 7ff3792e0640 -1 osd.0 0 waiting for initial osdmap 2024-03-09T14:28:09.309 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-09T14:28:09.307+0000 7fb1b2f9b640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T14:28:09.321 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-09T14:28:09.320+0000 7f0fdadf0640 -1 osd.6 0 waiting for initial osdmap 2024-03-09T14:28:09.321 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-09T14:28:09.319+0000 7ff371927640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T14:28:09.336 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-09T14:28:09.336+0000 7f0fd1c01640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T14:28:09.353 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-09T14:28:09.352+0000 7f65dfb9c640 -1 osd.5 0 waiting for initial osdmap 2024-03-09T14:28:09.380 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-09T14:28:09.380+0000 7f65d69ad640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T14:28:09.828 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-09T14:28:10.124 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-09T14:28:10.124 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":12,"fsid":"fe441b42-34a9-46d6-87e9-ac75455181ab","created":"2024-03-09T14:27:59.514149+0000","modified":"2024-03-09T14:28:07.695016+0000","last_up_change":"0.000000","last_in_change":"2024-03-09T14:28:01.753379+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"94cb6315-8a11-4b66-8842-2d1925ae9e28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5007a43-9e9d-4bc4-bc3f-d8996bf2d160","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8ef94bd2-a269-44a0-8870-eecdf5a35a1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"878834c0-0f19-4e43-bea6-4e8b4f972c67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"787d3950-2e1f-4714-af31-12d811b3b8c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8cb1f2fa-3eec-4750-a028-b8b3c07846d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"094803a5-1cb6-452c-abe9-890105c65516","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6bbf6c2e-f1fe-4644-918d-53a25c557cb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T14:28:10.141 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-09T14:28:16.947 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-09T14:28:17.250 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-09T14:28:17.250 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":16,"fsid":"fe441b42-34a9-46d6-87e9-ac75455181ab","created":"2024-03-09T14:27:59.514149+0000","modified":"2024-03-09T14:28:13.190829+0000","last_up_change":"2024-03-09T14:28:10.177363+0000","last_in_change":"2024-03-09T14:28:01.753379+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-09T14:28:10.677796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94cb6315-8a11-4b66-8842-2d1925ae9e28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6809","nonce":3097686011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6813","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6815","nonce":3097686011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6823","nonce":3097686011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6819","nonce":3097686011}]},"public_addr":"172.21.15.1:6809/3097686011","cluster_addr":"172.21.15.1:6815/3097686011","heartbeat_back_addr":"172.21.15.1:6823/3097686011","heartbeat_front_addr":"172.21.15.1:6819/3097686011","state":["exists","up"]},{"osd":1,"uuid":"c5007a43-9e9d-4bc4-bc3f-d8996bf2d160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6805","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6808","nonce":3585289272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6814","nonce":3585289272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6822","nonce":3585289272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6818","nonce":3585289272}]},"public_addr":"172.21.15.1:6808/3585289272","cluster_addr":"172.21.15.1:6814/3585289272","heartbeat_back_addr":"172.21.15.1:6822/3585289272","heartbeat_front_addr":"172.21.15.1:6818/3585289272","state":["exists","up"]},{"osd":2,"uuid":"8ef94bd2-a269-44a0-8870-eecdf5a35a1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6825","nonce":1666056338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6827","nonce":1666056338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6831","nonce":1666056338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6829","nonce":1666056338}]},"public_addr":"172.21.15.1:6825/1666056338","cluster_addr":"172.21.15.1:6827/1666056338","heartbeat_back_addr":"172.21.15.1:6831/1666056338","heartbeat_front_addr":"172.21.15.1:6829/1666056338","state":["exists","up"]},{"osd":3,"uuid":"878834c0-0f19-4e43-bea6-4e8b4f972c67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6801","nonce":4091251076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6803","nonce":4091251076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6812","nonce":4091251076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6807","nonce":4091251076}]},"public_addr":"172.21.15.1:6801/4091251076","cluster_addr":"172.21.15.1:6803/4091251076","heartbeat_back_addr":"172.21.15.1:6812/4091251076","heartbeat_front_addr":"172.21.15.1:6807/4091251076","state":["exists","up"]},{"osd":4,"uuid":"787d3950-2e1f-4714-af31-12d811b3b8c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6817","nonce":859385596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6819","nonce":859385596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6823","nonce":859385596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6821","nonce":859385596}]},"public_addr":"172.21.15.53:6817/859385596","cluster_addr":"172.21.15.53:6819/859385596","heartbeat_back_addr":"172.21.15.53:6823/859385596","heartbeat_front_addr":"172.21.15.53:6821/859385596","state":["exists","up"]},{"osd":5,"uuid":"8cb1f2fa-3eec-4750-a028-b8b3c07846d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6825","nonce":256097109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6827","nonce":256097109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6831","nonce":256097109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6829","nonce":256097109}]},"public_addr":"172.21.15.53:6825/256097109","cluster_addr":"172.21.15.53:6827/256097109","heartbeat_back_addr":"172.21.15.53:6831/256097109","heartbeat_front_addr":"172.21.15.53:6829/256097109","state":["exists","up"]},{"osd":6,"uuid":"094803a5-1cb6-452c-abe9-890105c65516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6805","nonce":1545189291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6807","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6809","nonce":1545189291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6815","nonce":1545189291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6813","nonce":1545189291}]},"public_addr":"172.21.15.53:6805/1545189291","cluster_addr":"172.21.15.53:6809/1545189291","heartbeat_back_addr":"172.21.15.53:6815/1545189291","heartbeat_front_addr":"172.21.15.53:6813/1545189291","state":["exists","up"]},{"osd":7,"uuid":"6bbf6c2e-f1fe-4644-918d-53a25c557cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6801","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6802","nonce":315236776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6803","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6804","nonce":315236776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6812","nonce":315236776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6808","nonce":315236776}]},"public_addr":"172.21.15.53:6802/315236776","cluster_addr":"172.21.15.53:6804/315236776","heartbeat_back_addr":"172.21.15.53:6812/315236776","heartbeat_front_addr":"172.21.15.53:6808/315236776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.295939+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.158214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.073987+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.239564+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.283109+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.389164+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.144938+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.031541+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T14:28:17.266 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-09T14:28:17.267 INFO:tasks.ceph:Creating RBD pool 2024-03-09T14:28:17.267 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-09T14:28:18.215 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'rbd' created 2024-03-09T14:28:18.239 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-09T14:28:19.231 INFO:teuthology.orchestra.run.smithi001.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-09T14:28:19.257 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-09T14:28:19.257 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-09T14:28:19.257 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-09T14:28:19.257 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-09T14:28:19.258 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-09T14:28:19.592 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:28:19.609 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1277640141},{"type":"v1","addr":"172.21.15.53:6833","nonce":1277640141}]},"active_addr":"172.21.15.53:6833/1277640141","active_change":"2024-03-09T14:28:04.629862+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4109,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1722149062}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1348701381}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1234332213}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":621674302}]}]} 2024-03-09T14:28:19.613 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-09T14:28:19.613 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-09T14:28:19.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T14:28:19.901 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:28:19.901 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":18,"fsid":"fe441b42-34a9-46d6-87e9-ac75455181ab","created":"2024-03-09T14:27:59.514149+0000","modified":"2024-03-09T14:28:19.215865+0000","last_up_change":"2024-03-09T14:28:10.177363+0000","last_in_change":"2024-03-09T14:28:01.753379+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-09T14:28:10.677796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-09T14:28:17.557699+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94cb6315-8a11-4b66-8842-2d1925ae9e28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6809","nonce":3097686011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6813","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6815","nonce":3097686011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6823","nonce":3097686011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6819","nonce":3097686011}]},"public_addr":"172.21.15.1:6809/3097686011","cluster_addr":"172.21.15.1:6815/3097686011","heartbeat_back_addr":"172.21.15.1:6823/3097686011","heartbeat_front_addr":"172.21.15.1:6819/3097686011","state":["exists","up"]},{"osd":1,"uuid":"c5007a43-9e9d-4bc4-bc3f-d8996bf2d160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6805","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6808","nonce":3585289272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6814","nonce":3585289272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6822","nonce":3585289272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6818","nonce":3585289272}]},"public_addr":"172.21.15.1:6808/3585289272","cluster_addr":"172.21.15.1:6814/3585289272","heartbeat_back_addr":"172.21.15.1:6822/3585289272","heartbeat_front_addr":"172.21.15.1:6818/3585289272","state":["exists","up"]},{"osd":2,"uuid":"8ef94bd2-a269-44a0-8870-eecdf5a35a1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6825","nonce":1666056338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6827","nonce":1666056338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6831","nonce":1666056338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6829","nonce":1666056338}]},"public_addr":"172.21.15.1:6825/1666056338","cluster_addr":"172.21.15.1:6827/1666056338","heartbeat_back_addr":"172.21.15.1:6831/1666056338","heartbeat_front_addr":"172.21.15.1:6829/1666056338","state":["exists","up"]},{"osd":3,"uuid":"878834c0-0f19-4e43-bea6-4e8b4f972c67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6801","nonce":4091251076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6803","nonce":4091251076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6812","nonce":4091251076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6807","nonce":4091251076}]},"public_addr":"172.21.15.1:6801/4091251076","cluster_addr":"172.21.15.1:6803/4091251076","heartbeat_back_addr":"172.21.15.1:6812/4091251076","heartbeat_front_addr":"172.21.15.1:6807/4091251076","state":["exists","up"]},{"osd":4,"uuid":"787d3950-2e1f-4714-af31-12d811b3b8c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6817","nonce":859385596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6819","nonce":859385596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6823","nonce":859385596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6821","nonce":859385596}]},"public_addr":"172.21.15.53:6817/859385596","cluster_addr":"172.21.15.53:6819/859385596","heartbeat_back_addr":"172.21.15.53:6823/859385596","heartbeat_front_addr":"172.21.15.53:6821/859385596","state":["exists","up"]},{"osd":5,"uuid":"8cb1f2fa-3eec-4750-a028-b8b3c07846d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6825","nonce":256097109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6827","nonce":256097109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6831","nonce":256097109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6829","nonce":256097109}]},"public_addr":"172.21.15.53:6825/256097109","cluster_addr":"172.21.15.53:6827/256097109","heartbeat_back_addr":"172.21.15.53:6831/256097109","heartbeat_front_addr":"172.21.15.53:6829/256097109","state":["exists","up"]},{"osd":6,"uuid":"094803a5-1cb6-452c-abe9-890105c65516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6805","nonce":1545189291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6807","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6809","nonce":1545189291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6815","nonce":1545189291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6813","nonce":1545189291}]},"public_addr":"172.21.15.53:6805/1545189291","cluster_addr":"172.21.15.53:6809/1545189291","heartbeat_back_addr":"172.21.15.53:6815/1545189291","heartbeat_front_addr":"172.21.15.53:6813/1545189291","state":["exists","up"]},{"osd":7,"uuid":"6bbf6c2e-f1fe-4644-918d-53a25c557cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6801","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6802","nonce":315236776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6803","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6804","nonce":315236776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6812","nonce":315236776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6808","nonce":315236776}]},"public_addr":"172.21.15.53:6802/315236776","cluster_addr":"172.21.15.53:6804/315236776","heartbeat_back_addr":"172.21.15.53:6812/315236776","heartbeat_front_addr":"172.21.15.53:6808/315236776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.295939+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.158214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.073987+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.239564+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.283109+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.389164+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.144938+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.031541+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T14:28:19.920 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-09T14:28:19.920 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T14:28:20.217 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:28:20.217 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":18,"fsid":"fe441b42-34a9-46d6-87e9-ac75455181ab","created":"2024-03-09T14:27:59.514149+0000","modified":"2024-03-09T14:28:19.215865+0000","last_up_change":"2024-03-09T14:28:10.177363+0000","last_in_change":"2024-03-09T14:28:01.753379+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-09T14:28:10.677796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-09T14:28:17.557699+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94cb6315-8a11-4b66-8842-2d1925ae9e28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6809","nonce":3097686011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6813","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6815","nonce":3097686011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6823","nonce":3097686011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6819","nonce":3097686011}]},"public_addr":"172.21.15.1:6809/3097686011","cluster_addr":"172.21.15.1:6815/3097686011","heartbeat_back_addr":"172.21.15.1:6823/3097686011","heartbeat_front_addr":"172.21.15.1:6819/3097686011","state":["exists","up"]},{"osd":1,"uuid":"c5007a43-9e9d-4bc4-bc3f-d8996bf2d160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6805","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6808","nonce":3585289272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6814","nonce":3585289272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6822","nonce":3585289272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6818","nonce":3585289272}]},"public_addr":"172.21.15.1:6808/3585289272","cluster_addr":"172.21.15.1:6814/3585289272","heartbeat_back_addr":"172.21.15.1:6822/3585289272","heartbeat_front_addr":"172.21.15.1:6818/3585289272","state":["exists","up"]},{"osd":2,"uuid":"8ef94bd2-a269-44a0-8870-eecdf5a35a1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6825","nonce":1666056338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6827","nonce":1666056338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6831","nonce":1666056338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6829","nonce":1666056338}]},"public_addr":"172.21.15.1:6825/1666056338","cluster_addr":"172.21.15.1:6827/1666056338","heartbeat_back_addr":"172.21.15.1:6831/1666056338","heartbeat_front_addr":"172.21.15.1:6829/1666056338","state":["exists","up"]},{"osd":3,"uuid":"878834c0-0f19-4e43-bea6-4e8b4f972c67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6801","nonce":4091251076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6803","nonce":4091251076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6812","nonce":4091251076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6807","nonce":4091251076}]},"public_addr":"172.21.15.1:6801/4091251076","cluster_addr":"172.21.15.1:6803/4091251076","heartbeat_back_addr":"172.21.15.1:6812/4091251076","heartbeat_front_addr":"172.21.15.1:6807/4091251076","state":["exists","up"]},{"osd":4,"uuid":"787d3950-2e1f-4714-af31-12d811b3b8c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6817","nonce":859385596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6819","nonce":859385596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6823","nonce":859385596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6821","nonce":859385596}]},"public_addr":"172.21.15.53:6817/859385596","cluster_addr":"172.21.15.53:6819/859385596","heartbeat_back_addr":"172.21.15.53:6823/859385596","heartbeat_front_addr":"172.21.15.53:6821/859385596","state":["exists","up"]},{"osd":5,"uuid":"8cb1f2fa-3eec-4750-a028-b8b3c07846d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6825","nonce":256097109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6827","nonce":256097109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6831","nonce":256097109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6829","nonce":256097109}]},"public_addr":"172.21.15.53:6825/256097109","cluster_addr":"172.21.15.53:6827/256097109","heartbeat_back_addr":"172.21.15.53:6831/256097109","heartbeat_front_addr":"172.21.15.53:6829/256097109","state":["exists","up"]},{"osd":6,"uuid":"094803a5-1cb6-452c-abe9-890105c65516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6805","nonce":1545189291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6807","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6809","nonce":1545189291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6815","nonce":1545189291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6813","nonce":1545189291}]},"public_addr":"172.21.15.53:6805/1545189291","cluster_addr":"172.21.15.53:6809/1545189291","heartbeat_back_addr":"172.21.15.53:6815/1545189291","heartbeat_front_addr":"172.21.15.53:6813/1545189291","state":["exists","up"]},{"osd":7,"uuid":"6bbf6c2e-f1fe-4644-918d-53a25c557cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6801","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6802","nonce":315236776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6803","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6804","nonce":315236776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6812","nonce":315236776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6808","nonce":315236776}]},"public_addr":"172.21.15.53:6802/315236776","cluster_addr":"172.21.15.53:6804/315236776","heartbeat_back_addr":"172.21.15.53:6812/315236776","heartbeat_front_addr":"172.21.15.53:6808/315236776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.295939+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.158214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.073987+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.239564+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.283109+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.389164+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.144938+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.031541+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T14:28:20.237 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-09T14:28:20.238 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-09T14:28:20.238 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-09T14:28:20.239 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-09T14:28:20.239 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-09T14:28:20.240 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-09T14:28:20.240 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-09T14:28:20.240 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-09T14:28:20.493 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:20.493 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-09T14:28:20.498 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:20.498 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-09T14:28:20.508 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:20.508 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-09T14:28:20.510 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:20.511 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-09T14:28:20.513 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-09T14:28:20.513 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-09T14:28:20.516 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-09T14:28:20.517 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-09T14:28:20.517 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:20.518 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-09T14:28:20.523 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-09T14:28:20.523 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-09T14:28:21.005 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:21.010 INFO:teuthology.orchestra.run.smithi001.stdout:55834574851 2024-03-09T14:28:21.013 INFO:teuthology.orchestra.run.smithi001.stdout:55834574851 2024-03-09T14:28:21.031 INFO:teuthology.orchestra.run.smithi001.stdout:55834574851 2024-03-09T14:28:21.035 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.7 2024-03-09T14:28:21.039 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.3 2024-03-09T14:28:21.043 INFO:teuthology.orchestra.run.smithi001.stdout:55834574851 2024-03-09T14:28:21.044 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.6 2024-03-09T14:28:21.061 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.1 2024-03-09T14:28:21.062 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.2 2024-03-09T14:28:21.070 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:21.071 INFO:teuthology.orchestra.run.smithi001.stdout:55834574851 2024-03-09T14:28:21.090 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2024-03-09T14:28:21.090 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.5 2024-03-09T14:28:21.097 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:21.113 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2024-03-09T14:28:22.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-09T14:28:22.040 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-09T14:28:22.045 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-09T14:28:22.062 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-09T14:28:22.063 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-09T14:28:22.090 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-09T14:28:22.091 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-09T14:28:22.114 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-09T14:28:22.582 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-09T14:28:22.601 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:22.601 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.7 2024-03-09T14:28:22.602 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:22.602 DEBUG:teuthology.parallel:result is None 2024-03-09T14:28:22.607 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:22.624 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.2 2024-03-09T14:28:22.624 DEBUG:teuthology.parallel:result is None 2024-03-09T14:28:22.634 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.3 2024-03-09T14:28:22.634 DEBUG:teuthology.parallel:result is None 2024-03-09T14:28:22.637 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:22.638 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.1 2024-03-09T14:28:22.638 DEBUG:teuthology.parallel:result is None 2024-03-09T14:28:22.649 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-09T14:28:22.655 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2024-03-09T14:28:22.655 DEBUG:teuthology.parallel:result is None 2024-03-09T14:28:22.660 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-09T14:28:22.668 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2024-03-09T14:28:22.668 DEBUG:teuthology.parallel:result is None 2024-03-09T14:28:22.678 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.5 2024-03-09T14:28:22.678 DEBUG:teuthology.parallel:result is None 2024-03-09T14:28:22.681 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-09T14:28:22.698 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2024-03-09T14:28:22.698 DEBUG:teuthology.parallel:result is None 2024-03-09T14:28:22.698 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-09T14:28:22.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T14:28:23.014 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:28:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-09T14:28:23.031 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-09T14:28:22.643358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":252056,"kb_used_data":2580,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722664,"statfs":{"total":773094113280,"available":772836007936,"internally_reserved":0,"allocated":2641920,"data_stored":1645012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001972"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.538659+0000","last_change":"2024-03-09T14:28:19.242561+0000","last_active":"2024-03-09T14:28:20.538659+0000","last_peered":"2024-03-09T14:28:20.538659+0000","last_clean":"2024-03-09T14:28:20.538659+0000","last_became_active":"2024-03-09T14:28:19.241700+0000","last_became_peered":"2024-03-09T14:28:19.241700+0000","last_unstale":"2024-03-09T14:28:20.538659+0000","last_undegraded":"2024-03-09T14:28:20.538659+0000","last_fullsized":"2024-03-09T14:28:20.538659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:15.462353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.227771+0000","last_change":"2024-03-09T14:28:19.238362+0000","last_active":"2024-03-09T14:28:20.227771+0000","last_peered":"2024-03-09T14:28:20.227771+0000","last_clean":"2024-03-09T14:28:20.227771+0000","last_became_active":"2024-03-09T14:28:19.237750+0000","last_became_peered":"2024-03-09T14:28:19.237750+0000","last_unstale":"2024-03-09T14:28:20.227771+0000","last_undegraded":"2024-03-09T14:28:20.227771+0000","last_fullsized":"2024-03-09T14:28:20.227771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:19.699145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.230915+0000","last_change":"2024-03-09T14:28:19.241475+0000","last_active":"2024-03-09T14:28:20.230915+0000","last_peered":"2024-03-09T14:28:20.230915+0000","last_clean":"2024-03-09T14:28:20.230915+0000","last_became_active":"2024-03-09T14:28:19.240704+0000","last_became_peered":"2024-03-09T14:28:19.240704+0000","last_unstale":"2024-03-09T14:28:20.230915+0000","last_undegraded":"2024-03-09T14:28:20.230915+0000","last_fullsized":"2024-03-09T14:28:20.230915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:07:08.993017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.227957+0000","last_change":"2024-03-09T14:28:19.236670+0000","last_active":"2024-03-09T14:28:20.227957+0000","last_peered":"2024-03-09T14:28:20.227957+0000","last_clean":"2024-03-09T14:28:20.227957+0000","last_became_active":"2024-03-09T14:28:19.236240+0000","last_became_peered":"2024-03-09T14:28:19.236240+0000","last_unstale":"2024-03-09T14:28:20.227957+0000","last_undegraded":"2024-03-09T14:28:20.227957+0000","last_fullsized":"2024-03-09T14:28:20.227957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:00:09.869968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.230656+0000","last_change":"2024-03-09T14:28:19.238720+0000","last_active":"2024-03-09T14:28:20.230656+0000","last_peered":"2024-03-09T14:28:20.230656+0000","last_clean":"2024-03-09T14:28:20.230656+0000","last_became_active":"2024-03-09T14:28:19.237937+0000","last_became_peered":"2024-03-09T14:28:19.237937+0000","last_unstale":"2024-03-09T14:28:20.230656+0000","last_undegraded":"2024-03-09T14:28:20.230656+0000","last_fullsized":"2024-03-09T14:28:20.230656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:29:01.306246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.535012+0000","last_change":"2024-03-09T14:28:19.437446+0000","last_active":"2024-03-09T14:28:20.535012+0000","last_peered":"2024-03-09T14:28:20.535012+0000","last_clean":"2024-03-09T14:28:20.535012+0000","last_became_active":"2024-03-09T14:28:19.436985+0000","last_became_peered":"2024-03-09T14:28:19.436985+0000","last_unstale":"2024-03-09T14:28:20.535012+0000","last_undegraded":"2024-03-09T14:28:20.535012+0000","last_fullsized":"2024-03-09T14:28:20.535012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:46:38.528697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.230948+0000","last_change":"2024-03-09T14:28:19.242176+0000","last_active":"2024-03-09T14:28:20.230948+0000","last_peered":"2024-03-09T14:28:20.230948+0000","last_clean":"2024-03-09T14:28:20.230948+0000","last_became_active":"2024-03-09T14:28:19.241110+0000","last_became_peered":"2024-03-09T14:28:19.241110+0000","last_unstale":"2024-03-09T14:28:20.230948+0000","last_undegraded":"2024-03-09T14:28:20.230948+0000","last_fullsized":"2024-03-09T14:28:20.230948+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:28:52.238193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.230625+0000","last_change":"2024-03-09T14:28:19.233336+0000","last_active":"2024-03-09T14:28:20.230625+0000","last_peered":"2024-03-09T14:28:20.230625+0000","last_clean":"2024-03-09T14:28:20.230625+0000","last_became_active":"2024-03-09T14:28:19.232517+0000","last_became_peered":"2024-03-09T14:28:19.232517+0000","last_unstale":"2024-03-09T14:28:20.230625+0000","last_undegraded":"2024-03-09T14:28:20.230625+0000","last_fullsized":"2024-03-09T14:28:20.230625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:53:39.430154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.230481+0000","last_change":"2024-03-09T14:28:12.265518+0000","last_active":"2024-03-09T14:28:20.230481+0000","last_peered":"2024-03-09T14:28:20.230481+0000","last_clean":"2024-03-09T14:28:20.230481+0000","last_became_active":"2024-03-09T14:28:12.262430+0000","last_became_peered":"2024-03-09T14:28:12.262430+0000","last_unstale":"2024-03-09T14:28:20.230481+0000","last_undegraded":"2024-03-09T14:28:20.230481+0000","last_fullsized":"2024-03-09T14:28:20.230481+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_clean_scrub_stamp":"2024-03-09T14:28:11.179640+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:14:17.888044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":774144,"data_stored":647684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":180224,"data_stored":57316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":192512,"data_stored":63064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":180224,"data_stored":57316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":180224,"data_stored":57316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":180224,"data_stored":57316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":180224,"data_stored":57316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":774144,"data_stored":647684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T14:28:23.031 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T14:28:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:28:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-09T14:28:23.336 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-09T14:28:22.643358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":252056,"kb_used_data":2580,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722664,"statfs":{"total":773094113280,"available":772836007936,"internally_reserved":0,"allocated":2641920,"data_stored":1645012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001972"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.538659+0000","last_change":"2024-03-09T14:28:19.242561+0000","last_active":"2024-03-09T14:28:20.538659+0000","last_peered":"2024-03-09T14:28:20.538659+0000","last_clean":"2024-03-09T14:28:20.538659+0000","last_became_active":"2024-03-09T14:28:19.241700+0000","last_became_peered":"2024-03-09T14:28:19.241700+0000","last_unstale":"2024-03-09T14:28:20.538659+0000","last_undegraded":"2024-03-09T14:28:20.538659+0000","last_fullsized":"2024-03-09T14:28:20.538659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:15.462353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.227771+0000","last_change":"2024-03-09T14:28:19.238362+0000","last_active":"2024-03-09T14:28:20.227771+0000","last_peered":"2024-03-09T14:28:20.227771+0000","last_clean":"2024-03-09T14:28:20.227771+0000","last_became_active":"2024-03-09T14:28:19.237750+0000","last_became_peered":"2024-03-09T14:28:19.237750+0000","last_unstale":"2024-03-09T14:28:20.227771+0000","last_undegraded":"2024-03-09T14:28:20.227771+0000","last_fullsized":"2024-03-09T14:28:20.227771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:19.699145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.230915+0000","last_change":"2024-03-09T14:28:19.241475+0000","last_active":"2024-03-09T14:28:20.230915+0000","last_peered":"2024-03-09T14:28:20.230915+0000","last_clean":"2024-03-09T14:28:20.230915+0000","last_became_active":"2024-03-09T14:28:19.240704+0000","last_became_peered":"2024-03-09T14:28:19.240704+0000","last_unstale":"2024-03-09T14:28:20.230915+0000","last_undegraded":"2024-03-09T14:28:20.230915+0000","last_fullsized":"2024-03-09T14:28:20.230915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:07:08.993017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.227957+0000","last_change":"2024-03-09T14:28:19.236670+0000","last_active":"2024-03-09T14:28:20.227957+0000","last_peered":"2024-03-09T14:28:20.227957+0000","last_clean":"2024-03-09T14:28:20.227957+0000","last_became_active":"2024-03-09T14:28:19.236240+0000","last_became_peered":"2024-03-09T14:28:19.236240+0000","last_unstale":"2024-03-09T14:28:20.227957+0000","last_undegraded":"2024-03-09T14:28:20.227957+0000","last_fullsized":"2024-03-09T14:28:20.227957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:00:09.869968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.230656+0000","last_change":"2024-03-09T14:28:19.238720+0000","last_active":"2024-03-09T14:28:20.230656+0000","last_peered":"2024-03-09T14:28:20.230656+0000","last_clean":"2024-03-09T14:28:20.230656+0000","last_became_active":"2024-03-09T14:28:19.237937+0000","last_became_peered":"2024-03-09T14:28:19.237937+0000","last_unstale":"2024-03-09T14:28:20.230656+0000","last_undegraded":"2024-03-09T14:28:20.230656+0000","last_fullsized":"2024-03-09T14:28:20.230656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:29:01.306246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.535012+0000","last_change":"2024-03-09T14:28:19.437446+0000","last_active":"2024-03-09T14:28:20.535012+0000","last_peered":"2024-03-09T14:28:20.535012+0000","last_clean":"2024-03-09T14:28:20.535012+0000","last_became_active":"2024-03-09T14:28:19.436985+0000","last_became_peered":"2024-03-09T14:28:19.436985+0000","last_unstale":"2024-03-09T14:28:20.535012+0000","last_undegraded":"2024-03-09T14:28:20.535012+0000","last_fullsized":"2024-03-09T14:28:20.535012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:46:38.528697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.230948+0000","last_change":"2024-03-09T14:28:19.242176+0000","last_active":"2024-03-09T14:28:20.230948+0000","last_peered":"2024-03-09T14:28:20.230948+0000","last_clean":"2024-03-09T14:28:20.230948+0000","last_became_active":"2024-03-09T14:28:19.241110+0000","last_became_peered":"2024-03-09T14:28:19.241110+0000","last_unstale":"2024-03-09T14:28:20.230948+0000","last_undegraded":"2024-03-09T14:28:20.230948+0000","last_fullsized":"2024-03-09T14:28:20.230948+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:28:52.238193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.230625+0000","last_change":"2024-03-09T14:28:19.233336+0000","last_active":"2024-03-09T14:28:20.230625+0000","last_peered":"2024-03-09T14:28:20.230625+0000","last_clean":"2024-03-09T14:28:20.230625+0000","last_became_active":"2024-03-09T14:28:19.232517+0000","last_became_peered":"2024-03-09T14:28:19.232517+0000","last_unstale":"2024-03-09T14:28:20.230625+0000","last_undegraded":"2024-03-09T14:28:20.230625+0000","last_fullsized":"2024-03-09T14:28:20.230625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:53:39.430154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-09T14:28:20.230481+0000","last_change":"2024-03-09T14:28:12.265518+0000","last_active":"2024-03-09T14:28:20.230481+0000","last_peered":"2024-03-09T14:28:20.230481+0000","last_clean":"2024-03-09T14:28:20.230481+0000","last_became_active":"2024-03-09T14:28:12.262430+0000","last_became_peered":"2024-03-09T14:28:12.262430+0000","last_unstale":"2024-03-09T14:28:20.230481+0000","last_undegraded":"2024-03-09T14:28:20.230481+0000","last_fullsized":"2024-03-09T14:28:20.230481+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_clean_scrub_stamp":"2024-03-09T14:28:11.179640+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:14:17.888044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":774144,"data_stored":647684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":180224,"data_stored":57316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":192512,"data_stored":63064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":180224,"data_stored":57316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":180224,"data_stored":57316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":180224,"data_stored":57316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":180224,"data_stored":57316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":774144,"data_stored":647684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T14:28:23.337 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-09T14:28:23.337 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-09T14:28:23.337 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-09T14:28:23.337 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-09T14:28:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:28:23.653 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-09T14:28:23.670 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-09T14:28:23.671 INFO:teuthology.run_tasks:Running task openssl_keys... 2024-03-09T14:28:23.683 DEBUG:teuthology.task:Applying overrides for task openssl_keys: {'rgw.client.0': {'ca': 'root', 'client': 'client.0', 'embed-key': True}, 'root': {'client': 'client.0', 'cn': 'teuthology', 'install': ['client.0'], 'key-type': 'rsa:4096'}} 2024-03-09T14:28:23.683 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p /home/ubuntu/cephtest/ca 2024-03-09T14:28:23.688 DEBUG:teuthology.orchestra.run.smithi001:> openssl req -x509 -nodes -newkey rsa:4096 -keyout /home/ubuntu/cephtest/ca/root.key -days 365 -out /home/ubuntu/cephtest/ca/root.crt -subj /CN=teuthology 2024-03-09T14:28:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:....+..+.............+...+..+...+...+......+.+..+......+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.........+.+..+.......+..+...+...+............................+........................+...+.....+...+......+.+...+...........+.+........+....+..+.........+...............+................+...+.....+.+............+..+.......+......+...............+..+.+.....+......+...+.............+.....+............+...+....+...+......+..........................+.+...+..............+......+.+........+...+.......+..+......+.......+......+..+....+.....+....+.........+.................+....+......+.........+......+.....+...............+................+..+..................+.............+..............+.+...........+.+.........+..................+..................+..+.+..+.........+...+.............+..+...+...+....+............+...........+............+..........+......+.....+....+.....+................+..+....+.....................+..+............+.+.....+..........+..+.......+..............+......+.............+...............+...+.........+...+......+......+..............+.......+.....+.........+....+.....+................+........+.+.....+...............................+....................+..........+...........+............+.+...........+....+......+.........+.....+.........+...............+......+.......+..+............+......+.........+.........................+......+.........+...+...+........+...................+..+.+.........+.........+..+...+....+..+.+.....+.......+............+.........+...........+....+......+.....+...+.+..+.......+........+...+....+..+....+.........+.....+...+..................+.+......+..+.+.....+.+.........+.................+.......+...+.........+.........+.....+..........+..+.........+....+..+.......+.....+...+....+......+...+.....+............+.......+...........+.+......+..............+..........+........................+......+.....+.+..............+...+............+.+..................+..+.............+..+.......+..+.........+.+...+.....+......................+.....+.+...+.....+.......+.....+.........+.........+.......+..+...+....+........+...............+.+..............+.....................+.+.....+.+.....+.......+...........+......+...+.+.................+.............+..+.......+..+...+......+.+.....+...+...+...+.........+...+..................+..........+.........+..+......+.......+...+..+......+.+...+...+..............+....+..+...+.+...........+...+.+..............+.......+...+...........+......+..................................+........+...+....+.....................+...............+...+..+....+...+........+...............+.......+.....................+....................+......+.+....................+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2024-03-09T14:28:26.963 INFO:teuthology.orchestra.run.smithi001.stderr:......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.+..+..........+...+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..+...+......+.+.....+.............+......+...............+...+.....................+..+.......+..+..........+.....+....+..+...+......+....+..+.............+.....+................+.....+................+.....+...............+...............+...+..........+......+...+.....+......+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2024-03-09T14:28:26.988 INFO:teuthology.orchestra.run.smithi001.stderr:----- 2024-03-09T14:28:26.999 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /home/ubuntu/cephtest/ca/root.crt /usr/local/share/ca-certificates/root.crt 2024-03-09T14:28:27.066 DEBUG:teuthology.orchestra.run.smithi001:> sudo update-ca-certificates 2024-03-09T14:28:27.124 INFO:teuthology.orchestra.run.smithi001.stdout:Updating certificates in /etc/ssl/certs... 2024-03-09T14:28:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL 2024-03-09T14:28:27.862 INFO:teuthology.orchestra.run.smithi001.stdout:1 added, 0 removed; done. 2024-03-09T14:28:27.873 INFO:teuthology.orchestra.run.smithi001.stdout:Running hooks in /etc/ca-certificates/update.d... 2024-03-09T14:28:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:28:29.464 INFO:teuthology.orchestra.run.smithi001.stdout:Adding debian:root.pem 2024-03-09T14:28:29.488 INFO:teuthology.orchestra.run.smithi001.stdout:done. 2024-03-09T14:28:29.488 INFO:teuthology.orchestra.run.smithi001.stdout:done. 2024-03-09T14:28:29.492 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p /home/ubuntu/cephtest/ca 2024-03-09T14:28:29.538 DEBUG:teuthology.orchestra.run.smithi001:> openssl req -nodes -newkey rsa:2048 -keyout /home/ubuntu/cephtest/ca/rgw.client.0.key -out /home/ubuntu/cephtest/ca/rgw.client.0.csr -subj /CN=smithi001.front.sepia.ceph.com 2024-03-09T14:28:30.095 INFO:teuthology.orchestra.run.smithi001.stderr:......+...+..+....+...+...+..+...+...+.+...........+.......+...+............+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+......................+..+...+...+.........+.........+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.............................+....+......+...........+.+.........+...+..+.+..+......+....+...+........+....+..+...+.+..+....+...+......+..+....+.........+..+...+..........+......+.....+...+............+...+...+.......+..+....+..............+.+..+.............+.....+....+.....+..........+.....+.......+....................+......+...+..........+..+...+......+...+................+.....+.......+...+....................+......+...............+.+..+...+....+......+........+...................+...............+........+...+.+.....+....+.....+...............+.+...+..+...+...+....+...........+.........+...+..........+..+...+...+..........+...+.....+.......+........+.+......+..+.+..+...+.......+..+.......+...........+.........+.+.................+.+......+......+...+..+....+......+...+...........+...+.+..+...+.+..............+.......+...+..+...+....+........+....+...+............+...........+.......+......+.................+............+..........+..+...............+.+..+.+..+....+.....+...+.+..+....+...+........+...+......+....+.....+......+...+......+....+.................+.+...+.........+..+.+.........+...+.....+.............+..+.+......+..............+.......+.....+.........+..........+.....+.........+....+............+...+........+.......+.....+.+..+...+....+.....+.............+...+.........+.................+.+........+.......+..+...+...................+......+..+...+.........+...+.......+......+...+..+.........+.+........+....+...+...+.........+......+..+....+...+..+......+....+...+...+...............+..+......+......+.+..+.+..+...+.........+.............+............+..+...+.+..+....+.........+........................+...+.........+..................+..+.+.........+.....+....+..............+............+...+.+.....+.+........+.+...............+.....+......+...+............+.+..+......+.+...+...+........+..........+.................+..........+...+..+....+...+..+.+..+......+...............+......+............................+...+...+..+....+...+........+...................+.....+.........+.+......+..+......+.+......+.....+..........+...+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2024-03-09T14:28:30.450 INFO:teuthology.orchestra.run.smithi001.stderr:.....+.........+.....+.......+..+...+..........+.....+...+.+.........+.........+...+...+..+....+.....+...+...+...+.......+..+.+.....+......+...+.+.....+.......+..+...+.......+..+...+......+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+...............+...+.+....................+......+.......+......+............+..+...+....+......+..+.............+...+..+...............+...+....+..+.+............+..+..........+...+........+...+.+......+..+.+.........+..+.+...+..............+......+....+...........+...............+......+.+............+........+...+.+..............+.......+...+............+........+....+.....+......+......+.+...+..+...+....+............+.....+...+.............+..+............+.+..+.+.....+.........+............+.............+..+....+.....+.......+..............+..........+........+.+..+.......+..+.........+......+....+......+...+..+......+.......+...............+.....+.+..+.........+......+..................+.+...+.....+......+.......+..+...+...+.......+.....+....+......+........+.......+..+..........+..+....+..+...............+...+....+......+......+.....+..........+.....+...+.......+...+......+...........+...+..........+.....+......+.......+.........+..............+.......+...+..+...+.+...........+...+.+.........+..+...+......+...+............+...+......+.+........+.......+.........+.....+......+................+.....+.......+..+......+...+....+......+..+....+...+...........+....+...........+....+..+....+............+........+......+...+.......+.....+.+............+...+..+.+......+...........+...+.......+...+..+...+....+..+.........+.......+.....+...+....+..+.+..+......+....+......+.............................+.............+..+...+....+......+..+.+......+.....+...+.+.....+......+......+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2024-03-09T14:28:30.454 INFO:teuthology.orchestra.run.smithi001.stderr:----- 2024-03-09T14:28:30.457 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:28:30.457 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ca/rgw.client.0.ext 2024-03-09T14:28:30.505 DEBUG:teuthology.orchestra.run.smithi001:> openssl x509 -req -in /home/ubuntu/cephtest/ca/rgw.client.0.csr -CA /home/ubuntu/cephtest/ca/root.crt -CAkey /home/ubuntu/cephtest/ca/root.key -CAcreateserial -out /home/ubuntu/cephtest/ca/rgw.client.0.crt -days 365 -sha256 -extfile /home/ubuntu/cephtest/ca/rgw.client.0.ext 2024-03-09T14:28:30.555 INFO:teuthology.orchestra.run.smithi001.stderr:Certificate request self-signature ok 2024-03-09T14:28:30.555 INFO:teuthology.orchestra.run.smithi001.stderr:subject=CN = smithi001.front.sepia.ceph.com 2024-03-09T14:28:30.567 DEBUG:teuthology.orchestra.run.smithi001:> rm -f /home/ubuntu/cephtest/ca/rgw.client.0.csr /home/ubuntu/cephtest/ca/root.srl /home/ubuntu/cephtest/ca/rgw.client.0.ext 2024-03-09T14:28:30.615 DEBUG:teuthology.orchestra.run.smithi001:> openssl verify -CAfile /home/ubuntu/cephtest/ca/root.crt /home/ubuntu/cephtest/ca/rgw.client.0.crt 2024-03-09T14:28:30.666 INFO:teuthology.orchestra.run.smithi001.stdout:/home/ubuntu/cephtest/ca/rgw.client.0.crt: OK 2024-03-09T14:28:30.666 DEBUG:teuthology.orchestra.run.smithi001:> cat /home/ubuntu/cephtest/ca/rgw.client.0.key >> /home/ubuntu/cephtest/ca/rgw.client.0.crt 2024-03-09T14:28:30.714 INFO:teuthology.run_tasks:Running task rgw... 2024-03-09T14:28:30.729 DEBUG:tasks.rgw:config is {'client.0': {'ssl certificate': 'rgw.client.0', 'valgrind': ['--tool=memcheck', '--max-threads=1024']}} 2024-03-09T14:28:30.729 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2024-03-09T14:28:30.729 INFO:tasks.rgw:Creating data pools 2024-03-09T14:28:30.729 DEBUG:tasks.rgw:Obtaining remote for client client.0 2024-03-09T14:28:30.729 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2024-03-09T14:28:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'default.rgw.buckets.data' created 2024-03-09T14:28:31.315 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2024-03-09T14:28:32.281 INFO:teuthology.orchestra.run.smithi001.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-03-09T14:28:32.331 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2024-03-09T14:28:33.277 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'default.rgw.buckets.index' created 2024-03-09T14:28:33.316 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2024-03-09T14:28:34.280 INFO:teuthology.orchestra.run.smithi001.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-03-09T14:28:34.325 DEBUG:tasks.rgw:Pools created 2024-03-09T14:28:34.325 INFO:tasks.rgw:Configuring compression type = random 2024-03-09T14:28:34.326 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2024-03-09T14:28:34.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2024-03-09T14:28:34.326 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2024-03-09T14:28:34.376 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:28:34.376 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:28:36.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.374+0000 7f1bb65c0640 0 -- >> [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] conn(0x55ad8e158230 msgr2=0x55ad8e158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send injecting socket failure 2024-03-09T14:28:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.378+0000 7f1bb82a7980 20 rados->read ofs=0 len=0 2024-03-09T14:28:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.378+0000 7f1bb55be640 0 -- 172.21.15.1:0/1084985534 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55ad8e16fc40 msgr2=0x55ad8e1720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:28:36.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.582+0000 7f1bb82a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:36.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.582+0000 7f1bb82a7980 20 realm 2024-03-09T14:28:36.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.582+0000 7f1bb82a7980 20 rados->read ofs=0 len=0 2024-03-09T14:28:36.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.582+0000 7f1bb82a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:36.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.582+0000 7f1bb82a7980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:28:36.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.582+0000 7f1bb82a7980 20 rados->read ofs=0 len=0 2024-03-09T14:28:36.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.582+0000 7f1bb82a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:36.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.582+0000 7f1bb82a7980 20 rados->read ofs=0 len=0 2024-03-09T14:28:36.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.586+0000 7f1bb82a7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:28:36.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.586+0000 7f1bb82a7980 20 rados->read ofs=0 len=0 2024-03-09T14:28:36.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.586+0000 7f1bb82a7980 20 rados_obj.operate() r=0 bl.length=957 2024-03-09T14:28:36.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.586+0000 7f1bb82a7980 20 searching for the correct realm 2024-03-09T14:28:36.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:28:36.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:28:36.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:28:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:28:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:28:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:28:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 rados->read ofs=0 len=0 2024-03-09T14:28:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 rados->read ofs=0 len=0 2024-03-09T14:28:36.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:28:36.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 rados->read ofs=0 len=0 2024-03-09T14:28:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 rados_obj.operate() r=0 bl.length=398 2024-03-09T14:28:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 20 zone default found 2024-03-09T14:28:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 4 Realm: () 2024-03-09T14:28:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.622+0000 7f1bb82a7980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:28:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.626+0000 7f1bb82a7980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:28:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.626+0000 7f1bb82a7980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:28:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.626+0000 7f1bb82a7980 20 zonegroup default 2024-03-09T14:28:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.626+0000 7f1bb82a7980 20 rados->read ofs=0 len=0 2024-03-09T14:28:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.626+0000 7f1bb82a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:36.626+0000 7f1bb82a7980 20 rados->read ofs=0 len=0 2024-03-09T14:28:38.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:38.346+0000 7f1bb82a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:38.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:38.346+0000 7f1bb82a7980 20 rados->read ofs=0 len=0 2024-03-09T14:28:38.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:38.346+0000 7f1bb82a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:38.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:38.346+0000 7f1bb82a7980 20 started sync module instance, tier type = 2024-03-09T14:28:38.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:38.346+0000 7f1bb82a7980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:28:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.398+0000 7f1bb82a7980 20 add_watcher() i=0 2024-03-09T14:28:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.398+0000 7f1bb82a7980 20 add_watcher() i=1 2024-03-09T14:28:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.398+0000 7f1bb82a7980 20 add_watcher() i=2 2024-03-09T14:28:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.398+0000 7f1bb82a7980 20 add_watcher() i=3 2024-03-09T14:28:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.398+0000 7f1bb82a7980 20 add_watcher() i=4 2024-03-09T14:28:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.398+0000 7f1bb82a7980 20 add_watcher() i=5 2024-03-09T14:28:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.398+0000 7f1bb82a7980 20 add_watcher() i=6 2024-03-09T14:28:40.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.402+0000 7f1bb82a7980 20 add_watcher() i=7 2024-03-09T14:28:40.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.402+0000 7f1bb82a7980 2 all 8 watchers are set, enabling cache 2024-03-09T14:28:40.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.402+0000 7f1bb82a7980 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-03-09T14:28:40.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.402+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-03-09T14:28:40.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.402+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.402+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-03-09T14:28:40.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.402+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-03-09T14:28:40.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.402+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-03-09T14:28:40.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-03-09T14:28:40.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-03-09T14:28:40.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-03-09T14:28:40.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-03-09T14:28:40.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.4 2024-03-09T14:28:40.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.4 does not exist 2024-03-09T14:28:40.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.5 2024-03-09T14:28:40.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.5 does not exist 2024-03-09T14:28:40.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.6 2024-03-09T14:28:40.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.6 does not exist 2024-03-09T14:28:40.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.7 2024-03-09T14:28:40.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.406+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.7 does not exist 2024-03-09T14:28:40.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.8 2024-03-09T14:28:40.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.8 does not exist 2024-03-09T14:28:40.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.9 2024-03-09T14:28:40.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.9 does not exist 2024-03-09T14:28:40.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.10 2024-03-09T14:28:40.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.10 does not exist 2024-03-09T14:28:40.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.11 2024-03-09T14:28:40.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.11 does not exist 2024-03-09T14:28:40.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.12 2024-03-09T14:28:40.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.12 does not exist 2024-03-09T14:28:40.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.13 2024-03-09T14:28:40.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.410+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.13 does not exist 2024-03-09T14:28:40.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.14 2024-03-09T14:28:40.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.14 does not exist 2024-03-09T14:28:40.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.15 2024-03-09T14:28:40.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.15 does not exist 2024-03-09T14:28:40.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.16 2024-03-09T14:28:40.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.16 does not exist 2024-03-09T14:28:40.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.17 2024-03-09T14:28:40.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.17 does not exist 2024-03-09T14:28:40.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.18 2024-03-09T14:28:40.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.18 does not exist 2024-03-09T14:28:40.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.19 2024-03-09T14:28:40.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.414+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.19 does not exist 2024-03-09T14:28:40.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.20 2024-03-09T14:28:40.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.20 does not exist 2024-03-09T14:28:40.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.21 2024-03-09T14:28:40.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.21 does not exist 2024-03-09T14:28:40.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.22 2024-03-09T14:28:40.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.22 does not exist 2024-03-09T14:28:40.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.23 2024-03-09T14:28:40.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.23 does not exist 2024-03-09T14:28:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.24 2024-03-09T14:28:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.24 does not exist 2024-03-09T14:28:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.25 2024-03-09T14:28:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.418+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.25 does not exist 2024-03-09T14:28:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.26 2024-03-09T14:28:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.26 does not exist 2024-03-09T14:28:40.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.27 2024-03-09T14:28:40.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.27 does not exist 2024-03-09T14:28:40.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.28 2024-03-09T14:28:40.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.28 does not exist 2024-03-09T14:28:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.29 2024-03-09T14:28:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.29 does not exist 2024-03-09T14:28:40.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.30 2024-03-09T14:28:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.30 does not exist 2024-03-09T14:28:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.31 2024-03-09T14:28:40.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.422+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.31 does not exist 2024-03-09T14:28:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.32 2024-03-09T14:28:40.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.32 does not exist 2024-03-09T14:28:40.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.33 2024-03-09T14:28:40.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.33 does not exist 2024-03-09T14:28:40.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.34 2024-03-09T14:28:40.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.34 does not exist 2024-03-09T14:28:40.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.35 2024-03-09T14:28:40.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.35 does not exist 2024-03-09T14:28:40.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.36 2024-03-09T14:28:40.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.36 does not exist 2024-03-09T14:28:40.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.37 2024-03-09T14:28:40.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.426+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.430+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.37 does not exist 2024-03-09T14:28:40.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.430+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.38 2024-03-09T14:28:40.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.430+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.430+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.38 does not exist 2024-03-09T14:28:40.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.430+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.39 2024-03-09T14:28:40.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.430+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.430+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.39 does not exist 2024-03-09T14:28:40.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.430+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.40 2024-03-09T14:28:40.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.430+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.430+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.40 does not exist 2024-03-09T14:28:40.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.430+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.41 2024-03-09T14:28:40.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.430+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.41 does not exist 2024-03-09T14:28:40.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.42 2024-03-09T14:28:40.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.42 does not exist 2024-03-09T14:28:40.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.43 2024-03-09T14:28:40.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.43 does not exist 2024-03-09T14:28:40.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.44 2024-03-09T14:28:40.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.44 does not exist 2024-03-09T14:28:40.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.45 2024-03-09T14:28:40.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.45 does not exist 2024-03-09T14:28:40.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.46 2024-03-09T14:28:40.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.434+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.46 does not exist 2024-03-09T14:28:40.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.47 2024-03-09T14:28:40.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.47 does not exist 2024-03-09T14:28:40.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.48 2024-03-09T14:28:40.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.48 does not exist 2024-03-09T14:28:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.49 2024-03-09T14:28:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.49 does not exist 2024-03-09T14:28:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.50 2024-03-09T14:28:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.50 does not exist 2024-03-09T14:28:40.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.51 2024-03-09T14:28:40.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.51 does not exist 2024-03-09T14:28:40.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.52 2024-03-09T14:28:40.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.52 does not exist 2024-03-09T14:28:40.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.53 2024-03-09T14:28:40.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.438+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.53 does not exist 2024-03-09T14:28:40.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.54 2024-03-09T14:28:40.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.54 does not exist 2024-03-09T14:28:40.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.55 2024-03-09T14:28:40.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.55 does not exist 2024-03-09T14:28:40.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.56 2024-03-09T14:28:40.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.56 does not exist 2024-03-09T14:28:40.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.57 2024-03-09T14:28:40.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.57 does not exist 2024-03-09T14:28:40.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.58 2024-03-09T14:28:40.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.58 does not exist 2024-03-09T14:28:40.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.59 2024-03-09T14:28:40.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.59 does not exist 2024-03-09T14:28:40.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.60 2024-03-09T14:28:40.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.442+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.60 does not exist 2024-03-09T14:28:40.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.61 2024-03-09T14:28:40.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.61 does not exist 2024-03-09T14:28:40.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.62 2024-03-09T14:28:40.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.62 does not exist 2024-03-09T14:28:40.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.63 2024-03-09T14:28:40.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.63 does not exist 2024-03-09T14:28:40.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.64 2024-03-09T14:28:40.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.64 does not exist 2024-03-09T14:28:40.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.65 2024-03-09T14:28:40.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.65 does not exist 2024-03-09T14:28:40.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.66 2024-03-09T14:28:40.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.446+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.66 does not exist 2024-03-09T14:28:40.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.67 2024-03-09T14:28:40.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.67 does not exist 2024-03-09T14:28:40.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.68 2024-03-09T14:28:40.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.68 does not exist 2024-03-09T14:28:40.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.69 2024-03-09T14:28:40.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.69 does not exist 2024-03-09T14:28:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.70 2024-03-09T14:28:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.70 does not exist 2024-03-09T14:28:40.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.71 2024-03-09T14:28:40.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.71 does not exist 2024-03-09T14:28:40.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.72 2024-03-09T14:28:40.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.450+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.72 does not exist 2024-03-09T14:28:40.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.73 2024-03-09T14:28:40.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.73 does not exist 2024-03-09T14:28:40.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.74 2024-03-09T14:28:40.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.74 does not exist 2024-03-09T14:28:40.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.75 2024-03-09T14:28:40.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.75 does not exist 2024-03-09T14:28:40.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.76 2024-03-09T14:28:40.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.76 does not exist 2024-03-09T14:28:40.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.77 2024-03-09T14:28:40.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.77 does not exist 2024-03-09T14:28:40.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.78 2024-03-09T14:28:40.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.454+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.78 does not exist 2024-03-09T14:28:40.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.79 2024-03-09T14:28:40.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.79 does not exist 2024-03-09T14:28:40.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.80 2024-03-09T14:28:40.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.80 does not exist 2024-03-09T14:28:40.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.81 2024-03-09T14:28:40.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.81 does not exist 2024-03-09T14:28:40.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.82 2024-03-09T14:28:40.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.82 does not exist 2024-03-09T14:28:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.83 2024-03-09T14:28:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.83 does not exist 2024-03-09T14:28:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.84 2024-03-09T14:28:40.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.458+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.84 does not exist 2024-03-09T14:28:40.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.85 2024-03-09T14:28:40.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.85 does not exist 2024-03-09T14:28:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.86 2024-03-09T14:28:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.86 does not exist 2024-03-09T14:28:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.87 2024-03-09T14:28:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.87 does not exist 2024-03-09T14:28:40.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.88 2024-03-09T14:28:40.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.88 does not exist 2024-03-09T14:28:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.89 2024-03-09T14:28:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.89 does not exist 2024-03-09T14:28:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.90 2024-03-09T14:28:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.462+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.90 does not exist 2024-03-09T14:28:40.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.91 2024-03-09T14:28:40.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.91 does not exist 2024-03-09T14:28:40.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.92 2024-03-09T14:28:40.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.92 does not exist 2024-03-09T14:28:40.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.93 2024-03-09T14:28:40.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.93 does not exist 2024-03-09T14:28:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.94 2024-03-09T14:28:40.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.94 does not exist 2024-03-09T14:28:40.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.95 2024-03-09T14:28:40.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.95 does not exist 2024-03-09T14:28:40.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.96 2024-03-09T14:28:40.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.466+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.96 does not exist 2024-03-09T14:28:40.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.97 2024-03-09T14:28:40.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.97 does not exist 2024-03-09T14:28:40.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.98 2024-03-09T14:28:40.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.98 does not exist 2024-03-09T14:28:40.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.99 2024-03-09T14:28:40.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.99 does not exist 2024-03-09T14:28:40.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.100 2024-03-09T14:28:40.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.100 does not exist 2024-03-09T14:28:40.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.101 2024-03-09T14:28:40.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.101 does not exist 2024-03-09T14:28:40.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.102 2024-03-09T14:28:40.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.470+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.102 does not exist 2024-03-09T14:28:40.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.103 2024-03-09T14:28:40.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.103 does not exist 2024-03-09T14:28:40.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.104 2024-03-09T14:28:40.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.104 does not exist 2024-03-09T14:28:40.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.105 2024-03-09T14:28:40.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.105 does not exist 2024-03-09T14:28:40.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.106 2024-03-09T14:28:40.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.106 does not exist 2024-03-09T14:28:40.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.107 2024-03-09T14:28:40.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.107 does not exist 2024-03-09T14:28:40.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.108 2024-03-09T14:28:40.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.474+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.108 does not exist 2024-03-09T14:28:40.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.109 2024-03-09T14:28:40.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.109 does not exist 2024-03-09T14:28:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.110 2024-03-09T14:28:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.110 does not exist 2024-03-09T14:28:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.111 2024-03-09T14:28:40.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.111 does not exist 2024-03-09T14:28:40.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.112 2024-03-09T14:28:40.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.112 does not exist 2024-03-09T14:28:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.113 2024-03-09T14:28:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.113 does not exist 2024-03-09T14:28:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.114 2024-03-09T14:28:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.478+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.114 does not exist 2024-03-09T14:28:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.115 2024-03-09T14:28:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.115 does not exist 2024-03-09T14:28:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.116 2024-03-09T14:28:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.116 does not exist 2024-03-09T14:28:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.117 2024-03-09T14:28:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.117 does not exist 2024-03-09T14:28:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.118 2024-03-09T14:28:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.118 does not exist 2024-03-09T14:28:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.119 2024-03-09T14:28:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.119 does not exist 2024-03-09T14:28:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.120 2024-03-09T14:28:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.482+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.120 does not exist 2024-03-09T14:28:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.121 2024-03-09T14:28:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.121 does not exist 2024-03-09T14:28:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.122 2024-03-09T14:28:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.122 does not exist 2024-03-09T14:28:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.123 2024-03-09T14:28:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.123 does not exist 2024-03-09T14:28:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.124 2024-03-09T14:28:40.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.124 does not exist 2024-03-09T14:28:40.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.125 2024-03-09T14:28:40.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.486+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.490+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.125 does not exist 2024-03-09T14:28:40.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.490+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.126 2024-03-09T14:28:40.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.490+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.490+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.126 does not exist 2024-03-09T14:28:40.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.490+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.127 2024-03-09T14:28:40.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.490+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.490+0000 7f1bb82a7980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.127 does not exist 2024-03-09T14:28:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.490+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1211 entering 2024-03-09T14:28:40.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.494+0000 7f1bb82a7980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-03-09T14:28:40.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.502+0000 7f1a67fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:28:40.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.502+0000 7f1bb82a7980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:28:40.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:40.502+0000 7f1bb82a7980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:28:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.378+0000 7f1bb82a7980 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-03-09T14:28:43.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.378+0000 7f1bb82a7980 5 note: GC not initialized 2024-03-09T14:28:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.390+0000 7f1a66fe5640 20 reqs_thread_entry: start 2024-03-09T14:28:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.390+0000 7f1bb82a7980 20 init_complete bucket index max shards: 1 2024-03-09T14:28:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.390+0000 7f1bb82a7980 20 Filter name: none 2024-03-09T14:28:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.390+0000 7f1a64fe1640 20 reqs_thread_entry: start 2024-03-09T14:28:43.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.434+0000 7f1a67fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:28:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.438+0000 7f1bb82a7980 20 remove_watcher() i=0 2024-03-09T14:28:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.438+0000 7f1bb82a7980 2 removed watcher, disabling cache 2024-03-09T14:28:43.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.442+0000 7f1bb82a7980 20 remove_watcher() i=1 2024-03-09T14:28:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.442+0000 7f1bb82a7980 20 remove_watcher() i=2 2024-03-09T14:28:43.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.450+0000 7f1bb82a7980 20 remove_watcher() i=3 2024-03-09T14:28:43.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.454+0000 7f1bb82a7980 20 remove_watcher() i=4 2024-03-09T14:28:43.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.454+0000 7f1bb82a7980 20 remove_watcher() i=5 2024-03-09T14:28:43.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.458+0000 7f1bb82a7980 20 remove_watcher() i=6 2024-03-09T14:28:43.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.458+0000 7f1bb82a7980 20 remove_watcher() i=7 2024-03-09T14:28:43.484 INFO:teuthology.orchestra.run.smithi001.stdout:[] 2024-03-09T14:28:43.484 DEBUG:tasks.util.rgw: json result: [] 2024-03-09T14:28:43.484 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-09T14:28:43.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-03-09T14:28:43.485 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-03-09T14:28:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:28:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:28:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.558+0000 7f5c9b071980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:28:43.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.558+0000 7f5c9b071980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:28:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.562+0000 7f5b56ffd640 20 reqs_thread_entry: start 2024-03-09T14:28:43.587 INFO:teuthology.orchestra.run.smithi001.stdout:{"id":"8c209108-e1cd-4d45-9608-6f491550b24c","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups"} 2024-03-09T14:28:43.588 DEBUG:tasks.util.rgw: json result: {'id': '8c209108-e1cd-4d45-9608-6f491550b24c', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups'} 2024-03-09T14:28:43.588 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2024-03-09T14:28:43.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2024-03-09T14:28:43.588 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2024-03-09T14:28:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:28:43.616 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:28:43.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.650+0000 7fd15ed68980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.650+0000 7fd15ed68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:43.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.650+0000 7fd15ed68980 20 realm 2024-03-09T14:28:43.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.650+0000 7fd15ed68980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.650+0000 7fd15ed68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:43.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.650+0000 7fd15ed68980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:28:43.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.650+0000 7fd15ed68980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.650+0000 7fd15ed68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:43.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.650+0000 7fd15ed68980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.654+0000 7fd15ed68980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:28:43.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.654+0000 7fd15ed68980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.654+0000 7fd15ed68980 20 rados_obj.operate() r=0 bl.length=973 2024-03-09T14:28:43.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.654+0000 7fd15ed68980 20 searching for the correct realm 2024-03-09T14:28:43.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.662+0000 7fd15c07f640 0 -- 172.21.15.1:0/77063672 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fd14c10d030 msgr2=0x7fd14c0cf8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:28:43.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.666+0000 7fd15c07f640 0 -- 172.21.15.1:0/77063672 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fd0280730c0 msgr2=0x7fd0280754b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:28:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.674+0000 7fd15ed68980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:28:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.674+0000 7fd15ed68980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:28:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.674+0000 7fd15ed68980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:28:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.674+0000 7fd15ed68980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:28:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.674+0000 7fd15ed68980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:28:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.674+0000 7fd15ed68980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:28:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.674+0000 7fd15ed68980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:43.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:28:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 rados_obj.operate() r=0 bl.length=398 2024-03-09T14:28:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 zone default found 2024-03-09T14:28:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 4 Realm: () 2024-03-09T14:28:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:28:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:28:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:28:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 zonegroup default 2024-03-09T14:28:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:43.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:43.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 started sync module instance, tier type = 2024-03-09T14:28:43.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.678+0000 7fd15ed68980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:28:43.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.694+0000 7fd15ed68980 20 add_watcher() i=0 2024-03-09T14:28:43.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.694+0000 7fd15ed68980 20 add_watcher() i=1 2024-03-09T14:28:43.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.694+0000 7fd15ed68980 20 add_watcher() i=2 2024-03-09T14:28:43.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.694+0000 7fd15ed68980 20 add_watcher() i=3 2024-03-09T14:28:43.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.694+0000 7fd15ed68980 20 add_watcher() i=4 2024-03-09T14:28:43.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.694+0000 7fd15ed68980 20 add_watcher() i=5 2024-03-09T14:28:43.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.694+0000 7fd15ed68980 20 add_watcher() i=6 2024-03-09T14:28:43.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.698+0000 7fd15ed68980 20 add_watcher() i=7 2024-03-09T14:28:43.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.698+0000 7fd15ed68980 2 all 8 watchers are set, enabling cache 2024-03-09T14:28:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.698+0000 7fd012fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:28:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.698+0000 7fd15ed68980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:28:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.698+0000 7fd15ed68980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:28:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.698+0000 7fd15ed68980 5 note: GC not initialized 2024-03-09T14:28:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.698+0000 7fd011fe3640 20 reqs_thread_entry: start 2024-03-09T14:28:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.698+0000 7fd15ed68980 20 init_complete bucket index max shards: 1 2024-03-09T14:28:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.698+0000 7fd15ed68980 20 Filter name: none 2024-03-09T14:28:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.698+0000 7fcf837fe640 20 reqs_thread_entry: start 2024-03-09T14:28:43.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.718+0000 7fd012fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:28:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.722+0000 7fd15ed68980 20 remove_watcher() i=0 2024-03-09T14:28:43.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.722+0000 7fd15ed68980 2 removed watcher, disabling cache 2024-03-09T14:28:43.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.726+0000 7fd15ed68980 20 remove_watcher() i=1 2024-03-09T14:28:43.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.730+0000 7fd15ed68980 20 remove_watcher() i=2 2024-03-09T14:28:43.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.730+0000 7fd15ed68980 20 remove_watcher() i=3 2024-03-09T14:28:43.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.734+0000 7fd15ed68980 20 remove_watcher() i=4 2024-03-09T14:28:43.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.734+0000 7fd15ed68980 20 remove_watcher() i=5 2024-03-09T14:28:43.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.734+0000 7fd15ed68980 20 remove_watcher() i=6 2024-03-09T14:28:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.738+0000 7fd15ed68980 20 remove_watcher() i=7 2024-03-09T14:28:43.755 INFO:teuthology.orchestra.run.smithi001.stdout:[] 2024-03-09T14:28:43.755 DEBUG:tasks.util.rgw: json result: [] 2024-03-09T14:28:43.755 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--placement-inline-data', 'false'] 2024-03-09T14:28:43.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--placement-inline-data', 'false'] 2024-03-09T14:28:43.756 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --placement-inline-data false 2024-03-09T14:28:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:28:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:28:43.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.866+0000 7f6582e95980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:28:43.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.866+0000 7f6582e95980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:28:43.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.870+0000 7f63c6ffd640 20 reqs_thread_entry: start 2024-03-09T14:28:43.898 INFO:teuthology.orchestra.run.smithi001.stdout:{"id":"8c209108-e1cd-4d45-9608-6f491550b24c","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":false}}],"realm_id":"","notif_pool":"default.rgw.log:notif","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups"} 2024-03-09T14:28:43.898 DEBUG:tasks.util.rgw: json result: {'id': '8c209108-e1cd-4d45-9608-6f491550b24c', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': False}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups'} 2024-03-09T14:28:43.898 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2024-03-09T14:28:43.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2024-03-09T14:28:43.899 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2024-03-09T14:28:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:28:43.926 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:28:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.958+0000 7f686ebba980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.962+0000 7f686ebba980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:43.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.962+0000 7f686ebba980 20 realm 2024-03-09T14:28:43.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.962+0000 7f686ebba980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.962+0000 7f686ebba980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:43.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.962+0000 7f686ebba980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:28:43.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.962+0000 7f686ebba980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.962+0000 7f686ebba980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:43.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.962+0000 7f686ebba980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.962+0000 7f686ebba980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:28:43.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.962+0000 7f686ebba980 20 rados->read ofs=0 len=0 2024-03-09T14:28:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.962+0000 7f686ebba980 20 rados_obj.operate() r=0 bl.length=973 2024-03-09T14:28:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.962+0000 7f686ebba980 20 searching for the correct realm 2024-03-09T14:28:43.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:43.970+0000 7f686ced3640 0 -- 172.21.15.1:0/492031211 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f686001b940 msgr2=0x7f6860043410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:28:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.186+0000 7f686c6d2640 0 -- 172.21.15.1:0/492031211 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x559d4e9ddb10 msgr2=0x559d4e9dffd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:28:44.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.194+0000 7f686ebba980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:28:44.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.194+0000 7f686ebba980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:28:44.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.194+0000 7f686ebba980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:28:44.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.194+0000 7f686ebba980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:28:44.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.194+0000 7f686ebba980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:28:44.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.194+0000 7f686ebba980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:28:44.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.194+0000 7f686ebba980 20 rados->read ofs=0 len=0 2024-03-09T14:28:44.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.194+0000 7f686ebba980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:44.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.194+0000 7f686ebba980 20 rados->read ofs=0 len=0 2024-03-09T14:28:44.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.194+0000 7f686ebba980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:28:44.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.194+0000 7f686ebba980 20 rados->read ofs=0 len=0 2024-03-09T14:28:44.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 20 rados_obj.operate() r=0 bl.length=398 2024-03-09T14:28:44.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 20 zone default found 2024-03-09T14:28:44.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 4 Realm: () 2024-03-09T14:28:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:28:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:28:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:28:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 20 zonegroup default 2024-03-09T14:28:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 20 rados->read ofs=0 len=0 2024-03-09T14:28:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 20 rados->read ofs=0 len=0 2024-03-09T14:28:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 20 rados->read ofs=0 len=0 2024-03-09T14:28:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:28:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 20 started sync module instance, tier type = 2024-03-09T14:28:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.198+0000 7f686ebba980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:28:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.222+0000 7f686ebba980 20 add_watcher() i=0 2024-03-09T14:28:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.222+0000 7f686ebba980 20 add_watcher() i=1 2024-03-09T14:28:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.222+0000 7f686ebba980 20 add_watcher() i=2 2024-03-09T14:28:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.222+0000 7f686ebba980 20 add_watcher() i=3 2024-03-09T14:28:44.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.222+0000 7f686ebba980 20 add_watcher() i=4 2024-03-09T14:28:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.222+0000 7f686ebba980 20 add_watcher() i=5 2024-03-09T14:28:44.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.222+0000 7f686ebba980 20 add_watcher() i=6 2024-03-09T14:28:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.222+0000 7f686ebba980 20 add_watcher() i=7 2024-03-09T14:28:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.222+0000 7f686ebba980 2 all 8 watchers are set, enabling cache 2024-03-09T14:28:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.226+0000 7f6722fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:28:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.226+0000 7f686ebba980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:28:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.226+0000 7f686ebba980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:28:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.226+0000 7f686ebba980 5 note: GC not initialized 2024-03-09T14:28:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.226+0000 7f6721fdb640 20 reqs_thread_entry: start 2024-03-09T14:28:44.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.226+0000 7f686ebba980 20 init_complete bucket index max shards: 1 2024-03-09T14:28:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.230+0000 7f686ebba980 20 Filter name: none 2024-03-09T14:28:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.230+0000 7f66937fe640 20 reqs_thread_entry: start 2024-03-09T14:28:44.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.250+0000 7f6722fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:28:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.258+0000 7f686ebba980 20 remove_watcher() i=0 2024-03-09T14:28:44.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.258+0000 7f686ebba980 2 removed watcher, disabling cache 2024-03-09T14:28:44.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.258+0000 7f686ebba980 20 remove_watcher() i=1 2024-03-09T14:28:44.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.262+0000 7f686ebba980 20 remove_watcher() i=2 2024-03-09T14:28:44.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.266+0000 7f686ebba980 20 remove_watcher() i=3 2024-03-09T14:28:44.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.266+0000 7f686ebba980 20 remove_watcher() i=4 2024-03-09T14:28:44.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.270+0000 7f686ebba980 20 remove_watcher() i=5 2024-03-09T14:28:44.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.270+0000 7f686ebba980 20 remove_watcher() i=6 2024-03-09T14:28:44.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.274+0000 7f686ebba980 20 remove_watcher() i=7 2024-03-09T14:28:44.302 INFO:teuthology.orchestra.run.smithi001.stdout:[] 2024-03-09T14:28:44.302 DEBUG:tasks.util.rgw: json result: [] 2024-03-09T14:28:44.303 INFO:tasks.rgw:Configuring storage class type = LUKEWARM 2024-03-09T14:28:44.303 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2024-03-09T14:28:44.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2024-03-09T14:28:44.303 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM 2024-03-09T14:28:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:28:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:28:44.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.414+0000 7f6d6c574980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:28:44.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.414+0000 7f6d6c574980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:28:44.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.418+0000 7f6c2cfe1640 20 reqs_thread_entry: start 2024-03-09T14:28:44.441 INFO:teuthology.orchestra.run.smithi001.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["LUKEWARM","STANDARD"]}}] 2024-03-09T14:28:44.441 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['LUKEWARM', 'STANDARD']}}] 2024-03-09T14:28:44.441 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2024-03-09T14:28:44.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2024-03-09T14:28:44.442 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2024-03-09T14:28:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:28:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:28:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.502+0000 7f42d9352980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:28:44.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.502+0000 7f42d9352980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:28:44.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.506+0000 7f4194ff9640 20 reqs_thread_entry: start 2024-03-09T14:28:44.529 INFO:teuthology.orchestra.run.smithi001.stdout:{"id":"8c209108-e1cd-4d45-9608-6f491550b24c","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":false}}],"realm_id":"","notif_pool":"default.rgw.log:notif","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups"} 2024-03-09T14:28:44.530 DEBUG:tasks.util.rgw: json result: {'id': '8c209108-e1cd-4d45-9608-6f491550b24c', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': False}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups'} 2024-03-09T14:28:44.530 INFO:tasks.rgw:Configuring storage class type = FROZEN 2024-03-09T14:28:44.530 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2024-03-09T14:28:44.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2024-03-09T14:28:44.530 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN 2024-03-09T14:28:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:28:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:28:44.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.630+0000 7ff6d8e2e980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:28:44.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.630+0000 7ff6d8e2e980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:28:44.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.634+0000 7ff594ff9640 20 reqs_thread_entry: start 2024-03-09T14:28:44.656 INFO:teuthology.orchestra.run.smithi001.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","LUKEWARM","STANDARD"]}}] 2024-03-09T14:28:44.656 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2024-03-09T14:28:44.656 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2024-03-09T14:28:44.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2024-03-09T14:28:44.656 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2024-03-09T14:28:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:28:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:28:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.766+0000 7f5be67d2980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:28:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.766+0000 7f5be67d2980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:28:44.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:28:44.770+0000 7f5aa27fc640 20 reqs_thread_entry: start 2024-03-09T14:28:44.798 INFO:teuthology.orchestra.run.smithi001.stdout:{"id":"8c209108-e1cd-4d45-9608-6f491550b24c","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":false}}],"realm_id":"","notif_pool":"default.rgw.log:notif","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups"} 2024-03-09T14:28:44.799 DEBUG:tasks.util.rgw: json result: {'id': '8c209108-e1cd-4d45-9608-6f491550b24c', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': False}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups'} 2024-03-09T14:28:44.799 INFO:tasks.rgw:Starting rgw... 2024-03-09T14:28:44.799 INFO:tasks.rgw:rgw client.0 config is {'ssl certificate': 'rgw.client.0', 'valgrind': ['--tool=memcheck', '--max-threads=1024']} 2024-03-09T14:28:44.799 INFO:tasks.rgw:Using beast as radosgw frontend 2024-03-09T14:28:44.799 DEBUG:teuthology.orchestra.run.smithi001:> sudo echo -n https://smithi001.front.sepia.ceph.com:443 | sudo tee /home/ubuntu/cephtest/url_file 2024-03-09T14:28:44.810 INFO:teuthology.orchestra.run.smithi001.stdout:https://smithi001.front.sepia.ceph.com:443 2024-03-09T14:28:44.810 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-03-09T14:28:44.862 DEBUG:teuthology.orchestra.run.smithi001:> sudo keytool -import -alias smithi001.front.sepia.ceph.com -keystore $(readlink -e $(dirname $(readlink -e $(which keytool)))/../lib/security/cacerts) -file /home/ubuntu/cephtest/ca/rgw.client.0.crt -storepass changeit 2024-03-09T14:28:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:Warning: use -cacerts option to access cacerts keystore 2024-03-09T14:28:45.573 INFO:teuthology.orchestra.run.smithi001.stderr:Certificate was added to keystore 2024-03-09T14:28:45.606 DEBUG:tasks.ceph_manager:running ceph.client.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/ceph.client.0.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--max-threads=1024'] 2024-03-09T14:28:45.606 INFO:tasks.rgw.client.0:Restarting daemon 2024-03-09T14:28:45.606 DEBUG:teuthology.orchestra.run.smithi001:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/ceph.client.0.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast ssl_certificate=/home/ubuntu/cephtest/ca/rgw.client.0.crt ssl_port=443' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2024-03-09T14:28:45.609 INFO:tasks.rgw.client.0:Started 2024-03-09T14:28:45.609 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on https://smithi001.front.sepia.ceph.com:443/ 2024-03-09T14:28:45.609 DEBUG:teuthology.orchestra.run.smithi001:> curl https://smithi001.front.sepia.ceph.com:443/ 2024-03-09T14:28:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-09T14:28:45.642 INFO:teuthology.orchestra.run.smithi001.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-09T14:28:45.642 INFO:teuthology.orchestra.run.smithi001.stderr: Dload Upload Total Spent Left Speed 2024-03-09T14:28:45.642 INFO:teuthology.orchestra.run.smithi001.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-09T14:28:45.642 INFO:teuthology.orchestra.run.smithi001.stderr:curl: (7) Failed to connect to smithi001.front.sepia.ceph.com port 443 after 0 ms: Connection refused 2024-03-09T14:28:46.643 DEBUG:teuthology.orchestra.run.smithi001:> curl https://smithi001.front.sepia.ceph.com:443/ 2024-03-09T14:28:46.654 INFO:teuthology.orchestra.run.smithi001.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-09T14:28:46.654 INFO:teuthology.orchestra.run.smithi001.stderr: Dload Upload Total Spent Left Speed 2024-03-09T14:28:46.654 INFO:teuthology.orchestra.run.smithi001.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-09T14:28:46.655 INFO:teuthology.orchestra.run.smithi001.stderr:curl: (7) Failed to connect to smithi001.front.sepia.ceph.com port 443 after 0 ms: Connection refused 2024-03-09T14:28:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-09T14:28:48.657 DEBUG:teuthology.orchestra.run.smithi001:> curl https://smithi001.front.sepia.ceph.com:443/ 2024-03-09T14:28:48.668 INFO:teuthology.orchestra.run.smithi001.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-09T14:28:48.668 INFO:teuthology.orchestra.run.smithi001.stderr: Dload Upload Total Spent Left Speed 2024-03-09T14:28:48.668 INFO:teuthology.orchestra.run.smithi001.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-09T14:28:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-09T14:28:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:curl: (7) Failed to connect to smithi001.front.sepia.ceph.com port 443 after 0 ms: Connection refused 2024-03-09T14:28:52.670 DEBUG:teuthology.orchestra.run.smithi001:> curl https://smithi001.front.sepia.ceph.com:443/ 2024-03-09T14:28:52.682 INFO:teuthology.orchestra.run.smithi001.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-09T14:28:52.682 INFO:teuthology.orchestra.run.smithi001.stderr: Dload Upload Total Spent Left Speed 2024-03-09T14:28:52.683 INFO:teuthology.orchestra.run.smithi001.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-09T14:28:52.683 INFO:teuthology.orchestra.run.smithi001.stderr:curl: (7) Failed to connect to smithi001.front.sepia.ceph.com port 443 after 0 ms: Connection refused 2024-03-09T14:28:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-09T14:29:00.684 DEBUG:teuthology.orchestra.run.smithi001:> curl https://smithi001.front.sepia.ceph.com:443/ 2024-03-09T14:29:00.695 INFO:teuthology.orchestra.run.smithi001.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-09T14:29:00.695 INFO:teuthology.orchestra.run.smithi001.stderr: Dload Upload Total Spent Left Speed 2024-03-09T14:29:00.695 INFO:teuthology.orchestra.run.smithi001.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-03-09T14:29:00.696 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-03-09T14:29:00.696 INFO:teuthology.orchestra.run.smithi001.stderr:curl: (7) Failed to connect to smithi001.front.sepia.ceph.com port 443 after 0 ms: Connection refused 2024-03-09T14:29:12.101 INFO:tasks.rgw.client.0.smithi001.stdout:2024-03-09T14:29:11.738+0000 8e90e80 -1 ssl_private_key was not found: rgw/cert//default.key 2024-03-09T14:29:16.697 DEBUG:teuthology.orchestra.run.smithi001:> curl https://smithi001.front.sepia.ceph.com:443/ 2024-03-09T14:29:16.708 INFO:teuthology.orchestra.run.smithi001.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-03-09T14:29:16.709 INFO:teuthology.orchestra.run.smithi001.stderr: Dload Upload Total Spent Left Speed 2024-03-09T14:29:17.891 INFO:teuthology.orchestra.run.smithi001.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 158 0 --:--:-- 0:00:01 --:--:-- 158 2024-03-09T14:29:17.896 INFO:teuthology.orchestra.run.smithi001.stdout:anonymous 2024-03-09T14:29:17.897 INFO:teuthology.run_tasks:Running task tox... 2024-03-09T14:29:17.908 INFO:tasks.tox:Deploying tox from pip... 2024-03-09T14:29:17.908 DEBUG:teuthology.orchestra.run.smithi001:> python3 -m venv /home/ubuntu/cephtest/tox-venv 2024-03-09T14:29:20.969 DEBUG:teuthology.orchestra.run.smithi001:> source /home/ubuntu/cephtest/tox-venv/bin/activate && pip install tox==3.15.0 2024-03-09T14:29:21.691 INFO:teuthology.orchestra.run.smithi001.stdout:Collecting tox==3.15.0 2024-03-09T14:29:21.800 INFO:teuthology.orchestra.run.smithi001.stdout: Downloading tox-3.15.0-py2.py3-none-any.whl (82 kB) 2024-03-09T14:29:21.873 INFO:teuthology.orchestra.run.smithi001.stdout: ---------------------------------------- 82.1/82.1 KB 1.1 MB/s eta 0:00:00 2024-03-09T14:29:22.047 INFO:teuthology.orchestra.run.smithi001.stdout:Collecting filelock<4,>=3.0.0 2024-03-09T14:29:22.065 INFO:teuthology.orchestra.run.smithi001.stdout: Downloading filelock-3.13.1-py3-none-any.whl (11 kB) 2024-03-09T14:29:22.185 INFO:teuthology.orchestra.run.smithi001.stdout:Collecting pluggy<1,>=0.12.0 2024-03-09T14:29:22.203 INFO:teuthology.orchestra.run.smithi001.stdout: Downloading pluggy-0.13.1-py2.py3-none-any.whl (18 kB) 2024-03-09T14:29:22.488 INFO:teuthology.orchestra.run.smithi001.stdout:Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 2024-03-09T14:29:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: Downloading virtualenv-20.25.1-py3-none-any.whl (3.8 MB) 2024-03-09T14:29:22.677 INFO:teuthology.orchestra.run.smithi001.stdout: ---------------------------------------- 3.8/3.8 MB 22.9 MB/s eta 0:00:00 2024-03-09T14:29:22.812 INFO:teuthology.orchestra.run.smithi001.stdout:Collecting packaging>=14 2024-03-09T14:29:22.830 INFO:teuthology.orchestra.run.smithi001.stdout: Downloading packaging-23.2-py3-none-any.whl (53 kB) 2024-03-09T14:29:22.867 INFO:teuthology.orchestra.run.smithi001.stdout: ---------------------------------------- 53.0/53.0 KB 1.3 MB/s eta 0:00:00 2024-03-09T14:29:22.961 INFO:teuthology.orchestra.run.smithi001.stdout:Collecting py<2,>=1.4.17 2024-03-09T14:29:22.987 INFO:teuthology.orchestra.run.smithi001.stdout: Downloading py-1.11.0-py2.py3-none-any.whl (98 kB) 2024-03-09T14:29:23.035 INFO:teuthology.orchestra.run.smithi001.stdout: ---------------------------------------- 98.7/98.7 KB 2.1 MB/s eta 0:00:00 2024-03-09T14:29:23.127 INFO:teuthology.orchestra.run.smithi001.stdout:Collecting six<2,>=1.14.0 2024-03-09T14:29:23.146 INFO:teuthology.orchestra.run.smithi001.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2024-03-09T14:29:23.249 INFO:teuthology.orchestra.run.smithi001.stdout:Collecting toml>=0.9.4 2024-03-09T14:29:23.270 INFO:teuthology.orchestra.run.smithi001.stdout: Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-03-09T14:29:23.495 INFO:teuthology.orchestra.run.smithi001.stdout:Collecting platformdirs<5,>=3.9.1 2024-03-09T14:29:23.514 INFO:teuthology.orchestra.run.smithi001.stdout: Downloading platformdirs-4.2.0-py3-none-any.whl (17 kB) 2024-03-09T14:29:23.618 INFO:teuthology.orchestra.run.smithi001.stdout:Collecting distlib<1,>=0.3.7 2024-03-09T14:29:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: Downloading distlib-0.3.8-py2.py3-none-any.whl (468 kB) 2024-03-09T14:29:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: --------------------------------------- 468.9/468.9 KB 7.0 MB/s eta 0:00:00 2024-03-09T14:29:23.878 INFO:teuthology.orchestra.run.smithi001.stdout:Installing collected packages: distlib, toml, six, py, pluggy, platformdirs, packaging, filelock, virtualenv, tox 2024-03-09T14:29:25.155 INFO:teuthology.orchestra.run.smithi001.stdout:Successfully installed distlib-0.3.8 filelock-3.13.1 packaging-23.2 platformdirs-4.2.0 pluggy-0.13.1 py-1.11.0 six-1.16.0 toml-0.10.2 tox-3.15.0 virtualenv-20.25.1 2024-03-09T14:29:25.241 INFO:teuthology.run_tasks:Running task workunit... 2024-03-09T14:29:25.254 INFO:tasks.workunit:Pulling workunits from ref c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:29:25.255 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-09T14:29:25.255 DEBUG:teuthology.orchestra.run.smithi001:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-09T14:29:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:29:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-09T14:29:25.261 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-09T14:29:25.309 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-09T14:29:25.310 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-09T14:29:25.358 INFO:tasks.workunit:timeout=3h 2024-03-09T14:29:25.358 INFO:tasks.workunit:cleanup=True 2024-03-09T14:29:25.358 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:29:25.407 INFO:tasks.workunit.client.0.smithi001.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-09T14:30:39.778 INFO:tasks.workunit.client.0.smithi001.stderr:Updating files: 96% (11799/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-09T14:30:40.480 INFO:tasks.workunit.client.0.smithi001.stderr:Note: switching to 'c00d9c609fb42a2a25f1c196bed064ca8bb02413'. 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr:state without impacting any branches by switching back to a branch. 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr: git switch -c 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr:Or undo this operation with: 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr: git switch - 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:30:40.481 INFO:tasks.workunit.client.0.smithi001.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-09T14:30:40.482 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:30:40.482 INFO:tasks.workunit.client.0.smithi001.stderr:HEAD is now at c00d9c609fb DNM: point s3tests at branch cbodley:wip-iam-user-apis 2024-03-09T14:30:40.485 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-09T14:30:40.533 INFO:tasks.workunit.client.0.smithi001.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-09T14:30:40.537 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-09T14:30:40.537 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-09T14:30:41.343 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-09T14:30:41.423 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-09T14:30:41.484 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-09T14:30:41.486 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-09T14:30:41.486 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-09T14:30:41.533 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-09T14:30:41.537 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:30:41.537 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-09T14:30:41.589 INFO:tasks.workunit:Running workunits matching rgw/run-bucket-check.sh on client.0... 2024-03-09T14:30:41.590 INFO:tasks.workunit:Running workunit rgw/run-bucket-check.sh... 2024-03-09T14:30:41.591 DEBUG:teuthology.orchestra.run.smithi001:workunit test rgw/run-bucket-check.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-bucket-check.sh 2024-03-09T14:30:41.644 INFO:tasks.workunit.client.0.smithi001.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-bucket-check.sh 2024-03-09T14:30:41.645 INFO:tasks.workunit.client.0.smithi001.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2024-03-09T14:30:41.645 INFO:tasks.workunit.client.0.smithi001.stderr:+ python3 -m venv /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2024-03-09T14:30:44.733 INFO:tasks.workunit.client.0.smithi001.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/activate 2024-03-09T14:30:44.733 INFO:tasks.workunit.client.0.smithi001.stderr:++ deactivate nondestructive 2024-03-09T14:30:44.733 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -n '' ']' 2024-03-09T14:30:44.733 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -n '' ']' 2024-03-09T14:30:44.733 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2024-03-09T14:30:44.733 INFO:tasks.workunit.client.0.smithi001.stderr:++ hash -r 2024-03-09T14:30:44.733 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -n '' ']' 2024-03-09T14:30:44.733 INFO:tasks.workunit.client.0.smithi001.stderr:++ unset VIRTUAL_ENV 2024-03-09T14:30:44.733 INFO:tasks.workunit.client.0.smithi001.stderr:++ unset VIRTUAL_ENV_PROMPT 2024-03-09T14:30:44.734 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' '!' nondestructive = nondestructive ']' 2024-03-09T14:30:44.734 INFO:tasks.workunit.client.0.smithi001.stderr:++ VIRTUAL_ENV=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2024-03-09T14:30:44.735 INFO:tasks.workunit.client.0.smithi001.stderr:++ export VIRTUAL_ENV 2024-03-09T14:30:44.735 INFO:tasks.workunit.client.0.smithi001.stderr:++ _OLD_VIRTUAL_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-09T14:30:44.735 INFO:tasks.workunit.client.0.smithi001.stderr:++ PATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-09T14:30:44.735 INFO:tasks.workunit.client.0.smithi001.stderr:++ export PATH 2024-03-09T14:30:44.735 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -n '' ']' 2024-03-09T14:30:44.735 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -z '' ']' 2024-03-09T14:30:44.735 INFO:tasks.workunit.client.0.smithi001.stderr:++ _OLD_VIRTUAL_PS1= 2024-03-09T14:30:44.735 INFO:tasks.workunit.client.0.smithi001.stderr:++ PS1='(rgw) ' 2024-03-09T14:30:44.735 INFO:tasks.workunit.client.0.smithi001.stderr:++ export PS1 2024-03-09T14:30:44.735 INFO:tasks.workunit.client.0.smithi001.stderr:++ VIRTUAL_ENV_PROMPT='(rgw) ' 2024-03-09T14:30:44.735 INFO:tasks.workunit.client.0.smithi001.stderr:++ export VIRTUAL_ENV_PROMPT 2024-03-09T14:30:44.736 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2024-03-09T14:30:44.736 INFO:tasks.workunit.client.0.smithi001.stderr:++ hash -r 2024-03-09T14:30:44.736 INFO:tasks.workunit.client.0.smithi001.stderr:+ pip install pip --upgrade 2024-03-09T14:30:45.134 INFO:tasks.workunit.client.0.smithi001.stdout:Requirement already satisfied: pip in /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages (22.0.2) 2024-03-09T14:30:45.389 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting pip 2024-03-09T14:30:45.503 INFO:tasks.workunit.client.0.smithi001.stdout: Downloading pip-24.0-py3-none-any.whl (2.1 MB) 2024-03-09T14:30:45.647 INFO:tasks.workunit.client.0.smithi001.stdout: ---------------------------------------- 2.1/2.1 MB 14.9 MB/s eta 0:00:00 2024-03-09T14:30:45.686 INFO:tasks.workunit.client.0.smithi001.stdout:Installing collected packages: pip 2024-03-09T14:30:45.686 INFO:tasks.workunit.client.0.smithi001.stdout: Attempting uninstall: pip 2024-03-09T14:30:45.686 INFO:tasks.workunit.client.0.smithi001.stdout: Found existing installation: pip 22.0.2 2024-03-09T14:30:45.979 INFO:tasks.workunit.client.0.smithi001.stdout: Uninstalling pip-22.0.2: 2024-03-09T14:30:45.986 INFO:tasks.workunit.client.0.smithi001.stdout: Successfully uninstalled pip-22.0.2 2024-03-09T14:30:46.967 INFO:tasks.workunit.client.0.smithi001.stdout:Successfully installed pip-24.0 2024-03-09T14:30:47.027 INFO:tasks.workunit.client.0.smithi001.stderr:+ pip install boto3 2024-03-09T14:30:48.093 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting boto3 2024-03-09T14:30:48.210 INFO:tasks.workunit.client.0.smithi001.stdout: Downloading boto3-1.34.59-py3-none-any.whl.metadata (6.6 kB) 2024-03-09T14:30:48.949 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting botocore<1.35.0,>=1.34.59 (from boto3) 2024-03-09T14:30:48.966 INFO:tasks.workunit.client.0.smithi001.stdout: Downloading botocore-1.34.59-py3-none-any.whl.metadata (5.7 kB) 2024-03-09T14:30:49.133 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting jmespath<2.0.0,>=0.7.1 (from boto3) 2024-03-09T14:30:49.153 INFO:tasks.workunit.client.0.smithi001.stdout: Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB) 2024-03-09T14:30:49.304 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting s3transfer<0.11.0,>=0.10.0 (from boto3) 2024-03-09T14:30:49.323 INFO:tasks.workunit.client.0.smithi001.stdout: Downloading s3transfer-0.10.0-py3-none-any.whl.metadata (1.7 kB) 2024-03-09T14:30:49.509 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.35.0,>=1.34.59->boto3) 2024-03-09T14:30:49.528 INFO:tasks.workunit.client.0.smithi001.stdout: Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB) 2024-03-09T14:30:49.870 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting urllib3<2.1,>=1.25.4 (from botocore<1.35.0,>=1.34.59->boto3) 2024-03-09T14:30:49.888 INFO:tasks.workunit.client.0.smithi001.stdout: Downloading urllib3-2.0.7-py3-none-any.whl.metadata (6.6 kB) 2024-03-09T14:30:50.307 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.35.0,>=1.34.59->boto3) 2024-03-09T14:30:50.324 INFO:tasks.workunit.client.0.smithi001.stdout: Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB) 2024-03-09T14:30:50.409 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading boto3-1.34.59-py3-none-any.whl (139 kB) 2024-03-09T14:30:50.608 INFO:tasks.workunit.client.0.smithi001.stdout: --------------------------------------- 139.3/139.3 kB 669.2 kB/s eta 0:00:00 2024-03-09T14:30:50.631 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading botocore-1.34.59-py3-none-any.whl (12.0 MB) 2024-03-09T14:30:50.954 INFO:tasks.workunit.client.0.smithi001.stdout: ---------------------------------------- 12.0/12.0 MB 38.7 MB/s eta 0:00:00 2024-03-09T14:30:50.973 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading jmespath-1.0.1-py3-none-any.whl (20 kB) 2024-03-09T14:30:51.041 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading s3transfer-0.10.0-py3-none-any.whl (82 kB) 2024-03-09T14:30:51.101 INFO:tasks.workunit.client.0.smithi001.stdout: ---------------------------------------- 82.1/82.1 kB 1.3 MB/s eta 0:00:00 2024-03-09T14:30:51.122 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) 2024-03-09T14:30:51.184 INFO:tasks.workunit.client.0.smithi001.stdout: ---------------------------------------- 229.9/229.9 kB 3.8 MB/s eta 0:00:00 2024-03-09T14:30:51.205 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading urllib3-2.0.7-py3-none-any.whl (124 kB) 2024-03-09T14:30:51.260 INFO:tasks.workunit.client.0.smithi001.stdout: ---------------------------------------- 124.2/124.2 kB 2.3 MB/s eta 0:00:00 2024-03-09T14:30:51.280 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2024-03-09T14:30:51.489 INFO:tasks.workunit.client.0.smithi001.stdout:Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3 2024-03-09T14:30:52.819 INFO:tasks.workunit.client.0.smithi001.stdout:Successfully installed boto3-1.34.59 botocore-1.34.59 jmespath-1.0.1 python-dateutil-2.9.0.post0 s3transfer-0.10.0 six-1.16.0 urllib3-2.0.7 2024-03-09T14:30:53.336 INFO:tasks.workunit.client.0.smithi001.stderr:+ /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py 2024-03-09T14:30:53.486 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin user create --uid check-tester --display-name "Check Testing" --access-key OJODXSLNX4LUNHQG99PA --secret 3l6ffld34qaymfomuh832j94738aie2x4p2o8h6n 2024-03-09T14:30:53.524 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:30:53.524 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:30:53.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.569+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.569+0000 7f1cb3d91980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:53.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.569+0000 7f1cb3d91980 20 realm 2024-03-09T14:30:53.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.569+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.569+0000 7f1cb3d91980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:53.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.569+0000 7f1cb3d91980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:30:53.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.569+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.569+0000 7f1cb3d91980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:53.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.569+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.573+0000 7f1cb3d91980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:30:53.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.573+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.574 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.573+0000 7f1cb3d91980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:30:53.574 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.573+0000 7f1cb3d91980 20 searching for the correct realm 2024-03-09T14:30:53.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.581+0000 7f1cb20aa640 0 -- 172.21.15.1:0/1677532964 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55d20e0f0e60 msgr2=0x55d20e0f32c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:30:53.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.585+0000 7f1cb10a8640 0 -- 172.21.15.1:0/1677532964 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f1ca4019040 msgr2=0x7f1ca4008940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:30:53.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:30:53.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:30:53.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:30:53.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:30:53.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:30:53.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:30:53.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:53.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:30:53.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:30:53.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 zone default found 2024-03-09T14:30:53.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 4 Realm: () 2024-03-09T14:30:53.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:30:53.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:30:53.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:30:53.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 zonegroup default 2024-03-09T14:30:53.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:53.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.593+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.597+0000 7f1cb3d91980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:53.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.597+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.597+0000 7f1cb3d91980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:53.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.597+0000 7f1cb3d91980 20 started sync module instance, tier type = 2024-03-09T14:30:53.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.597+0000 7f1cb3d91980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:30:53.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.613+0000 7f1cb3d91980 20 add_watcher() i=0 2024-03-09T14:30:53.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.613+0000 7f1cb3d91980 20 add_watcher() i=1 2024-03-09T14:30:53.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.613+0000 7f1cb3d91980 20 add_watcher() i=2 2024-03-09T14:30:53.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.613+0000 7f1cb3d91980 20 add_watcher() i=3 2024-03-09T14:30:53.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.613+0000 7f1cb3d91980 20 add_watcher() i=4 2024-03-09T14:30:53.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.613+0000 7f1cb3d91980 20 add_watcher() i=5 2024-03-09T14:30:53.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.613+0000 7f1cb3d91980 20 add_watcher() i=6 2024-03-09T14:30:53.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.613+0000 7f1cb3d91980 20 add_watcher() i=7 2024-03-09T14:30:53.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.613+0000 7f1cb3d91980 2 all 8 watchers are set, enabling cache 2024-03-09T14:30:53.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1b637fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:30:53.618 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:30:53.618 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:30:53.618 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 5 note: GC not initialized 2024-03-09T14:30:53.618 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1b627fc640 20 reqs_thread_entry: start 2024-03-09T14:30:53.618 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 20 init_complete bucket index max shards: 1 2024-03-09T14:30:53.618 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 20 Filter name: none 2024-03-09T14:30:53.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1ad3fff640 20 reqs_thread_entry: start 2024-03-09T14:30:53.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 10 cache get: name=default.rgw.meta+users.uid+check-tester : miss 2024-03-09T14:30:53.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:53.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 10 cache put: name=default.rgw.meta+users.uid+check-tester info.flags=0x0 2024-03-09T14:30:53.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 10 adding default.rgw.meta+users.uid+check-tester to cache LRU end 2024-03-09T14:30:53.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 10 cache get: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA : miss 2024-03-09T14:30:53.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 20 rados->read ofs=0 len=0 2024-03-09T14:30:53.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:53.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 10 cache put: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA info.flags=0x0 2024-03-09T14:30:53.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 10 adding default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA to cache LRU end 2024-03-09T14:30:53.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 10 cache get: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA : hit (negative entry) 2024-03-09T14:30:53.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.617+0000 7f1cb3d91980 10 cache get: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA : hit (negative entry) 2024-03-09T14:30:53.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.621+0000 7f1cb3d91980 10 cache put: name=default.rgw.meta+users.uid+check-tester info.flags=0x17 2024-03-09T14:30:53.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.621+0000 7f1cb3d91980 10 moving default.rgw.meta+users.uid+check-tester to cache LRU end 2024-03-09T14:30:53.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.621+0000 7f1cb3d91980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:check-tester, ofs0, ns] 2024-03-09T14:30:53.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.621+0000 7f1b957fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888064 cookie 94360667415888 notifier 4662 bl.length()=593 2024-03-09T14:30:53.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.625+0000 7f1b957fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+check-tester info.flags=0x17 2024-03-09T14:30:53.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.625+0000 7f1b957fa640 10 rgw watcher librados: moving default.rgw.meta+users.uid+check-tester to cache LRU end 2024-03-09T14:30:53.654 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.653+0000 7f1cb3d91980 10 cache put: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA info.flags=0x7 2024-03-09T14:30:53.654 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.653+0000 7f1cb3d91980 10 moving default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA to cache LRU end 2024-03-09T14:30:53.654 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.653+0000 7f1cb3d91980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.keys:OJODXSLNX4LUNHQG99PA, ofs0, ns] 2024-03-09T14:30:53.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.653+0000 7f1b957fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888064 cookie 94360667366640 notifier 4662 bl.length()=186 2024-03-09T14:30:53.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.653+0000 7f1b957fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA info.flags=0x7 2024-03-09T14:30:53.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.653+0000 7f1b957fa640 10 rgw watcher librados: moving default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA to cache LRU end 2024-03-09T14:30:53.659 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.657+0000 7f1b637fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:30:53.664 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.661+0000 7f1cb3d91980 20 remove_watcher() i=0 2024-03-09T14:30:53.664 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.661+0000 7f1cb3d91980 2 removed watcher, disabling cache 2024-03-09T14:30:53.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.665+0000 7f1cb3d91980 20 remove_watcher() i=1 2024-03-09T14:30:53.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.669+0000 7f1cb3d91980 20 remove_watcher() i=2 2024-03-09T14:30:53.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.669+0000 7f1cb3d91980 20 remove_watcher() i=3 2024-03-09T14:30:53.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.673+0000 7f1cb3d91980 20 remove_watcher() i=4 2024-03-09T14:30:53.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.673+0000 7f1cb3d91980 20 remove_watcher() i=5 2024-03-09T14:30:53.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.677+0000 7f1cb3d91980 20 remove_watcher() i=6 2024-03-09T14:30:53.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:53.677+0000 7f1cb3d91980 20 remove_watcher() i=7 2024-03-09T14:30:53.960 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:53.960 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:54.227 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:54.227 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:54.410 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:54.411 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:54.963 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:54.963 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:55.766 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:55.766 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:56.024 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:56.024 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:56.152 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:56.152 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:56.230 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:56.230 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:56.431 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:56.431 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:56.819 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:56.819 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:56.924 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:56.924 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:57.113 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:57.113 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:57.191 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:57.191 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:57.338 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:57.338 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:57.412 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:57.412 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:57.558 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:30:57.558 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:30:57.629 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: recalculated bucket check stats are correct 2024-03-09T14:30:57.630 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:30:57.630 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket check --fix --bucket check-bucket 2024-03-09T14:30:57.667 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:30:57.667 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:30:57.714 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.713+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.713+0000 7f1e5d5ab980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:57.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.713+0000 7f1e5d5ab980 20 realm 2024-03-09T14:30:57.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.713+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.713+0000 7f1e5d5ab980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:57.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.713+0000 7f1e5d5ab980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:30:57.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.713+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.713+0000 7f1e5d5ab980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:57.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.713+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.717+0000 7f1e5d5ab980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:30:57.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.717+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.717+0000 7f1e5d5ab980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:30:57.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.717+0000 7f1e5d5ab980 20 searching for the correct realm 2024-03-09T14:30:57.724 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.721+0000 7f1e5b8c4640 0 -- 172.21.15.1:0/1317150409 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f1e500e4040 msgr2=0x7f1e500e5940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:30:57.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.729+0000 7f1e5b0c3640 0 -- 172.21.15.1:0/1317150409 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f1e500e3920 msgr2=0x7f1e500d7420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:30:57.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.945+0000 7f1e5d5ab980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:30:57.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.945+0000 7f1e5d5ab980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:30:57.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.945+0000 7f1e5d5ab980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:30:57.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.945+0000 7f1e5d5ab980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:30:57.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.945+0000 7f1e5d5ab980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:30:57.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.945+0000 7f1e5d5ab980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:30:57.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.945+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.945+0000 7f1e5d5ab980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:57.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.945+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.949+0000 7f1e5d5ab980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:30:57.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.949+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.950 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.949+0000 7f1e5d5ab980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:30:57.950 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.949+0000 7f1e5d5ab980 20 zone default found 2024-03-09T14:30:57.950 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.949+0000 7f1e5d5ab980 4 Realm: () 2024-03-09T14:30:57.951 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.949+0000 7f1e5d5ab980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:30:57.951 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.949+0000 7f1e5d5ab980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:30:57.951 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.949+0000 7f1e5d5ab980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:30:57.951 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.949+0000 7f1e5d5ab980 20 zonegroup default 2024-03-09T14:30:57.951 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.949+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.951 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.949+0000 7f1e5d5ab980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:57.951 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.949+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.953+0000 7f1e5d5ab980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:57.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.953+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.953+0000 7f1e5d5ab980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:57.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.953+0000 7f1e5d5ab980 20 started sync module instance, tier type = 2024-03-09T14:30:57.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.953+0000 7f1e5d5ab980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:30:57.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.977+0000 7f1e5d5ab980 20 add_watcher() i=0 2024-03-09T14:30:57.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.977+0000 7f1e5d5ab980 20 add_watcher() i=1 2024-03-09T14:30:57.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.977+0000 7f1e5d5ab980 20 add_watcher() i=2 2024-03-09T14:30:57.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.977+0000 7f1e5d5ab980 20 add_watcher() i=3 2024-03-09T14:30:57.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.977+0000 7f1e5d5ab980 20 add_watcher() i=4 2024-03-09T14:30:57.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.977+0000 7f1e5d5ab980 20 add_watcher() i=5 2024-03-09T14:30:57.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.977+0000 7f1e5d5ab980 20 add_watcher() i=6 2024-03-09T14:30:57.982 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.981+0000 7f1e5d5ab980 20 add_watcher() i=7 2024-03-09T14:30:57.982 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.981+0000 7f1e5d5ab980 2 all 8 watchers are set, enabling cache 2024-03-09T14:30:57.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.985+0000 7f1d11fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:30:57.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.985+0000 7f1e5d5ab980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:30:57.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.985+0000 7f1e5d5ab980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:30:57.988 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.985+0000 7f1e5d5ab980 5 note: GC not initialized 2024-03-09T14:30:57.988 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.985+0000 7f1d10fe1640 20 reqs_thread_entry: start 2024-03-09T14:30:57.988 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 20 init_complete bucket index max shards: 1 2024-03-09T14:30:57.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 20 Filter name: none 2024-03-09T14:30:57.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1c8a7fc640 20 reqs_thread_entry: start 2024-03-09T14:30:57.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2024-03-09T14:30:57.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:30:57.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2024-03-09T14:30:57.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2024-03-09T14:30:57.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 : miss 2024-03-09T14:30:57.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 20 rados->read ofs=0 len=0 2024-03-09T14:30:57.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:30:57.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 info.flags=0x17 2024-03-09T14:30:57.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 to cache LRU end 2024-03-09T14:30:57.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:30:57.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 10 updating xattr: name=user.rgw.acl bl.length()=179 2024-03-09T14:30:57.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 2024-03-09T14:30:57.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:30:57.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:30:57.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:30:57.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:30:57.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:30:57.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.989+0000 7f1e5d5ab980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:30:57.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:57.997+0000 7f1d11fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:30:58.005 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.005+0000 7f1e5d5ab980 20 remove_watcher() i=0 2024-03-09T14:30:58.005 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.005+0000 7f1e5d5ab980 2 removed watcher, disabling cache 2024-03-09T14:30:58.007 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.005+0000 7f1e5d5ab980 20 remove_watcher() i=1 2024-03-09T14:30:58.009 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.009+0000 7f1e5d5ab980 20 remove_watcher() i=2 2024-03-09T14:30:58.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.009+0000 7f1e5d5ab980 20 remove_watcher() i=3 2024-03-09T14:30:58.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.013+0000 7f1e5d5ab980 20 remove_watcher() i=4 2024-03-09T14:30:58.017 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.017+0000 7f1e5d5ab980 20 remove_watcher() i=5 2024-03-09T14:30:58.018 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.017+0000 7f1e5d5ab980 20 remove_watcher() i=6 2024-03-09T14:30:58.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.021+0000 7f1e5d5ab980 20 remove_watcher() i=7 2024-03-09T14:30:58.044 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket check-bucket 2024-03-09T14:30:58.076 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:30:58.076 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:30:58.123 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.121+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.125+0000 7ff943704980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.125+0000 7ff943704980 20 realm 2024-03-09T14:30:58.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.125+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.125+0000 7ff943704980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.125+0000 7ff943704980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:30:58.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.125+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.125+0000 7ff943704980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.125+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.125+0000 7ff943704980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:30:58.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.125+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.125+0000 7ff943704980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:30:58.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.125+0000 7ff943704980 20 searching for the correct realm 2024-03-09T14:30:58.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.129+0000 7ff94121c640 0 -- 172.21.15.1:0/3115500323 >> [v2:172.21.15.1:6824/1666056338,v1:172.21.15.1:6825/1666056338] conn(0x55f636d32d50 msgr2=0x7ff93c0f06d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:30:58.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.137+0000 7ff941a1d640 0 -- 172.21.15.1:0/3115500323 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7ff938039070 msgr2=0x7ff938039450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:30:58.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:30:58.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:30:58.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:30:58.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:30:58.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:30:58.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:30:58.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:30:58.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:30:58.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 zone default found 2024-03-09T14:30:58.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 4 Realm: () 2024-03-09T14:30:58.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:30:58.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:30:58.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:30:58.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 zonegroup default 2024-03-09T14:30:58.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.149+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.153+0000 7ff943704980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.153+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.153+0000 7ff943704980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.153+0000 7ff943704980 20 started sync module instance, tier type = 2024-03-09T14:30:58.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.153+0000 7ff943704980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:30:58.169 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.169+0000 7ff943704980 20 add_watcher() i=0 2024-03-09T14:30:58.169 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.169+0000 7ff943704980 20 add_watcher() i=1 2024-03-09T14:30:58.169 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.169+0000 7ff943704980 20 add_watcher() i=2 2024-03-09T14:30:58.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.169+0000 7ff943704980 20 add_watcher() i=3 2024-03-09T14:30:58.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.169+0000 7ff943704980 20 add_watcher() i=4 2024-03-09T14:30:58.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.169+0000 7ff943704980 20 add_watcher() i=5 2024-03-09T14:30:58.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.169+0000 7ff943704980 20 add_watcher() i=6 2024-03-09T14:30:58.171 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.169+0000 7ff943704980 20 add_watcher() i=7 2024-03-09T14:30:58.171 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.169+0000 7ff943704980 2 all 8 watchers are set, enabling cache 2024-03-09T14:30:58.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff7f77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:30:58.174 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:30:58.174 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:30:58.174 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 5 note: GC not initialized 2024-03-09T14:30:58.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff7f67e4640 20 reqs_thread_entry: start 2024-03-09T14:30:58.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 20 init_complete bucket index max shards: 1 2024-03-09T14:30:58.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 20 Filter name: none 2024-03-09T14:30:58.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff767fff640 20 reqs_thread_entry: start 2024-03-09T14:30:58.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:30:58.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:30:58.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:30:58.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:30:58.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.173+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.177+0000 7ff943704980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:30:58.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.177+0000 7ff943704980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:30:58.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.177+0000 7ff943704980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.177+0000 7ff943704980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.177+0000 7ff7f77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:30:58.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.181+0000 7ff943704980 20 remove_watcher() i=0 2024-03-09T14:30:58.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.181+0000 7ff943704980 2 removed watcher, disabling cache 2024-03-09T14:30:58.186 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.185+0000 7ff943704980 20 remove_watcher() i=1 2024-03-09T14:30:58.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.189+0000 7ff943704980 20 remove_watcher() i=2 2024-03-09T14:30:58.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.189+0000 7ff943704980 20 remove_watcher() i=3 2024-03-09T14:30:58.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.193+0000 7ff943704980 20 remove_watcher() i=4 2024-03-09T14:30:58.194 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.193+0000 7ff943704980 20 remove_watcher() i=5 2024-03-09T14:30:58.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.193+0000 7ff943704980 20 remove_watcher() i=6 2024-03-09T14:30:58.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.197+0000 7ff943704980 20 remove_watcher() i=7 2024-03-09T14:30:58.211 INFO:tasks.workunit.client.0.smithi001.stderr:{'rgw.main': {'size': 54, 'size_actual': 24576, 'size_utilized': 100, 'size_kb': 1, 'size_kb_actual': 24, 'size_kb_utilized': 1, 'num_objects': 6}} 2024-03-09T14:30:58.211 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: bucket check unlinked does not report normal entries 2024-03-09T14:30:58.211 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:30:58.211 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --min-age-hours 0 --dump-keys 2024-03-09T14:30:58.241 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:30:58.241 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:30:58.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.281+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.285+0000 7ffbbf56b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.285+0000 7ffbbf56b980 20 realm 2024-03-09T14:30:58.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.285+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.285+0000 7ffbbf56b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.285+0000 7ffbbf56b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:30:58.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.285+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.285+0000 7ffbbf56b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.285+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.285+0000 7ffbbf56b980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:30:58.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.285+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.288 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.285+0000 7ffbbf56b980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:30:58.288 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.285+0000 7ffbbf56b980 20 searching for the correct realm 2024-03-09T14:30:58.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.293+0000 7ffbbd884640 0 -- 172.21.15.1:0/610596678 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55adc108e8f0 msgr2=0x55adc1090ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:30:58.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.301+0000 7ffbb4882640 0 -- 172.21.15.1:0/610596678 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7ffbb0074070 msgr2=0x7ffbb005be10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:30:58.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:30:58.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:30:58.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:30:58.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:30:58.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:30:58.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:30:58.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:30:58.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:30:58.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 zone default found 2024-03-09T14:30:58.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 4 Realm: () 2024-03-09T14:30:58.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:30:58.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:30:58.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:30:58.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 zonegroup default 2024-03-09T14:30:58.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:30:58.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 started sync module instance, tier type = 2024-03-09T14:30:58.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.309+0000 7ffbbf56b980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:30:58.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.329+0000 7ffbbf56b980 20 add_watcher() i=0 2024-03-09T14:30:58.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.329+0000 7ffbbf56b980 20 add_watcher() i=1 2024-03-09T14:30:58.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.329+0000 7ffbbf56b980 20 add_watcher() i=2 2024-03-09T14:30:58.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.329+0000 7ffbbf56b980 20 add_watcher() i=3 2024-03-09T14:30:58.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.329+0000 7ffbbf56b980 20 add_watcher() i=4 2024-03-09T14:30:58.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.329+0000 7ffbbf56b980 20 add_watcher() i=5 2024-03-09T14:30:58.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.329+0000 7ffbbf56b980 20 add_watcher() i=6 2024-03-09T14:30:58.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.329+0000 7ffbbf56b980 20 add_watcher() i=7 2024-03-09T14:30:58.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.329+0000 7ffbbf56b980 2 all 8 watchers are set, enabling cache 2024-03-09T14:30:58.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffa737e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:30:58.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:30:58.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:30:58.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 5 note: GC not initialized 2024-03-09T14:30:58.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffa727e4640 20 reqs_thread_entry: start 2024-03-09T14:30:58.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 20 init_complete bucket index max shards: 1 2024-03-09T14:30:58.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 20 Filter name: none 2024-03-09T14:30:58.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ff9e3fff640 20 reqs_thread_entry: start 2024-03-09T14:30:58.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2024-03-09T14:30:58.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:30:58.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2024-03-09T14:30:58.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2024-03-09T14:30:58.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 : miss 2024-03-09T14:30:58.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 20 rados->read ofs=0 len=0 2024-03-09T14:30:58.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:30:58.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 info.flags=0x17 2024-03-09T14:30:58.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 to cache LRU end 2024-03-09T14:30:58.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:30:58.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 10 updating xattr: name=user.rgw.acl bl.length()=179 2024-03-09T14:30:58.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 2024-03-09T14:30:58.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.333+0000 7ffbbf56b980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:30:58.337 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.337+0000 7ffbbf56b980 1 NOTICE: finished shard 0 (0 entries found) 2024-03-09T14:30:58.337 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.337+0000 7ffbbf56b980 1 NOTICE: finished all shards (0 entries found) 2024-03-09T14:30:58.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.337+0000 7ffa737e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:30:58.342 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.341+0000 7ffbbf56b980 20 remove_watcher() i=0 2024-03-09T14:30:58.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.341+0000 7ffbbf56b980 2 removed watcher, disabling cache 2024-03-09T14:30:58.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.345+0000 7ffbbf56b980 20 remove_watcher() i=1 2024-03-09T14:30:58.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.345+0000 7ffbbf56b980 20 remove_watcher() i=2 2024-03-09T14:30:58.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.349+0000 7ffbbf56b980 20 remove_watcher() i=3 2024-03-09T14:30:58.351 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.349+0000 7ffbbf56b980 20 remove_watcher() i=4 2024-03-09T14:30:58.352 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.353+0000 7ffbbf56b980 20 remove_watcher() i=5 2024-03-09T14:30:58.354 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.353+0000 7ffbbf56b980 20 remove_watcher() i=6 2024-03-09T14:30:58.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:30:58.353+0000 7ffbbf56b980 20 remove_watcher() i=7 2024-03-09T14:30:58.369 INFO:tasks.workunit.client.0.smithi001.stderr:ceph config set client rgw_debug_inject_set_olh_err 2 2024-03-09T14:30:58.785 INFO:tasks.workunit.client.0.smithi001.stderr:ceph config set client rgw_debug_inject_olh_cancel_modification_err true 2024-03-09T14:31:00.183 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:00.183 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:00.381 INFO:tasks.workunit.client.0.smithi001.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2024-03-09T14:31:00.382 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket check-bucket --object c 2024-03-09T14:31:00.419 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:00.419 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:00.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.465+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.465+0000 7f8c30b21980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.465+0000 7f8c30b21980 20 realm 2024-03-09T14:31:00.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.465+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.465+0000 7f8c30b21980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.465+0000 7f8c30b21980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:00.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.465+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.465+0000 7f8c30b21980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.465+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.469+0000 7f8c30b21980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:00.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.469+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.469+0000 7f8c30b21980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:00.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.469+0000 7f8c30b21980 20 searching for the correct realm 2024-03-09T14:31:00.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.477+0000 7f8c2de38640 0 -- 172.21.15.1:0/150459 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55a36255c0b0 msgr2=0x55a36255e500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:00.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.481+0000 7f8c2e639640 0 -- 172.21.15.1:0/150459 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f8c2403c0e0 msgr2=0x7f8c2403e960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:00.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.489+0000 7f8c30b21980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:00.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.489+0000 7f8c30b21980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:00.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.489+0000 7f8c30b21980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:00.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.489+0000 7f8c30b21980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:00.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.489+0000 7f8c30b21980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:00.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.489+0000 7f8c30b21980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:00.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.489+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.489+0000 7f8c30b21980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.489+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.489+0000 7f8c30b21980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:00.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.489+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:00.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 20 zone default found 2024-03-09T14:31:00.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 4 Realm: () 2024-03-09T14:31:00.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:00.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:00.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:00.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 20 zonegroup default 2024-03-09T14:31:00.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 20 started sync module instance, tier type = 2024-03-09T14:31:00.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.493+0000 7f8c30b21980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:00.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.509+0000 7f8c30b21980 20 add_watcher() i=0 2024-03-09T14:31:00.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.509+0000 7f8c30b21980 20 add_watcher() i=1 2024-03-09T14:31:00.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.509+0000 7f8c30b21980 20 add_watcher() i=2 2024-03-09T14:31:00.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.509+0000 7f8c30b21980 20 add_watcher() i=3 2024-03-09T14:31:00.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.509+0000 7f8c30b21980 20 add_watcher() i=4 2024-03-09T14:31:00.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.509+0000 7f8c30b21980 20 add_watcher() i=5 2024-03-09T14:31:00.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.509+0000 7f8c30b21980 20 add_watcher() i=6 2024-03-09T14:31:00.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.509+0000 7f8c30b21980 20 add_watcher() i=7 2024-03-09T14:31:00.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.509+0000 7f8c30b21980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:00.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8ae4fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:00.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:00.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:00.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 5 note: GC not initialized 2024-03-09T14:31:00.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8a5f7fe640 20 reqs_thread_entry: start 2024-03-09T14:31:00.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:00.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 20 Filter name: none 2024-03-09T14:31:00.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8a5d7fa640 20 reqs_thread_entry: start 2024-03-09T14:31:00.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:00.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:00.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:00.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2024-03-09T14:31:00.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 20 INFO: main: starting shard=0 2024-03-09T14:31:00.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.513+0000 7f8c30b21980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:00.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.517+0000 7f8c30b21980 20 INFO: main: bi_list() returned without error; entries.size()=5, is_truncated=0, marker= 2024-03-09T14:31:00.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.517+0000 7f8c30b21980 20 INFO: main: done 2024-03-09T14:31:00.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.517+0000 7f8ae4fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:00.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.521+0000 7f8c30b21980 20 remove_watcher() i=0 2024-03-09T14:31:00.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.521+0000 7f8c30b21980 2 removed watcher, disabling cache 2024-03-09T14:31:00.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.525+0000 7f8c30b21980 20 remove_watcher() i=1 2024-03-09T14:31:00.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.525+0000 7f8c30b21980 20 remove_watcher() i=2 2024-03-09T14:31:00.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.529+0000 7f8c30b21980 20 remove_watcher() i=3 2024-03-09T14:31:00.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.529+0000 7f8c30b21980 20 remove_watcher() i=4 2024-03-09T14:31:00.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.533+0000 7f8c30b21980 20 remove_watcher() i=5 2024-03-09T14:31:00.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.533+0000 7f8c30b21980 20 remove_watcher() i=6 2024-03-09T14:31:00.534 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.533+0000 7f8c30b21980 20 remove_watcher() i=7 2024-03-09T14:31:00.548 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:00.548 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:00.576 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:00.576 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:00.617 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:00.617 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:00.781 INFO:tasks.workunit.client.0.smithi001.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2024-03-09T14:31:00.782 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket check-bucket --object d 2024-03-09T14:31:00.817 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:00.817 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:00.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.861+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.861+0000 7f8e4a6a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.861+0000 7f8e4a6a6980 20 realm 2024-03-09T14:31:00.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.861+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.861+0000 7f8e4a6a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.861+0000 7f8e4a6a6980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:00.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.861+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.861+0000 7f8e4a6a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.861+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.865+0000 7f8e4a6a6980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:00.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.865+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.865+0000 7f8e4a6a6980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:00.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.865+0000 7f8e4a6a6980 20 searching for the correct realm 2024-03-09T14:31:00.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.873+0000 7f8e479bd640 0 -- 172.21.15.1:0/4119145964 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x55ae6ba55a50 msgr2=0x55ae6ba57e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:00.879 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.877+0000 7f8e481be640 0 -- 172.21.15.1:0/4119145964 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f8e3c0fc280 msgr2=0x7f8e3c0f4760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:00.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.885+0000 7f8e4a6a6980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:00.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.885+0000 7f8e4a6a6980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:00.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.885+0000 7f8e4a6a6980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:00.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.885+0000 7f8e4a6a6980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:00.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.885+0000 7f8e4a6a6980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:00.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.885+0000 7f8e4a6a6980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:00.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:00.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.890 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:00.890 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 zone default found 2024-03-09T14:31:00.890 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 4 Realm: () 2024-03-09T14:31:00.890 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:00.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:00.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:00.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 zonegroup default 2024-03-09T14:31:00.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:00.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 started sync module instance, tier type = 2024-03-09T14:31:00.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.889+0000 7f8e4a6a6980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:00.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.905+0000 7f8e4a6a6980 20 add_watcher() i=0 2024-03-09T14:31:00.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.905+0000 7f8e4a6a6980 20 add_watcher() i=1 2024-03-09T14:31:00.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.905+0000 7f8e4a6a6980 20 add_watcher() i=2 2024-03-09T14:31:00.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.905+0000 7f8e4a6a6980 20 add_watcher() i=3 2024-03-09T14:31:00.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.905+0000 7f8e4a6a6980 20 add_watcher() i=4 2024-03-09T14:31:00.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.905+0000 7f8e4a6a6980 20 add_watcher() i=5 2024-03-09T14:31:00.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.905+0000 7f8e4a6a6980 20 add_watcher() i=6 2024-03-09T14:31:00.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.909+0000 7f8e4a6a6980 20 add_watcher() i=7 2024-03-09T14:31:00.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.909+0000 7f8e4a6a6980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:00.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.909+0000 7f8cfefe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:00.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.909+0000 7f8e4a6a6980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:00.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.909+0000 7f8e4a6a6980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:00.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.909+0000 7f8e4a6a6980 5 note: GC not initialized 2024-03-09T14:31:00.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8cfdfe3640 20 reqs_thread_entry: start 2024-03-09T14:31:00.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8e4a6a6980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:00.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8e4a6a6980 20 Filter name: none 2024-03-09T14:31:00.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8cf6ffd640 20 reqs_thread_entry: start 2024-03-09T14:31:00.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8e4a6a6980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:00.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8e4a6a6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:00.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8e4a6a6980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:00.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8e4a6a6980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:00.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8e4a6a6980 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2024-03-09T14:31:00.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8e4a6a6980 20 INFO: main: starting shard=0 2024-03-09T14:31:00.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8e4a6a6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:00.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8e4a6a6980 20 INFO: main: bi_list() returned without error; entries.size()=5, is_truncated=0, marker= 2024-03-09T14:31:00.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8e4a6a6980 20 INFO: main: done 2024-03-09T14:31:00.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.913+0000 7f8cfefe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:00.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.921+0000 7f8e4a6a6980 20 remove_watcher() i=0 2024-03-09T14:31:00.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.921+0000 7f8e4a6a6980 2 removed watcher, disabling cache 2024-03-09T14:31:00.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.921+0000 7f8e4a6a6980 20 remove_watcher() i=1 2024-03-09T14:31:00.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.925+0000 7f8e4a6a6980 20 remove_watcher() i=2 2024-03-09T14:31:00.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.925+0000 7f8e4a6a6980 20 remove_watcher() i=3 2024-03-09T14:31:00.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.929+0000 7f8e4a6a6980 20 remove_watcher() i=4 2024-03-09T14:31:00.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.929+0000 7f8e4a6a6980 20 remove_watcher() i=5 2024-03-09T14:31:00.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.929+0000 7f8e4a6a6980 20 remove_watcher() i=6 2024-03-09T14:31:00.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:00.933+0000 7f8e4a6a6980 20 remove_watcher() i=7 2024-03-09T14:31:00.947 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:00.947 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:00.973 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:00.973 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:01.089 INFO:tasks.workunit.client.0.smithi001.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2024-03-09T14:31:01.090 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket check-bucket --object e 2024-03-09T14:31:01.124 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:01.124 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:01.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.165+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.165+0000 7fab879f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.165+0000 7fab879f9980 20 realm 2024-03-09T14:31:01.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.165+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.165+0000 7fab879f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.165+0000 7fab879f9980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:01.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.165+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.165+0000 7fab879f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.165+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.169 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.169+0000 7fab879f9980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:01.169 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.169+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.169+0000 7fab879f9980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:01.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.169+0000 7fab879f9980 20 searching for the correct realm 2024-03-09T14:31:01.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.173+0000 7fab84d10640 0 -- 172.21.15.1:0/1446443243 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x560e53895740 msgr2=0x560e53897b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:01.182 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.181+0000 7fab85511640 0 -- 172.21.15.1:0/1446443243 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fab7c076030 msgr2=0x7fab7c05f770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:01.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:01.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:01.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:01.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:01.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:01.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:01.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:01.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:01.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 zone default found 2024-03-09T14:31:01.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 4 Realm: () 2024-03-09T14:31:01.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:01.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:01.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:01.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 zonegroup default 2024-03-09T14:31:01.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.189+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.193+0000 7fab879f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.193+0000 7fab879f9980 20 started sync module instance, tier type = 2024-03-09T14:31:01.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.193+0000 7fab879f9980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:01.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.209+0000 7fab879f9980 20 add_watcher() i=0 2024-03-09T14:31:01.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.209+0000 7fab879f9980 20 add_watcher() i=1 2024-03-09T14:31:01.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.209+0000 7fab879f9980 20 add_watcher() i=2 2024-03-09T14:31:01.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.209+0000 7fab879f9980 20 add_watcher() i=3 2024-03-09T14:31:01.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.209+0000 7fab879f9980 20 add_watcher() i=4 2024-03-09T14:31:01.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.209+0000 7fab879f9980 20 add_watcher() i=5 2024-03-09T14:31:01.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.209+0000 7fab879f9980 20 add_watcher() i=6 2024-03-09T14:31:01.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.209+0000 7fab879f9980 20 add_watcher() i=7 2024-03-09T14:31:01.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.209+0000 7fab879f9980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:01.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.209+0000 7faa3b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:01.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:01.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:01.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 5 note: GC not initialized 2024-03-09T14:31:01.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7faa3a7e4640 20 reqs_thread_entry: start 2024-03-09T14:31:01.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:01.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 Filter name: none 2024-03-09T14:31:01.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fa9abfff640 20 reqs_thread_entry: start 2024-03-09T14:31:01.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:01.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:01.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:01.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2024-03-09T14:31:01.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 INFO: main: starting shard=0 2024-03-09T14:31:01.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:01.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2024-03-09T14:31:01.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7fab879f9980 20 INFO: main: done 2024-03-09T14:31:01.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.213+0000 7faa3b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:01.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.221+0000 7fab879f9980 20 remove_watcher() i=0 2024-03-09T14:31:01.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.221+0000 7fab879f9980 2 removed watcher, disabling cache 2024-03-09T14:31:01.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.221+0000 7fab879f9980 20 remove_watcher() i=1 2024-03-09T14:31:01.225 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.225+0000 7fab879f9980 20 remove_watcher() i=2 2024-03-09T14:31:01.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.225+0000 7fab879f9980 20 remove_watcher() i=3 2024-03-09T14:31:01.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.229+0000 7fab879f9980 20 remove_watcher() i=4 2024-03-09T14:31:01.230 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.229+0000 7fab879f9980 20 remove_watcher() i=5 2024-03-09T14:31:01.232 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.229+0000 7fab879f9980 20 remove_watcher() i=6 2024-03-09T14:31:01.233 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.233+0000 7fab879f9980 20 remove_watcher() i=7 2024-03-09T14:31:01.246 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:01.246 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:01.273 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:01.274 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:01.401 INFO:tasks.workunit.client.0.smithi001.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2024-03-09T14:31:01.402 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket check-bucket --object f 2024-03-09T14:31:01.438 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:01.438 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:01.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.481+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.481+0000 7f601295c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.481+0000 7f601295c980 20 realm 2024-03-09T14:31:01.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.481+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.481+0000 7f601295c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.481+0000 7f601295c980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:01.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.481+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.481+0000 7f601295c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.481+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.485+0000 7f601295c980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:01.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.485+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.485+0000 7f601295c980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:01.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.485+0000 7f601295c980 20 searching for the correct realm 2024-03-09T14:31:01.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.493+0000 7f6010c75640 0 -- 172.21.15.1:0/3958274235 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f6004008040 msgr2=0x7f6004052910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:01.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.709+0000 7f6010474640 0 -- 172.21.15.1:0/3958274235 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x5574e06802f0 msgr2=0x5574e06827b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:01.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.717+0000 7f601295c980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:01.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.717+0000 7f601295c980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:01.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.717+0000 7f601295c980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:01.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.717+0000 7f601295c980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:01.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.717+0000 7f601295c980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:01.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.717+0000 7f601295c980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:01.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.717+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.717+0000 7f601295c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.717+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:01.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:01.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 zone default found 2024-03-09T14:31:01.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 4 Realm: () 2024-03-09T14:31:01.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:01.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:01.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:01.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 zonegroup default 2024-03-09T14:31:01.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:01.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 started sync module instance, tier type = 2024-03-09T14:31:01.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.721+0000 7f601295c980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:01.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.749+0000 7f601295c980 20 add_watcher() i=0 2024-03-09T14:31:01.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.749+0000 7f601295c980 20 add_watcher() i=1 2024-03-09T14:31:01.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.749+0000 7f601295c980 20 add_watcher() i=2 2024-03-09T14:31:01.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.749+0000 7f601295c980 20 add_watcher() i=3 2024-03-09T14:31:01.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.749+0000 7f601295c980 20 add_watcher() i=4 2024-03-09T14:31:01.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.749+0000 7f601295c980 20 add_watcher() i=5 2024-03-09T14:31:01.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.749+0000 7f601295c980 20 add_watcher() i=6 2024-03-09T14:31:01.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.749+0000 7f601295c980 20 add_watcher() i=7 2024-03-09T14:31:01.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.749+0000 7f601295c980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:01.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.757+0000 7f5ec67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:01.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.757+0000 7f601295c980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:01.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.757+0000 7f601295c980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:01.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.757+0000 7f601295c980 5 note: GC not initialized 2024-03-09T14:31:01.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.757+0000 7f5ec57e2640 20 reqs_thread_entry: start 2024-03-09T14:31:01.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.757+0000 7f601295c980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:01.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.757+0000 7f601295c980 20 Filter name: none 2024-03-09T14:31:01.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.757+0000 7f5e36ffd640 20 reqs_thread_entry: start 2024-03-09T14:31:01.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.757+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.757+0000 7f601295c980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:01.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.757+0000 7f601295c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:01.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.761+0000 7f601295c980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:01.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.761+0000 7f601295c980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:01.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.761+0000 7f601295c980 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2024-03-09T14:31:01.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.761+0000 7f601295c980 20 INFO: main: starting shard=0 2024-03-09T14:31:01.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.761+0000 7f601295c980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:01.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.761+0000 7f601295c980 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2024-03-09T14:31:01.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.761+0000 7f601295c980 20 INFO: main: done 2024-03-09T14:31:01.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.761+0000 7f5ec67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:01.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.769+0000 7f601295c980 20 remove_watcher() i=0 2024-03-09T14:31:01.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.769+0000 7f601295c980 2 removed watcher, disabling cache 2024-03-09T14:31:01.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.769+0000 7f601295c980 20 remove_watcher() i=1 2024-03-09T14:31:01.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.773+0000 7f601295c980 20 remove_watcher() i=2 2024-03-09T14:31:01.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.777+0000 7f601295c980 20 remove_watcher() i=3 2024-03-09T14:31:01.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.777+0000 7f601295c980 20 remove_watcher() i=4 2024-03-09T14:31:01.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.781+0000 7f601295c980 20 remove_watcher() i=5 2024-03-09T14:31:01.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.781+0000 7f601295c980 20 remove_watcher() i=6 2024-03-09T14:31:01.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:01.785+0000 7f601295c980 20 remove_watcher() i=7 2024-03-09T14:31:01.802 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:01.802 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:01.838 INFO:tasks.workunit.client.0.smithi001.stderr:ceph config rm client rgw_debug_inject_set_olh_err 2024-03-09T14:31:02.270 INFO:tasks.workunit.client.0.smithi001.stderr:ceph config rm client rgw_debug_inject_olh_cancel_modification_err 2024-03-09T14:31:02.695 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: bucket check unlinked finds unlistable entries 2024-03-09T14:31:02.695 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:31:02.696 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --min-age-hours 0 --dump-keys 2024-03-09T14:31:02.733 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:02.733 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:02.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.773+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:02.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.777+0000 7fbc7bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:02.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.777+0000 7fbc7bbb3980 20 realm 2024-03-09T14:31:02.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.777+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:02.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.777+0000 7fbc7bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:02.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.777+0000 7fbc7bbb3980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:02.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.777+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:02.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.777+0000 7fbc7bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:02.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.777+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:02.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.777+0000 7fbc7bbb3980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:02.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.777+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:02.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.777+0000 7fbc7bbb3980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:02.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.777+0000 7fbc7bbb3980 20 searching for the correct realm 2024-03-09T14:31:02.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:02.785+0000 7fbc79ecc640 0 -- 172.21.15.1:0/715414189 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7fbc70049950 msgr2=0x7fbc70018410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:03.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.001+0000 7fbc796cb640 0 -- 172.21.15.1:0/715414189 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x55d46befd330 msgr2=0x55d46beff760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:03.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.009+0000 7fbc7bbb3980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:03.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.009+0000 7fbc7bbb3980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:03.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.009+0000 7fbc7bbb3980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:03.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.009+0000 7fbc7bbb3980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:03.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.009+0000 7fbc7bbb3980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:03.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.009+0000 7fbc7bbb3980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:03.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.009+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.009+0000 7fbc7bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:03.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.009+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:03.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:03.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 zone default found 2024-03-09T14:31:03.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 4 Realm: () 2024-03-09T14:31:03.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:03.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:03.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:03.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 zonegroup default 2024-03-09T14:31:03.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:03.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:03.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:03.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 started sync module instance, tier type = 2024-03-09T14:31:03.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.013+0000 7fbc7bbb3980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:03.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.041+0000 7fbc7bbb3980 20 add_watcher() i=0 2024-03-09T14:31:03.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.041+0000 7fbc7bbb3980 20 add_watcher() i=1 2024-03-09T14:31:03.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.041+0000 7fbc7bbb3980 20 add_watcher() i=2 2024-03-09T14:31:03.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.041+0000 7fbc7bbb3980 20 add_watcher() i=3 2024-03-09T14:31:03.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.041+0000 7fbc7bbb3980 20 add_watcher() i=4 2024-03-09T14:31:03.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.041+0000 7fbc7bbb3980 20 add_watcher() i=5 2024-03-09T14:31:03.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.041+0000 7fbc7bbb3980 20 add_watcher() i=6 2024-03-09T14:31:03.044 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.045+0000 7fbc7bbb3980 20 add_watcher() i=7 2024-03-09T14:31:03.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.045+0000 7fbc7bbb3980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:03.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbb2ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:03.050 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbc7bbb3980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:03.050 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbc7bbb3980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:03.050 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbc7bbb3980 5 note: GC not initialized 2024-03-09T14:31:03.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbb2efe5640 20 reqs_thread_entry: start 2024-03-09T14:31:03.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbc7bbb3980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:03.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbc7bbb3980 20 Filter name: none 2024-03-09T14:31:03.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbb2cfe1640 20 reqs_thread_entry: start 2024-03-09T14:31:03.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbc7bbb3980 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2024-03-09T14:31:03.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbc7bbb3980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:03.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbc7bbb3980 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2024-03-09T14:31:03.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbc7bbb3980 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2024-03-09T14:31:03.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbc7bbb3980 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 : miss 2024-03-09T14:31:03.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.049+0000 7fbc7bbb3980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.053+0000 7fbc7bbb3980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:03.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.053+0000 7fbc7bbb3980 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 info.flags=0x17 2024-03-09T14:31:03.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.053+0000 7fbc7bbb3980 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 to cache LRU end 2024-03-09T14:31:03.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.053+0000 7fbc7bbb3980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:31:03.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.053+0000 7fbc7bbb3980 10 updating xattr: name=user.rgw.acl bl.length()=179 2024-03-09T14:31:03.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.053+0000 7fbc7bbb3980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 2024-03-09T14:31:03.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.053+0000 7fbc7bbb3980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.057+0000 7fbc7bbb3980 1 NOTICE: finished shard 0 (4 entries found) 2024-03-09T14:31:03.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.057+0000 7fbc7bbb3980 1 NOTICE: finished all shards (4 entries found) 2024-03-09T14:31:03.061 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.061+0000 7fbb2ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:03.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.065+0000 7fbc7bbb3980 20 remove_watcher() i=0 2024-03-09T14:31:03.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.065+0000 7fbc7bbb3980 2 removed watcher, disabling cache 2024-03-09T14:31:03.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.069+0000 7fbc7bbb3980 20 remove_watcher() i=1 2024-03-09T14:31:03.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.069+0000 7fbc7bbb3980 20 remove_watcher() i=2 2024-03-09T14:31:03.074 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.073+0000 7fbc7bbb3980 20 remove_watcher() i=3 2024-03-09T14:31:03.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.077+0000 7fbc7bbb3980 20 remove_watcher() i=4 2024-03-09T14:31:03.078 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.077+0000 7fbc7bbb3980 20 remove_watcher() i=5 2024-03-09T14:31:03.080 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.077+0000 7fbc7bbb3980 20 remove_watcher() i=6 2024-03-09T14:31:03.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.081+0000 7fbc7bbb3980 20 remove_watcher() i=7 2024-03-09T14:31:03.099 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: unlinked entries are not listable 2024-03-09T14:31:03.099 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:31:03.102 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:03.102 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:03.293 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: GET returns 404 for unlinked entry keys that have no other versions 2024-03-09T14:31:03.293 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:31:03.296 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:03.296 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:03.406 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:03.406 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:03.505 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: bucket check unlinked fixes unlistable entries 2024-03-09T14:31:03.505 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:31:03.506 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --fix --min-age-hours 0 --rgw-olh-pending-timeout-sec 0 --dump-keys 2024-03-09T14:31:03.546 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:03.546 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:03.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.593+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.593+0000 7f6c80d12980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:03.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.593+0000 7f6c80d12980 20 realm 2024-03-09T14:31:03.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.593+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.593+0000 7f6c80d12980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:03.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.593+0000 7f6c80d12980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:03.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.593+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.593+0000 7f6c80d12980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:03.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.593+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.597+0000 7f6c80d12980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:03.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.597+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.597+0000 7f6c80d12980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:03.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.597+0000 7f6c80d12980 20 searching for the correct realm 2024-03-09T14:31:03.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.601+0000 7f6c7f02b640 0 -- 172.21.15.1:0/3497794624 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f6c6c008040 msgr2=0x7f6c6c075040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:03.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.609+0000 7f6c7e82a640 0 -- 172.21.15.1:0/3497794624 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f6c6c087040 msgr2=0x7f6c6c0718a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:03.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.825+0000 7f6c80d12980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:03.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.825+0000 7f6c80d12980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:03.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.825+0000 7f6c80d12980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:03.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.825+0000 7f6c80d12980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:03.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.825+0000 7f6c80d12980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:03.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.825+0000 7f6c80d12980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:03.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.825+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.825+0000 7f6c80d12980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:03.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.825+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:03.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:03.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 zone default found 2024-03-09T14:31:03.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 4 Realm: () 2024-03-09T14:31:03.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:03.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:03.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:03.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 zonegroup default 2024-03-09T14:31:03.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:03.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:03.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:03.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 started sync module instance, tier type = 2024-03-09T14:31:03.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.829+0000 7f6c80d12980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:03.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.853+0000 7f6c80d12980 20 add_watcher() i=0 2024-03-09T14:31:03.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.853+0000 7f6c80d12980 20 add_watcher() i=1 2024-03-09T14:31:03.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.853+0000 7f6c80d12980 20 add_watcher() i=2 2024-03-09T14:31:03.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.853+0000 7f6c80d12980 20 add_watcher() i=3 2024-03-09T14:31:03.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.853+0000 7f6c80d12980 20 add_watcher() i=4 2024-03-09T14:31:03.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.853+0000 7f6c80d12980 20 add_watcher() i=5 2024-03-09T14:31:03.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.853+0000 7f6c80d12980 20 add_watcher() i=6 2024-03-09T14:31:03.858 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.857+0000 7f6c80d12980 20 add_watcher() i=7 2024-03-09T14:31:03.858 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.857+0000 7f6c80d12980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:03.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6b357e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:03.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:03.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:03.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 5 note: GC not initialized 2024-03-09T14:31:03.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6b2f7fe640 20 reqs_thread_entry: start 2024-03-09T14:31:03.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:03.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 20 Filter name: none 2024-03-09T14:31:03.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6b2d7fa640 20 reqs_thread_entry: start 2024-03-09T14:31:03.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2024-03-09T14:31:03.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:03.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2024-03-09T14:31:03.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2024-03-09T14:31:03.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 : miss 2024-03-09T14:31:03.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 20 rados->read ofs=0 len=0 2024-03-09T14:31:03.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:03.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 info.flags=0x17 2024-03-09T14:31:03.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 to cache LRU end 2024-03-09T14:31:03.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:31:03.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 10 updating xattr: name=user.rgw.acl bl.length()=179 2024-03-09T14:31:03.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.861+0000 7f6c80d12980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 2024-03-09T14:31:03.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.865+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.867 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.865+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.867 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.865+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:c state=0x557caaffd318 s->prefetch_data=0 2024-03-09T14:31:03.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.865+0000 7f6c80d12980 20 get_obj_state: setting s->obj_tag to jqzeqelhtio537zp0lj948lfyxg303vw 2024-03-09T14:31:03.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.865+0000 7f6c80d12980 20 get_obj_state_impl: setting s->olh_tag to zgzi4ul8jd6y0hyr9w1jef4psmfhynde 2024-03-09T14:31:03.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.865+0000 7f6c80d12980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=zgzi4ul8jd6y0hyr9w1jef4psmfhynde 2024-03-09T14:31:03.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.869+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.873+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.873+0000 7f6c80d12980 20 olh_log_entry: epoch=3 op=3 key=c[GOkh3csWwMb7XmpZFv9zztpFIccdevl] 2024-03-09T14:31:03.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.873+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:_:GOkh3csWwMb7XmpZFv9zztpFIccdevl_c state=0x557caaff69e8 s->prefetch_data=0 2024-03-09T14:31:03.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.873+0000 7f6c80d12980 10 manifest: total_size = 22 2024-03-09T14:31:03.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.873+0000 7f6c80d12980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9409853794963343504 2024-03-09T14:31:03.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.873+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:_:GOkh3csWwMb7XmpZFv9zztpFIccdevl_c state=0x557caaff69e8 s->prefetch_data=0 2024-03-09T14:31:03.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.873+0000 7f6c80d12980 20 prepare_atomic_modification: state is not atomic. state=0x557caaff69e8 2024-03-09T14:31:03.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.873+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.881+0000 7f6c80d12980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:31:03.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.881+0000 7f6c80d12980 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2024-03-09T14:31:03.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.881+0000 7f6c80d12980 0 deleting objects inline since gc isn't initialized 2024-03-09T14:31:03.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.881+0000 7f6c80d12980 5 delete_objs_inline: removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1__shadow:GOkh3csWwMb7XmpZFv9zztpFIccdevl_.yZmmnmucKbOxuYHWr1rHuhHSqPazjzB_0 2024-03-09T14:31:03.884 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.881+0000 7f6c80d12980 5 delete_objs_inline: refcount put returned error -2 2024-03-09T14:31:03.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.885+0000 7f6b61ffb640 20 handle_completion(): completion ok for obj=c[GOkh3csWwMb7XmpZFv9zztpFIccdevl] 2024-03-09T14:31:03.886 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.885+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.886 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.885+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.889+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.889+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.889+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:d state=0x557caaff7868 s->prefetch_data=0 2024-03-09T14:31:03.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.893+0000 7f6c80d12980 20 get_obj_state: setting s->obj_tag to okii0w19xmpe6919ur213uvzrxt27wnh 2024-03-09T14:31:03.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.893+0000 7f6c80d12980 20 get_obj_state_impl: setting s->olh_tag to 673118glelcl7guosmd38uua1369ocxb 2024-03-09T14:31:03.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.893+0000 7f6c80d12980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=673118glelcl7guosmd38uua1369ocxb 2024-03-09T14:31:03.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.893+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.897+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.901+0000 7f6c80d12980 20 olh_log_entry: epoch=3 op=3 key=d[QVMjWUKeZk5.hFwfIp7EJL7EIWb-4Oj] 2024-03-09T14:31:03.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.901+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:_:QVMjWUKeZk5.hFwfIp7EJL7EIWb-4Oj_d state=0x557caaff1998 s->prefetch_data=0 2024-03-09T14:31:03.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.901+0000 7f6c80d12980 10 manifest: total_size = 22 2024-03-09T14:31:03.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.901+0000 7f6c80d12980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2097840931093174468 2024-03-09T14:31:03.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.901+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:_:QVMjWUKeZk5.hFwfIp7EJL7EIWb-4Oj_d state=0x557caaff1998 s->prefetch_data=0 2024-03-09T14:31:03.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.901+0000 7f6c80d12980 20 prepare_atomic_modification: state is not atomic. state=0x557caaff1998 2024-03-09T14:31:03.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.901+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.905+0000 7f6c80d12980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:31:03.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.905+0000 7f6c80d12980 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2024-03-09T14:31:03.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.905+0000 7f6c80d12980 0 deleting objects inline since gc isn't initialized 2024-03-09T14:31:03.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.905+0000 7f6c80d12980 5 delete_objs_inline: removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1__shadow:QVMjWUKeZk5.hFwfIp7EJL7EIWb-4Oj_.2k6_dZZMorM91TAHT5jnKHsTSJNiX01_0 2024-03-09T14:31:03.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.909+0000 7f6c80d12980 5 delete_objs_inline: refcount put returned error -2 2024-03-09T14:31:03.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.909+0000 7f6b617fa640 20 handle_completion(): completion ok for obj=d[QVMjWUKeZk5.hFwfIp7EJL7EIWb-4Oj] 2024-03-09T14:31:03.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.913+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.913+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.913+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.917 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.917+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.917+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:e state=0x557caaff7868 s->prefetch_data=0 2024-03-09T14:31:03.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.917+0000 7f6c80d12980 20 get_obj_state: setting s->obj_tag to 7wbmoch0htgnov80o7uuoa8dzs9o8hz0 2024-03-09T14:31:03.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.917+0000 7f6c80d12980 20 get_obj_state_impl: setting s->olh_tag to dcnsv0n21li6jcdjwd4e3jgy5ize2ruo 2024-03-09T14:31:03.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.917+0000 7f6c80d12980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=dcnsv0n21li6jcdjwd4e3jgy5ize2ruo 2024-03-09T14:31:03.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.921+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.925+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.925+0000 7f6c80d12980 20 olh_log_entry: epoch=2 op=2 key=e[] 2024-03-09T14:31:03.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.925+0000 7f6c80d12980 20 olh_log_entry: epoch=2 op=3 key=e[SJjiahGZhNM-SXIChLPsQnGBqnWmZAz] 2024-03-09T14:31:03.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.925+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:_:SJjiahGZhNM-SXIChLPsQnGBqnWmZAz_e state=0x557caaff29a8 s->prefetch_data=0 2024-03-09T14:31:03.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.929+0000 7f6c80d12980 10 manifest: total_size = 9 2024-03-09T14:31:03.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.929+0000 7f6c80d12980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11883556171727333401 2024-03-09T14:31:03.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.929+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:_:SJjiahGZhNM-SXIChLPsQnGBqnWmZAz_e state=0x557caaff29a8 s->prefetch_data=0 2024-03-09T14:31:03.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.929+0000 7f6c80d12980 20 prepare_atomic_modification: state is not atomic. state=0x557caaff29a8 2024-03-09T14:31:03.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.929+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.937+0000 7f6c80d12980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:31:03.937 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.937+0000 7f6c80d12980 20 RGWObjManifest::operator++(): result: ofs=9 stripe_ofs=9 part_ofs=0 rule->part_size=0 2024-03-09T14:31:03.937 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.937+0000 7f6c80d12980 0 deleting objects inline since gc isn't initialized 2024-03-09T14:31:03.937 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.937+0000 7f6c80d12980 5 delete_objs_inline: removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1__shadow:SJjiahGZhNM-SXIChLPsQnGBqnWmZAz_.0vMNOjRPKR4WYQs6kGkpRmSxkBVxIO5_0 2024-03-09T14:31:03.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.937+0000 7f6b61ffb640 20 handle_completion(): completion ok for obj=e[SJjiahGZhNM-SXIChLPsQnGBqnWmZAz] 2024-03-09T14:31:03.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.937+0000 7f6c80d12980 5 delete_objs_inline: refcount put returned error -2 2024-03-09T14:31:03.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.941+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:e state=0x557caaff7868 s->prefetch_data=0 2024-03-09T14:31:03.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.941+0000 7f6c80d12980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=dcnsv0n21li6jcdjwd4e3jgy5ize2ruo 2024-03-09T14:31:03.950 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.949+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.953+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.953+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.953+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:f state=0x557cab02d168 s->prefetch_data=0 2024-03-09T14:31:03.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.957+0000 7f6c80d12980 20 get_obj_state: setting s->obj_tag to tj8meofkobykhg4lc2uwonbhg47vkgtb 2024-03-09T14:31:03.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.957+0000 7f6c80d12980 20 get_obj_state_impl: setting s->olh_tag to a7ea3uqunuxfop8dv0aeovnm9f9kuo08 2024-03-09T14:31:03.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.957+0000 7f6c80d12980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=a7ea3uqunuxfop8dv0aeovnm9f9kuo08 2024-03-09T14:31:03.960 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.957+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.961+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.965+0000 7f6c80d12980 20 olh_log_entry: epoch=2 op=2 key=f[] 2024-03-09T14:31:03.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.965+0000 7f6c80d12980 20 olh_log_entry: epoch=2 op=3 key=f[0TfHjfBtsEarC5RdKOItUDb9L94Q6IA] 2024-03-09T14:31:03.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.965+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:_:0TfHjfBtsEarC5RdKOItUDb9L94Q6IA_f state=0x557caaff7868 s->prefetch_data=0 2024-03-09T14:31:03.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.965+0000 7f6c80d12980 10 manifest: total_size = 22 2024-03-09T14:31:03.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.965+0000 7f6c80d12980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.8983138823444694503 2024-03-09T14:31:03.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.965+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:_:0TfHjfBtsEarC5RdKOItUDb9L94Q6IA_f state=0x557caaff7868 s->prefetch_data=0 2024-03-09T14:31:03.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.965+0000 7f6c80d12980 20 prepare_atomic_modification: state is not atomic. state=0x557caaff7868 2024-03-09T14:31:03.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.965+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.973+0000 7f6c80d12980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:31:03.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.973+0000 7f6c80d12980 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2024-03-09T14:31:03.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.973+0000 7f6c80d12980 0 deleting objects inline since gc isn't initialized 2024-03-09T14:31:03.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.973+0000 7f6c80d12980 5 delete_objs_inline: removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1__shadow:0TfHjfBtsEarC5RdKOItUDb9L94Q6IA_.t3GSPoDBS_ejl5QakgB_Jg_KDJs63mG_0 2024-03-09T14:31:03.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.973+0000 7f6c80d12980 5 delete_objs_inline: refcount put returned error -2 2024-03-09T14:31:03.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.977+0000 7f6b617fa640 20 handle_completion(): completion ok for obj=f[0TfHjfBtsEarC5RdKOItUDb9L94Q6IA] 2024-03-09T14:31:03.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.977+0000 7f6c80d12980 20 get_obj_state: octx=0x557caad4ae30 obj=check-bucket:f state=0x557cab02d168 s->prefetch_data=0 2024-03-09T14:31:03.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.977+0000 7f6c80d12980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=a7ea3uqunuxfop8dv0aeovnm9f9kuo08 2024-03-09T14:31:03.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.985+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.989+0000 7f6c80d12980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:03.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.989+0000 7f6c80d12980 1 NOTICE: finished shard 0 (4 entries removed) 2024-03-09T14:31:03.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.989+0000 7f6c80d12980 1 NOTICE: finished all shards (4 entries removed) 2024-03-09T14:31:03.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.989+0000 7f6b357e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:03.997 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.997+0000 7f6c80d12980 20 remove_watcher() i=0 2024-03-09T14:31:03.997 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.997+0000 7f6c80d12980 2 removed watcher, disabling cache 2024-03-09T14:31:03.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:03.997+0000 7f6c80d12980 20 remove_watcher() i=1 2024-03-09T14:31:04.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.001+0000 7f6c80d12980 20 remove_watcher() i=2 2024-03-09T14:31:04.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.005+0000 7f6c80d12980 20 remove_watcher() i=3 2024-03-09T14:31:04.007 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.005+0000 7f6c80d12980 20 remove_watcher() i=4 2024-03-09T14:31:04.009 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.009+0000 7f6c80d12980 20 remove_watcher() i=5 2024-03-09T14:31:04.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.009+0000 7f6c80d12980 20 remove_watcher() i=6 2024-03-09T14:31:04.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.013+0000 7f6c80d12980 20 remove_watcher() i=7 2024-03-09T14:31:04.036 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:04.036 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:04.076 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:04.076 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:04.118 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:04.118 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:04.165 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:04.165 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:04.224 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: bucket check unlinked does not affect normal entries 2024-03-09T14:31:04.224 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:31:04.228 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:04.228 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:04.352 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:04.352 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:04.396 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:04.396 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:04.467 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:04.467 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:04.541 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:04.541 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:04.612 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: bucket check unlinked does not find new unlistable entries after fix 2024-03-09T14:31:04.612 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:31:04.612 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --min-age-hours 0 --dump-keys 2024-03-09T14:31:04.652 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:04.652 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:04.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.697+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.697+0000 7ff9a8cbe980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:04.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.697+0000 7ff9a8cbe980 20 realm 2024-03-09T14:31:04.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.697+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.697+0000 7ff9a8cbe980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:04.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.697+0000 7ff9a8cbe980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:04.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.697+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.701+0000 7ff9a8cbe980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:04.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.701+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.702 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.701+0000 7ff9a8cbe980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:04.702 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.701+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.703 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.701+0000 7ff9a8cbe980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:04.703 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.701+0000 7ff9a8cbe980 20 searching for the correct realm 2024-03-09T14:31:04.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.709+0000 7ff9a6fd7640 0 -- 172.21.15.1:0/3985025649 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7ff99c117b00 msgr2=0x7ff99c0ef160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:04.715 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.713+0000 7ff9a67d6640 0 -- 172.21.15.1:0/3985025649 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7ff99c0b9910 msgr2=0x7ff99c10e0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:04.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.929+0000 7ff9a8cbe980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:04.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.929+0000 7ff9a8cbe980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:04.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.929+0000 7ff9a8cbe980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:04.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.929+0000 7ff9a8cbe980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:04.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.929+0000 7ff9a8cbe980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:04.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.929+0000 7ff9a8cbe980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:04.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.929+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.929+0000 7ff9a8cbe980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:04.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.929+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.929+0000 7ff9a8cbe980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:04.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.929+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:04.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 20 zone default found 2024-03-09T14:31:04.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 4 Realm: () 2024-03-09T14:31:04.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:04.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:04.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:04.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 20 zonegroup default 2024-03-09T14:31:04.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.934 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:04.934 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.934 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:04.934 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.935 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:04.935 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 20 started sync module instance, tier type = 2024-03-09T14:31:04.935 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.933+0000 7ff9a8cbe980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:04.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.961+0000 7ff9a8cbe980 20 add_watcher() i=0 2024-03-09T14:31:04.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.961+0000 7ff9a8cbe980 20 add_watcher() i=1 2024-03-09T14:31:04.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.961+0000 7ff9a8cbe980 20 add_watcher() i=2 2024-03-09T14:31:04.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.961+0000 7ff9a8cbe980 20 add_watcher() i=3 2024-03-09T14:31:04.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.961+0000 7ff9a8cbe980 20 add_watcher() i=4 2024-03-09T14:31:04.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.961+0000 7ff9a8cbe980 20 add_watcher() i=5 2024-03-09T14:31:04.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.961+0000 7ff9a8cbe980 20 add_watcher() i=6 2024-03-09T14:31:04.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.961+0000 7ff9a8cbe980 20 add_watcher() i=7 2024-03-09T14:31:04.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.961+0000 7ff9a8cbe980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:04.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.965+0000 7ff85cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:04.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:04.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:04.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 5 note: GC not initialized 2024-03-09T14:31:04.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff7d77fe640 20 reqs_thread_entry: start 2024-03-09T14:31:04.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:04.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 20 Filter name: none 2024-03-09T14:31:04.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff7d57fa640 20 reqs_thread_entry: start 2024-03-09T14:31:04.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2024-03-09T14:31:04.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:04.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2024-03-09T14:31:04.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2024-03-09T14:31:04.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 : miss 2024-03-09T14:31:04.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 20 rados->read ofs=0 len=0 2024-03-09T14:31:04.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:04.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 info.flags=0x17 2024-03-09T14:31:04.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 to cache LRU end 2024-03-09T14:31:04.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:31:04.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 10 updating xattr: name=user.rgw.acl bl.length()=179 2024-03-09T14:31:04.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 2024-03-09T14:31:04.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.969+0000 7ff9a8cbe980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:04.974 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.973+0000 7ff9a8cbe980 1 NOTICE: finished shard 0 (0 entries found) 2024-03-09T14:31:04.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.973+0000 7ff9a8cbe980 1 NOTICE: finished all shards (0 entries found) 2024-03-09T14:31:04.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.973+0000 7ff85cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:04.981 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.981+0000 7ff9a8cbe980 20 remove_watcher() i=0 2024-03-09T14:31:04.981 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.981+0000 7ff9a8cbe980 2 removed watcher, disabling cache 2024-03-09T14:31:04.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.981+0000 7ff9a8cbe980 20 remove_watcher() i=1 2024-03-09T14:31:04.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.985+0000 7ff9a8cbe980 20 remove_watcher() i=2 2024-03-09T14:31:04.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.989+0000 7ff9a8cbe980 20 remove_watcher() i=3 2024-03-09T14:31:04.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.989+0000 7ff9a8cbe980 20 remove_watcher() i=4 2024-03-09T14:31:04.994 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.993+0000 7ff9a8cbe980 20 remove_watcher() i=5 2024-03-09T14:31:04.996 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.993+0000 7ff9a8cbe980 20 remove_watcher() i=6 2024-03-09T14:31:04.997 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:04.997+0000 7ff9a8cbe980 20 remove_watcher() i=7 2024-03-09T14:31:05.019 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:05.019 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:05.323 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:05.324 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:05.461 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket check-bucket --object c 2024-03-09T14:31:05.496 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:05.496 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:05.541 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.541+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.541+0000 7ff97b386980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:05.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.541+0000 7ff97b386980 20 realm 2024-03-09T14:31:05.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.541+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.541+0000 7ff97b386980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:05.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.541+0000 7ff97b386980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:05.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.541+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.541+0000 7ff97b386980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:05.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.541+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.545+0000 7ff97b386980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:05.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.545+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.545+0000 7ff97b386980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:05.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.545+0000 7ff97b386980 20 searching for the correct realm 2024-03-09T14:31:05.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.553+0000 7ff97969f640 0 -- 172.21.15.1:0/200028487 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7ff96c021040 msgr2=0x7ff96c021930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:05.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.557+0000 7ff978e9e640 0 -- 172.21.15.1:0/200028487 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7ff96c008040 msgr2=0x7ff96c044960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:05.775 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.773+0000 7ff97b386980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:05.775 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.773+0000 7ff97b386980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:05.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.773+0000 7ff97b386980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:05.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.773+0000 7ff97b386980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:05.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.773+0000 7ff97b386980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:05.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.773+0000 7ff97b386980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:05.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.773+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.773+0000 7ff97b386980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:05.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.773+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:05.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:05.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 zone default found 2024-03-09T14:31:05.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 4 Realm: () 2024-03-09T14:31:05.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:05.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:05.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:05.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 zonegroup default 2024-03-09T14:31:05.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:05.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:05.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:05.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 started sync module instance, tier type = 2024-03-09T14:31:05.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.777+0000 7ff97b386980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:05.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.805+0000 7ff97b386980 20 add_watcher() i=0 2024-03-09T14:31:05.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.805+0000 7ff97b386980 20 add_watcher() i=1 2024-03-09T14:31:05.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.805+0000 7ff97b386980 20 add_watcher() i=2 2024-03-09T14:31:05.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.805+0000 7ff97b386980 20 add_watcher() i=3 2024-03-09T14:31:05.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.805+0000 7ff97b386980 20 add_watcher() i=4 2024-03-09T14:31:05.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.805+0000 7ff97b386980 20 add_watcher() i=5 2024-03-09T14:31:05.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.805+0000 7ff97b386980 20 add_watcher() i=6 2024-03-09T14:31:05.808 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.809+0000 7ff97b386980 20 add_watcher() i=7 2024-03-09T14:31:05.808 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.809+0000 7ff97b386980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:05.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.813+0000 7ff82efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:05.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.813+0000 7ff97b386980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:05.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.813+0000 7ff97b386980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:05.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.813+0000 7ff97b386980 5 note: GC not initialized 2024-03-09T14:31:05.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.813+0000 7ff82dfe3640 20 reqs_thread_entry: start 2024-03-09T14:31:05.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.813+0000 7ff97b386980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:05.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.813+0000 7ff97b386980 20 Filter name: none 2024-03-09T14:31:05.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.813+0000 7ff79f7fe640 20 reqs_thread_entry: start 2024-03-09T14:31:05.816 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.813+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.816 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.813+0000 7ff97b386980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:05.816 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.813+0000 7ff97b386980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.817+0000 7ff97b386980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:05.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.817+0000 7ff97b386980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:05.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.817+0000 7ff97b386980 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2024-03-09T14:31:05.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.817+0000 7ff97b386980 20 INFO: main: starting shard=0 2024-03-09T14:31:05.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.817+0000 7ff97b386980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:05.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.817+0000 7ff97b386980 20 INFO: main: bi_list() returned without error; entries.size()=2, is_truncated=0, marker= 2024-03-09T14:31:05.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.817+0000 7ff97b386980 20 INFO: main: done 2024-03-09T14:31:05.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.817+0000 7ff82efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:05.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.825+0000 7ff97b386980 20 remove_watcher() i=0 2024-03-09T14:31:05.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.825+0000 7ff97b386980 2 removed watcher, disabling cache 2024-03-09T14:31:05.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.825+0000 7ff97b386980 20 remove_watcher() i=1 2024-03-09T14:31:05.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.829+0000 7ff97b386980 20 remove_watcher() i=2 2024-03-09T14:31:05.832 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.833+0000 7ff97b386980 20 remove_watcher() i=3 2024-03-09T14:31:05.835 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.833+0000 7ff97b386980 20 remove_watcher() i=4 2024-03-09T14:31:05.836 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.837+0000 7ff97b386980 20 remove_watcher() i=5 2024-03-09T14:31:05.838 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.837+0000 7ff97b386980 20 remove_watcher() i=6 2024-03-09T14:31:05.840 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.837+0000 7ff97b386980 20 remove_watcher() i=7 2024-03-09T14:31:05.862 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket check-bucket --object d 2024-03-09T14:31:05.893 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:05.893 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:05.938 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.937+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.937+0000 7fe0ddcc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:05.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.937+0000 7fe0ddcc0980 20 realm 2024-03-09T14:31:05.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.937+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.941+0000 7fe0ddcc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:05.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.941+0000 7fe0ddcc0980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:05.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.941+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.941+0000 7fe0ddcc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:05.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.941+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.941+0000 7fe0ddcc0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:05.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.941+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:05.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.941+0000 7fe0ddcc0980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:05.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.941+0000 7fe0ddcc0980 20 searching for the correct realm 2024-03-09T14:31:05.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:05.945+0000 7fe0dafd7640 0 -- 172.21.15.1:0/1066519515 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fe0cc0b9070 msgr2=0x7fe0cc0f8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:06.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.161+0000 7fe0db7d8640 0 -- 172.21.15.1:0/1066519515 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fe0c4003100 msgr2=0x7fe0d003a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:06.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.377+0000 7fe0ddcc0980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:06.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.377+0000 7fe0ddcc0980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:06.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.377+0000 7fe0ddcc0980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:06.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.377+0000 7fe0ddcc0980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:06.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.377+0000 7fe0ddcc0980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:06.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.377+0000 7fe0ddcc0980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:06.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.377+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.381 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.377+0000 7fe0ddcc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.381 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.377+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.381 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:06.381 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:06.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 20 zone default found 2024-03-09T14:31:06.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 4 Realm: () 2024-03-09T14:31:06.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:06.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:06.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:06.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 20 zonegroup default 2024-03-09T14:31:06.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.381+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.385+0000 7fe0ddcc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.385+0000 7fe0ddcc0980 20 started sync module instance, tier type = 2024-03-09T14:31:06.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.385+0000 7fe0ddcc0980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:06.411 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.409+0000 7fe0ddcc0980 20 add_watcher() i=0 2024-03-09T14:31:06.411 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.409+0000 7fe0ddcc0980 20 add_watcher() i=1 2024-03-09T14:31:06.411 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.409+0000 7fe0ddcc0980 20 add_watcher() i=2 2024-03-09T14:31:06.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.409+0000 7fe0ddcc0980 20 add_watcher() i=3 2024-03-09T14:31:06.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.409+0000 7fe0ddcc0980 20 add_watcher() i=4 2024-03-09T14:31:06.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.409+0000 7fe0ddcc0980 20 add_watcher() i=5 2024-03-09T14:31:06.414 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.413+0000 7fe0ddcc0980 20 add_watcher() i=6 2024-03-09T14:31:06.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.417+0000 7fe0ddcc0980 20 add_watcher() i=7 2024-03-09T14:31:06.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.417+0000 7fe0ddcc0980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:06.424 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.421+0000 7fdf91fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:06.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:06.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:06.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 5 note: GC not initialized 2024-03-09T14:31:06.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fdf90fe1640 20 reqs_thread_entry: start 2024-03-09T14:31:06.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:06.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 20 Filter name: none 2024-03-09T14:31:06.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fdf0a7fc640 20 reqs_thread_entry: start 2024-03-09T14:31:06.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:06.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:06.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:06.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2024-03-09T14:31:06.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 20 INFO: main: starting shard=0 2024-03-09T14:31:06.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.425+0000 7fe0ddcc0980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:06.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.429+0000 7fe0ddcc0980 20 INFO: main: bi_list() returned without error; entries.size()=2, is_truncated=0, marker= 2024-03-09T14:31:06.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.429+0000 7fe0ddcc0980 20 INFO: main: done 2024-03-09T14:31:06.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.429+0000 7fdf91fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:06.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.433+0000 7fe0ddcc0980 20 remove_watcher() i=0 2024-03-09T14:31:06.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.433+0000 7fe0ddcc0980 2 removed watcher, disabling cache 2024-03-09T14:31:06.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.437+0000 7fe0ddcc0980 20 remove_watcher() i=1 2024-03-09T14:31:06.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.441+0000 7fe0ddcc0980 20 remove_watcher() i=2 2024-03-09T14:31:06.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.441+0000 7fe0ddcc0980 20 remove_watcher() i=3 2024-03-09T14:31:06.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.445+0000 7fe0ddcc0980 20 remove_watcher() i=4 2024-03-09T14:31:06.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.445+0000 7fe0ddcc0980 20 remove_watcher() i=5 2024-03-09T14:31:06.449 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.449+0000 7fe0ddcc0980 20 remove_watcher() i=6 2024-03-09T14:31:06.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.449+0000 7fe0ddcc0980 20 remove_watcher() i=7 2024-03-09T14:31:06.466 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: bucket check olh finds leftover OLH entries 2024-03-09T14:31:06.466 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:31:06.467 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket check olh --bucket check-bucket --dump-keys 2024-03-09T14:31:06.495 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:06.496 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:06.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.537+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.541+0000 7f02f7588980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.541+0000 7f02f7588980 20 realm 2024-03-09T14:31:06.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.541+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.541+0000 7f02f7588980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.541+0000 7f02f7588980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:06.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.541+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.541+0000 7f02f7588980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.541+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.541+0000 7f02f7588980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:06.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.541+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.545+0000 7f02f7588980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:06.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.545+0000 7f02f7588980 20 searching for the correct realm 2024-03-09T14:31:06.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.549+0000 7f02f58a1640 0 -- 172.21.15.1:0/3120488650 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x5591d77001e0 msgr2=0x5591d77025f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:06.557 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.557+0000 7f02f50a0640 0 -- 172.21.15.1:0/3120488650 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f02ec064660 msgr2=0x7f02ec066d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:06.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:06.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:06.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:06.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:06.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:06.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:06.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:06.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:06.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 zone default found 2024-03-09T14:31:06.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 4 Realm: () 2024-03-09T14:31:06.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:06.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:06.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:06.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 zonegroup default 2024-03-09T14:31:06.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.565+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.569+0000 7f02f7588980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.569+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.569+0000 7f02f7588980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.569+0000 7f02f7588980 20 started sync module instance, tier type = 2024-03-09T14:31:06.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.569+0000 7f02f7588980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:06.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.585+0000 7f02f7588980 20 add_watcher() i=0 2024-03-09T14:31:06.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.585+0000 7f02f7588980 20 add_watcher() i=1 2024-03-09T14:31:06.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.585+0000 7f02f7588980 20 add_watcher() i=2 2024-03-09T14:31:06.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.585+0000 7f02f7588980 20 add_watcher() i=3 2024-03-09T14:31:06.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.585+0000 7f02f7588980 20 add_watcher() i=4 2024-03-09T14:31:06.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.585+0000 7f02f7588980 20 add_watcher() i=5 2024-03-09T14:31:06.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.585+0000 7f02f7588980 20 add_watcher() i=6 2024-03-09T14:31:06.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.589+0000 7f02f7588980 20 add_watcher() i=7 2024-03-09T14:31:06.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.589+0000 7f02f7588980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:06.591 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.589+0000 7f01abfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:06.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.589+0000 7f02f7588980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:06.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.589+0000 7f02f7588980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:06.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.589+0000 7f02f7588980 5 note: GC not initialized 2024-03-09T14:31:06.593 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.589+0000 7f01aafe5640 20 reqs_thread_entry: start 2024-03-09T14:31:06.593 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.589+0000 7f02f7588980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:06.593 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.589+0000 7f02f7588980 20 Filter name: none 2024-03-09T14:31:06.593 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f01a8fe1640 20 reqs_thread_entry: start 2024-03-09T14:31:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2024-03-09T14:31:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2024-03-09T14:31:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2024-03-09T14:31:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 : miss 2024-03-09T14:31:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 info.flags=0x17 2024-03-09T14:31:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 to cache LRU end 2024-03-09T14:31:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:31:06.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 10 updating xattr: name=user.rgw.acl bl.length()=179 2024-03-09T14:31:06.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 2024-03-09T14:31:06.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:06.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 1 NOTICE: finished shard 0 (2 entries found) 2024-03-09T14:31:06.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f02f7588980 1 NOTICE: finished all shards (2 entries found) 2024-03-09T14:31:06.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.593+0000 7f01abfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:06.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.601+0000 7f02f7588980 20 remove_watcher() i=0 2024-03-09T14:31:06.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.601+0000 7f02f7588980 2 removed watcher, disabling cache 2024-03-09T14:31:06.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.601+0000 7f02f7588980 20 remove_watcher() i=1 2024-03-09T14:31:06.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.605+0000 7f02f7588980 20 remove_watcher() i=2 2024-03-09T14:31:06.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.605+0000 7f02f7588980 20 remove_watcher() i=3 2024-03-09T14:31:06.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.609+0000 7f02f7588980 20 remove_watcher() i=4 2024-03-09T14:31:06.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.609+0000 7f02f7588980 20 remove_watcher() i=5 2024-03-09T14:31:06.612 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.613+0000 7f02f7588980 20 remove_watcher() i=6 2024-03-09T14:31:06.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.613+0000 7f02f7588980 20 remove_watcher() i=7 2024-03-09T14:31:06.627 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: bucket check olh fixes leftover OLH entries 2024-03-09T14:31:06.627 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:31:06.628 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket check olh --bucket check-bucket --fix --rgw-olh-pending-timeout-sec 0 --dump-keys 2024-03-09T14:31:06.659 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:06.659 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:06.702 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.701+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.701+0000 7f6eb67b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.701+0000 7f6eb67b0980 20 realm 2024-03-09T14:31:06.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.701+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.701+0000 7f6eb67b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.701+0000 7f6eb67b0980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:06.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.701+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.701+0000 7f6eb67b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.701+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.705 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.705+0000 7f6eb67b0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:06.705 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.705+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.706 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.705+0000 7f6eb67b0980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:06.706 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.705+0000 7f6eb67b0980 20 searching for the correct realm 2024-03-09T14:31:06.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.709+0000 7f6eb3ac7640 0 -- 172.21.15.1:0/875275325 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x5562d72bb360 msgr2=0x5562d72bd7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:31:06.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.717+0000 7f6eb42c8640 0 -- 172.21.15.1:0/875275325 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f6ea803c410 msgr2=0x7f6ea803c890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:06.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.725+0000 7f6eb67b0980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:06.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.725+0000 7f6eb67b0980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:06.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.725+0000 7f6eb67b0980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:06.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.725+0000 7f6eb67b0980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:06.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.725+0000 7f6eb67b0980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:06.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.725+0000 7f6eb67b0980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:06.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.725+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.725+0000 7f6eb67b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.725+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.725+0000 7f6eb67b0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:06.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.725+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:06.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 20 zone default found 2024-03-09T14:31:06.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 4 Realm: () 2024-03-09T14:31:06.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:06.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:06.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:06.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 20 zonegroup default 2024-03-09T14:31:06.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 20 started sync module instance, tier type = 2024-03-09T14:31:06.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.729+0000 7f6eb67b0980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:06.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.745+0000 7f6eb67b0980 20 add_watcher() i=0 2024-03-09T14:31:06.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.745+0000 7f6eb67b0980 20 add_watcher() i=1 2024-03-09T14:31:06.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.745+0000 7f6eb67b0980 20 add_watcher() i=2 2024-03-09T14:31:06.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.745+0000 7f6eb67b0980 20 add_watcher() i=3 2024-03-09T14:31:06.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.745+0000 7f6eb67b0980 20 add_watcher() i=4 2024-03-09T14:31:06.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.745+0000 7f6eb67b0980 20 add_watcher() i=5 2024-03-09T14:31:06.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.745+0000 7f6eb67b0980 20 add_watcher() i=6 2024-03-09T14:31:06.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.745+0000 7f6eb67b0980 20 add_watcher() i=7 2024-03-09T14:31:06.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.745+0000 7f6eb67b0980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:06.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6d6a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:06.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:06.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:06.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 5 note: GC not initialized 2024-03-09T14:31:06.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6d697e2640 20 reqs_thread_entry: start 2024-03-09T14:31:06.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:06.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 20 Filter name: none 2024-03-09T14:31:06.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6cdaffd640 20 reqs_thread_entry: start 2024-03-09T14:31:06.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2024-03-09T14:31:06.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:06.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2024-03-09T14:31:06.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2024-03-09T14:31:06.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 : miss 2024-03-09T14:31:06.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:06.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 info.flags=0x17 2024-03-09T14:31:06.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 to cache LRU end 2024-03-09T14:31:06.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:31:06.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 10 updating xattr: name=user.rgw.acl bl.length()=179 2024-03-09T14:31:06.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 2024-03-09T14:31:06.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.749+0000 7f6eb67b0980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:06.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.753+0000 7f6eb67b0980 20 get_obj_state: octx=0x5562d704fe20 obj=check-bucket:c state=0x5562d72f6288 s->prefetch_data=0 2024-03-09T14:31:06.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.753+0000 7f6eb67b0980 20 get_obj_state: setting s->obj_tag to jqzeqelhtio537zp0lj948lfyxg303vw 2024-03-09T14:31:06.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.753+0000 7f6eb67b0980 20 get_obj_state_impl: setting s->olh_tag to zgzi4ul8jd6y0hyr9w1jef4psmfhynde 2024-03-09T14:31:06.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.753+0000 7f6eb67b0980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:06.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.753+0000 7f6eb67b0980 20 olh_log_entry: epoch=4 op=2 key=c[] 2024-03-09T14:31:06.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.753+0000 7f6eb67b0980 20 olh_log_entry: epoch=4 op=3 key=c[FN40PUMc0.in5IVaboyOr4wTEYagYC-] 2024-03-09T14:31:06.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.753+0000 7f6eb67b0980 20 get_obj_state: octx=0x5562d73232b0 obj=check-bucket:_:FN40PUMc0.in5IVaboyOr4wTEYagYC-_c state=0x5562d72f5428 s->prefetch_data=0 2024-03-09T14:31:06.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.757+0000 7f6eb67b0980 20 get_obj_state: octx=0x5562d73232b0 obj=check-bucket:c state=0x5562d72f78f8 s->prefetch_data=0 2024-03-09T14:31:06.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.757+0000 7f6eb67b0980 20 get_obj_state: setting s->obj_tag to jqzeqelhtio537zp0lj948lfyxg303vw 2024-03-09T14:31:06.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.757+0000 7f6eb67b0980 20 get_obj_state_impl: setting s->olh_tag to zgzi4ul8jd6y0hyr9w1jef4psmfhynde 2024-03-09T14:31:06.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.757+0000 7f6eb67b0980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=zgzi4ul8jd6y0hyr9w1jef4psmfhynde 2024-03-09T14:31:06.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.761+0000 7f6eb67b0980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:06.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.765+0000 7f6eb67b0980 20 get_obj_state: octx=0x5562d704fe20 obj=check-bucket:d state=0x5562d72f6288 s->prefetch_data=0 2024-03-09T14:31:06.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.765+0000 7f6eb67b0980 20 get_obj_state: setting s->obj_tag to okii0w19xmpe6919ur213uvzrxt27wnh 2024-03-09T14:31:06.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.765+0000 7f6eb67b0980 20 get_obj_state_impl: setting s->olh_tag to 673118glelcl7guosmd38uua1369ocxb 2024-03-09T14:31:06.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.765+0000 7f6eb67b0980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:06.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.765+0000 7f6eb67b0980 20 olh_log_entry: epoch=4 op=2 key=d[] 2024-03-09T14:31:06.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.765+0000 7f6eb67b0980 20 olh_log_entry: epoch=4 op=3 key=d[T3DM9BEsCn.lF4rnZh4FNR1.QHKHWLh] 2024-03-09T14:31:06.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.765+0000 7f6eb67b0980 20 get_obj_state: octx=0x5562d73232b0 obj=check-bucket:_:T3DM9BEsCn.lF4rnZh4FNR1.QHKHWLh_d state=0x5562d7324ab8 s->prefetch_data=0 2024-03-09T14:31:06.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.769+0000 7f6eb67b0980 20 get_obj_state: octx=0x5562d73232b0 obj=check-bucket:d state=0x5562d72f8458 s->prefetch_data=0 2024-03-09T14:31:06.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.769+0000 7f6eb67b0980 20 get_obj_state: setting s->obj_tag to okii0w19xmpe6919ur213uvzrxt27wnh 2024-03-09T14:31:06.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.769+0000 7f6eb67b0980 20 get_obj_state_impl: setting s->olh_tag to 673118glelcl7guosmd38uua1369ocxb 2024-03-09T14:31:06.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.769+0000 7f6eb67b0980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=673118glelcl7guosmd38uua1369ocxb 2024-03-09T14:31:06.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.769+0000 7f6eb67b0980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:06.775 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.773+0000 7f6eb67b0980 1 NOTICE: finished shard 0 (2 entries removed) 2024-03-09T14:31:06.775 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.773+0000 7f6eb67b0980 1 NOTICE: finished all shards (2 entries removed) 2024-03-09T14:31:06.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.773+0000 7f6d6a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:06.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.781+0000 7f6eb67b0980 20 remove_watcher() i=0 2024-03-09T14:31:06.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.781+0000 7f6eb67b0980 2 removed watcher, disabling cache 2024-03-09T14:31:06.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.781+0000 7f6eb67b0980 20 remove_watcher() i=1 2024-03-09T14:31:06.786 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.785+0000 7f6eb67b0980 20 remove_watcher() i=2 2024-03-09T14:31:06.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.785+0000 7f6eb67b0980 20 remove_watcher() i=3 2024-03-09T14:31:06.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.789+0000 7f6eb67b0980 20 remove_watcher() i=4 2024-03-09T14:31:06.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.789+0000 7f6eb67b0980 20 remove_watcher() i=5 2024-03-09T14:31:06.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.793+0000 7f6eb67b0980 20 remove_watcher() i=6 2024-03-09T14:31:06.794 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.793+0000 7f6eb67b0980 20 remove_watcher() i=7 2024-03-09T14:31:06.808 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket check-bucket --object c 2024-03-09T14:31:06.835 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:06.835 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:06.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.877+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.877+0000 7fd8c57ca980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.877+0000 7fd8c57ca980 20 realm 2024-03-09T14:31:06.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.877+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.877+0000 7fd8c57ca980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.877+0000 7fd8c57ca980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:06.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.877+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.881+0000 7fd8c57ca980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:06.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.881+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.881+0000 7fd8c57ca980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:06.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.881+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:06.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.881+0000 7fd8c57ca980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:06.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.881+0000 7fd8c57ca980 20 searching for the correct realm 2024-03-09T14:31:06.890 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:06.889+0000 7fd8c3ae3640 0 -- 172.21.15.1:0/4192842179 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7fd8b80f8860 msgr2=0x7fd8b81247d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:31:07.106 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.105+0000 7fd8c32e2640 0 -- 172.21.15.1:0/4192842179 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x55bf9138d230 msgr2=0x55bf9138f6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:07.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.113+0000 7fd8c57ca980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:07.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.113+0000 7fd8c57ca980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:07.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.113+0000 7fd8c57ca980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:07.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.113+0000 7fd8c57ca980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:07.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.113+0000 7fd8c57ca980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:07.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.113+0000 7fd8c57ca980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:07.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.113+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.113+0000 7fd8c57ca980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.113+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.113+0000 7fd8c57ca980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:07.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.113+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:07.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 20 zone default found 2024-03-09T14:31:07.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 4 Realm: () 2024-03-09T14:31:07.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:07.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:07.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:07.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 20 zonegroup default 2024-03-09T14:31:07.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 20 started sync module instance, tier type = 2024-03-09T14:31:07.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.117+0000 7fd8c57ca980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:07.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.145+0000 7fd8c57ca980 20 add_watcher() i=0 2024-03-09T14:31:07.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.145+0000 7fd8c57ca980 20 add_watcher() i=1 2024-03-09T14:31:07.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.145+0000 7fd8c57ca980 20 add_watcher() i=2 2024-03-09T14:31:07.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.145+0000 7fd8c57ca980 20 add_watcher() i=3 2024-03-09T14:31:07.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.145+0000 7fd8c57ca980 20 add_watcher() i=4 2024-03-09T14:31:07.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.145+0000 7fd8c57ca980 20 add_watcher() i=5 2024-03-09T14:31:07.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.145+0000 7fd8c57ca980 20 add_watcher() i=6 2024-03-09T14:31:07.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.145+0000 7fd8c57ca980 20 add_watcher() i=7 2024-03-09T14:31:07.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.145+0000 7fd8c57ca980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:07.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.149+0000 7fd77a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:07.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:07.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:07.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 5 note: GC not initialized 2024-03-09T14:31:07.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd7797e2640 20 reqs_thread_entry: start 2024-03-09T14:31:07.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:07.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 20 Filter name: none 2024-03-09T14:31:07.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd7727fc640 20 reqs_thread_entry: start 2024-03-09T14:31:07.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:07.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:07.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:07.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2024-03-09T14:31:07.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 20 INFO: main: starting shard=0 2024-03-09T14:31:07.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.153+0000 7fd8c57ca980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:07.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.157+0000 7fd8c57ca980 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2024-03-09T14:31:07.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.157+0000 7fd8c57ca980 20 INFO: main: done 2024-03-09T14:31:07.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.157+0000 7fd77a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:07.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.161+0000 7fd8c57ca980 20 remove_watcher() i=0 2024-03-09T14:31:07.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.161+0000 7fd8c57ca980 2 removed watcher, disabling cache 2024-03-09T14:31:07.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.165+0000 7fd8c57ca980 20 remove_watcher() i=1 2024-03-09T14:31:07.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.169+0000 7fd8c57ca980 20 remove_watcher() i=2 2024-03-09T14:31:07.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.169+0000 7fd8c57ca980 20 remove_watcher() i=3 2024-03-09T14:31:07.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.173+0000 7fd8c57ca980 20 remove_watcher() i=4 2024-03-09T14:31:07.174 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.173+0000 7fd8c57ca980 20 remove_watcher() i=5 2024-03-09T14:31:07.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.177+0000 7fd8c57ca980 20 remove_watcher() i=6 2024-03-09T14:31:07.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.177+0000 7fd8c57ca980 20 remove_watcher() i=7 2024-03-09T14:31:07.194 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket check-bucket --object d 2024-03-09T14:31:07.229 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:07.229 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:07.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.273+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.277+0000 7f99239fb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.277+0000 7f99239fb980 20 realm 2024-03-09T14:31:07.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.277+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.277+0000 7f99239fb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.277+0000 7f99239fb980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:07.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.277+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.277+0000 7f99239fb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.277+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.277+0000 7f99239fb980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:07.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.277+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.277+0000 7f99239fb980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:07.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.277+0000 7f99239fb980 20 searching for the correct realm 2024-03-09T14:31:07.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.281+0000 7f9920d12640 0 -- 172.21.15.1:0/3707395373 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f99140d9290 msgr2=0x7f9914007ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:07.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.497+0000 7f9921513640 0 -- 172.21.15.1:0/3707395373 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f991805af20 msgr2=0x7f99180731d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:07.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.713+0000 7f99239fb980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:07.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:07.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:07.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:07.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:07.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:07.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:07.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:07.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 zone default found 2024-03-09T14:31:07.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 4 Realm: () 2024-03-09T14:31:07.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:07.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:07.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:07.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 zonegroup default 2024-03-09T14:31:07.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.717+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.721+0000 7f99239fb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.721+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.721+0000 7f99239fb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.721+0000 7f99239fb980 20 started sync module instance, tier type = 2024-03-09T14:31:07.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.721+0000 7f99239fb980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:07.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.745+0000 7f99239fb980 20 add_watcher() i=0 2024-03-09T14:31:07.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.745+0000 7f99239fb980 20 add_watcher() i=1 2024-03-09T14:31:07.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.745+0000 7f99239fb980 20 add_watcher() i=2 2024-03-09T14:31:07.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.745+0000 7f99239fb980 20 add_watcher() i=3 2024-03-09T14:31:07.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.745+0000 7f99239fb980 20 add_watcher() i=4 2024-03-09T14:31:07.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.745+0000 7f99239fb980 20 add_watcher() i=5 2024-03-09T14:31:07.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.745+0000 7f99239fb980 20 add_watcher() i=6 2024-03-09T14:31:07.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.749+0000 7f99239fb980 20 add_watcher() i=7 2024-03-09T14:31:07.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.749+0000 7f99239fb980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:07.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.753+0000 7f97d77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:07.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.753+0000 7f99239fb980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:07.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.753+0000 7f99239fb980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:07.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.753+0000 7f99239fb980 5 note: GC not initialized 2024-03-09T14:31:07.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.753+0000 7f97d67e4640 20 reqs_thread_entry: start 2024-03-09T14:31:07.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.757+0000 7f99239fb980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:07.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.757+0000 7f99239fb980 20 Filter name: none 2024-03-09T14:31:07.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.757+0000 7f9747fff640 20 reqs_thread_entry: start 2024-03-09T14:31:07.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.757+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.757+0000 7f99239fb980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:07.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.757+0000 7f99239fb980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.757+0000 7f99239fb980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:07.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.757+0000 7f99239fb980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:07.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.757+0000 7f99239fb980 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2024-03-09T14:31:07.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.757+0000 7f99239fb980 20 INFO: main: starting shard=0 2024-03-09T14:31:07.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.757+0000 7f99239fb980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:07.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.757+0000 7f99239fb980 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2024-03-09T14:31:07.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.761+0000 7f99239fb980 20 INFO: main: done 2024-03-09T14:31:07.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.761+0000 7f97d77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:07.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.765+0000 7f99239fb980 20 remove_watcher() i=0 2024-03-09T14:31:07.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.765+0000 7f99239fb980 2 removed watcher, disabling cache 2024-03-09T14:31:07.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.769+0000 7f99239fb980 20 remove_watcher() i=1 2024-03-09T14:31:07.773 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.773+0000 7f99239fb980 20 remove_watcher() i=2 2024-03-09T14:31:07.775 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.773+0000 7f99239fb980 20 remove_watcher() i=3 2024-03-09T14:31:07.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.777+0000 7f99239fb980 20 remove_watcher() i=4 2024-03-09T14:31:07.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.777+0000 7f99239fb980 20 remove_watcher() i=5 2024-03-09T14:31:07.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.781+0000 7f99239fb980 20 remove_watcher() i=6 2024-03-09T14:31:07.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.781+0000 7f99239fb980 20 remove_watcher() i=7 2024-03-09T14:31:07.805 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: bucket check olh does not find new leftover OLH entries after fix 2024-03-09T14:31:07.805 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:31:07.805 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket check olh --bucket check-bucket --dump-keys 2024-03-09T14:31:07.835 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:07.835 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:07.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.877+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.881+0000 7f667301c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.881+0000 7f667301c980 20 realm 2024-03-09T14:31:07.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.881+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.881+0000 7f667301c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.881+0000 7f667301c980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:07.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.881+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.881+0000 7f667301c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.881+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.881+0000 7f667301c980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:07.884 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.881+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.884 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.881+0000 7f667301c980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:07.884 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.881+0000 7f667301c980 20 searching for the correct realm 2024-03-09T14:31:07.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.889+0000 7f666bfff640 0 -- 172.21.15.1:0/2897531926 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f6664078040 msgr2=0x7f666407c9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:07.898 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.897+0000 7f6671335640 0 -- 172.21.15.1:0/2897531926 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f665c08e040 msgr2=0x7f665c06e040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:07.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:07.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:07.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:07.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:07.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:07.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:07.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:07.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:07.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 zone default found 2024-03-09T14:31:07.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 4 Realm: () 2024-03-09T14:31:07.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:07.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:07.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:07.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 zonegroup default 2024-03-09T14:31:07.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:07.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 started sync module instance, tier type = 2024-03-09T14:31:07.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.905+0000 7f667301c980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:07.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.921+0000 7f667301c980 20 add_watcher() i=0 2024-03-09T14:31:07.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.921+0000 7f667301c980 20 add_watcher() i=1 2024-03-09T14:31:07.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.921+0000 7f667301c980 20 add_watcher() i=2 2024-03-09T14:31:07.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.921+0000 7f667301c980 20 add_watcher() i=3 2024-03-09T14:31:07.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.921+0000 7f667301c980 20 add_watcher() i=4 2024-03-09T14:31:07.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.921+0000 7f667301c980 20 add_watcher() i=5 2024-03-09T14:31:07.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.921+0000 7f667301c980 20 add_watcher() i=6 2024-03-09T14:31:07.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.925+0000 7f667301c980 20 add_watcher() i=7 2024-03-09T14:31:07.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.925+0000 7f667301c980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:07.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.925+0000 7f65277e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:07.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.925+0000 7f667301c980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:07.928 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.925+0000 7f667301c980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:07.928 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.925+0000 7f667301c980 5 note: GC not initialized 2024-03-09T14:31:07.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.925+0000 7f65267e4640 20 reqs_thread_entry: start 2024-03-09T14:31:07.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.925+0000 7f667301c980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:07.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.925+0000 7f667301c980 20 Filter name: none 2024-03-09T14:31:07.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f651f7fe640 20 reqs_thread_entry: start 2024-03-09T14:31:07.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2024-03-09T14:31:07.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:07.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2024-03-09T14:31:07.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2024-03-09T14:31:07.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 : miss 2024-03-09T14:31:07.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 20 rados->read ofs=0 len=0 2024-03-09T14:31:07.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:07.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 info.flags=0x17 2024-03-09T14:31:07.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 to cache LRU end 2024-03-09T14:31:07.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:31:07.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 10 updating xattr: name=user.rgw.acl bl.length()=179 2024-03-09T14:31:07.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 2024-03-09T14:31:07.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:07.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 1 NOTICE: finished shard 0 (0 entries found) 2024-03-09T14:31:07.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f667301c980 1 NOTICE: finished all shards (0 entries found) 2024-03-09T14:31:07.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.929+0000 7f65277e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:07.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.937+0000 7f667301c980 20 remove_watcher() i=0 2024-03-09T14:31:07.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.937+0000 7f667301c980 2 removed watcher, disabling cache 2024-03-09T14:31:07.938 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.937+0000 7f667301c980 20 remove_watcher() i=1 2024-03-09T14:31:07.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.941+0000 7f667301c980 20 remove_watcher() i=2 2024-03-09T14:31:07.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.941+0000 7f667301c980 20 remove_watcher() i=3 2024-03-09T14:31:07.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.945+0000 7f667301c980 20 remove_watcher() i=4 2024-03-09T14:31:07.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.945+0000 7f667301c980 20 remove_watcher() i=5 2024-03-09T14:31:07.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.945+0000 7f667301c980 20 remove_watcher() i=6 2024-03-09T14:31:07.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:07.949+0000 7f667301c980 20 remove_watcher() i=7 2024-03-09T14:31:07.963 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: verify that bucket check fixes do not affect null version objects 2024-03-09T14:31:07.963 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:31:07.964 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:07.964 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:07.991 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:07.991 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:08.020 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:08.020 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:08.096 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: bucket check stats are correct in the presence of unlinked entries 2024-03-09T14:31:08.096 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:31:08.098 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:08.098 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:08.208 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:08.208 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:08.779 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:08.779 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:08.948 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:08.948 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:09.021 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:09.022 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:09.169 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:09.169 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:09.243 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:09.243 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:09.387 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:09.387 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:09.462 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:09.462 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:09.624 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:09.624 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:09.699 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:09.699 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:09.847 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:09.847 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:09.920 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:09.920 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:10.069 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:10.069 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:10.138 INFO:tasks.workunit.client.0.smithi001.stderr:ceph config set client rgw_debug_inject_set_olh_err 2 2024-03-09T14:31:10.589 INFO:tasks.workunit.client.0.smithi001.stderr:ceph config set client rgw_debug_inject_olh_cancel_modification_err true 2024-03-09T14:31:12.029 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:12.030 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:12.189 INFO:tasks.workunit.client.0.smithi001.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2024-03-09T14:31:12.190 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket check-bucket --object c 2024-03-09T14:31:12.225 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:12.226 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:12.272 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.273+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.273+0000 7f666c344980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:12.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.273+0000 7f666c344980 20 realm 2024-03-09T14:31:12.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.273+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.273+0000 7f666c344980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:12.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.273+0000 7f666c344980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:12.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.273+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.273+0000 7f666c344980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:12.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.273+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.277+0000 7f666c344980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:12.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.277+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.277+0000 7f666c344980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:12.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.277+0000 7f666c344980 20 searching for the correct realm 2024-03-09T14:31:12.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.281+0000 7f666965b640 0 -- 172.21.15.1:0/2823708286 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f665c028040 msgr2=0x7f665c0297e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:12.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.497+0000 7f6669e5c640 0 -- 172.21.15.1:0/2823708286 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f66600630b0 msgr2=0x7f66600594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:12.715 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.713+0000 7f666c344980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:12.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.713+0000 7f666c344980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:12.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.713+0000 7f666c344980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:12.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.713+0000 7f666c344980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:12.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.713+0000 7f666c344980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:12.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.713+0000 7f666c344980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:12.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.713+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.713+0000 7f666c344980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:12.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.713+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:12.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:12.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 zone default found 2024-03-09T14:31:12.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 4 Realm: () 2024-03-09T14:31:12.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:12.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:12.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:12.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 zonegroup default 2024-03-09T14:31:12.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:12.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:12.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:12.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 started sync module instance, tier type = 2024-03-09T14:31:12.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.717+0000 7f666c344980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:12.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.745+0000 7f666c344980 20 add_watcher() i=0 2024-03-09T14:31:12.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.745+0000 7f666c344980 20 add_watcher() i=1 2024-03-09T14:31:12.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.745+0000 7f666c344980 20 add_watcher() i=2 2024-03-09T14:31:12.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.745+0000 7f666c344980 20 add_watcher() i=3 2024-03-09T14:31:12.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.745+0000 7f666c344980 20 add_watcher() i=4 2024-03-09T14:31:12.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.745+0000 7f666c344980 20 add_watcher() i=5 2024-03-09T14:31:12.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.745+0000 7f666c344980 20 add_watcher() i=6 2024-03-09T14:31:12.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.749+0000 7f666c344980 20 add_watcher() i=7 2024-03-09T14:31:12.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.749+0000 7f666c344980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:12.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.753+0000 7f651ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:12.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.753+0000 7f666c344980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:12.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.753+0000 7f666c344980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:12.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.753+0000 7f666c344980 5 note: GC not initialized 2024-03-09T14:31:12.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.753+0000 7f651efe5640 20 reqs_thread_entry: start 2024-03-09T14:31:12.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.753+0000 7f666c344980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:12.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.753+0000 7f666c344980 20 Filter name: none 2024-03-09T14:31:12.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.753+0000 7f651cfe1640 20 reqs_thread_entry: start 2024-03-09T14:31:12.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.753+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.753+0000 7f666c344980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:12.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.753+0000 7f666c344980 20 rados->read ofs=0 len=0 2024-03-09T14:31:12.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.757+0000 7f666c344980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:12.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.757+0000 7f666c344980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:12.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.757+0000 7f666c344980 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2024-03-09T14:31:12.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.757+0000 7f666c344980 20 INFO: main: starting shard=0 2024-03-09T14:31:12.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.757+0000 7f666c344980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:12.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.757+0000 7f666c344980 20 INFO: main: bi_list() returned without error; entries.size()=5, is_truncated=0, marker= 2024-03-09T14:31:12.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.757+0000 7f666c344980 20 INFO: main: done 2024-03-09T14:31:12.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.757+0000 7f651ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:12.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.765+0000 7f666c344980 20 remove_watcher() i=0 2024-03-09T14:31:12.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.765+0000 7f666c344980 2 removed watcher, disabling cache 2024-03-09T14:31:12.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.769+0000 7f666c344980 20 remove_watcher() i=1 2024-03-09T14:31:12.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.769+0000 7f666c344980 20 remove_watcher() i=2 2024-03-09T14:31:12.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.773+0000 7f666c344980 20 remove_watcher() i=3 2024-03-09T14:31:12.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.777+0000 7f666c344980 20 remove_watcher() i=4 2024-03-09T14:31:12.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.777+0000 7f666c344980 20 remove_watcher() i=5 2024-03-09T14:31:12.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.781+0000 7f666c344980 20 remove_watcher() i=6 2024-03-09T14:31:12.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:12.781+0000 7f666c344980 20 remove_watcher() i=7 2024-03-09T14:31:12.801 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:12.802 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:12.835 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:12.835 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:13.001 INFO:tasks.workunit.client.0.smithi001.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2024-03-09T14:31:13.002 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket check-bucket --object d 2024-03-09T14:31:13.038 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:13.038 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:13.084 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.085+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.085+0000 7fe91f80e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.085+0000 7fe91f80e980 20 realm 2024-03-09T14:31:13.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.085+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.085+0000 7fe91f80e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.085+0000 7fe91f80e980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:13.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.085+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.085+0000 7fe91f80e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.085+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.089+0000 7fe91f80e980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:13.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.089+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.089+0000 7fe91f80e980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:13.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.089+0000 7fe91f80e980 20 searching for the correct realm 2024-03-09T14:31:13.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.093+0000 7fe91d326640 0 -- 172.21.15.1:0/1314153243 >> [v2:172.21.15.1:6824/1666056338,v1:172.21.15.1:6825/1666056338] conn(0x55fd76de2f50 msgr2=0x7fe9180c3920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:13.100 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.097+0000 7fe91d326640 0 -- 172.21.15.1:0/1314153243 >> [v2:172.21.15.1:6824/1666056338,v1:172.21.15.1:6825/1666056338] conn(0x7fe7e80733a0 msgr2=0x7fe7e80757f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:31:13.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:13.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:13.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:13.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:13.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:13.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:13.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:13.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:13.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 zone default found 2024-03-09T14:31:13.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 4 Realm: () 2024-03-09T14:31:13.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:13.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:13.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:13.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 zonegroup default 2024-03-09T14:31:13.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.109+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.113+0000 7fe91f80e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.113+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.113+0000 7fe91f80e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.113+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.113+0000 7fe91f80e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.113+0000 7fe91f80e980 20 started sync module instance, tier type = 2024-03-09T14:31:13.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.113+0000 7fe91f80e980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:13.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.129+0000 7fe91f80e980 20 add_watcher() i=0 2024-03-09T14:31:13.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.129+0000 7fe91f80e980 20 add_watcher() i=1 2024-03-09T14:31:13.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.129+0000 7fe91f80e980 20 add_watcher() i=2 2024-03-09T14:31:13.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.129+0000 7fe91f80e980 20 add_watcher() i=3 2024-03-09T14:31:13.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.129+0000 7fe91f80e980 20 add_watcher() i=4 2024-03-09T14:31:13.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.129+0000 7fe91f80e980 20 add_watcher() i=5 2024-03-09T14:31:13.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.129+0000 7fe91f80e980 20 add_watcher() i=6 2024-03-09T14:31:13.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.133+0000 7fe91f80e980 20 add_watcher() i=7 2024-03-09T14:31:13.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.133+0000 7fe91f80e980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:13.135 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.133+0000 7fe7d37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:13.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.133+0000 7fe91f80e980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:13.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.133+0000 7fe91f80e980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:13.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.133+0000 7fe91f80e980 5 note: GC not initialized 2024-03-09T14:31:13.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe7d27e4640 20 reqs_thread_entry: start 2024-03-09T14:31:13.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe91f80e980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:13.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe91f80e980 20 Filter name: none 2024-03-09T14:31:13.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe743fff640 20 reqs_thread_entry: start 2024-03-09T14:31:13.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe91f80e980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:13.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe91f80e980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe91f80e980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:13.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe91f80e980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:13.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe91f80e980 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2024-03-09T14:31:13.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe91f80e980 20 INFO: main: starting shard=0 2024-03-09T14:31:13.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe91f80e980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:13.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe91f80e980 20 INFO: main: bi_list() returned without error; entries.size()=5, is_truncated=0, marker= 2024-03-09T14:31:13.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.137+0000 7fe91f80e980 20 INFO: main: done 2024-03-09T14:31:13.140 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.141+0000 7fe7d37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:13.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.145+0000 7fe91f80e980 20 remove_watcher() i=0 2024-03-09T14:31:13.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.145+0000 7fe91f80e980 2 removed watcher, disabling cache 2024-03-09T14:31:13.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.149+0000 7fe91f80e980 20 remove_watcher() i=1 2024-03-09T14:31:13.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.149+0000 7fe91f80e980 20 remove_watcher() i=2 2024-03-09T14:31:13.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.153+0000 7fe91f80e980 20 remove_watcher() i=3 2024-03-09T14:31:13.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.153+0000 7fe91f80e980 20 remove_watcher() i=4 2024-03-09T14:31:13.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.153+0000 7fe91f80e980 20 remove_watcher() i=5 2024-03-09T14:31:13.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.157+0000 7fe91f80e980 20 remove_watcher() i=6 2024-03-09T14:31:13.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.157+0000 7fe91f80e980 20 remove_watcher() i=7 2024-03-09T14:31:13.174 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:13.174 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:13.204 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:13.204 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:13.357 INFO:tasks.workunit.client.0.smithi001.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2024-03-09T14:31:13.358 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket check-bucket --object e 2024-03-09T14:31:13.393 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:13.394 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:13.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.437+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.441+0000 7f34ef1e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.441+0000 7f34ef1e1980 20 realm 2024-03-09T14:31:13.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.441+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.441+0000 7f34ef1e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.441+0000 7f34ef1e1980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:13.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.441+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.441+0000 7f34ef1e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.441+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.441+0000 7f34ef1e1980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:13.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.441+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.441+0000 7f34ef1e1980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:13.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.441+0000 7f34ef1e1980 20 searching for the correct realm 2024-03-09T14:31:13.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.449+0000 7f34ed4fa640 0 -- 172.21.15.1:0/3672278931 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x562111b460e0 msgr2=0x562111b48530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:13.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.457+0000 7f34ec4f8640 0 -- 172.21.15.1:0/3672278931 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f34dc0cb040 msgr2=0x7f34dc0eedc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:13.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:13.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:13.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:13.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:13.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:13.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:13.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 zone default found 2024-03-09T14:31:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 4 Realm: () 2024-03-09T14:31:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 zonegroup default 2024-03-09T14:31:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 started sync module instance, tier type = 2024-03-09T14:31:13.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.465+0000 7f34ef1e1980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:13.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.485+0000 7f34ef1e1980 20 add_watcher() i=0 2024-03-09T14:31:13.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.485+0000 7f34ef1e1980 20 add_watcher() i=1 2024-03-09T14:31:13.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.485+0000 7f34ef1e1980 20 add_watcher() i=2 2024-03-09T14:31:13.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.485+0000 7f34ef1e1980 20 add_watcher() i=3 2024-03-09T14:31:13.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.485+0000 7f34ef1e1980 20 add_watcher() i=4 2024-03-09T14:31:13.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.485+0000 7f34ef1e1980 20 add_watcher() i=5 2024-03-09T14:31:13.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.485+0000 7f34ef1e1980 20 add_watcher() i=6 2024-03-09T14:31:13.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.485+0000 7f34ef1e1980 20 add_watcher() i=7 2024-03-09T14:31:13.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.485+0000 7f34ef1e1980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:13.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f33a2fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:13.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:13.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:13.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 5 note: GC not initialized 2024-03-09T14:31:13.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f33a1fe3640 20 reqs_thread_entry: start 2024-03-09T14:31:13.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:13.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 Filter name: none 2024-03-09T14:31:13.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f33137fe640 20 reqs_thread_entry: start 2024-03-09T14:31:13.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:13.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:13.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:13.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2024-03-09T14:31:13.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 INFO: main: starting shard=0 2024-03-09T14:31:13.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:13.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2024-03-09T14:31:13.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.489+0000 7f34ef1e1980 20 INFO: main: done 2024-03-09T14:31:13.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.493+0000 7f33a2fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:13.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.497+0000 7f34ef1e1980 20 remove_watcher() i=0 2024-03-09T14:31:13.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.497+0000 7f34ef1e1980 2 removed watcher, disabling cache 2024-03-09T14:31:13.500 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.501+0000 7f34ef1e1980 20 remove_watcher() i=1 2024-03-09T14:31:13.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.501+0000 7f34ef1e1980 20 remove_watcher() i=2 2024-03-09T14:31:13.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.505+0000 7f34ef1e1980 20 remove_watcher() i=3 2024-03-09T14:31:13.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.505+0000 7f34ef1e1980 20 remove_watcher() i=4 2024-03-09T14:31:13.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.505+0000 7f34ef1e1980 20 remove_watcher() i=5 2024-03-09T14:31:13.509 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.509+0000 7f34ef1e1980 20 remove_watcher() i=6 2024-03-09T14:31:13.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.509+0000 7f34ef1e1980 20 remove_watcher() i=7 2024-03-09T14:31:13.530 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:13.530 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:13.552 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:13.553 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:13.677 INFO:tasks.workunit.client.0.smithi001.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2024-03-09T14:31:13.678 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket check-bucket --object f 2024-03-09T14:31:13.715 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:13.715 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:13.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.757+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.761+0000 7f5b7e61d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.761+0000 7f5b7e61d980 20 realm 2024-03-09T14:31:13.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.761+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.761+0000 7f5b7e61d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.761+0000 7f5b7e61d980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:13.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.761+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.761+0000 7f5b7e61d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.761+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.765+0000 7f5b7e61d980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:13.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.765+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.765+0000 7f5b7e61d980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:13.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.765+0000 7f5b7e61d980 20 searching for the correct realm 2024-03-09T14:31:13.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.773+0000 7f5b7b934640 0 -- 172.21.15.1:0/3018239433 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f5b6c0d0030 msgr2=0x7f5b6c0ba9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:13.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.777+0000 7f5b7b934640 0 -- 172.21.15.1:0/3018239433 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f5a44073400 msgr2=0x7f5a44075850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:13.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.785+0000 7f5b7e61d980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:13.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.785+0000 7f5b7e61d980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:13.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.785+0000 7f5b7e61d980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:13.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.785+0000 7f5b7e61d980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:13.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.785+0000 7f5b7e61d980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:13.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.785+0000 7f5b7e61d980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:13.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.785+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.785+0000 7f5b7e61d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.785+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.785+0000 7f5b7e61d980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:13.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.785+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:13.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 20 zone default found 2024-03-09T14:31:13.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 4 Realm: () 2024-03-09T14:31:13.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:13.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:13.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:13.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 20 zonegroup default 2024-03-09T14:31:13.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:13.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 20 started sync module instance, tier type = 2024-03-09T14:31:13.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.789+0000 7f5b7e61d980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:13.807 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.805+0000 7f5b7e61d980 20 add_watcher() i=0 2024-03-09T14:31:13.807 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.805+0000 7f5b7e61d980 20 add_watcher() i=1 2024-03-09T14:31:13.807 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.805+0000 7f5b7e61d980 20 add_watcher() i=2 2024-03-09T14:31:13.807 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.805+0000 7f5b7e61d980 20 add_watcher() i=3 2024-03-09T14:31:13.807 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.805+0000 7f5b7e61d980 20 add_watcher() i=4 2024-03-09T14:31:13.807 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.805+0000 7f5b7e61d980 20 add_watcher() i=5 2024-03-09T14:31:13.807 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.805+0000 7f5b7e61d980 20 add_watcher() i=6 2024-03-09T14:31:13.808 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.805+0000 7f5b7e61d980 20 add_watcher() i=7 2024-03-09T14:31:13.808 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.805+0000 7f5b7e61d980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:13.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.809+0000 7f5a32fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:13.811 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.809+0000 7f5b7e61d980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:13.812 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.809+0000 7f5b7e61d980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:13.812 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.809+0000 7f5b7e61d980 5 note: GC not initialized 2024-03-09T14:31:13.812 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.809+0000 7f5a31fdb640 20 reqs_thread_entry: start 2024-03-09T14:31:13.812 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.809+0000 7f5b7e61d980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:13.812 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.809+0000 7f5b7e61d980 20 Filter name: none 2024-03-09T14:31:13.812 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.809+0000 7f59a37fe640 20 reqs_thread_entry: start 2024-03-09T14:31:13.812 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.809+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.809+0000 7f5b7e61d980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:13.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.813+0000 7f5b7e61d980 20 rados->read ofs=0 len=0 2024-03-09T14:31:13.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.813+0000 7f5b7e61d980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:13.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.813+0000 7f5b7e61d980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:13.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.813+0000 7f5b7e61d980 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2024-03-09T14:31:13.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.813+0000 7f5b7e61d980 20 INFO: main: starting shard=0 2024-03-09T14:31:13.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.813+0000 7f5b7e61d980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:13.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.813+0000 7f5b7e61d980 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2024-03-09T14:31:13.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.813+0000 7f5b7e61d980 20 INFO: main: done 2024-03-09T14:31:13.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.813+0000 7f5a32fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:13.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.817+0000 7f5b7e61d980 20 remove_watcher() i=0 2024-03-09T14:31:13.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.817+0000 7f5b7e61d980 2 removed watcher, disabling cache 2024-03-09T14:31:13.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.821+0000 7f5b7e61d980 20 remove_watcher() i=1 2024-03-09T14:31:13.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.825+0000 7f5b7e61d980 20 remove_watcher() i=2 2024-03-09T14:31:13.826 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.825+0000 7f5b7e61d980 20 remove_watcher() i=3 2024-03-09T14:31:13.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.829+0000 7f5b7e61d980 20 remove_watcher() i=4 2024-03-09T14:31:13.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.829+0000 7f5b7e61d980 20 remove_watcher() i=5 2024-03-09T14:31:13.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.829+0000 7f5b7e61d980 20 remove_watcher() i=6 2024-03-09T14:31:13.833 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:13.833+0000 7f5b7e61d980 20 remove_watcher() i=7 2024-03-09T14:31:13.848 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:13.849 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:13.878 INFO:tasks.workunit.client.0.smithi001.stderr:ceph config rm client rgw_debug_inject_set_olh_err 2024-03-09T14:31:14.298 INFO:tasks.workunit.client.0.smithi001.stderr:ceph config rm client rgw_debug_inject_olh_cancel_modification_err 2024-03-09T14:31:14.713 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket check --fix --bucket check-bucket 2024-03-09T14:31:14.746 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:14.746 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:14.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.789+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:14.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.789+0000 7f0f6ddc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:14.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.789+0000 7f0f6ddc0980 20 realm 2024-03-09T14:31:14.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.789+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:14.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.789+0000 7f0f6ddc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:14.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.789+0000 7f0f6ddc0980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:14.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.789+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:14.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.789+0000 7f0f6ddc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:14.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.789+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:14.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.793+0000 7f0f6ddc0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:14.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.793+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:14.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.793+0000 7f0f6ddc0980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:14.794 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.793+0000 7f0f6ddc0980 20 searching for the correct realm 2024-03-09T14:31:14.801 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:14.801+0000 7f0f6c0d9640 0 -- 172.21.15.1:0/701725111 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f0f60078d00 msgr2=0x7f0f6008d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:31:15.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.017+0000 7f0f6b8d8640 0 -- 172.21.15.1:0/701725111 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x55e51613ef60 msgr2=0x55e5161413b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:15.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.025+0000 7f0f6ddc0980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:15.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.025+0000 7f0f6ddc0980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:15.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.025+0000 7f0f6ddc0980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:15.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.025+0000 7f0f6ddc0980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:15.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.025+0000 7f0f6ddc0980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:15.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.025+0000 7f0f6ddc0980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:15.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.025+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.025+0000 7f0f6ddc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.025+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:15.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:15.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 20 zone default found 2024-03-09T14:31:15.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 4 Realm: () 2024-03-09T14:31:15.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:15.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:15.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:15.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 20 zonegroup default 2024-03-09T14:31:15.031 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.031 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.031 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.029+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.033+0000 7f0f6ddc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.033+0000 7f0f6ddc0980 20 started sync module instance, tier type = 2024-03-09T14:31:15.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.033+0000 7f0f6ddc0980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:15.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.057+0000 7f0f6ddc0980 20 add_watcher() i=0 2024-03-09T14:31:15.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.057+0000 7f0f6ddc0980 20 add_watcher() i=1 2024-03-09T14:31:15.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.057+0000 7f0f6ddc0980 20 add_watcher() i=2 2024-03-09T14:31:15.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.057+0000 7f0f6ddc0980 20 add_watcher() i=3 2024-03-09T14:31:15.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.057+0000 7f0f6ddc0980 20 add_watcher() i=4 2024-03-09T14:31:15.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.057+0000 7f0f6ddc0980 20 add_watcher() i=5 2024-03-09T14:31:15.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.057+0000 7f0f6ddc0980 20 add_watcher() i=6 2024-03-09T14:31:15.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.061+0000 7f0f6ddc0980 20 add_watcher() i=7 2024-03-09T14:31:15.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.061+0000 7f0f6ddc0980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:15.065 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0e21fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:15.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0f6ddc0980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:15.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0f6ddc0980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:15.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0f6ddc0980 5 note: GC not initialized 2024-03-09T14:31:15.067 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0e20fe1640 20 reqs_thread_entry: start 2024-03-09T14:31:15.067 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0f6ddc0980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:15.067 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0f6ddc0980 20 Filter name: none 2024-03-09T14:31:15.067 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0d9a7fc640 20 reqs_thread_entry: start 2024-03-09T14:31:15.067 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0f6ddc0980 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2024-03-09T14:31:15.067 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.068 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0f6ddc0980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:15.068 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0f6ddc0980 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2024-03-09T14:31:15.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0f6ddc0980 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2024-03-09T14:31:15.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0f6ddc0980 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 : miss 2024-03-09T14:31:15.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.065+0000 7f0f6ddc0980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.069+0000 7f0f6ddc0980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:15.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.069+0000 7f0f6ddc0980 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 info.flags=0x17 2024-03-09T14:31:15.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.069+0000 7f0f6ddc0980 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 to cache LRU end 2024-03-09T14:31:15.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.069+0000 7f0f6ddc0980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:31:15.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.069+0000 7f0f6ddc0980 10 updating xattr: name=user.rgw.acl bl.length()=179 2024-03-09T14:31:15.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.069+0000 7f0f6ddc0980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 2024-03-09T14:31:15.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.069+0000 7f0f6ddc0980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:31:15.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.069+0000 7f0f6ddc0980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:31:15.070 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.069+0000 7f0f6ddc0980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:31:15.070 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.069+0000 7f0f6ddc0980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:31:15.070 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.069+0000 7f0f6ddc0980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:31:15.070 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.069+0000 7f0f6ddc0980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:31:15.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.077+0000 7f0e21fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:15.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.081+0000 7f0f6ddc0980 20 remove_watcher() i=0 2024-03-09T14:31:15.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.081+0000 7f0f6ddc0980 2 removed watcher, disabling cache 2024-03-09T14:31:15.084 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.081+0000 7f0f6ddc0980 20 remove_watcher() i=1 2024-03-09T14:31:15.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.085+0000 7f0f6ddc0980 20 remove_watcher() i=2 2024-03-09T14:31:15.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.085+0000 7f0f6ddc0980 20 remove_watcher() i=3 2024-03-09T14:31:15.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.089+0000 7f0f6ddc0980 20 remove_watcher() i=4 2024-03-09T14:31:15.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.093+0000 7f0f6ddc0980 20 remove_watcher() i=5 2024-03-09T14:31:15.094 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.093+0000 7f0f6ddc0980 20 remove_watcher() i=6 2024-03-09T14:31:15.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.093+0000 7f0f6ddc0980 20 remove_watcher() i=7 2024-03-09T14:31:15.111 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --fix --min-age-hours 0 --rgw-olh-pending-timeout-sec 0 --dump-keys 2024-03-09T14:31:15.144 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:15.144 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:15.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.189+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.189+0000 7f462bce6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.189+0000 7f462bce6980 20 realm 2024-03-09T14:31:15.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.189+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.189+0000 7f462bce6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.189+0000 7f462bce6980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:15.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.189+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.189+0000 7f462bce6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.189+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.189+0000 7f462bce6980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:15.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.189+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.193+0000 7f462bce6980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:15.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.193+0000 7f462bce6980 20 searching for the correct realm 2024-03-09T14:31:15.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.197+0000 7f4628ffd640 0 -- 172.21.15.1:0/1820632967 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x55d2128dbc50 msgr2=0x55d2128de110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:15.205 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.205+0000 7f46297fe640 0 -- 172.21.15.1:0/1820632967 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f462001cab0 msgr2=0x7f462002b420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:31:15.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:15.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:15.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:15.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:15.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:15.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:15.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:15.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:15.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 zone default found 2024-03-09T14:31:15.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 4 Realm: () 2024-03-09T14:31:15.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:15.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:15.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:15.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 zonegroup default 2024-03-09T14:31:15.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.213+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.217 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.217+0000 7f462bce6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.217 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.217+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.217 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.217+0000 7f462bce6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.217 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.217+0000 7f462bce6980 20 started sync module instance, tier type = 2024-03-09T14:31:15.217 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.217+0000 7f462bce6980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:15.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.233+0000 7f462bce6980 20 add_watcher() i=0 2024-03-09T14:31:15.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.233+0000 7f462bce6980 20 add_watcher() i=1 2024-03-09T14:31:15.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.233+0000 7f462bce6980 20 add_watcher() i=2 2024-03-09T14:31:15.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.233+0000 7f462bce6980 20 add_watcher() i=3 2024-03-09T14:31:15.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.233+0000 7f462bce6980 20 add_watcher() i=4 2024-03-09T14:31:15.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.233+0000 7f462bce6980 20 add_watcher() i=5 2024-03-09T14:31:15.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.233+0000 7f462bce6980 20 add_watcher() i=6 2024-03-09T14:31:15.235 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.233+0000 7f462bce6980 20 add_watcher() i=7 2024-03-09T14:31:15.235 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.233+0000 7f462bce6980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:15.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f44df7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:15.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:15.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:15.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 5 note: GC not initialized 2024-03-09T14:31:15.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f44de7e4640 20 reqs_thread_entry: start 2024-03-09T14:31:15.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:15.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 20 Filter name: none 2024-03-09T14:31:15.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f444ffff640 20 reqs_thread_entry: start 2024-03-09T14:31:15.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2024-03-09T14:31:15.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:15.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2024-03-09T14:31:15.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2024-03-09T14:31:15.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 : miss 2024-03-09T14:31:15.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:15.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 info.flags=0x17 2024-03-09T14:31:15.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 to cache LRU end 2024-03-09T14:31:15.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:31:15.242 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 10 updating xattr: name=user.rgw.acl bl.length()=179 2024-03-09T14:31:15.242 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1 2024-03-09T14:31:15.242 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.237+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.242 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.241+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.243 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.241+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:c state=0x55d212912fd8 s->prefetch_data=0 2024-03-09T14:31:15.244 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.241+0000 7f462bce6980 20 get_obj_state: setting s->obj_tag to ry8zj0w1t9q8g7slchopydvgbj1j3p7w 2024-03-09T14:31:15.244 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.241+0000 7f462bce6980 20 get_obj_state_impl: setting s->olh_tag to co103favg8ure1i0tv90fyw074i7gb65 2024-03-09T14:31:15.244 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.241+0000 7f462bce6980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=co103favg8ure1i0tv90fyw074i7gb65 2024-03-09T14:31:15.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.245+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.249+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.249+0000 7f462bce6980 20 olh_log_entry: epoch=3 op=3 key=c[oIqZIryi-wqkPLEaq4PBZkF5YQXid-j] 2024-03-09T14:31:15.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.249+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:_:oIqZIryi-wqkPLEaq4PBZkF5YQXid-j_c state=0x55d21290c648 s->prefetch_data=0 2024-03-09T14:31:15.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.249+0000 7f462bce6980 10 manifest: total_size = 22 2024-03-09T14:31:15.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.249+0000 7f462bce6980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.1566176098442176211 2024-03-09T14:31:15.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.249+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:_:oIqZIryi-wqkPLEaq4PBZkF5YQXid-j_c state=0x55d21290c648 s->prefetch_data=0 2024-03-09T14:31:15.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.249+0000 7f462bce6980 20 prepare_atomic_modification: state is not atomic. state=0x55d21290c648 2024-03-09T14:31:15.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.249+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.253+0000 7f462bce6980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:31:15.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.253+0000 7f462bce6980 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2024-03-09T14:31:15.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.253+0000 7f462bce6980 0 deleting objects inline since gc isn't initialized 2024-03-09T14:31:15.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.253+0000 7f462bce6980 5 delete_objs_inline: removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1__shadow:oIqZIryi-wqkPLEaq4PBZkF5YQXid-j_.dzJ0TxKCNJ83ZjvAB8w62nSIeo-a3BT_0 2024-03-09T14:31:15.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.257+0000 7f462bce6980 5 delete_objs_inline: refcount put returned error -2 2024-03-09T14:31:15.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.257+0000 7f450cff9640 20 handle_completion(): completion ok for obj=c[oIqZIryi-wqkPLEaq4PBZkF5YQXid-j] 2024-03-09T14:31:15.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.257+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.257+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.265+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.267 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.265+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.265+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:d state=0x55d21290f018 s->prefetch_data=0 2024-03-09T14:31:15.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.269+0000 7f462bce6980 20 get_obj_state: setting s->obj_tag to 6pkz5w1zzrvy31f16dz9w7tgseqdx063 2024-03-09T14:31:15.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.269+0000 7f462bce6980 20 get_obj_state_impl: setting s->olh_tag to 1lzhsjg32aoppig2aroeznhfkuq4tw84 2024-03-09T14:31:15.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.269+0000 7f462bce6980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=1lzhsjg32aoppig2aroeznhfkuq4tw84 2024-03-09T14:31:15.270 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.269+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.273+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.273+0000 7f462bce6980 20 olh_log_entry: epoch=3 op=3 key=d[6U3-AXQPS7jJnPPlbDSf5h8iKan.jm4] 2024-03-09T14:31:15.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.273+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:_:6U3-AXQPS7jJnPPlbDSf5h8iKan.jm4_d state=0x55d2129075f8 s->prefetch_data=0 2024-03-09T14:31:15.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.273+0000 7f462bce6980 10 manifest: total_size = 11 2024-03-09T14:31:15.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.273+0000 7f462bce6980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13155223920587027782 2024-03-09T14:31:15.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.273+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:_:6U3-AXQPS7jJnPPlbDSf5h8iKan.jm4_d state=0x55d2129075f8 s->prefetch_data=0 2024-03-09T14:31:15.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.273+0000 7f462bce6980 20 prepare_atomic_modification: state is not atomic. state=0x55d2129075f8 2024-03-09T14:31:15.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.273+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.277+0000 7f462bce6980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:31:15.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.277+0000 7f462bce6980 20 RGWObjManifest::operator++(): result: ofs=11 stripe_ofs=11 part_ofs=0 rule->part_size=0 2024-03-09T14:31:15.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.277+0000 7f462bce6980 0 deleting objects inline since gc isn't initialized 2024-03-09T14:31:15.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.277+0000 7f462bce6980 5 delete_objs_inline: removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1__shadow:6U3-AXQPS7jJnPPlbDSf5h8iKan.jm4_.r6_9YEYBdqdyrEp9ZlRM6jx8Wj8G-L7_0 2024-03-09T14:31:15.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.281+0000 7f462bce6980 5 delete_objs_inline: refcount put returned error -2 2024-03-09T14:31:15.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.281+0000 7f450cff9640 20 handle_completion(): completion ok for obj=d[6U3-AXQPS7jJnPPlbDSf5h8iKan.jm4] 2024-03-09T14:31:15.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.285+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.285+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.288 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.285+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.289+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.289+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:e state=0x55d21290d4f8 s->prefetch_data=0 2024-03-09T14:31:15.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.289+0000 7f462bce6980 20 get_obj_state: setting s->obj_tag to 10enh6wc7mg57pprnq8qkmqoyyq3zmsl 2024-03-09T14:31:15.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.289+0000 7f462bce6980 20 get_obj_state_impl: setting s->olh_tag to 7u702ii2bi5mbjum3l6vr14danirevfn 2024-03-09T14:31:15.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.289+0000 7f462bce6980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=7u702ii2bi5mbjum3l6vr14danirevfn 2024-03-09T14:31:15.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.293+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.293+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.297+0000 7f462bce6980 20 olh_log_entry: epoch=2 op=2 key=e[] 2024-03-09T14:31:15.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.297+0000 7f462bce6980 20 olh_log_entry: epoch=2 op=3 key=e[ze0nDX1EsJgqZeRloVyU2IXWhI50pqJ] 2024-03-09T14:31:15.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.297+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:_:ze0nDX1EsJgqZeRloVyU2IXWhI50pqJ_e state=0x55d212941588 s->prefetch_data=0 2024-03-09T14:31:15.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.297+0000 7f462bce6980 10 manifest: total_size = 11 2024-03-09T14:31:15.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.297+0000 7f462bce6980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4528769209146649671 2024-03-09T14:31:15.299 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.297+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:_:ze0nDX1EsJgqZeRloVyU2IXWhI50pqJ_e state=0x55d212941588 s->prefetch_data=0 2024-03-09T14:31:15.299 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.297+0000 7f462bce6980 20 prepare_atomic_modification: state is not atomic. state=0x55d212941588 2024-03-09T14:31:15.299 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.297+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.301+0000 7f462bce6980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:31:15.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.301+0000 7f462bce6980 20 RGWObjManifest::operator++(): result: ofs=11 stripe_ofs=11 part_ofs=0 rule->part_size=0 2024-03-09T14:31:15.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.301+0000 7f462bce6980 0 deleting objects inline since gc isn't initialized 2024-03-09T14:31:15.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.301+0000 7f462bce6980 5 delete_objs_inline: removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1__shadow:ze0nDX1EsJgqZeRloVyU2IXWhI50pqJ_.eVDvtvUeCYKox4Gj-lMpsIlbCN-39-g_0 2024-03-09T14:31:15.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.305+0000 7f462bce6980 5 delete_objs_inline: refcount put returned error -2 2024-03-09T14:31:15.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.305+0000 7f44fffff640 20 handle_completion(): completion ok for obj=e[ze0nDX1EsJgqZeRloVyU2IXWhI50pqJ] 2024-03-09T14:31:15.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.305+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:e state=0x55d21290d4f8 s->prefetch_data=0 2024-03-09T14:31:15.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.305+0000 7f462bce6980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=7u702ii2bi5mbjum3l6vr14danirevfn 2024-03-09T14:31:15.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.309+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.313+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.316 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.317+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.317+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:f state=0x55d21290d4f8 s->prefetch_data=0 2024-03-09T14:31:15.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.317+0000 7f462bce6980 20 get_obj_state: setting s->obj_tag to ulnoq5752d4m7zjw8t3qolm8v4nhx58b 2024-03-09T14:31:15.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.317+0000 7f462bce6980 20 get_obj_state_impl: setting s->olh_tag to 5tfi3r41f6uxfcf53nmzwjcqqob9idfg 2024-03-09T14:31:15.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.317+0000 7f462bce6980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=5tfi3r41f6uxfcf53nmzwjcqqob9idfg 2024-03-09T14:31:15.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.321+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.324 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.321+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.324 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.325+0000 7f462bce6980 20 olh_log_entry: epoch=2 op=2 key=f[] 2024-03-09T14:31:15.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.325+0000 7f462bce6980 20 olh_log_entry: epoch=2 op=3 key=f[XlxD2gCwXwS2ua0znqme9OV5QI8oqAA] 2024-03-09T14:31:15.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.325+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:_:XlxD2gCwXwS2ua0znqme9OV5QI8oqAA_f state=0x55d212941548 s->prefetch_data=0 2024-03-09T14:31:15.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.325+0000 7f462bce6980 10 manifest: total_size = 12 2024-03-09T14:31:15.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.325+0000 7f462bce6980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.17434083002254113861 2024-03-09T14:31:15.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.325+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:_:XlxD2gCwXwS2ua0znqme9OV5QI8oqAA_f state=0x55d212941548 s->prefetch_data=0 2024-03-09T14:31:15.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.325+0000 7f462bce6980 20 prepare_atomic_modification: state is not atomic. state=0x55d212941548 2024-03-09T14:31:15.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.325+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.333+0000 7f462bce6980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:31:15.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.333+0000 7f462bce6980 20 RGWObjManifest::operator++(): result: ofs=12 stripe_ofs=12 part_ofs=0 rule->part_size=0 2024-03-09T14:31:15.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.333+0000 7f462bce6980 0 deleting objects inline since gc isn't initialized 2024-03-09T14:31:15.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.333+0000 7f462bce6980 5 delete_objs_inline: removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.1__shadow:XlxD2gCwXwS2ua0znqme9OV5QI8oqAA_.ynoqlk6f8UQCPoCabOTHx-gRIcUUe0i_0 2024-03-09T14:31:15.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.333+0000 7f462bce6980 5 delete_objs_inline: refcount put returned error -2 2024-03-09T14:31:15.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.333+0000 7f450cff9640 20 handle_completion(): completion ok for obj=f[XlxD2gCwXwS2ua0znqme9OV5QI8oqAA] 2024-03-09T14:31:15.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.337+0000 7f462bce6980 20 get_obj_state: octx=0x55d212664e30 obj=check-bucket:f state=0x55d21290d4f8 s->prefetch_data=0 2024-03-09T14:31:15.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.337+0000 7f462bce6980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=5tfi3r41f6uxfcf53nmzwjcqqob9idfg 2024-03-09T14:31:15.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.341+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.345+0000 7f462bce6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.1.0 2024-03-09T14:31:15.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.345+0000 7f462bce6980 1 NOTICE: finished shard 0 (4 entries removed) 2024-03-09T14:31:15.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.345+0000 7f462bce6980 1 NOTICE: finished all shards (4 entries removed) 2024-03-09T14:31:15.348 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.345+0000 7f44df7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:15.353 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.353+0000 7f462bce6980 20 remove_watcher() i=0 2024-03-09T14:31:15.353 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.353+0000 7f462bce6980 2 removed watcher, disabling cache 2024-03-09T14:31:15.356 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.353+0000 7f462bce6980 20 remove_watcher() i=1 2024-03-09T14:31:15.359 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.357+0000 7f462bce6980 20 remove_watcher() i=2 2024-03-09T14:31:15.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.361+0000 7f462bce6980 20 remove_watcher() i=3 2024-03-09T14:31:15.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.361+0000 7f462bce6980 20 remove_watcher() i=4 2024-03-09T14:31:15.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.365+0000 7f462bce6980 20 remove_watcher() i=5 2024-03-09T14:31:15.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.365+0000 7f462bce6980 20 remove_watcher() i=6 2024-03-09T14:31:15.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.369+0000 7f462bce6980 20 remove_watcher() i=7 2024-03-09T14:31:15.388 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:15.388 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:15.447 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:15.447 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:15.917 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket check-bucket 2024-03-09T14:31:15.949 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:31:15.949 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:31:15.998 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.993+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.997+0000 7f919d008980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.997+0000 7f919d008980 20 realm 2024-03-09T14:31:15.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.997+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.997+0000 7f919d008980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.997+0000 7f919d008980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:31:15.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.997+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:15.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.997+0000 7f919d008980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:15.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.997+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.000 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.997+0000 7f919d008980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:16.000 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.997+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.000 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.997+0000 7f919d008980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:31:16.000 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:15.997+0000 7f919d008980 20 searching for the correct realm 2024-03-09T14:31:16.006 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.005+0000 7f919a31f640 0 -- 172.21.15.1:0/570151106 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x564c49017e10 msgr2=0x564c4901a220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:31:16.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.013+0000 7f919a31f640 0 -- 172.21.15.1:0/570151106 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f918c0eb470 msgr2=0x7f918c0eb8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:31:16.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:31:16.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:31:16.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:31:16.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:31:16.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:31:16.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:31:16.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:16.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:31:16.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:31:16.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 zone default found 2024-03-09T14:31:16.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 4 Realm: () 2024-03-09T14:31:16.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:31:16.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:31:16.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:31:16.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 zonegroup default 2024-03-09T14:31:16.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:16.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:16.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:16.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 started sync module instance, tier type = 2024-03-09T14:31:16.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.021+0000 7f919d008980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:31:16.041 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.041+0000 7f919d008980 20 add_watcher() i=0 2024-03-09T14:31:16.041 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.041+0000 7f919d008980 20 add_watcher() i=1 2024-03-09T14:31:16.041 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.041+0000 7f919d008980 20 add_watcher() i=2 2024-03-09T14:31:16.041 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.041+0000 7f919d008980 20 add_watcher() i=3 2024-03-09T14:31:16.041 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.041+0000 7f919d008980 20 add_watcher() i=4 2024-03-09T14:31:16.041 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.041+0000 7f919d008980 20 add_watcher() i=5 2024-03-09T14:31:16.041 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.041+0000 7f919d008980 20 add_watcher() i=6 2024-03-09T14:31:16.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.041+0000 7f919d008980 20 add_watcher() i=7 2024-03-09T14:31:16.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.041+0000 7f919d008980 2 all 8 watchers are set, enabling cache 2024-03-09T14:31:16.044 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f90517e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:16.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:31:16.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:31:16.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 5 note: GC not initialized 2024-03-09T14:31:16.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f8fcbfff640 20 reqs_thread_entry: start 2024-03-09T14:31:16.046 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 init_complete bucket index max shards: 1 2024-03-09T14:31:16.046 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 Filter name: none 2024-03-09T14:31:16.046 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f8fc9ffb640 20 reqs_thread_entry: start 2024-03-09T14:31:16.046 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:16.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:16.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:16.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 rados_obj.operate() r=0 bl.length=190 2024-03-09T14:31:16.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 rados_obj.operate() r=0 bl.length=404 2024-03-09T14:31:16.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.045+0000 7f919d008980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:31:16.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.049+0000 7f919d008980 20 rados->read ofs=0 len=0 2024-03-09T14:31:16.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.049+0000 7f919d008980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:31:16.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.049+0000 7f90517e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:31:16.054 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.053+0000 7f919d008980 20 remove_watcher() i=0 2024-03-09T14:31:16.054 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.053+0000 7f919d008980 2 removed watcher, disabling cache 2024-03-09T14:31:16.056 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.057+0000 7f919d008980 20 remove_watcher() i=1 2024-03-09T14:31:16.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.057+0000 7f919d008980 20 remove_watcher() i=2 2024-03-09T14:31:16.061 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.061+0000 7f919d008980 20 remove_watcher() i=3 2024-03-09T14:31:16.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.061+0000 7f919d008980 20 remove_watcher() i=4 2024-03-09T14:31:16.064 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.065+0000 7f919d008980 20 remove_watcher() i=5 2024-03-09T14:31:16.065 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.065+0000 7f919d008980 20 remove_watcher() i=6 2024-03-09T14:31:16.067 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:31:16.065+0000 7f919d008980 20 remove_watcher() i=7 2024-03-09T14:31:16.081 INFO:tasks.workunit.client.0.smithi001.stderr:{'rgw.main': {'size': 0, 'size_actual': 0, 'size_utilized': 0, 'size_kb': 0, 'size_kb_actual': 0, 'size_kb_utilized': 0, 'num_objects': 0}} 2024-03-09T14:31:16.081 INFO:tasks.workunit.client.0.smithi001.stderr:Deleting bucket check-bucket 2024-03-09T14:31:16.082 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:16.082 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:31:16.112 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:31:16.435 INFO:tasks.workunit.client.0.smithi001.stderr:Completed bucket check tests 2024-03-09T14:31:16.435 INFO:tasks.workunit.client.0.smithi001.stdout:Could not connect to the endpoint URL: "http://localhost:80/" 2024-03-09T14:31:16.435 INFO:tasks.workunit.client.0.smithi001.stdout:Could not connect to the endpoint URL: "http://localhost:8000/" 2024-03-09T14:31:16.435 INFO:tasks.workunit.client.0.smithi001.stdout:connected to https://localhost:443 2024-03-09T14:31:16.539 INFO:tasks.workunit.client.0.smithi001.stderr:+ deactivate 2024-03-09T14:31:16.539 INFO:tasks.workunit.client.0.smithi001.stderr:+ '[' -n /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin ']' 2024-03-09T14:31:16.539 INFO:tasks.workunit.client.0.smithi001.stderr:+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-09T14:31:16.539 INFO:tasks.workunit.client.0.smithi001.stderr:+ export PATH 2024-03-09T14:31:16.539 INFO:tasks.workunit.client.0.smithi001.stderr:+ unset _OLD_VIRTUAL_PATH 2024-03-09T14:31:16.539 INFO:tasks.workunit.client.0.smithi001.stderr:+ '[' -n '' ']' 2024-03-09T14:31:16.539 INFO:tasks.workunit.client.0.smithi001.stderr:+ '[' -n /usr/bin/bash -o -n '' ']' 2024-03-09T14:31:16.539 INFO:tasks.workunit.client.0.smithi001.stderr:+ hash -r 2024-03-09T14:31:16.539 INFO:tasks.workunit.client.0.smithi001.stderr:+ '[' -n '' ']' 2024-03-09T14:31:16.539 INFO:tasks.workunit.client.0.smithi001.stderr:+ unset VIRTUAL_ENV 2024-03-09T14:31:16.540 INFO:tasks.workunit.client.0.smithi001.stderr:+ unset VIRTUAL_ENV_PROMPT 2024-03-09T14:31:16.540 INFO:tasks.workunit.client.0.smithi001.stderr:+ '[' '!' '' = nondestructive ']' 2024-03-09T14:31:16.540 INFO:tasks.workunit.client.0.smithi001.stderr:+ unset -f deactivate 2024-03-09T14:31:16.540 INFO:tasks.workunit.client.0.smithi001.stderr:+ echo OK. 2024-03-09T14:31:16.541 INFO:tasks.workunit.client.0.smithi001.stdout:OK. 2024-03-09T14:31:16.541 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:31:16.541 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:31:16.602 INFO:tasks.workunit:Stopping ['rgw/run-bucket-check.sh'] on client.0... 2024-03-09T14:31:16.602 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-09T14:31:17.005 DEBUG:teuthology.parallel:result is None 2024-03-09T14:31:17.006 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:31:17.020 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:31:17.020 DEBUG:teuthology.orchestra.run.smithi001:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-09T14:31:17.077 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:31:17.077 INFO:teuthology.run_tasks:Running task workunit... 2024-03-09T14:31:17.087 INFO:tasks.workunit:Pulling workunits from ref c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:31:17.087 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-09T14:31:17.087 DEBUG:teuthology.orchestra.run.smithi001:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-09T14:31:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:31:17.125 INFO:teuthology.orchestra.run.smithi001.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-09T14:31:17.125 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-09T14:31:17.173 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-09T14:31:17.173 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-09T14:31:17.222 INFO:tasks.workunit:timeout=3h 2024-03-09T14:31:17.223 INFO:tasks.workunit:cleanup=True 2024-03-09T14:31:17.223 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:31:17.271 INFO:tasks.workunit.client.0.smithi001.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-09T14:32:29.712 INFO:tasks.workunit.client.0.smithi001.stderr:Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-09T14:32:30.379 INFO:tasks.workunit.client.0.smithi001.stderr:Note: switching to 'c00d9c609fb42a2a25f1c196bed064ca8bb02413'. 2024-03-09T14:32:30.379 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr:state without impacting any branches by switching back to a branch. 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr: git switch -c 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr:Or undo this operation with: 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr: git switch - 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:32:30.380 INFO:tasks.workunit.client.0.smithi001.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-09T14:32:30.381 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:32:30.381 INFO:tasks.workunit.client.0.smithi001.stderr:HEAD is now at c00d9c609fb DNM: point s3tests at branch cbodley:wip-iam-user-apis 2024-03-09T14:32:30.385 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-09T14:32:30.433 INFO:tasks.workunit.client.0.smithi001.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-09T14:32:30.436 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-09T14:32:30.437 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-09T14:32:30.493 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-09T14:32:30.549 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-09T14:32:30.593 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-09T14:32:30.594 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-09T14:32:30.595 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-09T14:32:30.640 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-09T14:32:30.643 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:32:30.643 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-09T14:32:30.689 INFO:tasks.workunit:Running workunits matching cls/test_cls_lock.sh on client.0... 2024-03-09T14:32:30.690 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2024-03-09T14:32:30.690 DEBUG:teuthology.orchestra.run.smithi001:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2024-03-09T14:32:30.753 INFO:tasks.workunit.client.0.smithi001.stdout:Running main() from gmock_main.cc 2024-03-09T14:32:30.753 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 11 tests from 1 test suite. 2024-03-09T14:32:30.754 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:32:30.754 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 11 tests from ClsLock 2024-03-09T14:32:30.754 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsLock.TestMultiLocking 2024-03-09T14:32:32.143 INFO:tasks.workunit.client.0.smithi001.stdout:lock: mylock 2024-03-09T14:32:32.143 INFO:tasks.workunit.client.0.smithi001.stdout: lock_type: exclusive 2024-03-09T14:32:32.143 INFO:tasks.workunit.client.0.smithi001.stdout: tag: 2024-03-09T14:32:32.143 INFO:tasks.workunit.client.0.smithi001.stdout: lockers: 2024-03-09T14:32:32.143 INFO:tasks.workunit.client.0.smithi001.stdout: client.5175 expiration=2024-03-09T14:34:32.136006+0000 addr=v1:172.21.15.1:0/1097203202 cookie= 2024-03-09T14:32:32.151 INFO:tasks.workunit.client.0.smithi001.stdout:lock: mylock 2024-03-09T14:32:32.151 INFO:tasks.workunit.client.0.smithi001.stdout: lock_type: shared 2024-03-09T14:32:32.151 INFO:tasks.workunit.client.0.smithi001.stdout: tag: 2024-03-09T14:32:32.151 INFO:tasks.workunit.client.0.smithi001.stdout: lockers: 2024-03-09T14:32:32.151 INFO:tasks.workunit.client.0.smithi001.stdout: client.4979 expiration=0.000000 addr=v1:172.21.15.1:0/70309129 cookie= 2024-03-09T14:32:32.151 INFO:tasks.workunit.client.0.smithi001.stdout: client.5175 expiration=2024-03-09T14:34:32.149748+0000 addr=v1:172.21.15.1:0/1097203202 cookie= 2024-03-09T14:32:32.154 INFO:tasks.workunit.client.0.smithi001.stdout:lock: mylock 2024-03-09T14:32:32.154 INFO:tasks.workunit.client.0.smithi001.stdout: lock_type: shared 2024-03-09T14:32:32.154 INFO:tasks.workunit.client.0.smithi001.stdout: tag: 2024-03-09T14:32:32.154 INFO:tasks.workunit.client.0.smithi001.stdout: lockers: 2024-03-09T14:32:32.154 INFO:tasks.workunit.client.0.smithi001.stdout: client.4979 expiration=0.000000 addr=v1:172.21.15.1:0/70309129 cookie= 2024-03-09T14:32:33.082 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsLock.TestMultiLocking (2329 ms) 2024-03-09T14:32:33.082 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsLock.TestMeta 2024-03-09T14:32:35.147 INFO:tasks.workunit.client.0.smithi001.stdout:lock: mylock 2024-03-09T14:32:35.147 INFO:tasks.workunit.client.0.smithi001.stdout: lock_type: shared 2024-03-09T14:32:35.147 INFO:tasks.workunit.client.0.smithi001.stdout: tag: 2024-03-09T14:32:35.147 INFO:tasks.workunit.client.0.smithi001.stdout: lockers: 2024-03-09T14:32:35.147 INFO:tasks.workunit.client.0.smithi001.stdout: client.4988 expiration=0.000000 addr=v1:172.21.15.1:0/4139328329 cookie= 2024-03-09T14:32:35.151 INFO:tasks.workunit.client.0.smithi001.stdout:lock: mylock 2024-03-09T14:32:35.151 INFO:tasks.workunit.client.0.smithi001.stdout: lock_type: exclusive 2024-03-09T14:32:35.151 INFO:tasks.workunit.client.0.smithi001.stdout: tag: new_tag 2024-03-09T14:32:35.151 INFO:tasks.workunit.client.0.smithi001.stdout: lockers: 2024-03-09T14:32:35.151 INFO:tasks.workunit.client.0.smithi001.stdout: client.5187 expiration=0.000000 addr=v1:172.21.15.1:0/350460629 cookie= 2024-03-09T14:32:36.105 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsLock.TestMeta (3022 ms) 2024-03-09T14:32:36.105 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsLock.TestCookie 2024-03-09T14:32:38.159 INFO:tasks.workunit.client.0.smithi001.stdout:lock: mylock 2024-03-09T14:32:38.159 INFO:tasks.workunit.client.0.smithi001.stdout: lock_type: exclusive 2024-03-09T14:32:38.159 INFO:tasks.workunit.client.0.smithi001.stdout: tag: 2024-03-09T14:32:38.159 INFO:tasks.workunit.client.0.smithi001.stdout: lockers: 2024-03-09T14:32:38.167 INFO:tasks.workunit.client.0.smithi001.stdout:lock: mylock 2024-03-09T14:32:38.167 INFO:tasks.workunit.client.0.smithi001.stdout: lock_type: shared 2024-03-09T14:32:38.167 INFO:tasks.workunit.client.0.smithi001.stdout: tag: 2024-03-09T14:32:38.167 INFO:tasks.workunit.client.0.smithi001.stdout: lockers: 2024-03-09T14:32:38.167 INFO:tasks.workunit.client.0.smithi001.stdout: client.4994 expiration=0.000000 addr=v1:172.21.15.1:0/2919089312 cookie= 2024-03-09T14:32:38.167 INFO:tasks.workunit.client.0.smithi001.stdout: client.4994 expiration=0.000000 addr=v1:172.21.15.1:0/2919089312 cookie=new cookie 2024-03-09T14:32:39.125 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsLock.TestCookie (3020 ms) 2024-03-09T14:32:39.125 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsLock.TestMultipleLocks 2024-03-09T14:32:42.137 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsLock.TestMultipleLocks (3010 ms) 2024-03-09T14:32:42.138 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsLock.TestLockDuration 2024-03-09T14:32:50.213 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsLock.TestLockDuration (8079 ms) 2024-03-09T14:32:50.213 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsLock.TestAssertLocked 2024-03-09T14:32:53.225 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsLock.TestAssertLocked (3011 ms) 2024-03-09T14:32:53.225 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsLock.TestSetCookie 2024-03-09T14:32:56.274 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsLock.TestSetCookie (3048 ms) 2024-03-09T14:32:56.274 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsLock.TestRenew 2024-03-09T14:33:17.352 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsLock.TestRenew (21078 ms) 2024-03-09T14:33:17.352 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralBasic 2024-03-09T14:33:24.412 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsLock.TestExclusiveEphemeralBasic (7057 ms) 2024-03-09T14:33:24.412 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealEphemeral 2024-03-09T14:33:32.438 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealEphemeral (8029 ms) 2024-03-09T14:33:32.438 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealExclusive 2024-03-09T14:33:40.512 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealExclusive (8074 ms) 2024-03-09T14:33:40.512 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 11 tests from ClsLock (69758 ms total) 2024-03-09T14:33:40.512 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:33:40.512 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:33:40.512 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 11 tests from 1 test suite ran. (69758 ms total) 2024-03-09T14:33:40.512 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 11 tests. 2024-03-09T14:33:40.519 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:33:40.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:33:40.577 INFO:tasks.workunit:Running workunits matching cls/test_cls_log.sh on client.0... 2024-03-09T14:33:40.578 INFO:tasks.workunit:Running workunit cls/test_cls_log.sh... 2024-03-09T14:33:40.578 DEBUG:teuthology.orchestra.run.smithi001:workunit test cls/test_cls_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_log.sh 2024-03-09T14:33:40.643 INFO:tasks.workunit.client.0.smithi001.stdout:Running main() from gmock_main.cc 2024-03-09T14:33:40.643 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 4 tests from 1 test suite. 2024-03-09T14:33:40.643 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:33:40.643 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 4 tests from cls_log 2024-03-09T14:33:40.643 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_log.test_log_add_same_time 2024-03-09T14:33:43.520 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_log.test_log_add_same_time (2878 ms) 2024-03-09T14:33:43.520 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_log.test_log_add_different_time 2024-03-09T14:33:46.581 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_log.test_log_add_different_time (3061 ms) 2024-03-09T14:33:46.581 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_log.trim_by_time 2024-03-09T14:33:49.622 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_log.trim_by_time (3040 ms) 2024-03-09T14:33:49.622 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_log.trim_by_marker 2024-03-09T14:33:52.645 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_log.trim_by_marker (3024 ms) 2024-03-09T14:33:52.645 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 4 tests from cls_log (12003 ms total) 2024-03-09T14:33:52.645 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:33:52.645 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:33:52.645 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 4 tests from 1 test suite ran. (12003 ms total) 2024-03-09T14:33:52.646 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 4 tests. 2024-03-09T14:33:52.650 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:33:52.650 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:33:52.705 INFO:tasks.workunit:Running workunits matching cls/test_cls_refcount.sh on client.0... 2024-03-09T14:33:52.706 INFO:tasks.workunit:Running workunit cls/test_cls_refcount.sh... 2024-03-09T14:33:52.706 DEBUG:teuthology.orchestra.run.smithi001:workunit test cls/test_cls_refcount.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh 2024-03-09T14:33:52.771 INFO:tasks.workunit.client.0.smithi001.stdout:Running main() from gmock_main.cc 2024-03-09T14:33:52.771 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 10 tests from 1 test suite. 2024-03-09T14:33:52.771 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:33:52.771 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 10 tests from cls_refcount 2024-03-09T14:33:52.772 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_refcount.test_implicit 2024-03-09T14:33:55.704 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_refcount.test_implicit (2933 ms) 2024-03-09T14:33:55.705 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_refcount.test_implicit_idempotent 2024-03-09T14:33:58.736 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_refcount.test_implicit_idempotent (3032 ms) 2024-03-09T14:33:58.736 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_refcount.test_put_snap 2024-03-09T14:34:06.926 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_refcount.test_put_snap (8189 ms) 2024-03-09T14:34:06.926 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_refcount.test_explicit 2024-03-09T14:34:09.932 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_refcount.test_explicit (3006 ms) 2024-03-09T14:34:09.932 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_refcount.set 2024-03-09T14:34:12.963 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_refcount.set (3032 ms) 2024-03-09T14:34:12.963 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_refcount.test_implicit_ec 2024-03-09T14:34:20.007 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_refcount.test_implicit_ec (7044 ms) 2024-03-09T14:34:20.007 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_refcount.test_implicit_idempotent_ec 2024-03-09T14:34:27.061 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_refcount.test_implicit_idempotent_ec (7054 ms) 2024-03-09T14:34:27.061 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_refcount.test_put_snap_ec 2024-03-09T14:34:39.176 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_refcount.test_put_snap_ec (12115 ms) 2024-03-09T14:34:39.176 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_refcount.test_explicit_ec 2024-03-09T14:34:46.256 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_refcount.test_explicit_ec (7079 ms) 2024-03-09T14:34:46.256 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_refcount.set_ec 2024-03-09T14:34:53.305 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_refcount.set_ec (7049 ms) 2024-03-09T14:34:53.305 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 10 tests from cls_refcount (60533 ms total) 2024-03-09T14:34:53.305 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:34:53.305 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:34:53.305 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 10 tests from 1 test suite ran. (60533 ms total) 2024-03-09T14:34:53.305 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 10 tests. 2024-03-09T14:34:53.310 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:34:53.310 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:34:53.366 INFO:tasks.workunit:Running workunits matching cls/test_cls_rgw.sh on client.0... 2024-03-09T14:34:53.367 INFO:tasks.workunit:Running workunit cls/test_cls_rgw.sh... 2024-03-09T14:34:53.368 DEBUG:teuthology.orchestra.run.smithi001:workunit test cls/test_cls_rgw.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh 2024-03-09T14:34:53.431 INFO:tasks.workunit.client.0.smithi001.stdout:Running main() from gmock_main.cc 2024-03-09T14:34:53.431 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 16 tests from 1 test suite. 2024-03-09T14:34:53.431 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:34:53.431 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 16 tests from cls_rgw 2024-03-09T14:34:55.327 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.index_basic 2024-03-09T14:34:55.394 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.index_basic (67 ms) 2024-03-09T14:34:55.395 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.index_multiple_obj_writers 2024-03-09T14:34:55.475 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.index_multiple_obj_writers (81 ms) 2024-03-09T14:34:55.476 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.index_remove_object 2024-03-09T14:34:55.591 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.index_remove_object (116 ms) 2024-03-09T14:34:55.591 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.index_suggest 2024-03-09T14:34:58.096 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.index_suggest (2505 ms) 2024-03-09T14:34:58.096 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.index_suggest_complete 2024-03-09T14:34:58.110 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.index_suggest_complete (14 ms) 2024-03-09T14:34:58.110 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.index_list 2024-03-09T14:34:58.175 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.index_list (65 ms) 2024-03-09T14:34:58.175 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.index_list_delimited 2024-03-09T14:35:50.296 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.index_list_delimited (52120 ms) 2024-03-09T14:35:50.296 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.bi_list 2024-03-09T14:35:50.488 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.bi_list (193 ms) 2024-03-09T14:35:50.488 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.gc_set 2024-03-09T14:35:50.512 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.gc_set (24 ms) 2024-03-09T14:35:50.512 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.gc_list 2024-03-09T14:35:50.536 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.gc_list (24 ms) 2024-03-09T14:35:50.536 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.gc_defer 2024-03-09T14:35:58.515 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.gc_defer (7978 ms) 2024-03-09T14:35:58.515 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.usage_basic 2024-03-09T14:35:58.926 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.usage_basic (411 ms) 2024-03-09T14:35:58.926 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.usage_clear_no_obj 2024-03-09T14:35:58.928 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.usage_clear_no_obj (3 ms) 2024-03-09T14:35:58.928 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.usage_clear 2024-03-09T14:35:59.146 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.usage_clear (218 ms) 2024-03-09T14:35:59.146 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.bi_log_trim 2024-03-09T14:35:59.274 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.bi_log_trim (128 ms) 2024-03-09T14:35:59.274 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw.index_racing_removes 2024-03-09T14:35:59.308 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw.index_racing_removes (34 ms) 2024-03-09T14:35:59.516 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 16 tests from cls_rgw (63981 ms total) 2024-03-09T14:35:59.516 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:35:59.517 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:35:59.517 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 16 tests from 1 test suite ran. (66085 ms total) 2024-03-09T14:35:59.517 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 16 tests. 2024-03-09T14:35:59.525 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:35:59.525 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:35:59.580 INFO:tasks.workunit:Running workunits matching cls/test_cls_rgw_gc.sh on client.0... 2024-03-09T14:35:59.582 INFO:tasks.workunit:Running workunit cls/test_cls_rgw_gc.sh... 2024-03-09T14:35:59.582 DEBUG:teuthology.orchestra.run.smithi001:workunit test cls/test_cls_rgw_gc.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw_gc.sh 2024-03-09T14:35:59.648 INFO:tasks.workunit.client.0.smithi001.stdout:Running main() from gmock_main.cc 2024-03-09T14:35:59.648 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 9 tests from 1 test suite. 2024-03-09T14:35:59.648 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:35:59.648 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 9 tests from cls_rgw_gc 2024-03-09T14:35:59.648 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw_gc.init 2024-03-09T14:36:01.521 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw_gc.init (1873 ms) 2024-03-09T14:36:01.521 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops1 2024-03-09T14:36:01.565 INFO:tasks.workunit.client.0.smithi001.stderr:[ ] list info tag = chain-0 2024-03-09T14:36:01.566 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw_gc.gc_queue_ops1 (44 ms) 2024-03-09T14:36:01.566 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops2 2024-03-09T14:36:01.582 INFO:tasks.workunit.client.0.smithi001.stderr:[ ] next_marker is: = 0/4263 2024-03-09T14:36:01.582 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw_gc.gc_queue_ops2 (18 ms) 2024-03-09T14:36:01.582 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops5 2024-03-09T14:36:01.600 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw_gc.gc_queue_ops5 (17 ms) 2024-03-09T14:36:01.600 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops6 2024-03-09T14:36:01.618 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw_gc.gc_queue_ops6 (18 ms) 2024-03-09T14:36:01.618 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops7 2024-03-09T14:36:01.641 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw_gc.gc_queue_ops7 (23 ms) 2024-03-09T14:36:01.641 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops8 2024-03-09T14:36:01.659 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw_gc.gc_queue_ops8 (18 ms) 2024-03-09T14:36:01.659 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops9 2024-03-09T14:36:01.678 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw_gc.gc_queue_ops9 (19 ms) 2024-03-09T14:36:01.678 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw_gc.finalize 2024-03-09T14:36:02.543 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw_gc.finalize (865 ms) 2024-03-09T14:36:02.544 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 9 tests from cls_rgw_gc (2895 ms total) 2024-03-09T14:36:02.544 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:36:02.544 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:36:02.544 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 9 tests from 1 test suite ran. (2895 ms total) 2024-03-09T14:36:02.544 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 9 tests. 2024-03-09T14:36:02.549 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:36:02.549 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:36:02.605 INFO:tasks.workunit:Running workunits matching cls/test_cls_rgw_stats.sh on client.0... 2024-03-09T14:36:02.606 INFO:tasks.workunit:Running workunit cls/test_cls_rgw_stats.sh... 2024-03-09T14:36:02.606 DEBUG:teuthology.orchestra.run.smithi001:workunit test cls/test_cls_rgw_stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw_stats.sh 2024-03-09T14:36:02.686 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:36:02.686 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:36:02.686 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 1 test from 1 test suite. 2024-03-09T14:36:02.686 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:36:04.571 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 1 test from cls_rgw_stats 2024-03-09T14:36:04.571 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] cls_rgw_stats.simulate 2024-03-09T14:36:04.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.594+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~GRRU0HGUUURO.0 size=651860 2024-03-09T14:36:04.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.594+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=GRRU0HGUUURO tag=CO46CRQRBF8Y type=write size=1700436 2024-03-09T14:36:04.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.598+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=28G45SE6GCH2 type=del size=2209228 2024-03-09T14:36:04.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.602+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=P304JC2UV2IS type=del size=521826 2024-03-09T14:36:04.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.606+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~I7WLCS08AHB1.0 size=1048576 2024-03-09T14:36:04.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.606+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~I7WLCS08AHB1.1 size=548397 2024-03-09T14:36:04.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.606+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=I7WLCS08AHB1 tag=ST82F4IDU10Z type=write size=2645549 2024-03-09T14:36:04.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.610+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~ZXUXG0PE9RU5.0 size=1048576 2024-03-09T14:36:04.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.614+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~ZXUXG0PE9RU5.1 size=657229 2024-03-09T14:36:04.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.614+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=ZXUXG0PE9RU5 tag=5N6FTAXKSLK3 type=write size=2754381 2024-03-09T14:36:04.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.618+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~GWV26K5CX34Y.0 size=370765 2024-03-09T14:36:04.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.618+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=GWV26K5CX34Y tag=MQSIMZVN02X6 type=write size=1419341 2024-03-09T14:36:04.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.618+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=0IT893JMN3XH type=del size=1906401 2024-03-09T14:36:04.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.622+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=77SDZEJ8RWIN type=del size=3056018 2024-03-09T14:36:04.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.626+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~RQA685BVUFFW.0 size=1048576 2024-03-09T14:36:04.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.626+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~RQA685BVUFFW.1 size=1048576 2024-03-09T14:36:04.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.630+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~RQA685BVUFFW.2 size=667462 2024-03-09T14:36:04.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.630+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=RQA685BVUFFW tag=YEU0ITJBHQ84 type=write size=3813190 2024-03-09T14:36:04.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.634+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~DFJJITM0PSFY.0 size=1048576 2024-03-09T14:36:04.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.634+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~DFJJITM0PSFY.1 size=1048576 2024-03-09T14:36:04.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.638+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~DFJJITM0PSFY.2 size=118473 2024-03-09T14:36:04.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.638+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=DFJJITM0PSFY tag=WNHR1DZYAXNO type=write size=3264201 2024-03-09T14:36:04.644 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.642+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~1PY0SVKF9Q45.0 size=1048576 2024-03-09T14:36:04.646 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.642+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~1PY0SVKF9Q45.1 size=359699 2024-03-09T14:36:04.646 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.642+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=1PY0SVKF9Q45 tag=11LPLS29HFQW type=write size=2456851 2024-03-09T14:36:04.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.646+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~CFVRS7H7ODOO.0 size=1048576 2024-03-09T14:36:04.652 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.650+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~CFVRS7H7ODOO.1 size=704922 2024-03-09T14:36:04.652 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.650+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=CFVRS7H7ODOO tag=OYP0UXIL55MB type=write size=2802074 2024-03-09T14:36:04.654 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.650+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=XSR1L4QZL68H type=write size=788145 2024-03-09T14:36:04.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.654+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~7F1IU00C4HXV.0 size=1048576 2024-03-09T14:36:04.660 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.658+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~7F1IU00C4HXV.1 size=46315 2024-03-09T14:36:04.660 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.658+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=7F1IU00C4HXV tag=6O04SC79IUDR type=write size=2143467 2024-03-09T14:36:04.663 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.658+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=UJ8BCET4YMZ4 type=del size=3290415 2024-03-09T14:36:04.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.662+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=FWJ7YYYU3OSP type=del size=3876704 2024-03-09T14:36:04.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.662+0000 7f73315818c0 1 < canceled part key=_multipart_8O5OP8C5NX12.2~GRRU0HGUUURO.0 size=651860 2024-03-09T14:36:04.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.666+0000 7f73315818c0 1 < canceled multipart upload key=8O5OP8C5NX12 upload=GRRU0HGUUURO tag=CO46CRQRBF8Y type=write size=1700436 2024-03-09T14:36:04.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.670+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~LWWADSXCG18D.0 size=506671 2024-03-09T14:36:04.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.670+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=LWWADSXCG18D tag=BOZR5OONFQPI type=write size=1555247 2024-03-09T14:36:04.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.674+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=28G45SE6GCH2 type=del 2024-03-09T14:36:04.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.674+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=ASSE5VR4YDKD type=del size=195037 2024-03-09T14:36:04.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.678+0000 7f73315818c0 1 < canceled operation key=LZC7NPF0GRO7 tag=P304JC2UV2IS type=del size=521826 2024-03-09T14:36:04.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.682+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=HCZSU754ZD5J type=del size=259468 2024-03-09T14:36:04.688 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.686+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~I7WLCS08AHB1.0 size=1048576 2024-03-09T14:36:04.689 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.686+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~I7WLCS08AHB1.1 size=548397 2024-03-09T14:36:04.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.690+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=I7WLCS08AHB1 tag=ST82F4IDU10Z type=write size=2645549 2024-03-09T14:36:04.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.694+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=BQJ03UQ8LKB2 type=del size=2374476 2024-03-09T14:36:04.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.698+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~ZXUXG0PE9RU5.0 size=1048576 2024-03-09T14:36:04.702 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.698+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~ZXUXG0PE9RU5.1 size=657229 2024-03-09T14:36:04.706 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.702+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=ZXUXG0PE9RU5 tag=5N6FTAXKSLK3 type=write size=2754381 2024-03-09T14:36:04.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.706+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=EAJT6O0MZEWS type=del size=2081118 2024-03-09T14:36:04.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.706+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~GWV26K5CX34Y.0 size=370765 2024-03-09T14:36:04.715 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.714+0000 7f73315818c0 1 < completed multipart upload key=0IB00CZ92VV4 upload=GWV26K5CX34Y tag=MQSIMZVN02X6 type=write size=1419341 2024-03-09T14:36:04.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.714+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=ZBVJ5SI0E4TZ type=del size=2118127 2024-03-09T14:36:04.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.718+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=0IT893JMN3XH type=del 2024-03-09T14:36:04.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.722+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=12K8Z49V57PS type=del size=3287801 2024-03-09T14:36:04.726 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.722+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=77SDZEJ8RWIN type=del 2024-03-09T14:36:04.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.726+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~THA4RZ0FFOLP.0 size=1048576 2024-03-09T14:36:04.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.730+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~THA4RZ0FFOLP.1 size=1048576 2024-03-09T14:36:04.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.730+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~THA4RZ0FFOLP.2 size=448373 2024-03-09T14:36:04.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.730+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=THA4RZ0FFOLP tag=D1V3MYN6AYW4 type=write size=3594101 2024-03-09T14:36:04.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.734+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~RQA685BVUFFW.0 size=1048576 2024-03-09T14:36:04.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.734+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~RQA685BVUFFW.1 size=1048576 2024-03-09T14:36:04.741 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.738+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~RQA685BVUFFW.2 size=667462 2024-03-09T14:36:04.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.742+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=RQA685BVUFFW tag=YEU0ITJBHQ84 type=write size=3813190 2024-03-09T14:36:04.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.746+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=CRF1V1AMEJ46 type=del size=1070654 2024-03-09T14:36:04.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.746+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~DFJJITM0PSFY.0 size=1048576 2024-03-09T14:36:04.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.750+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~DFJJITM0PSFY.1 size=1048576 2024-03-09T14:36:04.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.750+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~DFJJITM0PSFY.2 size=118473 2024-03-09T14:36:04.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.754+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=DFJJITM0PSFY tag=WNHR1DZYAXNO type=write size=3264201 2024-03-09T14:36:04.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.758+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~R0THZ9Q1ADGY.0 size=1048576 2024-03-09T14:36:04.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.762+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~R0THZ9Q1ADGY.1 size=1048576 2024-03-09T14:36:04.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.762+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~R0THZ9Q1ADGY.2 size=647026 2024-03-09T14:36:04.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.762+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=R0THZ9Q1ADGY tag=KSE8VLT2WNUI type=write size=3792754 2024-03-09T14:36:04.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.766+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~1PY0SVKF9Q45.0 size=1048576 2024-03-09T14:36:04.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.766+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~1PY0SVKF9Q45.1 size=359699 2024-03-09T14:36:04.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.770+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=1PY0SVKF9Q45 tag=11LPLS29HFQW type=write size=2456851 2024-03-09T14:36:04.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.774+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=OAUCOGO986B2 type=del size=1086689 2024-03-09T14:36:04.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.778+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~CFVRS7H7ODOO.0 size=1048576 2024-03-09T14:36:04.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.778+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~CFVRS7H7ODOO.1 size=704922 2024-03-09T14:36:04.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.782+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=CFVRS7H7ODOO tag=OYP0UXIL55MB type=write size=2802074 2024-03-09T14:36:04.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.782+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=H30IMQUBG5IZ type=write size=229236 2024-03-09T14:36:04.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.786+0000 7f73315818c0 1 < completed write operation key=Y8EQAHMTL66U tag=XSR1L4QZL68H type=write size=788145 2024-03-09T14:36:04.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.790+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=E3W5IIPUT1N8 type=del size=3030637 2024-03-09T14:36:04.794 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.790+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~7F1IU00C4HXV.0 size=1048576 2024-03-09T14:36:04.796 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.794+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~7F1IU00C4HXV.1 size=46315 2024-03-09T14:36:04.800 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.798+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=7F1IU00C4HXV tag=6O04SC79IUDR type=write size=2143467 2024-03-09T14:36:04.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.798+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=XU4RVIMIV9QN type=del size=1631877 2024-03-09T14:36:04.805 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.802+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=UJ8BCET4YMZ4 type=del 2024-03-09T14:36:04.807 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.802+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=ZJ5IHZXZ8Z55 type=del size=3608185 2024-03-09T14:36:04.811 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.810+0000 7f73315818c0 1 < suggested update operation key=Y8EQAHMTL66U tag=FWJ7YYYU3OSP type=del size=3876704 2024-03-09T14:36:04.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.814+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~BTLGE3O1D4YQ.0 size=239877 2024-03-09T14:36:04.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.814+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=BTLGE3O1D4YQ tag=31SW8TIDKSPV type=write size=1288453 2024-03-09T14:36:04.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.814+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~LWWADSXCG18D.0 size=506671 2024-03-09T14:36:04.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.822+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=LWWADSXCG18D tag=BOZR5OONFQPI type=write size=1555247 2024-03-09T14:36:04.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.822+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=50FXEHN5DVI8 type=del size=3877320 2024-03-09T14:36:04.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.826+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=ASSE5VR4YDKD type=del 2024-03-09T14:36:04.835 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.834+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~PMP34NK42E1J.0 size=1048576 2024-03-09T14:36:04.838 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.834+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~PMP34NK42E1J.1 size=507624 2024-03-09T14:36:04.838 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.834+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=PMP34NK42E1J tag=ZUS5BQAPHHF2 type=write size=2604776 2024-03-09T14:36:04.845 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.842+0000 7f73315818c0 1 < suggested remove operation key=K3JNN1NCKBXN tag=HCZSU754ZD5J type=del 2024-03-09T14:36:04.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.846+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=S692YTKBIH2A type=del size=3454803 2024-03-09T14:36:04.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.846+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=BQJ03UQ8LKB2 type=del 2024-03-09T14:36:04.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.854+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~YIV14P1UC6NU.0 size=1048576 2024-03-09T14:36:04.858 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.854+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~YIV14P1UC6NU.1 size=659618 2024-03-09T14:36:04.858 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.854+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=YIV14P1UC6NU tag=WA7O7W3WOID0 type=write size=2756770 2024-03-09T14:36:04.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.858+0000 7f73315818c0 1 < canceled operation key=P8R5RJI8LGZ9 tag=EAJT6O0MZEWS type=del size=2081118 2024-03-09T14:36:04.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.862+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~6GSGYQWA37PM.0 size=1048576 2024-03-09T14:36:04.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.866+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~6GSGYQWA37PM.1 size=466321 2024-03-09T14:36:04.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.866+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=6GSGYQWA37PM tag=YCPEAC5NSX97 type=write size=2563473 2024-03-09T14:36:04.871 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.866+0000 7f73315818c0 1 < canceled operation key=1NYNL8M5DVER tag=ZBVJ5SI0E4TZ type=del size=2118127 2024-03-09T14:36:04.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.870+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=OAR9S3Z158KF type=del size=1557438 2024-03-09T14:36:04.877 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.874+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=12K8Z49V57PS type=del 2024-03-09T14:36:04.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.878+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=N9BWAUYI3VWX type=del size=1189580 2024-03-09T14:36:04.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.878+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~THA4RZ0FFOLP.0 size=1048576 2024-03-09T14:36:04.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.882+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~THA4RZ0FFOLP.1 size=1048576 2024-03-09T14:36:04.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.886+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~THA4RZ0FFOLP.2 size=448373 2024-03-09T14:36:04.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.890+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=THA4RZ0FFOLP tag=D1V3MYN6AYW4 type=write size=3594101 2024-03-09T14:36:04.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.894+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=TUC5SGHF2OU3 type=del size=1102879 2024-03-09T14:36:04.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.894+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=CRF1V1AMEJ46 type=del 2024-03-09T14:36:04.904 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.902+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~B15GSUDXL0ZV.0 size=1048576 2024-03-09T14:36:04.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.902+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~B15GSUDXL0ZV.1 size=1048576 2024-03-09T14:36:04.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.906+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~B15GSUDXL0ZV.2 size=685824 2024-03-09T14:36:04.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.906+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=B15GSUDXL0ZV tag=FR4N2ZJ7O19C type=write size=3831552 2024-03-09T14:36:04.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.910+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~R0THZ9Q1ADGY.0 size=1048576 2024-03-09T14:36:04.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.910+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~R0THZ9Q1ADGY.1 size=1048576 2024-03-09T14:36:04.917 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.914+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~R0THZ9Q1ADGY.2 size=647026 2024-03-09T14:36:04.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.918+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=R0THZ9Q1ADGY tag=KSE8VLT2WNUI type=write size=3792754 2024-03-09T14:36:04.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.922+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=F0MHTPHBER0F type=del size=2989412 2024-03-09T14:36:04.928 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.926+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=OAUCOGO986B2 type=del 2024-03-09T14:36:04.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.930+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=ZUEO092U21SO type=del size=2338072 2024-03-09T14:36:04.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.934+0000 7f73315818c0 1 < completed write operation key=S302JFYEDEL7 tag=H30IMQUBG5IZ type=write size=229236 2024-03-09T14:36:04.939 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.938+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=BE1F2IHOEV7M type=del size=1782793 2024-03-09T14:36:04.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.938+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=E3W5IIPUT1N8 type=del 2024-03-09T14:36:04.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.942+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=MQL404DLJHTE type=del size=1722988 2024-03-09T14:36:04.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.946+0000 7f73315818c0 1 < completed delete operation key=M1B6O6N8N84T tag=XU4RVIMIV9QN type=del 2024-03-09T14:36:04.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.954+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~5OJYNIZJN2Z1.0 size=1048576 2024-03-09T14:36:04.958 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.954+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~5OJYNIZJN2Z1.1 size=1048576 2024-03-09T14:36:04.960 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.958+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~5OJYNIZJN2Z1.2 size=400048 2024-03-09T14:36:04.960 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.958+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=5OJYNIZJN2Z1 tag=Y4DU3RFDUKWF type=write size=3545776 2024-03-09T14:36:04.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.962+0000 7f73315818c0 1 < completed delete operation key=M1B6O6N8N84T tag=ZJ5IHZXZ8Z55 type=del 2024-03-09T14:36:04.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.966+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~S12PEW21IB3L.0 size=1009962 2024-03-09T14:36:04.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.966+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=S12PEW21IB3L tag=UMYIE224EY3A type=write size=2058538 2024-03-09T14:36:04.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.970+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~BTLGE3O1D4YQ.0 size=239877 2024-03-09T14:36:04.978 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.974+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=BTLGE3O1D4YQ tag=31SW8TIDKSPV type=write size=1288453 2024-03-09T14:36:04.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.982+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~520YCFYKMD8B.0 size=1048576 2024-03-09T14:36:04.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.982+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~520YCFYKMD8B.1 size=1048576 2024-03-09T14:36:04.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.986+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~520YCFYKMD8B.2 size=973124 2024-03-09T14:36:04.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.986+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=520YCFYKMD8B tag=1FWXS8YAON56 type=write size=4118852 2024-03-09T14:36:04.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.990+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=50FXEHN5DVI8 type=del 2024-03-09T14:36:04.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.994+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~RIBX3UUQKUW5.0 size=691363 2024-03-09T14:36:04.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.994+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=RIBX3UUQKUW5 tag=6X21TBN0I77H type=write size=1739939 2024-03-09T14:36:05.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:04.998+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~PMP34NK42E1J.0 size=1048576 2024-03-09T14:36:05.005 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.002+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~PMP34NK42E1J.1 size=507624 2024-03-09T14:36:05.010 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.006+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=PMP34NK42E1J tag=ZUS5BQAPHHF2 type=write size=2604776 2024-03-09T14:36:05.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.010+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=7ZH0JMM07I7J type=del size=2096567 2024-03-09T14:36:05.017 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.014+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=S692YTKBIH2A type=del 2024-03-09T14:36:05.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.022+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~1KJC61OZGZZQ.0 size=627056 2024-03-09T14:36:05.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.022+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=1KJC61OZGZZQ tag=19GFD3E3JR3J type=write size=1675632 2024-03-09T14:36:05.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.022+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~YIV14P1UC6NU.0 size=1048576 2024-03-09T14:36:05.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.026+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~YIV14P1UC6NU.1 size=659618 2024-03-09T14:36:05.037 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.034+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=YIV14P1UC6NU tag=WA7O7W3WOID0 type=write size=2756770 2024-03-09T14:36:05.040 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.038+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=L8XIDY6HN0EW type=del size=3186074 2024-03-09T14:36:05.044 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.042+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~6GSGYQWA37PM.0 size=1048576 2024-03-09T14:36:05.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.042+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~6GSGYQWA37PM.1 size=466321 2024-03-09T14:36:05.054 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.050+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=6GSGYQWA37PM tag=YCPEAC5NSX97 type=write size=2563473 2024-03-09T14:36:05.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.054+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=OBXDDJUUE2ZN type=del size=825423 2024-03-09T14:36:05.061 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.058+0000 7f73315818c0 1 < canceled operation key=S302JFYEDEL7 tag=OAR9S3Z158KF type=del size=1557438 2024-03-09T14:36:05.067 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.062+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~4V97EZN1AQK7.0 size=1048576 2024-03-09T14:36:05.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.066+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~4V97EZN1AQK7.1 size=1048576 2024-03-09T14:36:05.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.070+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~4V97EZN1AQK7.2 size=445241 2024-03-09T14:36:05.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.070+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=4V97EZN1AQK7 tag=26UMQV8HYNVO type=write size=3590969 2024-03-09T14:36:05.078 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.074+0000 7f73315818c0 1 < suggested remove operation key=CVAWRZMAX997 tag=N9BWAUYI3VWX type=del 2024-03-09T14:36:05.084 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.082+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~DHAYK4U7X7Z4.0 size=1048576 2024-03-09T14:36:05.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.082+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~DHAYK4U7X7Z4.1 size=1048576 2024-03-09T14:36:05.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.086+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~DHAYK4U7X7Z4.2 size=408015 2024-03-09T14:36:05.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.086+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=DHAYK4U7X7Z4 tag=JTYYEMURGZ4O type=write size=3553743 2024-03-09T14:36:05.092 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.090+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=TUC5SGHF2OU3 type=del 2024-03-09T14:36:05.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.094+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~A47XHVJ18U5G.0 size=1048576 2024-03-09T14:36:05.101 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.098+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~A47XHVJ18U5G.1 size=577666 2024-03-09T14:36:05.101 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.098+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=A47XHVJ18U5G tag=HPPJ232NPF4L type=write size=2674818 2024-03-09T14:36:05.105 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.102+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~B15GSUDXL0ZV.0 size=1048576 2024-03-09T14:36:05.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.102+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~B15GSUDXL0ZV.1 size=1048576 2024-03-09T14:36:05.110 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.106+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~B15GSUDXL0ZV.2 size=685824 2024-03-09T14:36:05.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.114+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=B15GSUDXL0ZV tag=FR4N2ZJ7O19C type=write size=3831552 2024-03-09T14:36:05.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.114+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=8AJM4BQZ0W1N type=write size=480680 2024-03-09T14:36:05.122 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.118+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=F0MHTPHBER0F type=del 2024-03-09T14:36:05.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.126+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~QP4C8UG2PJ2H.0 size=1048576 2024-03-09T14:36:05.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.126+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~QP4C8UG2PJ2H.1 size=587565 2024-03-09T14:36:05.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.126+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=QP4C8UG2PJ2H tag=7ZET6OPVJC43 type=write size=2684717 2024-03-09T14:36:05.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.130+0000 7f73315818c0 1 < completed delete operation key=DQOY1JJ1YQ2W tag=ZUEO092U21SO type=del 2024-03-09T14:36:05.141 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.138+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=GWKDKUL9PKQN type=del size=486004 2024-03-09T14:36:05.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.142+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=BE1F2IHOEV7M type=del 2024-03-09T14:36:05.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.146+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~W0CTQV9L3W37.0 size=597253 2024-03-09T14:36:05.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.146+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=W0CTQV9L3W37 tag=AE1VLZJ8PEX5 type=write size=1645829 2024-03-09T14:36:05.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.154+0000 7f73315818c0 1 < suggested update operation key=3ADG4GWVARC1 tag=MQL404DLJHTE type=del size=1722988 2024-03-09T14:36:05.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.158+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~T1QQ796H9JKF.0 size=1048576 2024-03-09T14:36:05.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.162+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~T1QQ796H9JKF.1 size=11133 2024-03-09T14:36:05.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.162+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=T1QQ796H9JKF tag=HP1EXKPVYXAR type=write size=2108285 2024-03-09T14:36:05.169 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.166+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~5OJYNIZJN2Z1.0 size=1048576 2024-03-09T14:36:05.172 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.170+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~5OJYNIZJN2Z1.1 size=1048576 2024-03-09T14:36:05.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.174+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~5OJYNIZJN2Z1.2 size=400048 2024-03-09T14:36:05.182 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.178+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=5OJYNIZJN2Z1 tag=Y4DU3RFDUKWF type=write size=3545776 2024-03-09T14:36:05.189 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.186+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~PBMDGW46XW03.0 size=1048576 2024-03-09T14:36:05.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.190+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~PBMDGW46XW03.1 size=1048576 2024-03-09T14:36:05.194 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.190+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~PBMDGW46XW03.2 size=296190 2024-03-09T14:36:05.194 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.190+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=PBMDGW46XW03 tag=12LGFLPZOHTE type=write size=3441918 2024-03-09T14:36:05.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.194+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~S12PEW21IB3L.0 size=1009962 2024-03-09T14:36:05.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.202+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=S12PEW21IB3L tag=UMYIE224EY3A type=write size=2058538 2024-03-09T14:36:05.207 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.206+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=BUBYPZHD2UB1 type=del size=1856238 2024-03-09T14:36:05.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.206+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~520YCFYKMD8B.0 size=1048576 2024-03-09T14:36:05.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.210+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~520YCFYKMD8B.1 size=1048576 2024-03-09T14:36:05.217 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.214+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~520YCFYKMD8B.2 size=973124 2024-03-09T14:36:05.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.218+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=520YCFYKMD8B tag=1FWXS8YAON56 type=write size=4118852 2024-03-09T14:36:05.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.222+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=YAYWX68G97OP type=del size=3989243 2024-03-09T14:36:05.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.222+0000 7f73315818c0 1 < canceled part key=_multipart_K3JNN1NCKBXN.2~RIBX3UUQKUW5.0 size=691363 2024-03-09T14:36:05.233 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.230+0000 7f73315818c0 1 < canceled multipart upload key=K3JNN1NCKBXN upload=RIBX3UUQKUW5 tag=6X21TBN0I77H type=write size=1739939 2024-03-09T14:36:05.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.238+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~1Z3YLTH60KMC.0 size=203422 2024-03-09T14:36:05.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.238+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=1Z3YLTH60KMC tag=KIGXBN1QPW6C type=write size=1251998 2024-03-09T14:36:05.243 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.242+0000 7f73315818c0 1 < canceled operation key=8O5OP8C5NX12 tag=7ZH0JMM07I7J type=del size=2096567 2024-03-09T14:36:05.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.246+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~3YMWJ571O25Z.0 size=1048576 2024-03-09T14:36:05.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.250+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~3YMWJ571O25Z.1 size=942661 2024-03-09T14:36:05.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.250+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=3YMWJ571O25Z tag=FPUUI96IHBDF type=write size=3039813 2024-03-09T14:36:05.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.250+0000 7f73315818c0 1 < canceled part key=_multipart_QG4O2L526B8E.2~1KJC61OZGZZQ.0 size=627056 2024-03-09T14:36:05.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.254+0000 7f73315818c0 1 < canceled multipart upload key=QG4O2L526B8E upload=1KJC61OZGZZQ tag=19GFD3E3JR3J type=write size=1675632 2024-03-09T14:36:05.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.258+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=G0XHE7DG17KJ type=del size=268324 2024-03-09T14:36:05.267 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.262+0000 7f73315818c0 1 < canceled operation key=0WGGMHTEF2LB tag=L8XIDY6HN0EW type=del size=3186074 2024-03-09T14:36:05.270 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.266+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=02RFPEEZAAE4 type=write size=337199 2024-03-09T14:36:05.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.270+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=OBXDDJUUE2ZN type=del 2024-03-09T14:36:05.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.274+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=RDMDMHZFUHRN type=del size=2244598 2024-03-09T14:36:05.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.278+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~4V97EZN1AQK7.0 size=1048576 2024-03-09T14:36:05.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.282+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~4V97EZN1AQK7.1 size=1048576 2024-03-09T14:36:05.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.282+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~4V97EZN1AQK7.2 size=445241 2024-03-09T14:36:05.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.290+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=4V97EZN1AQK7 tag=26UMQV8HYNVO type=write size=3590969 2024-03-09T14:36:05.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.294+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~16C1Q3379U89.0 size=440250 2024-03-09T14:36:05.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.294+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=16C1Q3379U89 tag=21UXHRORYZGX type=write size=1488826 2024-03-09T14:36:05.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.298+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~DHAYK4U7X7Z4.0 size=1048576 2024-03-09T14:36:05.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.302+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~DHAYK4U7X7Z4.1 size=1048576 2024-03-09T14:36:05.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.302+0000 7f73315818c0 1 < canceled part key=_multipart_W6KVP8EQFQ8J.2~DHAYK4U7X7Z4.2 size=408015 2024-03-09T14:36:05.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.306+0000 7f73315818c0 1 < canceled multipart upload key=W6KVP8EQFQ8J upload=DHAYK4U7X7Z4 tag=JTYYEMURGZ4O type=write size=3553743 2024-03-09T14:36:05.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.310+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=OUZXAL9GC4YH type=del size=2252790 2024-03-09T14:36:05.316 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.314+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~A47XHVJ18U5G.0 size=1048576 2024-03-09T14:36:05.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.314+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~A47XHVJ18U5G.1 size=577666 2024-03-09T14:36:05.323 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.318+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=A47XHVJ18U5G tag=HPPJ232NPF4L type=write size=2674818 2024-03-09T14:36:05.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.322+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=RVHXRYVR0EXB type=del size=2321180 2024-03-09T14:36:05.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.326+0000 7f73315818c0 1 < canceled operation key=92GWY5B0K4SX tag=8AJM4BQZ0W1N type=write size=480680 2024-03-09T14:36:05.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.330+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=KH4YOR4KRK00 type=del size=3058688 2024-03-09T14:36:05.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.334+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~QP4C8UG2PJ2H.0 size=1048576 2024-03-09T14:36:05.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.334+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~QP4C8UG2PJ2H.1 size=587565 2024-03-09T14:36:05.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.342+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=QP4C8UG2PJ2H tag=7ZET6OPVJC43 type=write size=2684717 2024-03-09T14:36:05.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.346+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=OY3EGLUZAOX2 type=del size=560253 2024-03-09T14:36:05.351 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.346+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=GWKDKUL9PKQN type=del 2024-03-09T14:36:05.357 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.354+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~EBOVI8D8W0Y9.0 size=1048576 2024-03-09T14:36:05.359 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.358+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~EBOVI8D8W0Y9.1 size=1048576 2024-03-09T14:36:05.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.358+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~EBOVI8D8W0Y9.2 size=243352 2024-03-09T14:36:05.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.358+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=EBOVI8D8W0Y9 tag=5TLRAXGZAJH0 type=write size=3389080 2024-03-09T14:36:05.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.362+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~W0CTQV9L3W37.0 size=597253 2024-03-09T14:36:05.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.366+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=W0CTQV9L3W37 tag=AE1VLZJ8PEX5 type=write size=1645829 2024-03-09T14:36:05.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.370+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=29WMYXL15CEY type=write size=379550 2024-03-09T14:36:05.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.374+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~T1QQ796H9JKF.0 size=1048576 2024-03-09T14:36:05.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.374+0000 7f73315818c0 1 < canceled part key=_multipart_8O5OP8C5NX12.2~T1QQ796H9JKF.1 size=11133 2024-03-09T14:36:05.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.382+0000 7f73315818c0 1 < canceled multipart upload key=8O5OP8C5NX12 upload=T1QQ796H9JKF tag=HP1EXKPVYXAR type=write size=2108285 2024-03-09T14:36:05.390 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.386+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~Q0QJRS1LT3KR.0 size=963615 2024-03-09T14:36:05.390 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.386+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=Q0QJRS1LT3KR tag=PBAQWXZZ8TBH type=write size=2012191 2024-03-09T14:36:05.394 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.390+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~PBMDGW46XW03.0 size=1048576 2024-03-09T14:36:05.396 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.394+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~PBMDGW46XW03.1 size=1048576 2024-03-09T14:36:05.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.394+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~PBMDGW46XW03.2 size=296190 2024-03-09T14:36:05.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.402+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=PBMDGW46XW03 tag=12LGFLPZOHTE type=write size=3441918 2024-03-09T14:36:05.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.406+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=RJ230YFNM2Q9 type=del size=1307958 2024-03-09T14:36:05.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.410+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=BUBYPZHD2UB1 type=del 2024-03-09T14:36:05.416 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.414+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=LY63A37IPWH2 type=del size=660848 2024-03-09T14:36:05.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.414+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=YAYWX68G97OP type=del 2024-03-09T14:36:05.423 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.418+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=NNFM4GUVXU5K type=del size=3629725 2024-03-09T14:36:05.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.422+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~1Z3YLTH60KMC.0 size=203422 2024-03-09T14:36:05.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.430+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=1Z3YLTH60KMC tag=KIGXBN1QPW6C type=write size=1251998 2024-03-09T14:36:05.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.434+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=EG6UFJ226K6N type=del size=596212 2024-03-09T14:36:05.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.438+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~3YMWJ571O25Z.0 size=1048576 2024-03-09T14:36:05.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.438+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~3YMWJ571O25Z.1 size=942661 2024-03-09T14:36:05.449 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.446+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=3YMWJ571O25Z tag=FPUUI96IHBDF type=write size=3039813 2024-03-09T14:36:05.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.450+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=FQVH6FUBEKGN type=del size=2876517 2024-03-09T14:36:05.459 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.458+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=G0XHE7DG17KJ type=del 2024-03-09T14:36:05.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.462+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~XWHIUBBA2KOG.0 size=1048576 2024-03-09T14:36:05.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.466+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~XWHIUBBA2KOG.1 size=199094 2024-03-09T14:36:05.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.466+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=XWHIUBBA2KOG tag=QEGF30CDXAZ8 type=write size=2296246 2024-03-09T14:36:05.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.470+0000 7f73315818c0 1 < completed write operation key=S302JFYEDEL7 tag=02RFPEEZAAE4 type=write size=337199 2024-03-09T14:36:05.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.474+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=U7GGSDK0CZ1X type=del size=3655801 2024-03-09T14:36:05.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.474+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=RDMDMHZFUHRN type=del 2024-03-09T14:36:05.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.478+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=0CNOK3KBSUBF type=del size=1230528 2024-03-09T14:36:05.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.482+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~16C1Q3379U89.0 size=440250 2024-03-09T14:36:05.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.490+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=16C1Q3379U89 tag=21UXHRORYZGX type=write size=1488826 2024-03-09T14:36:05.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.494+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=9UD5X8DJO3N2 type=del size=3182081 2024-03-09T14:36:05.499 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.494+0000 7f73315818c0 1 < canceled operation key=P9LI5ESR3N9G tag=OUZXAL9GC4YH type=del size=2252790 2024-03-09T14:36:05.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.498+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=914ZDYEFKTVP type=write size=933957 2024-03-09T14:36:05.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.506+0000 7f73315818c0 1 < suggested update operation key=QG4O2L526B8E tag=RVHXRYVR0EXB type=del size=2321180 2024-03-09T14:36:05.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.514+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~6Q40REEH6DS2.0 size=1048576 2024-03-09T14:36:05.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.514+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~6Q40REEH6DS2.1 size=1048576 2024-03-09T14:36:05.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.518+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~6Q40REEH6DS2.2 size=584616 2024-03-09T14:36:05.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.518+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=6Q40REEH6DS2 tag=AX06XNONN4VQ type=write size=3730344 2024-03-09T14:36:05.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.522+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=KH4YOR4KRK00 type=del 2024-03-09T14:36:05.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.526+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~TXDANX4MJ5PR.0 size=576144 2024-03-09T14:36:05.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.526+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=TXDANX4MJ5PR tag=8Z9K7SLR442I type=write size=1624720 2024-03-09T14:36:05.535 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.530+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=OY3EGLUZAOX2 type=del 2024-03-09T14:36:05.538 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.534+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=ED8E4K4529EO type=del size=3833397 2024-03-09T14:36:05.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.538+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~EBOVI8D8W0Y9.0 size=1048576 2024-03-09T14:36:05.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.538+0000 7f73315818c0 1 < canceled part key=_multipart_K3JNN1NCKBXN.2~EBOVI8D8W0Y9.1 size=1048576 2024-03-09T14:36:05.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.542+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~EBOVI8D8W0Y9.2 size=243352 2024-03-09T14:36:05.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.546+0000 7f73315818c0 1 < canceled multipart upload key=K3JNN1NCKBXN upload=EBOVI8D8W0Y9 tag=5TLRAXGZAJH0 type=write size=3389080 2024-03-09T14:36:05.558 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.554+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~FWCG8OE9DGQS.0 size=1048576 2024-03-09T14:36:05.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.558+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~FWCG8OE9DGQS.1 size=1048576 2024-03-09T14:36:05.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.578+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~FWCG8OE9DGQS.2 size=162526 2024-03-09T14:36:05.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.578+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=FWCG8OE9DGQS tag=FVCG9N1RYIRX type=write size=3308254 2024-03-09T14:36:05.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.582+0000 7f73315818c0 1 < completed write operation key=XELG1AWATMVL tag=29WMYXL15CEY type=write size=379550 2024-03-09T14:36:05.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.582+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=HTNLT8RHIDQL type=del size=79362 2024-03-09T14:36:05.590 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.586+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~Q0QJRS1LT3KR.0 size=963615 2024-03-09T14:36:05.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.594+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=Q0QJRS1LT3KR tag=PBAQWXZZ8TBH type=write size=2012191 2024-03-09T14:36:05.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.598+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~YWSF1RVESR47.0 size=1048576 2024-03-09T14:36:05.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.602+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~YWSF1RVESR47.1 size=171612 2024-03-09T14:36:05.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.602+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=YWSF1RVESR47 tag=LQP8WG501W81 type=write size=2268764 2024-03-09T14:36:05.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.606+0000 7f73315818c0 1 < completed delete operation key=M1B6O6N8N84T tag=RJ230YFNM2Q9 type=del 2024-03-09T14:36:05.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.610+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~CU1K6N6WUG7K.0 size=522720 2024-03-09T14:36:05.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.610+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=CU1K6N6WUG7K tag=XPO3O7L21Y6Q type=write size=1571296 2024-03-09T14:36:05.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.614+0000 7f73315818c0 1 < canceled operation key=0IB00CZ92VV4 tag=LY63A37IPWH2 type=del size=660848 2024-03-09T14:36:05.618 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.614+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=WMTC2SPY3PFC type=del size=3772287 2024-03-09T14:36:05.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.618+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=NNFM4GUVXU5K type=del 2024-03-09T14:36:05.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.622+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~LBM3Y02R69D6.0 size=1048576 2024-03-09T14:36:05.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.626+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~LBM3Y02R69D6.1 size=1048576 2024-03-09T14:36:05.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.626+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~LBM3Y02R69D6.2 size=400322 2024-03-09T14:36:05.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.626+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=LBM3Y02R69D6 tag=0X64386RRQ3H type=write size=3546050 2024-03-09T14:36:05.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.630+0000 7f73315818c0 -1 < failed to complete operation key=QG4O2L526B8E tag=EG6UFJ226K6N type=del size=596212: (22) Invalid argument 2024-03-09T14:36:05.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.630+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=MZ3D3BDNIWBJ type=del size=2861236 2024-03-09T14:36:05.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.634+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=FQVH6FUBEKGN type=del 2024-03-09T14:36:05.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.638+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~N04E5OUI7VR0.0 size=332963 2024-03-09T14:36:05.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.638+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=N04E5OUI7VR0 tag=2NYIORMVCQ9R type=write size=1381539 2024-03-09T14:36:05.646 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.642+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~XWHIUBBA2KOG.0 size=1048576 2024-03-09T14:36:05.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.646+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~XWHIUBBA2KOG.1 size=199094 2024-03-09T14:36:05.653 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.650+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=XWHIUBBA2KOG tag=QEGF30CDXAZ8 type=write size=2296246 2024-03-09T14:36:05.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.654+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~AR7UCP8NBFDC.0 size=1048576 2024-03-09T14:36:05.660 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.658+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~AR7UCP8NBFDC.1 size=1020298 2024-03-09T14:36:05.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.658+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=AR7UCP8NBFDC tag=J62F7XSZQ1VF type=write size=3117450 2024-03-09T14:36:05.663 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.658+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=U7GGSDK0CZ1X type=del 2024-03-09T14:36:05.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.662+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=EYN44XBJAS1I type=del size=2407092 2024-03-09T14:36:05.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.666+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=0CNOK3KBSUBF type=del 2024-03-09T14:36:05.673 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.670+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~5S5I1XPH3NXK.0 size=40843 2024-03-09T14:36:05.673 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.670+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=5S5I1XPH3NXK tag=A9SMDOI9SVT3 type=write size=1089419 2024-03-09T14:36:05.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.674+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=9UD5X8DJO3N2 type=del 2024-03-09T14:36:05.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.674+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=J6GHJAB9M4LZ type=del size=2291727 2024-03-09T14:36:05.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.682+0000 7f73315818c0 1 < suggested remove operation key=S302JFYEDEL7 tag=914ZDYEFKTVP type=write 2024-03-09T14:36:05.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.682+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=I20IAXRPRKJ4 type=del size=4042791 2024-03-09T14:36:05.690 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.686+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~6Q40REEH6DS2.0 size=1048576 2024-03-09T14:36:05.692 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.690+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~6Q40REEH6DS2.1 size=1048576 2024-03-09T14:36:05.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.690+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~6Q40REEH6DS2.2 size=584616 2024-03-09T14:36:05.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.694+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=6Q40REEH6DS2 tag=AX06XNONN4VQ type=write size=3730344 2024-03-09T14:36:05.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.698+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=ZXQCOCBSBEB1 type=del size=592850 2024-03-09T14:36:05.703 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.698+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~TXDANX4MJ5PR.0 size=576144 2024-03-09T14:36:05.707 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.702+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=TXDANX4MJ5PR tag=8Z9K7SLR442I type=write size=1624720 2024-03-09T14:36:05.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.706+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=KFSC8ZT0C0NK type=write size=388202 2024-03-09T14:36:05.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.710+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=ED8E4K4529EO type=del 2024-03-09T14:36:05.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.714+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~CR8GJZC03OUX.0 size=766758 2024-03-09T14:36:05.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.714+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=CR8GJZC03OUX tag=VOUG2AJAH7XB type=write size=1815334 2024-03-09T14:36:05.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.714+0000 7f73315818c0 1 < canceled part key=_multipart_OU0730HGI176.2~FWCG8OE9DGQS.0 size=1048576 2024-03-09T14:36:05.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.714+0000 7f73315818c0 1 < canceled part key=_multipart_OU0730HGI176.2~FWCG8OE9DGQS.1 size=1048576 2024-03-09T14:36:05.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.718+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~FWCG8OE9DGQS.2 size=162526 2024-03-09T14:36:05.724 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.722+0000 7f73315818c0 1 < canceled multipart upload key=OU0730HGI176 upload=FWCG8OE9DGQS tag=FVCG9N1RYIRX type=write size=3308254 2024-03-09T14:36:05.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.726+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~UHRAG6K0IWZ7.0 size=1048576 2024-03-09T14:36:05.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.730+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~UHRAG6K0IWZ7.1 size=817290 2024-03-09T14:36:05.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.730+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=UHRAG6K0IWZ7 tag=ROW58178H04M type=write size=2914442 2024-03-09T14:36:05.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.730+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=HTNLT8RHIDQL type=del 2024-03-09T14:36:05.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.734+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=7FCK39I40BTO type=del size=802849 2024-03-09T14:36:05.741 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.738+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~YWSF1RVESR47.0 size=1048576 2024-03-09T14:36:05.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.742+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~YWSF1RVESR47.1 size=171612 2024-03-09T14:36:05.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.746+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=YWSF1RVESR47 tag=LQP8WG501W81 type=write size=2268764 2024-03-09T14:36:05.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.750+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=2SD17QUPNQHO type=del size=455584 2024-03-09T14:36:05.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.750+0000 7f73315818c0 1 < canceled part key=_multipart_OU0730HGI176.2~CU1K6N6WUG7K.0 size=522720 2024-03-09T14:36:05.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.758+0000 7f73315818c0 1 < canceled multipart upload key=OU0730HGI176 upload=CU1K6N6WUG7K tag=XPO3O7L21Y6Q type=write size=1571296 2024-03-09T14:36:05.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.758+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=WH4163UTWW8V type=del size=1328707 2024-03-09T14:36:05.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.762+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=WMTC2SPY3PFC type=del 2024-03-09T14:36:05.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.766+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=5AEPYCTLD5UQ type=del size=2212651 2024-03-09T14:36:05.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.770+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~LBM3Y02R69D6.0 size=1048576 2024-03-09T14:36:05.775 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.770+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~LBM3Y02R69D6.1 size=1048576 2024-03-09T14:36:05.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.774+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~LBM3Y02R69D6.2 size=400322 2024-03-09T14:36:05.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.782+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=LBM3Y02R69D6 tag=0X64386RRQ3H type=write size=3546050 2024-03-09T14:36:05.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.786+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~H3TH7WMNVPYC.0 size=1048576 2024-03-09T14:36:05.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.790+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~H3TH7WMNVPYC.1 size=694326 2024-03-09T14:36:05.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.790+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=H3TH7WMNVPYC tag=P1P9X7EDE1DH type=write size=2791478 2024-03-09T14:36:05.796 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.794+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=MZ3D3BDNIWBJ type=del 2024-03-09T14:36:05.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.798+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~G8Q1NQWGNY9X.0 size=1048576 2024-03-09T14:36:05.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.802+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~G8Q1NQWGNY9X.1 size=157859 2024-03-09T14:36:05.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.802+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=G8Q1NQWGNY9X tag=GOF4D3FB88MY type=write size=2255011 2024-03-09T14:36:05.808 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.806+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~N04E5OUI7VR0.0 size=332963 2024-03-09T14:36:05.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.810+0000 7f73315818c0 1 < completed multipart upload key=IA0AB7UVLFEJ upload=N04E5OUI7VR0 tag=2NYIORMVCQ9R type=write size=1381539 2024-03-09T14:36:05.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.814+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=B62NYXNM0ALS type=del size=1196674 2024-03-09T14:36:05.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.818+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~AR7UCP8NBFDC.0 size=1048576 2024-03-09T14:36:05.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.818+0000 7f73315818c0 1 < canceled part key=_multipart_3ADG4GWVARC1.2~AR7UCP8NBFDC.1 size=1020298 2024-03-09T14:36:05.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.822+0000 7f73315818c0 1 < canceled multipart upload key=3ADG4GWVARC1 upload=AR7UCP8NBFDC tag=J62F7XSZQ1VF type=write size=3117450 2024-03-09T14:36:05.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.826+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~2XTLSQKUQWG5.0 size=1048576 2024-03-09T14:36:05.833 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.830+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~2XTLSQKUQWG5.1 size=1048576 2024-03-09T14:36:05.836 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.834+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~2XTLSQKUQWG5.2 size=150876 2024-03-09T14:36:05.836 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.834+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=2XTLSQKUQWG5 tag=48B28UJH32CM type=write size=3296604 2024-03-09T14:36:05.839 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.838+0000 7f73315818c0 1 < canceled operation key=0WGGMHTEF2LB tag=EYN44XBJAS1I type=del size=2407092 2024-03-09T14:36:05.844 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.842+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=DFVWRBMV8JPR type=del size=3730363 2024-03-09T14:36:05.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.846+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~5S5I1XPH3NXK.0 size=40843 2024-03-09T14:36:05.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.850+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=5S5I1XPH3NXK tag=A9SMDOI9SVT3 type=write size=1089419 2024-03-09T14:36:05.860 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.858+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~5FXLQC7SWGH9.0 size=1048576 2024-03-09T14:36:05.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.858+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~5FXLQC7SWGH9.1 size=146359 2024-03-09T14:36:05.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.858+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=5FXLQC7SWGH9 tag=N3AHPZOH508F type=write size=2243511 2024-03-09T14:36:05.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.862+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=J6GHJAB9M4LZ type=del 2024-03-09T14:36:05.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.870+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~2TTOBBBK73ZW.0 size=1048576 2024-03-09T14:36:05.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.870+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~2TTOBBBK73ZW.1 size=1048576 2024-03-09T14:36:05.877 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.874+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~2TTOBBBK73ZW.2 size=312886 2024-03-09T14:36:05.877 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.874+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=2TTOBBBK73ZW tag=Y73QXO7N1RJY type=write size=3458614 2024-03-09T14:36:05.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.878+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=I20IAXRPRKJ4 type=del 2024-03-09T14:36:05.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.882+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=OC87ETU15NFN type=del size=1577366 2024-03-09T14:36:05.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.886+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=ZXQCOCBSBEB1 type=del 2024-03-09T14:36:05.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.890+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=7M75Z9TUD9YT type=del size=53577 2024-03-09T14:36:05.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.894+0000 7f73315818c0 1 < suggested remove operation key=3ADG4GWVARC1 tag=KFSC8ZT0C0NK type=write 2024-03-09T14:36:05.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.898+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=W1NWXCUS5L1L type=write size=479582 2024-03-09T14:36:05.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.902+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~CR8GJZC03OUX.0 size=766758 2024-03-09T14:36:05.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.910+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=CR8GJZC03OUX tag=VOUG2AJAH7XB type=write size=1815334 2024-03-09T14:36:05.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.918+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~RGBXQQSA7CKO.0 size=1048576 2024-03-09T14:36:05.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.918+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~RGBXQQSA7CKO.1 size=453090 2024-03-09T14:36:05.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.918+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=RGBXQQSA7CKO tag=SLD0XW0UAF4F type=write size=2550242 2024-03-09T14:36:05.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.922+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~UHRAG6K0IWZ7.0 size=1048576 2024-03-09T14:36:05.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.926+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~UHRAG6K0IWZ7.1 size=817290 2024-03-09T14:36:05.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.934+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=UHRAG6K0IWZ7 tag=ROW58178H04M type=write size=2914442 2024-03-09T14:36:05.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.938+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~4NMQ27YNGRQX.0 size=1048576 2024-03-09T14:36:05.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.942+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~4NMQ27YNGRQX.1 size=943317 2024-03-09T14:36:05.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.942+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=4NMQ27YNGRQX tag=2YN2BAXBS3WX type=write size=3040469 2024-03-09T14:36:05.950 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.946+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=7FCK39I40BTO type=del 2024-03-09T14:36:05.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.950+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=3ZP7ORIY202A type=write size=168316 2024-03-09T14:36:05.960 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.958+0000 7f73315818c0 1 < suggested update operation key=DQOY1JJ1YQ2W tag=2SD17QUPNQHO type=del size=455584 2024-03-09T14:36:05.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.962+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~R191R1TTJGSV.0 size=1041997 2024-03-09T14:36:05.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.962+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=R191R1TTJGSV tag=SIF9WDXZE3JM type=write size=2090573 2024-03-09T14:36:05.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.966+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=WH4163UTWW8V type=del 2024-03-09T14:36:05.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.970+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=LUEG2BFIBH4C type=del size=505159 2024-03-09T14:36:05.980 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.978+0000 7f73315818c0 1 < suggested remove operation key=4P89277EEYJ3 tag=5AEPYCTLD5UQ type=del 2024-03-09T14:36:05.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.982+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~5MWTOHS54CF2.0 size=1048576 2024-03-09T14:36:05.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.986+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~5MWTOHS54CF2.1 size=1048576 2024-03-09T14:36:05.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.990+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~5MWTOHS54CF2.2 size=251026 2024-03-09T14:36:05.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.990+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=5MWTOHS54CF2 tag=S7LR9F4ZJU3F type=write size=3396754 2024-03-09T14:36:05.995 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.990+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~H3TH7WMNVPYC.0 size=1048576 2024-03-09T14:36:05.997 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:05.994+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~H3TH7WMNVPYC.1 size=694326 2024-03-09T14:36:06.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.002+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=H3TH7WMNVPYC tag=P1P9X7EDE1DH type=write size=2791478 2024-03-09T14:36:06.010 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.006+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~9NEZ8UOFBNBA.0 size=1048576 2024-03-09T14:36:06.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.010+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~9NEZ8UOFBNBA.1 size=1048576 2024-03-09T14:36:06.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.010+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~9NEZ8UOFBNBA.2 size=377792 2024-03-09T14:36:06.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.010+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=9NEZ8UOFBNBA tag=M1HMFYNJVMVF type=write size=3523520 2024-03-09T14:36:06.018 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.014+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~G8Q1NQWGNY9X.0 size=1048576 2024-03-09T14:36:06.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.018+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~G8Q1NQWGNY9X.1 size=157859 2024-03-09T14:36:06.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.022+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=G8Q1NQWGNY9X tag=GOF4D3FB88MY type=write size=2255011 2024-03-09T14:36:06.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.026+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=22K61RJS1J0T type=write size=562391 2024-03-09T14:36:06.036 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.034+0000 7f73315818c0 1 < suggested update operation key=0LP8KKU53X0A tag=B62NYXNM0ALS type=del size=1196674 2024-03-09T14:36:06.039 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.034+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=MTMYP6Q7K445 type=del size=2153130 2024-03-09T14:36:06.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.038+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~2XTLSQKUQWG5.0 size=1048576 2024-03-09T14:36:06.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.042+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~2XTLSQKUQWG5.1 size=1048576 2024-03-09T14:36:06.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.046+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~2XTLSQKUQWG5.2 size=150876 2024-03-09T14:36:06.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.050+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=2XTLSQKUQWG5 tag=48B28UJH32CM type=write size=3296604 2024-03-09T14:36:06.056 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.054+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=1X8AE566B7OQ type=del size=1256764 2024-03-09T14:36:06.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.058+0000 7f73315818c0 1 < suggested update operation key=M1B6O6N8N84T tag=DFVWRBMV8JPR type=del size=3730363 2024-03-09T14:36:06.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.062+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=97L1VDAJ62KC type=write size=311040 2024-03-09T14:36:06.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.066+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~5FXLQC7SWGH9.0 size=1048576 2024-03-09T14:36:06.075 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.070+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~5FXLQC7SWGH9.1 size=146359 2024-03-09T14:36:06.080 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.078+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=5FXLQC7SWGH9 tag=N3AHPZOH508F type=write size=2243511 2024-03-09T14:36:06.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.086+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~39MLTDOX0ZGZ.0 size=1048576 2024-03-09T14:36:06.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.086+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~39MLTDOX0ZGZ.1 size=742629 2024-03-09T14:36:06.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.086+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=39MLTDOX0ZGZ tag=M3XJDRB517J5 type=write size=2839781 2024-03-09T14:36:06.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.086+0000 7f73315818c0 1 < canceled part key=_multipart_PAJYGD2GXHDT.2~2TTOBBBK73ZW.0 size=1048576 2024-03-09T14:36:06.094 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.090+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~2TTOBBBK73ZW.1 size=1048576 2024-03-09T14:36:06.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.094+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~2TTOBBBK73ZW.2 size=312886 2024-03-09T14:36:06.102 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.098+0000 7f73315818c0 1 < canceled multipart upload key=PAJYGD2GXHDT upload=2TTOBBBK73ZW tag=Y73QXO7N1RJY type=write size=3458614 2024-03-09T14:36:06.108 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.106+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~KZJO5DHMIFTG.0 size=1048576 2024-03-09T14:36:06.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.106+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~KZJO5DHMIFTG.1 size=1048576 2024-03-09T14:36:06.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.110+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~KZJO5DHMIFTG.2 size=159891 2024-03-09T14:36:06.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.110+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=KZJO5DHMIFTG tag=CJJ89QT74RZQ type=write size=3305619 2024-03-09T14:36:06.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.114+0000 7f73315818c0 1 < canceled operation key=162AOMHO2BQL tag=OC87ETU15NFN type=del size=1577366 2024-03-09T14:36:06.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.122+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~JX6MDD0LYA7D.0 size=1048576 2024-03-09T14:36:06.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.126+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~JX6MDD0LYA7D.1 size=334847 2024-03-09T14:36:06.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.126+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=JX6MDD0LYA7D tag=J5ERPS1DQJSW type=write size=2431999 2024-03-09T14:36:06.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.126+0000 7f73315818c0 -1 < no bi entry to suggest for operation key=4P89277EEYJ3 tag=7M75Z9TUD9YT type=del size=53577: (2) No such file or directory 2024-03-09T14:36:06.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.130+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=T23OLD8KTUO8 type=write size=100176 2024-03-09T14:36:06.135 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.130+0000 7f73315818c0 1 < completed write operation key=92GWY5B0K4SX tag=W1NWXCUS5L1L type=write size=479582 2024-03-09T14:36:06.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.134+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=9DTY3GYR4725 type=del size=4123999 2024-03-09T14:36:06.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.134+0000 7f73315818c0 1 < canceled part key=_multipart_3ADG4GWVARC1.2~RGBXQQSA7CKO.0 size=1048576 2024-03-09T14:36:06.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.134+0000 7f73315818c0 1 < canceled part key=_multipart_3ADG4GWVARC1.2~RGBXQQSA7CKO.1 size=453090 2024-03-09T14:36:06.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.142+0000 7f73315818c0 1 < canceled multipart upload key=3ADG4GWVARC1 upload=RGBXQQSA7CKO tag=SLD0XW0UAF4F type=write size=2550242 2024-03-09T14:36:06.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.146+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~1J1R10I0T5SS.0 size=1003803 2024-03-09T14:36:06.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.146+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=1J1R10I0T5SS tag=E7PXTK3GMUFW type=write size=2052379 2024-03-09T14:36:06.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.150+0000 7f73315818c0 1 < canceled part key=_multipart_9B0RAHZZGLYU.2~4NMQ27YNGRQX.0 size=1048576 2024-03-09T14:36:06.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.150+0000 7f73315818c0 1 < canceled part key=_multipart_9B0RAHZZGLYU.2~4NMQ27YNGRQX.1 size=943317 2024-03-09T14:36:06.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.154+0000 7f73315818c0 1 < canceled multipart upload key=9B0RAHZZGLYU upload=4NMQ27YNGRQX tag=2YN2BAXBS3WX type=write size=3040469 2024-03-09T14:36:06.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.158+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~Z240OROGFGAQ.0 size=1048576 2024-03-09T14:36:06.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.162+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~Z240OROGFGAQ.1 size=1048576 2024-03-09T14:36:06.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.166+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~Z240OROGFGAQ.2 size=468265 2024-03-09T14:36:06.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.166+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=Z240OROGFGAQ tag=EBH9BJ44W0QU type=write size=3613993 2024-03-09T14:36:06.172 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.170+0000 7f73315818c0 1 < completed write operation key=LZC7NPF0GRO7 tag=3ZP7ORIY202A type=write size=168316 2024-03-09T14:36:06.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.174+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~Y9APZXN7HJ1X.0 size=1048576 2024-03-09T14:36:06.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.178+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~Y9APZXN7HJ1X.1 size=409020 2024-03-09T14:36:06.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.178+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=Y9APZXN7HJ1X tag=C2QIR9ZNQHNQ type=write size=2506172 2024-03-09T14:36:06.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.182+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~R191R1TTJGSV.0 size=1041997 2024-03-09T14:36:06.189 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.186+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=R191R1TTJGSV tag=SIF9WDXZE3JM type=write size=2090573 2024-03-09T14:36:06.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.190+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=JPBSYXWYWJ59 type=write size=576557 2024-03-09T14:36:06.196 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.194+0000 7f73315818c0 1 < completed delete operation key=DQOY1JJ1YQ2W tag=LUEG2BFIBH4C type=del 2024-03-09T14:36:06.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.198+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=UT8GF1JOBFUP type=write size=774447 2024-03-09T14:36:06.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.198+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~5MWTOHS54CF2.0 size=1048576 2024-03-09T14:36:06.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.202+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~5MWTOHS54CF2.1 size=1048576 2024-03-09T14:36:06.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.206+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~5MWTOHS54CF2.2 size=251026 2024-03-09T14:36:06.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.210+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=5MWTOHS54CF2 tag=S7LR9F4ZJU3F type=write size=3396754 2024-03-09T14:36:06.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.218+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~E2DJIM7U9I2P.0 size=1048576 2024-03-09T14:36:06.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.222+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~E2DJIM7U9I2P.1 size=431710 2024-03-09T14:36:06.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.222+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=E2DJIM7U9I2P tag=EN7FJH6NQR66 type=write size=2528862 2024-03-09T14:36:06.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.226+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~9NEZ8UOFBNBA.0 size=1048576 2024-03-09T14:36:06.231 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.226+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~9NEZ8UOFBNBA.1 size=1048576 2024-03-09T14:36:06.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.230+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~9NEZ8UOFBNBA.2 size=377792 2024-03-09T14:36:06.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.238+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=9NEZ8UOFBNBA tag=M1HMFYNJVMVF type=write size=3523520 2024-03-09T14:36:06.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.246+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~KZCTKU8R9OGO.0 size=1048576 2024-03-09T14:36:06.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.250+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~KZCTKU8R9OGO.1 size=624277 2024-03-09T14:36:06.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.250+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=KZCTKU8R9OGO tag=PPXCULSNG3YF type=write size=2721429 2024-03-09T14:36:06.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.254+0000 7f73315818c0 1 < completed write operation key=PXFZLDSAR1EG tag=22K61RJS1J0T type=write size=562391 2024-03-09T14:36:06.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.258+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=INHIHUI8VPHU type=del size=3969718 2024-03-09T14:36:06.264 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.262+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=MTMYP6Q7K445 type=del 2024-03-09T14:36:06.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.266+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=GBQTS7TQQ9R8 type=del size=909545 2024-03-09T14:36:06.272 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.270+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=1X8AE566B7OQ type=del 2024-03-09T14:36:06.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.274+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=I95Y8JVZW442 type=del size=4062538 2024-03-09T14:36:06.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.278+0000 7f73315818c0 1 < suggested remove operation key=0LP8KKU53X0A tag=97L1VDAJ62KC type=write 2024-03-09T14:36:06.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.286+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~08UHMLA5OA3V.0 size=1048576 2024-03-09T14:36:06.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.290+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~08UHMLA5OA3V.1 size=1048576 2024-03-09T14:36:06.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.294+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~08UHMLA5OA3V.2 size=577951 2024-03-09T14:36:06.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.294+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=08UHMLA5OA3V tag=C9T4KU41W8OC type=write size=3723679 2024-03-09T14:36:06.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.294+0000 7f73315818c0 1 < canceled part key=_multipart_CVAWRZMAX997.2~39MLTDOX0ZGZ.0 size=1048576 2024-03-09T14:36:06.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.298+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~39MLTDOX0ZGZ.1 size=742629 2024-03-09T14:36:06.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.302+0000 7f73315818c0 1 < canceled multipart upload key=CVAWRZMAX997 upload=39MLTDOX0ZGZ tag=M3XJDRB517J5 type=write size=2839781 2024-03-09T14:36:06.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.310+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~VH9SCE7TPVQC.0 size=1048576 2024-03-09T14:36:06.316 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.314+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~VH9SCE7TPVQC.1 size=1048576 2024-03-09T14:36:06.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.314+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~VH9SCE7TPVQC.2 size=493887 2024-03-09T14:36:06.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.314+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=VH9SCE7TPVQC tag=SZTMVOOPBUND type=write size=3639615 2024-03-09T14:36:06.323 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.318+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~KZJO5DHMIFTG.0 size=1048576 2024-03-09T14:36:06.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.322+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~KZJO5DHMIFTG.1 size=1048576 2024-03-09T14:36:06.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.326+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~KZJO5DHMIFTG.2 size=159891 2024-03-09T14:36:06.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.334+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=KZJO5DHMIFTG tag=CJJ89QT74RZQ type=write size=3305619 2024-03-09T14:36:06.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.338+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=MVAHYTHV3DRS type=write size=626240 2024-03-09T14:36:06.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.342+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~JX6MDD0LYA7D.0 size=1048576 2024-03-09T14:36:06.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.346+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~JX6MDD0LYA7D.1 size=334847 2024-03-09T14:36:06.353 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.350+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=JX6MDD0LYA7D tag=J5ERPS1DQJSW type=write size=2431999 2024-03-09T14:36:06.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.358+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~H27L9GWTT9UB.0 size=1048576 2024-03-09T14:36:06.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.358+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~H27L9GWTT9UB.1 size=741350 2024-03-09T14:36:06.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.358+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=H27L9GWTT9UB tag=GSKC8OG8T91Y type=write size=2838502 2024-03-09T14:36:06.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.362+0000 7f73315818c0 1 < completed write operation key=QG4O2L526B8E tag=T23OLD8KTUO8 type=write size=100176 2024-03-09T14:36:06.373 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.370+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~DH6Y8KKWGFKN.0 size=916833 2024-03-09T14:36:06.373 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.370+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=DH6Y8KKWGFKN tag=QLE7425PEK7A type=write size=1965409 2024-03-09T14:36:06.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.378+0000 7f73315818c0 1 < suggested remove operation key=4P89277EEYJ3 tag=9DTY3GYR4725 type=del 2024-03-09T14:36:06.387 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.382+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~ISJTONR35NRA.0 size=1045869 2024-03-09T14:36:06.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.382+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=ISJTONR35NRA tag=X5VLO083QXHS type=write size=2094445 2024-03-09T14:36:06.391 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.386+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~1J1R10I0T5SS.0 size=1003803 2024-03-09T14:36:06.396 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.394+0000 7f73315818c0 1 < completed multipart upload key=LZC7NPF0GRO7 upload=1J1R10I0T5SS tag=E7PXTK3GMUFW type=write size=2052379 2024-03-09T14:36:06.400 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.398+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=ZVTGQ0LR9PRB type=del size=4016069 2024-03-09T14:36:06.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.402+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~Z240OROGFGAQ.0 size=1048576 2024-03-09T14:36:06.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.406+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~Z240OROGFGAQ.1 size=1048576 2024-03-09T14:36:06.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.410+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~Z240OROGFGAQ.2 size=468265 2024-03-09T14:36:06.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.414+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=Z240OROGFGAQ tag=EBH9BJ44W0QU type=write size=3613993 2024-03-09T14:36:06.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.422+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~KXX6IAEZKZ1V.0 size=479703 2024-03-09T14:36:06.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.422+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=KXX6IAEZKZ1V tag=SJIDH5JBUW5P type=write size=1528279 2024-03-09T14:36:06.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.422+0000 7f73315818c0 1 < canceled part key=_multipart_MD7FXGYXB823.2~Y9APZXN7HJ1X.0 size=1048576 2024-03-09T14:36:06.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.426+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~Y9APZXN7HJ1X.1 size=409020 2024-03-09T14:36:06.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.430+0000 7f73315818c0 1 < canceled multipart upload key=MD7FXGYXB823 upload=Y9APZXN7HJ1X tag=C2QIR9ZNQHNQ type=write size=2506172 2024-03-09T14:36:06.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.434+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=PO6RZ2RSG44L type=del size=41111 2024-03-09T14:36:06.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.438+0000 7f73315818c0 1 < completed write operation key=PXFZLDSAR1EG tag=JPBSYXWYWJ59 type=write size=576557 2024-03-09T14:36:06.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.446+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~FG55099VEF65.0 size=1048576 2024-03-09T14:36:06.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.446+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~FG55099VEF65.1 size=1048576 2024-03-09T14:36:06.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.450+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~FG55099VEF65.2 size=357721 2024-03-09T14:36:06.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.450+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=FG55099VEF65 tag=CYDNSGUU5F2S type=write size=3503449 2024-03-09T14:36:06.461 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.458+0000 7f73315818c0 1 < suggested remove operation key=Y8EQAHMTL66U tag=UT8GF1JOBFUP type=write 2024-03-09T14:36:06.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.462+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=FZCSNQG0MA0I type=write size=522782 2024-03-09T14:36:06.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.466+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~E2DJIM7U9I2P.0 size=1048576 2024-03-09T14:36:06.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.470+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~E2DJIM7U9I2P.1 size=431710 2024-03-09T14:36:06.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.474+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=E2DJIM7U9I2P tag=EN7FJH6NQR66 type=write size=2528862 2024-03-09T14:36:06.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.478+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=DC44AN8I2KRD type=del size=4107187 2024-03-09T14:36:06.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.482+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~KZCTKU8R9OGO.0 size=1048576 2024-03-09T14:36:06.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.486+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~KZCTKU8R9OGO.1 size=624277 2024-03-09T14:36:06.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.490+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=KZCTKU8R9OGO tag=PPXCULSNG3YF type=write size=2721429 2024-03-09T14:36:06.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.494+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=URPUTQTQ0CJC type=del size=515186 2024-03-09T14:36:06.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.498+0000 7f73315818c0 -1 < failed to complete operation key=Y8EQAHMTL66U tag=INHIHUI8VPHU type=del size=3969718: (22) Invalid argument 2024-03-09T14:36:06.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.502+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=H81VS0JNZBU9 type=del size=3252392 2024-03-09T14:36:06.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.506+0000 7f73315818c0 1 < suggested remove operation key=W6KVP8EQFQ8J tag=GBQTS7TQQ9R8 type=del 2024-03-09T14:36:06.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.514+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~2KYQ0ASDBLG9.0 size=1048576 2024-03-09T14:36:06.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.518+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~2KYQ0ASDBLG9.1 size=1048576 2024-03-09T14:36:06.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.518+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~2KYQ0ASDBLG9.2 size=902315 2024-03-09T14:36:06.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.518+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=2KYQ0ASDBLG9 tag=XSQXYISPYROV type=write size=4048043 2024-03-09T14:36:06.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.526+0000 7f73315818c0 1 < suggested remove operation key=PAJYGD2GXHDT tag=I95Y8JVZW442 type=del 2024-03-09T14:36:06.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.534+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~3ONEMA6KM92C.0 size=1048576 2024-03-09T14:36:06.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.538+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~3ONEMA6KM92C.1 size=327939 2024-03-09T14:36:06.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.538+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=3ONEMA6KM92C tag=YSU7A7ZXWYKH type=write size=2425091 2024-03-09T14:36:06.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.542+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~08UHMLA5OA3V.0 size=1048576 2024-03-09T14:36:06.547 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.542+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~08UHMLA5OA3V.1 size=1048576 2024-03-09T14:36:06.549 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.546+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~08UHMLA5OA3V.2 size=577951 2024-03-09T14:36:06.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.550+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=08UHMLA5OA3V tag=C9T4KU41W8OC type=write size=3723679 2024-03-09T14:36:06.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.558+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~JSNLVN3BYATQ.0 size=1048576 2024-03-09T14:36:06.565 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.562+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~JSNLVN3BYATQ.1 size=67354 2024-03-09T14:36:06.565 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.562+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=JSNLVN3BYATQ tag=16RYNYPHZK7I type=write size=2164506 2024-03-09T14:36:06.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.566+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~VH9SCE7TPVQC.0 size=1048576 2024-03-09T14:36:06.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.566+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~VH9SCE7TPVQC.1 size=1048576 2024-03-09T14:36:06.574 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.570+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~VH9SCE7TPVQC.2 size=493887 2024-03-09T14:36:06.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.574+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=VH9SCE7TPVQC tag=SZTMVOOPBUND type=write size=3639615 2024-03-09T14:36:06.585 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.582+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~TWSKN3I9IAFD.0 size=1048576 2024-03-09T14:36:06.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.582+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~TWSKN3I9IAFD.1 size=393621 2024-03-09T14:36:06.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.582+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=TWSKN3I9IAFD tag=YBKCP85V1XA4 type=write size=2490773 2024-03-09T14:36:06.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.586+0000 7f73315818c0 -1 < no bi entry to suggest for operation key=PAJYGD2GXHDT tag=MVAHYTHV3DRS type=write size=626240: (2) No such file or directory 2024-03-09T14:36:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.590+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~QKKYC0BBULYR.0 size=187968 2024-03-09T14:36:06.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.590+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=QKKYC0BBULYR tag=67WFI4813Y82 type=write size=1236544 2024-03-09T14:36:06.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.594+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~H27L9GWTT9UB.0 size=1048576 2024-03-09T14:36:06.600 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.598+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~H27L9GWTT9UB.1 size=741350 2024-03-09T14:36:06.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.602+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=H27L9GWTT9UB tag=GSKC8OG8T91Y type=write size=2838502 2024-03-09T14:36:06.612 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.610+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~EG6RRI90IDAL.0 size=129254 2024-03-09T14:36:06.612 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.610+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=EG6RRI90IDAL tag=UNZAWOU40XEI type=write size=1177830 2024-03-09T14:36:06.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.614+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~DH6Y8KKWGFKN.0 size=916833 2024-03-09T14:36:06.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.618+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=DH6Y8KKWGFKN tag=QLE7425PEK7A type=write size=1965409 2024-03-09T14:36:06.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.622+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~QEFFOSYS5277.0 size=1048576 2024-03-09T14:36:06.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.626+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~QEFFOSYS5277.1 size=1048576 2024-03-09T14:36:06.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.630+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~QEFFOSYS5277.2 size=713483 2024-03-09T14:36:06.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.630+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=QEFFOSYS5277 tag=0SVGQK332MMZ type=write size=3859211 2024-03-09T14:36:06.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.630+0000 7f73315818c0 1 < canceled part key=_multipart_9B0RAHZZGLYU.2~ISJTONR35NRA.0 size=1045869 2024-03-09T14:36:06.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.634+0000 7f73315818c0 1 < canceled multipart upload key=9B0RAHZZGLYU upload=ISJTONR35NRA tag=X5VLO083QXHS type=write size=2094445 2024-03-09T14:36:06.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.638+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=WAH8BLLTDL20 type=del size=723161 2024-03-09T14:36:06.647 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.642+0000 7f73315818c0 1 < suggested update operation key=MD7FXGYXB823 tag=ZVTGQ0LR9PRB type=del size=4016069 2024-03-09T14:36:06.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.646+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=Y0COVJ5DWGQI type=del size=842239 2024-03-09T14:36:06.654 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.650+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~KXX6IAEZKZ1V.0 size=479703 2024-03-09T14:36:06.659 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.654+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=KXX6IAEZKZ1V tag=SJIDH5JBUW5P type=write size=1528279 2024-03-09T14:36:06.663 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.658+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=BYRVQ20Z41DE type=write size=163361 2024-03-09T14:36:06.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.662+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=PO6RZ2RSG44L type=del 2024-03-09T14:36:06.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.666+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=G9QCMB09N46J type=write size=646396 2024-03-09T14:36:06.673 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.670+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~FG55099VEF65.0 size=1048576 2024-03-09T14:36:06.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.670+0000 7f73315818c0 1 < canceled part key=_multipart_MD7FXGYXB823.2~FG55099VEF65.1 size=1048576 2024-03-09T14:36:06.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.674+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~FG55099VEF65.2 size=357721 2024-03-09T14:36:06.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.678+0000 7f73315818c0 1 < canceled multipart upload key=MD7FXGYXB823 upload=FG55099VEF65 tag=CYDNSGUU5F2S type=write size=3503449 2024-03-09T14:36:06.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.682+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=V2SWO0FE4LJ1 type=del size=277394 2024-03-09T14:36:06.688 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.686+0000 7f73315818c0 1 < completed write operation key=K3JNN1NCKBXN tag=FZCSNQG0MA0I type=write size=522782 2024-03-09T14:36:06.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.690+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=I3097N7TLJ03 type=del size=114598 2024-03-09T14:36:06.695 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.690+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=DC44AN8I2KRD type=del 2024-03-09T14:36:06.702 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.698+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~WQHTUGZXR3WN.0 size=1048576 2024-03-09T14:36:06.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.702+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~WQHTUGZXR3WN.1 size=875813 2024-03-09T14:36:06.705 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.702+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=WQHTUGZXR3WN tag=DZ6X08MX90BI type=write size=2972965 2024-03-09T14:36:06.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.706+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=URPUTQTQ0CJC type=del 2024-03-09T14:36:06.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.706+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=PS7TH9IY2MJD type=del size=2102342 2024-03-09T14:36:06.714 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.710+0000 7f73315818c0 -1 < failed to complete operation key=W6KVP8EQFQ8J tag=H81VS0JNZBU9 type=del size=3252392: (22) Invalid argument 2024-03-09T14:36:06.724 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.722+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~ZTE2Q8FO6KKI.0 size=304838 2024-03-09T14:36:06.724 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.722+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=ZTE2Q8FO6KKI tag=3V146MJO61MJ type=write size=1353414 2024-03-09T14:36:06.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.722+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~2KYQ0ASDBLG9.0 size=1048576 2024-03-09T14:36:06.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.726+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~2KYQ0ASDBLG9.1 size=1048576 2024-03-09T14:36:06.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.730+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~2KYQ0ASDBLG9.2 size=902315 2024-03-09T14:36:06.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.734+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=2KYQ0ASDBLG9 tag=XSQXYISPYROV type=write size=4048043 2024-03-09T14:36:06.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.742+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~6JDGCM4AMM1L.0 size=93916 2024-03-09T14:36:06.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.742+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=6JDGCM4AMM1L tag=LTEE747K6OFR type=write size=1142492 2024-03-09T14:36:06.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.746+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~3ONEMA6KM92C.0 size=1048576 2024-03-09T14:36:06.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.746+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~3ONEMA6KM92C.1 size=327939 2024-03-09T14:36:06.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.754+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=3ONEMA6KM92C tag=YSU7A7ZXWYKH type=write size=2425091 2024-03-09T14:36:06.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.758+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~N3NJL66IC9E3.0 size=1048576 2024-03-09T14:36:06.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.762+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~N3NJL66IC9E3.1 size=1048576 2024-03-09T14:36:06.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.766+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~N3NJL66IC9E3.2 size=768728 2024-03-09T14:36:06.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.766+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=N3NJL66IC9E3 tag=W9QLVKQ78CN2 type=write size=3914456 2024-03-09T14:36:06.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.766+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~JSNLVN3BYATQ.0 size=1048576 2024-03-09T14:36:06.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.770+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~JSNLVN3BYATQ.1 size=67354 2024-03-09T14:36:06.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.778+0000 7f73315818c0 1 < completed multipart upload key=IA0AB7UVLFEJ upload=JSNLVN3BYATQ tag=16RYNYPHZK7I type=write size=2164506 2024-03-09T14:36:06.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.778+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=9NTWJF0KCOO5 type=write size=911016 2024-03-09T14:36:06.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.782+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~TWSKN3I9IAFD.0 size=1048576 2024-03-09T14:36:06.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.782+0000 7f73315818c0 1 < canceled part key=_multipart_162AOMHO2BQL.2~TWSKN3I9IAFD.1 size=393621 2024-03-09T14:36:06.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.790+0000 7f73315818c0 1 < canceled multipart upload key=162AOMHO2BQL upload=TWSKN3I9IAFD tag=YBKCP85V1XA4 type=write size=2490773 2024-03-09T14:36:06.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.794+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~YPIWOMGE49YJ.0 size=101825 2024-03-09T14:36:06.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.794+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=YPIWOMGE49YJ tag=8H3Q46QN2915 type=write size=1150401 2024-03-09T14:36:06.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.798+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~QKKYC0BBULYR.0 size=187968 2024-03-09T14:36:06.807 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.806+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=QKKYC0BBULYR tag=67WFI4813Y82 type=write size=1236544 2024-03-09T14:36:06.811 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.806+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=F9LIO7O31X0F type=del size=1358530 2024-03-09T14:36:06.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.810+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~EG6RRI90IDAL.0 size=129254 2024-03-09T14:36:06.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.818+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=EG6RRI90IDAL tag=UNZAWOU40XEI type=write size=1177830 2024-03-09T14:36:06.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.822+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=45OA2RB6ZX96 type=write size=612992 2024-03-09T14:36:06.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.826+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~QEFFOSYS5277.0 size=1048576 2024-03-09T14:36:06.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.826+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~QEFFOSYS5277.1 size=1048576 2024-03-09T14:36:06.834 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.830+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~QEFFOSYS5277.2 size=713483 2024-03-09T14:36:06.839 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.838+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=QEFFOSYS5277 tag=0SVGQK332MMZ type=write size=3859211 2024-03-09T14:36:06.843 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.838+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=Q3TWHIVO7SZN type=del size=3063803 2024-03-09T14:36:06.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.842+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=WAH8BLLTDL20 type=del 2024-03-09T14:36:06.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.846+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=P04ZR5A2I5WT type=del size=3100812 2024-03-09T14:36:06.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.850+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=Y0COVJ5DWGQI type=del 2024-03-09T14:36:06.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.854+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=88IXMJGACIQ6 type=del size=1050639 2024-03-09T14:36:06.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.858+0000 7f73315818c0 1 < completed write operation key=MSMD3ANK4T7J tag=BYRVQ20Z41DE type=write size=163361 2024-03-09T14:36:06.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.858+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=GXOCMYLEVB6Z type=del size=1017881 2024-03-09T14:36:06.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.862+0000 7f73315818c0 1 < completed write operation key=MSMD3ANK4T7J tag=G9QCMB09N46J type=write size=646396 2024-03-09T14:36:06.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.866+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=JFWWFLK48PC0 type=del size=2554747 2024-03-09T14:36:06.877 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.874+0000 7f73315818c0 1 < suggested update operation key=92GWY5B0K4SX tag=V2SWO0FE4LJ1 type=del size=277394 2024-03-09T14:36:06.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.878+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=MNKG8EEZCWLM type=del size=776282 2024-03-09T14:36:06.884 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.882+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=I3097N7TLJ03 type=del 2024-03-09T14:36:06.890 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.886+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~BE2D3SZKKNV3.0 size=1048576 2024-03-09T14:36:06.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.890+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~BE2D3SZKKNV3.1 size=1048576 2024-03-09T14:36:06.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.890+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~BE2D3SZKKNV3.2 size=521989 2024-03-09T14:36:06.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.890+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=BE2D3SZKKNV3 tag=VS1JS2DCX1YT type=write size=3667717 2024-03-09T14:36:06.898 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.894+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~WQHTUGZXR3WN.0 size=1048576 2024-03-09T14:36:06.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.898+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~WQHTUGZXR3WN.1 size=875813 2024-03-09T14:36:06.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.902+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=WQHTUGZXR3WN tag=DZ6X08MX90BI type=write size=2972965 2024-03-09T14:36:06.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.910+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~1RRES9H4S5GL.0 size=1048576 2024-03-09T14:36:06.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.910+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~1RRES9H4S5GL.1 size=373803 2024-03-09T14:36:06.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.910+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=1RRES9H4S5GL tag=CI4GE14QDSIE type=write size=2470955 2024-03-09T14:36:06.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.914+0000 7f73315818c0 1 < canceled operation key=4P89277EEYJ3 tag=PS7TH9IY2MJD type=del size=2102342 2024-03-09T14:36:06.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.918+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=MV1ACL5XLIL8 type=del size=2794200 2024-03-09T14:36:06.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.922+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~ZTE2Q8FO6KKI.0 size=304838 2024-03-09T14:36:06.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.926+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=ZTE2Q8FO6KKI tag=3V146MJO61MJ type=write size=1353414 2024-03-09T14:36:06.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.930+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=958037JZ3UH3 type=write size=19379 2024-03-09T14:36:06.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.934+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~6JDGCM4AMM1L.0 size=93916 2024-03-09T14:36:06.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.938+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=6JDGCM4AMM1L tag=LTEE747K6OFR type=write size=1142492 2024-03-09T14:36:06.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.942+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=XU0DHL2GVYEC type=del size=3910173 2024-03-09T14:36:06.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.946+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~N3NJL66IC9E3.0 size=1048576 2024-03-09T14:36:06.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.946+0000 7f73315818c0 1 < canceled part key=_multipart_92GWY5B0K4SX.2~N3NJL66IC9E3.1 size=1048576 2024-03-09T14:36:06.951 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.946+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~N3NJL66IC9E3.2 size=768728 2024-03-09T14:36:06.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.954+0000 7f73315818c0 1 < canceled multipart upload key=92GWY5B0K4SX upload=N3NJL66IC9E3 tag=W9QLVKQ78CN2 type=write size=3914456 2024-03-09T14:36:06.960 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.958+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=UP6BF5BBFYF4 type=del size=3251668 2024-03-09T14:36:06.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.958+0000 7f73315818c0 1 < completed write operation key=P9LI5ESR3N9G tag=9NTWJF0KCOO5 type=write size=911016 2024-03-09T14:36:06.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.962+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=51SN3R3ZXUX2 type=write size=539253 2024-03-09T14:36:06.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.966+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~YPIWOMGE49YJ.0 size=101825 2024-03-09T14:36:06.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.974+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=YPIWOMGE49YJ tag=8H3Q46QN2915 type=write size=1150401 2024-03-09T14:36:06.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.974+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=4VN6LVB1XC3K type=write size=1013808 2024-03-09T14:36:06.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.978+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=F9LIO7O31X0F type=del 2024-03-09T14:36:06.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.982+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=AR665KWL11JB type=del size=431213 2024-03-09T14:36:06.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.986+0000 7f73315818c0 1 < completed write operation key=P8R5RJI8LGZ9 tag=45OA2RB6ZX96 type=write size=612992 2024-03-09T14:36:06.993 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.990+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=X0VXD1B3IOJZ type=del size=239511 2024-03-09T14:36:06.996 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.994+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=Q3TWHIVO7SZN type=del 2024-03-09T14:36:06.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.998+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=S3YL0JR31CKH type=del size=1448983 2024-03-09T14:36:07.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:06.998+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=P04ZR5A2I5WT type=del 2024-03-09T14:36:07.006 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.002+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=9ZL6WETGB33F type=del size=3031816 2024-03-09T14:36:07.010 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.006+0000 7f73315818c0 1 < canceled operation key=M8F9TA80VME6 tag=88IXMJGACIQ6 type=del size=1050639 2024-03-09T14:36:07.016 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.014+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~Z3UNJV284H2V.0 size=1048576 2024-03-09T14:36:07.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.014+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~Z3UNJV284H2V.1 size=558604 2024-03-09T14:36:07.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.014+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=Z3UNJV284H2V tag=AJWV1YCBC9FL type=write size=2655756 2024-03-09T14:36:07.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.018+0000 7f73315818c0 1 < canceled operation key=IA0AB7UVLFEJ tag=GXOCMYLEVB6Z type=del size=1017881 2024-03-09T14:36:07.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.022+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=KJEBBDEB6UII type=del size=3575747 2024-03-09T14:36:07.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.026+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=JFWWFLK48PC0 type=del 2024-03-09T14:36:07.039 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.034+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~K6R7RR7Y5A45.0 size=1048576 2024-03-09T14:36:07.041 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.038+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~K6R7RR7Y5A45.1 size=514517 2024-03-09T14:36:07.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.038+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=K6R7RR7Y5A45 tag=KUY4AIBANLOH type=write size=2611669 2024-03-09T14:36:07.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.042+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=MNKG8EEZCWLM type=del 2024-03-09T14:36:07.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.046+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=VUKWUB0CNJ2O type=del size=3610068 2024-03-09T14:36:07.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.050+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~BE2D3SZKKNV3.0 size=1048576 2024-03-09T14:36:07.056 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.054+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~BE2D3SZKKNV3.1 size=1048576 2024-03-09T14:36:07.056 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.054+0000 7f73315818c0 1 < canceled part key=_multipart_M8F9TA80VME6.2~BE2D3SZKKNV3.2 size=521989 2024-03-09T14:36:07.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.058+0000 7f73315818c0 1 < canceled multipart upload key=M8F9TA80VME6 upload=BE2D3SZKKNV3 tag=VS1JS2DCX1YT type=write size=3667717 2024-03-09T14:36:07.065 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.062+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=VYENZXKQ9PRE type=del size=1097692 2024-03-09T14:36:07.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.066+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~1RRES9H4S5GL.0 size=1048576 2024-03-09T14:36:07.071 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.070+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~1RRES9H4S5GL.1 size=373803 2024-03-09T14:36:07.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.074+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=1RRES9H4S5GL tag=CI4GE14QDSIE type=write size=2470955 2024-03-09T14:36:07.084 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.082+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~9OQ9WHY0FMFF.0 size=1048576 2024-03-09T14:36:07.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.086+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~9OQ9WHY0FMFF.1 size=462753 2024-03-09T14:36:07.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.086+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=9OQ9WHY0FMFF tag=Y3AXUIG811NF type=write size=2559905 2024-03-09T14:36:07.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.086+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=MV1ACL5XLIL8 type=del 2024-03-09T14:36:07.094 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.090+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=9853EHQ7WIN1 type=del size=283483 2024-03-09T14:36:07.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.094+0000 7f73315818c0 1 < completed write operation key=8O5OP8C5NX12 tag=958037JZ3UH3 type=write size=19379 2024-03-09T14:36:07.104 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.102+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~FEUZK4JSCBTB.0 size=548114 2024-03-09T14:36:07.105 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.102+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=FEUZK4JSCBTB tag=17O3JL7LJN5S type=write size=1596690 2024-03-09T14:36:07.108 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.106+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=XU0DHL2GVYEC type=del 2024-03-09T14:36:07.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.110+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~CTXY5MOF2162.0 size=1048576 2024-03-09T14:36:07.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.114+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~CTXY5MOF2162.1 size=1048576 2024-03-09T14:36:07.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.118+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~CTXY5MOF2162.2 size=315436 2024-03-09T14:36:07.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.118+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=CTXY5MOF2162 tag=0OWETTP3J5ZP type=write size=3461164 2024-03-09T14:36:07.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.122+0000 7f73315818c0 1 < suggested update operation key=Y8EQAHMTL66U tag=UP6BF5BBFYF4 type=del size=3251668 2024-03-09T14:36:07.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.130+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~WT6KXOJXHE2P.0 size=1048576 2024-03-09T14:36:07.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.134+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~WT6KXOJXHE2P.1 size=811458 2024-03-09T14:36:07.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.134+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=WT6KXOJXHE2P tag=KMC5X6KM0VL2 type=write size=2908610 2024-03-09T14:36:07.141 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.138+0000 7f73315818c0 1 < completed write operation key=4E3QMBHEAEHO tag=51SN3R3ZXUX2 type=write size=539253 2024-03-09T14:36:07.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.142+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=A53F4C5EA88X type=del size=2590291 2024-03-09T14:36:07.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.150+0000 7f73315818c0 1 < suggested remove operation key=Y8EQAHMTL66U tag=4VN6LVB1XC3K type=write 2024-03-09T14:36:07.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.154+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~MVI92XVIGZTD.0 size=79688 2024-03-09T14:36:07.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.154+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=MVI92XVIGZTD tag=YOUP62RFLMTP type=write size=1128264 2024-03-09T14:36:07.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.162+0000 7f73315818c0 1 < suggested remove operation key=92GWY5B0K4SX tag=AR665KWL11JB type=del 2024-03-09T14:36:07.171 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.166+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~HB8601U7N34D.0 size=1048576 2024-03-09T14:36:07.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.170+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~HB8601U7N34D.1 size=1048576 2024-03-09T14:36:07.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.174+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~HB8601U7N34D.2 size=178122 2024-03-09T14:36:07.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.174+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=HB8601U7N34D tag=09IY98VMNEGV type=write size=3323850 2024-03-09T14:36:07.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.178+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=X0VXD1B3IOJZ type=del 2024-03-09T14:36:07.183 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.182+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=EP62FUAGWEJF type=write size=519421 2024-03-09T14:36:07.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.182+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=S3YL0JR31CKH type=del 2024-03-09T14:36:07.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.186+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=FXHC27NIKEUI type=write size=911482 2024-03-09T14:36:07.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.194+0000 7f73315818c0 1 < suggested remove operation key=8O5OP8C5NX12 tag=9ZL6WETGB33F type=del 2024-03-09T14:36:07.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.198+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=3T0BOF6ST7EY type=del size=2921599 2024-03-09T14:36:07.205 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.202+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~Z3UNJV284H2V.0 size=1048576 2024-03-09T14:36:07.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.206+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~Z3UNJV284H2V.1 size=558604 2024-03-09T14:36:07.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.210+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=Z3UNJV284H2V tag=AJWV1YCBC9FL type=write size=2655756 2024-03-09T14:36:07.222 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.218+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~0HDKS1SO9203.0 size=1048576 2024-03-09T14:36:07.224 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.222+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~0HDKS1SO9203.1 size=861953 2024-03-09T14:36:07.224 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.222+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=0HDKS1SO9203 tag=TQUSXS8NKVJK type=write size=2959105 2024-03-09T14:36:07.231 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.230+0000 7f73315818c0 1 < suggested remove operation key=DQOY1JJ1YQ2W tag=KJEBBDEB6UII type=del 2024-03-09T14:36:07.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.234+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~RDE239QRFS2S.0 size=1048576 2024-03-09T14:36:07.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.238+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~RDE239QRFS2S.1 size=1048576 2024-03-09T14:36:07.243 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.242+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~RDE239QRFS2S.2 size=145319 2024-03-09T14:36:07.243 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.242+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=RDE239QRFS2S tag=35TJZ9DVC2O9 type=write size=3291047 2024-03-09T14:36:07.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.242+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~K6R7RR7Y5A45.0 size=1048576 2024-03-09T14:36:07.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.246+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~K6R7RR7Y5A45.1 size=514517 2024-03-09T14:36:07.255 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.250+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=K6R7RR7Y5A45 tag=KUY4AIBANLOH type=write size=2611669 2024-03-09T14:36:07.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.254+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=34QV9HFQ85BO type=del size=3772255 2024-03-09T14:36:07.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.258+0000 7f73315818c0 -1 < no bi entry to suggest for operation key=DQOY1JJ1YQ2W tag=VUKWUB0CNJ2O type=del size=3610068: (2) No such file or directory 2024-03-09T14:36:07.267 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.266+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~P9R8R9IJXNU7.0 size=390252 2024-03-09T14:36:07.267 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.266+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=P9R8R9IJXNU7 tag=GZOM2YXKAPZX type=write size=1438828 2024-03-09T14:36:07.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.270+0000 7f73315818c0 1 < suggested remove operation key=UMSNM53HTY1J tag=VYENZXKQ9PRE type=del 2024-03-09T14:36:07.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.278+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~OBSPDAUMMXB2.0 size=1048576 2024-03-09T14:36:07.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.282+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~OBSPDAUMMXB2.1 size=1048576 2024-03-09T14:36:07.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.282+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~OBSPDAUMMXB2.2 size=8554 2024-03-09T14:36:07.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.282+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=OBSPDAUMMXB2 tag=LM6RKFVPX8BO type=write size=3154282 2024-03-09T14:36:07.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.286+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~9OQ9WHY0FMFF.0 size=1048576 2024-03-09T14:36:07.293 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.290+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~9OQ9WHY0FMFF.1 size=462753 2024-03-09T14:36:07.299 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.294+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=9OQ9WHY0FMFF tag=Y3AXUIG811NF type=write size=2559905 2024-03-09T14:36:07.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.298+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=F5ZNZ0GXM047 type=del size=3900112 2024-03-09T14:36:07.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.302+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=9853EHQ7WIN1 type=del 2024-03-09T14:36:07.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.310+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~Y31S8HFP0T4F.0 size=1048576 2024-03-09T14:36:07.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.310+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~Y31S8HFP0T4F.1 size=803359 2024-03-09T14:36:07.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.310+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=Y31S8HFP0T4F tag=6IRD4JVKZKIL type=write size=2900511 2024-03-09T14:36:07.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.314+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~FEUZK4JSCBTB.0 size=548114 2024-03-09T14:36:07.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.322+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=FEUZK4JSCBTB tag=17O3JL7LJN5S type=write size=1596690 2024-03-09T14:36:07.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.326+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=EY2GP097TQDV type=write size=66923 2024-03-09T14:36:07.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.330+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~CTXY5MOF2162.0 size=1048576 2024-03-09T14:36:07.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.334+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~CTXY5MOF2162.1 size=1048576 2024-03-09T14:36:07.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.334+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~CTXY5MOF2162.2 size=315436 2024-03-09T14:36:07.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.342+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=CTXY5MOF2162 tag=0OWETTP3J5ZP type=write size=3461164 2024-03-09T14:36:07.348 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.346+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=2HCLE98IGZLV type=del size=3215634 2024-03-09T14:36:07.351 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.350+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~WT6KXOJXHE2P.0 size=1048576 2024-03-09T14:36:07.354 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.350+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~WT6KXOJXHE2P.1 size=811458 2024-03-09T14:36:07.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.358+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=WT6KXOJXHE2P tag=KMC5X6KM0VL2 type=write size=2908610 2024-03-09T14:36:07.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.366+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~925SPP4V2O5L.0 size=1048576 2024-03-09T14:36:07.372 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.370+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~925SPP4V2O5L.1 size=267508 2024-03-09T14:36:07.372 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.370+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=925SPP4V2O5L tag=SO0UH2LO7B7Z type=write size=2364660 2024-03-09T14:36:07.375 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.374+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=A53F4C5EA88X type=del 2024-03-09T14:36:07.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.378+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~FJNCELCBTAUO.0 size=1048576 2024-03-09T14:36:07.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.382+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~FJNCELCBTAUO.1 size=1048576 2024-03-09T14:36:07.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.386+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~FJNCELCBTAUO.2 size=543687 2024-03-09T14:36:07.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.386+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=FJNCELCBTAUO tag=IJ1UHGLG6GIH type=write size=3689415 2024-03-09T14:36:07.391 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.390+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~MVI92XVIGZTD.0 size=79688 2024-03-09T14:36:07.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.394+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=MVI92XVIGZTD tag=YOUP62RFLMTP type=write size=1128264 2024-03-09T14:36:07.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.398+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=H9E47QNNZ1SL type=del size=4098853 2024-03-09T14:36:07.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.402+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~HB8601U7N34D.0 size=1048576 2024-03-09T14:36:07.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.406+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~HB8601U7N34D.1 size=1048576 2024-03-09T14:36:07.411 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.406+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~HB8601U7N34D.2 size=178122 2024-03-09T14:36:07.417 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.414+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=HB8601U7N34D tag=09IY98VMNEGV type=write size=3323850 2024-03-09T14:36:07.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.418+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=I3UYS0A3OLJA type=del size=360116 2024-03-09T14:36:07.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.426+0000 7f73315818c0 1 < suggested update operation key=Y8EQAHMTL66U tag=EP62FUAGWEJF type=write size=519421 2024-03-09T14:36:07.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.430+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~Q9M86BIFQOTT.0 size=1048576 2024-03-09T14:36:07.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.434+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~Q9M86BIFQOTT.1 size=785116 2024-03-09T14:36:07.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.434+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=Q9M86BIFQOTT tag=A3G5WZXN24PH type=write size=2882268 2024-03-09T14:36:07.440 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.438+0000 7f73315818c0 -1 < failed to complete operation key=DQOY1JJ1YQ2W tag=FXHC27NIKEUI type=write size=911482: (22) Invalid argument 2024-03-09T14:36:07.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.442+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~U3E8E7KP0IQ6.0 size=1048576 2024-03-09T14:36:07.449 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.446+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~U3E8E7KP0IQ6.1 size=1048576 2024-03-09T14:36:07.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.450+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~U3E8E7KP0IQ6.2 size=251561 2024-03-09T14:36:07.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.450+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=U3E8E7KP0IQ6 tag=CRV9VD61IC7F type=write size=3397289 2024-03-09T14:36:07.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.454+0000 7f73315818c0 1 < completed delete operation key=M1B6O6N8N84T tag=3T0BOF6ST7EY type=del 2024-03-09T14:36:07.462 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.458+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~TJZF5Y8154GE.0 size=1048576 2024-03-09T14:36:07.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.462+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~TJZF5Y8154GE.1 size=1048576 2024-03-09T14:36:07.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.466+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~TJZF5Y8154GE.2 size=436426 2024-03-09T14:36:07.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.466+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=TJZF5Y8154GE tag=IXX20K0E4DE9 type=write size=3582154 2024-03-09T14:36:07.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.466+0000 7f73315818c0 1 < canceled part key=_multipart_M1B6O6N8N84T.2~0HDKS1SO9203.0 size=1048576 2024-03-09T14:36:07.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.466+0000 7f73315818c0 1 < canceled part key=_multipart_M1B6O6N8N84T.2~0HDKS1SO9203.1 size=861953 2024-03-09T14:36:07.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.470+0000 7f73315818c0 1 < canceled multipart upload key=M1B6O6N8N84T upload=0HDKS1SO9203 tag=TQUSXS8NKVJK type=write size=2959105 2024-03-09T14:36:07.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.478+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~C9A7EMV25A8X.0 size=1048576 2024-03-09T14:36:07.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.478+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~C9A7EMV25A8X.1 size=342775 2024-03-09T14:36:07.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.478+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=C9A7EMV25A8X tag=42WJEJNFCRH0 type=write size=2439927 2024-03-09T14:36:07.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.482+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~RDE239QRFS2S.0 size=1048576 2024-03-09T14:36:07.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.486+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~RDE239QRFS2S.1 size=1048576 2024-03-09T14:36:07.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.490+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~RDE239QRFS2S.2 size=145319 2024-03-09T14:36:07.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.494+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=RDE239QRFS2S tag=35TJZ9DVC2O9 type=write size=3291047 2024-03-09T14:36:07.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.498+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=Q9WX303CISFG type=del size=3881885 2024-03-09T14:36:07.505 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.502+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=34QV9HFQ85BO type=del 2024-03-09T14:36:07.509 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.506+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=7L066AYZ5HDX type=del size=821940 2024-03-09T14:36:07.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.510+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~P9R8R9IJXNU7.0 size=390252 2024-03-09T14:36:07.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.514+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=P9R8R9IJXNU7 tag=GZOM2YXKAPZX type=write size=1438828 2024-03-09T14:36:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.518+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=1HFFOARXVLTA type=del size=1972931 2024-03-09T14:36:07.526 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.522+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~OBSPDAUMMXB2.0 size=1048576 2024-03-09T14:36:07.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.526+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~OBSPDAUMMXB2.1 size=1048576 2024-03-09T14:36:07.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.530+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~OBSPDAUMMXB2.2 size=8554 2024-03-09T14:36:07.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.534+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=OBSPDAUMMXB2 tag=LM6RKFVPX8BO type=write size=3154282 2024-03-09T14:36:07.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.542+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~KX7AOOAG4RZM.0 size=1048576 2024-03-09T14:36:07.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.542+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~KX7AOOAG4RZM.1 size=709767 2024-03-09T14:36:07.547 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.542+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=KX7AOOAG4RZM tag=0E1UZQDHH80F type=write size=2806919 2024-03-09T14:36:07.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.546+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=F5ZNZ0GXM047 type=del 2024-03-09T14:36:07.556 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.554+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~29PVUBCPW11N.0 size=1048576 2024-03-09T14:36:07.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.554+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~29PVUBCPW11N.1 size=223702 2024-03-09T14:36:07.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.554+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=29PVUBCPW11N tag=EHNIWX847P6U type=write size=2320854 2024-03-09T14:36:07.564 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.562+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~Y31S8HFP0T4F.0 size=1048576 2024-03-09T14:36:07.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.562+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~Y31S8HFP0T4F.1 size=803359 2024-03-09T14:36:07.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.570+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=Y31S8HFP0T4F tag=6IRD4JVKZKIL type=write size=2900511 2024-03-09T14:36:07.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.574+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=VHKJVZFXLWQZ type=del size=1884203 2024-03-09T14:36:07.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.578+0000 7f73315818c0 1 < completed write operation key=M1B6O6N8N84T tag=EY2GP097TQDV type=write size=66923 2024-03-09T14:36:07.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.582+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=SZNQ7Y10W14N type=del size=501558 2024-03-09T14:36:07.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.582+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=2HCLE98IGZLV type=del 2024-03-09T14:36:07.593 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.590+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~5VI73S8SJJHZ.0 size=1048576 2024-03-09T14:36:07.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.594+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~5VI73S8SJJHZ.1 size=1048576 2024-03-09T14:36:07.599 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.594+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~5VI73S8SJJHZ.2 size=804954 2024-03-09T14:36:07.599 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.594+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=5VI73S8SJJHZ tag=TY7SJ6WW466L type=write size=3950682 2024-03-09T14:36:07.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.598+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~925SPP4V2O5L.0 size=1048576 2024-03-09T14:36:07.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.598+0000 7f73315818c0 1 < canceled part key=_multipart_QG4O2L526B8E.2~925SPP4V2O5L.1 size=267508 2024-03-09T14:36:07.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.606+0000 7f73315818c0 1 < canceled multipart upload key=QG4O2L526B8E upload=925SPP4V2O5L tag=SO0UH2LO7B7Z type=write size=2364660 2024-03-09T14:36:07.615 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.614+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~NZTJDG1T7A4L.0 size=843881 2024-03-09T14:36:07.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.614+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=NZTJDG1T7A4L tag=Y7GMK97Y3XD5 type=write size=1892457 2024-03-09T14:36:07.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.614+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~FJNCELCBTAUO.0 size=1048576 2024-03-09T14:36:07.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.618+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~FJNCELCBTAUO.1 size=1048576 2024-03-09T14:36:07.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.622+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~FJNCELCBTAUO.2 size=543687 2024-03-09T14:36:07.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.630+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=FJNCELCBTAUO tag=IJ1UHGLG6GIH type=write size=3689415 2024-03-09T14:36:07.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.630+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=ODROA3P3S0X4 type=del size=2422775 2024-03-09T14:36:07.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.634+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=H9E47QNNZ1SL type=del 2024-03-09T14:36:07.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.638+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=7G2XF3FOFML6 type=del size=1370700 2024-03-09T14:36:07.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.642+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=I3UYS0A3OLJA type=del 2024-03-09T14:36:07.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.650+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~4GI80WIIY115.0 size=847942 2024-03-09T14:36:07.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.650+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=4GI80WIIY115 tag=S3P9HQNF9CU5 type=write size=1896518 2024-03-09T14:36:07.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.650+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~Q9M86BIFQOTT.0 size=1048576 2024-03-09T14:36:07.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.654+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~Q9M86BIFQOTT.1 size=785116 2024-03-09T14:36:07.664 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.662+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=Q9M86BIFQOTT tag=A3G5WZXN24PH type=write size=2882268 2024-03-09T14:36:07.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.666+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=7U5OCYPEPS80 type=write size=972772 2024-03-09T14:36:07.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.670+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~U3E8E7KP0IQ6.0 size=1048576 2024-03-09T14:36:07.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.670+0000 7f73315818c0 1 < canceled part key=_multipart_QG4O2L526B8E.2~U3E8E7KP0IQ6.1 size=1048576 2024-03-09T14:36:07.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.670+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~U3E8E7KP0IQ6.2 size=251561 2024-03-09T14:36:07.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.678+0000 7f73315818c0 1 < canceled multipart upload key=QG4O2L526B8E upload=U3E8E7KP0IQ6 tag=CRV9VD61IC7F type=write size=3397289 2024-03-09T14:36:07.688 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.682+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~5YATSH1L8PB6.0 size=1048576 2024-03-09T14:36:07.692 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.690+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~5YATSH1L8PB6.1 size=255328 2024-03-09T14:36:07.692 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.690+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=5YATSH1L8PB6 tag=P8UC206401IY type=write size=2352480 2024-03-09T14:36:07.695 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.694+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~TJZF5Y8154GE.0 size=1048576 2024-03-09T14:36:07.699 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.694+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~TJZF5Y8154GE.1 size=1048576 2024-03-09T14:36:07.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.698+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~TJZF5Y8154GE.2 size=436426 2024-03-09T14:36:07.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.706+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=TJZF5Y8154GE tag=IXX20K0E4DE9 type=write size=3582154 2024-03-09T14:36:07.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.710+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=G2UP0AVGA5WR type=del size=1050207 2024-03-09T14:36:07.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.710+0000 7f73315818c0 1 < canceled part key=_multipart_OU0730HGI176.2~C9A7EMV25A8X.0 size=1048576 2024-03-09T14:36:07.715 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.710+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~C9A7EMV25A8X.1 size=342775 2024-03-09T14:36:07.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.718+0000 7f73315818c0 1 < canceled multipart upload key=OU0730HGI176 upload=C9A7EMV25A8X tag=42WJEJNFCRH0 type=write size=2439927 2024-03-09T14:36:07.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.726+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~SAG1MPWNP2K8.0 size=1048576 2024-03-09T14:36:07.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.730+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~SAG1MPWNP2K8.1 size=1048576 2024-03-09T14:36:07.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.730+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~SAG1MPWNP2K8.2 size=984669 2024-03-09T14:36:07.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.730+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=SAG1MPWNP2K8 tag=VWQ271B3UXDP type=write size=4130397 2024-03-09T14:36:07.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.734+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=Q9WX303CISFG type=del 2024-03-09T14:36:07.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.742+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~FHMRKBP8FSRD.0 size=516903 2024-03-09T14:36:07.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.742+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=FHMRKBP8FSRD tag=1ANLJK8PB425 type=write size=1565479 2024-03-09T14:36:07.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.746+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=7L066AYZ5HDX type=del 2024-03-09T14:36:07.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.754+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~W0MW63HHRIVK.0 size=1048576 2024-03-09T14:36:07.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.758+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~W0MW63HHRIVK.1 size=1048576 2024-03-09T14:36:07.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.758+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~W0MW63HHRIVK.2 size=792196 2024-03-09T14:36:07.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.758+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=W0MW63HHRIVK tag=1DFC1A1I35SE type=write size=3937924 2024-03-09T14:36:07.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.762+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=1HFFOARXVLTA type=del 2024-03-09T14:36:07.773 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.770+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~3QE0729AB7X4.0 size=546460 2024-03-09T14:36:07.773 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.770+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=3QE0729AB7X4 tag=6RWSWXH2GNZ4 type=write size=1595036 2024-03-09T14:36:07.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.774+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~KX7AOOAG4RZM.0 size=1048576 2024-03-09T14:36:07.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.774+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~KX7AOOAG4RZM.1 size=709767 2024-03-09T14:36:07.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.782+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=KX7AOOAG4RZM tag=0E1UZQDHH80F type=write size=2806919 2024-03-09T14:36:07.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.782+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=APCOVYI1G8WO type=del size=1812422 2024-03-09T14:36:07.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.786+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~29PVUBCPW11N.0 size=1048576 2024-03-09T14:36:07.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.790+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~29PVUBCPW11N.1 size=223702 2024-03-09T14:36:07.798 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.794+0000 7f73315818c0 1 < completed multipart upload key=0IB00CZ92VV4 upload=29PVUBCPW11N tag=EHNIWX847P6U type=write size=2320854 2024-03-09T14:36:07.803 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.802+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~QTV7KCRCHR3X.0 size=1048576 2024-03-09T14:36:07.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.802+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~QTV7KCRCHR3X.1 size=678344 2024-03-09T14:36:07.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.802+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=QTV7KCRCHR3X tag=CFL1QYZLRP37 type=write size=2775496 2024-03-09T14:36:07.809 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.806+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=VHKJVZFXLWQZ type=del 2024-03-09T14:36:07.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.810+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~O0UX2RSCT4UM.0 size=1048576 2024-03-09T14:36:07.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.814+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~O0UX2RSCT4UM.1 size=1048576 2024-03-09T14:36:07.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.818+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~O0UX2RSCT4UM.2 size=744859 2024-03-09T14:36:07.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.818+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=O0UX2RSCT4UM tag=5B8ZXJXEPUKJ type=write size=3890587 2024-03-09T14:36:07.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.818+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=SZNQ7Y10W14N type=del 2024-03-09T14:36:07.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.826+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~VKI7X9XH1EFR.0 size=1048576 2024-03-09T14:36:07.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.826+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~VKI7X9XH1EFR.1 size=401733 2024-03-09T14:36:07.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.826+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=VKI7X9XH1EFR tag=K3DGXK1BABEW type=write size=2498885 2024-03-09T14:36:07.834 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.830+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~5VI73S8SJJHZ.0 size=1048576 2024-03-09T14:36:07.837 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.834+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~5VI73S8SJJHZ.1 size=1048576 2024-03-09T14:36:07.840 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.838+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~5VI73S8SJJHZ.2 size=804954 2024-03-09T14:36:07.845 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.842+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=5VI73S8SJJHZ tag=TY7SJ6WW466L type=write size=3950682 2024-03-09T14:36:07.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.846+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=09128N7SO3O2 type=write size=211628 2024-03-09T14:36:07.852 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.850+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~NZTJDG1T7A4L.0 size=843881 2024-03-09T14:36:07.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.854+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=NZTJDG1T7A4L tag=Y7GMK97Y3XD5 type=write size=1892457 2024-03-09T14:36:07.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.858+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=2ANZWFUHX4MG type=del size=417214 2024-03-09T14:36:07.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.862+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=ODROA3P3S0X4 type=del 2024-03-09T14:36:07.871 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.870+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~XFJYT8UOX034.0 size=1048576 2024-03-09T14:36:07.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.870+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~XFJYT8UOX034.1 size=684664 2024-03-09T14:36:07.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.870+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=XFJYT8UOX034 tag=NVLXAPF89JLA type=write size=2781816 2024-03-09T14:36:07.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.878+0000 7f73315818c0 1 < suggested remove operation key=3ADG4GWVARC1 tag=7G2XF3FOFML6 type=del 2024-03-09T14:36:07.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.882+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=9PS1DX5B28WV type=write size=664033 2024-03-09T14:36:07.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.886+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~4GI80WIIY115.0 size=847942 2024-03-09T14:36:07.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.890+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=4GI80WIIY115 tag=S3P9HQNF9CU5 type=write size=1896518 2024-03-09T14:36:07.898 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.894+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=3WZN0BZXQ9RG type=del size=692973 2024-03-09T14:36:07.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.898+0000 7f73315818c0 1 < completed write operation key=S302JFYEDEL7 tag=7U5OCYPEPS80 type=write size=972772 2024-03-09T14:36:07.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.902+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=5EAAJNWVG29Q type=del size=3356299 2024-03-09T14:36:07.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.906+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~5YATSH1L8PB6.0 size=1048576 2024-03-09T14:36:07.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.910+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~5YATSH1L8PB6.1 size=255328 2024-03-09T14:36:07.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.914+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=5YATSH1L8PB6 tag=P8UC206401IY type=write size=2352480 2024-03-09T14:36:07.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.918+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=22LOU87WF9LS type=del size=2684620 2024-03-09T14:36:07.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.922+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=G2UP0AVGA5WR type=del 2024-03-09T14:36:07.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.930+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~F94EZWLKSD5Y.0 size=602028 2024-03-09T14:36:07.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.930+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=F94EZWLKSD5Y tag=DOD9PV9NGVGI type=write size=1650604 2024-03-09T14:36:07.935 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.930+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~SAG1MPWNP2K8.0 size=1048576 2024-03-09T14:36:07.938 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.934+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~SAG1MPWNP2K8.1 size=1048576 2024-03-09T14:36:07.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.938+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~SAG1MPWNP2K8.2 size=984669 2024-03-09T14:36:07.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.942+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=SAG1MPWNP2K8 tag=VWQ271B3UXDP type=write size=4130397 2024-03-09T14:36:07.950 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.946+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=1YX512F4N5H9 type=del size=4047026 2024-03-09T14:36:07.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.950+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~FHMRKBP8FSRD.0 size=516903 2024-03-09T14:36:07.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.954+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=FHMRKBP8FSRD tag=1ANLJK8PB425 type=write size=1565479 2024-03-09T14:36:07.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.958+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=KH19QBP7Q0QD type=del size=2330477 2024-03-09T14:36:07.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.962+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~W0MW63HHRIVK.0 size=1048576 2024-03-09T14:36:07.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.966+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~W0MW63HHRIVK.1 size=1048576 2024-03-09T14:36:07.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.966+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~W0MW63HHRIVK.2 size=792196 2024-03-09T14:36:07.978 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.974+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=W0MW63HHRIVK tag=1DFC1A1I35SE type=write size=3937924 2024-03-09T14:36:07.981 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.978+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=ENM6GP39H2FE type=write size=500492 2024-03-09T14:36:07.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.982+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~3QE0729AB7X4.0 size=546460 2024-03-09T14:36:07.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.986+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=3QE0729AB7X4 tag=6RWSWXH2GNZ4 type=write size=1595036 2024-03-09T14:36:07.993 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.990+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=EYUSE25V6L8U type=del size=3644470 2024-03-09T14:36:07.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.998+0000 7f73315818c0 1 < suggested remove operation key=MSMD3ANK4T7J tag=APCOVYI1G8WO type=del 2024-03-09T14:36:08.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:07.998+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=XSZYEBSW8PE2 type=del size=2582600 2024-03-09T14:36:08.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.006+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~QTV7KCRCHR3X.0 size=1048576 2024-03-09T14:36:08.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.010+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~QTV7KCRCHR3X.1 size=678344 2024-03-09T14:36:08.018 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.014+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=QTV7KCRCHR3X tag=CFL1QYZLRP37 type=write size=2775496 2024-03-09T14:36:08.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.022+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~B5SLGIDP9NSU.0 size=1048576 2024-03-09T14:36:08.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.022+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~B5SLGIDP9NSU.1 size=1048576 2024-03-09T14:36:08.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.026+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~B5SLGIDP9NSU.2 size=730963 2024-03-09T14:36:08.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.026+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=B5SLGIDP9NSU tag=M9EY62UOK6E5 type=write size=3876691 2024-03-09T14:36:08.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.030+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~O0UX2RSCT4UM.0 size=1048576 2024-03-09T14:36:08.035 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.034+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~O0UX2RSCT4UM.1 size=1048576 2024-03-09T14:36:08.038 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.034+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~O0UX2RSCT4UM.2 size=744859 2024-03-09T14:36:08.044 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.042+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=O0UX2RSCT4UM tag=5B8ZXJXEPUKJ type=write size=3890587 2024-03-09T14:36:08.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.046+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~LW2G1E1P7H12.0 size=987835 2024-03-09T14:36:08.050 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.046+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=LW2G1E1P7H12 tag=XYEW88132743 type=write size=2036411 2024-03-09T14:36:08.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.050+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~VKI7X9XH1EFR.0 size=1048576 2024-03-09T14:36:08.056 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.054+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~VKI7X9XH1EFR.1 size=401733 2024-03-09T14:36:08.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.058+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=VKI7X9XH1EFR tag=K3DGXK1BABEW type=write size=2498885 2024-03-09T14:36:08.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.066+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~EJD18HTT605Q.0 size=1048576 2024-03-09T14:36:08.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.070+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~EJD18HTT605Q.1 size=927655 2024-03-09T14:36:08.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.070+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=EJD18HTT605Q tag=5SCJK9G320JH type=write size=3024807 2024-03-09T14:36:08.075 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.070+0000 7f73315818c0 1 < canceled operation key=162AOMHO2BQL tag=09128N7SO3O2 type=write size=211628 2024-03-09T14:36:08.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.078+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~753Z8YJPZW8Z.0 size=699464 2024-03-09T14:36:08.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.078+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=753Z8YJPZW8Z tag=NFAH2VYKGNQK type=write size=1748040 2024-03-09T14:36:08.085 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.082+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=2ANZWFUHX4MG type=del 2024-03-09T14:36:08.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.086+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~HEX63G48R5V2.0 size=387333 2024-03-09T14:36:08.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.086+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=HEX63G48R5V2 tag=2ITMRK00V2IU type=write size=1435909 2024-03-09T14:36:08.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.090+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~XFJYT8UOX034.0 size=1048576 2024-03-09T14:36:08.097 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.094+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~XFJYT8UOX034.1 size=684664 2024-03-09T14:36:08.102 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.098+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=XFJYT8UOX034 tag=NVLXAPF89JLA type=write size=2781816 2024-03-09T14:36:08.105 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.102+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=U2RB3ZVZS5NC type=del size=4062821 2024-03-09T14:36:08.108 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.106+0000 7f73315818c0 1 < completed write operation key=QG4O2L526B8E tag=9PS1DX5B28WV type=write size=664033 2024-03-09T14:36:08.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.106+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=PHG51GZHCNVY type=del size=422623 2024-03-09T14:36:08.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.114+0000 7f73315818c0 1 < suggested update operation key=S302JFYEDEL7 tag=3WZN0BZXQ9RG type=del size=692973 2024-03-09T14:36:08.122 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.118+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~FBYPNJN1RFS9.0 size=1048576 2024-03-09T14:36:08.124 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.122+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~FBYPNJN1RFS9.1 size=1048576 2024-03-09T14:36:08.126 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.122+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~FBYPNJN1RFS9.2 size=24768 2024-03-09T14:36:08.126 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.122+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=FBYPNJN1RFS9 tag=STE4J5PPEJED type=write size=3170496 2024-03-09T14:36:08.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.126+0000 7f73315818c0 1 < completed delete operation key=DQOY1JJ1YQ2W tag=5EAAJNWVG29Q type=del 2024-03-09T14:36:08.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.130+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=01NTNOWS6Q5Z type=del size=1450771 2024-03-09T14:36:08.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.134+0000 7f73315818c0 1 < canceled operation key=0LP8KKU53X0A tag=22LOU87WF9LS type=del size=2684620 2024-03-09T14:36:08.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.134+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=6OL2LEFCFV47 type=del size=2908299 2024-03-09T14:36:08.140 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.138+0000 7f73315818c0 1 < canceled part key=_multipart_4E3QMBHEAEHO.2~F94EZWLKSD5Y.0 size=602028 2024-03-09T14:36:08.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.142+0000 7f73315818c0 1 < canceled multipart upload key=4E3QMBHEAEHO upload=F94EZWLKSD5Y tag=DOD9PV9NGVGI type=write size=1650604 2024-03-09T14:36:08.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.150+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~P4MXP8GJ104B.0 size=1048576 2024-03-09T14:36:08.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.150+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~P4MXP8GJ104B.1 size=1048576 2024-03-09T14:36:08.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.154+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~P4MXP8GJ104B.2 size=637883 2024-03-09T14:36:08.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.154+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=P4MXP8GJ104B tag=91E1QIJPTJEJ type=write size=3783611 2024-03-09T14:36:08.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.158+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=1YX512F4N5H9 type=del 2024-03-09T14:36:08.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.162+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=0CPUNXYFW1OP type=write size=848562 2024-03-09T14:36:08.169 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.166+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=KH19QBP7Q0QD type=del 2024-03-09T14:36:08.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.174+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~JLKI7BLXFBHY.0 size=1048576 2024-03-09T14:36:08.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.174+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~JLKI7BLXFBHY.1 size=1048576 2024-03-09T14:36:08.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.178+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~JLKI7BLXFBHY.2 size=955340 2024-03-09T14:36:08.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.178+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=JLKI7BLXFBHY tag=6XBY8940I0NH type=write size=4101068 2024-03-09T14:36:08.186 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.182+0000 7f73315818c0 1 < suggested remove operation key=K3JNN1NCKBXN tag=ENM6GP39H2FE type=write 2024-03-09T14:36:08.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.190+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~ZL2ZUTWEN96Y.0 size=1048576 2024-03-09T14:36:08.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.194+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~ZL2ZUTWEN96Y.1 size=25542 2024-03-09T14:36:08.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.194+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=ZL2ZUTWEN96Y tag=09M5AWZAB4KB type=write size=2122694 2024-03-09T14:36:08.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.198+0000 7f73315818c0 1 < suggested remove operation key=P8R5RJI8LGZ9 tag=EYUSE25V6L8U type=del 2024-03-09T14:36:08.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.202+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=QBMM8SDAY1MI type=del size=1712398 2024-03-09T14:36:08.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.206+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=XSZYEBSW8PE2 type=del 2024-03-09T14:36:08.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.210+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=32DYEYSD88BZ type=del size=2400429 2024-03-09T14:36:08.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.210+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~B5SLGIDP9NSU.0 size=1048576 2024-03-09T14:36:08.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.214+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~B5SLGIDP9NSU.1 size=1048576 2024-03-09T14:36:08.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.214+0000 7f73315818c0 1 < canceled part key=_multipart_PAJYGD2GXHDT.2~B5SLGIDP9NSU.2 size=730963 2024-03-09T14:36:08.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.222+0000 7f73315818c0 1 < canceled multipart upload key=PAJYGD2GXHDT upload=B5SLGIDP9NSU tag=M9EY62UOK6E5 type=write size=3876691 2024-03-09T14:36:08.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.226+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=W9ZMW702WK4T type=del size=1833516 2024-03-09T14:36:08.230 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.226+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~LW2G1E1P7H12.0 size=987835 2024-03-09T14:36:08.236 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.234+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=LW2G1E1P7H12 tag=XYEW88132743 type=write size=2036411 2024-03-09T14:36:08.242 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.238+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~QVR6ZWHDDJB2.0 size=1048576 2024-03-09T14:36:08.245 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.242+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~QVR6ZWHDDJB2.1 size=713674 2024-03-09T14:36:08.245 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.242+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=QVR6ZWHDDJB2 tag=M174A0VYGKZS type=write size=2810826 2024-03-09T14:36:08.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.246+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~EJD18HTT605Q.0 size=1048576 2024-03-09T14:36:08.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.246+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~EJD18HTT605Q.1 size=927655 2024-03-09T14:36:08.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.254+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=EJD18HTT605Q tag=5SCJK9G320JH type=write size=3024807 2024-03-09T14:36:08.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.258+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=UYBVEBHXB9YQ type=del size=1305015 2024-03-09T14:36:08.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.262+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~753Z8YJPZW8Z.0 size=699464 2024-03-09T14:36:08.271 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.266+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=753Z8YJPZW8Z tag=NFAH2VYKGNQK type=write size=1748040 2024-03-09T14:36:08.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.270+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=5CLSL4K99CFN type=del size=2456767 2024-03-09T14:36:08.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.278+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~HEX63G48R5V2.0 size=387333 2024-03-09T14:36:08.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.282+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=HEX63G48R5V2 tag=2ITMRK00V2IU type=write size=1435909 2024-03-09T14:36:08.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.290+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~8FFQVKGDYG0Y.0 size=1048576 2024-03-09T14:36:08.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.290+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~8FFQVKGDYG0Y.1 size=706315 2024-03-09T14:36:08.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.290+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=8FFQVKGDYG0Y tag=1AM2X43H035P type=write size=2803467 2024-03-09T14:36:08.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.294+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=U2RB3ZVZS5NC type=del 2024-03-09T14:36:08.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.298+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=P5OHUFDR6BX6 type=del size=2043180 2024-03-09T14:36:08.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.306+0000 7f73315818c0 1 < suggested remove operation key=MSMD3ANK4T7J tag=PHG51GZHCNVY type=del 2024-03-09T14:36:08.314 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.310+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~HB80VPPB36X3.0 size=1048576 2024-03-09T14:36:08.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.314+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~HB80VPPB36X3.1 size=1048576 2024-03-09T14:36:08.321 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.318+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~HB80VPPB36X3.2 size=16077 2024-03-09T14:36:08.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.318+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=HB80VPPB36X3 tag=WN9W2A6VYX3V type=write size=3161805 2024-03-09T14:36:08.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.322+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~FBYPNJN1RFS9.0 size=1048576 2024-03-09T14:36:08.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.326+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~FBYPNJN1RFS9.1 size=1048576 2024-03-09T14:36:08.331 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.326+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~FBYPNJN1RFS9.2 size=24768 2024-03-09T14:36:08.337 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.334+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=FBYPNJN1RFS9 tag=STE4J5PPEJED type=write size=3170496 2024-03-09T14:36:08.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.338+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=KO125EK0GOL2 type=del size=3914267 2024-03-09T14:36:08.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.342+0000 7f73315818c0 1 < suggested remove operation key=8O5OP8C5NX12 tag=01NTNOWS6Q5Z type=del 2024-03-09T14:36:08.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.350+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~13750MFK9EPS.0 size=1048576 2024-03-09T14:36:08.357 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.354+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~13750MFK9EPS.1 size=636481 2024-03-09T14:36:08.357 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.354+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=13750MFK9EPS tag=BA7Y2GGK3VLX type=write size=2733633 2024-03-09T14:36:08.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.358+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=6OL2LEFCFV47 type=del 2024-03-09T14:36:08.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.362+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=H8NYUOAXY105 type=write size=549132 2024-03-09T14:36:08.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.362+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~P4MXP8GJ104B.0 size=1048576 2024-03-09T14:36:08.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.366+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~P4MXP8GJ104B.1 size=1048576 2024-03-09T14:36:08.372 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.370+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~P4MXP8GJ104B.2 size=637883 2024-03-09T14:36:08.377 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.374+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=P4MXP8GJ104B tag=91E1QIJPTJEJ type=write size=3783611 2024-03-09T14:36:08.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.382+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~NVJ9XKGJ539O.0 size=1048576 2024-03-09T14:36:08.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.382+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~NVJ9XKGJ539O.1 size=1048576 2024-03-09T14:36:08.389 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.386+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~NVJ9XKGJ539O.2 size=499586 2024-03-09T14:36:08.389 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.386+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=NVJ9XKGJ539O tag=GBJGZ79ULPW4 type=write size=3645314 2024-03-09T14:36:08.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.390+0000 7f73315818c0 1 < completed write operation key=M8F9TA80VME6 tag=0CPUNXYFW1OP type=write size=848562 2024-03-09T14:36:08.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.394+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=8A0ZLMXLQW1V type=del size=3238085 2024-03-09T14:36:08.398 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.394+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~JLKI7BLXFBHY.0 size=1048576 2024-03-09T14:36:08.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.394+0000 7f73315818c0 1 < canceled part key=_multipart_92GWY5B0K4SX.2~JLKI7BLXFBHY.1 size=1048576 2024-03-09T14:36:08.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.398+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~JLKI7BLXFBHY.2 size=955340 2024-03-09T14:36:08.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.402+0000 7f73315818c0 1 < canceled multipart upload key=92GWY5B0K4SX upload=JLKI7BLXFBHY tag=6XBY8940I0NH type=write size=4101068 2024-03-09T14:36:08.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.406+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=1PPWLFCXXA3R type=write size=541819 2024-03-09T14:36:08.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.410+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~ZL2ZUTWEN96Y.0 size=1048576 2024-03-09T14:36:08.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.410+0000 7f73315818c0 1 < canceled part key=_multipart_S302JFYEDEL7.2~ZL2ZUTWEN96Y.1 size=25542 2024-03-09T14:36:08.416 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.414+0000 7f73315818c0 1 < canceled multipart upload key=S302JFYEDEL7 upload=ZL2ZUTWEN96Y tag=09M5AWZAB4KB type=write size=2122694 2024-03-09T14:36:08.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.414+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=GM333541PX5J type=write size=82818 2024-03-09T14:36:08.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.422+0000 7f73315818c0 1 < suggested remove operation key=0LP8KKU53X0A tag=QBMM8SDAY1MI type=del 2024-03-09T14:36:08.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.426+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=4B3RJO8EZ1IW type=del size=75250 2024-03-09T14:36:08.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.426+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=32DYEYSD88BZ type=del 2024-03-09T14:36:08.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.430+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=FUK60KFFUGAA type=write size=170024 2024-03-09T14:36:08.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.434+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=W9ZMW702WK4T type=del 2024-03-09T14:36:08.440 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.438+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=R5VW0NFJPTBS type=del size=3456980 2024-03-09T14:36:08.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.438+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~QVR6ZWHDDJB2.0 size=1048576 2024-03-09T14:36:08.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.438+0000 7f73315818c0 1 < canceled part key=_multipart_QG4O2L526B8E.2~QVR6ZWHDDJB2.1 size=713674 2024-03-09T14:36:08.449 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.446+0000 7f73315818c0 1 < canceled multipart upload key=QG4O2L526B8E upload=QVR6ZWHDDJB2 tag=M174A0VYGKZS type=write size=2810826 2024-03-09T14:36:08.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.450+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=ETIIL2IAIJLR type=del size=3180489 2024-03-09T14:36:08.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.454+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=UYBVEBHXB9YQ type=del 2024-03-09T14:36:08.459 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.454+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=I5XY2RF58DZ0 type=del size=3136255 2024-03-09T14:36:08.462 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.458+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=5CLSL4K99CFN type=del 2024-03-09T14:36:08.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.462+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=3HFCRHPDZ0PU type=del size=2723567 2024-03-09T14:36:08.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.466+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~8FFQVKGDYG0Y.0 size=1048576 2024-03-09T14:36:08.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.470+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~8FFQVKGDYG0Y.1 size=706315 2024-03-09T14:36:08.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.474+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=8FFQVKGDYG0Y tag=1AM2X43H035P type=write size=2803467 2024-03-09T14:36:08.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.478+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=R71A0XTJ8MSQ type=del size=2755442 2024-03-09T14:36:08.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.482+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=P5OHUFDR6BX6 type=del 2024-03-09T14:36:08.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.486+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=TRXFXNU9A320 type=write size=943746 2024-03-09T14:36:08.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.490+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~HB80VPPB36X3.0 size=1048576 2024-03-09T14:36:08.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.490+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~HB80VPPB36X3.1 size=1048576 2024-03-09T14:36:08.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.494+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~HB80VPPB36X3.2 size=16077 2024-03-09T14:36:08.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.498+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=HB80VPPB36X3 tag=WN9W2A6VYX3V type=write size=3161805 2024-03-09T14:36:08.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.506+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~53U04WPQHHPL.0 size=741596 2024-03-09T14:36:08.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.506+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=53U04WPQHHPL tag=84XHU3HGQIXT type=write size=1790172 2024-03-09T14:36:08.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.510+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=KO125EK0GOL2 type=del 2024-03-09T14:36:08.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.514+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=N8COAD1NCIOS type=del size=233673 2024-03-09T14:36:08.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.514+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~13750MFK9EPS.0 size=1048576 2024-03-09T14:36:08.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.514+0000 7f73315818c0 1 < canceled part key=_multipart_LZC7NPF0GRO7.2~13750MFK9EPS.1 size=636481 2024-03-09T14:36:08.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.522+0000 7f73315818c0 1 < canceled multipart upload key=LZC7NPF0GRO7 upload=13750MFK9EPS tag=BA7Y2GGK3VLX type=write size=2733633 2024-03-09T14:36:08.528 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.526+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=1OIR5K3W0OYV type=del size=3741083 2024-03-09T14:36:08.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.526+0000 7f73315818c0 1 < completed write operation key=1NYNL8M5DVER tag=H8NYUOAXY105 type=write size=549132 2024-03-09T14:36:08.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.534+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~S38E8NAL04ZG.0 size=1048576 2024-03-09T14:36:08.539 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.538+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~S38E8NAL04ZG.1 size=1048576 2024-03-09T14:36:08.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.538+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~S38E8NAL04ZG.2 size=650420 2024-03-09T14:36:08.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.538+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=S38E8NAL04ZG tag=WAMX0KISTWX7 type=write size=3796148 2024-03-09T14:36:08.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.542+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~NVJ9XKGJ539O.0 size=1048576 2024-03-09T14:36:08.549 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.546+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~NVJ9XKGJ539O.1 size=1048576 2024-03-09T14:36:08.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.546+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~NVJ9XKGJ539O.2 size=499586 2024-03-09T14:36:08.557 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.554+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=NVJ9XKGJ539O tag=GBJGZ79ULPW4 type=write size=3645314 2024-03-09T14:36:08.561 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.558+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=7I5LN1HU3UW7 type=del size=1926583 2024-03-09T14:36:08.564 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.562+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=8A0ZLMXLQW1V type=del 2024-03-09T14:36:08.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.566+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=Z04HX0FQ4LZ8 type=del size=3828334 2024-03-09T14:36:08.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.570+0000 7f73315818c0 1 < completed write operation key=IA0AB7UVLFEJ tag=1PPWLFCXXA3R type=write size=541819 2024-03-09T14:36:08.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.574+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=7DLOQK24IH9L type=del size=3593036 2024-03-09T14:36:08.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.578+0000 7f73315818c0 1 < completed write operation key=S302JFYEDEL7 tag=GM333541PX5J type=write size=82818 2024-03-09T14:36:08.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.582+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~4ACCP6CWISU1.0 size=1048576 2024-03-09T14:36:08.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.586+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~4ACCP6CWISU1.1 size=59802 2024-03-09T14:36:08.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.586+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=4ACCP6CWISU1 tag=7RH1U5SI7EU0 type=write size=2156954 2024-03-09T14:36:08.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.590+0000 7f73315818c0 1 < suggested remove operation key=UMSNM53HTY1J tag=4B3RJO8EZ1IW type=del 2024-03-09T14:36:08.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.594+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=H0LBGS42I7L9 type=write size=740833 2024-03-09T14:36:08.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.598+0000 7f73315818c0 1 < completed write operation key=3ADG4GWVARC1 tag=FUK60KFFUGAA type=write size=170024 2024-03-09T14:36:08.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.602+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~GWJ3B8H0X0A0.0 size=1048576 2024-03-09T14:36:08.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.606+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~GWJ3B8H0X0A0.1 size=1048576 2024-03-09T14:36:08.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.606+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~GWJ3B8H0X0A0.2 size=827208 2024-03-09T14:36:08.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.606+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=GWJ3B8H0X0A0 tag=8Z0UG0SR04ZJ type=write size=3972936 2024-03-09T14:36:08.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.610+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=R5VW0NFJPTBS type=del 2024-03-09T14:36:08.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.618+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~A0223LSG4KFP.0 size=1048576 2024-03-09T14:36:08.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.622+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~A0223LSG4KFP.1 size=1048576 2024-03-09T14:36:08.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.626+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~A0223LSG4KFP.2 size=92194 2024-03-09T14:36:08.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.626+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=A0223LSG4KFP tag=6W0WYP56GTAI type=write size=3237922 2024-03-09T14:36:08.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.630+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=ETIIL2IAIJLR type=del 2024-03-09T14:36:08.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.638+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~WZQI9HJF23ZH.0 size=371510 2024-03-09T14:36:08.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.638+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=WZQI9HJF23ZH tag=3RJ9GN857I9D type=write size=1420086 2024-03-09T14:36:08.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.642+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=I5XY2RF58DZ0 type=del 2024-03-09T14:36:08.647 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.642+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=OBQAAVD4839V type=del size=2809925 2024-03-09T14:36:08.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.646+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=3HFCRHPDZ0PU type=del 2024-03-09T14:36:08.656 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.654+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~F4GLLG8N3DD5.0 size=1048576 2024-03-09T14:36:08.659 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.654+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~F4GLLG8N3DD5.1 size=455193 2024-03-09T14:36:08.659 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.654+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=F4GLLG8N3DD5 tag=BEXG5MXZNWIM type=write size=2552345 2024-03-09T14:36:08.663 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.658+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=R71A0XTJ8MSQ type=del 2024-03-09T14:36:08.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.662+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=JNNS9FXJGHDH type=del size=846690 2024-03-09T14:36:08.673 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.670+0000 7f73315818c0 1 < suggested update operation key=QG4O2L526B8E tag=TRXFXNU9A320 type=write size=943746 2024-03-09T14:36:08.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.674+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~VP836EDB94DS.0 size=1048576 2024-03-09T14:36:08.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.678+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~VP836EDB94DS.1 size=1048576 2024-03-09T14:36:08.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.682+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~VP836EDB94DS.2 size=94694 2024-03-09T14:36:08.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.682+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=VP836EDB94DS tag=LP2A2VM9Y088 type=write size=3240422 2024-03-09T14:36:08.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.686+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~53U04WPQHHPL.0 size=741596 2024-03-09T14:36:08.693 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.690+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=53U04WPQHHPL tag=84XHU3HGQIXT type=write size=1790172 2024-03-09T14:36:08.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.694+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=BCRRWYUZ7R80 type=del size=4017916 2024-03-09T14:36:08.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.698+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=N8COAD1NCIOS type=del 2024-03-09T14:36:08.703 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.698+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=TWG1M71DQJO6 type=del size=1701762 2024-03-09T14:36:08.707 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.702+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=1OIR5K3W0OYV type=del 2024-03-09T14:36:08.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.710+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~2H3ZWM46M5WV.0 size=781415 2024-03-09T14:36:08.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.710+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=2H3ZWM46M5WV tag=NYU7GH7ZXHBM type=write size=1829991 2024-03-09T14:36:08.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.714+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~S38E8NAL04ZG.0 size=1048576 2024-03-09T14:36:08.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.714+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~S38E8NAL04ZG.1 size=1048576 2024-03-09T14:36:08.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.718+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~S38E8NAL04ZG.2 size=650420 2024-03-09T14:36:08.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.726+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=S38E8NAL04ZG tag=WAMX0KISTWX7 type=write size=3796148 2024-03-09T14:36:08.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.730+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=YPJGNK3V51Q6 type=del size=1611512 2024-03-09T14:36:08.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.730+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=7I5LN1HU3UW7 type=del 2024-03-09T14:36:08.742 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.738+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~FTCUPRSESCV5.0 size=1048576 2024-03-09T14:36:08.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.742+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~FTCUPRSESCV5.1 size=296773 2024-03-09T14:36:08.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.742+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=FTCUPRSESCV5 tag=VH75G1728SBT type=write size=2393925 2024-03-09T14:36:08.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.746+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=Z04HX0FQ4LZ8 type=del 2024-03-09T14:36:08.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.750+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~AAZS63WH1NGZ.0 size=1048576 2024-03-09T14:36:08.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.754+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~AAZS63WH1NGZ.1 size=1048576 2024-03-09T14:36:08.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.754+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~AAZS63WH1NGZ.2 size=633752 2024-03-09T14:36:08.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.754+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=AAZS63WH1NGZ tag=8L6KD4DEIA62 type=write size=3779480 2024-03-09T14:36:08.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.762+0000 7f73315818c0 1 < suggested update operation key=M8F9TA80VME6 tag=7DLOQK24IH9L type=del size=3593036 2024-03-09T14:36:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.766+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=OC7M89I03TY1 type=del size=244333 2024-03-09T14:36:08.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.770+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~4ACCP6CWISU1.0 size=1048576 2024-03-09T14:36:08.775 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.770+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~4ACCP6CWISU1.1 size=59802 2024-03-09T14:36:08.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.778+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=4ACCP6CWISU1 tag=7RH1U5SI7EU0 type=write size=2156954 2024-03-09T14:36:08.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.782+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~H7UOA05OY9XD.0 size=1048576 2024-03-09T14:36:08.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.786+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~H7UOA05OY9XD.1 size=1048576 2024-03-09T14:36:08.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.790+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~H7UOA05OY9XD.2 size=978773 2024-03-09T14:36:08.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.790+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=H7UOA05OY9XD tag=9A5T8VS7GP5M type=write size=4124501 2024-03-09T14:36:08.796 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.794+0000 7f73315818c0 1 < canceled operation key=P9LI5ESR3N9G tag=H0LBGS42I7L9 type=write size=740833 2024-03-09T14:36:08.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.794+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=TR7LPJEFOCK6 type=del size=2364220 2024-03-09T14:36:08.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.798+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~GWJ3B8H0X0A0.0 size=1048576 2024-03-09T14:36:08.805 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.802+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~GWJ3B8H0X0A0.1 size=1048576 2024-03-09T14:36:08.808 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.806+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~GWJ3B8H0X0A0.2 size=827208 2024-03-09T14:36:08.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.810+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=GWJ3B8H0X0A0 tag=8Z0UG0SR04ZJ type=write size=3972936 2024-03-09T14:36:08.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.814+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=B0QW7ICRTC59 type=del size=3345497 2024-03-09T14:36:08.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.818+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~A0223LSG4KFP.0 size=1048576 2024-03-09T14:36:08.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.818+0000 7f73315818c0 1 < canceled part key=_multipart_QG4O2L526B8E.2~A0223LSG4KFP.1 size=1048576 2024-03-09T14:36:08.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.822+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~A0223LSG4KFP.2 size=92194 2024-03-09T14:36:08.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.826+0000 7f73315818c0 1 < canceled multipart upload key=QG4O2L526B8E upload=A0223LSG4KFP tag=6W0WYP56GTAI type=write size=3237922 2024-03-09T14:36:08.838 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.834+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~3VCXJQFJEXX0.0 size=459351 2024-03-09T14:36:08.838 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.834+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=3VCXJQFJEXX0 tag=LZYLP9KVDS2F type=write size=1507927 2024-03-09T14:36:08.839 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.838+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~WZQI9HJF23ZH.0 size=371510 2024-03-09T14:36:08.845 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.842+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=WZQI9HJF23ZH tag=3RJ9GN857I9D type=write size=1420086 2024-03-09T14:36:08.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.846+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=9KGMVBDLBPAX type=del size=3653534 2024-03-09T14:36:08.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.850+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=OBQAAVD4839V type=del 2024-03-09T14:36:08.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.854+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=QX697O4TEPEF type=del size=1463882 2024-03-09T14:36:08.860 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.858+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~F4GLLG8N3DD5.0 size=1048576 2024-03-09T14:36:08.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.858+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~F4GLLG8N3DD5.1 size=455193 2024-03-09T14:36:08.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.866+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=F4GLLG8N3DD5 tag=BEXG5MXZNWIM type=write size=2552345 2024-03-09T14:36:08.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.870+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=N90YWFPV748A type=del size=3321657 2024-03-09T14:36:08.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.878+0000 7f73315818c0 1 < suggested remove operation key=8O5OP8C5NX12 tag=JNNS9FXJGHDH type=del 2024-03-09T14:36:08.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.882+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~076ONVWLMV19.0 size=805411 2024-03-09T14:36:08.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.882+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=076ONVWLMV19 tag=EM8JFIUWA88M type=write size=1853987 2024-03-09T14:36:08.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.890+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~VP836EDB94DS.0 size=1048576 2024-03-09T14:36:08.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.890+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~VP836EDB94DS.1 size=1048576 2024-03-09T14:36:08.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.890+0000 7f73315818c0 1 < canceled part key=_multipart_QG4O2L526B8E.2~VP836EDB94DS.2 size=94694 2024-03-09T14:36:08.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.898+0000 7f73315818c0 1 < canceled multipart upload key=QG4O2L526B8E upload=VP836EDB94DS tag=LP2A2VM9Y088 type=write size=3240422 2024-03-09T14:36:08.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.902+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=3US3VXG2W723 type=del size=1452248 2024-03-09T14:36:08.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.906+0000 7f73315818c0 1 < completed delete operation key=M1B6O6N8N84T tag=BCRRWYUZ7R80 type=del 2024-03-09T14:36:08.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.906+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=9GO8COLB7FAZ type=del size=61196 2024-03-09T14:36:08.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.910+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=TWG1M71DQJO6 type=del 2024-03-09T14:36:08.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.914+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=UUXKFSE7VS35 type=del size=1882361 2024-03-09T14:36:08.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.918+0000 7f73315818c0 1 < canceled part key=_multipart_9B0RAHZZGLYU.2~2H3ZWM46M5WV.0 size=781415 2024-03-09T14:36:08.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.922+0000 7f73315818c0 1 < canceled multipart upload key=9B0RAHZZGLYU upload=2H3ZWM46M5WV tag=NYU7GH7ZXHBM type=write size=1829991 2024-03-09T14:36:08.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.926+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=3931SM0P3217 type=del size=3569934 2024-03-09T14:36:08.937 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.934+0000 7f73315818c0 1 < suggested update operation key=3ADG4GWVARC1 tag=YPJGNK3V51Q6 type=del size=1611512 2024-03-09T14:36:08.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.938+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~RL3HXT9Y3U9E.0 size=1048576 2024-03-09T14:36:08.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.942+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~RL3HXT9Y3U9E.1 size=747474 2024-03-09T14:36:08.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.942+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=RL3HXT9Y3U9E tag=XOOUPENLGU2J type=write size=2844626 2024-03-09T14:36:08.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.950+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~FTCUPRSESCV5.0 size=1048576 2024-03-09T14:36:08.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.954+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~FTCUPRSESCV5.1 size=296773 2024-03-09T14:36:08.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.958+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=FTCUPRSESCV5 tag=VH75G1728SBT type=write size=2393925 2024-03-09T14:36:08.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.962+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=QUUY62YBV0WX type=del size=2309697 2024-03-09T14:36:08.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.966+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~AAZS63WH1NGZ.0 size=1048576 2024-03-09T14:36:08.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.970+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~AAZS63WH1NGZ.1 size=1048576 2024-03-09T14:36:08.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.970+0000 7f73315818c0 1 < canceled part key=_multipart_M1B6O6N8N84T.2~AAZS63WH1NGZ.2 size=633752 2024-03-09T14:36:08.977 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.974+0000 7f73315818c0 1 < canceled multipart upload key=M1B6O6N8N84T upload=AAZS63WH1NGZ tag=8L6KD4DEIA62 type=write size=3779480 2024-03-09T14:36:08.980 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.978+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=2XSKOYSTN1AN type=del size=186656 2024-03-09T14:36:08.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.982+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=OC7M89I03TY1 type=del 2024-03-09T14:36:08.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.986+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~GHQZ4171859O.0 size=1048576 2024-03-09T14:36:08.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.990+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~GHQZ4171859O.1 size=873381 2024-03-09T14:36:08.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.990+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=GHQZ4171859O tag=85LYTOWB1UE9 type=write size=2970533 2024-03-09T14:36:08.996 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.994+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~H7UOA05OY9XD.0 size=1048576 2024-03-09T14:36:08.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.994+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~H7UOA05OY9XD.1 size=1048576 2024-03-09T14:36:09.001 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:08.998+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~H7UOA05OY9XD.2 size=978773 2024-03-09T14:36:09.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.006+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=H7UOA05OY9XD tag=9A5T8VS7GP5M type=write size=4124501 2024-03-09T14:36:09.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.010+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=N8BLAHPY66LN type=del size=3814553 2024-03-09T14:36:09.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.010+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=TR7LPJEFOCK6 type=del 2024-03-09T14:36:09.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.014+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=WPKBDA2KK46G type=write size=707087 2024-03-09T14:36:09.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.018+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=B0QW7ICRTC59 type=del 2024-03-09T14:36:09.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.022+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=O7EUK38R6ZNK type=write size=320302 2024-03-09T14:36:09.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.026+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~3VCXJQFJEXX0.0 size=459351 2024-03-09T14:36:09.035 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.030+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=3VCXJQFJEXX0 tag=LZYLP9KVDS2F type=write size=1507927 2024-03-09T14:36:09.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.038+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~3ZKWE8BSZILU.0 size=1048576 2024-03-09T14:36:09.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.042+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~3ZKWE8BSZILU.1 size=968662 2024-03-09T14:36:09.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.042+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=3ZKWE8BSZILU tag=Y6NNQKDWFMAE type=write size=3065814 2024-03-09T14:36:09.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.046+0000 7f73315818c0 1 < canceled operation key=M1B6O6N8N84T tag=9KGMVBDLBPAX type=del size=3653534 2024-03-09T14:36:09.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.050+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=2O6FW3OF72UX type=del size=2330111 2024-03-09T14:36:09.055 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.054+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=QX697O4TEPEF type=del 2024-03-09T14:36:09.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.058+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~CIPA5EUBG71P.0 size=830200 2024-03-09T14:36:09.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.058+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=CIPA5EUBG71P tag=BXWXG06EM2UE type=write size=1878776 2024-03-09T14:36:09.065 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.062+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=N90YWFPV748A type=del 2024-03-09T14:36:09.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.066+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=2GXE4FR2RQL0 type=del size=452658 2024-03-09T14:36:09.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.070+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~076ONVWLMV19.0 size=805411 2024-03-09T14:36:09.078 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.074+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=076ONVWLMV19 tag=EM8JFIUWA88M type=write size=1853987 2024-03-09T14:36:09.085 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.082+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~SA84NE5U6ESI.0 size=1048576 2024-03-09T14:36:09.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.086+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~SA84NE5U6ESI.1 size=527932 2024-03-09T14:36:09.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.086+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=SA84NE5U6ESI tag=Q61O2QCEYKMH type=write size=2625084 2024-03-09T14:36:09.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.090+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=3US3VXG2W723 type=del 2024-03-09T14:36:09.099 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.094+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~DGT2R8JI21ZQ.0 size=1048576 2024-03-09T14:36:09.101 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.098+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~DGT2R8JI21ZQ.1 size=1048576 2024-03-09T14:36:09.104 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.102+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~DGT2R8JI21ZQ.2 size=1033054 2024-03-09T14:36:09.104 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.102+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=DGT2R8JI21ZQ tag=SMDXZ3I3FPM3 type=write size=4178782 2024-03-09T14:36:09.108 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.106+0000 7f73315818c0 1 < canceled operation key=QG4O2L526B8E tag=9GO8COLB7FAZ type=del size=61196 2024-03-09T14:36:09.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.110+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~CL92FDOJ5RMQ.0 size=1048576 2024-03-09T14:36:09.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.114+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~CL92FDOJ5RMQ.1 size=1048576 2024-03-09T14:36:09.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.118+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~CL92FDOJ5RMQ.2 size=61353 2024-03-09T14:36:09.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.118+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=CL92FDOJ5RMQ tag=U5AQ0WDIYSDE type=write size=3207081 2024-03-09T14:36:09.123 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.118+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=UUXKFSE7VS35 type=del 2024-03-09T14:36:09.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.122+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=70A9EWE97UD3 type=del size=2801250 2024-03-09T14:36:09.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.126+0000 7f73315818c0 -1 < failed to complete operation key=3ADG4GWVARC1 tag=3931SM0P3217 type=del size=3569934: (22) Invalid argument 2024-03-09T14:36:09.135 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.130+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=J2VF8BYEBHXC type=del size=3384763 2024-03-09T14:36:09.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.134+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~RL3HXT9Y3U9E.0 size=1048576 2024-03-09T14:36:09.141 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.138+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~RL3HXT9Y3U9E.1 size=747474 2024-03-09T14:36:09.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.146+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=RL3HXT9Y3U9E tag=XOOUPENLGU2J type=write size=2844626 2024-03-09T14:36:09.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.150+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~QZRSX7A1FD3J.0 size=1027824 2024-03-09T14:36:09.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.150+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=QZRSX7A1FD3J tag=4LLC693GZRM9 type=write size=2076400 2024-03-09T14:36:09.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.154+0000 7f73315818c0 1 < canceled operation key=0LP8KKU53X0A tag=QUUY62YBV0WX type=del size=2309697 2024-03-09T14:36:09.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.158+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=KHJKKJ7FT3BR type=del size=346712 2024-03-09T14:36:09.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.162+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=2XSKOYSTN1AN type=del 2024-03-09T14:36:09.172 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.170+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~I9MN4HYYE403.0 size=147064 2024-03-09T14:36:09.172 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.170+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=I9MN4HYYE403 tag=XQZMR1VJXGRC type=write size=1195640 2024-03-09T14:36:09.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.174+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~GHQZ4171859O.0 size=1048576 2024-03-09T14:36:09.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.174+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~GHQZ4171859O.1 size=873381 2024-03-09T14:36:09.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.182+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=GHQZ4171859O tag=85LYTOWB1UE9 type=write size=2970533 2024-03-09T14:36:09.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.186+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=846OHE94QBYK type=del size=182476 2024-03-09T14:36:09.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.190+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=N8BLAHPY66LN type=del 2024-03-09T14:36:09.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.194+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~G220RR24P974.0 size=1048576 2024-03-09T14:36:09.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.198+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~G220RR24P974.1 size=1048576 2024-03-09T14:36:09.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.198+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~G220RR24P974.2 size=1004296 2024-03-09T14:36:09.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.198+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=G220RR24P974 tag=S9UWJRWBD414 type=write size=4150024 2024-03-09T14:36:09.207 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.202+0000 7f73315818c0 1 < canceled operation key=P9LI5ESR3N9G tag=WPKBDA2KK46G type=write size=707087 2024-03-09T14:36:09.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.210+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~M544F861UUB9.0 size=1048576 2024-03-09T14:36:09.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.214+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~M544F861UUB9.1 size=483186 2024-03-09T14:36:09.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.214+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=M544F861UUB9 tag=XCAQJ7BYLT3A type=write size=2580338 2024-03-09T14:36:09.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.214+0000 7f73315818c0 1 < completed write operation key=S302JFYEDEL7 tag=O7EUK38R6ZNK type=write size=320302 2024-03-09T14:36:09.222 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.218+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=BWMM1F0DW7PB type=del size=3434678 2024-03-09T14:36:09.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.222+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~3ZKWE8BSZILU.0 size=1048576 2024-03-09T14:36:09.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.226+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~3ZKWE8BSZILU.1 size=968662 2024-03-09T14:36:09.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.230+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=3ZKWE8BSZILU tag=Y6NNQKDWFMAE type=write size=3065814 2024-03-09T14:36:09.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.238+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~WBKPK82YPP33.0 size=1048576 2024-03-09T14:36:09.243 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.238+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~WBKPK82YPP33.1 size=1048576 2024-03-09T14:36:09.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.242+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~WBKPK82YPP33.2 size=760693 2024-03-09T14:36:09.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.242+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=WBKPK82YPP33 tag=ZUS9G8EAIN2B type=write size=3906421 2024-03-09T14:36:09.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.246+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=2O6FW3OF72UX type=del 2024-03-09T14:36:09.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.250+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=VKOWAQWQ45I7 type=del size=2405723 2024-03-09T14:36:09.258 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.254+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~CIPA5EUBG71P.0 size=830200 2024-03-09T14:36:09.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.262+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=CIPA5EUBG71P tag=BXWXG06EM2UE type=write size=1878776 2024-03-09T14:36:09.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.266+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=NHCSJJGPACA5 type=del size=705262 2024-03-09T14:36:09.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.270+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=2GXE4FR2RQL0 type=del 2024-03-09T14:36:09.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.282+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~1867CEBOMEMH.0 size=1048576 2024-03-09T14:36:09.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.282+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~1867CEBOMEMH.1 size=1048576 2024-03-09T14:36:09.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.286+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~1867CEBOMEMH.2 size=306762 2024-03-09T14:36:09.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.286+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=1867CEBOMEMH tag=K5SACE78ND78 type=write size=3452490 2024-03-09T14:36:09.293 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.290+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~SA84NE5U6ESI.0 size=1048576 2024-03-09T14:36:09.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.294+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~SA84NE5U6ESI.1 size=527932 2024-03-09T14:36:09.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.298+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=SA84NE5U6ESI tag=Q61O2QCEYKMH type=write size=2625084 2024-03-09T14:36:09.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.302+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=E3THF9DE42P0 type=del size=2199891 2024-03-09T14:36:09.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.306+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~DGT2R8JI21ZQ.0 size=1048576 2024-03-09T14:36:09.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.310+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~DGT2R8JI21ZQ.1 size=1048576 2024-03-09T14:36:09.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.310+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~DGT2R8JI21ZQ.2 size=1033054 2024-03-09T14:36:09.321 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.318+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=DGT2R8JI21ZQ tag=SMDXZ3I3FPM3 type=write size=4178782 2024-03-09T14:36:09.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.322+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=DALZ6AZ66JV3 type=write size=11804 2024-03-09T14:36:09.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.326+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~CL92FDOJ5RMQ.0 size=1048576 2024-03-09T14:36:09.331 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.330+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~CL92FDOJ5RMQ.1 size=1048576 2024-03-09T14:36:09.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.330+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~CL92FDOJ5RMQ.2 size=61353 2024-03-09T14:36:09.341 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.338+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=CL92FDOJ5RMQ tag=U5AQ0WDIYSDE type=write size=3207081 2024-03-09T14:36:09.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.342+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=ULMRBEYJ327P type=del size=2782368 2024-03-09T14:36:09.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.346+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=70A9EWE97UD3 type=del 2024-03-09T14:36:09.354 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.350+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=EEIL2EKLT1LZ type=del size=2426712 2024-03-09T14:36:09.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.354+0000 7f73315818c0 1 < canceled operation key=P8R5RJI8LGZ9 tag=J2VF8BYEBHXC type=del size=3384763 2024-03-09T14:36:09.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.358+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=MNM9WATGXMM6 type=del size=1547849 2024-03-09T14:36:09.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.362+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~QZRSX7A1FD3J.0 size=1027824 2024-03-09T14:36:09.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.370+0000 7f73315818c0 1 < completed multipart upload key=IA0AB7UVLFEJ upload=QZRSX7A1FD3J tag=4LLC693GZRM9 type=write size=2076400 2024-03-09T14:36:09.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.374+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=CCA4MRIS333L type=del size=396525 2024-03-09T14:36:09.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.378+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=KHJKKJ7FT3BR type=del 2024-03-09T14:36:09.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.382+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=VB327P0RN6P7 type=del size=2524998 2024-03-09T14:36:09.389 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.386+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~I9MN4HYYE403.0 size=147064 2024-03-09T14:36:09.396 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.394+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=I9MN4HYYE403 tag=XQZMR1VJXGRC type=write size=1195640 2024-03-09T14:36:09.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.394+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=1QGDOIH2TSI2 type=del size=2596241 2024-03-09T14:36:09.403 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.398+0000 7f73315818c0 1 < canceled operation key=QG4O2L526B8E tag=846OHE94QBYK type=del size=182476 2024-03-09T14:36:09.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.402+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=ROIRZ7IGBZSB type=write size=380844 2024-03-09T14:36:09.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.402+0000 7f73315818c0 1 < canceled part key=_multipart_PAJYGD2GXHDT.2~G220RR24P974.0 size=1048576 2024-03-09T14:36:09.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.406+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~G220RR24P974.1 size=1048576 2024-03-09T14:36:09.413 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.410+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~G220RR24P974.2 size=1004296 2024-03-09T14:36:09.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.414+0000 7f73315818c0 1 < canceled multipart upload key=PAJYGD2GXHDT upload=G220RR24P974 tag=S9UWJRWBD414 type=write size=4150024 2024-03-09T14:36:09.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.418+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=55IA1Z79V0G5 type=del size=3123351 2024-03-09T14:36:09.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.422+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~M544F861UUB9.0 size=1048576 2024-03-09T14:36:09.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.426+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~M544F861UUB9.1 size=483186 2024-03-09T14:36:09.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.430+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=M544F861UUB9 tag=XCAQJ7BYLT3A type=write size=2580338 2024-03-09T14:36:09.440 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.434+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=5EPLNP6VQG27 type=del size=2046177 2024-03-09T14:36:09.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.442+0000 7f73315818c0 1 < suggested remove operation key=MD7FXGYXB823 tag=BWMM1F0DW7PB type=del 2024-03-09T14:36:09.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.450+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~0AMD38CQH73S.0 size=1048576 2024-03-09T14:36:09.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.454+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~0AMD38CQH73S.1 size=558338 2024-03-09T14:36:09.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.454+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=0AMD38CQH73S tag=8L2M0GHTDMG6 type=write size=2655490 2024-03-09T14:36:09.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.454+0000 7f73315818c0 1 < canceled part key=_multipart_92GWY5B0K4SX.2~WBKPK82YPP33.0 size=1048576 2024-03-09T14:36:09.459 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.454+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~WBKPK82YPP33.1 size=1048576 2024-03-09T14:36:09.462 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.458+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~WBKPK82YPP33.2 size=760693 2024-03-09T14:36:09.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.466+0000 7f73315818c0 1 < canceled multipart upload key=92GWY5B0K4SX upload=WBKPK82YPP33 tag=ZUS9G8EAIN2B type=write size=3906421 2024-03-09T14:36:09.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.470+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~TLKJVS5M7C3D.0 size=1048576 2024-03-09T14:36:09.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.474+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~TLKJVS5M7C3D.1 size=1048576 2024-03-09T14:36:09.480 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.478+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~TLKJVS5M7C3D.2 size=626490 2024-03-09T14:36:09.480 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.478+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=TLKJVS5M7C3D tag=S4OSBKGO2BL0 type=write size=3772218 2024-03-09T14:36:09.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.486+0000 7f73315818c0 1 < suggested update operation key=FYFVW9R0M1PX tag=VKOWAQWQ45I7 type=del size=2405723 2024-03-09T14:36:09.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.490+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~1J0CACRGMDXF.0 size=1048576 2024-03-09T14:36:09.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.494+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~1J0CACRGMDXF.1 size=642601 2024-03-09T14:36:09.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.494+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=1J0CACRGMDXF tag=N0C7W18G1MJA type=write size=2739753 2024-03-09T14:36:09.500 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.498+0000 7f73315818c0 1 < canceled operation key=9B0RAHZZGLYU tag=NHCSJJGPACA5 type=del size=705262 2024-03-09T14:36:09.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.502+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=HW307EHPNZFD type=del size=3143992 2024-03-09T14:36:09.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.506+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~1867CEBOMEMH.0 size=1048576 2024-03-09T14:36:09.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.506+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~1867CEBOMEMH.1 size=1048576 2024-03-09T14:36:09.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.506+0000 7f73315818c0 1 < canceled part key=_multipart_PXFZLDSAR1EG.2~1867CEBOMEMH.2 size=306762 2024-03-09T14:36:09.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.514+0000 7f73315818c0 1 < canceled multipart upload key=PXFZLDSAR1EG upload=1867CEBOMEMH tag=K5SACE78ND78 type=write size=3452490 2024-03-09T14:36:09.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.518+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=9W8BEATGUXSQ type=del size=3335175 2024-03-09T14:36:09.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.522+0000 7f73315818c0 1 < suggested remove operation key=4E3QMBHEAEHO tag=E3THF9DE42P0 type=del 2024-03-09T14:36:09.534 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.530+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~B2RABQZHVWXQ.0 size=1048576 2024-03-09T14:36:09.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.534+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~B2RABQZHVWXQ.1 size=312687 2024-03-09T14:36:09.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.534+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=B2RABQZHVWXQ tag=Y9TQ36M7N1WB type=write size=2409839 2024-03-09T14:36:09.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.538+0000 7f73315818c0 1 < completed write operation key=M8F9TA80VME6 tag=DALZ6AZ66JV3 type=write size=11804 2024-03-09T14:36:09.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.542+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=YOCD3EJYLH04 type=del size=1319409 2024-03-09T14:36:09.547 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.546+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=ULMRBEYJ327P type=del 2024-03-09T14:36:09.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.546+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=GUYOUIQ4516K type=del size=2293964 2024-03-09T14:36:09.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.550+0000 7f73315818c0 -1 < no bi entry to suggest for operation key=MD7FXGYXB823 tag=EEIL2EKLT1LZ type=del size=2426712: (2) No such file or directory 2024-03-09T14:36:09.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.554+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~30AP639VPB28.0 size=1048576 2024-03-09T14:36:09.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.558+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~30AP639VPB28.1 size=351758 2024-03-09T14:36:09.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.558+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=30AP639VPB28 tag=KC5X0BIV28ZL type=write size=2448910 2024-03-09T14:36:09.565 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.562+0000 7f73315818c0 -1 < failed to complete operation key=MD7FXGYXB823 tag=MNM9WATGXMM6 type=del size=1547849: (22) Invalid argument 2024-03-09T14:36:09.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.570+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~WX8FG67AVEDU.0 size=643468 2024-03-09T14:36:09.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.570+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=WX8FG67AVEDU tag=ZLOL26G8I7Y9 type=write size=1692044 2024-03-09T14:36:09.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.574+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=CCA4MRIS333L type=del 2024-03-09T14:36:09.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.574+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=Q6XU6EC43OEC type=del size=981121 2024-03-09T14:36:09.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.578+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=VB327P0RN6P7 type=del 2024-03-09T14:36:09.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.582+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=CJO35US1Q7D6 type=del size=3994461 2024-03-09T14:36:09.590 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.586+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=1QGDOIH2TSI2 type=del 2024-03-09T14:36:09.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.594+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~KXEXOMVO5DZ4.0 size=1048576 2024-03-09T14:36:09.599 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.594+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~KXEXOMVO5DZ4.1 size=248534 2024-03-09T14:36:09.599 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.594+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=KXEXOMVO5DZ4 tag=REWW46R4Q0SL type=write size=2345686 2024-03-09T14:36:09.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.598+0000 7f73315818c0 1 < canceled operation key=P9LI5ESR3N9G tag=ROIRZ7IGBZSB type=write size=380844 2024-03-09T14:36:09.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.602+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=828108EZLL0X type=del size=3905273 2024-03-09T14:36:09.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.606+0000 7f73315818c0 -1 < no bi entry to suggest for operation key=MD7FXGYXB823 tag=55IA1Z79V0G5 type=del size=3123351: (2) No such file or directory 2024-03-09T14:36:09.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.606+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=1XU0TIKG25W2 type=del size=3120489 2024-03-09T14:36:09.615 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.610+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=5EPLNP6VQG27 type=del 2024-03-09T14:36:09.618 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.614+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=CFZUO5X4WWWB type=del size=1717672 2024-03-09T14:36:09.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.618+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~0AMD38CQH73S.0 size=1048576 2024-03-09T14:36:09.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.626+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~0AMD38CQH73S.1 size=558338 2024-03-09T14:36:09.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.630+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=0AMD38CQH73S tag=8L2M0GHTDMG6 type=write size=2655490 2024-03-09T14:36:09.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.638+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~RF5NSC3Y4WQT.0 size=798168 2024-03-09T14:36:09.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.638+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=RF5NSC3Y4WQT tag=9OO1J8Q3S93Q type=write size=1846744 2024-03-09T14:36:09.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.638+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~TLKJVS5M7C3D.0 size=1048576 2024-03-09T14:36:09.646 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.642+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~TLKJVS5M7C3D.1 size=1048576 2024-03-09T14:36:09.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.646+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~TLKJVS5M7C3D.2 size=626490 2024-03-09T14:36:09.654 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.650+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=TLKJVS5M7C3D tag=S4OSBKGO2BL0 type=write size=3772218 2024-03-09T14:36:09.660 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.658+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~R5YECSF37FR5.0 size=175926 2024-03-09T14:36:09.660 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.658+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=R5YECSF37FR5 tag=EY1KVMBDHLQ8 type=write size=1224502 2024-03-09T14:36:09.664 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.662+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~1J0CACRGMDXF.0 size=1048576 2024-03-09T14:36:09.664 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.662+0000 7f73315818c0 1 < canceled part key=_multipart_9B0RAHZZGLYU.2~1J0CACRGMDXF.1 size=642601 2024-03-09T14:36:09.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.666+0000 7f73315818c0 1 < canceled multipart upload key=9B0RAHZZGLYU upload=1J0CACRGMDXF tag=N0C7W18G1MJA type=write size=2739753 2024-03-09T14:36:09.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.674+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~LC9GH5G9U4X1.0 size=1048576 2024-03-09T14:36:09.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.678+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~LC9GH5G9U4X1.1 size=202845 2024-03-09T14:36:09.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.678+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=LC9GH5G9U4X1 tag=1XQWXWWUUC29 type=write size=2299997 2024-03-09T14:36:09.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.678+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=HW307EHPNZFD type=del 2024-03-09T14:36:09.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.682+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=0SEZIJ9RMS6Z type=del size=2328283 2024-03-09T14:36:09.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.686+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=9W8BEATGUXSQ type=del 2024-03-09T14:36:09.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.694+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~KWRLWD99LSS2.0 size=1048576 2024-03-09T14:36:09.699 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.698+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~KWRLWD99LSS2.1 size=1048576 2024-03-09T14:36:09.702 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.698+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~KWRLWD99LSS2.2 size=381672 2024-03-09T14:36:09.702 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.698+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=KWRLWD99LSS2 tag=M5YRYGKJIKR9 type=write size=3527400 2024-03-09T14:36:09.703 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.698+0000 7f73315818c0 1 < canceled part key=_multipart_K3JNN1NCKBXN.2~B2RABQZHVWXQ.0 size=1048576 2024-03-09T14:36:09.706 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.702+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~B2RABQZHVWXQ.1 size=312687 2024-03-09T14:36:09.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.710+0000 7f73315818c0 1 < canceled multipart upload key=K3JNN1NCKBXN upload=B2RABQZHVWXQ tag=Y9TQ36M7N1WB type=write size=2409839 2024-03-09T14:36:09.715 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.710+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=6G2PQRPRP5W8 type=del size=1785018 2024-03-09T14:36:09.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.714+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=YOCD3EJYLH04 type=del 2024-03-09T14:36:09.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.718+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=DR7R5S2YFGIT type=del size=1363041 2024-03-09T14:36:09.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.726+0000 7f73315818c0 1 < suggested update operation key=0WGGMHTEF2LB tag=GUYOUIQ4516K type=del size=2293964 2024-03-09T14:36:09.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.730+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=WLJ7Q5EYX7Q5 type=del size=3888318 2024-03-09T14:36:09.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.734+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~30AP639VPB28.0 size=1048576 2024-03-09T14:36:09.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.738+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~30AP639VPB28.1 size=351758 2024-03-09T14:36:09.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.742+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=30AP639VPB28 tag=KC5X0BIV28ZL type=write size=2448910 2024-03-09T14:36:09.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.746+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=PC4PV02F1MI3 type=del size=183021 2024-03-09T14:36:09.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.750+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~WX8FG67AVEDU.0 size=643468 2024-03-09T14:36:09.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.754+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=WX8FG67AVEDU tag=ZLOL26G8I7Y9 type=write size=1692044 2024-03-09T14:36:09.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.758+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=CGKKSDXXFQNF type=write size=77203 2024-03-09T14:36:09.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.762+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=Q6XU6EC43OEC type=del 2024-03-09T14:36:09.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.770+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~DVKZN3HMVJMT.0 size=1048576 2024-03-09T14:36:09.775 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.770+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~DVKZN3HMVJMT.1 size=1048576 2024-03-09T14:36:09.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.774+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~DVKZN3HMVJMT.2 size=962760 2024-03-09T14:36:09.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.774+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=DVKZN3HMVJMT tag=V5YT3X8I242A type=write size=4108488 2024-03-09T14:36:09.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.778+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=CJO35US1Q7D6 type=del 2024-03-09T14:36:09.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.786+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~D1KXVC3XEJZG.0 size=340620 2024-03-09T14:36:09.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.786+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=D1KXVC3XEJZG tag=SWS44WI0ZR95 type=write size=1389196 2024-03-09T14:36:09.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.790+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~KXEXOMVO5DZ4.0 size=1048576 2024-03-09T14:36:09.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.790+0000 7f73315818c0 1 < canceled part key=_multipart_PXFZLDSAR1EG.2~KXEXOMVO5DZ4.1 size=248534 2024-03-09T14:36:09.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.794+0000 7f73315818c0 1 < canceled multipart upload key=PXFZLDSAR1EG upload=KXEXOMVO5DZ4 tag=REWW46R4Q0SL type=write size=2345686 2024-03-09T14:36:09.803 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.798+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=QKS0BI7JSXXS type=del size=1428018 2024-03-09T14:36:09.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.802+0000 7f73315818c0 1 < canceled operation key=W6KVP8EQFQ8J tag=828108EZLL0X type=del size=3905273 2024-03-09T14:36:09.809 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.806+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=Q4E1VZ0CMEEM type=del size=3412638 2024-03-09T14:36:09.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.810+0000 7f73315818c0 1 < canceled operation key=QG4O2L526B8E tag=1XU0TIKG25W2 type=del size=3120489 2024-03-09T14:36:09.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.814+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~KHM12Y3BARAU.0 size=1048576 2024-03-09T14:36:09.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.818+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~KHM12Y3BARAU.1 size=1048576 2024-03-09T14:36:09.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.822+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~KHM12Y3BARAU.2 size=529822 2024-03-09T14:36:09.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.822+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=KHM12Y3BARAU tag=YA5EMD02TKIA type=write size=3675550 2024-03-09T14:36:09.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.826+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=CFZUO5X4WWWB type=del 2024-03-09T14:36:09.832 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.830+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=2JTQDJM67B2P type=del size=1590904 2024-03-09T14:36:09.835 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.830+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~RF5NSC3Y4WQT.0 size=798168 2024-03-09T14:36:09.841 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.838+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=RF5NSC3Y4WQT tag=9OO1J8Q3S93Q type=write size=1846744 2024-03-09T14:36:09.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.842+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~4OTJ7ST79K1R.0 size=1048576 2024-03-09T14:36:09.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.846+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~4OTJ7ST79K1R.1 size=942530 2024-03-09T14:36:09.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.846+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=4OTJ7ST79K1R tag=JD0USJLTT1A5 type=write size=3039682 2024-03-09T14:36:09.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.850+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~R5YECSF37FR5.0 size=175926 2024-03-09T14:36:09.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.858+0000 7f73315818c0 1 < completed multipart upload key=0IB00CZ92VV4 upload=R5YECSF37FR5 tag=EY1KVMBDHLQ8 type=write size=1224502 2024-03-09T14:36:09.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.858+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=L5NFCZNNDMQF type=del size=1904536 2024-03-09T14:36:09.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.862+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~LC9GH5G9U4X1.0 size=1048576 2024-03-09T14:36:09.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.866+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~LC9GH5G9U4X1.1 size=202845 2024-03-09T14:36:09.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.874+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=LC9GH5G9U4X1 tag=1XQWXWWUUC29 type=write size=2299997 2024-03-09T14:36:09.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.878+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=ZXTFEHL4JUBU type=del size=2550290 2024-03-09T14:36:09.884 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.882+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=0SEZIJ9RMS6Z type=del 2024-03-09T14:36:09.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.886+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=BSWHFCXFRHI3 type=del size=148796 2024-03-09T14:36:09.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.886+0000 7f73315818c0 1 < canceled part key=_multipart_K3JNN1NCKBXN.2~KWRLWD99LSS2.0 size=1048576 2024-03-09T14:36:09.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.886+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~KWRLWD99LSS2.1 size=1048576 2024-03-09T14:36:09.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.886+0000 7f73315818c0 1 < canceled part key=_multipart_K3JNN1NCKBXN.2~KWRLWD99LSS2.2 size=381672 2024-03-09T14:36:09.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.894+0000 7f73315818c0 1 < canceled multipart upload key=K3JNN1NCKBXN upload=KWRLWD99LSS2 tag=M5YRYGKJIKR9 type=write size=3527400 2024-03-09T14:36:09.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.898+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~4BPTVRAWFLIH.0 size=1048576 2024-03-09T14:36:09.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.902+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~4BPTVRAWFLIH.1 size=1048576 2024-03-09T14:36:09.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.906+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~4BPTVRAWFLIH.2 size=818126 2024-03-09T14:36:09.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.906+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=4BPTVRAWFLIH tag=LIP7HICY5KDA type=write size=3963854 2024-03-09T14:36:09.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.906+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=6G2PQRPRP5W8 type=del 2024-03-09T14:36:09.917 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.914+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~XXDDOW4EUE3E.0 size=427998 2024-03-09T14:36:09.917 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.914+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=XXDDOW4EUE3E tag=UMEI2OQKZMFU type=write size=1476574 2024-03-09T14:36:09.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.922+0000 7f73315818c0 1 < suggested remove operation key=0WGGMHTEF2LB tag=DR7R5S2YFGIT type=del 2024-03-09T14:36:09.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.926+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~AJRXJLRYEGU7.0 size=1048576 2024-03-09T14:36:09.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.930+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~AJRXJLRYEGU7.1 size=1048576 2024-03-09T14:36:09.935 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.934+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~AJRXJLRYEGU7.2 size=103463 2024-03-09T14:36:09.935 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.934+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=AJRXJLRYEGU7 tag=F2W1SFIAKQ7N type=write size=3249191 2024-03-09T14:36:09.939 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.934+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=WLJ7Q5EYX7Q5 type=del 2024-03-09T14:36:09.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.942+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~N0I3JZ49FNDK.0 size=1048576 2024-03-09T14:36:09.950 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.946+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~N0I3JZ49FNDK.1 size=1048576 2024-03-09T14:36:09.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.950+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~N0I3JZ49FNDK.2 size=1032729 2024-03-09T14:36:09.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.950+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=N0I3JZ49FNDK tag=UG39D4NDC083 type=write size=4178457 2024-03-09T14:36:09.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.954+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=PC4PV02F1MI3 type=del 2024-03-09T14:36:09.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.958+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~D0NEWBPVE6AT.0 size=1048576 2024-03-09T14:36:09.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.962+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~D0NEWBPVE6AT.1 size=1020802 2024-03-09T14:36:09.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.962+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=D0NEWBPVE6AT tag=G8I1Z467LPFU type=write size=3117954 2024-03-09T14:36:09.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.966+0000 7f73315818c0 1 < completed write operation key=IA0AB7UVLFEJ tag=CGKKSDXXFQNF type=write size=77203 2024-03-09T14:36:09.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.970+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=IXA7G0HISU60 type=write size=120892 2024-03-09T14:36:09.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.970+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~DVKZN3HMVJMT.0 size=1048576 2024-03-09T14:36:09.977 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.974+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~DVKZN3HMVJMT.1 size=1048576 2024-03-09T14:36:09.980 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.978+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~DVKZN3HMVJMT.2 size=962760 2024-03-09T14:36:09.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.982+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=DVKZN3HMVJMT tag=V5YT3X8I242A type=write size=4108488 2024-03-09T14:36:09.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.986+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=6HJ8O3CKZLAT type=del size=364548 2024-03-09T14:36:09.994 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.990+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~D1KXVC3XEJZG.0 size=340620 2024-03-09T14:36:09.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:09.994+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=D1KXVC3XEJZG tag=SWS44WI0ZR95 type=write size=1389196 2024-03-09T14:36:10.005 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.002+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~VR1O15H09A56.0 size=1048576 2024-03-09T14:36:10.007 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.006+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~VR1O15H09A56.1 size=1048576 2024-03-09T14:36:10.010 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.006+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~VR1O15H09A56.2 size=85436 2024-03-09T14:36:10.010 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.006+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=VR1O15H09A56 tag=A8Q1ME3RDK7J type=write size=3231164 2024-03-09T14:36:10.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.010+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=QKS0BI7JSXXS type=del 2024-03-09T14:36:10.016 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.014+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=WNOY259HI87B type=del size=2981645 2024-03-09T14:36:10.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.014+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=Q4E1VZ0CMEEM type=del 2024-03-09T14:36:10.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.018+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=XA39I9YWW6QJ type=write size=134842 2024-03-09T14:36:10.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.022+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~KHM12Y3BARAU.0 size=1048576 2024-03-09T14:36:10.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.022+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~KHM12Y3BARAU.1 size=1048576 2024-03-09T14:36:10.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.026+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~KHM12Y3BARAU.2 size=529822 2024-03-09T14:36:10.035 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.030+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=KHM12Y3BARAU tag=YA5EMD02TKIA type=write size=3675550 2024-03-09T14:36:10.038 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.034+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=KYVX92EGKLQZ type=write size=664153 2024-03-09T14:36:10.044 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.042+0000 7f73315818c0 1 < suggested update operation key=S302JFYEDEL7 tag=2JTQDJM67B2P type=del size=1590904 2024-03-09T14:36:10.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.042+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=AVB8UQRS7B0V type=del size=4029638 2024-03-09T14:36:10.050 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.046+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~4OTJ7ST79K1R.0 size=1048576 2024-03-09T14:36:10.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.050+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~4OTJ7ST79K1R.1 size=942530 2024-03-09T14:36:10.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.054+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=4OTJ7ST79K1R tag=JD0USJLTT1A5 type=write size=3039682 2024-03-09T14:36:10.064 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.058+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~SID8VD1QQ2OV.0 size=1048576 2024-03-09T14:36:10.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.062+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~SID8VD1QQ2OV.1 size=446869 2024-03-09T14:36:10.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.062+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=SID8VD1QQ2OV tag=D1KQWGQZ084E type=write size=2544021 2024-03-09T14:36:10.073 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.070+0000 7f73315818c0 1 < suggested remove operation key=PAJYGD2GXHDT tag=L5NFCZNNDMQF type=del 2024-03-09T14:36:10.080 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.078+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~D7IGHM8CYXTZ.0 size=906321 2024-03-09T14:36:10.080 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.078+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=D7IGHM8CYXTZ tag=AA45DSN1FF20 type=write size=1954897 2024-03-09T14:36:10.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.082+0000 7f73315818c0 1 < suggested update operation key=QG4O2L526B8E tag=ZXTFEHL4JUBU type=del size=2550290 2024-03-09T14:36:10.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.086+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=32M1SD3T9NEX type=del size=3849470 2024-03-09T14:36:10.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.090+0000 7f73315818c0 1 < canceled operation key=1NYNL8M5DVER tag=BSWHFCXFRHI3 type=del size=148796 2024-03-09T14:36:10.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.094+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=64T32Q4LSARY type=write size=899856 2024-03-09T14:36:10.100 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.098+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~4BPTVRAWFLIH.0 size=1048576 2024-03-09T14:36:10.102 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.098+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~4BPTVRAWFLIH.1 size=1048576 2024-03-09T14:36:10.105 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.102+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~4BPTVRAWFLIH.2 size=818126 2024-03-09T14:36:10.109 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.106+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=4BPTVRAWFLIH tag=LIP7HICY5KDA type=write size=3963854 2024-03-09T14:36:10.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.110+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=FH4VUY28D206 type=del size=2853724 2024-03-09T14:36:10.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.114+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~XXDDOW4EUE3E.0 size=427998 2024-03-09T14:36:10.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.118+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=XXDDOW4EUE3E tag=UMEI2OQKZMFU type=write size=1476574 2024-03-09T14:36:10.126 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.122+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~FQOGEK3H5LJM.0 size=844289 2024-03-09T14:36:10.126 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.122+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=FQOGEK3H5LJM tag=F51BMWKC3TCW type=write size=1892865 2024-03-09T14:36:10.126 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.122+0000 7f73315818c0 1 < canceled part key=_multipart_162AOMHO2BQL.2~AJRXJLRYEGU7.0 size=1048576 2024-03-09T14:36:10.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.126+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~AJRXJLRYEGU7.1 size=1048576 2024-03-09T14:36:10.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.130+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~AJRXJLRYEGU7.2 size=103463 2024-03-09T14:36:10.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.134+0000 7f73315818c0 1 < canceled multipart upload key=162AOMHO2BQL upload=AJRXJLRYEGU7 tag=F2W1SFIAKQ7N type=write size=3249191 2024-03-09T14:36:10.140 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.138+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=SD5XUSXN277Z type=del size=3521098 2024-03-09T14:36:10.143 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.138+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~N0I3JZ49FNDK.0 size=1048576 2024-03-09T14:36:10.143 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.138+0000 7f73315818c0 1 < canceled part key=_multipart_4P89277EEYJ3.2~N0I3JZ49FNDK.1 size=1048576 2024-03-09T14:36:10.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.142+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~N0I3JZ49FNDK.2 size=1032729 2024-03-09T14:36:10.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.150+0000 7f73315818c0 1 < canceled multipart upload key=4P89277EEYJ3 upload=N0I3JZ49FNDK tag=UG39D4NDC083 type=write size=4178457 2024-03-09T14:36:10.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.150+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=8U4UPX6KUENY type=write size=387065 2024-03-09T14:36:10.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.154+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~D0NEWBPVE6AT.0 size=1048576 2024-03-09T14:36:10.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.154+0000 7f73315818c0 1 < canceled part key=_multipart_DQOY1JJ1YQ2W.2~D0NEWBPVE6AT.1 size=1020802 2024-03-09T14:36:10.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.162+0000 7f73315818c0 1 < canceled multipart upload key=DQOY1JJ1YQ2W upload=D0NEWBPVE6AT tag=G8I1Z467LPFU type=write size=3117954 2024-03-09T14:36:10.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.166+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=RHQM18OLQ4H3 type=del size=3705181 2024-03-09T14:36:10.171 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.166+0000 7f73315818c0 1 < canceled operation key=8O5OP8C5NX12 tag=IXA7G0HISU60 type=write size=120892 2024-03-09T14:36:10.174 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.170+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=6A3I6EYPQQDV type=del size=1789044 2024-03-09T14:36:10.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.174+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=6HJ8O3CKZLAT type=del 2024-03-09T14:36:10.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.178+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=E1EVYFT553P1 type=del size=3758192 2024-03-09T14:36:10.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.178+0000 7f73315818c0 1 < canceled part key=_multipart_LZC7NPF0GRO7.2~VR1O15H09A56.0 size=1048576 2024-03-09T14:36:10.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.178+0000 7f73315818c0 1 < canceled part key=_multipart_LZC7NPF0GRO7.2~VR1O15H09A56.1 size=1048576 2024-03-09T14:36:10.183 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.182+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~VR1O15H09A56.2 size=85436 2024-03-09T14:36:10.189 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.186+0000 7f73315818c0 1 < canceled multipart upload key=LZC7NPF0GRO7 upload=VR1O15H09A56 tag=A8Q1ME3RDK7J type=write size=3231164 2024-03-09T14:36:10.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.190+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~P3QUDJ348NKU.0 size=1048576 2024-03-09T14:36:10.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.194+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~P3QUDJ348NKU.1 size=161010 2024-03-09T14:36:10.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.194+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=P3QUDJ348NKU tag=ZDZST0KG1H2K type=write size=2258162 2024-03-09T14:36:10.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.198+0000 7f73315818c0 1 < canceled operation key=XELG1AWATMVL tag=WNOY259HI87B type=del size=2981645 2024-03-09T14:36:10.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.202+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=2H1JTDOIKE2X type=write size=715064 2024-03-09T14:36:10.207 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.202+0000 7f73315818c0 1 < completed write operation key=P8R5RJI8LGZ9 tag=XA39I9YWW6QJ type=write size=134842 2024-03-09T14:36:10.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.206+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=9MUD4KXKX10T type=del size=3793584 2024-03-09T14:36:10.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.210+0000 7f73315818c0 1 < completed write operation key=IA0AB7UVLFEJ tag=KYVX92EGKLQZ type=write size=664153 2024-03-09T14:36:10.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.214+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=NQGFU8L6NCYJ type=del size=2766181 2024-03-09T14:36:10.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.218+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=AVB8UQRS7B0V type=del 2024-03-09T14:36:10.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.222+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=XWBH27BR6LA5 type=write size=804263 2024-03-09T14:36:10.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.226+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~SID8VD1QQ2OV.0 size=1048576 2024-03-09T14:36:10.232 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.230+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~SID8VD1QQ2OV.1 size=446869 2024-03-09T14:36:10.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.234+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=SID8VD1QQ2OV tag=D1KQWGQZ084E type=write size=2544021 2024-03-09T14:36:10.244 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.242+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~AU2GO2O3FTG0.0 size=1048576 2024-03-09T14:36:10.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.242+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~AU2GO2O3FTG0.1 size=388987 2024-03-09T14:36:10.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.242+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=AU2GO2O3FTG0 tag=XVB8XT7BKPON type=write size=2486139 2024-03-09T14:36:10.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.250+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~D7IGHM8CYXTZ.0 size=906321 2024-03-09T14:36:10.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.258+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=D7IGHM8CYXTZ tag=AA45DSN1FF20 type=write size=1954897 2024-03-09T14:36:10.267 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.262+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~1Q3WZ3UG72UV.0 size=1048576 2024-03-09T14:36:10.270 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.266+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~1Q3WZ3UG72UV.1 size=1048576 2024-03-09T14:36:10.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.270+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~1Q3WZ3UG72UV.2 size=1024826 2024-03-09T14:36:10.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.270+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=1Q3WZ3UG72UV tag=QA7CYS2E62KQ type=write size=4170554 2024-03-09T14:36:10.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.274+0000 7f73315818c0 1 < suggested update operation key=92GWY5B0K4SX tag=32M1SD3T9NEX type=del size=3849470 2024-03-09T14:36:10.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.282+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~8P91ZHVE76SZ.0 size=200688 2024-03-09T14:36:10.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.282+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=8P91ZHVE76SZ tag=4MKQ7S92Y2M9 type=write size=1249264 2024-03-09T14:36:10.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.290+0000 7f73315818c0 1 < suggested update operation key=IA0AB7UVLFEJ tag=64T32Q4LSARY type=write size=899856 2024-03-09T14:36:10.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.294+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~4XLHKGVAM7Q8.0 size=1048576 2024-03-09T14:36:10.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.298+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~4XLHKGVAM7Q8.1 size=773068 2024-03-09T14:36:10.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.298+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=4XLHKGVAM7Q8 tag=H4I28UXT43RX type=write size=2870220 2024-03-09T14:36:10.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.302+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=FH4VUY28D206 type=del 2024-03-09T14:36:10.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.306+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=J9H6TXHB6HC6 type=del size=1487302 2024-03-09T14:36:10.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.310+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~FQOGEK3H5LJM.0 size=844289 2024-03-09T14:36:10.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.314+0000 7f73315818c0 1 < completed multipart upload key=LZC7NPF0GRO7 upload=FQOGEK3H5LJM tag=F51BMWKC3TCW type=write size=1892865 2024-03-09T14:36:10.324 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.322+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~PWOF5VGRBH42.0 size=1048576 2024-03-09T14:36:10.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.322+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~PWOF5VGRBH42.1 size=1048576 2024-03-09T14:36:10.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.326+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~PWOF5VGRBH42.2 size=490516 2024-03-09T14:36:10.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.326+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=PWOF5VGRBH42 tag=JCETL72DYJG0 type=write size=3636244 2024-03-09T14:36:10.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.330+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=SD5XUSXN277Z type=del 2024-03-09T14:36:10.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.334+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~56DZ564I9J68.0 size=195091 2024-03-09T14:36:10.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.334+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=56DZ564I9J68 tag=FYXA4CT8K10L type=write size=1243667 2024-03-09T14:36:10.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.338+0000 7f73315818c0 1 < completed write operation key=3ADG4GWVARC1 tag=8U4UPX6KUENY type=write size=387065 2024-03-09T14:36:10.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.342+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=9PF0MDC0YQSY type=del size=2395082 2024-03-09T14:36:10.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.346+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=RHQM18OLQ4H3 type=del 2024-03-09T14:36:10.353 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.350+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=F5IZ3IA0OEWY type=del size=1487678 2024-03-09T14:36:10.357 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.354+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=6A3I6EYPQQDV type=del 2024-03-09T14:36:10.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.358+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=S22JZODWFILY type=write size=89862 2024-03-09T14:36:10.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.362+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=E1EVYFT553P1 type=del 2024-03-09T14:36:10.370 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.366+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~CSYO1GXDYH1Y.0 size=1048576 2024-03-09T14:36:10.372 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.370+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~CSYO1GXDYH1Y.1 size=1048576 2024-03-09T14:36:10.375 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.374+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~CSYO1GXDYH1Y.2 size=597261 2024-03-09T14:36:10.375 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.374+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=CSYO1GXDYH1Y tag=UE8SMZG21PWJ type=write size=3742989 2024-03-09T14:36:10.379 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.374+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~P3QUDJ348NKU.0 size=1048576 2024-03-09T14:36:10.381 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.378+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~P3QUDJ348NKU.1 size=161010 2024-03-09T14:36:10.387 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.382+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=P3QUDJ348NKU tag=ZDZST0KG1H2K type=write size=2258162 2024-03-09T14:36:10.393 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.390+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~OXMJXNRXQ1JP.0 size=159710 2024-03-09T14:36:10.393 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.390+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=OXMJXNRXQ1JP tag=FY0318217TAO type=write size=1208286 2024-03-09T14:36:10.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.398+0000 7f73315818c0 1 < suggested remove operation key=PAJYGD2GXHDT tag=2H1JTDOIKE2X type=write 2024-03-09T14:36:10.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.402+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=WG2PBQ9M1I0F type=del size=3966235 2024-03-09T14:36:10.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.406+0000 7f73315818c0 1 < canceled operation key=0LP8KKU53X0A tag=9MUD4KXKX10T type=del size=3793584 2024-03-09T14:36:10.416 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.414+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~Q0JH89423ABR.0 size=422103 2024-03-09T14:36:10.416 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.414+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=Q0JH89423ABR tag=DT1K00B5VS69 type=write size=1470679 2024-03-09T14:36:10.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.418+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=NQGFU8L6NCYJ type=del 2024-03-09T14:36:10.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.422+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~ITUET1B38HXT.0 size=1048576 2024-03-09T14:36:10.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.426+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~ITUET1B38HXT.1 size=65074 2024-03-09T14:36:10.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.426+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=ITUET1B38HXT tag=JLWDQX9VZSGW type=write size=2162226 2024-03-09T14:36:10.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.430+0000 7f73315818c0 1 < completed write operation key=OU0730HGI176 tag=XWBH27BR6LA5 type=write size=804263 2024-03-09T14:36:10.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.430+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=5IHT8PPH34VS type=del size=3513450 2024-03-09T14:36:10.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.434+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~AU2GO2O3FTG0.0 size=1048576 2024-03-09T14:36:10.440 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.438+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~AU2GO2O3FTG0.1 size=388987 2024-03-09T14:36:10.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.442+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=AU2GO2O3FTG0 tag=XVB8XT7BKPON type=write size=2486139 2024-03-09T14:36:10.449 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.446+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=5NM9EKBRUC3J type=del size=3648884 2024-03-09T14:36:10.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.450+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~1Q3WZ3UG72UV.0 size=1048576 2024-03-09T14:36:10.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.454+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~1Q3WZ3UG72UV.1 size=1048576 2024-03-09T14:36:10.458 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.454+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~1Q3WZ3UG72UV.2 size=1024826 2024-03-09T14:36:10.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.462+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=1Q3WZ3UG72UV tag=QA7CYS2E62KQ type=write size=4170554 2024-03-09T14:36:10.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.466+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~8N250F058PYC.0 size=448447 2024-03-09T14:36:10.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.466+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=8N250F058PYC tag=P6W1F5UTKBUX type=write size=1497023 2024-03-09T14:36:10.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.466+0000 7f73315818c0 1 < canceled part key=_multipart_DQOY1JJ1YQ2W.2~8P91ZHVE76SZ.0 size=200688 2024-03-09T14:36:10.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.474+0000 7f73315818c0 1 < canceled multipart upload key=DQOY1JJ1YQ2W upload=8P91ZHVE76SZ tag=4MKQ7S92Y2M9 type=write size=1249264 2024-03-09T14:36:10.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.478+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~CSY6M42116KS.0 size=1048576 2024-03-09T14:36:10.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.482+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~CSY6M42116KS.1 size=1048576 2024-03-09T14:36:10.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.486+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~CSY6M42116KS.2 size=472226 2024-03-09T14:36:10.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.486+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=CSY6M42116KS tag=EBW5D1L30AZK type=write size=3617954 2024-03-09T14:36:10.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.486+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~4XLHKGVAM7Q8.0 size=1048576 2024-03-09T14:36:10.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.490+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~4XLHKGVAM7Q8.1 size=773068 2024-03-09T14:36:10.500 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.498+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=4XLHKGVAM7Q8 tag=H4I28UXT43RX type=write size=2870220 2024-03-09T14:36:10.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.502+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~HI5U53SALZW9.0 size=1048576 2024-03-09T14:36:10.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.506+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~HI5U53SALZW9.1 size=1048576 2024-03-09T14:36:10.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.506+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~HI5U53SALZW9.2 size=850339 2024-03-09T14:36:10.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.506+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=HI5U53SALZW9 tag=WHUNGCF9ZUJW type=write size=3996067 2024-03-09T14:36:10.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.510+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=J9H6TXHB6HC6 type=del 2024-03-09T14:36:10.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.518+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~137S1WWG30OA.0 size=1048576 2024-03-09T14:36:10.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.522+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~137S1WWG30OA.1 size=772949 2024-03-09T14:36:10.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.522+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=137S1WWG30OA tag=X33YJT4QWESR type=write size=2870101 2024-03-09T14:36:10.528 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.522+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~PWOF5VGRBH42.0 size=1048576 2024-03-09T14:36:10.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.526+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~PWOF5VGRBH42.1 size=1048576 2024-03-09T14:36:10.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.530+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~PWOF5VGRBH42.2 size=490516 2024-03-09T14:36:10.538 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.534+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=PWOF5VGRBH42 tag=JCETL72DYJG0 type=write size=3636244 2024-03-09T14:36:10.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.538+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=HJH47OQ4D317 type=del size=663303 2024-03-09T14:36:10.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.542+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~56DZ564I9J68.0 size=195091 2024-03-09T14:36:10.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.550+0000 7f73315818c0 1 < completed multipart upload key=IA0AB7UVLFEJ upload=56DZ564I9J68 tag=FYXA4CT8K10L type=write size=1243667 2024-03-09T14:36:10.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.554+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=U3S0IZXHNNIH type=del size=4057972 2024-03-09T14:36:10.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.558+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=9PF0MDC0YQSY type=del 2024-03-09T14:36:10.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.562+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=HHLBJBXRWMEJ type=del size=1547833 2024-03-09T14:36:10.574 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.570+0000 7f73315818c0 1 < suggested remove operation key=4P89277EEYJ3 tag=F5IZ3IA0OEWY type=del 2024-03-09T14:36:10.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.574+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=KQBX6XB6PF0Z type=del size=1039595 2024-03-09T14:36:10.581 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.578+0000 7f73315818c0 1 < completed write operation key=W6KVP8EQFQ8J tag=S22JZODWFILY type=write size=89862 2024-03-09T14:36:10.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.582+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=34ZD6MHQJW62 type=del size=1762723 2024-03-09T14:36:10.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.586+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~CSYO1GXDYH1Y.0 size=1048576 2024-03-09T14:36:10.591 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.590+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~CSYO1GXDYH1Y.1 size=1048576 2024-03-09T14:36:10.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.590+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~CSYO1GXDYH1Y.2 size=597261 2024-03-09T14:36:10.600 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.598+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=CSYO1GXDYH1Y tag=UE8SMZG21PWJ type=write size=3742989 2024-03-09T14:36:10.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.602+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~AP6RFGUTWH69.0 size=1048576 2024-03-09T14:36:10.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.606+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~AP6RFGUTWH69.1 size=602360 2024-03-09T14:36:10.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.606+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=AP6RFGUTWH69 tag=5LKU0SNA1DHS type=write size=2699512 2024-03-09T14:36:10.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.606+0000 7f73315818c0 1 < canceled part key=_multipart_0WGGMHTEF2LB.2~OXMJXNRXQ1JP.0 size=159710 2024-03-09T14:36:10.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.614+0000 7f73315818c0 1 < canceled multipart upload key=0WGGMHTEF2LB upload=OXMJXNRXQ1JP tag=FY0318217TAO type=write size=1208286 2024-03-09T14:36:10.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.618+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~6SGPA81FZXAE.0 size=380897 2024-03-09T14:36:10.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.618+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=6SGPA81FZXAE tag=T5PPSS307FT6 type=write size=1429473 2024-03-09T14:36:10.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.622+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=WG2PBQ9M1I0F type=del 2024-03-09T14:36:10.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.626+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=FECJ2VXJPEIO type=del size=3596240 2024-03-09T14:36:10.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.630+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~Q0JH89423ABR.0 size=422103 2024-03-09T14:36:10.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.634+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=Q0JH89423ABR tag=DT1K00B5VS69 type=write size=1470679 2024-03-09T14:36:10.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.638+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=WKVJW74FQHAL type=del size=376290 2024-03-09T14:36:10.646 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.642+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~ITUET1B38HXT.0 size=1048576 2024-03-09T14:36:10.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.646+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~ITUET1B38HXT.1 size=65074 2024-03-09T14:36:10.654 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.650+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=ITUET1B38HXT tag=JLWDQX9VZSGW type=write size=2162226 2024-03-09T14:36:10.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.654+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=GFOORH34Z068 type=del size=1674455 2024-03-09T14:36:10.662 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.658+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=5IHT8PPH34VS type=del 2024-03-09T14:36:10.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.662+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=JEAWLYR2ZSJD type=write size=1033923 2024-03-09T14:36:10.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.666+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=5NM9EKBRUC3J type=del 2024-03-09T14:36:10.673 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.670+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=SDA7GNZD8S5I type=del size=2105995 2024-03-09T14:36:10.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.674+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~8N250F058PYC.0 size=448447 2024-03-09T14:36:10.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.678+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=8N250F058PYC tag=P6W1F5UTKBUX type=write size=1497023 2024-03-09T14:36:10.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.682+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=CVE9WQN99XB5 type=write size=335640 2024-03-09T14:36:10.689 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.686+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~CSY6M42116KS.0 size=1048576 2024-03-09T14:36:10.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.690+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~CSY6M42116KS.1 size=1048576 2024-03-09T14:36:10.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.690+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~CSY6M42116KS.2 size=472226 2024-03-09T14:36:10.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.698+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=CSY6M42116KS tag=EBW5D1L30AZK type=write size=3617954 2024-03-09T14:36:10.703 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.702+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=1H8G04KN4P6I type=del size=1812452 2024-03-09T14:36:10.707 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.702+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~HI5U53SALZW9.0 size=1048576 2024-03-09T14:36:10.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.706+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~HI5U53SALZW9.1 size=1048576 2024-03-09T14:36:10.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.710+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~HI5U53SALZW9.2 size=850339 2024-03-09T14:36:10.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.714+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=HI5U53SALZW9 tag=WHUNGCF9ZUJW type=write size=3996067 2024-03-09T14:36:10.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.718+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=OPT0YZM7CK0J type=write size=961512 2024-03-09T14:36:10.725 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.722+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~137S1WWG30OA.0 size=1048576 2024-03-09T14:36:10.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.726+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~137S1WWG30OA.1 size=772949 2024-03-09T14:36:10.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.730+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=137S1WWG30OA tag=X33YJT4QWESR type=write size=2870101 2024-03-09T14:36:10.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.734+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=CIN15QHPLSKJ type=del size=387047 2024-03-09T14:36:10.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.738+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=HJH47OQ4D317 type=del 2024-03-09T14:36:10.743 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.742+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=YS863DE10LCB type=del size=3269964 2024-03-09T14:36:10.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.742+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=U3S0IZXHNNIH type=del 2024-03-09T14:36:10.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.746+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=C6QO5CICZI9K type=del size=3675810 2024-03-09T14:36:10.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.750+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=HHLBJBXRWMEJ type=del 2024-03-09T14:36:10.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.758+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~VFARQVVZ00ZB.0 size=987234 2024-03-09T14:36:10.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.758+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=VFARQVVZ00ZB tag=HNPJUU22OSCT type=write size=2035810 2024-03-09T14:36:10.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.762+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=KQBX6XB6PF0Z type=del 2024-03-09T14:36:10.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.766+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=8N7I4U81NOMN type=del size=2948240 2024-03-09T14:36:10.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.766+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=34ZD6MHQJW62 type=del 2024-03-09T14:36:10.775 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.770+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=2J56M0J00CN1 type=del size=166658 2024-03-09T14:36:10.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.774+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~AP6RFGUTWH69.0 size=1048576 2024-03-09T14:36:10.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.778+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~AP6RFGUTWH69.1 size=602360 2024-03-09T14:36:10.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.786+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=AP6RFGUTWH69 tag=5LKU0SNA1DHS type=write size=2699512 2024-03-09T14:36:10.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.790+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=UZUIDRPBOIM6 type=write size=634232 2024-03-09T14:36:10.795 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.794+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~6SGPA81FZXAE.0 size=380897 2024-03-09T14:36:10.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.798+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=6SGPA81FZXAE tag=T5PPSS307FT6 type=write size=1429473 2024-03-09T14:36:10.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.802+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=P9PXXFQ2XMZV type=del size=1704577 2024-03-09T14:36:10.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.806+0000 7f73315818c0 1 < canceled operation key=4E3QMBHEAEHO tag=FECJ2VXJPEIO type=del size=3596240 2024-03-09T14:36:10.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.814+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~39M9B0WWM5J5.0 size=671344 2024-03-09T14:36:10.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.814+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=39M9B0WWM5J5 tag=90JEHQJY9TWE type=write size=1719920 2024-03-09T14:36:10.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.818+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=WKVJW74FQHAL type=del 2024-03-09T14:36:10.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.826+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~KQVA0NFS2PI9.0 size=1048576 2024-03-09T14:36:10.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.826+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~KQVA0NFS2PI9.1 size=1048576 2024-03-09T14:36:10.834 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.830+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~KQVA0NFS2PI9.2 size=839415 2024-03-09T14:36:10.834 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.830+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=KQVA0NFS2PI9 tag=JFS4T0CXGMVQ type=write size=3985143 2024-03-09T14:36:10.838 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.834+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=GFOORH34Z068 type=del 2024-03-09T14:36:10.841 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.838+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=PTD42JET0KHT type=write size=222378 2024-03-09T14:36:10.845 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.842+0000 7f73315818c0 1 < completed write operation key=OU0730HGI176 tag=JEAWLYR2ZSJD type=write size=1033923 2024-03-09T14:36:10.852 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.850+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~TMF987KF5EDR.0 size=1048576 2024-03-09T14:36:10.855 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.850+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~TMF987KF5EDR.1 size=1048576 2024-03-09T14:36:10.857 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.854+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~TMF987KF5EDR.2 size=248555 2024-03-09T14:36:10.857 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.854+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=TMF987KF5EDR tag=CUNX6ES4YY34 type=write size=3394283 2024-03-09T14:36:10.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.858+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=SDA7GNZD8S5I type=del 2024-03-09T14:36:10.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.862+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~IV2L382HQJHQ.0 size=1048576 2024-03-09T14:36:10.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.866+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~IV2L382HQJHQ.1 size=340645 2024-03-09T14:36:10.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.866+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=IV2L382HQJHQ tag=VL2W6EWQF0AK type=write size=2437797 2024-03-09T14:36:10.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.870+0000 7f73315818c0 1 < completed write operation key=DQOY1JJ1YQ2W tag=CVE9WQN99XB5 type=write size=335640 2024-03-09T14:36:10.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.878+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~1WVLC881ZFHW.0 size=230537 2024-03-09T14:36:10.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.878+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=1WVLC881ZFHW tag=V3860E7039LT type=write size=1279113 2024-03-09T14:36:10.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.882+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=1H8G04KN4P6I type=del 2024-03-09T14:36:10.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.882+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=YFNYW8XHFVUA type=write size=354609 2024-03-09T14:36:10.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.890+0000 7f73315818c0 1 < suggested update operation key=P9LI5ESR3N9G tag=OPT0YZM7CK0J type=write size=961512 2024-03-09T14:36:10.900 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.898+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~28Z9E7W4IIHN.0 size=1048576 2024-03-09T14:36:10.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.902+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~28Z9E7W4IIHN.1 size=164954 2024-03-09T14:36:10.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.902+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=28Z9E7W4IIHN tag=H0CUWQZRSZPY type=write size=2262106 2024-03-09T14:36:10.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.906+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=CIN15QHPLSKJ type=del 2024-03-09T14:36:10.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.910+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=V7KLFH6VQ3BX type=del size=254100 2024-03-09T14:36:10.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.918+0000 7f73315818c0 1 < suggested remove operation key=W6KVP8EQFQ8J tag=YS863DE10LCB type=del 2024-03-09T14:36:10.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.926+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~UW6WJTW12JVZ.0 size=1048576 2024-03-09T14:36:10.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.926+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~UW6WJTW12JVZ.1 size=602703 2024-03-09T14:36:10.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.926+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=UW6WJTW12JVZ tag=EPUE5KRIHFN7 type=write size=2699855 2024-03-09T14:36:10.934 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.930+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=C6QO5CICZI9K type=del 2024-03-09T14:36:10.937 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.934+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=QN0WZB82P8F6 type=del size=1618294 2024-03-09T14:36:10.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.938+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~VFARQVVZ00ZB.0 size=987234 2024-03-09T14:36:10.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.942+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=VFARQVVZ00ZB tag=HNPJUU22OSCT type=write size=2035810 2024-03-09T14:36:10.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.950+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~1V0CPSHAX00M.0 size=1048576 2024-03-09T14:36:10.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.954+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~1V0CPSHAX00M.1 size=274216 2024-03-09T14:36:10.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.954+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=1V0CPSHAX00M tag=5X6ZCD6NIY0M type=write size=2371368 2024-03-09T14:36:10.960 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.958+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=8N7I4U81NOMN type=del 2024-03-09T14:36:10.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.966+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~F7JPZQJMH4QY.0 size=636155 2024-03-09T14:36:10.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.966+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=F7JPZQJMH4QY tag=LGVIIL6JY4X7 type=write size=1684731 2024-03-09T14:36:10.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.970+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=2J56M0J00CN1 type=del 2024-03-09T14:36:10.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.970+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=G6GLY3NW7LN7 type=write size=1042030 2024-03-09T14:36:10.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.974+0000 7f73315818c0 1 < completed write operation key=0LP8KKU53X0A tag=UZUIDRPBOIM6 type=write size=634232 2024-03-09T14:36:10.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.978+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=PSGD1QOEKF6V type=del size=4152978 2024-03-09T14:36:10.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.982+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=P9PXXFQ2XMZV type=del 2024-03-09T14:36:10.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.986+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=U9NK1HIK2DU0 type=del size=643276 2024-03-09T14:36:10.994 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.990+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~39M9B0WWM5J5.0 size=671344 2024-03-09T14:36:11.000 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:10.998+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=39M9B0WWM5J5 tag=90JEHQJY9TWE type=write size=1719920 2024-03-09T14:36:11.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.002+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=N8O5IJSGEU4W type=write size=432599 2024-03-09T14:36:11.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.002+0000 7f73315818c0 1 < canceled part key=_multipart_PXFZLDSAR1EG.2~KQVA0NFS2PI9.0 size=1048576 2024-03-09T14:36:11.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.006+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~KQVA0NFS2PI9.1 size=1048576 2024-03-09T14:36:11.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.006+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~KQVA0NFS2PI9.2 size=839415 2024-03-09T14:36:11.018 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.014+0000 7f73315818c0 1 < canceled multipart upload key=PXFZLDSAR1EG upload=KQVA0NFS2PI9 tag=JFS4T0CXGMVQ type=write size=3985143 2024-03-09T14:36:11.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.022+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~UTKMSVZKE3K5.0 size=1048576 2024-03-09T14:36:11.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.026+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~UTKMSVZKE3K5.1 size=907678 2024-03-09T14:36:11.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.026+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=UTKMSVZKE3K5 tag=1TAY1BWD5E1U type=write size=3004830 2024-03-09T14:36:11.035 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.030+0000 7f73315818c0 1 < suggested update operation key=MSMD3ANK4T7J tag=PTD42JET0KHT type=write size=222378 2024-03-09T14:36:11.038 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.034+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=NX31IYG88GGN type=del size=2107127 2024-03-09T14:36:11.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.038+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~TMF987KF5EDR.0 size=1048576 2024-03-09T14:36:11.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.042+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~TMF987KF5EDR.1 size=1048576 2024-03-09T14:36:11.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.042+0000 7f73315818c0 1 < canceled part key=_multipart_3ADG4GWVARC1.2~TMF987KF5EDR.2 size=248555 2024-03-09T14:36:11.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.046+0000 7f73315818c0 1 < canceled multipart upload key=3ADG4GWVARC1 upload=TMF987KF5EDR tag=CUNX6ES4YY34 type=write size=3394283 2024-03-09T14:36:11.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.054+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~7KL5DJWFZ032.0 size=1048576 2024-03-09T14:36:11.061 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.058+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~7KL5DJWFZ032.1 size=430761 2024-03-09T14:36:11.061 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.058+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=7KL5DJWFZ032 tag=PFRWYF4R5059 type=write size=2527913 2024-03-09T14:36:11.065 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.062+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~IV2L382HQJHQ.0 size=1048576 2024-03-09T14:36:11.068 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.066+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~IV2L382HQJHQ.1 size=340645 2024-03-09T14:36:11.074 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.070+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=IV2L382HQJHQ tag=VL2W6EWQF0AK type=write size=2437797 2024-03-09T14:36:11.078 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.074+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=9USBCBKN1ZLP type=del size=2956702 2024-03-09T14:36:11.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.078+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~1WVLC881ZFHW.0 size=230537 2024-03-09T14:36:11.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.086+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=1WVLC881ZFHW tag=V3860E7039LT type=write size=1279113 2024-03-09T14:36:11.095 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.090+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~DRITV9GNN3DX.0 size=313632 2024-03-09T14:36:11.095 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.090+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=DRITV9GNN3DX tag=ENGP7YYU5GKR type=write size=1362208 2024-03-09T14:36:11.099 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.098+0000 7f73315818c0 1 < completed write operation key=UMSNM53HTY1J tag=YFNYW8XHFVUA type=write size=354609 2024-03-09T14:36:11.103 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.098+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=TY2RF6VJUXCO type=del size=1895183 2024-03-09T14:36:11.106 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.102+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~28Z9E7W4IIHN.0 size=1048576 2024-03-09T14:36:11.106 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.102+0000 7f73315818c0 1 < canceled part key=_multipart_PXFZLDSAR1EG.2~28Z9E7W4IIHN.1 size=164954 2024-03-09T14:36:11.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.110+0000 7f73315818c0 1 < canceled multipart upload key=PXFZLDSAR1EG upload=28Z9E7W4IIHN tag=H0CUWQZRSZPY type=write size=2262106 2024-03-09T14:36:11.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.114+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~BLPEJCBYA9A8.0 size=1048576 2024-03-09T14:36:11.121 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.118+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~BLPEJCBYA9A8.1 size=885470 2024-03-09T14:36:11.121 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.118+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=BLPEJCBYA9A8 tag=50OR7DENXMD4 type=write size=2982622 2024-03-09T14:36:11.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.122+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=V7KLFH6VQ3BX type=del 2024-03-09T14:36:11.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.126+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=CC4VC9XU9B09 type=del size=4033522 2024-03-09T14:36:11.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.130+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~UW6WJTW12JVZ.0 size=1048576 2024-03-09T14:36:11.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.130+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~UW6WJTW12JVZ.1 size=602703 2024-03-09T14:36:11.140 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.138+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=UW6WJTW12JVZ tag=EPUE5KRIHFN7 type=write size=2699855 2024-03-09T14:36:11.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.142+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=TF1ZSJRFFJU5 type=del size=534817 2024-03-09T14:36:11.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.146+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=QN0WZB82P8F6 type=del 2024-03-09T14:36:11.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.146+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=0UFF8TFZY72R type=write size=317879 2024-03-09T14:36:11.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.150+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~1V0CPSHAX00M.0 size=1048576 2024-03-09T14:36:11.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.150+0000 7f73315818c0 1 < canceled part key=_multipart_DQOY1JJ1YQ2W.2~1V0CPSHAX00M.1 size=274216 2024-03-09T14:36:11.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.158+0000 7f73315818c0 1 < canceled multipart upload key=DQOY1JJ1YQ2W upload=1V0CPSHAX00M tag=5X6ZCD6NIY0M type=write size=2371368 2024-03-09T14:36:11.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.162+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~VJZ1OAC7NAC2.0 size=1048576 2024-03-09T14:36:11.169 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.166+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~VJZ1OAC7NAC2.1 size=1048576 2024-03-09T14:36:11.172 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.170+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~VJZ1OAC7NAC2.2 size=778687 2024-03-09T14:36:11.172 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.170+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=VJZ1OAC7NAC2 tag=DPLSLF5ZKK6Q type=write size=3924415 2024-03-09T14:36:11.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.174+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~F7JPZQJMH4QY.0 size=636155 2024-03-09T14:36:11.182 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.178+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=F7JPZQJMH4QY tag=LGVIIL6JY4X7 type=write size=1684731 2024-03-09T14:36:11.189 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.186+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~MW5MPUAJ89PU.0 size=1048576 2024-03-09T14:36:11.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.190+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~MW5MPUAJ89PU.1 size=1048576 2024-03-09T14:36:11.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.190+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~MW5MPUAJ89PU.2 size=744565 2024-03-09T14:36:11.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.190+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=MW5MPUAJ89PU tag=9NH4AIR23W5L type=write size=3890293 2024-03-09T14:36:11.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.194+0000 7f73315818c0 1 < completed write operation key=LZC7NPF0GRO7 tag=G6GLY3NW7LN7 type=write size=1042030 2024-03-09T14:36:11.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.198+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=O82A3JRDNPBW type=del size=1191451 2024-03-09T14:36:11.207 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.202+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=PSGD1QOEKF6V type=del 2024-03-09T14:36:11.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.210+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~Y04VTGNV4IZO.0 size=1048576 2024-03-09T14:36:11.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.214+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~Y04VTGNV4IZO.1 size=794911 2024-03-09T14:36:11.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.214+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=Y04VTGNV4IZO tag=SFPIKZW364N7 type=write size=2892063 2024-03-09T14:36:11.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.218+0000 7f73315818c0 1 < suggested update operation key=QG4O2L526B8E tag=U9NK1HIK2DU0 type=del size=643276 2024-03-09T14:36:11.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.222+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=WQEBZCQYX8MH type=del size=245748 2024-03-09T14:36:11.230 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.226+0000 7f73315818c0 1 < completed write operation key=0WGGMHTEF2LB tag=N8O5IJSGEU4W type=write size=432599 2024-03-09T14:36:11.237 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.234+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~KBGOZVIKGITC.0 size=1007156 2024-03-09T14:36:11.237 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.234+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=KBGOZVIKGITC tag=1NP02INPB1RJ type=write size=2055732 2024-03-09T14:36:11.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.238+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~UTKMSVZKE3K5.0 size=1048576 2024-03-09T14:36:11.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.242+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~UTKMSVZKE3K5.1 size=907678 2024-03-09T14:36:11.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.250+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=UTKMSVZKE3K5 tag=1TAY1BWD5E1U type=write size=3004830 2024-03-09T14:36:11.255 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.254+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=YGXEQ97A74ZW type=del size=4154178 2024-03-09T14:36:11.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.254+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=NX31IYG88GGN type=del 2024-03-09T14:36:11.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.262+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~JY5IUJS8GZBJ.0 size=247712 2024-03-09T14:36:11.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.262+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=JY5IUJS8GZBJ tag=7C2Q5DR1J437 type=write size=1296288 2024-03-09T14:36:11.269 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.266+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~7KL5DJWFZ032.0 size=1048576 2024-03-09T14:36:11.272 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.270+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~7KL5DJWFZ032.1 size=430761 2024-03-09T14:36:11.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.274+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=7KL5DJWFZ032 tag=PFRWYF4R5059 type=write size=2527913 2024-03-09T14:36:11.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.278+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=31W393WG2YL1 type=write size=354744 2024-03-09T14:36:11.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.282+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=9USBCBKN1ZLP type=del 2024-03-09T14:36:11.288 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.286+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=P19ZJNVO9T98 type=del size=251774 2024-03-09T14:36:11.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.290+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~DRITV9GNN3DX.0 size=313632 2024-03-09T14:36:11.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.294+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=DRITV9GNN3DX tag=ENGP7YYU5GKR type=write size=1362208 2024-03-09T14:36:11.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.298+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=NNF3I1GTM153 type=del size=1857960 2024-03-09T14:36:11.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.302+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=TY2RF6VJUXCO type=del 2024-03-09T14:36:11.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.306+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=7D0SWN1BRAJ7 type=del size=950625 2024-03-09T14:36:11.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.310+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~BLPEJCBYA9A8.0 size=1048576 2024-03-09T14:36:11.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.310+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~BLPEJCBYA9A8.1 size=885470 2024-03-09T14:36:11.321 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.318+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=BLPEJCBYA9A8 tag=50OR7DENXMD4 type=write size=2982622 2024-03-09T14:36:11.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.322+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=4EXVRX4L5TDU type=write size=398102 2024-03-09T14:36:11.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.330+0000 7f73315818c0 1 < suggested update operation key=92GWY5B0K4SX tag=CC4VC9XU9B09 type=del size=4033522 2024-03-09T14:36:11.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.334+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~WQ7IQWTG1EP9.0 size=1048576 2024-03-09T14:36:11.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.338+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~WQ7IQWTG1EP9.1 size=642234 2024-03-09T14:36:11.341 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.338+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=WQ7IQWTG1EP9 tag=DD6YK2NYJCZI type=write size=2739386 2024-03-09T14:36:11.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.342+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=TF1ZSJRFFJU5 type=del 2024-03-09T14:36:11.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.346+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~AIOVXAUJZGQD.0 size=1048576 2024-03-09T14:36:11.352 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.350+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~AIOVXAUJZGQD.1 size=1048576 2024-03-09T14:36:11.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.350+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~AIOVXAUJZGQD.2 size=40127 2024-03-09T14:36:11.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.350+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=AIOVXAUJZGQD tag=LTR75MEUJY04 type=write size=3185855 2024-03-09T14:36:11.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.358+0000 7f73315818c0 1 < suggested update operation key=92GWY5B0K4SX tag=0UFF8TFZY72R type=write size=317879 2024-03-09T14:36:11.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.366+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~KI7LIW0JUSKJ.0 size=286614 2024-03-09T14:36:11.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.366+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=KI7LIW0JUSKJ tag=4IVIYML5UYN9 type=write size=1335190 2024-03-09T14:36:11.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.370+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~VJZ1OAC7NAC2.0 size=1048576 2024-03-09T14:36:11.377 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.374+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~VJZ1OAC7NAC2.1 size=1048576 2024-03-09T14:36:11.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.378+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~VJZ1OAC7NAC2.2 size=778687 2024-03-09T14:36:11.387 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.382+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=VJZ1OAC7NAC2 tag=DPLSLF5ZKK6Q type=write size=3924415 2024-03-09T14:36:11.394 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.390+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~RDHRNHZQ1VM1.0 size=1048576 2024-03-09T14:36:11.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.394+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~RDHRNHZQ1VM1.1 size=181105 2024-03-09T14:36:11.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.394+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=RDHRNHZQ1VM1 tag=76WQS343RK49 type=write size=2278257 2024-03-09T14:36:11.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.398+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~MW5MPUAJ89PU.0 size=1048576 2024-03-09T14:36:11.404 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.402+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~MW5MPUAJ89PU.1 size=1048576 2024-03-09T14:36:11.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.402+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~MW5MPUAJ89PU.2 size=744565 2024-03-09T14:36:11.413 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.410+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=MW5MPUAJ89PU tag=9NH4AIR23W5L type=write size=3890293 2024-03-09T14:36:11.416 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.414+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=N5YFS8983SXK type=del size=2193416 2024-03-09T14:36:11.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.418+0000 7f73315818c0 1 < canceled operation key=CVAWRZMAX997 tag=O82A3JRDNPBW type=del size=1191451 2024-03-09T14:36:11.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.422+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~YTU1FM9JMKG3.0 size=1048576 2024-03-09T14:36:11.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.426+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~YTU1FM9JMKG3.1 size=1048576 2024-03-09T14:36:11.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.430+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~YTU1FM9JMKG3.2 size=798541 2024-03-09T14:36:11.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.430+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=YTU1FM9JMKG3 tag=PU32Y0MSL1YR type=write size=3944269 2024-03-09T14:36:11.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.434+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~Y04VTGNV4IZO.0 size=1048576 2024-03-09T14:36:11.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.434+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~Y04VTGNV4IZO.1 size=794911 2024-03-09T14:36:11.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.442+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=Y04VTGNV4IZO tag=SFPIKZW364N7 type=write size=2892063 2024-03-09T14:36:11.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.446+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=VI6LQNZCY180 type=del size=4165691 2024-03-09T14:36:11.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.450+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=WQEBZCQYX8MH type=del 2024-03-09T14:36:11.455 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.454+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=AEU021Y5CG3E type=del size=654517 2024-03-09T14:36:11.459 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.454+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~KBGOZVIKGITC.0 size=1007156 2024-03-09T14:36:11.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.462+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=KBGOZVIKGITC tag=1NP02INPB1RJ type=write size=2055732 2024-03-09T14:36:11.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.466+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~A652V1DGMXHW.0 size=1048576 2024-03-09T14:36:11.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.470+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~A652V1DGMXHW.1 size=1048576 2024-03-09T14:36:11.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.474+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~A652V1DGMXHW.2 size=851583 2024-03-09T14:36:11.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.474+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=A652V1DGMXHW tag=ZYGJHQP208UU type=write size=3997311 2024-03-09T14:36:11.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.478+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=YGXEQ97A74ZW type=del 2024-03-09T14:36:11.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.478+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=Q11UP89K3K6M type=del size=3669357 2024-03-09T14:36:11.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.482+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~JY5IUJS8GZBJ.0 size=247712 2024-03-09T14:36:11.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.490+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=JY5IUJS8GZBJ tag=7C2Q5DR1J437 type=write size=1296288 2024-03-09T14:36:11.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.494+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~04L5VEP9JCIC.0 size=1048576 2024-03-09T14:36:11.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.498+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~04L5VEP9JCIC.1 size=954834 2024-03-09T14:36:11.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.498+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=04L5VEP9JCIC tag=8SPRHUNLRVY3 type=write size=3051986 2024-03-09T14:36:11.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.502+0000 7f73315818c0 1 < completed write operation key=UMSNM53HTY1J tag=31W393WG2YL1 type=write size=354744 2024-03-09T14:36:11.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.506+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~URPFR9KXZUXJ.0 size=968291 2024-03-09T14:36:11.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.506+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=URPFR9KXZUXJ tag=EI0OE2TJLAZL type=write size=2016867 2024-03-09T14:36:11.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.510+0000 7f73315818c0 1 < canceled operation key=PXFZLDSAR1EG tag=P19ZJNVO9T98 type=del size=251774 2024-03-09T14:36:11.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.518+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~5M7UEN3XG18D.0 size=1048576 2024-03-09T14:36:11.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.522+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~5M7UEN3XG18D.1 size=288113 2024-03-09T14:36:11.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.522+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=5M7UEN3XG18D tag=3KK4C2H2JUD7 type=write size=2385265 2024-03-09T14:36:11.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.522+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=NNF3I1GTM153 type=del 2024-03-09T14:36:11.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.526+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=OJJG9ZZC307B type=del size=1158466 2024-03-09T14:36:11.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.530+0000 7f73315818c0 1 < canceled operation key=1NYNL8M5DVER tag=7D0SWN1BRAJ7 type=del size=950625 2024-03-09T14:36:11.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.534+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=J0VA83QUZ1XY type=write size=626355 2024-03-09T14:36:11.539 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.538+0000 7f73315818c0 1 < completed write operation key=FYFVW9R0M1PX tag=4EXVRX4L5TDU type=write size=398102 2024-03-09T14:36:11.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.538+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=B1WEZKQ2I97Z type=del size=795301 2024-03-09T14:36:11.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.542+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~WQ7IQWTG1EP9.0 size=1048576 2024-03-09T14:36:11.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.542+0000 7f73315818c0 1 < canceled part key=_multipart_P8R5RJI8LGZ9.2~WQ7IQWTG1EP9.1 size=642234 2024-03-09T14:36:11.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.546+0000 7f73315818c0 1 < canceled multipart upload key=P8R5RJI8LGZ9 upload=WQ7IQWTG1EP9 tag=DD6YK2NYJCZI type=write size=2739386 2024-03-09T14:36:11.557 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.554+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~5LD3DNCAHFWA.0 size=1048576 2024-03-09T14:36:11.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.558+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~5LD3DNCAHFWA.1 size=1048576 2024-03-09T14:36:11.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.558+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~5LD3DNCAHFWA.2 size=944194 2024-03-09T14:36:11.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.558+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=5LD3DNCAHFWA tag=1YJOK9ZIELZM type=write size=4089922 2024-03-09T14:36:11.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.566+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~AIOVXAUJZGQD.0 size=1048576 2024-03-09T14:36:11.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.570+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~AIOVXAUJZGQD.1 size=1048576 2024-03-09T14:36:11.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.570+0000 7f73315818c0 1 < canceled part key=_multipart_K3JNN1NCKBXN.2~AIOVXAUJZGQD.2 size=40127 2024-03-09T14:36:11.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.574+0000 7f73315818c0 1 < canceled multipart upload key=K3JNN1NCKBXN upload=AIOVXAUJZGQD tag=LTR75MEUJY04 type=write size=3185855 2024-03-09T14:36:11.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.578+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~U0IOHT1MDPMC.0 size=1048576 2024-03-09T14:36:11.585 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.582+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~U0IOHT1MDPMC.1 size=1048576 2024-03-09T14:36:11.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.586+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~U0IOHT1MDPMC.2 size=806197 2024-03-09T14:36:11.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.586+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=U0IOHT1MDPMC tag=DMK7HTKI7JCO type=write size=3951925 2024-03-09T14:36:11.591 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.586+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~KI7LIW0JUSKJ.0 size=286614 2024-03-09T14:36:11.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.594+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=KI7LIW0JUSKJ tag=4IVIYML5UYN9 type=write size=1335190 2024-03-09T14:36:11.599 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.594+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=UIOITDDB8QNI type=write size=1003792 2024-03-09T14:36:11.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.598+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~RDHRNHZQ1VM1.0 size=1048576 2024-03-09T14:36:11.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.602+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~RDHRNHZQ1VM1.1 size=181105 2024-03-09T14:36:11.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.606+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=RDHRNHZQ1VM1 tag=76WQS343RK49 type=write size=2278257 2024-03-09T14:36:11.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.610+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~LM4NDB0EI94L.0 size=1043630 2024-03-09T14:36:11.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.610+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=LM4NDB0EI94L tag=HWVTRRTO6OLC type=write size=2092206 2024-03-09T14:36:11.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.614+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=N5YFS8983SXK type=del 2024-03-09T14:36:11.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.618+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~XZXDY5ZBHZ2G.0 size=1048576 2024-03-09T14:36:11.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.622+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~XZXDY5ZBHZ2G.1 size=1048576 2024-03-09T14:36:11.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.622+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~XZXDY5ZBHZ2G.2 size=554937 2024-03-09T14:36:11.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.622+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=XZXDY5ZBHZ2G tag=QINK1KCNTIVI type=write size=3700665 2024-03-09T14:36:11.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.626+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~YTU1FM9JMKG3.0 size=1048576 2024-03-09T14:36:11.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.630+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~YTU1FM9JMKG3.1 size=1048576 2024-03-09T14:36:11.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.630+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~YTU1FM9JMKG3.2 size=798541 2024-03-09T14:36:11.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.634+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=YTU1FM9JMKG3 tag=PU32Y0MSL1YR type=write size=3944269 2024-03-09T14:36:11.644 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.642+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~5O2HKGT3C09Y.0 size=1048576 2024-03-09T14:36:11.647 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.642+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~5O2HKGT3C09Y.1 size=1048576 2024-03-09T14:36:11.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.646+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~5O2HKGT3C09Y.2 size=941486 2024-03-09T14:36:11.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.646+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=5O2HKGT3C09Y tag=2KE0W7ABN5E2 type=write size=4087214 2024-03-09T14:36:11.652 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.650+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=VI6LQNZCY180 type=del 2024-03-09T14:36:11.654 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.650+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=80B2J0N1K1DY type=del size=1053539 2024-03-09T14:36:11.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.654+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=AEU021Y5CG3E type=del 2024-03-09T14:36:11.663 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.662+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~TT5CHVCUJIT4.0 size=1048576 2024-03-09T14:36:11.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.662+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~TT5CHVCUJIT4.1 size=717511 2024-03-09T14:36:11.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.662+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=TT5CHVCUJIT4 tag=CFQVDNEB0BK5 type=write size=2814663 2024-03-09T14:36:11.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.666+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~A652V1DGMXHW.0 size=1048576 2024-03-09T14:36:11.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.666+0000 7f73315818c0 1 < canceled part key=_multipart_CVAWRZMAX997.2~A652V1DGMXHW.1 size=1048576 2024-03-09T14:36:11.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.670+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~A652V1DGMXHW.2 size=851583 2024-03-09T14:36:11.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.674+0000 7f73315818c0 1 < canceled multipart upload key=CVAWRZMAX997 upload=A652V1DGMXHW tag=ZYGJHQP208UU type=write size=3997311 2024-03-09T14:36:11.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.678+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~MWLJ787MVUG4.0 size=1048576 2024-03-09T14:36:11.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.682+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~MWLJ787MVUG4.1 size=730365 2024-03-09T14:36:11.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.682+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=MWLJ787MVUG4 tag=6OAIJJ98CL1D type=write size=2827517 2024-03-09T14:36:11.689 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.686+0000 7f73315818c0 1 < canceled operation key=M1B6O6N8N84T tag=Q11UP89K3K6M type=del size=3669357 2024-03-09T14:36:11.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.690+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~WGEW0ASDFEVM.0 size=1048576 2024-03-09T14:36:11.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.694+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~WGEW0ASDFEVM.1 size=707647 2024-03-09T14:36:11.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.694+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=WGEW0ASDFEVM tag=9UYKXGXCD7V8 type=write size=2804799 2024-03-09T14:36:11.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.698+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~04L5VEP9JCIC.0 size=1048576 2024-03-09T14:36:11.703 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.698+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~04L5VEP9JCIC.1 size=954834 2024-03-09T14:36:11.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.706+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=04L5VEP9JCIC tag=8SPRHUNLRVY3 type=write size=3051986 2024-03-09T14:36:11.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.710+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=BUB7M3RPI67A type=del size=2033333 2024-03-09T14:36:11.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.714+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~URPFR9KXZUXJ.0 size=968291 2024-03-09T14:36:11.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.718+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=URPFR9KXZUXJ tag=EI0OE2TJLAZL type=write size=2016867 2024-03-09T14:36:11.726 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.722+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=VPCMTRNGN6Q3 type=write size=902817 2024-03-09T14:36:11.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.726+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~5M7UEN3XG18D.0 size=1048576 2024-03-09T14:36:11.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.730+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~5M7UEN3XG18D.1 size=288113 2024-03-09T14:36:11.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.734+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=5M7UEN3XG18D tag=3KK4C2H2JUD7 type=write size=2385265 2024-03-09T14:36:11.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.738+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=YL4HSUUN4M3A type=del size=3317233 2024-03-09T14:36:11.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.742+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=OJJG9ZZC307B type=del 2024-03-09T14:36:11.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.746+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=OLQ2LV0YH9D6 type=del size=3968841 2024-03-09T14:36:11.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.746+0000 7f73315818c0 1 < completed write operation key=0WGGMHTEF2LB tag=J0VA83QUZ1XY type=write size=626355 2024-03-09T14:36:11.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.754+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~3HIVN94BM95O.0 size=1048576 2024-03-09T14:36:11.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.758+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~3HIVN94BM95O.1 size=751647 2024-03-09T14:36:11.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.758+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=3HIVN94BM95O tag=NF0NW99ME4FS type=write size=2848799 2024-03-09T14:36:11.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.762+0000 7f73315818c0 1 < suggested update operation key=XELG1AWATMVL tag=B1WEZKQ2I97Z type=del size=795301 2024-03-09T14:36:11.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.766+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=Z3PI2SSKY0NL type=del size=3772581 2024-03-09T14:36:11.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.770+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~5LD3DNCAHFWA.0 size=1048576 2024-03-09T14:36:11.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.774+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~5LD3DNCAHFWA.1 size=1048576 2024-03-09T14:36:11.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.778+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~5LD3DNCAHFWA.2 size=944194 2024-03-09T14:36:11.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.782+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=5LD3DNCAHFWA tag=1YJOK9ZIELZM type=write size=4089922 2024-03-09T14:36:11.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.786+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=YL2RND839VG2 type=del size=2923648 2024-03-09T14:36:11.794 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.790+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~U0IOHT1MDPMC.0 size=1048576 2024-03-09T14:36:11.794 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.790+0000 7f73315818c0 1 < canceled part key=_multipart_M8F9TA80VME6.2~U0IOHT1MDPMC.1 size=1048576 2024-03-09T14:36:11.794 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.790+0000 7f73315818c0 1 < canceled part key=_multipart_M8F9TA80VME6.2~U0IOHT1MDPMC.2 size=806197 2024-03-09T14:36:11.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.798+0000 7f73315818c0 1 < canceled multipart upload key=M8F9TA80VME6 upload=U0IOHT1MDPMC tag=DMK7HTKI7JCO type=write size=3951925 2024-03-09T14:36:11.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.798+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=D0CQ0F19JFVM type=del size=1521744 2024-03-09T14:36:11.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.802+0000 7f73315818c0 1 < completed write operation key=3ADG4GWVARC1 tag=UIOITDDB8QNI type=write size=1003792 2024-03-09T14:36:11.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.806+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=74RJM4U9G1ZA type=del size=3021539 2024-03-09T14:36:11.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.810+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~LM4NDB0EI94L.0 size=1043630 2024-03-09T14:36:11.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.814+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=LM4NDB0EI94L tag=HWVTRRTO6OLC type=write size=2092206 2024-03-09T14:36:11.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.818+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=M6MO22SAR1CR type=write size=778177 2024-03-09T14:36:11.826 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.822+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~XZXDY5ZBHZ2G.0 size=1048576 2024-03-09T14:36:11.826 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.822+0000 7f73315818c0 1 < canceled part key=_multipart_DQOY1JJ1YQ2W.2~XZXDY5ZBHZ2G.1 size=1048576 2024-03-09T14:36:11.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.826+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~XZXDY5ZBHZ2G.2 size=554937 2024-03-09T14:36:11.835 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.830+0000 7f73315818c0 1 < canceled multipart upload key=DQOY1JJ1YQ2W upload=XZXDY5ZBHZ2G tag=QINK1KCNTIVI type=write size=3700665 2024-03-09T14:36:11.839 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.834+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=26HSNL5J9OXQ type=del size=2453121 2024-03-09T14:36:11.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.838+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~5O2HKGT3C09Y.0 size=1048576 2024-03-09T14:36:11.845 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.842+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~5O2HKGT3C09Y.1 size=1048576 2024-03-09T14:36:11.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.846+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~5O2HKGT3C09Y.2 size=941486 2024-03-09T14:36:11.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.850+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=5O2HKGT3C09Y tag=2KE0W7ABN5E2 type=write size=4087214 2024-03-09T14:36:11.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.854+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=5QTNV1MYVS0W type=write size=716879 2024-03-09T14:36:11.860 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.858+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=80B2J0N1K1DY type=del 2024-03-09T14:36:11.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.866+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~1PPUM4LZ6MY5.0 size=1048576 2024-03-09T14:36:11.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.866+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~1PPUM4LZ6MY5.1 size=1048576 2024-03-09T14:36:11.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.870+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~1PPUM4LZ6MY5.2 size=824542 2024-03-09T14:36:11.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.870+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=1PPUM4LZ6MY5 tag=JPHCJ2AKCS9C type=write size=3970270 2024-03-09T14:36:11.877 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.874+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~TT5CHVCUJIT4.0 size=1048576 2024-03-09T14:36:11.879 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.878+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~TT5CHVCUJIT4.1 size=717511 2024-03-09T14:36:11.886 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.882+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=TT5CHVCUJIT4 tag=CFQVDNEB0BK5 type=write size=2814663 2024-03-09T14:36:11.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.886+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=V77TJ7JTGEQH type=del size=1077737 2024-03-09T14:36:11.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.890+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~MWLJ787MVUG4.0 size=1048576 2024-03-09T14:36:11.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.890+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~MWLJ787MVUG4.1 size=730365 2024-03-09T14:36:11.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.898+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=MWLJ787MVUG4 tag=6OAIJJ98CL1D type=write size=2827517 2024-03-09T14:36:11.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.902+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=8A2IHRJMYFPU type=write size=272606 2024-03-09T14:36:11.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.902+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~WGEW0ASDFEVM.0 size=1048576 2024-03-09T14:36:11.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.906+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~WGEW0ASDFEVM.1 size=707647 2024-03-09T14:36:11.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.910+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=WGEW0ASDFEVM tag=9UYKXGXCD7V8 type=write size=2804799 2024-03-09T14:36:11.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.914+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=W8L095PD43EX type=del size=911007 2024-03-09T14:36:11.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.918+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=BUB7M3RPI67A type=del 2024-03-09T14:36:11.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.926+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~4HXIVAT2N5L1.0 size=1048576 2024-03-09T14:36:11.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.926+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~4HXIVAT2N5L1.1 size=1048576 2024-03-09T14:36:11.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.930+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~4HXIVAT2N5L1.2 size=1023690 2024-03-09T14:36:11.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.930+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=4HXIVAT2N5L1 tag=44NIHKYQIWFD type=write size=4169418 2024-03-09T14:36:11.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.934+0000 7f73315818c0 1 < completed write operation key=FYFVW9R0M1PX tag=VPCMTRNGN6Q3 type=write size=902817 2024-03-09T14:36:11.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.938+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~FALOJ9VCBBXH.0 size=818053 2024-03-09T14:36:11.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.938+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=FALOJ9VCBBXH tag=8V3ZXZFCLSKR type=write size=1866629 2024-03-09T14:36:11.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.942+0000 7f73315818c0 1 < suggested update operation key=92GWY5B0K4SX tag=YL4HSUUN4M3A type=del size=3317233 2024-03-09T14:36:11.950 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.946+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=A4P5RQ5TDVFG type=del size=166908 2024-03-09T14:36:11.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.950+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=OLQ2LV0YH9D6 type=del 2024-03-09T14:36:11.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.954+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=IX8HTI2SFFIW type=del size=59346 2024-03-09T14:36:11.960 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.958+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~3HIVN94BM95O.0 size=1048576 2024-03-09T14:36:11.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.958+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~3HIVN94BM95O.1 size=751647 2024-03-09T14:36:11.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.966+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=3HIVN94BM95O tag=NF0NW99ME4FS type=write size=2848799 2024-03-09T14:36:11.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.970+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=0D3OPB00W1AA type=del size=1224653 2024-03-09T14:36:11.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.970+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=Z3PI2SSKY0NL type=del 2024-03-09T14:36:11.978 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.974+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=JZLINMMHZE1D type=write size=982069 2024-03-09T14:36:11.982 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.978+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=YL2RND839VG2 type=del 2024-03-09T14:36:11.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.982+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=1SDW69DPFR58 type=del size=2649625 2024-03-09T14:36:11.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.986+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=D0CQ0F19JFVM type=del 2024-03-09T14:36:11.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.990+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=XQFK8IACT3MN type=write size=1028137 2024-03-09T14:36:11.995 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.994+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=74RJM4U9G1ZA type=del 2024-03-09T14:36:12.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:11.998+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~M6RSQWOUA7QL.0 size=1048576 2024-03-09T14:36:12.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.002+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~M6RSQWOUA7QL.1 size=214620 2024-03-09T14:36:12.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.002+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=M6RSQWOUA7QL tag=1LL1MX0MZY7A type=write size=2311772 2024-03-09T14:36:12.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.006+0000 7f73315818c0 1 < completed write operation key=W6KVP8EQFQ8J tag=M6MO22SAR1CR type=write size=778177 2024-03-09T14:36:12.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.010+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=2WU8UZQ8131C type=del size=4042667 2024-03-09T14:36:12.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.010+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=26HSNL5J9OXQ type=del 2024-03-09T14:36:12.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.014+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=Q6HZZTGWREKZ type=write size=168717 2024-03-09T14:36:12.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.018+0000 7f73315818c0 1 < completed write operation key=UMSNM53HTY1J tag=5QTNV1MYVS0W type=write size=716879 2024-03-09T14:36:12.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.022+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=1ZZ60UKVIA3U type=del size=3127045 2024-03-09T14:36:12.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.026+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~1PPUM4LZ6MY5.0 size=1048576 2024-03-09T14:36:12.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.030+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~1PPUM4LZ6MY5.1 size=1048576 2024-03-09T14:36:12.035 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.034+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~1PPUM4LZ6MY5.2 size=824542 2024-03-09T14:36:12.041 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.038+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=1PPUM4LZ6MY5 tag=JPHCJ2AKCS9C type=write size=3970270 2024-03-09T14:36:12.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.042+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=X60SX2DH8FK2 type=del size=2967892 2024-03-09T14:36:12.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.046+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=V77TJ7JTGEQH type=del 2024-03-09T14:36:12.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.050+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=6Q0Z4S7QA6KD type=del size=2608429 2024-03-09T14:36:12.056 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.054+0000 7f73315818c0 1 < completed write operation key=0IB00CZ92VV4 tag=8A2IHRJMYFPU type=write size=272606 2024-03-09T14:36:12.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.058+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~1E3UE6X1N6KM.0 size=1048576 2024-03-09T14:36:12.065 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.062+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~1E3UE6X1N6KM.1 size=1048576 2024-03-09T14:36:12.068 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.066+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~1E3UE6X1N6KM.2 size=679522 2024-03-09T14:36:12.068 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.066+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=1E3UE6X1N6KM tag=9TLQ66G6EBUH type=write size=3825250 2024-03-09T14:36:12.075 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.070+0000 7f73315818c0 1 < suggested remove operation key=K3JNN1NCKBXN tag=W8L095PD43EX type=del 2024-03-09T14:36:12.079 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.074+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=EGBGAUSAWBLG type=del size=2562172 2024-03-09T14:36:12.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.078+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~4HXIVAT2N5L1.0 size=1048576 2024-03-09T14:36:12.085 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.082+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~4HXIVAT2N5L1.1 size=1048576 2024-03-09T14:36:12.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.086+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~4HXIVAT2N5L1.2 size=1023690 2024-03-09T14:36:12.094 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.090+0000 7f73315818c0 1 < completed multipart upload key=LZC7NPF0GRO7 upload=4HXIVAT2N5L1 tag=44NIHKYQIWFD type=write size=4169418 2024-03-09T14:36:12.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.094+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=UDRCJ47WLPJC type=del size=2534380 2024-03-09T14:36:12.102 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.098+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~FALOJ9VCBBXH.0 size=818053 2024-03-09T14:36:12.108 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.106+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=FALOJ9VCBBXH tag=8V3ZXZFCLSKR type=write size=1866629 2024-03-09T14:36:12.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.110+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~M52RYLT2YBJS.0 size=1048576 2024-03-09T14:36:12.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.114+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~M52RYLT2YBJS.1 size=954605 2024-03-09T14:36:12.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.114+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=M52RYLT2YBJS tag=V4H2JW2DLPBA type=write size=3051757 2024-03-09T14:36:12.121 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.118+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=A4P5RQ5TDVFG type=del 2024-03-09T14:36:12.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.126+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~BWDGHO5P47DU.0 size=1048576 2024-03-09T14:36:12.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.126+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~BWDGHO5P47DU.1 size=339817 2024-03-09T14:36:12.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.126+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=BWDGHO5P47DU tag=FSXTJ6BE5LZ1 type=write size=2436969 2024-03-09T14:36:12.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.134+0000 7f73315818c0 1 < suggested update operation key=92GWY5B0K4SX tag=IX8HTI2SFFIW type=del size=59346 2024-03-09T14:36:12.141 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.138+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=2VIH0RF5Q7X2 type=del size=2858752 2024-03-09T14:36:12.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.142+0000 7f73315818c0 -1 < failed to complete operation key=K3JNN1NCKBXN tag=0D3OPB00W1AA type=del size=1224653: (22) Invalid argument 2024-03-09T14:36:12.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.146+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~F32CRSYYR3NY.0 size=978980 2024-03-09T14:36:12.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.146+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=F32CRSYYR3NY tag=N08IUIL38FSD type=write size=2027556 2024-03-09T14:36:12.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.150+0000 7f73315818c0 1 < completed write operation key=LZC7NPF0GRO7 tag=JZLINMMHZE1D type=write size=982069 2024-03-09T14:36:12.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.154+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=SAHMSB27XZX1 type=write size=452283 2024-03-09T14:36:12.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.158+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=1SDW69DPFR58 type=del 2024-03-09T14:36:12.169 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.166+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~SQEEW6PBE082.0 size=548738 2024-03-09T14:36:12.169 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.166+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=SQEEW6PBE082 tag=9GHTZDY83239 type=write size=1597314 2024-03-09T14:36:12.174 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.170+0000 7f73315818c0 1 < canceled operation key=PXFZLDSAR1EG tag=XQFK8IACT3MN type=write size=1028137 2024-03-09T14:36:12.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.174+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=Y6F2392UGEJI type=del size=2608239 2024-03-09T14:36:12.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.174+0000 7f73315818c0 1 < canceled part key=_multipart_PXFZLDSAR1EG.2~M6RSQWOUA7QL.0 size=1048576 2024-03-09T14:36:12.182 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.178+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~M6RSQWOUA7QL.1 size=214620 2024-03-09T14:36:12.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.190+0000 7f73315818c0 1 < canceled multipart upload key=PXFZLDSAR1EG upload=M6RSQWOUA7QL tag=1LL1MX0MZY7A type=write size=2311772 2024-03-09T14:36:12.196 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.194+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=817NK98A7BG9 type=del size=382081 2024-03-09T14:36:12.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.198+0000 7f73315818c0 1 < canceled operation key=0WGGMHTEF2LB tag=2WU8UZQ8131C type=del size=4042667 2024-03-09T14:36:12.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.198+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=41YH0ZFJ5VSN type=del size=3952423 2024-03-09T14:36:12.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.202+0000 7f73315818c0 1 < completed write operation key=0LP8KKU53X0A tag=Q6HZZTGWREKZ type=write size=168717 2024-03-09T14:36:12.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.206+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=V02URCSD5QPG type=del size=2293874 2024-03-09T14:36:12.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.210+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=1ZZ60UKVIA3U type=del 2024-03-09T14:36:12.220 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.218+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~Q0QK5AIW93TS.0 size=1048576 2024-03-09T14:36:12.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.218+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~Q0QK5AIW93TS.1 size=1048576 2024-03-09T14:36:12.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.222+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~Q0QK5AIW93TS.2 size=71535 2024-03-09T14:36:12.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.222+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=Q0QK5AIW93TS tag=LMFSHQL99FCC type=write size=3217263 2024-03-09T14:36:12.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.226+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=X60SX2DH8FK2 type=del 2024-03-09T14:36:12.236 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.234+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~4RHYFKM0VNWO.0 size=1048576 2024-03-09T14:36:12.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.234+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~4RHYFKM0VNWO.1 size=817931 2024-03-09T14:36:12.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.234+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=4RHYFKM0VNWO tag=IDA3I2PXQQSK type=write size=2915083 2024-03-09T14:36:12.245 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.242+0000 7f73315818c0 1 < suggested remove operation key=LZC7NPF0GRO7 tag=6Q0Z4S7QA6KD type=del 2024-03-09T14:36:12.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.250+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~FWS1PU3GX04R.0 size=45646 2024-03-09T14:36:12.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.250+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=FWS1PU3GX04R tag=X92TAU7FDEA1 type=write size=1094222 2024-03-09T14:36:12.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.250+0000 7f73315818c0 1 < canceled part key=_multipart_3ADG4GWVARC1.2~1E3UE6X1N6KM.0 size=1048576 2024-03-09T14:36:12.255 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.250+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~1E3UE6X1N6KM.1 size=1048576 2024-03-09T14:36:12.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.254+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~1E3UE6X1N6KM.2 size=679522 2024-03-09T14:36:12.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.262+0000 7f73315818c0 1 < canceled multipart upload key=3ADG4GWVARC1 upload=1E3UE6X1N6KM tag=9TLQ66G6EBUH type=write size=3825250 2024-03-09T14:36:12.269 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.266+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~UZC6TVH052AD.0 size=1048576 2024-03-09T14:36:12.272 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.270+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~UZC6TVH052AD.1 size=1048576 2024-03-09T14:36:12.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.270+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~UZC6TVH052AD.2 size=845104 2024-03-09T14:36:12.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.270+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=UZC6TVH052AD tag=90RX6BSH1D5I type=write size=3990832 2024-03-09T14:36:12.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.274+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=EGBGAUSAWBLG type=del 2024-03-09T14:36:12.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.282+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~AW0DDFH1L2QP.0 size=1048576 2024-03-09T14:36:12.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.282+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~AW0DDFH1L2QP.1 size=530281 2024-03-09T14:36:12.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.282+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=AW0DDFH1L2QP tag=DAIJ9MC8AXM4 type=write size=2627433 2024-03-09T14:36:12.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.286+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=UDRCJ47WLPJC type=del 2024-03-09T14:36:12.293 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.290+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=FSQ2PNNCPNF3 type=del size=3359995 2024-03-09T14:36:12.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.294+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~M52RYLT2YBJS.0 size=1048576 2024-03-09T14:36:12.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.298+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~M52RYLT2YBJS.1 size=954605 2024-03-09T14:36:12.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.302+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=M52RYLT2YBJS tag=V4H2JW2DLPBA type=write size=3051757 2024-03-09T14:36:12.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.310+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~6YM38LGH6TMC.0 size=1048576 2024-03-09T14:36:12.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.310+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~6YM38LGH6TMC.1 size=238510 2024-03-09T14:36:12.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.310+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=6YM38LGH6TMC tag=PZAUSJE27DCS type=write size=2335662 2024-03-09T14:36:12.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.314+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~BWDGHO5P47DU.0 size=1048576 2024-03-09T14:36:12.321 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.318+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~BWDGHO5P47DU.1 size=339817 2024-03-09T14:36:12.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.326+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=BWDGHO5P47DU tag=FSXTJ6BE5LZ1 type=write size=2436969 2024-03-09T14:36:12.331 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.326+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=2F76HA8AG00Q type=del size=3382504 2024-03-09T14:36:12.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.330+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=2VIH0RF5Q7X2 type=del 2024-03-09T14:36:12.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.338+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~Z8PKO3NQV4WE.0 size=1048576 2024-03-09T14:36:12.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.338+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~Z8PKO3NQV4WE.1 size=1048576 2024-03-09T14:36:12.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.342+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~Z8PKO3NQV4WE.2 size=200776 2024-03-09T14:36:12.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.342+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=Z8PKO3NQV4WE tag=GZIKTJEF66W5 type=write size=3346504 2024-03-09T14:36:12.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.346+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~F32CRSYYR3NY.0 size=978980 2024-03-09T14:36:12.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.350+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=F32CRSYYR3NY tag=N08IUIL38FSD type=write size=2027556 2024-03-09T14:36:12.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.358+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~RRESKSK5L2BM.0 size=1048576 2024-03-09T14:36:12.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.362+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~RRESKSK5L2BM.1 size=994294 2024-03-09T14:36:12.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.362+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=RRESKSK5L2BM tag=6Z0YUNN0Q50V type=write size=3091446 2024-03-09T14:36:12.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.366+0000 7f73315818c0 1 < completed write operation key=MSMD3ANK4T7J tag=SAHMSB27XZX1 type=write size=452283 2024-03-09T14:36:12.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.366+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=4QOQEG5H461R type=del size=2037332 2024-03-09T14:36:12.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.370+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~SQEEW6PBE082.0 size=548738 2024-03-09T14:36:12.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.378+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=SQEEW6PBE082 tag=9GHTZDY83239 type=write size=1597314 2024-03-09T14:36:12.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.382+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~GD6EVYYLTRKF.0 size=1048576 2024-03-09T14:36:12.389 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.386+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~GD6EVYYLTRKF.1 size=1048576 2024-03-09T14:36:12.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.390+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~GD6EVYYLTRKF.2 size=165667 2024-03-09T14:36:12.393 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.390+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=GD6EVYYLTRKF tag=T2HL4FSSHDA3 type=write size=3311395 2024-03-09T14:36:12.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.398+0000 7f73315818c0 1 < suggested update operation key=0WGGMHTEF2LB tag=Y6F2392UGEJI type=del size=2608239 2024-03-09T14:36:12.404 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.402+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=SITE7R0LQJIJ type=del size=3088516 2024-03-09T14:36:12.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.406+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=817NK98A7BG9 type=del 2024-03-09T14:36:12.416 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.414+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~CORVQKB0B2M0.0 size=1048576 2024-03-09T14:36:12.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.414+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~CORVQKB0B2M0.1 size=1048576 2024-03-09T14:36:12.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.418+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~CORVQKB0B2M0.2 size=60686 2024-03-09T14:36:12.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.418+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=CORVQKB0B2M0 tag=5CJZDUURX5LU type=write size=3206414 2024-03-09T14:36:12.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.422+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=41YH0ZFJ5VSN type=del 2024-03-09T14:36:12.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.426+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=I1RN4AW7REOM type=del size=3166280 2024-03-09T14:36:12.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.430+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=V02URCSD5QPG type=del 2024-03-09T14:36:12.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.434+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~3P4KW42744SV.0 size=1048576 2024-03-09T14:36:12.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.438+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~3P4KW42744SV.1 size=1048576 2024-03-09T14:36:12.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.442+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~3P4KW42744SV.2 size=603568 2024-03-09T14:36:12.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.442+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=3P4KW42744SV tag=UF6L6ACPCAN4 type=write size=3749296 2024-03-09T14:36:12.449 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.446+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~Q0QK5AIW93TS.0 size=1048576 2024-03-09T14:36:12.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.450+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~Q0QK5AIW93TS.1 size=1048576 2024-03-09T14:36:12.455 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.450+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~Q0QK5AIW93TS.2 size=71535 2024-03-09T14:36:12.461 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.458+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=Q0QK5AIW93TS tag=LMFSHQL99FCC type=write size=3217263 2024-03-09T14:36:12.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.462+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=C8T4428RDH4V type=del size=3916924 2024-03-09T14:36:12.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.466+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~4RHYFKM0VNWO.0 size=1048576 2024-03-09T14:36:12.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.466+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~4RHYFKM0VNWO.1 size=817931 2024-03-09T14:36:12.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.474+0000 7f73315818c0 1 < completed multipart upload key=LZC7NPF0GRO7 upload=4RHYFKM0VNWO tag=IDA3I2PXQQSK type=write size=2915083 2024-03-09T14:36:12.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.478+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=8ATIAGI8C71X type=write size=38037 2024-03-09T14:36:12.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.482+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~FWS1PU3GX04R.0 size=45646 2024-03-09T14:36:12.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.486+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=FWS1PU3GX04R tag=X92TAU7FDEA1 type=write size=1094222 2024-03-09T14:36:12.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.490+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=SHOE0NR9DT90 type=del size=304173 2024-03-09T14:36:12.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.494+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~UZC6TVH052AD.0 size=1048576 2024-03-09T14:36:12.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.498+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~UZC6TVH052AD.1 size=1048576 2024-03-09T14:36:12.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.502+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~UZC6TVH052AD.2 size=845104 2024-03-09T14:36:12.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.506+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=UZC6TVH052AD tag=90RX6BSH1D5I type=write size=3990832 2024-03-09T14:36:12.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.514+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=MU6E3TQIKF1I type=del size=1844649 2024-03-09T14:36:12.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.518+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~AW0DDFH1L2QP.0 size=1048576 2024-03-09T14:36:12.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.522+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~AW0DDFH1L2QP.1 size=530281 2024-03-09T14:36:12.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.526+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=AW0DDFH1L2QP tag=DAIJ9MC8AXM4 type=write size=2627433 2024-03-09T14:36:12.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.530+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=U1SCG3NP4847 type=del size=3124251 2024-03-09T14:36:12.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.534+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=FSQ2PNNCPNF3 type=del 2024-03-09T14:36:12.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.538+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=YFM7W5711GNI type=del size=1650987 2024-03-09T14:36:12.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.542+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~6YM38LGH6TMC.0 size=1048576 2024-03-09T14:36:12.547 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.542+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~6YM38LGH6TMC.1 size=238510 2024-03-09T14:36:12.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.550+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=6YM38LGH6TMC tag=PZAUSJE27DCS type=write size=2335662 2024-03-09T14:36:12.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.554+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~YH60QAUIOH7S.0 size=339965 2024-03-09T14:36:12.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.554+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=YH60QAUIOH7S tag=X1U7Y2RH0RG7 type=write size=1388541 2024-03-09T14:36:12.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.562+0000 7f73315818c0 1 < suggested update operation key=FYFVW9R0M1PX tag=2F76HA8AG00Q type=del size=3382504 2024-03-09T14:36:12.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.570+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~BBKOFC9EEYTE.0 size=1048576 2024-03-09T14:36:12.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.574+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~BBKOFC9EEYTE.1 size=226417 2024-03-09T14:36:12.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.574+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=BBKOFC9EEYTE tag=0VNID6O6V02Q type=write size=2323569 2024-03-09T14:36:12.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.574+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~Z8PKO3NQV4WE.0 size=1048576 2024-03-09T14:36:12.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.574+0000 7f73315818c0 1 < canceled part key=_multipart_9B0RAHZZGLYU.2~Z8PKO3NQV4WE.1 size=1048576 2024-03-09T14:36:12.581 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.578+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~Z8PKO3NQV4WE.2 size=200776 2024-03-09T14:36:12.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.582+0000 7f73315818c0 1 < canceled multipart upload key=9B0RAHZZGLYU upload=Z8PKO3NQV4WE tag=GZIKTJEF66W5 type=write size=3346504 2024-03-09T14:36:12.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.590+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~TM9EYBQ925OC.0 size=1000653 2024-03-09T14:36:12.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.590+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=TM9EYBQ925OC tag=8QNYWE3LPLX7 type=write size=2049229 2024-03-09T14:36:12.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.594+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~RRESKSK5L2BM.0 size=1048576 2024-03-09T14:36:12.599 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.594+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~RRESKSK5L2BM.1 size=994294 2024-03-09T14:36:12.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.602+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=RRESKSK5L2BM tag=6Z0YUNN0Q50V type=write size=3091446 2024-03-09T14:36:12.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.606+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=PUYUBFMWP903 type=del size=3554024 2024-03-09T14:36:12.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.610+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=4QOQEG5H461R type=del 2024-03-09T14:36:12.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.614+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~PRF1NS8AHIX9.0 size=1048576 2024-03-09T14:36:12.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.618+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~PRF1NS8AHIX9.1 size=1048576 2024-03-09T14:36:12.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.618+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~PRF1NS8AHIX9.2 size=294877 2024-03-09T14:36:12.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.618+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=PRF1NS8AHIX9 tag=A7X0OMZYHYVE type=write size=3440605 2024-03-09T14:36:12.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.622+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~GD6EVYYLTRKF.0 size=1048576 2024-03-09T14:36:12.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.626+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~GD6EVYYLTRKF.1 size=1048576 2024-03-09T14:36:12.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.630+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~GD6EVYYLTRKF.2 size=165667 2024-03-09T14:36:12.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.634+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=GD6EVYYLTRKF tag=T2HL4FSSHDA3 type=write size=3311395 2024-03-09T14:36:12.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.638+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=LBBG9EPO9L3P type=del size=2631949 2024-03-09T14:36:12.644 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.642+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=SITE7R0LQJIJ type=del 2024-03-09T14:36:12.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.650+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~98N6EPXP5RM3.0 size=133131 2024-03-09T14:36:12.652 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.650+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=98N6EPXP5RM3 tag=PJMOINLZ8IEL type=write size=1181707 2024-03-09T14:36:12.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.654+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~CORVQKB0B2M0.0 size=1048576 2024-03-09T14:36:12.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.654+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~CORVQKB0B2M0.1 size=1048576 2024-03-09T14:36:12.662 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.658+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~CORVQKB0B2M0.2 size=60686 2024-03-09T14:36:12.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.662+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=CORVQKB0B2M0 tag=5CJZDUURX5LU type=write size=3206414 2024-03-09T14:36:12.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.666+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=0RNMBAFJCRI5 type=del size=3598113 2024-03-09T14:36:12.673 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.670+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=I1RN4AW7REOM type=del 2024-03-09T14:36:12.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.674+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=KQMT0XIB5XAV type=del size=3138361 2024-03-09T14:36:12.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.678+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~3P4KW42744SV.0 size=1048576 2024-03-09T14:36:12.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.678+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~3P4KW42744SV.1 size=1048576 2024-03-09T14:36:12.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.682+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~3P4KW42744SV.2 size=603568 2024-03-09T14:36:12.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.690+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=3P4KW42744SV tag=UF6L6ACPCAN4 type=write size=3749296 2024-03-09T14:36:12.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.694+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~4CX0O3YGO75M.0 size=1048576 2024-03-09T14:36:12.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.698+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~4CX0O3YGO75M.1 size=515082 2024-03-09T14:36:12.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.698+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=4CX0O3YGO75M tag=7J7031T3ZA4P type=write size=2612234 2024-03-09T14:36:12.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.702+0000 7f73315818c0 1 < canceled operation key=QG4O2L526B8E tag=C8T4428RDH4V type=del size=3916924 2024-03-09T14:36:12.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.706+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~35U51H8SHKAI.0 size=537461 2024-03-09T14:36:12.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.706+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=35U51H8SHKAI tag=17OZ8J8BMPHD type=write size=1586037 2024-03-09T14:36:12.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.710+0000 7f73315818c0 1 < completed write operation key=8O5OP8C5NX12 tag=8ATIAGI8C71X type=write size=38037 2024-03-09T14:36:12.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.714+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=W2ZQHE9I2L7L type=write size=643812 2024-03-09T14:36:12.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.718+0000 7f73315818c0 1 < completed delete operation key=M1B6O6N8N84T tag=SHOE0NR9DT90 type=del 2024-03-09T14:36:12.726 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.722+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~X8NIU2767JZ0.0 size=1048576 2024-03-09T14:36:12.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.726+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~X8NIU2767JZ0.1 size=395615 2024-03-09T14:36:12.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.726+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=X8NIU2767JZ0 tag=RU1R7T7P1B5Q type=write size=2492767 2024-03-09T14:36:12.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.734+0000 7f73315818c0 1 < suggested remove operation key=P8R5RJI8LGZ9 tag=MU6E3TQIKF1I type=del 2024-03-09T14:36:12.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.734+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=U1GHF30P8V3Q type=del size=3745758 2024-03-09T14:36:12.742 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.738+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=U1SCG3NP4847 type=del 2024-03-09T14:36:12.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.746+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~TLMXR658DM3Z.0 size=1048576 2024-03-09T14:36:12.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.750+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~TLMXR658DM3Z.1 size=1048576 2024-03-09T14:36:12.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.750+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~TLMXR658DM3Z.2 size=491638 2024-03-09T14:36:12.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.750+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=TLMXR658DM3Z tag=U862DZ121517 type=write size=3637366 2024-03-09T14:36:12.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.754+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=YFM7W5711GNI type=del 2024-03-09T14:36:12.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.758+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=71MN0JUZXE4B type=del size=176582 2024-03-09T14:36:12.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.762+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~YH60QAUIOH7S.0 size=339965 2024-03-09T14:36:12.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.766+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=YH60QAUIOH7S tag=X1U7Y2RH0RG7 type=write size=1388541 2024-03-09T14:36:12.773 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.770+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=CHNQ4KCWX98J type=del size=2279969 2024-03-09T14:36:12.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.774+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~BBKOFC9EEYTE.0 size=1048576 2024-03-09T14:36:12.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.774+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~BBKOFC9EEYTE.1 size=226417 2024-03-09T14:36:12.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.782+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=BBKOFC9EEYTE tag=0VNID6O6V02Q type=write size=2323569 2024-03-09T14:36:12.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.786+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=ZO1OICJJI2H8 type=write size=531335 2024-03-09T14:36:12.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.790+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~TM9EYBQ925OC.0 size=1000653 2024-03-09T14:36:12.797 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.794+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=TM9EYBQ925OC tag=8QNYWE3LPLX7 type=write size=2049229 2024-03-09T14:36:12.801 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.798+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=PMGLY0CRM7ZQ type=del size=419376 2024-03-09T14:36:12.808 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.806+0000 7f73315818c0 1 < suggested update operation key=8O5OP8C5NX12 tag=PUYUBFMWP903 type=del size=3554024 2024-03-09T14:36:12.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.810+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~QXUDLHHOPZI2.0 size=104880 2024-03-09T14:36:12.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.810+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=QXUDLHHOPZI2 tag=T0JSUBZ4SOKI type=write size=1153456 2024-03-09T14:36:12.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.814+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~PRF1NS8AHIX9.0 size=1048576 2024-03-09T14:36:12.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.818+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~PRF1NS8AHIX9.1 size=1048576 2024-03-09T14:36:12.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.818+0000 7f73315818c0 1 < canceled part key=_multipart_PAJYGD2GXHDT.2~PRF1NS8AHIX9.2 size=294877 2024-03-09T14:36:12.826 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.822+0000 7f73315818c0 1 < canceled multipart upload key=PAJYGD2GXHDT upload=PRF1NS8AHIX9 tag=A7X0OMZYHYVE type=write size=3440605 2024-03-09T14:36:12.836 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.834+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~HDF2F52F2PRC.0 size=1048576 2024-03-09T14:36:12.839 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.834+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~HDF2F52F2PRC.1 size=1048576 2024-03-09T14:36:12.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.838+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~HDF2F52F2PRC.2 size=212943 2024-03-09T14:36:12.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.838+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=HDF2F52F2PRC tag=N69MKN5J5WSW type=write size=3358671 2024-03-09T14:36:12.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.846+0000 7f73315818c0 1 < suggested remove operation key=M1B6O6N8N84T tag=LBBG9EPO9L3P type=del 2024-03-09T14:36:12.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.854+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~NCDNS1THRFLZ.0 size=937825 2024-03-09T14:36:12.857 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.854+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=NCDNS1THRFLZ tag=4UIMIIC9LR3R type=write size=1986401 2024-03-09T14:36:12.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.858+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~98N6EPXP5RM3.0 size=133131 2024-03-09T14:36:12.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.862+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=98N6EPXP5RM3 tag=PJMOINLZ8IEL type=write size=1181707 2024-03-09T14:36:12.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.866+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=C6M3WCNVV0OL type=del size=1983839 2024-03-09T14:36:12.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.870+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=0RNMBAFJCRI5 type=del 2024-03-09T14:36:12.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.874+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~II1NIFUE3U9J.0 size=1048576 2024-03-09T14:36:12.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.878+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~II1NIFUE3U9J.1 size=610315 2024-03-09T14:36:12.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.878+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=II1NIFUE3U9J tag=J5W49O0BC02Q type=write size=2707467 2024-03-09T14:36:12.884 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.882+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=KQMT0XIB5XAV type=del 2024-03-09T14:36:12.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.886+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~S5DDI91230ZQ.0 size=1048576 2024-03-09T14:36:12.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.890+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~S5DDI91230ZQ.1 size=455489 2024-03-09T14:36:12.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.890+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=S5DDI91230ZQ tag=CFCAI72KA1HD type=write size=2552641 2024-03-09T14:36:12.897 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.894+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~4CX0O3YGO75M.0 size=1048576 2024-03-09T14:36:12.900 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.898+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~4CX0O3YGO75M.1 size=515082 2024-03-09T14:36:12.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.902+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=4CX0O3YGO75M tag=7J7031T3ZA4P type=write size=2612234 2024-03-09T14:36:12.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.910+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~EPM3S0Y92PUM.0 size=979670 2024-03-09T14:36:12.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.910+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=EPM3S0Y92PUM tag=ZTNIXWBX5PCM type=write size=2028246 2024-03-09T14:36:12.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.914+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~35U51H8SHKAI.0 size=537461 2024-03-09T14:36:12.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.918+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=35U51H8SHKAI tag=17OZ8J8BMPHD type=write size=1586037 2024-03-09T14:36:12.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.922+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=214N7IHTMX6O type=write size=667683 2024-03-09T14:36:12.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.930+0000 7f73315818c0 1 < suggested remove operation key=QG4O2L526B8E tag=W2ZQHE9I2L7L type=write 2024-03-09T14:36:12.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.938+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~HUD87LFLIU14.0 size=530528 2024-03-09T14:36:12.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.938+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=HUD87LFLIU14 tag=63UT1D80S87F type=write size=1579104 2024-03-09T14:36:12.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.938+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~X8NIU2767JZ0.0 size=1048576 2024-03-09T14:36:12.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.942+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~X8NIU2767JZ0.1 size=395615 2024-03-09T14:36:12.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.950+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=X8NIU2767JZ0 tag=RU1R7T7P1B5Q type=write size=2492767 2024-03-09T14:36:12.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.950+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=STYZO260TT4O type=del size=3409652 2024-03-09T14:36:12.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.958+0000 7f73315818c0 1 < suggested update operation key=P9LI5ESR3N9G tag=U1GHF30P8V3Q type=del size=3745758 2024-03-09T14:36:12.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.966+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~Z41S7ISEB31H.0 size=279608 2024-03-09T14:36:12.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.966+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=Z41S7ISEB31H tag=E7TY7GEPYSUV type=write size=1328184 2024-03-09T14:36:12.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.970+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~TLMXR658DM3Z.0 size=1048576 2024-03-09T14:36:12.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.974+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~TLMXR658DM3Z.1 size=1048576 2024-03-09T14:36:12.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.974+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~TLMXR658DM3Z.2 size=491638 2024-03-09T14:36:12.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.982+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=TLMXR658DM3Z tag=U862DZ121517 type=write size=3637366 2024-03-09T14:36:12.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.986+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=ITQRU60RQ25R type=del size=3426257 2024-03-09T14:36:12.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.986+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=71MN0JUZXE4B type=del 2024-03-09T14:36:12.997 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.994+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~CV8GHRYKUHJW.0 size=1048576 2024-03-09T14:36:12.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.998+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~CV8GHRYKUHJW.1 size=1048576 2024-03-09T14:36:13.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.998+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~CV8GHRYKUHJW.2 size=747392 2024-03-09T14:36:13.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:12.998+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=CV8GHRYKUHJW tag=9P4V8A0POMY7 type=write size=3893120 2024-03-09T14:36:13.005 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.002+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=CHNQ4KCWX98J type=del 2024-03-09T14:36:13.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.006+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=RHNVZOCQ2J2F type=del size=228911 2024-03-09T14:36:13.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.010+0000 7f73315818c0 1 < completed write operation key=LZC7NPF0GRO7 tag=ZO1OICJJI2H8 type=write size=531335 2024-03-09T14:36:13.017 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.014+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~Z03KA9VS8FJ0.0 size=985963 2024-03-09T14:36:13.017 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.014+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=Z03KA9VS8FJ0 tag=RQ8LCUMVYX27 type=write size=2034539 2024-03-09T14:36:13.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.018+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=PMGLY0CRM7ZQ type=del 2024-03-09T14:36:13.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.022+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~I72J34INBOAR.0 size=961932 2024-03-09T14:36:13.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.022+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=I72J34INBOAR tag=XP9E8X5BQNEM type=write size=2010508 2024-03-09T14:36:13.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.026+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~QXUDLHHOPZI2.0 size=104880 2024-03-09T14:36:13.036 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.034+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=QXUDLHHOPZI2 tag=T0JSUBZ4SOKI type=write size=1153456 2024-03-09T14:36:13.039 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.038+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=SB1QQBFTYADK type=del size=2440859 2024-03-09T14:36:13.043 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.038+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~HDF2F52F2PRC.0 size=1048576 2024-03-09T14:36:13.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.042+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~HDF2F52F2PRC.1 size=1048576 2024-03-09T14:36:13.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.046+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~HDF2F52F2PRC.2 size=212943 2024-03-09T14:36:13.054 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.050+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=HDF2F52F2PRC tag=N69MKN5J5WSW type=write size=3358671 2024-03-09T14:36:13.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.058+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~WJB5UHE063OV.0 size=758038 2024-03-09T14:36:13.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.058+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=WJB5UHE063OV tag=QX1616L12JQ5 type=write size=1806614 2024-03-09T14:36:13.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.058+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~NCDNS1THRFLZ.0 size=937825 2024-03-09T14:36:13.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.066+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=NCDNS1THRFLZ tag=4UIMIIC9LR3R type=write size=1986401 2024-03-09T14:36:13.075 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.074+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~GAB3EZTZ1Z7R.0 size=1048576 2024-03-09T14:36:13.078 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.074+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~GAB3EZTZ1Z7R.1 size=1048576 2024-03-09T14:36:13.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.078+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~GAB3EZTZ1Z7R.2 size=1045451 2024-03-09T14:36:13.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.078+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=GAB3EZTZ1Z7R tag=QA2UZULG9IPD type=write size=4191179 2024-03-09T14:36:13.085 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.082+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=C6M3WCNVV0OL type=del 2024-03-09T14:36:13.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.086+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=GII8MQXMQ8JX type=del size=2628718 2024-03-09T14:36:13.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.090+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~II1NIFUE3U9J.0 size=1048576 2024-03-09T14:36:13.095 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.094+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~II1NIFUE3U9J.1 size=610315 2024-03-09T14:36:13.101 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.098+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=II1NIFUE3U9J tag=J5W49O0BC02Q type=write size=2707467 2024-03-09T14:36:13.104 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.102+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=PXYBEYBJKKO1 type=del size=1886155 2024-03-09T14:36:13.108 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.106+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~S5DDI91230ZQ.0 size=1048576 2024-03-09T14:36:13.108 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.106+0000 7f73315818c0 1 < canceled part key=_multipart_MD7FXGYXB823.2~S5DDI91230ZQ.1 size=455489 2024-03-09T14:36:13.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.110+0000 7f73315818c0 1 < canceled multipart upload key=MD7FXGYXB823 upload=S5DDI91230ZQ tag=CFCAI72KA1HD type=write size=2552641 2024-03-09T14:36:13.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.114+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=24RJVOR14FRL type=del size=2021411 2024-03-09T14:36:13.122 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.118+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~EPM3S0Y92PUM.0 size=979670 2024-03-09T14:36:13.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.126+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=EPM3S0Y92PUM tag=ZTNIXWBX5PCM type=write size=2028246 2024-03-09T14:36:13.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.130+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~A4MJDDVNK0SG.0 size=1048576 2024-03-09T14:36:13.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.134+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~A4MJDDVNK0SG.1 size=846784 2024-03-09T14:36:13.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.134+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=A4MJDDVNK0SG tag=31EGR5ZCIV2A type=write size=2943936 2024-03-09T14:36:13.142 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.138+0000 7f73315818c0 1 < completed write operation key=P8R5RJI8LGZ9 tag=214N7IHTMX6O type=write size=667683 2024-03-09T14:36:13.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.142+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=DJV0C9D9RSK7 type=write size=726466 2024-03-09T14:36:13.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.146+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~HUD87LFLIU14.0 size=530528 2024-03-09T14:36:13.160 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.158+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=HUD87LFLIU14 tag=63UT1D80S87F type=write size=1579104 2024-03-09T14:36:13.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.158+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=TSQ2JA34MXPQ type=del size=2504530 2024-03-09T14:36:13.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.162+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=STYZO260TT4O type=del 2024-03-09T14:36:13.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.166+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=DTI3JQD9U7UE type=del size=2239874 2024-03-09T14:36:13.174 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.170+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~Z41S7ISEB31H.0 size=279608 2024-03-09T14:36:13.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.174+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=Z41S7ISEB31H tag=E7TY7GEPYSUV type=write size=1328184 2024-03-09T14:36:13.183 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.182+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=6SR5J5Q7ACP3 type=del size=2952405 2024-03-09T14:36:13.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.182+0000 7f73315818c0 1 < completed delete operation key=DQOY1JJ1YQ2W tag=ITQRU60RQ25R type=del 2024-03-09T14:36:13.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.186+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=Z4B1ORJ4XQWU type=del size=2979860 2024-03-09T14:36:13.194 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.190+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~CV8GHRYKUHJW.0 size=1048576 2024-03-09T14:36:13.194 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.190+0000 7f73315818c0 1 < canceled part key=_multipart_W6KVP8EQFQ8J.2~CV8GHRYKUHJW.1 size=1048576 2024-03-09T14:36:13.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.194+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~CV8GHRYKUHJW.2 size=747392 2024-03-09T14:36:13.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.198+0000 7f73315818c0 1 < canceled multipart upload key=W6KVP8EQFQ8J upload=CV8GHRYKUHJW tag=9P4V8A0POMY7 type=write size=3893120 2024-03-09T14:36:13.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.206+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~IQHZUTU8WL1T.0 size=1048576 2024-03-09T14:36:13.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.210+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~IQHZUTU8WL1T.1 size=1048576 2024-03-09T14:36:13.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.214+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~IQHZUTU8WL1T.2 size=304317 2024-03-09T14:36:13.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.214+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=IQHZUTU8WL1T tag=F5P09ND3P9SE type=write size=3450045 2024-03-09T14:36:13.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.214+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=RHNVZOCQ2J2F type=del 2024-03-09T14:36:13.224 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.222+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=VGQ8USW121BE type=del size=1973165 2024-03-09T14:36:13.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.226+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~Z03KA9VS8FJ0.0 size=985963 2024-03-09T14:36:13.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.230+0000 7f73315818c0 1 < completed multipart upload key=LZC7NPF0GRO7 upload=Z03KA9VS8FJ0 tag=RQ8LCUMVYX27 type=write size=2034539 2024-03-09T14:36:13.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.238+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~970HTLTGEDYU.0 size=1048576 2024-03-09T14:36:13.243 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.242+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~970HTLTGEDYU.1 size=919526 2024-03-09T14:36:13.243 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.242+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=970HTLTGEDYU tag=20P60TF6MJM2 type=write size=3016678 2024-03-09T14:36:13.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.242+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~I72J34INBOAR.0 size=961932 2024-03-09T14:36:13.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.250+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=I72J34INBOAR tag=XP9E8X5BQNEM type=write size=2010508 2024-03-09T14:36:13.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.254+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=1EHT5NQDDRYL type=write size=227172 2024-03-09T14:36:13.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.258+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=SB1QQBFTYADK type=del 2024-03-09T14:36:13.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.262+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~V4REMIJ86LPH.0 size=1048576 2024-03-09T14:36:13.269 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.266+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~V4REMIJ86LPH.1 size=837821 2024-03-09T14:36:13.269 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.266+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=V4REMIJ86LPH tag=SL94X47EGH3P type=write size=2934973 2024-03-09T14:36:13.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.270+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~WJB5UHE063OV.0 size=758038 2024-03-09T14:36:13.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.274+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=WJB5UHE063OV tag=QX1616L12JQ5 type=write size=1806614 2024-03-09T14:36:13.282 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.278+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=E3H70W8OUUHX type=write size=811810 2024-03-09T14:36:13.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.282+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~GAB3EZTZ1Z7R.0 size=1048576 2024-03-09T14:36:13.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.286+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~GAB3EZTZ1Z7R.1 size=1048576 2024-03-09T14:36:13.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.290+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~GAB3EZTZ1Z7R.2 size=1045451 2024-03-09T14:36:13.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.294+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=GAB3EZTZ1Z7R tag=QA2UZULG9IPD type=write size=4191179 2024-03-09T14:36:13.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.302+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~P5R5RAYUPKAA.0 size=1048576 2024-03-09T14:36:13.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.306+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~P5R5RAYUPKAA.1 size=1048576 2024-03-09T14:36:13.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.306+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~P5R5RAYUPKAA.2 size=204464 2024-03-09T14:36:13.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.306+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=P5R5RAYUPKAA tag=V649YWV1X6OG type=write size=3350192 2024-03-09T14:36:13.314 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.310+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=GII8MQXMQ8JX type=del 2024-03-09T14:36:13.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.314+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=J0WRRLVNVCAI type=del size=541221 2024-03-09T14:36:13.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.318+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=PXYBEYBJKKO1 type=del 2024-03-09T14:36:13.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.322+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=NOUDZ3XP7M4P type=del size=4015734 2024-03-09T14:36:13.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.326+0000 7f73315818c0 1 < canceled operation key=PXFZLDSAR1EG tag=24RJVOR14FRL type=del size=2021411 2024-03-09T14:36:13.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.330+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=3NFTWZ1DLNEM type=del size=1219393 2024-03-09T14:36:13.337 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.334+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~A4MJDDVNK0SG.0 size=1048576 2024-03-09T14:36:13.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.338+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~A4MJDDVNK0SG.1 size=846784 2024-03-09T14:36:13.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.342+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=A4MJDDVNK0SG tag=31EGR5ZCIV2A type=write size=2943936 2024-03-09T14:36:13.348 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.346+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=ZQWDVC9PG1FQ type=del size=1806946 2024-03-09T14:36:13.352 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.350+0000 7f73315818c0 1 < completed write operation key=3ADG4GWVARC1 tag=DJV0C9D9RSK7 type=write size=726466 2024-03-09T14:36:13.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.354+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=2EWWR1B805HP type=del size=698129 2024-03-09T14:36:13.359 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.354+0000 7f73315818c0 1 < canceled operation key=8O5OP8C5NX12 tag=TSQ2JA34MXPQ type=del size=2504530 2024-03-09T14:36:13.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.358+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=YE7F7SG7KDRE type=write size=756607 2024-03-09T14:36:13.366 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.362+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=DTI3JQD9U7UE type=del 2024-03-09T14:36:13.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.366+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=3GRY0OW72X4Q type=write size=547694 2024-03-09T14:36:13.373 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.370+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=6SR5J5Q7ACP3 type=del 2024-03-09T14:36:13.376 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.374+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=G6ASPBM0O6EW type=write size=304610 2024-03-09T14:36:13.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.378+0000 7f73315818c0 1 < suggested remove operation key=M1B6O6N8N84T tag=Z4B1ORJ4XQWU type=del 2024-03-09T14:36:13.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.386+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~XBIB0E6Y9084.0 size=1048576 2024-03-09T14:36:13.390 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.386+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~XBIB0E6Y9084.1 size=1048576 2024-03-09T14:36:13.393 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.390+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~XBIB0E6Y9084.2 size=17369 2024-03-09T14:36:13.393 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.390+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=XBIB0E6Y9084 tag=S0KIO1OUJVBN type=write size=3163097 2024-03-09T14:36:13.394 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.390+0000 7f73315818c0 1 < canceled part key=_multipart_9B0RAHZZGLYU.2~IQHZUTU8WL1T.0 size=1048576 2024-03-09T14:36:13.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.394+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~IQHZUTU8WL1T.1 size=1048576 2024-03-09T14:36:13.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.398+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~IQHZUTU8WL1T.2 size=304317 2024-03-09T14:36:13.404 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.402+0000 7f73315818c0 1 < canceled multipart upload key=9B0RAHZZGLYU upload=IQHZUTU8WL1T tag=F5P09ND3P9SE type=write size=3450045 2024-03-09T14:36:13.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.406+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~BBZ7JZPJEQST.0 size=387055 2024-03-09T14:36:13.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.406+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=BBZ7JZPJEQST tag=RAS8BX056JOE type=write size=1435631 2024-03-09T14:36:13.413 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.410+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=VGQ8USW121BE type=del 2024-03-09T14:36:13.417 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.414+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=0IBI932PK00F type=del size=507738 2024-03-09T14:36:13.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.418+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~970HTLTGEDYU.0 size=1048576 2024-03-09T14:36:13.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.418+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~970HTLTGEDYU.1 size=919526 2024-03-09T14:36:13.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.426+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=970HTLTGEDYU tag=20P60TF6MJM2 type=write size=3016678 2024-03-09T14:36:13.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.430+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~NOSEI3WZLLQF.0 size=1048576 2024-03-09T14:36:13.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.434+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~NOSEI3WZLLQF.1 size=763795 2024-03-09T14:36:13.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.434+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=NOSEI3WZLLQF tag=1BNZEKDX6JKX type=write size=2860947 2024-03-09T14:36:13.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.434+0000 7f73315818c0 1 < completed write operation key=PXFZLDSAR1EG tag=1EHT5NQDDRYL type=write size=227172 2024-03-09T14:36:13.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.442+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~Q34X2RTURZWN.0 size=648692 2024-03-09T14:36:13.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.442+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=Q34X2RTURZWN tag=X6QISTS84ZD4 type=write size=1697268 2024-03-09T14:36:13.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.446+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~V4REMIJ86LPH.0 size=1048576 2024-03-09T14:36:13.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.446+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~V4REMIJ86LPH.1 size=837821 2024-03-09T14:36:13.455 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.450+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=V4REMIJ86LPH tag=SL94X47EGH3P type=write size=2934973 2024-03-09T14:36:13.461 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.458+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~GTH99F884DYO.0 size=252061 2024-03-09T14:36:13.461 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.458+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=GTH99F884DYO tag=EGSQLO4GSCHX type=write size=1300637 2024-03-09T14:36:13.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.462+0000 7f73315818c0 1 < completed write operation key=DQOY1JJ1YQ2W tag=E3H70W8OUUHX type=write size=811810 2024-03-09T14:36:13.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.462+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=AKVEYIRZHK43 type=del size=1729543 2024-03-09T14:36:13.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.466+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~P5R5RAYUPKAA.0 size=1048576 2024-03-09T14:36:13.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.474+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~P5R5RAYUPKAA.1 size=1048576 2024-03-09T14:36:13.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.474+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~P5R5RAYUPKAA.2 size=204464 2024-03-09T14:36:13.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.482+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=P5R5RAYUPKAA tag=V649YWV1X6OG type=write size=3350192 2024-03-09T14:36:13.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.482+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=Y7G8UPBWYRBD type=del size=2149203 2024-03-09T14:36:13.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.490+0000 7f73315818c0 1 < suggested remove operation key=W6KVP8EQFQ8J tag=J0WRRLVNVCAI type=del 2024-03-09T14:36:13.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.494+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=WRL78XRC6JCL type=write size=341113 2024-03-09T14:36:13.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.502+0000 7f73315818c0 1 < suggested remove operation key=M8F9TA80VME6 tag=NOUDZ3XP7M4P type=del 2024-03-09T14:36:13.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.502+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=NU3MGMTZHPIL type=del size=2506422 2024-03-09T14:36:13.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.506+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=3NFTWZ1DLNEM type=del 2024-03-09T14:36:13.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.510+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=HNJ7DXAU509T type=del size=1797723 2024-03-09T14:36:13.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.514+0000 7f73315818c0 -1 < failed to complete operation key=W6KVP8EQFQ8J tag=ZQWDVC9PG1FQ type=del size=1806946: (22) Invalid argument 2024-03-09T14:36:13.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.518+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=CZYOKEADKBP3 type=del size=1473134 2024-03-09T14:36:13.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.522+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=2EWWR1B805HP type=del 2024-03-09T14:36:13.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.526+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~B90D4HNIZ4A2.0 size=119759 2024-03-09T14:36:13.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.526+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=B90D4HNIZ4A2 tag=RTO3PTAP1INO type=write size=1168335 2024-03-09T14:36:13.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.530+0000 7f73315818c0 1 < completed write operation key=1NYNL8M5DVER tag=YE7F7SG7KDRE type=write size=756607 2024-03-09T14:36:13.535 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.534+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=1XC9LCVG51GZ type=del size=843197 2024-03-09T14:36:13.541 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.538+0000 7f73315818c0 1 < suggested update operation key=1NYNL8M5DVER tag=3GRY0OW72X4Q type=write size=547694 2024-03-09T14:36:13.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.542+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=GNUJP1XIQTSF type=del size=193414 2024-03-09T14:36:13.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.546+0000 7f73315818c0 1 < completed write operation key=LZC7NPF0GRO7 tag=G6ASPBM0O6EW type=write size=304610 2024-03-09T14:36:13.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.550+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~97FRF2A6TERQ.0 size=550148 2024-03-09T14:36:13.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.550+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=97FRF2A6TERQ tag=XXSLMUX5H2DP type=write size=1598724 2024-03-09T14:36:13.557 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.554+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~XBIB0E6Y9084.0 size=1048576 2024-03-09T14:36:13.557 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.554+0000 7f73315818c0 1 < canceled part key=_multipart_M8F9TA80VME6.2~XBIB0E6Y9084.1 size=1048576 2024-03-09T14:36:13.560 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.558+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~XBIB0E6Y9084.2 size=17369 2024-03-09T14:36:13.565 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.562+0000 7f73315818c0 1 < canceled multipart upload key=M8F9TA80VME6 upload=XBIB0E6Y9084 tag=S0KIO1OUJVBN type=write size=3163097 2024-03-09T14:36:13.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.566+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=ITH6C7RCYQZM type=del size=2433652 2024-03-09T14:36:13.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.570+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~BBZ7JZPJEQST.0 size=387055 2024-03-09T14:36:13.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.574+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=BBZ7JZPJEQST tag=RAS8BX056JOE type=write size=1435631 2024-03-09T14:36:13.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.578+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=SNF1EGI0WZ34 type=del size=3063547 2024-03-09T14:36:13.585 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.582+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=0IBI932PK00F type=del 2024-03-09T14:36:13.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.586+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=HQ2KDHC45818 type=del size=1582764 2024-03-09T14:36:13.591 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.586+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~NOSEI3WZLLQF.0 size=1048576 2024-03-09T14:36:13.593 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.590+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~NOSEI3WZLLQF.1 size=763795 2024-03-09T14:36:13.599 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.594+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=NOSEI3WZLLQF tag=1BNZEKDX6JKX type=write size=2860947 2024-03-09T14:36:13.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.598+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=IPYYQ2F9J92U type=del size=3582613 2024-03-09T14:36:13.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.602+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~Q34X2RTURZWN.0 size=648692 2024-03-09T14:36:13.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.606+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=Q34X2RTURZWN tag=X6QISTS84ZD4 type=write size=1697268 2024-03-09T14:36:13.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.614+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~DNUXOGM9HKVC.0 size=1048576 2024-03-09T14:36:13.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.618+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~DNUXOGM9HKVC.1 size=1048576 2024-03-09T14:36:13.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.618+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~DNUXOGM9HKVC.2 size=838999 2024-03-09T14:36:13.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.618+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=DNUXOGM9HKVC tag=OFXHJ732KVOP type=write size=3984727 2024-03-09T14:36:13.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.622+0000 7f73315818c0 1 < canceled part key=_multipart_IA0AB7UVLFEJ.2~GTH99F884DYO.0 size=252061 2024-03-09T14:36:13.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.626+0000 7f73315818c0 1 < canceled multipart upload key=IA0AB7UVLFEJ upload=GTH99F884DYO tag=EGSQLO4GSCHX type=write size=1300637 2024-03-09T14:36:13.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.630+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=4ATRMRK7J7AY type=del size=3651989 2024-03-09T14:36:13.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.634+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=AKVEYIRZHK43 type=del 2024-03-09T14:36:13.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.638+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~FHSIPNIWO4ZU.0 size=1048576 2024-03-09T14:36:13.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.642+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~FHSIPNIWO4ZU.1 size=404708 2024-03-09T14:36:13.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.642+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=FHSIPNIWO4ZU tag=MC5C84CC9U4F type=write size=2501860 2024-03-09T14:36:13.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.646+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=Y7G8UPBWYRBD type=del 2024-03-09T14:36:13.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.650+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~EY6GRRZST49E.0 size=1048576 2024-03-09T14:36:13.657 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.654+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~EY6GRRZST49E.1 size=654095 2024-03-09T14:36:13.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.654+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=EY6GRRZST49E tag=ZWPZMDV1U3V3 type=write size=2751247 2024-03-09T14:36:13.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.658+0000 7f73315818c0 1 < completed write operation key=LZC7NPF0GRO7 tag=WRL78XRC6JCL type=write size=341113 2024-03-09T14:36:13.665 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.662+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=FYBJNF7WCZDE type=del size=2991939 2024-03-09T14:36:13.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.666+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=NU3MGMTZHPIL type=del 2024-03-09T14:36:13.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.670+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~BELRUZMJ2QPX.0 size=1048576 2024-03-09T14:36:13.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.674+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~BELRUZMJ2QPX.1 size=1048576 2024-03-09T14:36:13.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.678+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~BELRUZMJ2QPX.2 size=224834 2024-03-09T14:36:13.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.678+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=BELRUZMJ2QPX tag=5E3V5BEPM3X3 type=write size=3370562 2024-03-09T14:36:13.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.678+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=HNJ7DXAU509T type=del 2024-03-09T14:36:13.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.682+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=VGRPCTUYC3CN type=del size=2997180 2024-03-09T14:36:13.689 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.686+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=CZYOKEADKBP3 type=del 2024-03-09T14:36:13.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.694+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~3VXY24Z3NSMC.0 size=1048576 2024-03-09T14:36:13.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.694+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~3VXY24Z3NSMC.1 size=145027 2024-03-09T14:36:13.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.694+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=3VXY24Z3NSMC tag=CNTSA08T5Y8V type=write size=2242179 2024-03-09T14:36:13.699 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.694+0000 7f73315818c0 1 < canceled part key=_multipart_DQOY1JJ1YQ2W.2~B90D4HNIZ4A2.0 size=119759 2024-03-09T14:36:13.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.702+0000 7f73315818c0 1 < canceled multipart upload key=DQOY1JJ1YQ2W upload=B90D4HNIZ4A2 tag=RTO3PTAP1INO type=write size=1168335 2024-03-09T14:36:13.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.706+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=G058TDWHT8MT type=del size=65787 2024-03-09T14:36:13.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.706+0000 7f73315818c0 1 < canceled operation key=FYFVW9R0M1PX tag=1XC9LCVG51GZ type=del size=843197 2024-03-09T14:36:13.714 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.710+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=1P8NFT88516J type=del size=1875948 2024-03-09T14:36:13.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.714+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=GNUJP1XIQTSF type=del 2024-03-09T14:36:13.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.718+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=BGZ7017P3PQG type=del size=1824685 2024-03-09T14:36:13.724 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.722+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~97FRF2A6TERQ.0 size=550148 2024-03-09T14:36:13.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.726+0000 7f73315818c0 1 < completed multipart upload key=LZC7NPF0GRO7 upload=97FRF2A6TERQ tag=XXSLMUX5H2DP type=write size=1598724 2024-03-09T14:36:13.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.734+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~OF471VD40FTJ.0 size=1048576 2024-03-09T14:36:13.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.734+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~OF471VD40FTJ.1 size=1048576 2024-03-09T14:36:13.741 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.738+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~OF471VD40FTJ.2 size=79559 2024-03-09T14:36:13.741 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.738+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=OF471VD40FTJ tag=QZ9S2PD16NBY type=write size=3225287 2024-03-09T14:36:13.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.742+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=ITH6C7RCYQZM type=del 2024-03-09T14:36:13.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.746+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~4NIT4D9DZYTW.0 size=570229 2024-03-09T14:36:13.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.746+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=4NIT4D9DZYTW tag=TRSD8SLY99M9 type=write size=1618805 2024-03-09T14:36:13.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.750+0000 7f73315818c0 1 < canceled operation key=XELG1AWATMVL tag=SNF1EGI0WZ34 type=del size=3063547 2024-03-09T14:36:13.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.754+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=KV3NTVGHT6L1 type=del size=2469293 2024-03-09T14:36:13.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.758+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=HQ2KDHC45818 type=del 2024-03-09T14:36:13.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.762+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=6AQO3BL0RML2 type=del size=1020505 2024-03-09T14:36:13.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.770+0000 7f73315818c0 1 < suggested remove operation key=UMSNM53HTY1J tag=IPYYQ2F9J92U type=del 2024-03-09T14:36:13.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.774+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~P3C5AR0S6H1W.0 size=271447 2024-03-09T14:36:13.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.774+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=P3C5AR0S6H1W tag=0D98L6ZVP6WZ type=write size=1320023 2024-03-09T14:36:13.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.778+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~DNUXOGM9HKVC.0 size=1048576 2024-03-09T14:36:13.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.782+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~DNUXOGM9HKVC.1 size=1048576 2024-03-09T14:36:13.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.782+0000 7f73315818c0 1 < canceled part key=_multipart_4P89277EEYJ3.2~DNUXOGM9HKVC.2 size=838999 2024-03-09T14:36:13.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.790+0000 7f73315818c0 1 < canceled multipart upload key=4P89277EEYJ3 upload=DNUXOGM9HKVC tag=OFXHJ732KVOP type=write size=3984727 2024-03-09T14:36:13.796 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.794+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=USF4WUTJNOFB type=del size=2306290 2024-03-09T14:36:13.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.794+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=4ATRMRK7J7AY type=del 2024-03-09T14:36:13.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.798+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=RX1T7J5QQPT3 type=del size=3926727 2024-03-09T14:36:13.803 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.802+0000 7f73315818c0 1 < canceled part key=_multipart_DQOY1JJ1YQ2W.2~FHSIPNIWO4ZU.0 size=1048576 2024-03-09T14:36:13.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.802+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~FHSIPNIWO4ZU.1 size=404708 2024-03-09T14:36:13.812 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.810+0000 7f73315818c0 1 < canceled multipart upload key=DQOY1JJ1YQ2W upload=FHSIPNIWO4ZU tag=MC5C84CC9U4F type=write size=2501860 2024-03-09T14:36:13.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.814+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~U0NYJQKO0H0A.0 size=1048576 2024-03-09T14:36:13.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.818+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~U0NYJQKO0H0A.1 size=1048576 2024-03-09T14:36:13.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.822+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~U0NYJQKO0H0A.2 size=467554 2024-03-09T14:36:13.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.822+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=U0NYJQKO0H0A tag=VZ89PA72M5G5 type=write size=3613282 2024-03-09T14:36:13.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.822+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~EY6GRRZST49E.0 size=1048576 2024-03-09T14:36:13.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.822+0000 7f73315818c0 1 < canceled part key=_multipart_M8F9TA80VME6.2~EY6GRRZST49E.1 size=654095 2024-03-09T14:36:13.833 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.830+0000 7f73315818c0 1 < canceled multipart upload key=M8F9TA80VME6 upload=EY6GRRZST49E tag=ZWPZMDV1U3V3 type=write size=2751247 2024-03-09T14:36:13.836 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.834+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=KH59AF6SG5QJ type=del size=2985531 2024-03-09T14:36:13.840 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.838+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=FYBJNF7WCZDE type=del 2024-03-09T14:36:13.846 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.842+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~TT1Y5I3I0LJM.0 size=1048576 2024-03-09T14:36:13.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.846+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~TT1Y5I3I0LJM.1 size=1048576 2024-03-09T14:36:13.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.846+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~TT1Y5I3I0LJM.2 size=544187 2024-03-09T14:36:13.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.846+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=TT1Y5I3I0LJM tag=JAYN2YVISEQ9 type=write size=3689915 2024-03-09T14:36:13.854 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.850+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~BELRUZMJ2QPX.0 size=1048576 2024-03-09T14:36:13.857 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.854+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~BELRUZMJ2QPX.1 size=1048576 2024-03-09T14:36:13.860 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.858+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~BELRUZMJ2QPX.2 size=224834 2024-03-09T14:36:13.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.862+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=BELRUZMJ2QPX tag=5E3V5BEPM3X3 type=write size=3370562 2024-03-09T14:36:13.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.866+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=F2E5TPFUDLHE type=del size=1168393 2024-03-09T14:36:13.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.870+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=VGRPCTUYC3CN type=del 2024-03-09T14:36:13.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.874+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~XTEKRL7OL12U.0 size=1048576 2024-03-09T14:36:13.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.878+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~XTEKRL7OL12U.1 size=39066 2024-03-09T14:36:13.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.878+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=XTEKRL7OL12U tag=7YX314L1XOJH type=write size=2136218 2024-03-09T14:36:13.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.882+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~3VXY24Z3NSMC.0 size=1048576 2024-03-09T14:36:13.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.886+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~3VXY24Z3NSMC.1 size=145027 2024-03-09T14:36:13.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.890+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=3VXY24Z3NSMC tag=CNTSA08T5Y8V type=write size=2242179 2024-03-09T14:36:13.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.898+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~G6OCPEX3S12G.0 size=1048576 2024-03-09T14:36:13.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.898+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~G6OCPEX3S12G.1 size=276222 2024-03-09T14:36:13.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.898+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=G6OCPEX3S12G tag=A6X50VWDYPJ1 type=write size=2373374 2024-03-09T14:36:13.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.902+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=G058TDWHT8MT type=del 2024-03-09T14:36:13.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.910+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~14A2BRSPVIU3.0 size=1048576 2024-03-09T14:36:13.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.910+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~14A2BRSPVIU3.1 size=956753 2024-03-09T14:36:13.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.910+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=14A2BRSPVIU3 tag=PWXS07329KHG type=write size=3053905 2024-03-09T14:36:13.917 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.914+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=1P8NFT88516J type=del 2024-03-09T14:36:13.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.918+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=1SIO8IC2PRE6 type=del size=452781 2024-03-09T14:36:13.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.922+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=BGZ7017P3PQG type=del 2024-03-09T14:36:13.928 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.926+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=X0AGX9NARKD5 type=del size=1213567 2024-03-09T14:36:13.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.930+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~OF471VD40FTJ.0 size=1048576 2024-03-09T14:36:13.935 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.930+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~OF471VD40FTJ.1 size=1048576 2024-03-09T14:36:13.938 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.934+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~OF471VD40FTJ.2 size=79559 2024-03-09T14:36:13.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.942+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=OF471VD40FTJ tag=QZ9S2PD16NBY type=write size=3225287 2024-03-09T14:36:13.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.942+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=W0058L9H3DR2 type=del size=679575 2024-03-09T14:36:13.950 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.946+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~4NIT4D9DZYTW.0 size=570229 2024-03-09T14:36:13.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.954+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=4NIT4D9DZYTW tag=TRSD8SLY99M9 type=write size=1618805 2024-03-09T14:36:13.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.954+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=V6H33EL1Y6XR type=del size=2753574 2024-03-09T14:36:13.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.962+0000 7f73315818c0 1 < suggested remove operation key=3ADG4GWVARC1 tag=KV3NTVGHT6L1 type=del 2024-03-09T14:36:13.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.970+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~PW7M08J26HFU.0 size=1048576 2024-03-09T14:36:13.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.974+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~PW7M08J26HFU.1 size=1048576 2024-03-09T14:36:13.978 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.974+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~PW7M08J26HFU.2 size=158669 2024-03-09T14:36:13.978 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.974+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=PW7M08J26HFU tag=815F45SNHYCB type=write size=3304397 2024-03-09T14:36:13.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.982+0000 7f73315818c0 1 < suggested update operation key=0IB00CZ92VV4 tag=6AQO3BL0RML2 type=del size=1020505 2024-03-09T14:36:13.988 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.986+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=E3IQ43G8GTC1 type=write size=30072 2024-03-09T14:36:13.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.990+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~P3C5AR0S6H1W.0 size=271447 2024-03-09T14:36:13.998 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.994+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=P3C5AR0S6H1W tag=0D98L6ZVP6WZ type=write size=1320023 2024-03-09T14:36:14.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:13.998+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=SJ5I7LAJPKEM type=del size=3398815 2024-03-09T14:36:14.006 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.002+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=USF4WUTJNOFB type=del 2024-03-09T14:36:14.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.010+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~KSWBZASX2RB1.0 size=1048576 2024-03-09T14:36:14.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.010+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~KSWBZASX2RB1.1 size=333576 2024-03-09T14:36:14.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.010+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=KSWBZASX2RB1 tag=CO25TGAEDIK8 type=write size=2430728 2024-03-09T14:36:14.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.018+0000 7f73315818c0 1 < suggested update operation key=0WGGMHTEF2LB tag=RX1T7J5QQPT3 type=del size=3926727 2024-03-09T14:36:14.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.022+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=JDE8HM1LCEFQ type=del size=394834 2024-03-09T14:36:14.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.022+0000 7f73315818c0 1 < canceled part key=_multipart_4P89277EEYJ3.2~U0NYJQKO0H0A.0 size=1048576 2024-03-09T14:36:14.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.026+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~U0NYJQKO0H0A.1 size=1048576 2024-03-09T14:36:14.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.030+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~U0NYJQKO0H0A.2 size=467554 2024-03-09T14:36:14.038 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.034+0000 7f73315818c0 1 < canceled multipart upload key=4P89277EEYJ3 upload=U0NYJQKO0H0A tag=VZ89PA72M5G5 type=write size=3613282 2024-03-09T14:36:14.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.038+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=RDQZPTBYPNA8 type=del size=2270317 2024-03-09T14:36:14.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.042+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=KH59AF6SG5QJ type=del 2024-03-09T14:36:14.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.046+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=Q24QK7DFCGAW type=del size=684191 2024-03-09T14:36:14.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.050+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~TT1Y5I3I0LJM.0 size=1048576 2024-03-09T14:36:14.056 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.054+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~TT1Y5I3I0LJM.1 size=1048576 2024-03-09T14:36:14.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.054+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~TT1Y5I3I0LJM.2 size=544187 2024-03-09T14:36:14.065 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.062+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=TT1Y5I3I0LJM tag=JAYN2YVISEQ9 type=write size=3689915 2024-03-09T14:36:14.068 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.066+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=AMR4MLX5HTC1 type=del size=1968275 2024-03-09T14:36:14.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.070+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=F2E5TPFUDLHE type=del 2024-03-09T14:36:14.076 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.074+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=F7Y1MYCE5DUJ type=del size=566 2024-03-09T14:36:14.080 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.078+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~XTEKRL7OL12U.0 size=1048576 2024-03-09T14:36:14.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.078+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~XTEKRL7OL12U.1 size=39066 2024-03-09T14:36:14.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.086+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=XTEKRL7OL12U tag=7YX314L1XOJH type=write size=2136218 2024-03-09T14:36:14.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.090+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=V00I0I6SF3HZ type=del size=1208040 2024-03-09T14:36:14.097 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.094+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~G6OCPEX3S12G.0 size=1048576 2024-03-09T14:36:14.100 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.098+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~G6OCPEX3S12G.1 size=276222 2024-03-09T14:36:14.106 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.102+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=G6OCPEX3S12G tag=A6X50VWDYPJ1 type=write size=2373374 2024-03-09T14:36:14.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.110+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=1ZDCEHJDU80K type=del size=642870 2024-03-09T14:36:14.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.110+0000 7f73315818c0 1 < canceled part key=_multipart_DQOY1JJ1YQ2W.2~14A2BRSPVIU3.0 size=1048576 2024-03-09T14:36:14.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.114+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~14A2BRSPVIU3.1 size=956753 2024-03-09T14:36:14.122 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.118+0000 7f73315818c0 1 < canceled multipart upload key=DQOY1JJ1YQ2W upload=14A2BRSPVIU3 tag=PWXS07329KHG type=write size=3053905 2024-03-09T14:36:14.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.126+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~8QKG9IZR0488.0 size=1048576 2024-03-09T14:36:14.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.126+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~8QKG9IZR0488.1 size=350961 2024-03-09T14:36:14.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.126+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=8QKG9IZR0488 tag=6I3JUTLEWLYF type=write size=2448113 2024-03-09T14:36:14.135 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.130+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=1SIO8IC2PRE6 type=del 2024-03-09T14:36:14.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.134+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=G6LO067O4MLP type=write size=123928 2024-03-09T14:36:14.142 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.138+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=X0AGX9NARKD5 type=del 2024-03-09T14:36:14.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.142+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=F031J1Q1JUA3 type=del size=798326 2024-03-09T14:36:14.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.146+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=W0058L9H3DR2 type=del 2024-03-09T14:36:14.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.150+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=JL58U9SDCZUM type=del size=112167 2024-03-09T14:36:14.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.154+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=V6H33EL1Y6XR type=del 2024-03-09T14:36:14.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.158+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=RG20DUUGX34Q type=del size=3046974 2024-03-09T14:36:14.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.162+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~PW7M08J26HFU.0 size=1048576 2024-03-09T14:36:14.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.162+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~PW7M08J26HFU.1 size=1048576 2024-03-09T14:36:14.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.166+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~PW7M08J26HFU.2 size=158669 2024-03-09T14:36:14.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.174+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=PW7M08J26HFU tag=815F45SNHYCB type=write size=3304397 2024-03-09T14:36:14.182 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.178+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~P84IE1A3R47H.0 size=1048576 2024-03-09T14:36:14.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.182+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~P84IE1A3R47H.1 size=1048576 2024-03-09T14:36:14.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.186+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~P84IE1A3R47H.2 size=602610 2024-03-09T14:36:14.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.186+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=P84IE1A3R47H tag=20JJFFZ6R98H type=write size=3748338 2024-03-09T14:36:14.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.186+0000 7f73315818c0 1 < completed write operation key=K3JNN1NCKBXN tag=E3IQ43G8GTC1 type=write size=30072 2024-03-09T14:36:14.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.194+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~WAMN3QAPS2FT.0 size=1048576 2024-03-09T14:36:14.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.198+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~WAMN3QAPS2FT.1 size=33403 2024-03-09T14:36:14.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.198+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=WAMN3QAPS2FT tag=Q2BHMSFGO95Q type=write size=2130555 2024-03-09T14:36:14.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.198+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=SJ5I7LAJPKEM type=del 2024-03-09T14:36:14.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.202+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=11XQEZCMOW3G type=del size=2885246 2024-03-09T14:36:14.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.206+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~KSWBZASX2RB1.0 size=1048576 2024-03-09T14:36:14.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.210+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~KSWBZASX2RB1.1 size=333576 2024-03-09T14:36:14.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.214+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=KSWBZASX2RB1 tag=CO25TGAEDIK8 type=write size=2430728 2024-03-09T14:36:14.225 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.222+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~RQWDGU5HGVHU.0 size=1048576 2024-03-09T14:36:14.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.222+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~RQWDGU5HGVHU.1 size=1048576 2024-03-09T14:36:14.230 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.226+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~RQWDGU5HGVHU.2 size=789180 2024-03-09T14:36:14.230 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.226+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=RQWDGU5HGVHU tag=RZOQP0KZMYVO type=write size=3934908 2024-03-09T14:36:14.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.230+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=JDE8HM1LCEFQ type=del 2024-03-09T14:36:14.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.238+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~K0BHF9MRTYFG.0 size=1048576 2024-03-09T14:36:14.243 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.238+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~K0BHF9MRTYFG.1 size=86395 2024-03-09T14:36:14.243 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.238+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=K0BHF9MRTYFG tag=G6WE998JHNVN type=write size=2183547 2024-03-09T14:36:14.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.242+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=RDQZPTBYPNA8 type=del 2024-03-09T14:36:14.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.250+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~U9P9DYZGJKIE.0 size=1048576 2024-03-09T14:36:14.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.254+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~U9P9DYZGJKIE.1 size=777385 2024-03-09T14:36:14.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.254+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=U9P9DYZGJKIE tag=5E2LXCCY9KY7 type=write size=2874537 2024-03-09T14:36:14.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.258+0000 7f73315818c0 1 < canceled operation key=XELG1AWATMVL tag=Q24QK7DFCGAW type=del size=684191 2024-03-09T14:36:14.267 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.262+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~4V49TN7I8P04.0 size=923944 2024-03-09T14:36:14.267 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.262+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=4V49TN7I8P04 tag=0C65G0A5GM0M type=write size=1972520 2024-03-09T14:36:14.271 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.266+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=AMR4MLX5HTC1 type=del 2024-03-09T14:36:14.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.270+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=NZ595UQRKVYG type=del size=3328985 2024-03-09T14:36:14.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.274+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=F7Y1MYCE5DUJ type=del 2024-03-09T14:36:14.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.278+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=PADX8Y1ONSHQ type=del size=988281 2024-03-09T14:36:14.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.282+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=V00I0I6SF3HZ type=del 2024-03-09T14:36:14.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.290+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~Y8WWTJUT05WZ.0 size=1048576 2024-03-09T14:36:14.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.294+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~Y8WWTJUT05WZ.1 size=81823 2024-03-09T14:36:14.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.294+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=Y8WWTJUT05WZ tag=KI697FYB6DHA type=write size=2178975 2024-03-09T14:36:14.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.298+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=1ZDCEHJDU80K type=del 2024-03-09T14:36:14.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.302+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=AQ6DOSCMZL4S type=del size=747391 2024-03-09T14:36:14.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.302+0000 7f73315818c0 1 < canceled part key=_multipart_162AOMHO2BQL.2~8QKG9IZR0488.0 size=1048576 2024-03-09T14:36:14.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.302+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~8QKG9IZR0488.1 size=350961 2024-03-09T14:36:14.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.310+0000 7f73315818c0 1 < canceled multipart upload key=162AOMHO2BQL upload=8QKG9IZR0488 tag=6I3JUTLEWLYF type=write size=2448113 2024-03-09T14:36:14.316 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.314+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=0KYPBUKQT94P type=write size=14073 2024-03-09T14:36:14.323 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.318+0000 7f73315818c0 1 < suggested remove operation key=0IB00CZ92VV4 tag=G6LO067O4MLP type=write 2024-03-09T14:36:14.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.326+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~4ZQTN8IAN42H.0 size=254823 2024-03-09T14:36:14.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.326+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=4ZQTN8IAN42H tag=DKYKVX74NF0W type=write size=1303399 2024-03-09T14:36:14.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.330+0000 7f73315818c0 1 < completed delete operation key=DQOY1JJ1YQ2W tag=F031J1Q1JUA3 type=del 2024-03-09T14:36:14.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.330+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=TVI0ERPJLO9K type=del size=2139364 2024-03-09T14:36:14.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.334+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=JL58U9SDCZUM type=del 2024-03-09T14:36:14.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.342+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~2Q3LWFEVQ2QT.0 size=1048576 2024-03-09T14:36:14.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.342+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~2Q3LWFEVQ2QT.1 size=1048576 2024-03-09T14:36:14.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.346+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~2Q3LWFEVQ2QT.2 size=771897 2024-03-09T14:36:14.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.346+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=2Q3LWFEVQ2QT tag=H547ZT2T9KO1 type=write size=3917625 2024-03-09T14:36:14.353 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.350+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=RG20DUUGX34Q type=del 2024-03-09T14:36:14.359 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.358+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~P8W9TMU1P3AU.0 size=1048576 2024-03-09T14:36:14.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.358+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~P8W9TMU1P3AU.1 size=837992 2024-03-09T14:36:14.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.358+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=P8W9TMU1P3AU tag=F0EPMMQXFTXB type=write size=2935144 2024-03-09T14:36:14.366 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.362+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~P84IE1A3R47H.0 size=1048576 2024-03-09T14:36:14.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.366+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~P84IE1A3R47H.1 size=1048576 2024-03-09T14:36:14.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.366+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~P84IE1A3R47H.2 size=602610 2024-03-09T14:36:14.376 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.374+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=P84IE1A3R47H tag=20JJFFZ6R98H type=write size=3748338 2024-03-09T14:36:14.379 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.374+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=76T3I8OU2ERU type=del size=267006 2024-03-09T14:36:14.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.378+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~WAMN3QAPS2FT.0 size=1048576 2024-03-09T14:36:14.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.382+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~WAMN3QAPS2FT.1 size=33403 2024-03-09T14:36:14.391 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.386+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=WAMN3QAPS2FT tag=Q2BHMSFGO95Q type=write size=2130555 2024-03-09T14:36:14.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.390+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=L1077VCFDNOJ type=del size=2672179 2024-03-09T14:36:14.398 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.394+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=11XQEZCMOW3G type=del 2024-03-09T14:36:14.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.398+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=RZKS6KN6Q84M type=del size=2562011 2024-03-09T14:36:14.404 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.402+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~RQWDGU5HGVHU.0 size=1048576 2024-03-09T14:36:14.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.406+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~RQWDGU5HGVHU.1 size=1048576 2024-03-09T14:36:14.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.406+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~RQWDGU5HGVHU.2 size=789180 2024-03-09T14:36:14.416 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.414+0000 7f73315818c0 1 < completed multipart upload key=IA0AB7UVLFEJ upload=RQWDGU5HGVHU tag=RZOQP0KZMYVO type=write size=3934908 2024-03-09T14:36:14.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.418+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=JFR6QW1Q8UMR type=del size=2740073 2024-03-09T14:36:14.424 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.422+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~K0BHF9MRTYFG.0 size=1048576 2024-03-09T14:36:14.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.426+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~K0BHF9MRTYFG.1 size=86395 2024-03-09T14:36:14.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.434+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=K0BHF9MRTYFG tag=G6WE998JHNVN type=write size=2183547 2024-03-09T14:36:14.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.442+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~VWNYM5Z1PGYV.0 size=398744 2024-03-09T14:36:14.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.442+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=VWNYM5Z1PGYV tag=7XSHA52SK79N type=write size=1447320 2024-03-09T14:36:14.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.446+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~U9P9DYZGJKIE.0 size=1048576 2024-03-09T14:36:14.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.450+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~U9P9DYZGJKIE.1 size=777385 2024-03-09T14:36:14.459 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.458+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=U9P9DYZGJKIE tag=5E2LXCCY9KY7 type=write size=2874537 2024-03-09T14:36:14.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.462+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~W8099FSBTVYJ.0 size=1048576 2024-03-09T14:36:14.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.466+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~W8099FSBTVYJ.1 size=819999 2024-03-09T14:36:14.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.466+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=W8099FSBTVYJ tag=5PB80IVQWWDK type=write size=2917151 2024-03-09T14:36:14.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.470+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~4V49TN7I8P04.0 size=923944 2024-03-09T14:36:14.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.474+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=4V49TN7I8P04 tag=0C65G0A5GM0M type=write size=1972520 2024-03-09T14:36:14.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.478+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=J4X7KSBQE5JS type=del size=843789 2024-03-09T14:36:14.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.482+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=NZ595UQRKVYG type=del 2024-03-09T14:36:14.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.486+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~O93FW0R1LIDB.0 size=1048576 2024-03-09T14:36:14.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.490+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~O93FW0R1LIDB.1 size=195448 2024-03-09T14:36:14.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.490+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=O93FW0R1LIDB tag=M4BPXMMJPB8H type=write size=2292600 2024-03-09T14:36:14.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.494+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=PADX8Y1ONSHQ type=del 2024-03-09T14:36:14.503 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.498+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~43ESGLSXAQ1C.0 size=1048576 2024-03-09T14:36:14.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.502+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~43ESGLSXAQ1C.1 size=730015 2024-03-09T14:36:14.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.502+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=43ESGLSXAQ1C tag=OS6LFAO1248N type=write size=2827167 2024-03-09T14:36:14.509 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.506+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~Y8WWTJUT05WZ.0 size=1048576 2024-03-09T14:36:14.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.506+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~Y8WWTJUT05WZ.1 size=81823 2024-03-09T14:36:14.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.514+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=Y8WWTJUT05WZ tag=KI697FYB6DHA type=write size=2178975 2024-03-09T14:36:14.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.522+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~D0SPBH3P52WA.0 size=1048576 2024-03-09T14:36:14.526 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.522+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~D0SPBH3P52WA.1 size=1048576 2024-03-09T14:36:14.528 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.526+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~D0SPBH3P52WA.2 size=955461 2024-03-09T14:36:14.528 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.526+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=D0SPBH3P52WA tag=L3QHNPNN5YMQ type=write size=4101189 2024-03-09T14:36:14.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.530+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=AQ6DOSCMZL4S type=del 2024-03-09T14:36:14.535 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.530+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=PGABYVVBEA35 type=del size=72129 2024-03-09T14:36:14.538 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.534+0000 7f73315818c0 1 < completed write operation key=K3JNN1NCKBXN tag=0KYPBUKQT94P type=write size=14073 2024-03-09T14:36:14.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.538+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=QPRIH3YV0BJP type=write size=240511 2024-03-09T14:36:14.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.542+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~4ZQTN8IAN42H.0 size=254823 2024-03-09T14:36:14.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.546+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=4ZQTN8IAN42H tag=DKYKVX74NF0W type=write size=1303399 2024-03-09T14:36:14.557 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.554+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~3XT9MU1ASH9H.0 size=1048576 2024-03-09T14:36:14.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.554+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~3XT9MU1ASH9H.1 size=932582 2024-03-09T14:36:14.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.554+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=3XT9MU1ASH9H tag=73NQPPVADHXC type=write size=3029734 2024-03-09T14:36:14.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.558+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=TVI0ERPJLO9K type=del 2024-03-09T14:36:14.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.562+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=F7WQOUR0USZH type=del size=678924 2024-03-09T14:36:14.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.566+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~2Q3LWFEVQ2QT.0 size=1048576 2024-03-09T14:36:14.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.570+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~2Q3LWFEVQ2QT.1 size=1048576 2024-03-09T14:36:14.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.570+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~2Q3LWFEVQ2QT.2 size=771897 2024-03-09T14:36:14.581 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.578+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=2Q3LWFEVQ2QT tag=H547ZT2T9KO1 type=write size=3917625 2024-03-09T14:36:14.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.582+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~IEVN9BE8IJWZ.0 size=1048576 2024-03-09T14:36:14.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.586+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~IEVN9BE8IJWZ.1 size=354661 2024-03-09T14:36:14.590 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.586+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=IEVN9BE8IJWZ tag=2FP7FIAFVP3B type=write size=2451813 2024-03-09T14:36:14.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.590+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~P8W9TMU1P3AU.0 size=1048576 2024-03-09T14:36:14.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.590+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~P8W9TMU1P3AU.1 size=837992 2024-03-09T14:36:14.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.598+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=P8W9TMU1P3AU tag=F0EPMMQXFTXB type=write size=2935144 2024-03-09T14:36:14.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.602+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=03HX3KQMWRWF type=write size=138754 2024-03-09T14:36:14.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.602+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=76T3I8OU2ERU type=del 2024-03-09T14:36:14.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.606+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=8MXK7ZC5464M type=write size=613450 2024-03-09T14:36:14.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.610+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=L1077VCFDNOJ type=del 2024-03-09T14:36:14.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.614+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=TTUI562JW76H type=del size=119043 2024-03-09T14:36:14.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.618+0000 7f73315818c0 1 < canceled operation key=PXFZLDSAR1EG tag=RZKS6KN6Q84M type=del size=2562011 2024-03-09T14:36:14.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.618+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=UI8CIPCC7EXG type=del size=2405487 2024-03-09T14:36:14.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.626+0000 7f73315818c0 1 < suggested remove operation key=162AOMHO2BQL tag=JFR6QW1Q8UMR type=del 2024-03-09T14:36:14.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.630+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=Y2750C1UTCNO type=del size=2688131 2024-03-09T14:36:14.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.630+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~VWNYM5Z1PGYV.0 size=398744 2024-03-09T14:36:14.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.638+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=VWNYM5Z1PGYV tag=7XSHA52SK79N type=write size=1447320 2024-03-09T14:36:14.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.642+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=94IWP4NMP84L type=del size=161214 2024-03-09T14:36:14.647 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.642+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~W8099FSBTVYJ.0 size=1048576 2024-03-09T14:36:14.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.646+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~W8099FSBTVYJ.1 size=819999 2024-03-09T14:36:14.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.650+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=W8099FSBTVYJ tag=5PB80IVQWWDK type=write size=2917151 2024-03-09T14:36:14.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.654+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=7OTO2O1FK750 type=del size=3069525 2024-03-09T14:36:14.662 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.658+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=J4X7KSBQE5JS type=del 2024-03-09T14:36:14.665 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.662+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=BVXEXP9JMI2I type=write size=196624 2024-03-09T14:36:14.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.666+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~O93FW0R1LIDB.0 size=1048576 2024-03-09T14:36:14.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.670+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~O93FW0R1LIDB.1 size=195448 2024-03-09T14:36:14.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.674+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=O93FW0R1LIDB tag=M4BPXMMJPB8H type=write size=2292600 2024-03-09T14:36:14.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.678+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=IPCXPNVLP5YR type=del size=3128436 2024-03-09T14:36:14.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.682+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~43ESGLSXAQ1C.0 size=1048576 2024-03-09T14:36:14.688 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.686+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~43ESGLSXAQ1C.1 size=730015 2024-03-09T14:36:14.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.690+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=43ESGLSXAQ1C tag=OS6LFAO1248N type=write size=2827167 2024-03-09T14:36:14.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.694+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=HIG9VRJ374H6 type=del size=2082628 2024-03-09T14:36:14.702 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.698+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~D0SPBH3P52WA.0 size=1048576 2024-03-09T14:36:14.702 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.698+0000 7f73315818c0 1 < canceled part key=_multipart_UMSNM53HTY1J.2~D0SPBH3P52WA.1 size=1048576 2024-03-09T14:36:14.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.702+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~D0SPBH3P52WA.2 size=955461 2024-03-09T14:36:14.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.706+0000 7f73315818c0 1 < canceled multipart upload key=UMSNM53HTY1J upload=D0SPBH3P52WA tag=L3QHNPNN5YMQ type=write size=4101189 2024-03-09T14:36:14.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.710+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=CA4N6J05M3MO type=del size=3144619 2024-03-09T14:36:14.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.718+0000 7f73315818c0 1 < suggested update operation key=PAJYGD2GXHDT tag=PGABYVVBEA35 type=del size=72129 2024-03-09T14:36:14.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.722+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~B53RVJWCSETI.0 size=671765 2024-03-09T14:36:14.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.722+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=B53RVJWCSETI tag=8ANDVRNGNT37 type=write size=1720341 2024-03-09T14:36:14.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.730+0000 7f73315818c0 1 < suggested update operation key=0LP8KKU53X0A tag=QPRIH3YV0BJP type=write size=240511 2024-03-09T14:36:14.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.734+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=2SICPQ1OALB1 type=del size=1381850 2024-03-09T14:36:14.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.738+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~3XT9MU1ASH9H.0 size=1048576 2024-03-09T14:36:14.743 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.742+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~3XT9MU1ASH9H.1 size=932582 2024-03-09T14:36:14.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.750+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=3XT9MU1ASH9H tag=73NQPPVADHXC type=write size=3029734 2024-03-09T14:36:14.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.754+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=4RPMVHKXQI3F type=del size=3445497 2024-03-09T14:36:14.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.758+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=F7WQOUR0USZH type=del 2024-03-09T14:36:14.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.762+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=1CP12E7URZJ1 type=del size=3919600 2024-03-09T14:36:14.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.766+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~IEVN9BE8IJWZ.0 size=1048576 2024-03-09T14:36:14.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.770+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~IEVN9BE8IJWZ.1 size=354661 2024-03-09T14:36:14.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.774+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=IEVN9BE8IJWZ tag=2FP7FIAFVP3B type=write size=2451813 2024-03-09T14:36:14.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.778+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=PQK3RAKDYJ3I type=del size=1881338 2024-03-09T14:36:14.786 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.782+0000 7f73315818c0 1 < completed write operation key=4E3QMBHEAEHO tag=03HX3KQMWRWF type=write size=138754 2024-03-09T14:36:14.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.786+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=7HSKSKMY2ZH5 type=del size=679743 2024-03-09T14:36:14.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.790+0000 7f73315818c0 1 < completed write operation key=92GWY5B0K4SX tag=8MXK7ZC5464M type=write size=613450 2024-03-09T14:36:14.797 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.794+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=1E3DH0WBGS8Y type=del size=1210553 2024-03-09T14:36:14.801 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.798+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=TTUI562JW76H type=del 2024-03-09T14:36:14.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.802+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=RQM2NT3OIE1I type=write size=227428 2024-03-09T14:36:14.808 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.806+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=UI8CIPCC7EXG type=del 2024-03-09T14:36:14.811 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.806+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=50FYHWPDMODX type=del size=284426 2024-03-09T14:36:14.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.810+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=Y2750C1UTCNO type=del 2024-03-09T14:36:14.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.814+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=MTGIN1JFFO75 type=del size=1480828 2024-03-09T14:36:14.826 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.822+0000 7f73315818c0 1 < suggested remove operation key=4P89277EEYJ3 tag=94IWP4NMP84L type=del 2024-03-09T14:36:14.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.826+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=KMO1WZ1PW6GL type=del size=4181601 2024-03-09T14:36:14.834 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.830+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=7OTO2O1FK750 type=del 2024-03-09T14:36:14.837 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.834+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=N8L4ASHP2EHZ type=del size=3753742 2024-03-09T14:36:14.845 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.842+0000 7f73315818c0 1 < suggested remove operation key=CVAWRZMAX997 tag=BVXEXP9JMI2I type=write 2024-03-09T14:36:14.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.846+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=P889JLLHMA8A type=write size=398588 2024-03-09T14:36:14.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.850+0000 7f73315818c0 1 < canceled operation key=PXFZLDSAR1EG tag=IPCXPNVLP5YR type=del size=3128436 2024-03-09T14:36:14.860 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.858+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~E8NH8WSM0GY3.0 size=1048576 2024-03-09T14:36:14.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.858+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~E8NH8WSM0GY3.1 size=1048576 2024-03-09T14:36:14.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.862+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~E8NH8WSM0GY3.2 size=291942 2024-03-09T14:36:14.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.862+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=E8NH8WSM0GY3 tag=VZTH77HT4MTZ type=write size=3437670 2024-03-09T14:36:14.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.870+0000 7f73315818c0 1 < suggested update operation key=W6KVP8EQFQ8J tag=HIG9VRJ374H6 type=del size=2082628 2024-03-09T14:36:14.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.874+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~7ZP1E81QJWHJ.0 size=1048576 2024-03-09T14:36:14.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.878+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~7ZP1E81QJWHJ.1 size=966972 2024-03-09T14:36:14.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.878+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=7ZP1E81QJWHJ tag=2SZ9CO2SCH2R type=write size=3064124 2024-03-09T14:36:14.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.886+0000 7f73315818c0 1 < suggested remove operation key=M1B6O6N8N84T tag=CA4N6J05M3MO type=del 2024-03-09T14:36:14.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.890+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=32ROLDQW2CGI type=del size=2998739 2024-03-09T14:36:14.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.894+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~B53RVJWCSETI.0 size=671765 2024-03-09T14:36:14.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.898+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=B53RVJWCSETI tag=8ANDVRNGNT37 type=write size=1720341 2024-03-09T14:36:14.904 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.902+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=Z15NCH1MNY9A type=del size=3361666 2024-03-09T14:36:14.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.902+0000 7f73315818c0 -1 < no bi entry to suggest for operation key=4P89277EEYJ3 tag=2SICPQ1OALB1 type=del size=1381850: (2) No such file or directory 2024-03-09T14:36:14.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.906+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=8TYI35D8IUID type=del size=3581600 2024-03-09T14:36:14.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.910+0000 7f73315818c0 -1 < failed to complete operation key=4P89277EEYJ3 tag=4RPMVHKXQI3F type=del size=3445497: (22) Invalid argument 2024-03-09T14:36:14.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.918+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~2FY2RT224Z0P.0 size=1048576 2024-03-09T14:36:14.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.918+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~2FY2RT224Z0P.1 size=1048576 2024-03-09T14:36:14.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.922+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~2FY2RT224Z0P.2 size=176587 2024-03-09T14:36:14.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.922+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=2FY2RT224Z0P tag=3VPQX2BEPQIN type=write size=3322315 2024-03-09T14:36:14.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.926+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=1CP12E7URZJ1 type=del 2024-03-09T14:36:14.937 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.934+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~SR4GQULN8BDP.0 size=966917 2024-03-09T14:36:14.937 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.934+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=SR4GQULN8BDP tag=CXMQ5UGM8C4U type=write size=2015493 2024-03-09T14:36:14.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.938+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=PQK3RAKDYJ3I type=del 2024-03-09T14:36:14.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.942+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=ZEHE4BCXITBW type=del size=3168142 2024-03-09T14:36:14.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.950+0000 7f73315818c0 1 < suggested remove operation key=FYFVW9R0M1PX tag=7HSKSKMY2ZH5 type=del 2024-03-09T14:36:14.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.954+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=EOZN01P12LN7 type=del size=3111173 2024-03-09T14:36:14.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.958+0000 7f73315818c0 -1 < failed to complete operation key=M1B6O6N8N84T tag=1E3DH0WBGS8Y type=del size=1210553: (22) Invalid argument 2024-03-09T14:36:14.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.958+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=YIOMBRCWO17L type=del size=2454360 2024-03-09T14:36:14.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.962+0000 7f73315818c0 1 < completed write operation key=DQOY1JJ1YQ2W tag=RQM2NT3OIE1I type=write size=227428 2024-03-09T14:36:14.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.966+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=QM8L85FCU09U type=del size=999022 2024-03-09T14:36:14.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.970+0000 7f73315818c0 -1 < failed to complete operation key=CVAWRZMAX997 tag=50FYHWPDMODX type=del size=284426: (22) Invalid argument 2024-03-09T14:36:14.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.974+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=480XE4Q3AHAV type=write size=948666 2024-03-09T14:36:14.980 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.978+0000 7f73315818c0 -1 < failed to complete operation key=4P89277EEYJ3 tag=MTGIN1JFFO75 type=del size=1480828: (22) Invalid argument 2024-03-09T14:36:14.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.982+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=488SFZEJHPTA type=del size=2379601 2024-03-09T14:36:14.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.986+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=KMO1WZ1PW6GL type=del 2024-03-09T14:36:14.993 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.990+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~AV79WFGEIF5J.0 size=1048576 2024-03-09T14:36:14.996 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.994+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~AV79WFGEIF5J.1 size=286435 2024-03-09T14:36:14.996 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.994+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=AV79WFGEIF5J tag=3NBEHOKI4W2P type=write size=2383587 2024-03-09T14:36:14.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:14.994+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=N8L4ASHP2EHZ type=del 2024-03-09T14:36:15.005 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.002+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~TA8R26JSU8MA.0 size=1048576 2024-03-09T14:36:15.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.006+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~TA8R26JSU8MA.1 size=1024693 2024-03-09T14:36:15.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.006+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=TA8R26JSU8MA tag=N2AJLFPV00TF type=write size=3121845 2024-03-09T14:36:15.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.010+0000 7f73315818c0 1 < suggested update operation key=P9LI5ESR3N9G tag=P889JLLHMA8A type=write size=398588 2024-03-09T14:36:15.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.018+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~OV6UYVHLDM17.0 size=951080 2024-03-09T14:36:15.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.018+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=OV6UYVHLDM17 tag=U2ZZNO1V2JIT type=write size=1999656 2024-03-09T14:36:15.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.022+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~E8NH8WSM0GY3.0 size=1048576 2024-03-09T14:36:15.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.026+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~E8NH8WSM0GY3.1 size=1048576 2024-03-09T14:36:15.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.026+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~E8NH8WSM0GY3.2 size=291942 2024-03-09T14:36:15.036 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.034+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=E8NH8WSM0GY3 tag=VZTH77HT4MTZ type=write size=3437670 2024-03-09T14:36:15.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.038+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~RIT53BZD0JKI.0 size=1048576 2024-03-09T14:36:15.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.042+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~RIT53BZD0JKI.1 size=838793 2024-03-09T14:36:15.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.042+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=RIT53BZD0JKI tag=10SE4UCVT0O0 type=write size=2935945 2024-03-09T14:36:15.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.046+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~7ZP1E81QJWHJ.0 size=1048576 2024-03-09T14:36:15.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.046+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~7ZP1E81QJWHJ.1 size=966972 2024-03-09T14:36:15.056 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.054+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=7ZP1E81QJWHJ tag=2SZ9CO2SCH2R type=write size=3064124 2024-03-09T14:36:15.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.058+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~0U06D5V88Z39.0 size=945273 2024-03-09T14:36:15.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.058+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=0U06D5V88Z39 tag=UXGEO5QYDO0Z type=write size=1993849 2024-03-09T14:36:15.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.062+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=32ROLDQW2CGI type=del 2024-03-09T14:36:15.073 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.070+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~YX1NLRIZBINM.0 size=314185 2024-03-09T14:36:15.073 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.070+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=YX1NLRIZBINM tag=SC3X6WOI4SWA type=write size=1362761 2024-03-09T14:36:15.076 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.074+0000 7f73315818c0 1 < canceled operation key=S302JFYEDEL7 tag=Z15NCH1MNY9A type=del size=3361666 2024-03-09T14:36:15.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.078+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~LDH8ICL0VC8A.0 size=1048576 2024-03-09T14:36:15.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.082+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~LDH8ICL0VC8A.1 size=276205 2024-03-09T14:36:15.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.082+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=LDH8ICL0VC8A tag=7XRQP3PU1LR4 type=write size=2373357 2024-03-09T14:36:15.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.086+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=8TYI35D8IUID type=del 2024-03-09T14:36:15.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.094+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=9DL04VH2PDN1 type=write size=874826 2024-03-09T14:36:15.099 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.094+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~2FY2RT224Z0P.0 size=1048576 2024-03-09T14:36:15.102 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.098+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~2FY2RT224Z0P.1 size=1048576 2024-03-09T14:36:15.102 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.098+0000 7f73315818c0 1 < canceled part key=_multipart_XELG1AWATMVL.2~2FY2RT224Z0P.2 size=176587 2024-03-09T14:36:15.106 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.102+0000 7f73315818c0 1 < canceled multipart upload key=XELG1AWATMVL upload=2FY2RT224Z0P tag=3VPQX2BEPQIN type=write size=3322315 2024-03-09T14:36:15.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.110+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~YNJRG93FZ2C5.0 size=1048576 2024-03-09T14:36:15.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.110+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~YNJRG93FZ2C5.1 size=352554 2024-03-09T14:36:15.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.110+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=YNJRG93FZ2C5 tag=SXYC15QDU0E4 type=write size=2449706 2024-03-09T14:36:15.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.110+0000 7f73315818c0 1 < canceled part key=_multipart_XELG1AWATMVL.2~SR4GQULN8BDP.0 size=966917 2024-03-09T14:36:15.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.118+0000 7f73315818c0 1 < canceled multipart upload key=XELG1AWATMVL upload=SR4GQULN8BDP tag=CXMQ5UGM8C4U type=write size=2015493 2024-03-09T14:36:15.126 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.122+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~GD5X1H2OS9QD.0 size=1048576 2024-03-09T14:36:15.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.126+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~GD5X1H2OS9QD.1 size=893634 2024-03-09T14:36:15.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.126+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=GD5X1H2OS9QD tag=G4YD57X7AOO5 type=write size=2990786 2024-03-09T14:36:15.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.130+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=ZEHE4BCXITBW type=del 2024-03-09T14:36:15.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.134+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~LI30BY0U4Q8X.0 size=1048576 2024-03-09T14:36:15.141 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.138+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~LI30BY0U4Q8X.1 size=1048576 2024-03-09T14:36:15.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.142+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~LI30BY0U4Q8X.2 size=686461 2024-03-09T14:36:15.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.142+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=LI30BY0U4Q8X tag=K2CD0J41LSTM type=write size=3832189 2024-03-09T14:36:15.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.146+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=EOZN01P12LN7 type=del 2024-03-09T14:36:15.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.150+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~DR57R4F2473Q.0 size=1048576 2024-03-09T14:36:15.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.154+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~DR57R4F2473Q.1 size=297066 2024-03-09T14:36:15.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.154+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=DR57R4F2473Q tag=UJ1UAYCS9WVM type=write size=2394218 2024-03-09T14:36:15.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.162+0000 7f73315818c0 1 < suggested update operation key=P9LI5ESR3N9G tag=YIOMBRCWO17L type=del size=2454360 2024-03-09T14:36:15.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.166+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~L3X3Z4S1CH9T.0 size=1048576 2024-03-09T14:36:15.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.170+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~L3X3Z4S1CH9T.1 size=163682 2024-03-09T14:36:15.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.170+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=L3X3Z4S1CH9T tag=PCW7JRIA2DZQ type=write size=2260834 2024-03-09T14:36:15.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.174+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=QM8L85FCU09U type=del 2024-03-09T14:36:15.183 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.178+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~JZ42PJY5VMH9.0 size=1048576 2024-03-09T14:36:15.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.182+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~JZ42PJY5VMH9.1 size=910490 2024-03-09T14:36:15.186 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.182+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=JZ42PJY5VMH9 tag=V87RWERCBCHB type=write size=3007642 2024-03-09T14:36:15.189 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.186+0000 7f73315818c0 1 < completed write operation key=MSMD3ANK4T7J tag=480XE4Q3AHAV type=write size=948666 2024-03-09T14:36:15.196 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.194+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~CLRMEXFJJM53.0 size=1048576 2024-03-09T14:36:15.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.194+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~CLRMEXFJJM53.1 size=2170 2024-03-09T14:36:15.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.194+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=CLRMEXFJJM53 tag=ZEQZD05TO5JW type=write size=2099322 2024-03-09T14:36:15.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.198+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=488SFZEJHPTA type=del 2024-03-09T14:36:15.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.202+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=ULCBZF1FXPLW type=write size=1021714 2024-03-09T14:36:15.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.206+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~AV79WFGEIF5J.0 size=1048576 2024-03-09T14:36:15.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.210+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~AV79WFGEIF5J.1 size=286435 2024-03-09T14:36:15.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.214+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=AV79WFGEIF5J tag=3NBEHOKI4W2P type=write size=2383587 2024-03-09T14:36:15.222 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.218+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=L662J9D213HO type=del size=3264722 2024-03-09T14:36:15.225 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.222+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~TA8R26JSU8MA.0 size=1048576 2024-03-09T14:36:15.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.226+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~TA8R26JSU8MA.1 size=1024693 2024-03-09T14:36:15.235 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.230+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=TA8R26JSU8MA tag=N2AJLFPV00TF type=write size=3121845 2024-03-09T14:36:15.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.234+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=P3U3HI519HC9 type=del size=3204864 2024-03-09T14:36:15.242 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.238+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~OV6UYVHLDM17.0 size=951080 2024-03-09T14:36:15.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.246+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=OV6UYVHLDM17 tag=U2ZZNO1V2JIT type=write size=1999656 2024-03-09T14:36:15.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.250+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~DI30I4CIYQGX.0 size=1048576 2024-03-09T14:36:15.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.254+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~DI30I4CIYQGX.1 size=1048576 2024-03-09T14:36:15.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.258+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~DI30I4CIYQGX.2 size=671687 2024-03-09T14:36:15.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.258+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=DI30I4CIYQGX tag=BWDV7X5WZ3G4 type=write size=3817415 2024-03-09T14:36:15.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.258+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~RIT53BZD0JKI.0 size=1048576 2024-03-09T14:36:15.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.258+0000 7f73315818c0 1 < canceled part key=_multipart_DQOY1JJ1YQ2W.2~RIT53BZD0JKI.1 size=838793 2024-03-09T14:36:15.269 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.266+0000 7f73315818c0 1 < canceled multipart upload key=DQOY1JJ1YQ2W upload=RIT53BZD0JKI tag=10SE4UCVT0O0 type=write size=2935945 2024-03-09T14:36:15.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.270+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=0FNYXKYD27JH type=del size=398556 2024-03-09T14:36:15.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.274+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~0U06D5V88Z39.0 size=945273 2024-03-09T14:36:15.282 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.278+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=0U06D5V88Z39 tag=UXGEO5QYDO0Z type=write size=1993849 2024-03-09T14:36:15.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.282+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=KCZAYQ5VL2C5 type=del size=4148669 2024-03-09T14:36:15.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.286+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~YX1NLRIZBINM.0 size=314185 2024-03-09T14:36:15.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.290+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=YX1NLRIZBINM tag=SC3X6WOI4SWA type=write size=1362761 2024-03-09T14:36:15.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.294+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=DEA0035JMEVM type=del size=406753 2024-03-09T14:36:15.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.298+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~LDH8ICL0VC8A.0 size=1048576 2024-03-09T14:36:15.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.302+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~LDH8ICL0VC8A.1 size=276205 2024-03-09T14:36:15.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.306+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=LDH8ICL0VC8A tag=7XRQP3PU1LR4 type=write size=2373357 2024-03-09T14:36:15.314 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.310+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~6OF3YDZ2YN70.0 size=1048576 2024-03-09T14:36:15.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.314+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~6OF3YDZ2YN70.1 size=1048576 2024-03-09T14:36:15.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.318+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~6OF3YDZ2YN70.2 size=37513 2024-03-09T14:36:15.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.318+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=6OF3YDZ2YN70 tag=NBQNBMTF88WL type=write size=3183241 2024-03-09T14:36:15.323 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.322+0000 7f73315818c0 1 < completed write operation key=W6KVP8EQFQ8J tag=9DL04VH2PDN1 type=write size=874826 2024-03-09T14:36:15.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.326+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~FZ6CGV57XWYE.0 size=335591 2024-03-09T14:36:15.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.326+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=FZ6CGV57XWYE tag=31SMJSJ4ZUCR type=write size=1384167 2024-03-09T14:36:15.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.330+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~YNJRG93FZ2C5.0 size=1048576 2024-03-09T14:36:15.337 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.334+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~YNJRG93FZ2C5.1 size=352554 2024-03-09T14:36:15.342 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.338+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=YNJRG93FZ2C5 tag=SXYC15QDU0E4 type=write size=2449706 2024-03-09T14:36:15.348 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.346+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~RG92MDFDN3CX.0 size=1048576 2024-03-09T14:36:15.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.346+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~RG92MDFDN3CX.1 size=1048576 2024-03-09T14:36:15.353 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.350+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~RG92MDFDN3CX.2 size=749009 2024-03-09T14:36:15.353 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.350+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=RG92MDFDN3CX tag=JJ6KG5NFF1ES type=write size=3894737 2024-03-09T14:36:15.356 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.354+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~GD5X1H2OS9QD.0 size=1048576 2024-03-09T14:36:15.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.358+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~GD5X1H2OS9QD.1 size=893634 2024-03-09T14:36:15.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.362+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=GD5X1H2OS9QD tag=G4YD57X7AOO5 type=write size=2990786 2024-03-09T14:36:15.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.366+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=4GC9XI1ND5BV type=del size=2248390 2024-03-09T14:36:15.372 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.370+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~LI30BY0U4Q8X.0 size=1048576 2024-03-09T14:36:15.375 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.370+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~LI30BY0U4Q8X.1 size=1048576 2024-03-09T14:36:15.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.374+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~LI30BY0U4Q8X.2 size=686461 2024-03-09T14:36:15.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.378+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=LI30BY0U4Q8X tag=K2CD0J41LSTM type=write size=3832189 2024-03-09T14:36:15.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.382+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=RJVGXUA9412E type=del size=200462 2024-03-09T14:36:15.389 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.386+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~DR57R4F2473Q.0 size=1048576 2024-03-09T14:36:15.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.390+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~DR57R4F2473Q.1 size=297066 2024-03-09T14:36:15.398 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.394+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=DR57R4F2473Q tag=UJ1UAYCS9WVM type=write size=2394218 2024-03-09T14:36:15.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.402+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=XGSE2CN6OAV3 type=write size=452934 2024-03-09T14:36:15.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.406+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~L3X3Z4S1CH9T.0 size=1048576 2024-03-09T14:36:15.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.410+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~L3X3Z4S1CH9T.1 size=163682 2024-03-09T14:36:15.418 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.414+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=L3X3Z4S1CH9T tag=PCW7JRIA2DZQ type=write size=2260834 2024-03-09T14:36:15.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.422+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~JAJRIS3S567Q.0 size=1048576 2024-03-09T14:36:15.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.426+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~JAJRIS3S567Q.1 size=1036646 2024-03-09T14:36:15.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.426+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=JAJRIS3S567Q tag=0ILAHQ3L8QQJ type=write size=3133798 2024-03-09T14:36:15.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.430+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~JZ42PJY5VMH9.0 size=1048576 2024-03-09T14:36:15.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.430+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~JZ42PJY5VMH9.1 size=910490 2024-03-09T14:36:15.440 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.438+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=JZ42PJY5VMH9 tag=V87RWERCBCHB type=write size=3007642 2024-03-09T14:36:15.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.442+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~JPHRPNFN3OQG.0 size=181961 2024-03-09T14:36:15.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.442+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=JPHRPNFN3OQG tag=Z885O6DY06AN type=write size=1230537 2024-03-09T14:36:15.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.446+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~CLRMEXFJJM53.0 size=1048576 2024-03-09T14:36:15.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.450+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~CLRMEXFJJM53.1 size=2170 2024-03-09T14:36:15.458 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.454+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=CLRMEXFJJM53 tag=ZEQZD05TO5JW type=write size=2099322 2024-03-09T14:36:15.462 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.458+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=SSHXG2ZCJSI9 type=del size=1049894 2024-03-09T14:36:15.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.462+0000 7f73315818c0 1 < completed write operation key=0IB00CZ92VV4 tag=ULCBZF1FXPLW type=write size=1021714 2024-03-09T14:36:15.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.470+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~C1VDO92LAZND.0 size=1048576 2024-03-09T14:36:15.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.470+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~C1VDO92LAZND.1 size=1048576 2024-03-09T14:36:15.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.474+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~C1VDO92LAZND.2 size=435956 2024-03-09T14:36:15.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.474+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=C1VDO92LAZND tag=CSDJNA5H30EE type=write size=3581684 2024-03-09T14:36:15.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.482+0000 7f73315818c0 1 < suggested update operation key=162AOMHO2BQL tag=L662J9D213HO type=del size=3264722 2024-03-09T14:36:15.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.486+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~PT837T73ASZP.0 size=156041 2024-03-09T14:36:15.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.486+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=PT837T73ASZP tag=06XAI8ENLH9A type=write size=1204617 2024-03-09T14:36:15.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.490+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=P3U3HI519HC9 type=del 2024-03-09T14:36:15.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.494+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=126V3SXZDHC3 type=del size=1704280 2024-03-09T14:36:15.500 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.498+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~DI30I4CIYQGX.0 size=1048576 2024-03-09T14:36:15.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.498+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~DI30I4CIYQGX.1 size=1048576 2024-03-09T14:36:15.505 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.502+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~DI30I4CIYQGX.2 size=671687 2024-03-09T14:36:15.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.506+0000 7f73315818c0 1 < completed multipart upload key=IA0AB7UVLFEJ upload=DI30I4CIYQGX tag=BWDV7X5WZ3G4 type=write size=3817415 2024-03-09T14:36:15.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.510+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=4AT1ZU401I0T type=del size=1295047 2024-03-09T14:36:15.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.518+0000 7f73315818c0 1 < suggested remove operation key=QG4O2L526B8E tag=0FNYXKYD27JH type=del 2024-03-09T14:36:15.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.518+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=131AOIDFS8OB type=del size=1621086 2024-03-09T14:36:15.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.526+0000 7f73315818c0 1 < suggested update operation key=CVAWRZMAX997 tag=KCZAYQ5VL2C5 type=del size=4148669 2024-03-09T14:36:15.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.530+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=A4JOZEQ0WNCW type=write size=666249 2024-03-09T14:36:15.535 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.534+0000 7f73315818c0 -1 < failed to complete operation key=QG4O2L526B8E tag=DEA0035JMEVM type=del size=406753: (22) Invalid argument 2024-03-09T14:36:15.539 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.534+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=UO98K5M16WQE type=del size=330125 2024-03-09T14:36:15.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.538+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~6OF3YDZ2YN70.0 size=1048576 2024-03-09T14:36:15.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.542+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~6OF3YDZ2YN70.1 size=1048576 2024-03-09T14:36:15.547 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.546+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~6OF3YDZ2YN70.2 size=37513 2024-03-09T14:36:15.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.550+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=6OF3YDZ2YN70 tag=NBQNBMTF88WL type=write size=3183241 2024-03-09T14:36:15.556 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.554+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=BMO245GH4W39 type=del size=2416709 2024-03-09T14:36:15.556 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.554+0000 7f73315818c0 1 < canceled part key=_multipart_QG4O2L526B8E.2~FZ6CGV57XWYE.0 size=335591 2024-03-09T14:36:15.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.558+0000 7f73315818c0 1 < canceled multipart upload key=QG4O2L526B8E upload=FZ6CGV57XWYE tag=31SMJSJ4ZUCR type=write size=1384167 2024-03-09T14:36:15.565 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.562+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=4AP7PGF9BUWU type=write size=1541 2024-03-09T14:36:15.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.566+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~RG92MDFDN3CX.0 size=1048576 2024-03-09T14:36:15.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.566+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~RG92MDFDN3CX.1 size=1048576 2024-03-09T14:36:15.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.570+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~RG92MDFDN3CX.2 size=749009 2024-03-09T14:36:15.581 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.578+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=RG92MDFDN3CX tag=JJ6KG5NFF1ES type=write size=3894737 2024-03-09T14:36:15.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.582+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=TX7INDM2MYHO type=del size=1890823 2024-03-09T14:36:15.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.582+0000 7f73315818c0 1 < completed delete operation key=DQOY1JJ1YQ2W tag=4GC9XI1ND5BV type=del 2024-03-09T14:36:15.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.586+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=BNM0IEXXVV08 type=del size=1537777 2024-03-09T14:36:15.593 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.590+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=RJVGXUA9412E type=del 2024-03-09T14:36:15.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.594+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=KRYSBJ1KBVAF type=del size=3644479 2024-03-09T14:36:15.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.598+0000 7f73315818c0 1 < suggested remove operation key=0IB00CZ92VV4 tag=XGSE2CN6OAV3 type=write 2024-03-09T14:36:15.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.602+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=72LYOGYVD3K0 type=write size=248532 2024-03-09T14:36:15.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.602+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~JAJRIS3S567Q.0 size=1048576 2024-03-09T14:36:15.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.606+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~JAJRIS3S567Q.1 size=1036646 2024-03-09T14:36:15.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.610+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=JAJRIS3S567Q tag=0ILAHQ3L8QQJ type=write size=3133798 2024-03-09T14:36:15.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.614+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=S81GRYUDI7V5 type=del size=2160429 2024-03-09T14:36:15.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.618+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~JPHRPNFN3OQG.0 size=181961 2024-03-09T14:36:15.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.622+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=JPHRPNFN3OQG tag=Z885O6DY06AN type=write size=1230537 2024-03-09T14:36:15.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.622+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=3DPD2HAVSVTG type=del size=1131441 2024-03-09T14:36:15.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.626+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=SSHXG2ZCJSI9 type=del 2024-03-09T14:36:15.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.630+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=J48W5FEIOW4Z type=del size=1518765 2024-03-09T14:36:15.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.634+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~C1VDO92LAZND.0 size=1048576 2024-03-09T14:36:15.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.634+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~C1VDO92LAZND.1 size=1048576 2024-03-09T14:36:15.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.638+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~C1VDO92LAZND.2 size=435956 2024-03-09T14:36:15.647 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.642+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=C1VDO92LAZND tag=CSDJNA5H30EE type=write size=3581684 2024-03-09T14:36:15.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.646+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=3A1HBS71EBVF type=del size=502143 2024-03-09T14:36:15.654 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.650+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~PT837T73ASZP.0 size=156041 2024-03-09T14:36:15.660 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.658+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=PT837T73ASZP tag=06XAI8ENLH9A type=write size=1204617 2024-03-09T14:36:15.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.662+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~PPVIWJTQJRFX.0 size=1048576 2024-03-09T14:36:15.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.666+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~PPVIWJTQJRFX.1 size=458115 2024-03-09T14:36:15.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.666+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=PPVIWJTQJRFX tag=GZWKRD0K1YVV type=write size=2555267 2024-03-09T14:36:15.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.666+0000 7f73315818c0 -1 < no bi entry to suggest for operation key=QG4O2L526B8E tag=126V3SXZDHC3 type=del size=1704280: (2) No such file or directory 2024-03-09T14:36:15.673 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.670+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=CP0DNERP4KS7 type=write size=120075 2024-03-09T14:36:15.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.674+0000 7f73315818c0 1 < suggested remove operation key=3ADG4GWVARC1 tag=4AT1ZU401I0T type=del 2024-03-09T14:36:15.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.678+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=DH1VYQDSVOGJ type=del size=1573388 2024-03-09T14:36:15.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.682+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=131AOIDFS8OB type=del 2024-03-09T14:36:15.690 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.686+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~G3H9CYOHA8ME.0 size=512721 2024-03-09T14:36:15.690 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.686+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=G3H9CYOHA8ME tag=AKCRVVWNP90P type=write size=1561297 2024-03-09T14:36:15.693 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.690+0000 7f73315818c0 1 < canceled operation key=PXFZLDSAR1EG tag=A4JOZEQ0WNCW type=write size=666249 2024-03-09T14:36:15.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.694+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~W734PXM2C0BU.0 size=1048576 2024-03-09T14:36:15.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.698+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~W734PXM2C0BU.1 size=635622 2024-03-09T14:36:15.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.698+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=W734PXM2C0BU tag=LDSQPB4P025W type=write size=2732774 2024-03-09T14:36:15.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.706+0000 7f73315818c0 1 < suggested update operation key=9B0RAHZZGLYU tag=UO98K5M16WQE type=del size=330125 2024-03-09T14:36:15.715 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.714+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~Y1TRF788X23V.0 size=1048576 2024-03-09T14:36:15.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.714+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~Y1TRF788X23V.1 size=1048576 2024-03-09T14:36:15.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.718+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~Y1TRF788X23V.2 size=834504 2024-03-09T14:36:15.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.718+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=Y1TRF788X23V tag=AIGSR2YDY226 type=write size=3980232 2024-03-09T14:36:15.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.722+0000 7f73315818c0 1 < canceled operation key=M1B6O6N8N84T tag=BMO245GH4W39 type=del size=2416709 2024-03-09T14:36:15.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.726+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~ULFYUEGBLHWA.0 size=1048576 2024-03-09T14:36:15.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.730+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~ULFYUEGBLHWA.1 size=1048576 2024-03-09T14:36:15.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.734+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~ULFYUEGBLHWA.2 size=442830 2024-03-09T14:36:15.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.734+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=ULFYUEGBLHWA tag=A7JJ3X73OO6D type=write size=3588558 2024-03-09T14:36:15.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.734+0000 7f73315818c0 1 < completed write operation key=W6KVP8EQFQ8J tag=4AP7PGF9BUWU type=write size=1541 2024-03-09T14:36:15.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.742+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~72144SPTTQJJ.0 size=1048576 2024-03-09T14:36:15.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.746+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~72144SPTTQJJ.1 size=1048576 2024-03-09T14:36:15.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.746+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~72144SPTTQJJ.2 size=693294 2024-03-09T14:36:15.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.746+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=72144SPTTQJJ tag=PZRF5QMQIIJG type=write size=3839022 2024-03-09T14:36:15.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.750+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=TX7INDM2MYHO type=del 2024-03-09T14:36:15.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.754+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=QRI2Z0ZOFS4Y type=del size=2001706 2024-03-09T14:36:15.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.758+0000 7f73315818c0 1 < completed delete operation key=M1B6O6N8N84T tag=BNM0IEXXVV08 type=del 2024-03-09T14:36:15.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.762+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=LIS13OIPXPMC type=del size=1651281 2024-03-09T14:36:15.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.766+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=KRYSBJ1KBVAF type=del 2024-03-09T14:36:15.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.766+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=WQNXJOF5FHYF type=del size=1520945 2024-03-09T14:36:15.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.770+0000 7f73315818c0 1 < completed write operation key=MD7FXGYXB823 tag=72LYOGYVD3K0 type=write size=248532 2024-03-09T14:36:15.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.774+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=6CZ44H3ZTDIV type=del size=940220 2024-03-09T14:36:15.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.782+0000 7f73315818c0 1 < suggested update operation key=0LP8KKU53X0A tag=S81GRYUDI7V5 type=del size=2160429 2024-03-09T14:36:15.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.786+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=AWE9E57WXMCS type=del size=708602 2024-03-09T14:36:15.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.786+0000 7f73315818c0 1 < completed delete operation key=M1B6O6N8N84T tag=3DPD2HAVSVTG type=del 2024-03-09T14:36:15.797 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.794+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~M7LN1G0VBIAB.0 size=1048576 2024-03-09T14:36:15.800 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.798+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~M7LN1G0VBIAB.1 size=1048576 2024-03-09T14:36:15.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.798+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~M7LN1G0VBIAB.2 size=810206 2024-03-09T14:36:15.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.798+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=M7LN1G0VBIAB tag=Z67QRHOEN5PN type=write size=3955934 2024-03-09T14:36:15.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.802+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=J48W5FEIOW4Z type=del 2024-03-09T14:36:15.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.806+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=FVX5FU74N39B type=del size=107425 2024-03-09T14:36:15.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.810+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=3A1HBS71EBVF type=del 2024-03-09T14:36:15.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.818+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~788YLJ169APQ.0 size=1048576 2024-03-09T14:36:15.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.818+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~788YLJ169APQ.1 size=1048576 2024-03-09T14:36:15.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.822+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~788YLJ169APQ.2 size=60401 2024-03-09T14:36:15.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.822+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=788YLJ169APQ tag=2TN64HABFSF6 type=write size=3206129 2024-03-09T14:36:15.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.826+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~PPVIWJTQJRFX.0 size=1048576 2024-03-09T14:36:15.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.826+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~PPVIWJTQJRFX.1 size=458115 2024-03-09T14:36:15.837 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.834+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=PPVIWJTQJRFX tag=GZWKRD0K1YVV type=write size=2555267 2024-03-09T14:36:15.844 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.842+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~6W8ZW9P4X83G.0 size=683467 2024-03-09T14:36:15.844 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.842+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=6W8ZW9P4X83G tag=XDDVHGMQ5LNA type=write size=1732043 2024-03-09T14:36:15.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.846+0000 7f73315818c0 1 < completed write operation key=OU0730HGI176 tag=CP0DNERP4KS7 type=write size=120075 2024-03-09T14:36:15.852 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.850+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=NUW1Q4HANDV0 type=del size=1441492 2024-03-09T14:36:15.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.854+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=DH1VYQDSVOGJ type=del 2024-03-09T14:36:15.860 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.858+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=N57W6X31QOI5 type=del size=2881358 2024-03-09T14:36:15.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.862+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~G3H9CYOHA8ME.0 size=512721 2024-03-09T14:36:15.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.866+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=G3H9CYOHA8ME tag=AKCRVVWNP90P type=write size=1561297 2024-03-09T14:36:15.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.870+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=K6N6V2Z6KQ51 type=del size=3894428 2024-03-09T14:36:15.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.874+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~W734PXM2C0BU.0 size=1048576 2024-03-09T14:36:15.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.874+0000 7f73315818c0 1 < canceled part key=_multipart_Y8EQAHMTL66U.2~W734PXM2C0BU.1 size=635622 2024-03-09T14:36:15.884 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.882+0000 7f73315818c0 1 < canceled multipart upload key=Y8EQAHMTL66U upload=W734PXM2C0BU tag=LDSQPB4P025W type=write size=2732774 2024-03-09T14:36:15.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.886+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~7N1VGMEOV9YJ.0 size=1048576 2024-03-09T14:36:15.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.890+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~7N1VGMEOV9YJ.1 size=818627 2024-03-09T14:36:15.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.890+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=7N1VGMEOV9YJ tag=7KSBZTL8MV2Q type=write size=2915779 2024-03-09T14:36:15.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.890+0000 7f73315818c0 1 < canceled part key=_multipart_Y8EQAHMTL66U.2~Y1TRF788X23V.0 size=1048576 2024-03-09T14:36:15.898 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.894+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~Y1TRF788X23V.1 size=1048576 2024-03-09T14:36:15.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.898+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~Y1TRF788X23V.2 size=834504 2024-03-09T14:36:15.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.906+0000 7f73315818c0 1 < canceled multipart upload key=Y8EQAHMTL66U upload=Y1TRF788X23V tag=AIGSR2YDY226 type=write size=3980232 2024-03-09T14:36:15.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.910+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~ZFKHFKSZDOD3.0 size=1048576 2024-03-09T14:36:15.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.914+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~ZFKHFKSZDOD3.1 size=1048576 2024-03-09T14:36:15.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.914+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~ZFKHFKSZDOD3.2 size=473756 2024-03-09T14:36:15.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.914+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=ZFKHFKSZDOD3 tag=182H6TB3HVIT type=write size=3619484 2024-03-09T14:36:15.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.918+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~ULFYUEGBLHWA.0 size=1048576 2024-03-09T14:36:15.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.922+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~ULFYUEGBLHWA.1 size=1048576 2024-03-09T14:36:15.928 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.926+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~ULFYUEGBLHWA.2 size=442830 2024-03-09T14:36:15.935 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.930+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=ULFYUEGBLHWA tag=A7JJ3X73OO6D type=write size=3588558 2024-03-09T14:36:15.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.938+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~2VXMV1E3Z2K7.0 size=1048576 2024-03-09T14:36:15.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.942+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~2VXMV1E3Z2K7.1 size=1048576 2024-03-09T14:36:15.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.942+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~2VXMV1E3Z2K7.2 size=239665 2024-03-09T14:36:15.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.942+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=2VXMV1E3Z2K7 tag=8DZQ8TUWZJQ7 type=write size=3385393 2024-03-09T14:36:15.950 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.946+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~72144SPTTQJJ.0 size=1048576 2024-03-09T14:36:15.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.950+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~72144SPTTQJJ.1 size=1048576 2024-03-09T14:36:15.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.950+0000 7f73315818c0 1 < canceled part key=_multipart_FYFVW9R0M1PX.2~72144SPTTQJJ.2 size=693294 2024-03-09T14:36:15.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.954+0000 7f73315818c0 1 < canceled multipart upload key=FYFVW9R0M1PX upload=72144SPTTQJJ tag=PZRF5QMQIIJG type=write size=3839022 2024-03-09T14:36:15.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.962+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~R4UBLLDCLE6F.0 size=1048576 2024-03-09T14:36:15.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.966+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~R4UBLLDCLE6F.1 size=1048576 2024-03-09T14:36:15.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.966+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~R4UBLLDCLE6F.2 size=17037 2024-03-09T14:36:15.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.966+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=R4UBLLDCLE6F tag=6MC69SXVBM4X type=write size=3162765 2024-03-09T14:36:15.974 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.970+0000 7f73315818c0 -1 < failed to complete operation key=0LP8KKU53X0A tag=QRI2Z0ZOFS4Y type=del size=2001706: (22) Invalid argument 2024-03-09T14:36:15.980 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.978+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~HOWWAXOODOYL.0 size=1048576 2024-03-09T14:36:15.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.978+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~HOWWAXOODOYL.1 size=12164 2024-03-09T14:36:15.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.978+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=HOWWAXOODOYL tag=KKDXSMJ61030 type=write size=2109316 2024-03-09T14:36:15.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.982+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=LIS13OIPXPMC type=del 2024-03-09T14:36:15.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.986+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=256II4XYQWIE type=write size=825423 2024-03-09T14:36:15.993 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.990+0000 7f73315818c0 1 < canceled operation key=9B0RAHZZGLYU tag=WQNXJOF5FHYF type=del size=1520945 2024-03-09T14:36:15.997 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.994+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=UAQ1MV29CVO4 type=del size=2482459 2024-03-09T14:36:16.000 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:15.998+0000 7f73315818c0 1 < canceled operation key=PAJYGD2GXHDT tag=6CZ44H3ZTDIV type=del size=940220 2024-03-09T14:36:16.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.002+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=8FE1PSWY8SPL type=write size=186163 2024-03-09T14:36:16.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.006+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=AWE9E57WXMCS type=del 2024-03-09T14:36:16.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.010+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~YP5M6V0J7PMI.0 size=1048576 2024-03-09T14:36:16.017 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.014+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~YP5M6V0J7PMI.1 size=1006889 2024-03-09T14:36:16.017 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.014+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=YP5M6V0J7PMI tag=CS4JAH1YBN85 type=write size=3104041 2024-03-09T14:36:16.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.018+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~M7LN1G0VBIAB.0 size=1048576 2024-03-09T14:36:16.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.018+0000 7f73315818c0 1 < canceled part key=_multipart_0LP8KKU53X0A.2~M7LN1G0VBIAB.1 size=1048576 2024-03-09T14:36:16.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.018+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~M7LN1G0VBIAB.2 size=810206 2024-03-09T14:36:16.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.030+0000 7f73315818c0 1 < canceled multipart upload key=0LP8KKU53X0A upload=M7LN1G0VBIAB tag=Z67QRHOEN5PN type=write size=3955934 2024-03-09T14:36:16.039 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.034+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~ZB2UT237M2O7.0 size=1048576 2024-03-09T14:36:16.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.038+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~ZB2UT237M2O7.1 size=1048576 2024-03-09T14:36:16.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.042+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~ZB2UT237M2O7.2 size=78425 2024-03-09T14:36:16.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.042+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=ZB2UT237M2O7 tag=IOVO6688LB0M type=write size=3224153 2024-03-09T14:36:16.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.050+0000 7f73315818c0 1 < suggested remove operation key=DQOY1JJ1YQ2W tag=FVX5FU74N39B type=del 2024-03-09T14:36:16.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.054+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~CFVLOYFJZJ5W.0 size=857732 2024-03-09T14:36:16.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.054+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=CFVLOYFJZJ5W tag=J3S334GW11WS type=write size=1906308 2024-03-09T14:36:16.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.058+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~788YLJ169APQ.0 size=1048576 2024-03-09T14:36:16.065 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.062+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~788YLJ169APQ.1 size=1048576 2024-03-09T14:36:16.068 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.066+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~788YLJ169APQ.2 size=60401 2024-03-09T14:36:16.075 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.070+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=788YLJ169APQ tag=2TN64HABFSF6 type=write size=3206129 2024-03-09T14:36:16.079 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.074+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=TK42G0MNOZ0G type=del size=724499 2024-03-09T14:36:16.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.078+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~6W8ZW9P4X83G.0 size=683467 2024-03-09T14:36:16.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.086+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=6W8ZW9P4X83G tag=XDDVHGMQ5LNA type=write size=1732043 2024-03-09T14:36:16.092 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.090+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=B7UPZVZ80M09 type=del size=3958770 2024-03-09T14:36:16.100 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.098+0000 7f73315818c0 1 < suggested remove operation key=P9LI5ESR3N9G tag=NUW1Q4HANDV0 type=del 2024-03-09T14:36:16.103 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.098+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=DEKR85YWU7RY type=del size=1674505 2024-03-09T14:36:16.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.102+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=N57W6X31QOI5 type=del 2024-03-09T14:36:16.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.106+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=N0T6ROSZOX69 type=del size=3718160 2024-03-09T14:36:16.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.110+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=K6N6V2Z6KQ51 type=del 2024-03-09T14:36:16.121 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.118+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~65ZUUTEJERHZ.0 size=1048576 2024-03-09T14:36:16.124 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.122+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~65ZUUTEJERHZ.1 size=442956 2024-03-09T14:36:16.124 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.122+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=65ZUUTEJERHZ tag=2JJDK2KYSLYU type=write size=2540108 2024-03-09T14:36:16.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.126+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~7N1VGMEOV9YJ.0 size=1048576 2024-03-09T14:36:16.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.126+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~7N1VGMEOV9YJ.1 size=818627 2024-03-09T14:36:16.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.134+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=7N1VGMEOV9YJ tag=7KSBZTL8MV2Q type=write size=2915779 2024-03-09T14:36:16.140 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.138+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=PKU1BHL7CI1C type=del size=4022124 2024-03-09T14:36:16.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.142+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~ZFKHFKSZDOD3.0 size=1048576 2024-03-09T14:36:16.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.142+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~ZFKHFKSZDOD3.1 size=1048576 2024-03-09T14:36:16.149 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.146+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~ZFKHFKSZDOD3.2 size=473756 2024-03-09T14:36:16.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.154+0000 7f73315818c0 1 < completed multipart upload key=LZC7NPF0GRO7 upload=ZFKHFKSZDOD3 tag=182H6TB3HVIT type=write size=3619484 2024-03-09T14:36:16.160 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.158+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=U3Q1VSWUAYP6 type=write size=890226 2024-03-09T14:36:16.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.162+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~2VXMV1E3Z2K7.0 size=1048576 2024-03-09T14:36:16.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.162+0000 7f73315818c0 1 < canceled part key=_multipart_PAJYGD2GXHDT.2~2VXMV1E3Z2K7.1 size=1048576 2024-03-09T14:36:16.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.162+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~2VXMV1E3Z2K7.2 size=239665 2024-03-09T14:36:16.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.170+0000 7f73315818c0 1 < canceled multipart upload key=PAJYGD2GXHDT upload=2VXMV1E3Z2K7 tag=8DZQ8TUWZJQ7 type=write size=3385393 2024-03-09T14:36:16.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.178+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~VSF05A9RXBXB.0 size=1048576 2024-03-09T14:36:16.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.182+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~VSF05A9RXBXB.1 size=129445 2024-03-09T14:36:16.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.182+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=VSF05A9RXBXB tag=S20C1LA4EDWO type=write size=2226597 2024-03-09T14:36:16.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.186+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~R4UBLLDCLE6F.0 size=1048576 2024-03-09T14:36:16.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.186+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~R4UBLLDCLE6F.1 size=1048576 2024-03-09T14:36:16.194 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.190+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~R4UBLLDCLE6F.2 size=17037 2024-03-09T14:36:16.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.198+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=R4UBLLDCLE6F tag=6MC69SXVBM4X type=write size=3162765 2024-03-09T14:36:16.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.202+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=PSYB6LH8S0I3 type=del size=4090227 2024-03-09T14:36:16.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.206+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~HOWWAXOODOYL.0 size=1048576 2024-03-09T14:36:16.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.210+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~HOWWAXOODOYL.1 size=12164 2024-03-09T14:36:16.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.214+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=HOWWAXOODOYL tag=KKDXSMJ61030 type=write size=2109316 2024-03-09T14:36:16.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.218+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=YTWTI7S14AOV type=write size=822521 2024-03-09T14:36:16.225 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.222+0000 7f73315818c0 1 < completed write operation key=162AOMHO2BQL tag=256II4XYQWIE type=write size=825423 2024-03-09T14:36:16.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.226+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=0S6990ZUEU7P type=write size=320446 2024-03-09T14:36:16.232 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.230+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=UAQ1MV29CVO4 type=del 2024-03-09T14:36:16.235 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.234+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=902B5C06JMZX type=del size=2674202 2024-03-09T14:36:16.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.234+0000 7f73315818c0 1 < completed write operation key=K3JNN1NCKBXN tag=8FE1PSWY8SPL type=write size=186163 2024-03-09T14:36:16.242 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.238+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=XZTSXJGGN3KR type=del size=353873 2024-03-09T14:36:16.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.242+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~YP5M6V0J7PMI.0 size=1048576 2024-03-09T14:36:16.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.246+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~YP5M6V0J7PMI.1 size=1006889 2024-03-09T14:36:16.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.254+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=YP5M6V0J7PMI tag=CS4JAH1YBN85 type=write size=3104041 2024-03-09T14:36:16.264 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.262+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~G2O056JFRRG6.0 size=215027 2024-03-09T14:36:16.264 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.262+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=G2O056JFRRG6 tag=ZS5NANWVJ1ZG type=write size=1263603 2024-03-09T14:36:16.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.266+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~ZB2UT237M2O7.0 size=1048576 2024-03-09T14:36:16.271 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.266+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~ZB2UT237M2O7.1 size=1048576 2024-03-09T14:36:16.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.270+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~ZB2UT237M2O7.2 size=78425 2024-03-09T14:36:16.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.278+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=ZB2UT237M2O7 tag=IOVO6688LB0M type=write size=3224153 2024-03-09T14:36:16.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.282+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=3AKDJBZB6LZ6 type=del size=2181918 2024-03-09T14:36:16.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.282+0000 7f73315818c0 1 < canceled part key=_multipart_4E3QMBHEAEHO.2~CFVLOYFJZJ5W.0 size=857732 2024-03-09T14:36:16.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.286+0000 7f73315818c0 1 < canceled multipart upload key=4E3QMBHEAEHO upload=CFVLOYFJZJ5W tag=J3S334GW11WS type=write size=1906308 2024-03-09T14:36:16.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.290+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=UC24MOAK3XT8 type=del size=1414285 2024-03-09T14:36:16.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.298+0000 7f73315818c0 1 < suggested update operation key=P8R5RJI8LGZ9 tag=TK42G0MNOZ0G type=del size=724499 2024-03-09T14:36:16.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.302+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=NCQQ65USETN0 type=write size=287023 2024-03-09T14:36:16.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.306+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=B7UPZVZ80M09 type=del 2024-03-09T14:36:16.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.310+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~ES7RKDX1Z4YR.0 size=1048576 2024-03-09T14:36:16.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.314+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~ES7RKDX1Z4YR.1 size=1048576 2024-03-09T14:36:16.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.318+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~ES7RKDX1Z4YR.2 size=719043 2024-03-09T14:36:16.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.318+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=ES7RKDX1Z4YR tag=QE06W87B8WMM type=write size=3864771 2024-03-09T14:36:16.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.322+0000 7f73315818c0 1 < suggested remove operation key=PAJYGD2GXHDT tag=DEKR85YWU7RY type=del 2024-03-09T14:36:16.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.326+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~04VR0U4BNCM2.0 size=979222 2024-03-09T14:36:16.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.326+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=04VR0U4BNCM2 tag=BKLQV10DS3PB type=write size=2027798 2024-03-09T14:36:16.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.330+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=N0T6ROSZOX69 type=del 2024-03-09T14:36:16.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.334+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=LDIJ4WMUQ16D type=del size=4045365 2024-03-09T14:36:16.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.334+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~65ZUUTEJERHZ.0 size=1048576 2024-03-09T14:36:16.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.338+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~65ZUUTEJERHZ.1 size=442956 2024-03-09T14:36:16.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.342+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=65ZUUTEJERHZ tag=2JJDK2KYSLYU type=write size=2540108 2024-03-09T14:36:16.348 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.346+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=XFFQFH8MVYQO type=write size=353198 2024-03-09T14:36:16.351 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.346+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=PKU1BHL7CI1C type=del 2024-03-09T14:36:16.359 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.354+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~1E7F3MB19F85.0 size=1048576 2024-03-09T14:36:16.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.358+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~1E7F3MB19F85.1 size=1048576 2024-03-09T14:36:16.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.362+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~1E7F3MB19F85.2 size=59718 2024-03-09T14:36:16.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.362+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=1E7F3MB19F85 tag=TMOLE9D0YKQX type=write size=3205446 2024-03-09T14:36:16.366 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.362+0000 7f73315818c0 1 < completed write operation key=9B0RAHZZGLYU tag=U3Q1VSWUAYP6 type=write size=890226 2024-03-09T14:36:16.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.370+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~ECHRQ8JZXKAI.0 size=752574 2024-03-09T14:36:16.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.370+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=ECHRQ8JZXKAI tag=WIZ1WULNXJGB type=write size=1801150 2024-03-09T14:36:16.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.370+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~VSF05A9RXBXB.0 size=1048576 2024-03-09T14:36:16.376 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.374+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~VSF05A9RXBXB.1 size=129445 2024-03-09T14:36:16.381 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.378+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=VSF05A9RXBXB tag=S20C1LA4EDWO type=write size=2226597 2024-03-09T14:36:16.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.382+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=N81TYT6F5LNM type=del size=2827216 2024-03-09T14:36:16.387 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.382+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=PSYB6LH8S0I3 type=del 2024-03-09T14:36:16.390 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.386+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=OFY1L89DVQVF type=del size=3629380 2024-03-09T14:36:16.393 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.390+0000 7f73315818c0 1 < completed write operation key=M1B6O6N8N84T tag=YTWTI7S14AOV type=write size=822521 2024-03-09T14:36:16.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.394+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=72JJC44AZ0IN type=del size=1184588 2024-03-09T14:36:16.398 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.394+0000 7f73315818c0 1 < completed write operation key=M8F9TA80VME6 tag=0S6990ZUEU7P type=write size=320446 2024-03-09T14:36:16.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.398+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=98XJSXIW9BX0 type=write size=591871 2024-03-09T14:36:16.404 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.402+0000 7f73315818c0 1 < canceled operation key=K3JNN1NCKBXN tag=902B5C06JMZX type=del size=2674202 2024-03-09T14:36:16.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.402+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=MTQ6B3MIGCJ9 type=del size=3740331 2024-03-09T14:36:16.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.406+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=XZTSXJGGN3KR type=del 2024-03-09T14:36:16.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.410+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=HNWHMECX3UZ5 type=write size=390747 2024-03-09T14:36:16.415 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.414+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~G2O056JFRRG6.0 size=215027 2024-03-09T14:36:16.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.418+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=G2O056JFRRG6 tag=ZS5NANWVJ1ZG type=write size=1263603 2024-03-09T14:36:16.423 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.422+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=0MJ382SW74UO type=del size=2074465 2024-03-09T14:36:16.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.422+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=3AKDJBZB6LZ6 type=del 2024-03-09T14:36:16.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.426+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=X6T6H810TNZL type=del size=1373595 2024-03-09T14:36:16.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.430+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=UC24MOAK3XT8 type=del 2024-03-09T14:36:16.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.434+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~LYX4MIWZZY79.0 size=1048576 2024-03-09T14:36:16.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.438+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~LYX4MIWZZY79.1 size=1048576 2024-03-09T14:36:16.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.438+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~LYX4MIWZZY79.2 size=380466 2024-03-09T14:36:16.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.438+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=LYX4MIWZZY79 tag=YB8X5Y4NCF59 type=write size=3526194 2024-03-09T14:36:16.449 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.446+0000 7f73315818c0 1 < suggested remove operation key=MSMD3ANK4T7J tag=NCQQ65USETN0 type=write 2024-03-09T14:36:16.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.450+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=W2L53WH7U1RR type=del size=2328418 2024-03-09T14:36:16.455 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.454+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~ES7RKDX1Z4YR.0 size=1048576 2024-03-09T14:36:16.458 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.454+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~ES7RKDX1Z4YR.1 size=1048576 2024-03-09T14:36:16.461 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.458+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~ES7RKDX1Z4YR.2 size=719043 2024-03-09T14:36:16.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.462+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=ES7RKDX1Z4YR tag=QE06W87B8WMM type=write size=3864771 2024-03-09T14:36:16.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.466+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=XWC627NTZ92B type=del size=4007458 2024-03-09T14:36:16.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.470+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~04VR0U4BNCM2.0 size=979222 2024-03-09T14:36:16.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.474+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=04VR0U4BNCM2 tag=BKLQV10DS3PB type=write size=2027798 2024-03-09T14:36:16.480 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.478+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=H67LF0U25DYZ type=del size=1500487 2024-03-09T14:36:16.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.482+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=LDIJ4WMUQ16D type=del 2024-03-09T14:36:16.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.486+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~KB3UEY2K4OE3.0 size=948014 2024-03-09T14:36:16.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.486+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=KB3UEY2K4OE3 tag=9O516N3XVOJB type=write size=1996590 2024-03-09T14:36:16.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.490+0000 7f73315818c0 1 < completed write operation key=0IB00CZ92VV4 tag=XFFQFH8MVYQO type=write size=353198 2024-03-09T14:36:16.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.494+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~3S058R2WU6TG.0 size=970408 2024-03-09T14:36:16.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.494+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=3S058R2WU6TG tag=HC9IYL4DJOUQ type=write size=2018984 2024-03-09T14:36:16.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.498+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~1E7F3MB19F85.0 size=1048576 2024-03-09T14:36:16.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.502+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~1E7F3MB19F85.1 size=1048576 2024-03-09T14:36:16.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.502+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~1E7F3MB19F85.2 size=59718 2024-03-09T14:36:16.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.510+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=1E7F3MB19F85 tag=TMOLE9D0YKQX type=write size=3205446 2024-03-09T14:36:16.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.510+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=M1TAW71GVVZC type=del size=2826915 2024-03-09T14:36:16.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.514+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~ECHRQ8JZXKAI.0 size=752574 2024-03-09T14:36:16.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.522+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=ECHRQ8JZXKAI tag=WIZ1WULNXJGB type=write size=1801150 2024-03-09T14:36:16.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.526+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~18N4M6NG8OPZ.0 size=1048576 2024-03-09T14:36:16.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.530+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~18N4M6NG8OPZ.1 size=1048576 2024-03-09T14:36:16.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.530+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~18N4M6NG8OPZ.2 size=673115 2024-03-09T14:36:16.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.530+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=18N4M6NG8OPZ tag=9ZI7N7U2CQKP type=write size=3818843 2024-03-09T14:36:16.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.534+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=N81TYT6F5LNM type=del 2024-03-09T14:36:16.539 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.538+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=RJ4A3YKFZ9WQ type=del size=1681087 2024-03-09T14:36:16.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.542+0000 7f73315818c0 1 < suggested update operation key=PXFZLDSAR1EG tag=OFY1L89DVQVF type=del size=3629380 2024-03-09T14:36:16.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.546+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=5RBSAX8G7NIB type=del size=3958235 2024-03-09T14:36:16.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.546+0000 7f73315818c0 1 < canceled operation key=UMSNM53HTY1J tag=72JJC44AZ0IN type=del size=1184588 2024-03-09T14:36:16.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.550+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=4D4WZOBO6H98 type=del size=3956447 2024-03-09T14:36:16.557 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.554+0000 7f73315818c0 1 < completed write operation key=P9LI5ESR3N9G tag=98XJSXIW9BX0 type=write size=591871 2024-03-09T14:36:16.560 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.558+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=EXMGGK7WLDUI type=write size=508641 2024-03-09T14:36:16.565 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.562+0000 7f73315818c0 1 < suggested update operation key=K3JNN1NCKBXN tag=MTQ6B3MIGCJ9 type=del size=3740331 2024-03-09T14:36:16.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.566+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~QWV4J3ZCKBVW.0 size=1048576 2024-03-09T14:36:16.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.570+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~QWV4J3ZCKBVW.1 size=221335 2024-03-09T14:36:16.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.570+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=QWV4J3ZCKBVW tag=2E9GQLS0D716 type=write size=2318487 2024-03-09T14:36:16.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.578+0000 7f73315818c0 1 < suggested remove operation key=P8R5RJI8LGZ9 tag=HNWHMECX3UZ5 type=write 2024-03-09T14:36:16.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.582+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=0RS9VQ0K6LV8 type=del size=802600 2024-03-09T14:36:16.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.582+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=0MJ382SW74UO type=del 2024-03-09T14:36:16.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.590+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~81QUCBOQMJRR.0 size=179248 2024-03-09T14:36:16.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.590+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=81QUCBOQMJRR tag=RUPKITF7GO4X type=write size=1227824 2024-03-09T14:36:16.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.590+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=X6T6H810TNZL type=del 2024-03-09T14:36:16.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.594+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=MTA56FW7W011 type=write size=715658 2024-03-09T14:36:16.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.598+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~LYX4MIWZZY79.0 size=1048576 2024-03-09T14:36:16.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.598+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~LYX4MIWZZY79.1 size=1048576 2024-03-09T14:36:16.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.602+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~LYX4MIWZZY79.2 size=380466 2024-03-09T14:36:16.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.606+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=LYX4MIWZZY79 tag=YB8X5Y4NCF59 type=write size=3526194 2024-03-09T14:36:16.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.610+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=88PAZYPW5DLW type=del size=1173797 2024-03-09T14:36:16.618 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.614+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=W2L53WH7U1RR type=del 2024-03-09T14:36:16.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.618+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~F1HVM7I1KBO1.0 size=801907 2024-03-09T14:36:16.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.618+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=F1HVM7I1KBO1 tag=PJN2CWXGJBR4 type=write size=1850483 2024-03-09T14:36:16.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.622+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=XWC627NTZ92B type=del 2024-03-09T14:36:16.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.626+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=6TYPOW7GU6CM type=del size=1567446 2024-03-09T14:36:16.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.630+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=H67LF0U25DYZ type=del 2024-03-09T14:36:16.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.634+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~WN5VOY8JEHRO.0 size=1048576 2024-03-09T14:36:16.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.642+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~WN5VOY8JEHRO.1 size=959710 2024-03-09T14:36:16.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.642+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=WN5VOY8JEHRO tag=51XVR17H5H39 type=write size=3056862 2024-03-09T14:36:16.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.646+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~KB3UEY2K4OE3.0 size=948014 2024-03-09T14:36:16.652 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.650+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=KB3UEY2K4OE3 tag=9O516N3XVOJB type=write size=1996590 2024-03-09T14:36:16.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.650+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=P0ZVO3DK0A7S type=del size=2078668 2024-03-09T14:36:16.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.654+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~3S058R2WU6TG.0 size=970408 2024-03-09T14:36:16.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.658+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=3S058R2WU6TG tag=HC9IYL4DJOUQ type=write size=2018984 2024-03-09T14:36:16.664 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.662+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=UY1DCNOJEL5C type=write size=213145 2024-03-09T14:36:16.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.666+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=M1TAW71GVVZC type=del 2024-03-09T14:36:16.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.666+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=2XGB787M0GP9 type=del size=3719054 2024-03-09T14:36:16.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.670+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~18N4M6NG8OPZ.0 size=1048576 2024-03-09T14:36:16.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.674+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~18N4M6NG8OPZ.1 size=1048576 2024-03-09T14:36:16.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.674+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~18N4M6NG8OPZ.2 size=673115 2024-03-09T14:36:16.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.678+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=18N4M6NG8OPZ tag=9ZI7N7U2CQKP type=write size=3818843 2024-03-09T14:36:16.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.682+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=6W7H3J4JPFO6 type=del size=2162233 2024-03-09T14:36:16.689 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.686+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=RJ4A3YKFZ9WQ type=del 2024-03-09T14:36:16.695 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.690+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~9STTD7KIKUOL.0 size=744310 2024-03-09T14:36:16.695 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.690+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=9STTD7KIKUOL tag=JUR52XIPE2HD type=write size=1792886 2024-03-09T14:36:16.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.698+0000 7f73315818c0 1 < suggested remove operation key=3ADG4GWVARC1 tag=5RBSAX8G7NIB type=del 2024-03-09T14:36:16.707 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.702+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~1V0QFD8H5KMD.0 size=1048576 2024-03-09T14:36:16.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.706+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~1V0QFD8H5KMD.1 size=172265 2024-03-09T14:36:16.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.706+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=1V0QFD8H5KMD tag=PO1QPOTKYUAG type=write size=2269417 2024-03-09T14:36:16.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.710+0000 7f73315818c0 1 < canceled operation key=0IB00CZ92VV4 tag=4D4WZOBO6H98 type=del size=3956447 2024-03-09T14:36:16.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.714+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~47HT4S6NILGN.0 size=1048576 2024-03-09T14:36:16.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.714+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~47HT4S6NILGN.1 size=766801 2024-03-09T14:36:16.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.714+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=47HT4S6NILGN tag=YBA440GILL29 type=write size=2863953 2024-03-09T14:36:16.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.718+0000 7f73315818c0 1 < completed write operation key=8O5OP8C5NX12 tag=EXMGGK7WLDUI type=write size=508641 2024-03-09T14:36:16.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.722+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~DGQTXUFWAUXR.0 size=1048576 2024-03-09T14:36:16.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.726+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~DGQTXUFWAUXR.1 size=1048576 2024-03-09T14:36:16.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.730+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~DGQTXUFWAUXR.2 size=390990 2024-03-09T14:36:16.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.730+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=DGQTXUFWAUXR tag=EWLMI2T04QV1 type=write size=3536718 2024-03-09T14:36:16.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.730+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~QWV4J3ZCKBVW.0 size=1048576 2024-03-09T14:36:16.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.734+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~QWV4J3ZCKBVW.1 size=221335 2024-03-09T14:36:16.742 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.738+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=QWV4J3ZCKBVW tag=2E9GQLS0D716 type=write size=2318487 2024-03-09T14:36:16.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.742+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=HCQIE4QS31O7 type=del size=2741984 2024-03-09T14:36:16.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.746+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=0RS9VQ0K6LV8 type=del 2024-03-09T14:36:16.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.746+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=F1UA6Q9DR076 type=del size=106873 2024-03-09T14:36:16.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.750+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~81QUCBOQMJRR.0 size=179248 2024-03-09T14:36:16.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.754+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=81QUCBOQMJRR tag=RUPKITF7GO4X type=write size=1227824 2024-03-09T14:36:16.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.758+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=J4C8KAOHV9MB type=del size=2925564 2024-03-09T14:36:16.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.766+0000 7f73315818c0 1 < suggested update operation key=FYFVW9R0M1PX tag=MTA56FW7W011 type=write size=715658 2024-03-09T14:36:16.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.770+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~71LIZP5HTBR0.0 size=1048576 2024-03-09T14:36:16.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.774+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~71LIZP5HTBR0.1 size=1048576 2024-03-09T14:36:16.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.774+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~71LIZP5HTBR0.2 size=795315 2024-03-09T14:36:16.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.774+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=71LIZP5HTBR0 tag=B51O5TR9ARVX type=write size=3941043 2024-03-09T14:36:16.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.778+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=88PAZYPW5DLW type=del 2024-03-09T14:36:16.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.786+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~6MLLLQ0XX7EI.0 size=558076 2024-03-09T14:36:16.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.786+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=6MLLLQ0XX7EI tag=ADR08P4W6XM0 type=write size=1606652 2024-03-09T14:36:16.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.786+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~F1HVM7I1KBO1.0 size=801907 2024-03-09T14:36:16.795 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.794+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=F1HVM7I1KBO1 tag=PJN2CWXGJBR4 type=write size=1850483 2024-03-09T14:36:16.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.794+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=8ZO585PTZ5ET type=del size=3716322 2024-03-09T14:36:16.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.798+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=6TYPOW7GU6CM type=del 2024-03-09T14:36:16.807 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.806+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~JYLI09CG30F4.0 size=1048576 2024-03-09T14:36:16.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.806+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~JYLI09CG30F4.1 size=982134 2024-03-09T14:36:16.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.806+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=JYLI09CG30F4 tag=VXTLDT4IY2Z3 type=write size=3079286 2024-03-09T14:36:16.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.810+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~WN5VOY8JEHRO.0 size=1048576 2024-03-09T14:36:16.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.814+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~WN5VOY8JEHRO.1 size=959710 2024-03-09T14:36:16.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.818+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=WN5VOY8JEHRO tag=51XVR17H5H39 type=write size=3056862 2024-03-09T14:36:16.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.822+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=3TK1SLV2Q6JK type=del size=1586907 2024-03-09T14:36:16.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.826+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=P0ZVO3DK0A7S type=del 2024-03-09T14:36:16.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.826+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=28EM72XOR57S type=del size=3481277 2024-03-09T14:36:16.835 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.830+0000 7f73315818c0 1 < completed write operation key=0IB00CZ92VV4 tag=UY1DCNOJEL5C type=write size=213145 2024-03-09T14:36:16.838 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.834+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=SZ4VCNZ4VDHC type=del size=4118374 2024-03-09T14:36:16.841 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.838+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=2XGB787M0GP9 type=del 2024-03-09T14:36:16.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.842+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~35T13ZGF4KB0.0 size=1048576 2024-03-09T14:36:16.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.846+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~35T13ZGF4KB0.1 size=1048576 2024-03-09T14:36:16.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.850+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~35T13ZGF4KB0.2 size=803081 2024-03-09T14:36:16.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.850+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=35T13ZGF4KB0 tag=AUZCWKUX0BVO type=write size=3948809 2024-03-09T14:36:16.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.854+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=6W7H3J4JPFO6 type=del 2024-03-09T14:36:16.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.858+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~W1PQQQSOMQTZ.0 size=1048576 2024-03-09T14:36:16.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.862+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~W1PQQQSOMQTZ.1 size=119039 2024-03-09T14:36:16.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.862+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=W1PQQQSOMQTZ tag=R4EGBFNACZN6 type=write size=2216191 2024-03-09T14:36:16.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.866+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~9STTD7KIKUOL.0 size=744310 2024-03-09T14:36:16.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.870+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=9STTD7KIKUOL tag=JUR52XIPE2HD type=write size=1792886 2024-03-09T14:36:16.879 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.878+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=TG2YKQ9XW9Y7 type=del size=3672813 2024-03-09T14:36:16.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.878+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~1V0QFD8H5KMD.0 size=1048576 2024-03-09T14:36:16.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.882+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~1V0QFD8H5KMD.1 size=172265 2024-03-09T14:36:16.890 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.886+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=1V0QFD8H5KMD tag=PO1QPOTKYUAG type=write size=2269417 2024-03-09T14:36:16.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.894+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~6QF1A81EXL8M.0 size=1048576 2024-03-09T14:36:16.898 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.894+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~6QF1A81EXL8M.1 size=237818 2024-03-09T14:36:16.898 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.894+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=6QF1A81EXL8M tag=HK32BI6CSQYW type=write size=2334970 2024-03-09T14:36:16.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.898+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~47HT4S6NILGN.0 size=1048576 2024-03-09T14:36:16.904 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.902+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~47HT4S6NILGN.1 size=766801 2024-03-09T14:36:16.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.906+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=47HT4S6NILGN tag=YBA440GILL29 type=write size=2863953 2024-03-09T14:36:16.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.910+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=5FI1YFMJKQVA type=del size=37108 2024-03-09T14:36:16.917 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.914+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~DGQTXUFWAUXR.0 size=1048576 2024-03-09T14:36:16.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.914+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~DGQTXUFWAUXR.1 size=1048576 2024-03-09T14:36:16.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.918+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~DGQTXUFWAUXR.2 size=390990 2024-03-09T14:36:16.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.926+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=DGQTXUFWAUXR tag=EWLMI2T04QV1 type=write size=3536718 2024-03-09T14:36:16.934 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.930+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=3ZQSTQ072W15 type=del size=3076455 2024-03-09T14:36:16.937 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.934+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=HCQIE4QS31O7 type=del 2024-03-09T14:36:16.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.938+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~COR4PF78124N.0 size=1048576 2024-03-09T14:36:16.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.942+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~COR4PF78124N.1 size=968539 2024-03-09T14:36:16.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.942+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=COR4PF78124N tag=FQNJD17G8EM7 type=write size=3065691 2024-03-09T14:36:16.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.946+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=F1UA6Q9DR076 type=del 2024-03-09T14:36:16.954 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.950+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~N4YV5I82WQWX.0 size=1048576 2024-03-09T14:36:16.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.954+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~N4YV5I82WQWX.1 size=1048576 2024-03-09T14:36:16.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.954+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~N4YV5I82WQWX.2 size=1003422 2024-03-09T14:36:16.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.954+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=N4YV5I82WQWX tag=GIQ27ELGROUH type=write size=4149150 2024-03-09T14:36:16.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.958+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=J4C8KAOHV9MB type=del 2024-03-09T14:36:16.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.962+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=WHFO7W4SNICS type=del size=2487879 2024-03-09T14:36:16.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.966+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~71LIZP5HTBR0.0 size=1048576 2024-03-09T14:36:16.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.970+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~71LIZP5HTBR0.1 size=1048576 2024-03-09T14:36:16.974 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.970+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~71LIZP5HTBR0.2 size=795315 2024-03-09T14:36:16.980 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.978+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=71LIZP5HTBR0 tag=B51O5TR9ARVX type=write size=3941043 2024-03-09T14:36:16.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.982+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=X82I59DRO37K type=del size=2211373 2024-03-09T14:36:16.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.982+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~6MLLLQ0XX7EI.0 size=558076 2024-03-09T14:36:16.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.990+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=6MLLLQ0XX7EI tag=ADR08P4W6XM0 type=write size=1606652 2024-03-09T14:36:16.996 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.994+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=1YG91FEXP8L9 type=write size=256152 2024-03-09T14:36:16.998 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.994+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=8ZO585PTZ5ET type=del 2024-03-09T14:36:17.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:16.998+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=82TC95G48R74 type=del size=653772 2024-03-09T14:36:17.005 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.002+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~JYLI09CG30F4.0 size=1048576 2024-03-09T14:36:17.009 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.006+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~JYLI09CG30F4.1 size=982134 2024-03-09T14:36:17.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.010+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=JYLI09CG30F4 tag=VXTLDT4IY2Z3 type=write size=3079286 2024-03-09T14:36:17.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.018+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~72E6TX0SV902.0 size=1048576 2024-03-09T14:36:17.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.022+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~72E6TX0SV902.1 size=444590 2024-03-09T14:36:17.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.022+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=72E6TX0SV902 tag=KSRVCS43QVL9 type=write size=2541742 2024-03-09T14:36:17.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.030+0000 7f73315818c0 1 < suggested update operation key=PAJYGD2GXHDT tag=3TK1SLV2Q6JK type=del size=1586907 2024-03-09T14:36:17.035 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.030+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=6NJD9QNYLIGT type=del size=519895 2024-03-09T14:36:17.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.038+0000 7f73315818c0 1 < suggested remove operation key=0LP8KKU53X0A tag=28EM72XOR57S type=del 2024-03-09T14:36:17.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.042+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=L5UDT5IAR52U type=del size=3067366 2024-03-09T14:36:17.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.050+0000 7f73315818c0 1 < suggested remove operation key=PXFZLDSAR1EG tag=SZ4VCNZ4VDHC type=del 2024-03-09T14:36:17.056 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.054+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=B6YAUZEDP3DS type=del size=486399 2024-03-09T14:36:17.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.058+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~35T13ZGF4KB0.0 size=1048576 2024-03-09T14:36:17.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.058+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~35T13ZGF4KB0.1 size=1048576 2024-03-09T14:36:17.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.058+0000 7f73315818c0 1 < canceled part key=_multipart_UMSNM53HTY1J.2~35T13ZGF4KB0.2 size=803081 2024-03-09T14:36:17.068 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.066+0000 7f73315818c0 1 < canceled multipart upload key=UMSNM53HTY1J upload=35T13ZGF4KB0 tag=AUZCWKUX0BVO type=write size=3948809 2024-03-09T14:36:17.074 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.070+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~UY7XFI1MP4LT.0 size=1048576 2024-03-09T14:36:17.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.074+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~UY7XFI1MP4LT.1 size=1048576 2024-03-09T14:36:17.079 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.074+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~UY7XFI1MP4LT.2 size=391711 2024-03-09T14:36:17.079 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.074+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=UY7XFI1MP4LT tag=RD81Q6VAJCYX type=write size=3537439 2024-03-09T14:36:17.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.078+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~W1PQQQSOMQTZ.0 size=1048576 2024-03-09T14:36:17.085 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.082+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~W1PQQQSOMQTZ.1 size=119039 2024-03-09T14:36:17.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.086+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=W1PQQQSOMQTZ tag=R4EGBFNACZN6 type=write size=2216191 2024-03-09T14:36:17.142 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.094+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~T278EGCW2CD5.0 size=556138 2024-03-09T14:36:17.142 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.094+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=T278EGCW2CD5 tag=4CM80FLJ8HRF type=write size=1604714 2024-03-09T14:36:17.142 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.094+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=TG2YKQ9XW9Y7 type=del 2024-03-09T14:36:17.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.142+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~QQJZTG7XTXO3.0 size=1048576 2024-03-09T14:36:17.149 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.146+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~QQJZTG7XTXO3.1 size=1048576 2024-03-09T14:36:17.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.150+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~QQJZTG7XTXO3.2 size=539119 2024-03-09T14:36:17.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.150+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=QQJZTG7XTXO3 tag=GFTJW245D7M4 type=write size=3684847 2024-03-09T14:36:17.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.150+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~6QF1A81EXL8M.0 size=1048576 2024-03-09T14:36:17.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.154+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~6QF1A81EXL8M.1 size=237818 2024-03-09T14:36:17.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.162+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=6QF1A81EXL8M tag=HK32BI6CSQYW type=write size=2334970 2024-03-09T14:36:17.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.166+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~PBUOMX7I6ZX0.0 size=1048576 2024-03-09T14:36:17.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.170+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~PBUOMX7I6ZX0.1 size=1003553 2024-03-09T14:36:17.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.170+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=PBUOMX7I6ZX0 tag=D6BQV8U6WRZC type=write size=3100705 2024-03-09T14:36:17.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.174+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=5FI1YFMJKQVA type=del 2024-03-09T14:36:17.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.178+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=LW9P7RV1NN73 type=del size=652565 2024-03-09T14:36:17.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.182+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=3ZQSTQ072W15 type=del 2024-03-09T14:36:17.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.186+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~4O821T1G0GMX.0 size=628055 2024-03-09T14:36:17.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.186+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=4O821T1G0GMX tag=EXQV393D5WD3 type=write size=1676631 2024-03-09T14:36:17.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.190+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~COR4PF78124N.0 size=1048576 2024-03-09T14:36:17.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.194+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~COR4PF78124N.1 size=968539 2024-03-09T14:36:17.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.202+0000 7f73315818c0 1 < completed multipart upload key=IA0AB7UVLFEJ upload=COR4PF78124N tag=FQNJD17G8EM7 type=write size=3065691 2024-03-09T14:36:17.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.206+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=4DF31FMUQB7O type=del size=3277484 2024-03-09T14:36:17.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.210+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~N4YV5I82WQWX.0 size=1048576 2024-03-09T14:36:17.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.210+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~N4YV5I82WQWX.1 size=1048576 2024-03-09T14:36:17.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.210+0000 7f73315818c0 1 < canceled part key=_multipart_0LP8KKU53X0A.2~N4YV5I82WQWX.2 size=1003422 2024-03-09T14:36:17.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.218+0000 7f73315818c0 1 < canceled multipart upload key=0LP8KKU53X0A upload=N4YV5I82WQWX tag=GIQ27ELGROUH type=write size=4149150 2024-03-09T14:36:17.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.226+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~8LET1UQ1DIJJ.0 size=825079 2024-03-09T14:36:17.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.226+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=8LET1UQ1DIJJ tag=3CJDA229SW30 type=write size=1873655 2024-03-09T14:36:17.235 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.230+0000 7f73315818c0 1 < suggested remove operation key=8O5OP8C5NX12 tag=WHFO7W4SNICS type=del 2024-03-09T14:36:17.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.238+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~2PBDSMVPNUYX.0 size=1048576 2024-03-09T14:36:17.244 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.242+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~2PBDSMVPNUYX.1 size=1048576 2024-03-09T14:36:17.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.242+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~2PBDSMVPNUYX.2 size=391617 2024-03-09T14:36:17.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.242+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=2PBDSMVPNUYX tag=RGUUGP4HSXKL type=write size=3537345 2024-03-09T14:36:17.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.246+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=X82I59DRO37K type=del 2024-03-09T14:36:17.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.250+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=YXSXS2IU37Y4 type=del size=2678709 2024-03-09T14:36:17.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.254+0000 7f73315818c0 1 < completed write operation key=0IB00CZ92VV4 tag=1YG91FEXP8L9 type=write size=256152 2024-03-09T14:36:17.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.258+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=1G8GL2NZ7JPX type=del size=1136912 2024-03-09T14:36:17.264 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.262+0000 7f73315818c0 -1 < failed to complete operation key=8O5OP8C5NX12 tag=82TC95G48R74 type=del size=653772: (22) Invalid argument 2024-03-09T14:36:17.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.266+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=KIH0JBMREQ8Z type=del size=1995584 2024-03-09T14:36:17.271 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.270+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~72E6TX0SV902.0 size=1048576 2024-03-09T14:36:17.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.270+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~72E6TX0SV902.1 size=444590 2024-03-09T14:36:17.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.278+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=72E6TX0SV902 tag=KSRVCS43QVL9 type=write size=2541742 2024-03-09T14:36:17.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.282+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=QTUZY8YQ692D type=del size=2126362 2024-03-09T14:36:17.291 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.286+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=6NJD9QNYLIGT type=del 2024-03-09T14:36:17.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.290+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=9K9U0I621F98 type=write size=331265 2024-03-09T14:36:17.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.294+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=L5UDT5IAR52U type=del 2024-03-09T14:36:17.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.302+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~2JEFIAVFTNUB.0 size=1048576 2024-03-09T14:36:17.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.302+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~2JEFIAVFTNUB.1 size=1048576 2024-03-09T14:36:17.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.306+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~2JEFIAVFTNUB.2 size=657310 2024-03-09T14:36:17.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.306+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=2JEFIAVFTNUB tag=QTIFOBH4SK4W type=write size=3803038 2024-03-09T14:36:17.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.310+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=B6YAUZEDP3DS type=del 2024-03-09T14:36:17.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.314+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=VHYDXUVZKRW8 type=del size=2853100 2024-03-09T14:36:17.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.318+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~UY7XFI1MP4LT.0 size=1048576 2024-03-09T14:36:17.324 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.322+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~UY7XFI1MP4LT.1 size=1048576 2024-03-09T14:36:17.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.322+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~UY7XFI1MP4LT.2 size=391711 2024-03-09T14:36:17.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.330+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=UY7XFI1MP4LT tag=RD81Q6VAJCYX type=write size=3537439 2024-03-09T14:36:17.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.334+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~KZQ7CF619N1D.0 size=1048576 2024-03-09T14:36:17.342 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.338+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~KZQ7CF619N1D.1 size=1048576 2024-03-09T14:36:17.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.342+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~KZQ7CF619N1D.2 size=515415 2024-03-09T14:36:17.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.342+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=KZQ7CF619N1D tag=2J5IDCZGX967 type=write size=3661143 2024-03-09T14:36:17.348 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.346+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~T278EGCW2CD5.0 size=556138 2024-03-09T14:36:17.354 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.350+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=T278EGCW2CD5 tag=4CM80FLJ8HRF type=write size=1604714 2024-03-09T14:36:17.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.354+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=ZAROF85LI3HQ type=del size=1457855 2024-03-09T14:36:17.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.358+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~QQJZTG7XTXO3.0 size=1048576 2024-03-09T14:36:17.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.362+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~QQJZTG7XTXO3.1 size=1048576 2024-03-09T14:36:17.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.366+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~QQJZTG7XTXO3.2 size=539119 2024-03-09T14:36:17.373 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.370+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=QQJZTG7XTXO3 tag=GFTJW245D7M4 type=write size=3684847 2024-03-09T14:36:17.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.378+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~U51LSD3SEBJO.0 size=793514 2024-03-09T14:36:17.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.378+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=U51LSD3SEBJO tag=W4ZV796BBCJU type=write size=1842090 2024-03-09T14:36:17.381 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.378+0000 7f73315818c0 1 < canceled part key=_multipart_IA0AB7UVLFEJ.2~PBUOMX7I6ZX0.0 size=1048576 2024-03-09T14:36:17.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.382+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~PBUOMX7I6ZX0.1 size=1003553 2024-03-09T14:36:17.389 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.386+0000 7f73315818c0 1 < canceled multipart upload key=IA0AB7UVLFEJ upload=PBUOMX7I6ZX0 tag=D6BQV8U6WRZC type=write size=3100705 2024-03-09T14:36:17.396 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.394+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~2XAN21L0GHIH.0 size=1048576 2024-03-09T14:36:17.398 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.394+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~2XAN21L0GHIH.1 size=1048576 2024-03-09T14:36:17.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.398+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~2XAN21L0GHIH.2 size=124702 2024-03-09T14:36:17.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.398+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=2XAN21L0GHIH tag=7DU3K4ED0QIE type=write size=3270430 2024-03-09T14:36:17.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.402+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=LW9P7RV1NN73 type=del 2024-03-09T14:36:17.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.406+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=RI3RKU24SONC type=del size=465656 2024-03-09T14:36:17.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.410+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~4O821T1G0GMX.0 size=628055 2024-03-09T14:36:17.418 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.414+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=4O821T1G0GMX tag=EXQV393D5WD3 type=write size=1676631 2024-03-09T14:36:17.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.418+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=QW84R8N2PUPB type=write size=48047 2024-03-09T14:36:17.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.426+0000 7f73315818c0 1 < suggested update operation key=FYFVW9R0M1PX tag=4DF31FMUQB7O type=del size=3277484 2024-03-09T14:36:17.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.430+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=TI7C117PHWNP type=del size=823045 2024-03-09T14:36:17.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.430+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~8LET1UQ1DIJJ.0 size=825079 2024-03-09T14:36:17.440 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.438+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=8LET1UQ1DIJJ tag=3CJDA229SW30 type=write size=1873655 2024-03-09T14:36:17.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.442+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=FHB7JO0J0V4R type=del size=3960129 2024-03-09T14:36:17.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.446+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~2PBDSMVPNUYX.0 size=1048576 2024-03-09T14:36:17.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.446+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~2PBDSMVPNUYX.1 size=1048576 2024-03-09T14:36:17.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.450+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~2PBDSMVPNUYX.2 size=391617 2024-03-09T14:36:17.459 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.454+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=2PBDSMVPNUYX tag=RGUUGP4HSXKL type=write size=3537345 2024-03-09T14:36:17.462 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.458+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=F12W5K37KH9H type=del size=1233311 2024-03-09T14:36:17.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.462+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=YXSXS2IU37Y4 type=del 2024-03-09T14:36:17.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.470+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~YG9IZ385OOT3.0 size=1048576 2024-03-09T14:36:17.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.474+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~YG9IZ385OOT3.1 size=443642 2024-03-09T14:36:17.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.474+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=YG9IZ385OOT3 tag=2QJC0JUJY3H7 type=write size=2540794 2024-03-09T14:36:17.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.478+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=1G8GL2NZ7JPX type=del 2024-03-09T14:36:17.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.482+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=8R9BP2JEMRL5 type=del size=3037145 2024-03-09T14:36:17.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.486+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=KIH0JBMREQ8Z type=del 2024-03-09T14:36:17.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.490+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=019FITBLQ3FN type=del size=1808068 2024-03-09T14:36:17.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.490+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=QTUZY8YQ692D type=del 2024-03-09T14:36:17.499 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.494+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=HO4JR11SNPUY type=del size=1364847 2024-03-09T14:36:17.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.498+0000 7f73315818c0 1 < completed write operation key=8O5OP8C5NX12 tag=9K9U0I621F98 type=write size=331265 2024-03-09T14:36:17.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.502+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=6OH4JXIQW1HW type=del size=3303105 2024-03-09T14:36:17.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.502+0000 7f73315818c0 1 < canceled part key=_multipart_K3JNN1NCKBXN.2~2JEFIAVFTNUB.0 size=1048576 2024-03-09T14:36:17.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.506+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~2JEFIAVFTNUB.1 size=1048576 2024-03-09T14:36:17.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.510+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~2JEFIAVFTNUB.2 size=657310 2024-03-09T14:36:17.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.514+0000 7f73315818c0 1 < canceled multipart upload key=K3JNN1NCKBXN upload=2JEFIAVFTNUB tag=QTIFOBH4SK4W type=write size=3803038 2024-03-09T14:36:17.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.518+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=EESI52WT6MBV type=del size=2664756 2024-03-09T14:36:17.526 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.522+0000 7f73315818c0 1 < canceled operation key=0LP8KKU53X0A tag=VHYDXUVZKRW8 type=del size=2853100 2024-03-09T14:36:17.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.526+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=JH6FJEW2WO1N type=del size=4092868 2024-03-09T14:36:17.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.530+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~KZQ7CF619N1D.0 size=1048576 2024-03-09T14:36:17.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.534+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~KZQ7CF619N1D.1 size=1048576 2024-03-09T14:36:17.539 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.534+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~KZQ7CF619N1D.2 size=515415 2024-03-09T14:36:17.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.542+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=KZQ7CF619N1D tag=2J5IDCZGX967 type=write size=3661143 2024-03-09T14:36:17.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.546+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=RA0Y2QUDZUPL type=del size=2213011 2024-03-09T14:36:17.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.550+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=ZAROF85LI3HQ type=del 2024-03-09T14:36:17.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.550+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=9YB86ODGVDD9 type=del size=3868489 2024-03-09T14:36:17.558 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.554+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~U51LSD3SEBJO.0 size=793514 2024-03-09T14:36:17.564 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.562+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=U51LSD3SEBJO tag=W4ZV796BBCJU type=write size=1842090 2024-03-09T14:36:17.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.566+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=EPH3P4Z6SISE type=del size=993932 2024-03-09T14:36:17.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.566+0000 7f73315818c0 1 < canceled part key=_multipart_0WGGMHTEF2LB.2~2XAN21L0GHIH.0 size=1048576 2024-03-09T14:36:17.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.566+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~2XAN21L0GHIH.1 size=1048576 2024-03-09T14:36:17.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.570+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~2XAN21L0GHIH.2 size=124702 2024-03-09T14:36:17.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.574+0000 7f73315818c0 1 < canceled multipart upload key=0WGGMHTEF2LB upload=2XAN21L0GHIH tag=7DU3K4ED0QIE type=write size=3270430 2024-03-09T14:36:17.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.578+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=WHSZBQRV7DE4 type=del size=4100509 2024-03-09T14:36:17.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.582+0000 7f73315818c0 1 < canceled operation key=QG4O2L526B8E tag=RI3RKU24SONC type=del size=465656 2024-03-09T14:36:17.590 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.586+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=4SRCN4F2RJF2 type=del size=1511330 2024-03-09T14:36:17.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.594+0000 7f73315818c0 1 < suggested remove operation key=4P89277EEYJ3 tag=QW84R8N2PUPB type=write 2024-03-09T14:36:17.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.598+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=153WW086BVKE type=write size=338036 2024-03-09T14:36:17.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.602+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=TI7C117PHWNP type=del 2024-03-09T14:36:17.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.606+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~J7902CKJBKHO.0 size=410234 2024-03-09T14:36:17.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.606+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=J7902CKJBKHO tag=C22ICZ799F2C type=write size=1458810 2024-03-09T14:36:17.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.618+0000 7f73315818c0 1 < suggested update operation key=9B0RAHZZGLYU tag=FHB7JO0J0V4R type=del size=3960129 2024-03-09T14:36:17.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.626+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~QPWJDPVAN221.0 size=1048576 2024-03-09T14:36:17.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.630+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~QPWJDPVAN221.1 size=389794 2024-03-09T14:36:17.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.630+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=QPWJDPVAN221 tag=RLZ9GPSDORPT type=write size=2486946 2024-03-09T14:36:17.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.630+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=F12W5K37KH9H type=del 2024-03-09T14:36:17.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.634+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=STMK6FRK7BH7 type=write size=626193 2024-03-09T14:36:17.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.638+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~YG9IZ385OOT3.0 size=1048576 2024-03-09T14:36:17.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.642+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~YG9IZ385OOT3.1 size=443642 2024-03-09T14:36:17.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.646+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=YG9IZ385OOT3 tag=2QJC0JUJY3H7 type=write size=2540794 2024-03-09T14:36:17.654 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.650+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=JXZEPRVZIHVI type=write size=18441 2024-03-09T14:36:17.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.654+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=8R9BP2JEMRL5 type=del 2024-03-09T14:36:17.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.658+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=8Z5XTC70RRM7 type=del size=719640 2024-03-09T14:36:17.665 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.662+0000 7f73315818c0 1 < canceled operation key=Y8EQAHMTL66U tag=019FITBLQ3FN type=del size=1808068 2024-03-09T14:36:17.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.666+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=58FKXTZGNMR4 type=del size=3894911 2024-03-09T14:36:17.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.670+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=HO4JR11SNPUY type=del 2024-03-09T14:36:17.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.674+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=J4VYDWHIZ3Z4 type=write size=1011850 2024-03-09T14:36:17.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.674+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=6OH4JXIQW1HW type=del 2024-03-09T14:36:17.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.682+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~AWEPOIXSU5UB.0 size=1048576 2024-03-09T14:36:17.688 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.686+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~AWEPOIXSU5UB.1 size=775832 2024-03-09T14:36:17.688 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.686+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=AWEPOIXSU5UB tag=0HTQGMO50N3Y type=write size=2872984 2024-03-09T14:36:17.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.690+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=EESI52WT6MBV type=del 2024-03-09T14:36:17.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.694+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~1ZWR5NYFGHK7.0 size=512790 2024-03-09T14:36:17.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.694+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=1ZWR5NYFGHK7 tag=E6OSMAZ3AOZ3 type=write size=1561366 2024-03-09T14:36:17.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.698+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=JH6FJEW2WO1N type=del 2024-03-09T14:36:17.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.706+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~UVFWOWFPARPJ.0 size=1048576 2024-03-09T14:36:17.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.706+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~UVFWOWFPARPJ.1 size=401243 2024-03-09T14:36:17.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.706+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=UVFWOWFPARPJ tag=IJJLP2EMMA4C type=write size=2498395 2024-03-09T14:36:17.714 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.710+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=RA0Y2QUDZUPL type=del 2024-03-09T14:36:17.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.714+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=MDAPFDIA5MTY type=write size=1017096 2024-03-09T14:36:17.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.718+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=9YB86ODGVDD9 type=del 2024-03-09T14:36:17.725 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.722+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=PBQ5XLYBO2F7 type=del size=4178877 2024-03-09T14:36:17.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.726+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=EPH3P4Z6SISE type=del 2024-03-09T14:36:17.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.730+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~EMPWCRVO2SUG.0 size=1048576 2024-03-09T14:36:17.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.734+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~EMPWCRVO2SUG.1 size=1048576 2024-03-09T14:36:17.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.738+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~EMPWCRVO2SUG.2 size=175364 2024-03-09T14:36:17.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.738+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=EMPWCRVO2SUG tag=KUAXSL50XY76 type=write size=3321092 2024-03-09T14:36:17.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.742+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=WHSZBQRV7DE4 type=del 2024-03-09T14:36:17.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.746+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=OB25BY0J087L type=del size=714216 2024-03-09T14:36:17.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.746+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=4SRCN4F2RJF2 type=del 2024-03-09T14:36:17.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.750+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=O3Z27KXQPYI7 type=write size=773622 2024-03-09T14:36:17.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.754+0000 7f73315818c0 1 < completed write operation key=1NYNL8M5DVER tag=153WW086BVKE type=write size=338036 2024-03-09T14:36:17.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.762+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~YC42G1LZ9QAL.0 size=1048576 2024-03-09T14:36:17.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.766+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~YC42G1LZ9QAL.1 size=1048576 2024-03-09T14:36:17.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.766+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~YC42G1LZ9QAL.2 size=660693 2024-03-09T14:36:17.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.766+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=YC42G1LZ9QAL tag=TYD0JXNKD6KK type=write size=3806421 2024-03-09T14:36:17.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.770+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~J7902CKJBKHO.0 size=410234 2024-03-09T14:36:17.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.778+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=J7902CKJBKHO tag=C22ICZ799F2C type=write size=1458810 2024-03-09T14:36:17.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.782+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=UPE2IDI80QBN type=del size=426982 2024-03-09T14:36:17.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.786+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~QPWJDPVAN221.0 size=1048576 2024-03-09T14:36:17.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.786+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~QPWJDPVAN221.1 size=389794 2024-03-09T14:36:17.796 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.794+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=QPWJDPVAN221 tag=RLZ9GPSDORPT type=write size=2486946 2024-03-09T14:36:17.800 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.798+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=ZRUZWRPJ90P3 type=del size=2876605 2024-03-09T14:36:17.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.802+0000 7f73315818c0 1 < completed write operation key=UMSNM53HTY1J tag=STMK6FRK7BH7 type=write size=626193 2024-03-09T14:36:17.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.806+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~D8J7WE8QNP07.0 size=1048576 2024-03-09T14:36:17.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.810+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~D8J7WE8QNP07.1 size=389828 2024-03-09T14:36:17.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.810+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=D8J7WE8QNP07 tag=8DYKXIK0CQSX type=write size=2486980 2024-03-09T14:36:17.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.818+0000 7f73315818c0 1 < suggested update operation key=MSMD3ANK4T7J tag=JXZEPRVZIHVI type=write size=18441 2024-03-09T14:36:17.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.822+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=PW3ZNYF5CZCL type=write size=126557 2024-03-09T14:36:17.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.826+0000 7f73315818c0 1 < suggested update operation key=S302JFYEDEL7 tag=8Z5XTC70RRM7 type=del size=719640 2024-03-09T14:36:17.834 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.830+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=PGIXAC3HQTEZ type=del size=1654261 2024-03-09T14:36:17.838 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.834+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=58FKXTZGNMR4 type=del 2024-03-09T14:36:17.841 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.838+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=7WYY41M7YZ7Z type=del size=2344935 2024-03-09T14:36:17.845 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.842+0000 7f73315818c0 1 < canceled operation key=92GWY5B0K4SX tag=J4VYDWHIZ3Z4 type=write size=1011850 2024-03-09T14:36:17.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.846+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~8NL6XTMPXAVX.0 size=1048576 2024-03-09T14:36:17.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.850+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~8NL6XTMPXAVX.1 size=655621 2024-03-09T14:36:17.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.850+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=8NL6XTMPXAVX tag=CZKSN6MIWITL type=write size=2752773 2024-03-09T14:36:17.854 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.850+0000 7f73315818c0 1 < canceled part key=_multipart_4P89277EEYJ3.2~AWEPOIXSU5UB.0 size=1048576 2024-03-09T14:36:17.857 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.854+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~AWEPOIXSU5UB.1 size=775832 2024-03-09T14:36:17.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.858+0000 7f73315818c0 1 < canceled multipart upload key=4P89277EEYJ3 upload=AWEPOIXSU5UB tag=0HTQGMO50N3Y type=write size=2872984 2024-03-09T14:36:17.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.866+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~V9SBCYW02C3T.0 size=23726 2024-03-09T14:36:17.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.866+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=V9SBCYW02C3T tag=V6C7P6OYF5SW type=write size=1072302 2024-03-09T14:36:17.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.870+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~1ZWR5NYFGHK7.0 size=512790 2024-03-09T14:36:17.879 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.878+0000 7f73315818c0 1 < completed multipart upload key=0IB00CZ92VV4 upload=1ZWR5NYFGHK7 tag=E6OSMAZ3AOZ3 type=write size=1561366 2024-03-09T14:36:17.886 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.882+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~XNQVU6WLP2PJ.0 size=1048576 2024-03-09T14:36:17.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.886+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~XNQVU6WLP2PJ.1 size=1048576 2024-03-09T14:36:17.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.890+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~XNQVU6WLP2PJ.2 size=40707 2024-03-09T14:36:17.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.890+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=XNQVU6WLP2PJ tag=AWJ0HQAQW7NV type=write size=3186435 2024-03-09T14:36:17.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.894+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~UVFWOWFPARPJ.0 size=1048576 2024-03-09T14:36:17.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.894+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~UVFWOWFPARPJ.1 size=401243 2024-03-09T14:36:17.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.902+0000 7f73315818c0 1 < completed multipart upload key=0IB00CZ92VV4 upload=UVFWOWFPARPJ tag=IJJLP2EMMA4C type=write size=2498395 2024-03-09T14:36:17.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.910+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~Y4H3LYZ4NXYM.0 size=1048576 2024-03-09T14:36:17.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.910+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~Y4H3LYZ4NXYM.1 size=337300 2024-03-09T14:36:17.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.910+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=Y4H3LYZ4NXYM tag=ASGPM6DFJE1D type=write size=2434452 2024-03-09T14:36:17.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.914+0000 7f73315818c0 -1 < failed to complete operation key=S302JFYEDEL7 tag=MDAPFDIA5MTY type=write size=1017096: (22) Invalid argument 2024-03-09T14:36:17.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.918+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=DYOKRFB2JR3X type=del size=4052076 2024-03-09T14:36:17.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.922+0000 7f73315818c0 1 < completed delete operation key=DQOY1JJ1YQ2W tag=PBQ5XLYBO2F7 type=del 2024-03-09T14:36:17.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.930+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~E9GHSSC9GJL8.0 size=1048576 2024-03-09T14:36:17.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.934+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~E9GHSSC9GJL8.1 size=1048576 2024-03-09T14:36:17.939 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.934+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~E9GHSSC9GJL8.2 size=652974 2024-03-09T14:36:17.939 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.934+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=E9GHSSC9GJL8 tag=6CATUD4A0CVP type=write size=3798702 2024-03-09T14:36:17.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.938+0000 7f73315818c0 1 < canceled part key=_multipart_8O5OP8C5NX12.2~EMPWCRVO2SUG.0 size=1048576 2024-03-09T14:36:17.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.938+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~EMPWCRVO2SUG.1 size=1048576 2024-03-09T14:36:17.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.942+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~EMPWCRVO2SUG.2 size=175364 2024-03-09T14:36:17.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.950+0000 7f73315818c0 1 < canceled multipart upload key=8O5OP8C5NX12 upload=EMPWCRVO2SUG tag=KUAXSL50XY76 type=write size=3321092 2024-03-09T14:36:17.958 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.954+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=7Q2GTSQMUOSS type=del size=696423 2024-03-09T14:36:17.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.962+0000 7f73315818c0 1 < suggested update operation key=K3JNN1NCKBXN tag=OB25BY0J087L type=del size=714216 2024-03-09T14:36:17.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.966+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=4FQTOHWSHYQV type=del size=2254126 2024-03-09T14:36:17.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.970+0000 7f73315818c0 1 < completed write operation key=PXFZLDSAR1EG tag=O3Z27KXQPYI7 type=write size=773622 2024-03-09T14:36:17.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.974+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=2MRQLHDJNYJ7 type=del size=803919 2024-03-09T14:36:17.977 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.974+0000 7f73315818c0 1 < canceled part key=_multipart_QG4O2L526B8E.2~YC42G1LZ9QAL.0 size=1048576 2024-03-09T14:36:17.977 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.974+0000 7f73315818c0 1 < canceled part key=_multipart_QG4O2L526B8E.2~YC42G1LZ9QAL.1 size=1048576 2024-03-09T14:36:17.981 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.978+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~YC42G1LZ9QAL.2 size=660693 2024-03-09T14:36:17.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.986+0000 7f73315818c0 1 < canceled multipart upload key=QG4O2L526B8E upload=YC42G1LZ9QAL tag=TYD0JXNKD6KK type=write size=3806421 2024-03-09T14:36:17.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.986+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=VNSS8JLIB2F3 type=del size=2855366 2024-03-09T14:36:17.995 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.990+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=UPE2IDI80QBN type=del 2024-03-09T14:36:18.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:17.998+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~5BXNI42RANW2.0 size=1048576 2024-03-09T14:36:18.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.002+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~5BXNI42RANW2.1 size=1048576 2024-03-09T14:36:18.007 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.002+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~5BXNI42RANW2.2 size=307597 2024-03-09T14:36:18.007 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.002+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=5BXNI42RANW2 tag=SBP63SM8P8IO type=write size=3453325 2024-03-09T14:36:18.010 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.006+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=ZRUZWRPJ90P3 type=del 2024-03-09T14:36:18.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.010+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=V69AFLANN16T type=del size=858170 2024-03-09T14:36:18.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.010+0000 7f73315818c0 1 < canceled part key=_multipart_8O5OP8C5NX12.2~D8J7WE8QNP07.0 size=1048576 2024-03-09T14:36:18.016 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.014+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~D8J7WE8QNP07.1 size=389828 2024-03-09T14:36:18.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.018+0000 7f73315818c0 1 < canceled multipart upload key=8O5OP8C5NX12 upload=D8J7WE8QNP07 tag=8DYKXIK0CQSX type=write size=2486980 2024-03-09T14:36:18.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.026+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~8U6LRJ0FXJFO.0 size=1041161 2024-03-09T14:36:18.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.026+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=8U6LRJ0FXJFO tag=BS924QZN10K2 type=write size=2089737 2024-03-09T14:36:18.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.030+0000 7f73315818c0 1 < completed write operation key=8O5OP8C5NX12 tag=PW3ZNYF5CZCL type=write size=126557 2024-03-09T14:36:18.035 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.030+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=PWA53MMGS1B1 type=del size=295979 2024-03-09T14:36:18.039 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.034+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=PGIXAC3HQTEZ type=del 2024-03-09T14:36:18.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.042+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~FK15T66FYKXW.0 size=972019 2024-03-09T14:36:18.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.042+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=FK15T66FYKXW tag=3RQXBW4QANJW type=write size=2020595 2024-03-09T14:36:18.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.046+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=7WYY41M7YZ7Z type=del 2024-03-09T14:36:18.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.050+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=VLSGOJN4V2TN type=del size=3830868 2024-03-09T14:36:18.055 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.050+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~8NL6XTMPXAVX.0 size=1048576 2024-03-09T14:36:18.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.054+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~8NL6XTMPXAVX.1 size=655621 2024-03-09T14:36:18.065 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.062+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=8NL6XTMPXAVX tag=CZKSN6MIWITL type=write size=2752773 2024-03-09T14:36:18.067 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.066+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=DJ7YCCILBQTP type=del size=173374 2024-03-09T14:36:18.071 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.066+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~V9SBCYW02C3T.0 size=23726 2024-03-09T14:36:18.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.074+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=V9SBCYW02C3T tag=V6C7P6OYF5SW type=write size=1072302 2024-03-09T14:36:18.080 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.078+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=8BGK9093RDAR type=del size=3090271 2024-03-09T14:36:18.084 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.082+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~XNQVU6WLP2PJ.0 size=1048576 2024-03-09T14:36:18.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.082+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~XNQVU6WLP2PJ.1 size=1048576 2024-03-09T14:36:18.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.086+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~XNQVU6WLP2PJ.2 size=40707 2024-03-09T14:36:18.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.094+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=XNQVU6WLP2PJ tag=AWJ0HQAQW7NV type=write size=3186435 2024-03-09T14:36:18.099 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.098+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=CDRCOCFLS1JZ type=write size=96177 2024-03-09T14:36:18.103 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.098+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~Y4H3LYZ4NXYM.0 size=1048576 2024-03-09T14:36:18.106 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.102+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~Y4H3LYZ4NXYM.1 size=337300 2024-03-09T14:36:18.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.110+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=Y4H3LYZ4NXYM tag=ASGPM6DFJE1D type=write size=2434452 2024-03-09T14:36:18.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.110+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=2HYCZ1LK435W type=del size=1405215 2024-03-09T14:36:18.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.114+0000 7f73315818c0 -1 < failed to complete operation key=K3JNN1NCKBXN tag=DYOKRFB2JR3X type=del size=4052076: (22) Invalid argument 2024-03-09T14:36:18.122 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.118+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=7AWPUM2F1XOQ type=del size=3100248 2024-03-09T14:36:18.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.122+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~E9GHSSC9GJL8.0 size=1048576 2024-03-09T14:36:18.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.126+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~E9GHSSC9GJL8.1 size=1048576 2024-03-09T14:36:18.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.126+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~E9GHSSC9GJL8.2 size=652974 2024-03-09T14:36:18.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.134+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=E9GHSSC9GJL8 tag=6CATUD4A0CVP type=write size=3798702 2024-03-09T14:36:18.140 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.138+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=W9X2ZY4XRBHN type=del size=1773762 2024-03-09T14:36:18.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.142+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=7Q2GTSQMUOSS type=del 2024-03-09T14:36:18.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.142+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=RB9VE74KK01N type=del size=3314912 2024-03-09T14:36:18.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.150+0000 7f73315818c0 1 < suggested remove operation key=MSMD3ANK4T7J tag=4FQTOHWSHYQV type=del 2024-03-09T14:36:18.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.154+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=4AT1LTMY76KY type=del size=1013129 2024-03-09T14:36:18.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.158+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=2MRQLHDJNYJ7 type=del 2024-03-09T14:36:18.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.162+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~BI3F0B5D3139.0 size=136570 2024-03-09T14:36:18.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.162+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=BI3F0B5D3139 tag=O4AAUTDZEN3R type=write size=1185146 2024-03-09T14:36:18.171 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.166+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=VNSS8JLIB2F3 type=del 2024-03-09T14:36:18.174 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.170+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=WUQKPZZ6VAFJ type=del size=1011521 2024-03-09T14:36:18.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.174+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~5BXNI42RANW2.0 size=1048576 2024-03-09T14:36:18.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.178+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~5BXNI42RANW2.1 size=1048576 2024-03-09T14:36:18.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.182+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~5BXNI42RANW2.2 size=307597 2024-03-09T14:36:18.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.186+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=5BXNI42RANW2 tag=SBP63SM8P8IO type=write size=3453325 2024-03-09T14:36:18.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.190+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=YXWR2N9X5ZMO type=write size=486101 2024-03-09T14:36:18.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.194+0000 7f73315818c0 1 < canceled operation key=0IB00CZ92VV4 tag=V69AFLANN16T type=del size=858170 2024-03-09T14:36:18.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.198+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=F42PISJBKK79 type=del size=1965308 2024-03-09T14:36:18.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.202+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~8U6LRJ0FXJFO.0 size=1041161 2024-03-09T14:36:18.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.206+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=8U6LRJ0FXJFO tag=BS924QZN10K2 type=write size=2089737 2024-03-09T14:36:18.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.210+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=7QU6Z4LCVOZ1 type=del size=3297406 2024-03-09T14:36:18.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.218+0000 7f73315818c0 1 < suggested remove operation key=OU0730HGI176 tag=PWA53MMGS1B1 type=del 2024-03-09T14:36:18.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.222+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~OQZETJ4I7F1B.0 size=1048576 2024-03-09T14:36:18.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.226+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~OQZETJ4I7F1B.1 size=674094 2024-03-09T14:36:18.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.226+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=OQZETJ4I7F1B tag=JW6NBCGDW4L0 type=write size=2771246 2024-03-09T14:36:18.231 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.226+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~FK15T66FYKXW.0 size=972019 2024-03-09T14:36:18.237 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.234+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=FK15T66FYKXW tag=3RQXBW4QANJW type=write size=2020595 2024-03-09T14:36:18.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.238+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=ECW5GVUW0M9E type=del size=3461042 2024-03-09T14:36:18.244 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.242+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=VLSGOJN4V2TN type=del 2024-03-09T14:36:18.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.242+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=1GA7Q38XUL6X type=del size=2332669 2024-03-09T14:36:18.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.246+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=DJ7YCCILBQTP type=del 2024-03-09T14:36:18.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.250+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=MI89SVDTBLSL type=del size=1786851 2024-03-09T14:36:18.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.258+0000 7f73315818c0 1 < suggested update operation key=4P89277EEYJ3 tag=8BGK9093RDAR type=del size=3090271 2024-03-09T14:36:18.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.262+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~2LROBY61H55P.0 size=1048576 2024-03-09T14:36:18.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.266+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~2LROBY61H55P.1 size=55472 2024-03-09T14:36:18.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.266+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=2LROBY61H55P tag=4JT9UJORLWGD type=write size=2152624 2024-03-09T14:36:18.272 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.270+0000 7f73315818c0 1 < completed write operation key=Y8EQAHMTL66U tag=CDRCOCFLS1JZ type=write size=96177 2024-03-09T14:36:18.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.274+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=LV7NP305YY1U type=write size=724656 2024-03-09T14:36:18.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.282+0000 7f73315818c0 1 < canceled operation key=LZC7NPF0GRO7 tag=2HYCZ1LK435W type=del size=1405215 2024-03-09T14:36:18.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.286+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~UVL8BFN14XLY.0 size=1048576 2024-03-09T14:36:18.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.290+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~UVL8BFN14XLY.1 size=767474 2024-03-09T14:36:18.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.290+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=UVL8BFN14XLY tag=LPM0CN43OQKB type=write size=2864626 2024-03-09T14:36:18.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.294+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=7AWPUM2F1XOQ type=del 2024-03-09T14:36:18.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.298+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~H1MZ2S1RLHZ1.0 size=1048576 2024-03-09T14:36:18.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.302+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~H1MZ2S1RLHZ1.1 size=1048576 2024-03-09T14:36:18.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.306+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~H1MZ2S1RLHZ1.2 size=93280 2024-03-09T14:36:18.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.306+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=H1MZ2S1RLHZ1 tag=AN38NZMY2N35 type=write size=3239008 2024-03-09T14:36:18.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.306+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=W9X2ZY4XRBHN type=del 2024-03-09T14:36:18.314 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.310+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=1VJ73KD27KX5 type=del size=3837027 2024-03-09T14:36:18.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.314+0000 7f73315818c0 -1 < failed to complete operation key=OU0730HGI176 tag=RB9VE74KK01N type=del size=3314912: (22) Invalid argument 2024-03-09T14:36:18.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.318+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=3Z3I88BGGWG9 type=del size=2772314 2024-03-09T14:36:18.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.326+0000 7f73315818c0 1 < suggested remove operation key=UMSNM53HTY1J tag=4AT1LTMY76KY type=del 2024-03-09T14:36:18.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.330+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~989Y59DHNET0.0 size=1048576 2024-03-09T14:36:18.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.334+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~989Y59DHNET0.1 size=1048576 2024-03-09T14:36:18.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.334+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~989Y59DHNET0.2 size=259710 2024-03-09T14:36:18.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.334+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=989Y59DHNET0 tag=LFXJKQZ2RQ62 type=write size=3405438 2024-03-09T14:36:18.341 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.338+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~BI3F0B5D3139.0 size=136570 2024-03-09T14:36:18.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.342+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=BI3F0B5D3139 tag=O4AAUTDZEN3R type=write size=1185146 2024-03-09T14:36:18.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.346+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~918IQ3QPP4S3.0 size=1048576 2024-03-09T14:36:18.353 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.350+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~918IQ3QPP4S3.1 size=1048576 2024-03-09T14:36:18.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.354+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~918IQ3QPP4S3.2 size=1002044 2024-03-09T14:36:18.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.354+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=918IQ3QPP4S3 tag=ZICPD26F0VKZ type=write size=4147772 2024-03-09T14:36:18.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.354+0000 7f73315818c0 1 < canceled operation key=Y8EQAHMTL66U tag=WUQKPZZ6VAFJ type=del size=1011521 2024-03-09T14:36:18.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.358+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~MZP015OGJ880.0 size=1048576 2024-03-09T14:36:18.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.362+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~MZP015OGJ880.1 size=482955 2024-03-09T14:36:18.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.362+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=MZP015OGJ880 tag=PF8J9AJJM9QK type=write size=2580107 2024-03-09T14:36:18.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.366+0000 7f73315818c0 1 < completed write operation key=4E3QMBHEAEHO tag=YXWR2N9X5ZMO type=write size=486101 2024-03-09T14:36:18.370 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.366+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=IIMYA1DMWT23 type=del size=3333717 2024-03-09T14:36:18.373 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.370+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=F42PISJBKK79 type=del 2024-03-09T14:36:18.376 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.374+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=AXDO8QHIHY3H type=del size=1481507 2024-03-09T14:36:18.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.374+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=7QU6Z4LCVOZ1 type=del 2024-03-09T14:36:18.381 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.378+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=H9DUTM1FXH2K type=del size=3869107 2024-03-09T14:36:18.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.382+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~OQZETJ4I7F1B.0 size=1048576 2024-03-09T14:36:18.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.382+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~OQZETJ4I7F1B.1 size=674094 2024-03-09T14:36:18.390 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.386+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=OQZETJ4I7F1B tag=JW6NBCGDW4L0 type=write size=2771246 2024-03-09T14:36:18.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.390+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=N8BMXBQWD3Z4 type=del size=4074675 2024-03-09T14:36:18.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.390+0000 7f73315818c0 -1 < failed to complete operation key=4P89277EEYJ3 tag=ECW5GVUW0M9E type=del size=3461042: (22) Invalid argument 2024-03-09T14:36:18.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.398+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~0WPBCEB6QE4Z.0 size=421409 2024-03-09T14:36:18.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.398+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=0WPBCEB6QE4Z tag=K2JX7UQKZKV9 type=write size=1469985 2024-03-09T14:36:18.402 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.398+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=1GA7Q38XUL6X type=del 2024-03-09T14:36:18.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.402+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=A929YENJQ19Q type=del size=531097 2024-03-09T14:36:18.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.402+0000 7f73315818c0 1 < canceled operation key=MSMD3ANK4T7J tag=MI89SVDTBLSL type=del size=1786851 2024-03-09T14:36:18.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.406+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=P4Z86I62FH7G type=del size=2160894 2024-03-09T14:36:18.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.406+0000 7f73315818c0 1 < canceled part key=_multipart_QG4O2L526B8E.2~2LROBY61H55P.0 size=1048576 2024-03-09T14:36:18.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.410+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~2LROBY61H55P.1 size=55472 2024-03-09T14:36:18.416 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.414+0000 7f73315818c0 1 < canceled multipart upload key=QG4O2L526B8E upload=2LROBY61H55P tag=4JT9UJORLWGD type=write size=2152624 2024-03-09T14:36:18.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.414+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=YXSADD9EQX9P type=del size=117322 2024-03-09T14:36:18.423 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.422+0000 7f73315818c0 1 < suggested remove operation key=CVAWRZMAX997 tag=LV7NP305YY1U type=write 2024-03-09T14:36:18.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.422+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=4188KNVDBECC type=write size=567232 2024-03-09T14:36:18.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.426+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~UVL8BFN14XLY.0 size=1048576 2024-03-09T14:36:18.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.426+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~UVL8BFN14XLY.1 size=767474 2024-03-09T14:36:18.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.434+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=UVL8BFN14XLY tag=LPM0CN43OQKB type=write size=2864626 2024-03-09T14:36:18.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.434+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=CZPASLW7FVO0 type=del size=3603437 2024-03-09T14:36:18.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.438+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~H1MZ2S1RLHZ1.0 size=1048576 2024-03-09T14:36:18.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.438+0000 7f73315818c0 1 < canceled part key=_multipart_92GWY5B0K4SX.2~H1MZ2S1RLHZ1.1 size=1048576 2024-03-09T14:36:18.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.438+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~H1MZ2S1RLHZ1.2 size=93280 2024-03-09T14:36:18.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.446+0000 7f73315818c0 1 < canceled multipart upload key=92GWY5B0K4SX upload=H1MZ2S1RLHZ1 tag=AN38NZMY2N35 type=write size=3239008 2024-03-09T14:36:18.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.446+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=JR4I1L18UF19 type=del size=1316844 2024-03-09T14:36:18.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.450+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=1VJ73KD27KX5 type=del 2024-03-09T14:36:18.458 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.454+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=MDGBPP387PMU type=del size=2934038 2024-03-09T14:36:18.460 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.458+0000 7f73315818c0 1 < canceled operation key=LZC7NPF0GRO7 tag=3Z3I88BGGWG9 type=del size=2772314 2024-03-09T14:36:18.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.458+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=7022MHR1XD0U type=del size=295454 2024-03-09T14:36:18.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.462+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~989Y59DHNET0.0 size=1048576 2024-03-09T14:36:18.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.466+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~989Y59DHNET0.1 size=1048576 2024-03-09T14:36:18.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.466+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~989Y59DHNET0.2 size=259710 2024-03-09T14:36:18.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.470+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=989Y59DHNET0 tag=LFXJKQZ2RQ62 type=write size=3405438 2024-03-09T14:36:18.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.474+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=969EYKIM4T2S type=del size=1186869 2024-03-09T14:36:18.480 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.478+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~918IQ3QPP4S3.0 size=1048576 2024-03-09T14:36:18.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.478+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~918IQ3QPP4S3.1 size=1048576 2024-03-09T14:36:18.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.482+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~918IQ3QPP4S3.2 size=1002044 2024-03-09T14:36:18.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.486+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=918IQ3QPP4S3 tag=ZICPD26F0VKZ type=write size=4147772 2024-03-09T14:36:18.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.494+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~X7YAXG0E6GUR.0 size=1048576 2024-03-09T14:36:18.499 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.494+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~X7YAXG0E6GUR.1 size=938866 2024-03-09T14:36:18.499 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.494+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=X7YAXG0E6GUR tag=P3BFET3XFHC4 type=write size=3036018 2024-03-09T14:36:18.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.498+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~MZP015OGJ880.0 size=1048576 2024-03-09T14:36:18.505 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.502+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~MZP015OGJ880.1 size=482955 2024-03-09T14:36:18.509 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.506+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=MZP015OGJ880 tag=PF8J9AJJM9QK type=write size=2580107 2024-03-09T14:36:18.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.510+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~2A15266VMVDT.0 size=338056 2024-03-09T14:36:18.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.510+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=2A15266VMVDT tag=DOQ55OTJ3Q6N type=write size=1386632 2024-03-09T14:36:18.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.514+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=IIMYA1DMWT23 type=del 2024-03-09T14:36:18.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.518+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~9EYMAQ2I5B4A.0 size=920391 2024-03-09T14:36:18.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.518+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=9EYMAQ2I5B4A tag=O1KMUPCVXB5L type=write size=1968967 2024-03-09T14:36:18.526 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.522+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=AXDO8QHIHY3H type=del 2024-03-09T14:36:18.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.526+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=PB6B83F1K25G type=del size=1659718 2024-03-09T14:36:18.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.530+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=H9DUTM1FXH2K type=del 2024-03-09T14:36:18.539 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.534+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~EOVNUKYIVUSY.0 size=1048576 2024-03-09T14:36:18.541 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.538+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~EOVNUKYIVUSY.1 size=842166 2024-03-09T14:36:18.541 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.538+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=EOVNUKYIVUSY tag=9WV1S5I29O39 type=write size=2939318 2024-03-09T14:36:18.547 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.542+0000 7f73315818c0 1 < suggested update operation key=Y8EQAHMTL66U tag=N8BMXBQWD3Z4 type=del size=4074675 2024-03-09T14:36:18.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.550+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=B90CBI4JHCBH type=del size=562451 2024-03-09T14:36:18.558 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.554+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~0WPBCEB6QE4Z.0 size=421409 2024-03-09T14:36:18.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.558+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=0WPBCEB6QE4Z tag=K2JX7UQKZKV9 type=write size=1469985 2024-03-09T14:36:18.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.566+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~YCRITE1RF6JC.0 size=1048576 2024-03-09T14:36:18.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.566+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~YCRITE1RF6JC.1 size=156421 2024-03-09T14:36:18.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.566+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=YCRITE1RF6JC tag=065N20C61SI7 type=write size=2253573 2024-03-09T14:36:18.574 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.570+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=A929YENJQ19Q type=del 2024-03-09T14:36:18.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.578+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~4D8IU6PKWHOG.0 size=731260 2024-03-09T14:36:18.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.578+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=4D8IU6PKWHOG tag=AR7QYXHUO8XS type=write size=1779836 2024-03-09T14:36:18.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.582+0000 7f73315818c0 -1 < failed to complete operation key=CVAWRZMAX997 tag=P4Z86I62FH7G type=del size=2160894: (22) Invalid argument 2024-03-09T14:36:18.590 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.586+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~D8XY5H7RY948.0 size=1048576 2024-03-09T14:36:18.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.590+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~D8XY5H7RY948.1 size=243026 2024-03-09T14:36:18.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.590+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=D8XY5H7RY948 tag=X57YKSP6FF4Z type=write size=2340178 2024-03-09T14:36:18.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.590+0000 7f73315818c0 -1 < failed to complete operation key=CVAWRZMAX997 tag=YXSADD9EQX9P type=del size=117322: (22) Invalid argument 2024-03-09T14:36:18.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.594+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=115CAPRD818Z type=write size=462962 2024-03-09T14:36:18.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.598+0000 7f73315818c0 1 < completed write operation key=0IB00CZ92VV4 tag=4188KNVDBECC type=write size=567232 2024-03-09T14:36:18.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.602+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=RALANPUV8RJL type=write size=498603 2024-03-09T14:36:18.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.606+0000 7f73315818c0 1 < canceled operation key=W6KVP8EQFQ8J tag=CZPASLW7FVO0 type=del size=3603437 2024-03-09T14:36:18.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.610+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~V5GA7GTEVE08.0 size=1048576 2024-03-09T14:36:18.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.614+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~V5GA7GTEVE08.1 size=556991 2024-03-09T14:36:18.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.614+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=V5GA7GTEVE08 tag=RKGHKZJROZOG type=write size=2654143 2024-03-09T14:36:18.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.618+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=JR4I1L18UF19 type=del 2024-03-09T14:36:18.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.622+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~ZTOLYQGW6POB.0 size=1048576 2024-03-09T14:36:18.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.626+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~ZTOLYQGW6POB.1 size=1048576 2024-03-09T14:36:18.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.626+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~ZTOLYQGW6POB.2 size=1019071 2024-03-09T14:36:18.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.626+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=ZTOLYQGW6POB tag=ZJAN4H8DRFTU type=write size=4164799 2024-03-09T14:36:18.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.634+0000 7f73315818c0 1 < suggested update operation key=162AOMHO2BQL tag=MDGBPP387PMU type=del size=2934038 2024-03-09T14:36:18.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.634+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=DS9MPRGE98OB type=del size=3436417 2024-03-09T14:36:18.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.642+0000 7f73315818c0 1 < suggested update operation key=UMSNM53HTY1J tag=7022MHR1XD0U type=del size=295454 2024-03-09T14:36:18.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.646+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~2JXSN9LSRI9O.0 size=1048576 2024-03-09T14:36:18.653 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.650+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~2JXSN9LSRI9O.1 size=480411 2024-03-09T14:36:18.653 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.650+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=2JXSN9LSRI9O tag=B7R4DCTP372U type=write size=2577563 2024-03-09T14:36:18.657 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.654+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=969EYKIM4T2S type=del 2024-03-09T14:36:18.662 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.658+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~PJP4ARMKAG8Q.0 size=1048576 2024-03-09T14:36:18.665 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.662+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~PJP4ARMKAG8Q.1 size=196308 2024-03-09T14:36:18.665 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.662+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=PJP4ARMKAG8Q tag=LX7PAYZHOIYS type=write size=2293460 2024-03-09T14:36:18.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.666+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~X7YAXG0E6GUR.0 size=1048576 2024-03-09T14:36:18.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.666+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~X7YAXG0E6GUR.1 size=938866 2024-03-09T14:36:18.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.670+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=X7YAXG0E6GUR tag=P3BFET3XFHC4 type=write size=3036018 2024-03-09T14:36:18.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.678+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~HFJMP3LOMLXX.0 size=1048576 2024-03-09T14:36:18.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.678+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~HFJMP3LOMLXX.1 size=391665 2024-03-09T14:36:18.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.678+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=HFJMP3LOMLXX tag=BB9T58VO6W5E type=write size=2488817 2024-03-09T14:36:18.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.682+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~2A15266VMVDT.0 size=338056 2024-03-09T14:36:18.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.686+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=2A15266VMVDT tag=DOQ55OTJ3Q6N type=write size=1386632 2024-03-09T14:36:18.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.690+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=2YFHZXU6F07U type=del size=1003623 2024-03-09T14:36:18.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.694+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~9EYMAQ2I5B4A.0 size=920391 2024-03-09T14:36:18.702 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.698+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=9EYMAQ2I5B4A tag=O1KMUPCVXB5L type=write size=1968967 2024-03-09T14:36:18.705 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.702+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=UXPB0ZDLLR4M type=del size=2918343 2024-03-09T14:36:18.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.706+0000 7f73315818c0 1 < suggested remove operation key=MSMD3ANK4T7J tag=PB6B83F1K25G type=del 2024-03-09T14:36:18.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.714+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~YFV0QG3NTEW4.0 size=575099 2024-03-09T14:36:18.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.714+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=YFV0QG3NTEW4 tag=LAR767I80CYR type=write size=1623675 2024-03-09T14:36:18.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.718+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~EOVNUKYIVUSY.0 size=1048576 2024-03-09T14:36:18.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.718+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~EOVNUKYIVUSY.1 size=842166 2024-03-09T14:36:18.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.722+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=EOVNUKYIVUSY tag=9WV1S5I29O39 type=write size=2939318 2024-03-09T14:36:18.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.726+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=MKTQESDC2TBK type=write size=532548 2024-03-09T14:36:18.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.730+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=B90CBI4JHCBH type=del 2024-03-09T14:36:18.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.734+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=40I0ODP5K3UY type=del size=3984922 2024-03-09T14:36:18.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.738+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~YCRITE1RF6JC.0 size=1048576 2024-03-09T14:36:18.743 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.738+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~YCRITE1RF6JC.1 size=156421 2024-03-09T14:36:18.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.746+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=YCRITE1RF6JC tag=065N20C61SI7 type=write size=2253573 2024-03-09T14:36:18.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.750+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=TM8UQWBA6H11 type=del size=382129 2024-03-09T14:36:18.960 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.958+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~4D8IU6PKWHOG.0 size=731260 2024-03-09T14:36:18.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.962+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=4D8IU6PKWHOG tag=AR7QYXHUO8XS type=write size=1779836 2024-03-09T14:36:18.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.966+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=4L7VIFL7NTCK type=del size=269243 2024-03-09T14:36:18.974 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.970+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~D8XY5H7RY948.0 size=1048576 2024-03-09T14:36:18.977 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.974+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~D8XY5H7RY948.1 size=243026 2024-03-09T14:36:18.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.978+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=D8XY5H7RY948 tag=X57YKSP6FF4Z type=write size=2340178 2024-03-09T14:36:18.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.982+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=5GAG44OM5Z0V type=del size=3109708 2024-03-09T14:36:18.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.986+0000 7f73315818c0 1 < completed write operation key=0IB00CZ92VV4 tag=115CAPRD818Z type=write size=462962 2024-03-09T14:36:18.997 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.994+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~H0Q4NRJS657R.0 size=206629 2024-03-09T14:36:18.997 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.994+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=H0Q4NRJS657R tag=W5X1S5JWJX50 type=write size=1255205 2024-03-09T14:36:19.001 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:18.998+0000 7f73315818c0 1 < completed write operation key=K3JNN1NCKBXN tag=RALANPUV8RJL type=write size=498603 2024-03-09T14:36:19.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.002+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=S3PVTGFE9BAT type=del size=3609872 2024-03-09T14:36:19.009 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.006+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~V5GA7GTEVE08.0 size=1048576 2024-03-09T14:36:19.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.006+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~V5GA7GTEVE08.1 size=556991 2024-03-09T14:36:19.017 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.014+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=V5GA7GTEVE08 tag=RKGHKZJROZOG type=write size=2654143 2024-03-09T14:36:19.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.022+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~X0L5JCVE674N.0 size=1048576 2024-03-09T14:36:19.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.026+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~X0L5JCVE674N.1 size=879136 2024-03-09T14:36:19.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.026+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=X0L5JCVE674N tag=ITMMH999H13T type=write size=2976288 2024-03-09T14:36:19.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.030+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~ZTOLYQGW6POB.0 size=1048576 2024-03-09T14:36:19.035 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.030+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~ZTOLYQGW6POB.1 size=1048576 2024-03-09T14:36:19.038 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.034+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~ZTOLYQGW6POB.2 size=1019071 2024-03-09T14:36:19.044 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.042+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=ZTOLYQGW6POB tag=ZJAN4H8DRFTU type=write size=4164799 2024-03-09T14:36:19.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.046+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=DNPPNX39EBOK type=del size=3444431 2024-03-09T14:36:19.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.054+0000 7f73315818c0 1 < suggested remove operation key=OU0730HGI176 tag=DS9MPRGE98OB type=del 2024-03-09T14:36:19.068 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.066+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~92UMY9TXN3P3.0 size=1048576 2024-03-09T14:36:19.071 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.070+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~92UMY9TXN3P3.1 size=751343 2024-03-09T14:36:19.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.070+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=92UMY9TXN3P3 tag=OCOXQZH1K9TH type=write size=2848495 2024-03-09T14:36:19.076 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.074+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~2JXSN9LSRI9O.0 size=1048576 2024-03-09T14:36:19.078 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.074+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~2JXSN9LSRI9O.1 size=480411 2024-03-09T14:36:19.084 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.082+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=2JXSN9LSRI9O tag=B7R4DCTP372U type=write size=2577563 2024-03-09T14:36:19.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.086+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~FKIHQ4BYBQ4N.0 size=1048576 2024-03-09T14:36:19.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.090+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~FKIHQ4BYBQ4N.1 size=404379 2024-03-09T14:36:19.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.090+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=FKIHQ4BYBQ4N tag=7UAQUICEQ71S type=write size=2501531 2024-03-09T14:36:19.097 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.094+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~PJP4ARMKAG8Q.0 size=1048576 2024-03-09T14:36:19.100 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.098+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~PJP4ARMKAG8Q.1 size=196308 2024-03-09T14:36:19.106 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.102+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=PJP4ARMKAG8Q tag=LX7PAYZHOIYS type=write size=2293460 2024-03-09T14:36:19.110 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.106+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=UDAP8FCRN6DN type=del size=1471835 2024-03-09T14:36:19.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.110+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~HFJMP3LOMLXX.0 size=1048576 2024-03-09T14:36:19.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.114+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~HFJMP3LOMLXX.1 size=391665 2024-03-09T14:36:19.124 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.122+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=HFJMP3LOMLXX tag=BB9T58VO6W5E type=write size=2488817 2024-03-09T14:36:19.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.126+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~GZAG8QQ1U8KY.0 size=291880 2024-03-09T14:36:19.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.126+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=GZAG8QQ1U8KY tag=P9GJYPU6Z1IF type=write size=1340456 2024-03-09T14:36:19.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.130+0000 7f73315818c0 -1 < failed to complete operation key=MSMD3ANK4T7J tag=2YFHZXU6F07U type=del size=1003623: (22) Invalid argument 2024-03-09T14:36:19.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.134+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=IPLOYSPZU7KJ type=del size=696521 2024-03-09T14:36:19.141 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.138+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=UXPB0ZDLLR4M type=del 2024-03-09T14:36:19.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.142+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=ANKV1WEE0RGZ type=del size=354709 2024-03-09T14:36:19.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.146+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~YFV0QG3NTEW4.0 size=575099 2024-03-09T14:36:19.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.150+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=YFV0QG3NTEW4 tag=LAR767I80CYR type=write size=1623675 2024-03-09T14:36:19.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.154+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=E5PNB88NKDP4 type=del size=549871 2024-03-09T14:36:19.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.158+0000 7f73315818c0 1 < canceled operation key=P8R5RJI8LGZ9 tag=MKTQESDC2TBK type=write size=532548 2024-03-09T14:36:19.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.162+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=28Q6ESJ836UR type=del size=4041259 2024-03-09T14:36:19.172 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.170+0000 7f73315818c0 1 < suggested update operation key=W6KVP8EQFQ8J tag=40I0ODP5K3UY type=del size=3984922 2024-03-09T14:36:19.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.174+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=YQ7QLUDJ1MX7 type=del size=2916866 2024-03-09T14:36:19.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.178+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=TM8UQWBA6H11 type=del 2024-03-09T14:36:19.186 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.182+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~HR60MN59KBY9.0 size=1048576 2024-03-09T14:36:19.189 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.186+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~HR60MN59KBY9.1 size=1048576 2024-03-09T14:36:19.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.190+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~HR60MN59KBY9.2 size=917529 2024-03-09T14:36:19.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.190+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=HR60MN59KBY9 tag=30D71OI0TJQQ type=write size=4063257 2024-03-09T14:36:19.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.194+0000 7f73315818c0 1 < canceled operation key=MD7FXGYXB823 tag=4L7VIFL7NTCK type=del size=269243 2024-03-09T14:36:19.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.194+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=HF4MSO257LE4 type=del size=3745728 2024-03-09T14:36:19.205 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.202+0000 7f73315818c0 1 < suggested update operation key=XELG1AWATMVL tag=5GAG44OM5Z0V type=del size=3109708 2024-03-09T14:36:19.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.206+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=ZQITE8JC5TFX type=del size=565164 2024-03-09T14:36:19.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.210+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~H0Q4NRJS657R.0 size=206629 2024-03-09T14:36:19.217 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.214+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=H0Q4NRJS657R tag=W5X1S5JWJX50 type=write size=1255205 2024-03-09T14:36:19.220 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.218+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=QBZ5OCP5N8L6 type=del size=3334674 2024-03-09T14:36:19.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.226+0000 7f73315818c0 1 < suggested remove operation key=UMSNM53HTY1J tag=S3PVTGFE9BAT type=del 2024-03-09T14:36:19.231 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.226+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=4JR3B7NBRAEP type=del size=1535653 2024-03-09T14:36:19.235 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.230+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~X0L5JCVE674N.0 size=1048576 2024-03-09T14:36:19.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.234+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~X0L5JCVE674N.1 size=879136 2024-03-09T14:36:19.244 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.242+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=X0L5JCVE674N tag=ITMMH999H13T type=write size=2976288 2024-03-09T14:36:19.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.246+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=H3SZQWBVKA0M type=write size=659605 2024-03-09T14:36:19.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.246+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=DNPPNX39EBOK type=del 2024-03-09T14:36:19.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.250+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=02Q9JXY9QJ7X type=del size=3221758 2024-03-09T14:36:19.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.254+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~92UMY9TXN3P3.0 size=1048576 2024-03-09T14:36:19.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.258+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~92UMY9TXN3P3.1 size=751343 2024-03-09T14:36:19.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.266+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=92UMY9TXN3P3 tag=OCOXQZH1K9TH type=write size=2848495 2024-03-09T14:36:19.271 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.266+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=SAKZJWEGTVLR type=del size=2793933 2024-03-09T14:36:19.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.270+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~FKIHQ4BYBQ4N.0 size=1048576 2024-03-09T14:36:19.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.274+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~FKIHQ4BYBQ4N.1 size=404379 2024-03-09T14:36:19.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.282+0000 7f73315818c0 1 < completed multipart upload key=IA0AB7UVLFEJ upload=FKIHQ4BYBQ4N tag=7UAQUICEQ71S type=write size=2501531 2024-03-09T14:36:19.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.282+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=HP1C54DZ3CQH type=del size=1005099 2024-03-09T14:36:19.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.286+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=UDAP8FCRN6DN type=del 2024-03-09T14:36:19.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.294+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~QJZB556YW5LI.0 size=1048576 2024-03-09T14:36:19.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.298+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~QJZB556YW5LI.1 size=620757 2024-03-09T14:36:19.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.298+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=QJZB556YW5LI tag=ZMV98E73NTO4 type=write size=2717909 2024-03-09T14:36:19.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.298+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~GZAG8QQ1U8KY.0 size=291880 2024-03-09T14:36:19.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.306+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=GZAG8QQ1U8KY tag=P9GJYPU6Z1IF type=write size=1340456 2024-03-09T14:36:19.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.310+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=DG1PH89Y3DGU type=del size=143222 2024-03-09T14:36:19.316 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.314+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=IPLOYSPZU7KJ type=del 2024-03-09T14:36:19.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.318+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=22Y4MI2DSH04 type=del size=1321495 2024-03-09T14:36:19.324 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.322+0000 7f73315818c0 1 < canceled operation key=Y8EQAHMTL66U tag=ANKV1WEE0RGZ type=del size=354709 2024-03-09T14:36:19.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.326+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=MCIY7MVRB4T3 type=del size=2246942 2024-03-09T14:36:19.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.334+0000 7f73315818c0 1 < suggested update operation key=MD7FXGYXB823 tag=E5PNB88NKDP4 type=del size=549871 2024-03-09T14:36:19.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.338+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=K5KCF37QOE3P type=del size=792832 2024-03-09T14:36:19.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.342+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=28Q6ESJ836UR type=del 2024-03-09T14:36:19.348 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.346+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=K1UBFSNS0KQQ type=del size=1010423 2024-03-09T14:36:19.351 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.346+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=YQ7QLUDJ1MX7 type=del 2024-03-09T14:36:19.354 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.350+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=RCHVLRE4GV5L type=write size=1264 2024-03-09T14:36:19.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.354+0000 7f73315818c0 1 < canceled part key=_multipart_M8F9TA80VME6.2~HR60MN59KBY9.0 size=1048576 2024-03-09T14:36:19.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.354+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~HR60MN59KBY9.1 size=1048576 2024-03-09T14:36:19.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.358+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~HR60MN59KBY9.2 size=917529 2024-03-09T14:36:19.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.362+0000 7f73315818c0 1 < canceled multipart upload key=M8F9TA80VME6 upload=HR60MN59KBY9 tag=30D71OI0TJQQ type=write size=4063257 2024-03-09T14:36:19.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.370+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~PW6K85HEJ1T8.0 size=314235 2024-03-09T14:36:19.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.370+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=PW6K85HEJ1T8 tag=PNO4VCDE9N2A type=write size=1362811 2024-03-09T14:36:19.381 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.378+0000 7f73315818c0 1 < suggested remove operation key=8O5OP8C5NX12 tag=HF4MSO257LE4 type=del 2024-03-09T14:36:19.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.386+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~CNN5XW4CMXV5.0 size=1048576 2024-03-09T14:36:19.390 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.386+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~CNN5XW4CMXV5.1 size=1048576 2024-03-09T14:36:19.393 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.390+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~CNN5XW4CMXV5.2 size=555596 2024-03-09T14:36:19.393 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.390+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=CNN5XW4CMXV5 tag=40OEP7H9R5C6 type=write size=3701324 2024-03-09T14:36:19.398 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.394+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=ZQITE8JC5TFX type=del 2024-03-09T14:36:19.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.398+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=QWW4OT6T4IWE type=del size=1362219 2024-03-09T14:36:19.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.406+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=QBZ5OCP5N8L6 type=del 2024-03-09T14:36:19.416 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.414+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~L2M1MRIHXJ39.0 size=1048576 2024-03-09T14:36:19.418 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.414+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~L2M1MRIHXJ39.1 size=76200 2024-03-09T14:36:19.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.414+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=L2M1MRIHXJ39 tag=LFOSPY0TKLYY type=write size=2173352 2024-03-09T14:36:19.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.418+0000 7f73315818c0 -1 < failed to complete operation key=8O5OP8C5NX12 tag=4JR3B7NBRAEP type=del size=1535653: (22) Invalid argument 2024-03-09T14:36:19.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.426+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~1J3TYW2AK13U.0 size=1048576 2024-03-09T14:36:19.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.426+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~1J3TYW2AK13U.1 size=942842 2024-03-09T14:36:19.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.426+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=1J3TYW2AK13U tag=NCJEFCPTPHJ8 type=write size=3039994 2024-03-09T14:36:19.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.430+0000 7f73315818c0 1 < completed write operation key=162AOMHO2BQL tag=H3SZQWBVKA0M type=write size=659605 2024-03-09T14:36:19.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.434+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=3I356F03K5PM type=del size=882768 2024-03-09T14:36:19.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.438+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=02Q9JXY9QJ7X type=del 2024-03-09T14:36:19.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.442+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=17BEZ5TV78YA type=del size=951125 2024-03-09T14:36:19.449 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.446+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=SAKZJWEGTVLR type=del 2024-03-09T14:36:19.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.450+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=E8KVUHSJW18E type=del size=1099588 2024-03-09T14:36:19.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.454+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=HP1C54DZ3CQH type=del 2024-03-09T14:36:19.461 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.458+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=K4NW7CX3EY3F type=del size=3855300 2024-03-09T14:36:19.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.462+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~QJZB556YW5LI.0 size=1048576 2024-03-09T14:36:19.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.466+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~QJZB556YW5LI.1 size=620757 2024-03-09T14:36:19.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.470+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=QJZB556YW5LI tag=ZMV98E73NTO4 type=write size=2717909 2024-03-09T14:36:19.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.474+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=L6VM926XHN0E type=del size=587958 2024-03-09T14:36:19.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.478+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=DG1PH89Y3DGU type=del 2024-03-09T14:36:19.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.486+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~CE1BTSQCVC1Q.0 size=1048576 2024-03-09T14:36:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.486+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~CE1BTSQCVC1Q.1 size=1048576 2024-03-09T14:36:19.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.490+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~CE1BTSQCVC1Q.2 size=594292 2024-03-09T14:36:19.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.490+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=CE1BTSQCVC1Q tag=735QG53CQHIA type=write size=3740020 2024-03-09T14:36:19.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.494+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=22Y4MI2DSH04 type=del 2024-03-09T14:36:19.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.502+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~V0KGLMFEO18I.0 size=886968 2024-03-09T14:36:19.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.502+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=V0KGLMFEO18I tag=KBFVA3L9NFQ4 type=write size=1935544 2024-03-09T14:36:19.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.506+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=MCIY7MVRB4T3 type=del 2024-03-09T14:36:19.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.506+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=27XBH8ZDAWWY type=del size=2720513 2024-03-09T14:36:19.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.510+0000 7f73315818c0 -1 < failed to complete operation key=8O5OP8C5NX12 tag=K5KCF37QOE3P type=del size=792832: (22) Invalid argument 2024-03-09T14:36:19.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.514+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=35AT12UJC2UF type=del size=2680003 2024-03-09T14:36:19.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.518+0000 7f73315818c0 1 < canceled operation key=0IB00CZ92VV4 tag=K1UBFSNS0KQQ type=del size=1010423 2024-03-09T14:36:19.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.522+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=6Z1HUSWVCFHX type=write size=258129 2024-03-09T14:36:19.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.526+0000 7f73315818c0 1 < completed write operation key=M8F9TA80VME6 tag=RCHVLRE4GV5L type=write size=1264 2024-03-09T14:36:19.535 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.530+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~I1CEZGVJ6J5W.0 size=594543 2024-03-09T14:36:19.535 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.530+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=I1CEZGVJ6J5W tag=OR25NO1OLFTZ type=write size=1643119 2024-03-09T14:36:19.539 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.534+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~PW6K85HEJ1T8.0 size=314235 2024-03-09T14:36:19.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.542+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=PW6K85HEJ1T8 tag=PNO4VCDE9N2A type=write size=1362811 2024-03-09T14:36:19.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.546+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=KI4G77W0XG2L type=del size=1550212 2024-03-09T14:36:19.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.550+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~CNN5XW4CMXV5.0 size=1048576 2024-03-09T14:36:19.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.550+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~CNN5XW4CMXV5.1 size=1048576 2024-03-09T14:36:19.557 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.554+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~CNN5XW4CMXV5.2 size=555596 2024-03-09T14:36:19.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.558+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=CNN5XW4CMXV5 tag=40OEP7H9R5C6 type=write size=3701324 2024-03-09T14:36:19.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.562+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=JSEEA72F9Y9F type=del size=2672461 2024-03-09T14:36:19.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.566+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=QWW4OT6T4IWE type=del 2024-03-09T14:36:19.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.570+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=MCZQN72I2ZLG type=del size=3677309 2024-03-09T14:36:19.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.574+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~L2M1MRIHXJ39.0 size=1048576 2024-03-09T14:36:19.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.578+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~L2M1MRIHXJ39.1 size=76200 2024-03-09T14:36:19.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.582+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=L2M1MRIHXJ39 tag=LFOSPY0TKLYY type=write size=2173352 2024-03-09T14:36:19.590 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.586+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=ZSKKQK13PN14 type=del size=569506 2024-03-09T14:36:19.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.590+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~1J3TYW2AK13U.0 size=1048576 2024-03-09T14:36:19.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.590+0000 7f73315818c0 1 < canceled part key=_multipart_0WGGMHTEF2LB.2~1J3TYW2AK13U.1 size=942842 2024-03-09T14:36:19.599 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.594+0000 7f73315818c0 1 < canceled multipart upload key=0WGGMHTEF2LB upload=1J3TYW2AK13U tag=NCJEFCPTPHJ8 type=write size=3039994 2024-03-09T14:36:19.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.598+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=8EJOO6Z51UWN type=del size=1005532 2024-03-09T14:36:19.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.602+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=3I356F03K5PM type=del 2024-03-09T14:36:19.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.610+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~UB8H6K2CU0WB.0 size=1048576 2024-03-09T14:36:19.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.614+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~UB8H6K2CU0WB.1 size=1048576 2024-03-09T14:36:19.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.614+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~UB8H6K2CU0WB.2 size=980106 2024-03-09T14:36:19.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.614+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=UB8H6K2CU0WB tag=VKHS0N6X2X1F type=write size=4125834 2024-03-09T14:36:19.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.618+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=17BEZ5TV78YA type=del 2024-03-09T14:36:19.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.622+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=QE72JGRNNT17 type=del size=579731 2024-03-09T14:36:19.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.626+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=E8KVUHSJW18E type=del 2024-03-09T14:36:19.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.634+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~NFYH5VMJ897P.0 size=1048576 2024-03-09T14:36:19.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.634+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~NFYH5VMJ897P.1 size=1048576 2024-03-09T14:36:19.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.638+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~NFYH5VMJ897P.2 size=575746 2024-03-09T14:36:19.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.638+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=NFYH5VMJ897P tag=0J6M5YR9J9QL type=write size=3721474 2024-03-09T14:36:19.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.646+0000 7f73315818c0 1 < suggested update operation key=PXFZLDSAR1EG tag=K4NW7CX3EY3F type=del size=3855300 2024-03-09T14:36:19.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.650+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~CMXPXXPKIV3C.0 size=1048576 2024-03-09T14:36:19.657 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.654+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~CMXPXXPKIV3C.1 size=1048576 2024-03-09T14:36:19.660 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.658+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~CMXPXXPKIV3C.2 size=594040 2024-03-09T14:36:19.660 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.658+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=CMXPXXPKIV3C tag=K0HVB62N5YWH type=write size=3739768 2024-03-09T14:36:19.664 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.662+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=L6VM926XHN0E type=del 2024-03-09T14:36:19.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.666+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=PFS9ZQDVF51H type=write size=880548 2024-03-09T14:36:19.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.666+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~CE1BTSQCVC1Q.0 size=1048576 2024-03-09T14:36:19.673 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.670+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~CE1BTSQCVC1Q.1 size=1048576 2024-03-09T14:36:19.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.674+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~CE1BTSQCVC1Q.2 size=594292 2024-03-09T14:36:19.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.678+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=CE1BTSQCVC1Q tag=735QG53CQHIA type=write size=3740020 2024-03-09T14:36:19.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.682+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=14WU491WFW5D type=write size=822851 2024-03-09T14:36:19.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.686+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~V0KGLMFEO18I.0 size=886968 2024-03-09T14:36:19.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.694+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=V0KGLMFEO18I tag=KBFVA3L9NFQ4 type=write size=1935544 2024-03-09T14:36:19.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.698+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=8EB4MPJOQSRB type=del size=1837092 2024-03-09T14:36:19.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.702+0000 7f73315818c0 -1 < failed to complete operation key=PXFZLDSAR1EG tag=27XBH8ZDAWWY type=del size=2720513: (22) Invalid argument 2024-03-09T14:36:19.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.706+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=R66D2X5OEL2R type=del size=1405131 2024-03-09T14:36:19.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.710+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=35AT12UJC2UF type=del 2024-03-09T14:36:19.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.714+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=EK7UJEPZT63D type=del size=3160689 2024-03-09T14:36:19.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.718+0000 7f73315818c0 1 < completed write operation key=K3JNN1NCKBXN tag=6Z1HUSWVCFHX type=write size=258129 2024-03-09T14:36:19.725 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.722+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=M3XH1SJ2B7HX type=del size=1479865 2024-03-09T14:36:19.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.726+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~I1CEZGVJ6J5W.0 size=594543 2024-03-09T14:36:19.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.730+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=I1CEZGVJ6J5W tag=OR25NO1OLFTZ type=write size=1643119 2024-03-09T14:36:19.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.738+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~9I1KY2FQV6JK.0 size=1048576 2024-03-09T14:36:19.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.742+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~9I1KY2FQV6JK.1 size=1048576 2024-03-09T14:36:19.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.746+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~9I1KY2FQV6JK.2 size=463143 2024-03-09T14:36:19.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.746+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=9I1KY2FQV6JK tag=Z2H59RZ7WN0M type=write size=3608871 2024-03-09T14:36:19.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.750+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=KI4G77W0XG2L type=del 2024-03-09T14:36:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.754+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~XWR7DN0BXXF9.0 size=457756 2024-03-09T14:36:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.754+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=XWR7DN0BXXF9 tag=S5KLK7MV58JW type=write size=1506332 2024-03-09T14:36:19.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.762+0000 7f73315818c0 1 < suggested remove operation key=XELG1AWATMVL tag=JSEEA72F9Y9F type=del 2024-03-09T14:36:19.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.766+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~WGR1U6925H7I.0 size=1048576 2024-03-09T14:36:19.773 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.770+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~WGR1U6925H7I.1 size=1048576 2024-03-09T14:36:19.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.774+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~WGR1U6925H7I.2 size=535377 2024-03-09T14:36:19.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.774+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=WGR1U6925H7I tag=LW1L6B0XJP73 type=write size=3681105 2024-03-09T14:36:19.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.774+0000 7f73315818c0 -1 < failed to complete operation key=PXFZLDSAR1EG tag=MCZQN72I2ZLG type=del size=3677309: (22) Invalid argument 2024-03-09T14:36:19.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.778+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=BKE42XCULTP0 type=del size=3834978 2024-03-09T14:36:19.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.786+0000 7f73315818c0 1 < suggested update operation key=4P89277EEYJ3 tag=ZSKKQK13PN14 type=del size=569506 2024-03-09T14:36:19.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.790+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=Y55ZXPFMM0LK type=write size=196059 2024-03-09T14:36:19.794 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.790+0000 7f73315818c0 1 < canceled operation key=S302JFYEDEL7 tag=8EJOO6Z51UWN type=del size=1005532 2024-03-09T14:36:19.800 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.798+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~KPX1HROPTH31.0 size=1048576 2024-03-09T14:36:19.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.798+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~KPX1HROPTH31.1 size=1048576 2024-03-09T14:36:19.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.802+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~KPX1HROPTH31.2 size=221704 2024-03-09T14:36:19.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.802+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=KPX1HROPTH31 tag=J1BVQRDJLLI2 type=write size=3367432 2024-03-09T14:36:19.807 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.802+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~UB8H6K2CU0WB.0 size=1048576 2024-03-09T14:36:19.809 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.806+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~UB8H6K2CU0WB.1 size=1048576 2024-03-09T14:36:19.811 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.810+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~UB8H6K2CU0WB.2 size=980106 2024-03-09T14:36:19.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.814+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=UB8H6K2CU0WB tag=VKHS0N6X2X1F type=write size=4125834 2024-03-09T14:36:19.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.818+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=K7E7GAQPV57U type=del size=3427275 2024-03-09T14:36:19.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.818+0000 7f73315818c0 -1 < failed to complete operation key=4P89277EEYJ3 tag=QE72JGRNNT17 type=del size=579731: (22) Invalid argument 2024-03-09T14:36:19.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.822+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=A6RN2FMFP3G7 type=del size=1686647 2024-03-09T14:36:19.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.826+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~NFYH5VMJ897P.0 size=1048576 2024-03-09T14:36:19.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.826+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~NFYH5VMJ897P.1 size=1048576 2024-03-09T14:36:19.833 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.830+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~NFYH5VMJ897P.2 size=575746 2024-03-09T14:36:19.838 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.834+0000 7f73315818c0 1 < completed multipart upload key=0IB00CZ92VV4 upload=NFYH5VMJ897P tag=0J6M5YR9J9QL type=write size=3721474 2024-03-09T14:36:19.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.838+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=DNNEJF16TW0M type=del size=635420 2024-03-09T14:36:19.845 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.842+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~CMXPXXPKIV3C.0 size=1048576 2024-03-09T14:36:19.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.842+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~CMXPXXPKIV3C.1 size=1048576 2024-03-09T14:36:19.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.846+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~CMXPXXPKIV3C.2 size=594040 2024-03-09T14:36:19.855 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.850+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=CMXPXXPKIV3C tag=K0HVB62N5YWH type=write size=3739768 2024-03-09T14:36:19.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.858+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=P02RAZFH6NMN type=write size=80939 2024-03-09T14:36:19.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.858+0000 7f73315818c0 1 < completed write operation key=P9LI5ESR3N9G tag=PFS9ZQDVF51H type=write size=880548 2024-03-09T14:36:19.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.862+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=6HVTJLEKYYI3 type=del size=679549 2024-03-09T14:36:19.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.866+0000 7f73315818c0 1 < completed write operation key=92GWY5B0K4SX tag=14WU491WFW5D type=write size=822851 2024-03-09T14:36:19.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.870+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~SO0G1SGQZ13C.0 size=751478 2024-03-09T14:36:19.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.870+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=SO0G1SGQZ13C tag=M7RTOMMOFCUI type=write size=1800054 2024-03-09T14:36:19.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.874+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=8EB4MPJOQSRB type=del 2024-03-09T14:36:19.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.878+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=QWW9O4UBSJI8 type=del size=1279734 2024-03-09T14:36:19.884 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.882+0000 7f73315818c0 1 < canceled operation key=3ADG4GWVARC1 tag=R66D2X5OEL2R type=del size=1405131 2024-03-09T14:36:19.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.886+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=X9KZT4BKV2DY type=del size=4191856 2024-03-09T14:36:19.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.890+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=EK7UJEPZT63D type=del 2024-03-09T14:36:19.898 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.894+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~YK7CRDQER9C7.0 size=1048576 2024-03-09T14:36:19.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.898+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~YK7CRDQER9C7.1 size=1048576 2024-03-09T14:36:19.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.898+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~YK7CRDQER9C7.2 size=802548 2024-03-09T14:36:19.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.898+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=YK7CRDQER9C7 tag=I3P1U75YJ7YH type=write size=3948276 2024-03-09T14:36:19.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.906+0000 7f73315818c0 1 < suggested remove operation key=PAJYGD2GXHDT tag=M3XH1SJ2B7HX type=del 2024-03-09T14:36:19.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.914+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~YJQ6TEN17NH8.0 size=1048576 2024-03-09T14:36:19.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.914+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~YJQ6TEN17NH8.1 size=111684 2024-03-09T14:36:19.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.914+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=YJQ6TEN17NH8 tag=7BLOFQVYUVXX type=write size=2208836 2024-03-09T14:36:19.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.918+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~9I1KY2FQV6JK.0 size=1048576 2024-03-09T14:36:19.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.922+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~9I1KY2FQV6JK.1 size=1048576 2024-03-09T14:36:19.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.922+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~9I1KY2FQV6JK.2 size=463143 2024-03-09T14:36:19.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.930+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=9I1KY2FQV6JK tag=Z2H59RZ7WN0M type=write size=3608871 2024-03-09T14:36:19.938 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.934+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~DSMF6MXE3WX5.0 size=1048576 2024-03-09T14:36:19.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.938+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~DSMF6MXE3WX5.1 size=866581 2024-03-09T14:36:19.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.938+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=DSMF6MXE3WX5 tag=4JUIY1UDCU6P type=write size=2963733 2024-03-09T14:36:19.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.942+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~XWR7DN0BXXF9.0 size=457756 2024-03-09T14:36:19.950 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.946+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=XWR7DN0BXXF9 tag=S5KLK7MV58JW type=write size=1506332 2024-03-09T14:36:19.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.950+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=VKO4H8NJ0K8S type=del size=467678 2024-03-09T14:36:19.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.954+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~WGR1U6925H7I.0 size=1048576 2024-03-09T14:36:19.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.958+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~WGR1U6925H7I.1 size=1048576 2024-03-09T14:36:19.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.958+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~WGR1U6925H7I.2 size=535377 2024-03-09T14:36:19.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.966+0000 7f73315818c0 1 < completed multipart upload key=0IB00CZ92VV4 upload=WGR1U6925H7I tag=LW1L6B0XJP73 type=write size=3681105 2024-03-09T14:36:19.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.966+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=B0J0JN3OM11K type=del size=3464505 2024-03-09T14:36:19.974 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.970+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=BKE42XCULTP0 type=del 2024-03-09T14:36:19.981 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.978+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~784020SZDM9F.0 size=1048576 2024-03-09T14:36:19.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.978+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~784020SZDM9F.1 size=1048576 2024-03-09T14:36:19.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.982+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~784020SZDM9F.2 size=969278 2024-03-09T14:36:19.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.982+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=784020SZDM9F tag=27ALAOYF6F3D type=write size=4115006 2024-03-09T14:36:19.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.986+0000 7f73315818c0 1 < canceled operation key=162AOMHO2BQL tag=Y55ZXPFMM0LK type=write size=196059 2024-03-09T14:36:19.993 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.990+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=5S28K2XI6E9A type=del size=1749080 2024-03-09T14:36:19.994 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.990+0000 7f73315818c0 1 < canceled part key=_multipart_PAJYGD2GXHDT.2~KPX1HROPTH31.0 size=1048576 2024-03-09T14:36:19.996 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.994+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~KPX1HROPTH31.1 size=1048576 2024-03-09T14:36:19.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:19.998+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~KPX1HROPTH31.2 size=221704 2024-03-09T14:36:20.006 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.002+0000 7f73315818c0 1 < canceled multipart upload key=PAJYGD2GXHDT upload=KPX1HROPTH31 tag=J1BVQRDJLLI2 type=write size=3367432 2024-03-09T14:36:20.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.006+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=JO06M1WU20LH type=del size=3583000 2024-03-09T14:36:20.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.010+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=K7E7GAQPV57U type=del 2024-03-09T14:36:20.018 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.014+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~RYA60UBEB4RU.0 size=458889 2024-03-09T14:36:20.018 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.014+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=RYA60UBEB4RU tag=SNXEN2CC9GYG type=write size=1507465 2024-03-09T14:36:20.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.018+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=A6RN2FMFP3G7 type=del 2024-03-09T14:36:20.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.022+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=VPLZPFSYSPBC type=write size=417585 2024-03-09T14:36:20.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.026+0000 7f73315818c0 1 < canceled operation key=0WGGMHTEF2LB tag=DNNEJF16TW0M type=del size=635420 2024-03-09T14:36:20.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.030+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=GG0ERKOJGLTN type=write size=522419 2024-03-09T14:36:20.039 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.034+0000 7f73315818c0 1 < suggested remove operation key=K3JNN1NCKBXN tag=P02RAZFH6NMN type=write 2024-03-09T14:36:20.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.046+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~3TBMI8VJPTC1.0 size=1048576 2024-03-09T14:36:20.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.050+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~3TBMI8VJPTC1.1 size=1048576 2024-03-09T14:36:20.054 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.050+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~3TBMI8VJPTC1.2 size=872540 2024-03-09T14:36:20.054 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.050+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=3TBMI8VJPTC1 tag=W86X58JXLUQC type=write size=4018268 2024-03-09T14:36:20.055 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.054+0000 7f73315818c0 -1 < no bi entry to suggest for operation key=K3JNN1NCKBXN tag=6HVTJLEKYYI3 type=del size=679549: (2) No such file or directory 2024-03-09T14:36:20.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.054+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=R41LDBOLCT88 type=write size=916713 2024-03-09T14:36:20.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.058+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~SO0G1SGQZ13C.0 size=751478 2024-03-09T14:36:20.068 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.066+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=SO0G1SGQZ13C tag=M7RTOMMOFCUI type=write size=1800054 2024-03-09T14:36:20.071 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.070+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=PY84BOYFTW9W type=del size=1397062 2024-03-09T14:36:20.078 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.074+0000 7f73315818c0 1 < suggested remove operation key=OU0730HGI176 tag=QWW9O4UBSJI8 type=del 2024-03-09T14:36:20.084 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.082+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~FGR1H5U7BWEH.0 size=158322 2024-03-09T14:36:20.084 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.082+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=FGR1H5U7BWEH tag=6XWD13S9A3IZ type=write size=1206898 2024-03-09T14:36:20.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.086+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=X9KZT4BKV2DY type=del 2024-03-09T14:36:20.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.086+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=2J7MJ67L6KFB type=del size=1800696 2024-03-09T14:36:20.095 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.090+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~YK7CRDQER9C7.0 size=1048576 2024-03-09T14:36:20.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.094+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~YK7CRDQER9C7.1 size=1048576 2024-03-09T14:36:20.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.094+0000 7f73315818c0 1 < canceled part key=_multipart_UMSNM53HTY1J.2~YK7CRDQER9C7.2 size=802548 2024-03-09T14:36:20.103 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.102+0000 7f73315818c0 1 < canceled multipart upload key=UMSNM53HTY1J upload=YK7CRDQER9C7 tag=I3P1U75YJ7YH type=write size=3948276 2024-03-09T14:36:20.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.102+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=M13ZYDPPOYOM type=del size=757100 2024-03-09T14:36:20.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.106+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~YJQ6TEN17NH8.0 size=1048576 2024-03-09T14:36:20.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.110+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~YJQ6TEN17NH8.1 size=111684 2024-03-09T14:36:20.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.114+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=YJQ6TEN17NH8 tag=7BLOFQVYUVXX type=write size=2208836 2024-03-09T14:36:20.123 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.118+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=FWI537V0U4H6 type=del size=795664 2024-03-09T14:36:20.123 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.122+0000 7f73315818c0 1 < canceled part key=_multipart_S302JFYEDEL7.2~DSMF6MXE3WX5.0 size=1048576 2024-03-09T14:36:20.126 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.122+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~DSMF6MXE3WX5.1 size=866581 2024-03-09T14:36:20.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.130+0000 7f73315818c0 1 < canceled multipart upload key=S302JFYEDEL7 upload=DSMF6MXE3WX5 tag=4JUIY1UDCU6P type=write size=2963733 2024-03-09T14:36:20.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.134+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~HV2PM0V34136.0 size=306833 2024-03-09T14:36:20.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.134+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=HV2PM0V34136 tag=RHSKOQ41UING type=write size=1355409 2024-03-09T14:36:20.140 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.138+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=VKO4H8NJ0K8S type=del 2024-03-09T14:36:20.143 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.142+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=3HZ61N22DTLY type=write size=116196 2024-03-09T14:36:20.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.142+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=B0J0JN3OM11K type=del 2024-03-09T14:36:20.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.146+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=W7PKLHGH3OYC type=del size=1419024 2024-03-09T14:36:20.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.146+0000 7f73315818c0 1 < canceled part key=_multipart_P8R5RJI8LGZ9.2~784020SZDM9F.0 size=1048576 2024-03-09T14:36:20.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.150+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~784020SZDM9F.1 size=1048576 2024-03-09T14:36:20.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.154+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~784020SZDM9F.2 size=969278 2024-03-09T14:36:20.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.158+0000 7f73315818c0 1 < canceled multipart upload key=P8R5RJI8LGZ9 upload=784020SZDM9F tag=27ALAOYF6F3D type=write size=4115006 2024-03-09T14:36:20.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.162+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=XXZ116JGCPQB type=write size=796692 2024-03-09T14:36:20.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.170+0000 7f73315818c0 1 < suggested update operation key=K3JNN1NCKBXN tag=5S28K2XI6E9A type=del size=1749080 2024-03-09T14:36:20.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.174+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=HSUIJF8LUMJR type=del size=3608046 2024-03-09T14:36:20.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.178+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=JO06M1WU20LH type=del 2024-03-09T14:36:20.186 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.182+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~G7EM3AORKGG6.0 size=1048576 2024-03-09T14:36:20.189 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.186+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~G7EM3AORKGG6.1 size=1048576 2024-03-09T14:36:20.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.190+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~G7EM3AORKGG6.2 size=1038414 2024-03-09T14:36:20.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.190+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=G7EM3AORKGG6 tag=I3G13U38IBNW type=write size=4184142 2024-03-09T14:36:20.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.190+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~RYA60UBEB4RU.0 size=458889 2024-03-09T14:36:20.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.198+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=RYA60UBEB4RU tag=SNXEN2CC9GYG type=write size=1507465 2024-03-09T14:36:20.207 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.202+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~XYQRXILYRDGY.0 size=1048576 2024-03-09T14:36:20.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.206+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~XYQRXILYRDGY.1 size=1048576 2024-03-09T14:36:20.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.210+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~XYQRXILYRDGY.2 size=895682 2024-03-09T14:36:20.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.210+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=XYQRXILYRDGY tag=ZX8RN1FSLRAW type=write size=4041410 2024-03-09T14:36:20.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.214+0000 7f73315818c0 1 < completed write operation key=S302JFYEDEL7 tag=VPLZPFSYSPBC type=write size=417585 2024-03-09T14:36:20.222 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.218+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~24AO1NYUKR4D.0 size=1048576 2024-03-09T14:36:20.224 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.222+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~24AO1NYUKR4D.1 size=1048576 2024-03-09T14:36:20.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.222+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~24AO1NYUKR4D.2 size=299979 2024-03-09T14:36:20.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.222+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=24AO1NYUKR4D tag=BHBI5XLMDQAW type=write size=3445707 2024-03-09T14:36:20.230 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.226+0000 7f73315818c0 -1 < failed to complete operation key=OU0730HGI176 tag=GG0ERKOJGLTN type=write size=522419: (22) Invalid argument 2024-03-09T14:36:20.236 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.234+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~3P83OWNXNA59.0 size=222207 2024-03-09T14:36:20.236 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.234+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=3P83OWNXNA59 tag=21MABBV9H0KT type=write size=1270783 2024-03-09T14:36:20.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.238+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~3TBMI8VJPTC1.0 size=1048576 2024-03-09T14:36:20.242 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.238+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~3TBMI8VJPTC1.1 size=1048576 2024-03-09T14:36:20.245 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.242+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~3TBMI8VJPTC1.2 size=872540 2024-03-09T14:36:20.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.246+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=3TBMI8VJPTC1 tag=W86X58JXLUQC type=write size=4018268 2024-03-09T14:36:20.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.250+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=9HQG7UF2ENSC type=del size=322711 2024-03-09T14:36:20.258 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.254+0000 7f73315818c0 1 < completed write operation key=MD7FXGYXB823 tag=R41LDBOLCT88 type=write size=916713 2024-03-09T14:36:20.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.258+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=2KTQ7IZQQAM4 type=del size=1288597 2024-03-09T14:36:20.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.262+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=PY84BOYFTW9W type=del 2024-03-09T14:36:20.271 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.266+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~KFHBXNKWHQ8R.0 size=633529 2024-03-09T14:36:20.271 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.266+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=KFHBXNKWHQ8R tag=3X2KO71ZED97 type=write size=1682105 2024-03-09T14:36:20.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.270+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~FGR1H5U7BWEH.0 size=158322 2024-03-09T14:36:20.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.278+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=FGR1H5U7BWEH tag=6XWD13S9A3IZ type=write size=1206898 2024-03-09T14:36:20.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.278+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=9GT55YL0GXU8 type=write size=267683 2024-03-09T14:36:20.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.282+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=2J7MJ67L6KFB type=del 2024-03-09T14:36:20.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.286+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=YSFG4ATQKB8G type=del size=2704283 2024-03-09T14:36:20.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.294+0000 7f73315818c0 1 < suggested update operation key=K3JNN1NCKBXN tag=M13ZYDPPOYOM type=del size=757100 2024-03-09T14:36:20.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.298+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=8PCMO5GAARBO type=del size=3767378 2024-03-09T14:36:20.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.298+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=FWI537V0U4H6 type=del 2024-03-09T14:36:20.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.306+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~O6NETVZKBRPQ.0 size=1048576 2024-03-09T14:36:20.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.310+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~O6NETVZKBRPQ.1 size=600262 2024-03-09T14:36:20.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.310+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=O6NETVZKBRPQ tag=9BHUMVAMS75W type=write size=2697414 2024-03-09T14:36:20.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.310+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~HV2PM0V34136.0 size=306833 2024-03-09T14:36:20.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.318+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=HV2PM0V34136 tag=RHSKOQ41UING type=write size=1355409 2024-03-09T14:36:20.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.322+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~KBYBE8HQBF2M.0 size=565132 2024-03-09T14:36:20.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.322+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=KBYBE8HQBF2M tag=FXS9R5QND6B1 type=write size=1613708 2024-03-09T14:36:20.331 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.326+0000 7f73315818c0 1 < completed write operation key=162AOMHO2BQL tag=3HZ61N22DTLY type=write size=116196 2024-03-09T14:36:20.337 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.334+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~CISE542RVUAP.0 size=1048576 2024-03-09T14:36:20.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.334+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~CISE542RVUAP.1 size=351879 2024-03-09T14:36:20.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.334+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=CISE542RVUAP tag=ZCOZM1XTKDVN type=write size=2449031 2024-03-09T14:36:20.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.338+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=W7PKLHGH3OYC type=del 2024-03-09T14:36:20.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.346+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~X84O1VLA2GR9.0 size=1048576 2024-03-09T14:36:20.351 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.346+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~X84O1VLA2GR9.1 size=1048576 2024-03-09T14:36:20.354 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.350+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~X84O1VLA2GR9.2 size=590616 2024-03-09T14:36:20.354 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.350+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=X84O1VLA2GR9 tag=TONVMQ1VJKYX type=write size=3736344 2024-03-09T14:36:20.357 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.354+0000 7f73315818c0 1 < completed write operation key=IA0AB7UVLFEJ tag=XXZ116JGCPQB type=write size=796692 2024-03-09T14:36:20.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.366+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~N3JZ8O8YS05X.0 size=89958 2024-03-09T14:36:20.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.366+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=N3JZ8O8YS05X tag=3XM9YDPZASV2 type=write size=1138534 2024-03-09T14:36:20.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.366+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=HSUIJF8LUMJR type=del 2024-03-09T14:36:20.375 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.370+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=6HNBZC17QSS8 type=del size=874101 2024-03-09T14:36:20.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.374+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~G7EM3AORKGG6.0 size=1048576 2024-03-09T14:36:20.381 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.378+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~G7EM3AORKGG6.1 size=1048576 2024-03-09T14:36:20.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.382+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~G7EM3AORKGG6.2 size=1038414 2024-03-09T14:36:20.389 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.386+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=G7EM3AORKGG6 tag=I3G13U38IBNW type=write size=4184142 2024-03-09T14:36:20.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.390+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=7CK323Z7HNMK type=del size=2480566 2024-03-09T14:36:20.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.394+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~XYQRXILYRDGY.0 size=1048576 2024-03-09T14:36:20.398 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.394+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~XYQRXILYRDGY.1 size=1048576 2024-03-09T14:36:20.400 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.398+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~XYQRXILYRDGY.2 size=895682 2024-03-09T14:36:20.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.402+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=XYQRXILYRDGY tag=ZX8RN1FSLRAW type=write size=4041410 2024-03-09T14:36:20.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.406+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=SDHNRT13I915 type=write size=17396 2024-03-09T14:36:20.414 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.410+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~24AO1NYUKR4D.0 size=1048576 2024-03-09T14:36:20.417 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.414+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~24AO1NYUKR4D.1 size=1048576 2024-03-09T14:36:20.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.418+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~24AO1NYUKR4D.2 size=299979 2024-03-09T14:36:20.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.422+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=24AO1NYUKR4D tag=BHBI5XLMDQAW type=write size=3445707 2024-03-09T14:36:20.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.426+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~XOTU3K0RNS2W.0 size=1048576 2024-03-09T14:36:20.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.430+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~XOTU3K0RNS2W.1 size=1048576 2024-03-09T14:36:20.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.430+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~XOTU3K0RNS2W.2 size=182769 2024-03-09T14:36:20.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.430+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=XOTU3K0RNS2W tag=4FLKUELVL2TM type=write size=3328497 2024-03-09T14:36:20.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.434+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~3P83OWNXNA59.0 size=222207 2024-03-09T14:36:20.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.442+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=3P83OWNXNA59 tag=21MABBV9H0KT type=write size=1270783 2024-03-09T14:36:20.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.442+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=Q5P5RBIG414F type=del size=4035351 2024-03-09T14:36:20.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.446+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=9HQG7UF2ENSC type=del 2024-03-09T14:36:20.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.450+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=18F02G3L1LLD type=write size=16297 2024-03-09T14:36:20.460 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.458+0000 7f73315818c0 1 < suggested update operation key=K3JNN1NCKBXN tag=2KTQ7IZQQAM4 type=del size=1288597 2024-03-09T14:36:20.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.462+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~KGNHJXX50L3J.0 size=1048576 2024-03-09T14:36:20.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.466+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~KGNHJXX50L3J.1 size=1048576 2024-03-09T14:36:20.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.470+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~KGNHJXX50L3J.2 size=410476 2024-03-09T14:36:20.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.470+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=KGNHJXX50L3J tag=VLCD2IM8ORL6 type=write size=3556204 2024-03-09T14:36:20.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.470+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~KFHBXNKWHQ8R.0 size=633529 2024-03-09T14:36:20.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.478+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=KFHBXNKWHQ8R tag=3X2KO71ZED97 type=write size=1682105 2024-03-09T14:36:20.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.478+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=JEZ2UPRTL6U4 type=del size=457847 2024-03-09T14:36:20.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.482+0000 7f73315818c0 1 < canceled operation key=0LP8KKU53X0A tag=9GT55YL0GXU8 type=write size=267683 2024-03-09T14:36:20.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.486+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=5UWFEZNDM78V type=write size=1033928 2024-03-09T14:36:20.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.490+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=YSFG4ATQKB8G type=del 2024-03-09T14:36:20.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.494+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=R9KVHC5OYF78 type=write size=179963 2024-03-09T14:36:20.500 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.498+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=8PCMO5GAARBO type=del 2024-03-09T14:36:20.503 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.498+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=Z28PL24PZGO1 type=write size=89822 2024-03-09T14:36:20.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.502+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~O6NETVZKBRPQ.0 size=1048576 2024-03-09T14:36:20.509 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.506+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~O6NETVZKBRPQ.1 size=600262 2024-03-09T14:36:20.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.510+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=O6NETVZKBRPQ tag=9BHUMVAMS75W type=write size=2697414 2024-03-09T14:36:20.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.518+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~PMT8BHN2J62U.0 size=1048576 2024-03-09T14:36:20.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.518+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~PMT8BHN2J62U.1 size=867320 2024-03-09T14:36:20.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.518+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=PMT8BHN2J62U tag=HJNTIKNLHWX1 type=write size=2964472 2024-03-09T14:36:20.526 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.522+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~KBYBE8HQBF2M.0 size=565132 2024-03-09T14:36:20.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.530+0000 7f73315818c0 1 < completed multipart upload key=0IB00CZ92VV4 upload=KBYBE8HQBF2M tag=FXS9R5QND6B1 type=write size=1613708 2024-03-09T14:36:20.535 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.530+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=3OCNB0YYN6DK type=write size=958580 2024-03-09T14:36:20.538 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.534+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~CISE542RVUAP.0 size=1048576 2024-03-09T14:36:20.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.538+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~CISE542RVUAP.1 size=351879 2024-03-09T14:36:20.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.542+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=CISE542RVUAP tag=ZCOZM1XTKDVN type=write size=2449031 2024-03-09T14:36:20.549 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.546+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=EDS1CQXWJTF2 type=del size=2620268 2024-03-09T14:36:20.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.550+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~X84O1VLA2GR9.0 size=1048576 2024-03-09T14:36:20.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.550+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~X84O1VLA2GR9.1 size=1048576 2024-03-09T14:36:20.557 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.554+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~X84O1VLA2GR9.2 size=590616 2024-03-09T14:36:20.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.558+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=X84O1VLA2GR9 tag=TONVMQ1VJKYX type=write size=3736344 2024-03-09T14:36:20.564 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.562+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=JX7SJDG99N4W type=del size=346552 2024-03-09T14:36:20.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.566+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~N3JZ8O8YS05X.0 size=89958 2024-03-09T14:36:20.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.570+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=N3JZ8O8YS05X tag=3XM9YDPZASV2 type=write size=1138534 2024-03-09T14:36:20.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.574+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=7WOQYPIVS27A type=write size=554531 2024-03-09T14:36:20.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.578+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=6HNBZC17QSS8 type=del 2024-03-09T14:36:20.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.582+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~3ARJRACVPZC5.0 size=283182 2024-03-09T14:36:20.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.582+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=3ARJRACVPZC5 tag=RWXT6YSIOGHA type=write size=1331758 2024-03-09T14:36:20.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.586+0000 7f73315818c0 1 < canceled operation key=DQOY1JJ1YQ2W tag=7CK323Z7HNMK type=del size=2480566 2024-03-09T14:36:20.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.590+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~ACM6891AMKQ4.0 size=1048576 2024-03-09T14:36:20.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.594+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~ACM6891AMKQ4.1 size=53759 2024-03-09T14:36:20.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.594+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=ACM6891AMKQ4 tag=I02PXAU3HZK9 type=write size=2150911 2024-03-09T14:36:20.600 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.598+0000 7f73315818c0 1 < canceled operation key=1NYNL8M5DVER tag=SDHNRT13I915 type=write size=17396 2024-03-09T14:36:20.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.598+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=DT7AK1HE42K2 type=del size=1505982 2024-03-09T14:36:20.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.602+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~XOTU3K0RNS2W.0 size=1048576 2024-03-09T14:36:20.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.602+0000 7f73315818c0 1 < canceled part key=_multipart_LZC7NPF0GRO7.2~XOTU3K0RNS2W.1 size=1048576 2024-03-09T14:36:20.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.606+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~XOTU3K0RNS2W.2 size=182769 2024-03-09T14:36:20.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.610+0000 7f73315818c0 1 < canceled multipart upload key=LZC7NPF0GRO7 upload=XOTU3K0RNS2W tag=4FLKUELVL2TM type=write size=3328497 2024-03-09T14:36:20.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.614+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=8JO7GXBMX3Q0 type=del size=4063074 2024-03-09T14:36:20.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.614+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=Q5P5RBIG414F type=del 2024-03-09T14:36:20.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.618+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=9H7NCEGAK4YA type=del size=408921 2024-03-09T14:36:20.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.622+0000 7f73315818c0 1 < completed write operation key=162AOMHO2BQL tag=18F02G3L1LLD type=write size=16297 2024-03-09T14:36:20.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.626+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~KP7ZPO0VYY79.0 size=1048576 2024-03-09T14:36:20.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.630+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~KP7ZPO0VYY79.1 size=292887 2024-03-09T14:36:20.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.630+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=KP7ZPO0VYY79 tag=QUCGRKAC60FZ type=write size=2390039 2024-03-09T14:36:20.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.630+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~KGNHJXX50L3J.0 size=1048576 2024-03-09T14:36:20.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.634+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~KGNHJXX50L3J.1 size=1048576 2024-03-09T14:36:20.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.634+0000 7f73315818c0 1 < canceled part key=_multipart_CVAWRZMAX997.2~KGNHJXX50L3J.2 size=410476 2024-03-09T14:36:20.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.638+0000 7f73315818c0 1 < canceled multipart upload key=CVAWRZMAX997 upload=KGNHJXX50L3J tag=VLCD2IM8ORL6 type=write size=3556204 2024-03-09T14:36:20.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.642+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=424MX3W40U89 type=write size=834464 2024-03-09T14:36:20.652 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.650+0000 7f73315818c0 1 < suggested remove operation key=K3JNN1NCKBXN tag=JEZ2UPRTL6U4 type=del 2024-03-09T14:36:20.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.658+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~JDJMFIEZLVY4.0 size=1048576 2024-03-09T14:36:20.664 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.662+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~JDJMFIEZLVY4.1 size=1048576 2024-03-09T14:36:20.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.662+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~JDJMFIEZLVY4.2 size=503989 2024-03-09T14:36:20.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.662+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=JDJMFIEZLVY4 tag=J7OA11I41CI5 type=write size=3649717 2024-03-09T14:36:20.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.666+0000 7f73315818c0 1 < suggested remove operation key=3ADG4GWVARC1 tag=5UWFEZNDM78V type=write 2024-03-09T14:36:20.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.674+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~AHCWJ45FM7BD.0 size=1048576 2024-03-09T14:36:20.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.674+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~AHCWJ45FM7BD.1 size=1048576 2024-03-09T14:36:20.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.678+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~AHCWJ45FM7BD.2 size=590863 2024-03-09T14:36:20.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.678+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=AHCWJ45FM7BD tag=P9XFZTCMTG8Q type=write size=3736591 2024-03-09T14:36:20.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.682+0000 7f73315818c0 1 < completed write operation key=OU0730HGI176 tag=R9KVHC5OYF78 type=write size=179963 2024-03-09T14:36:20.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.682+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=A0TAUIC9YMQZ type=del size=1547136 2024-03-09T14:36:20.689 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.686+0000 7f73315818c0 1 < completed write operation key=1NYNL8M5DVER tag=Z28PL24PZGO1 type=write size=89822 2024-03-09T14:36:20.692 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.690+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=9D0Y4HSLGJ6T type=write size=136026 2024-03-09T14:36:20.693 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.690+0000 7f73315818c0 1 < canceled part key=_multipart_DQOY1JJ1YQ2W.2~PMT8BHN2J62U.0 size=1048576 2024-03-09T14:36:20.695 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.694+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~PMT8BHN2J62U.1 size=867320 2024-03-09T14:36:20.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.698+0000 7f73315818c0 1 < canceled multipart upload key=DQOY1JJ1YQ2W upload=PMT8BHN2J62U tag=HJNTIKNLHWX1 type=write size=2964472 2024-03-09T14:36:20.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.702+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~ORZQZUU0M473.0 size=1048576 2024-03-09T14:36:20.706 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.702+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~ORZQZUU0M473.1 size=906693 2024-03-09T14:36:20.707 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.702+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=ORZQZUU0M473 tag=TKLH93U0F62M type=write size=3003845 2024-03-09T14:36:20.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.710+0000 7f73315818c0 1 < suggested update operation key=MD7FXGYXB823 tag=3OCNB0YYN6DK type=write size=958580 2024-03-09T14:36:20.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.714+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~DHPDFL3U5PRU.0 size=978019 2024-03-09T14:36:20.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.714+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=DHPDFL3U5PRU tag=SAI9WIGZUXJJ type=write size=2026595 2024-03-09T14:36:20.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.714+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=EDS1CQXWJTF2 type=del 2024-03-09T14:36:20.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.718+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=XTF10ES7MSZC type=write size=716211 2024-03-09T14:36:20.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.718+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=JX7SJDG99N4W type=del 2024-03-09T14:36:20.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.726+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~N9MNFA5RGNTM.0 size=312672 2024-03-09T14:36:20.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.726+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=N9MNFA5RGNTM tag=X4VTRXH8D4I4 type=write size=1361248 2024-03-09T14:36:20.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.726+0000 7f73315818c0 1 < canceled operation key=IA0AB7UVLFEJ tag=7WOQYPIVS27A type=write size=554531 2024-03-09T14:36:20.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.730+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=FRMQ23NAQ0F0 type=del size=3213540 2024-03-09T14:36:20.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.730+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~3ARJRACVPZC5.0 size=283182 2024-03-09T14:36:20.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.734+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=3ARJRACVPZC5 tag=RWXT6YSIOGHA type=write size=1331758 2024-03-09T14:36:20.742 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.738+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~JKIDSKBGD0U8.0 size=1048576 2024-03-09T14:36:20.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.742+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~JKIDSKBGD0U8.1 size=1048576 2024-03-09T14:36:20.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.742+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~JKIDSKBGD0U8.2 size=448329 2024-03-09T14:36:20.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.742+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=JKIDSKBGD0U8 tag=SWB8Z0MI44C2 type=write size=3594057 2024-03-09T14:36:20.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.746+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~ACM6891AMKQ4.0 size=1048576 2024-03-09T14:36:20.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.746+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~ACM6891AMKQ4.1 size=53759 2024-03-09T14:36:20.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.750+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=ACM6891AMKQ4 tag=I02PXAU3HZK9 type=write size=2150911 2024-03-09T14:36:20.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.758+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~225NA8CND8AG.0 size=1048576 2024-03-09T14:36:20.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.758+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~225NA8CND8AG.1 size=1048576 2024-03-09T14:36:20.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.762+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~225NA8CND8AG.2 size=424307 2024-03-09T14:36:20.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.762+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=225NA8CND8AG tag=IJ5DPMB8E1KO type=write size=3570035 2024-03-09T14:36:20.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.762+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=DT7AK1HE42K2 type=del 2024-03-09T14:36:20.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.766+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=PLZRT3WQX530 type=del size=2846032 2024-03-09T14:36:20.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.766+0000 7f73315818c0 1 < canceled operation key=PXFZLDSAR1EG tag=8JO7GXBMX3Q0 type=del size=4063074 2024-03-09T14:36:20.775 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.774+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~POAYPTV4FPEY.0 size=1048576 2024-03-09T14:36:20.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.774+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~POAYPTV4FPEY.1 size=1048576 2024-03-09T14:36:20.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.778+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~POAYPTV4FPEY.2 size=491363 2024-03-09T14:36:20.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.778+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=POAYPTV4FPEY tag=7ZSO1T26VSG1 type=write size=3637091 2024-03-09T14:36:20.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.778+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=9H7NCEGAK4YA type=del 2024-03-09T14:36:20.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.782+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=LQNLMVL5GKDV type=del size=344760 2024-03-09T14:36:20.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.786+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~KP7ZPO0VYY79.0 size=1048576 2024-03-09T14:36:20.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.786+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~KP7ZPO0VYY79.1 size=292887 2024-03-09T14:36:20.795 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.790+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=KP7ZPO0VYY79 tag=QUCGRKAC60FZ type=write size=2390039 2024-03-09T14:36:20.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.798+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~4UTB7GO9PFXJ.0 size=698030 2024-03-09T14:36:20.800 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.798+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=4UTB7GO9PFXJ tag=YFO7CTSWUOYK type=write size=1746606 2024-03-09T14:36:20.805 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.802+0000 7f73315818c0 1 < suggested update operation key=MD7FXGYXB823 tag=424MX3W40U89 type=write size=834464 2024-03-09T14:36:20.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.806+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~R7FIRAZGV3LV.0 size=1048576 2024-03-09T14:36:20.812 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.810+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~R7FIRAZGV3LV.1 size=1048576 2024-03-09T14:36:20.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.810+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~R7FIRAZGV3LV.2 size=768358 2024-03-09T14:36:20.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.810+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=R7FIRAZGV3LV tag=FHKNPYBM53IH type=write size=3914086 2024-03-09T14:36:20.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.814+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~JDJMFIEZLVY4.0 size=1048576 2024-03-09T14:36:20.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.818+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~JDJMFIEZLVY4.1 size=1048576 2024-03-09T14:36:20.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.818+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~JDJMFIEZLVY4.2 size=503989 2024-03-09T14:36:20.826 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.822+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=JDJMFIEZLVY4 tag=J7OA11I41CI5 type=write size=3649717 2024-03-09T14:36:20.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.826+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=1IQZXMIM6DAF type=del size=3841318 2024-03-09T14:36:20.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.826+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~AHCWJ45FM7BD.0 size=1048576 2024-03-09T14:36:20.832 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.830+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~AHCWJ45FM7BD.1 size=1048576 2024-03-09T14:36:20.835 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.830+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~AHCWJ45FM7BD.2 size=590863 2024-03-09T14:36:20.840 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.838+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=AHCWJ45FM7BD tag=P9XFZTCMTG8Q type=write size=3736591 2024-03-09T14:36:20.843 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.838+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=53MFOFOW7FJ8 type=del size=1035171 2024-03-09T14:36:20.846 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.842+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=A0TAUIC9YMQZ type=del 2024-03-09T14:36:20.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.850+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~PQVLYSHRVZ1M.0 size=771743 2024-03-09T14:36:20.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.850+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=PQVLYSHRVZ1M tag=5TVHKKFV0JPK type=write size=1820319 2024-03-09T14:36:20.854 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.850+0000 7f73315818c0 1 < completed write operation key=LZC7NPF0GRO7 tag=9D0Y4HSLGJ6T type=write size=136026 2024-03-09T14:36:20.857 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.854+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=A0DKH2CECZ8S type=del size=97609 2024-03-09T14:36:20.860 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.858+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~ORZQZUU0M473.0 size=1048576 2024-03-09T14:36:20.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.858+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~ORZQZUU0M473.1 size=906693 2024-03-09T14:36:20.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.862+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=ORZQZUU0M473 tag=TKLH93U0F62M type=write size=3003845 2024-03-09T14:36:20.871 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.870+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~9OINTPVREMUA.0 size=1048576 2024-03-09T14:36:20.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.870+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~9OINTPVREMUA.1 size=1048576 2024-03-09T14:36:20.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.874+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~9OINTPVREMUA.2 size=29511 2024-03-09T14:36:20.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.874+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=9OINTPVREMUA tag=AMLQIP1A8FCA type=write size=3175239 2024-03-09T14:36:20.879 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.874+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~DHPDFL3U5PRU.0 size=978019 2024-03-09T14:36:20.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.878+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=DHPDFL3U5PRU tag=SAI9WIGZUXJJ type=write size=2026595 2024-03-09T14:36:20.886 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.882+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=1PM36AEDRP8S type=del size=2651815 2024-03-09T14:36:20.890 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.886+0000 7f73315818c0 1 < completed write operation key=M1B6O6N8N84T tag=XTF10ES7MSZC type=write size=716211 2024-03-09T14:36:20.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.890+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=3BMR4SJMXPVN type=del size=271193 2024-03-09T14:36:20.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.894+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~N9MNFA5RGNTM.0 size=312672 2024-03-09T14:36:20.904 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.902+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=N9MNFA5RGNTM tag=X4VTRXH8D4I4 type=write size=1361248 2024-03-09T14:36:20.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.906+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~1L3ZL4QV644F.0 size=1048576 2024-03-09T14:36:20.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.906+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~1L3ZL4QV644F.1 size=1048576 2024-03-09T14:36:20.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.910+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~1L3ZL4QV644F.2 size=496097 2024-03-09T14:36:20.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.910+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=1L3ZL4QV644F tag=JPGROOGVW7TT type=write size=3641825 2024-03-09T14:36:20.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.914+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=FRMQ23NAQ0F0 type=del 2024-03-09T14:36:20.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.918+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=PCHFOH54RB57 type=write size=404406 2024-03-09T14:36:20.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.918+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~JKIDSKBGD0U8.0 size=1048576 2024-03-09T14:36:20.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.922+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~JKIDSKBGD0U8.1 size=1048576 2024-03-09T14:36:20.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.926+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~JKIDSKBGD0U8.2 size=448329 2024-03-09T14:36:20.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.930+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=JKIDSKBGD0U8 tag=SWB8Z0MI44C2 type=write size=3594057 2024-03-09T14:36:20.935 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.930+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=RS5XFCAXAD6V type=del size=127143 2024-03-09T14:36:20.938 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.934+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~225NA8CND8AG.0 size=1048576 2024-03-09T14:36:20.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.938+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~225NA8CND8AG.1 size=1048576 2024-03-09T14:36:20.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.938+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~225NA8CND8AG.2 size=424307 2024-03-09T14:36:20.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.946+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=225NA8CND8AG tag=IJ5DPMB8E1KO type=write size=3570035 2024-03-09T14:36:20.951 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.950+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=A50HX39KERZK type=del size=1612396 2024-03-09T14:36:20.954 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.950+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=PLZRT3WQX530 type=del 2024-03-09T14:36:20.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.954+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=L4W74A4N9SKK type=del size=3158320 2024-03-09T14:36:20.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.958+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~POAYPTV4FPEY.0 size=1048576 2024-03-09T14:36:20.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.958+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~POAYPTV4FPEY.1 size=1048576 2024-03-09T14:36:20.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.962+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~POAYPTV4FPEY.2 size=491363 2024-03-09T14:36:20.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.966+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=POAYPTV4FPEY tag=7ZSO1T26VSG1 type=write size=3637091 2024-03-09T14:36:20.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.970+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=UQD5T1VBFRAX type=del size=2339300 2024-03-09T14:36:20.977 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.974+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=LQNLMVL5GKDV type=del 2024-03-09T14:36:20.980 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.978+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=E7QN20W9MSQF type=del size=2240490 2024-03-09T14:36:20.982 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.978+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~4UTB7GO9PFXJ.0 size=698030 2024-03-09T14:36:20.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.986+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=4UTB7GO9PFXJ tag=YFO7CTSWUOYK type=write size=1746606 2024-03-09T14:36:20.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.986+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=MLGL4TQ4EXV1 type=del size=1247618 2024-03-09T14:36:20.994 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.990+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~R7FIRAZGV3LV.0 size=1048576 2024-03-09T14:36:20.996 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.994+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~R7FIRAZGV3LV.1 size=1048576 2024-03-09T14:36:20.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:20.994+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~R7FIRAZGV3LV.2 size=768358 2024-03-09T14:36:21.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.002+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=R7FIRAZGV3LV tag=FHKNPYBM53IH type=write size=3914086 2024-03-09T14:36:21.010 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.006+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~X7V38KAOC6EV.0 size=465698 2024-03-09T14:36:21.010 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.006+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=X7V38KAOC6EV tag=263MC3CPRPE8 type=write size=1514274 2024-03-09T14:36:21.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.010+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=1IQZXMIM6DAF type=del 2024-03-09T14:36:21.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.014+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=CPN1YYZA2V8C type=del size=255757 2024-03-09T14:36:21.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.014+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=53MFOFOW7FJ8 type=del 2024-03-09T14:36:21.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.022+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~SI6C0RTGT1O4.0 size=1048576 2024-03-09T14:36:21.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.022+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~SI6C0RTGT1O4.1 size=501545 2024-03-09T14:36:21.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.022+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=SI6C0RTGT1O4 tag=LTOLOHF6L4ZU type=write size=2598697 2024-03-09T14:36:21.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.026+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~PQVLYSHRVZ1M.0 size=771743 2024-03-09T14:36:21.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.030+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=PQVLYSHRVZ1M tag=5TVHKKFV0JPK type=write size=1820319 2024-03-09T14:36:21.038 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.034+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~9NMQT1P23QSX.0 size=1048576 2024-03-09T14:36:21.040 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.038+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~9NMQT1P23QSX.1 size=1048576 2024-03-09T14:36:21.043 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.038+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~9NMQT1P23QSX.2 size=707215 2024-03-09T14:36:21.043 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.038+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=9NMQT1P23QSX tag=MWKG3WQM6F69 type=write size=3852943 2024-03-09T14:36:21.046 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.042+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=A0DKH2CECZ8S type=del 2024-03-09T14:36:21.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.050+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~D77WLEK0HN8L.0 size=1048576 2024-03-09T14:36:21.055 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.050+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~D77WLEK0HN8L.1 size=1048576 2024-03-09T14:36:21.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.054+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~D77WLEK0HN8L.2 size=623783 2024-03-09T14:36:21.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.054+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=D77WLEK0HN8L tag=64UQM3KB38HN type=write size=3769511 2024-03-09T14:36:21.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.058+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~9OINTPVREMUA.0 size=1048576 2024-03-09T14:36:21.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.058+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~9OINTPVREMUA.1 size=1048576 2024-03-09T14:36:21.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.062+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~9OINTPVREMUA.2 size=29511 2024-03-09T14:36:21.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.070+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=9OINTPVREMUA tag=AMLQIP1A8FCA type=write size=3175239 2024-03-09T14:36:21.075 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.070+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=4W4Q4NZFH9GL type=del size=626585 2024-03-09T14:36:21.079 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.074+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=1PM36AEDRP8S type=del 2024-03-09T14:36:21.085 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.082+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~2L8ZB759XTR9.0 size=405145 2024-03-09T14:36:21.085 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.082+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=2L8ZB759XTR9 tag=6K1ST5W7XKRC type=write size=1453721 2024-03-09T14:36:21.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.086+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=3BMR4SJMXPVN type=del 2024-03-09T14:36:21.094 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.090+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=P9K58S1RNZ68 type=write size=408397 2024-03-09T14:36:21.095 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.090+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~1L3ZL4QV644F.0 size=1048576 2024-03-09T14:36:21.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.094+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~1L3ZL4QV644F.1 size=1048576 2024-03-09T14:36:21.101 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.098+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~1L3ZL4QV644F.2 size=496097 2024-03-09T14:36:21.106 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.102+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=1L3ZL4QV644F tag=JPGROOGVW7TT type=write size=3641825 2024-03-09T14:36:21.110 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.106+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=O7U1A0T4902J type=del size=178370 2024-03-09T14:36:21.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.110+0000 7f73315818c0 1 < completed write operation key=0LP8KKU53X0A tag=PCHFOH54RB57 type=write size=404406 2024-03-09T14:36:21.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.114+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=OW3WCGHCICK6 type=del size=1115776 2024-03-09T14:36:21.123 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.122+0000 7f73315818c0 1 < suggested remove operation key=4P89277EEYJ3 tag=RS5XFCAXAD6V type=del 2024-03-09T14:36:21.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.122+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=35UGT47RN2A0 type=del size=334350 2024-03-09T14:36:21.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.130+0000 7f73315818c0 1 < suggested update operation key=M1B6O6N8N84T tag=A50HX39KERZK type=del size=1612396 2024-03-09T14:36:21.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.134+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=69NIP7LQM68B type=del size=1905557 2024-03-09T14:36:21.141 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.138+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=L4W74A4N9SKK type=del 2024-03-09T14:36:21.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.142+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=TJPDQZN3JI1V type=del size=4192780 2024-03-09T14:36:21.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.146+0000 7f73315818c0 1 < canceled operation key=4E3QMBHEAEHO tag=UQD5T1VBFRAX type=del size=2339300 2024-03-09T14:36:21.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.150+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~Q2VN8MAKZK6F.0 size=1048576 2024-03-09T14:36:21.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.154+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~Q2VN8MAKZK6F.1 size=1048576 2024-03-09T14:36:21.160 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.158+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~Q2VN8MAKZK6F.2 size=1016643 2024-03-09T14:36:21.160 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.158+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=Q2VN8MAKZK6F tag=Z09GJIGODWJ5 type=write size=4162371 2024-03-09T14:36:21.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.162+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=E7QN20W9MSQF type=del 2024-03-09T14:36:21.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.166+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=NU682O31DN0U type=del size=520140 2024-03-09T14:36:21.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.170+0000 7f73315818c0 1 < suggested update operation key=0LP8KKU53X0A tag=MLGL4TQ4EXV1 type=del size=1247618 2024-03-09T14:36:21.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.174+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=412FO2PFM13P type=del size=116522 2024-03-09T14:36:21.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.178+0000 7f73315818c0 1 < canceled part key=_multipart_K3JNN1NCKBXN.2~X7V38KAOC6EV.0 size=465698 2024-03-09T14:36:21.186 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.182+0000 7f73315818c0 1 < canceled multipart upload key=K3JNN1NCKBXN upload=X7V38KAOC6EV tag=263MC3CPRPE8 type=write size=1514274 2024-03-09T14:36:21.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.190+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~VY9B9VV2ZTSR.0 size=1048576 2024-03-09T14:36:21.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.194+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~VY9B9VV2ZTSR.1 size=1048576 2024-03-09T14:36:21.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.198+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~VY9B9VV2ZTSR.2 size=865049 2024-03-09T14:36:21.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.198+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=VY9B9VV2ZTSR tag=U5SX0VSAM58T type=write size=4010777 2024-03-09T14:36:21.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.202+0000 7f73315818c0 -1 < no bi entry to suggest for operation key=4P89277EEYJ3 tag=CPN1YYZA2V8C type=del size=255757: (2) No such file or directory 2024-03-09T14:36:21.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.206+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=3B0RF2BNVV0I type=del size=4023949 2024-03-09T14:36:21.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.210+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~SI6C0RTGT1O4.0 size=1048576 2024-03-09T14:36:21.217 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.214+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~SI6C0RTGT1O4.1 size=501545 2024-03-09T14:36:21.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.218+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=SI6C0RTGT1O4 tag=LTOLOHF6L4ZU type=write size=2598697 2024-03-09T14:36:21.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.222+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=U93003Q4Z2DN type=del size=1498568 2024-03-09T14:36:21.230 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.226+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~9NMQT1P23QSX.0 size=1048576 2024-03-09T14:36:21.233 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.230+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~9NMQT1P23QSX.1 size=1048576 2024-03-09T14:36:21.236 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.234+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~9NMQT1P23QSX.2 size=707215 2024-03-09T14:36:21.242 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.238+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=9NMQT1P23QSX tag=MWKG3WQM6F69 type=write size=3852943 2024-03-09T14:36:21.245 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.242+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=73GDUPT5IJP2 type=del size=700651 2024-03-09T14:36:21.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.242+0000 7f73315818c0 1 < canceled part key=_multipart_1NYNL8M5DVER.2~D77WLEK0HN8L.0 size=1048576 2024-03-09T14:36:21.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.246+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~D77WLEK0HN8L.1 size=1048576 2024-03-09T14:36:21.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.250+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~D77WLEK0HN8L.2 size=623783 2024-03-09T14:36:21.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.254+0000 7f73315818c0 1 < canceled multipart upload key=1NYNL8M5DVER upload=D77WLEK0HN8L tag=64UQM3KB38HN type=write size=3769511 2024-03-09T14:36:21.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.258+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=KE3UYQFSKBHL type=write size=338222 2024-03-09T14:36:21.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.262+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=4W4Q4NZFH9GL type=del 2024-03-09T14:36:21.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.266+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=5QJ2RHT0QJCA type=del size=1753215 2024-03-09T14:36:21.272 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.270+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~2L8ZB759XTR9.0 size=405145 2024-03-09T14:36:21.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.274+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=2L8ZB759XTR9 tag=6K1ST5W7XKRC type=write size=1453721 2024-03-09T14:36:21.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.278+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=R12ZESF8359I type=write size=595470 2024-03-09T14:36:21.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.282+0000 7f73315818c0 1 < completed write operation key=IA0AB7UVLFEJ tag=P9K58S1RNZ68 type=write size=408397 2024-03-09T14:36:21.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.286+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=P4K7KXUAVEDV type=write size=288559 2024-03-09T14:36:21.291 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.290+0000 7f73315818c0 1 < canceled operation key=8O5OP8C5NX12 tag=O7U1A0T4902J type=del size=178370 2024-03-09T14:36:21.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.294+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~NWWQPTK6R3Y0.0 size=1048576 2024-03-09T14:36:21.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.298+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~NWWQPTK6R3Y0.1 size=1048576 2024-03-09T14:36:21.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.298+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~NWWQPTK6R3Y0.2 size=245448 2024-03-09T14:36:21.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.298+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=NWWQPTK6R3Y0 tag=NWQ0E8BOGHRI type=write size=3391176 2024-03-09T14:36:21.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.306+0000 7f73315818c0 1 < suggested remove operation key=LZC7NPF0GRO7 tag=OW3WCGHCICK6 type=del 2024-03-09T14:36:21.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.310+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=QK6XEG4FZQUU type=del size=1318093 2024-03-09T14:36:21.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.314+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=35UGT47RN2A0 type=del 2024-03-09T14:36:21.324 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.322+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~E4TRZH2U8L4N.0 size=1048576 2024-03-09T14:36:21.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.322+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~E4TRZH2U8L4N.1 size=1048576 2024-03-09T14:36:21.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.326+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~E4TRZH2U8L4N.2 size=339291 2024-03-09T14:36:21.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.326+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=E4TRZH2U8L4N tag=ASKJ7H02PUH6 type=write size=3485019 2024-03-09T14:36:21.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.330+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=69NIP7LQM68B type=del 2024-03-09T14:36:21.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.334+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=28EPMT810FPW type=del size=2068769 2024-03-09T14:36:21.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.338+0000 7f73315818c0 -1 < failed to complete operation key=0LP8KKU53X0A tag=TJPDQZN3JI1V type=del size=4192780: (22) Invalid argument 2024-03-09T14:36:21.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.342+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~SRRNZO2MH5AI.0 size=1048576 2024-03-09T14:36:21.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.346+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~SRRNZO2MH5AI.1 size=1048576 2024-03-09T14:36:21.351 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.350+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~SRRNZO2MH5AI.2 size=317854 2024-03-09T14:36:21.351 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.350+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=SRRNZO2MH5AI tag=VBH00D3PRCYX type=write size=3463582 2024-03-09T14:36:21.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.350+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~Q2VN8MAKZK6F.0 size=1048576 2024-03-09T14:36:21.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.354+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~Q2VN8MAKZK6F.1 size=1048576 2024-03-09T14:36:21.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.358+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~Q2VN8MAKZK6F.2 size=1016643 2024-03-09T14:36:21.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.362+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=Q2VN8MAKZK6F tag=Z09GJIGODWJ5 type=write size=4162371 2024-03-09T14:36:21.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.366+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=LXRR5SM8M0X2 type=write size=883902 2024-03-09T14:36:21.375 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.370+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=NU682O31DN0U type=del 2024-03-09T14:36:21.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.374+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=GUG0A5RXRO6X type=del size=3192306 2024-03-09T14:36:21.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.378+0000 7f73315818c0 -1 < failed to complete operation key=LZC7NPF0GRO7 tag=412FO2PFM13P type=del size=116522: (22) Invalid argument 2024-03-09T14:36:21.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.382+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=SYCHZLL2FY31 type=del size=1345935 2024-03-09T14:36:21.389 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.386+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~VY9B9VV2ZTSR.0 size=1048576 2024-03-09T14:36:21.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.390+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~VY9B9VV2ZTSR.1 size=1048576 2024-03-09T14:36:21.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.390+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~VY9B9VV2ZTSR.2 size=865049 2024-03-09T14:36:21.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.398+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=VY9B9VV2ZTSR tag=U5SX0VSAM58T type=write size=4010777 2024-03-09T14:36:21.404 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.402+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=Y5BKFYRZ02JH type=write size=668801 2024-03-09T14:36:21.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.406+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=3B0RF2BNVV0I type=del 2024-03-09T14:36:21.414 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.410+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~H2XDNN41ECQJ.0 size=57298 2024-03-09T14:36:21.414 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.410+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=H2XDNN41ECQJ tag=NBFP9915PSFP type=write size=1105874 2024-03-09T14:36:21.418 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.414+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=U93003Q4Z2DN type=del 2024-03-09T14:36:21.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.422+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~XYOQ18PXW8FU.0 size=1048576 2024-03-09T14:36:21.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.422+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~XYOQ18PXW8FU.1 size=1048576 2024-03-09T14:36:21.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.426+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~XYOQ18PXW8FU.2 size=483315 2024-03-09T14:36:21.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.426+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=XYOQ18PXW8FU tag=0MWERWHE1KN2 type=write size=3629043 2024-03-09T14:36:21.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.430+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=73GDUPT5IJP2 type=del 2024-03-09T14:36:21.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.434+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=O59SZHBK6YN0 type=del size=2778692 2024-03-09T14:36:21.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.438+0000 7f73315818c0 1 < completed write operation key=PXFZLDSAR1EG tag=KE3UYQFSKBHL type=write size=338222 2024-03-09T14:36:21.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.442+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~S5QTBC7ZFS09.0 size=173604 2024-03-09T14:36:21.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.442+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=S5QTBC7ZFS09 tag=2JQALYGSON9M type=write size=1222180 2024-03-09T14:36:21.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.446+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=5QJ2RHT0QJCA type=del 2024-03-09T14:36:21.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.454+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~3MQCHMQL4K3S.0 size=124738 2024-03-09T14:36:21.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.454+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=3MQCHMQL4K3S tag=PVIE13QRYWKE type=write size=1173314 2024-03-09T14:36:21.460 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.458+0000 7f73315818c0 1 < completed write operation key=0LP8KKU53X0A tag=R12ZESF8359I type=write size=595470 2024-03-09T14:36:21.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.462+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~PKZPBVU36XX3.0 size=1048576 2024-03-09T14:36:21.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.466+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~PKZPBVU36XX3.1 size=1048576 2024-03-09T14:36:21.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.470+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~PKZPBVU36XX3.2 size=25366 2024-03-09T14:36:21.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.470+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=PKZPBVU36XX3 tag=WQV30A90BM6I type=write size=3171094 2024-03-09T14:36:21.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.470+0000 7f73315818c0 1 < canceled operation key=9B0RAHZZGLYU tag=P4K7KXUAVEDV type=write size=288559 2024-03-09T14:36:21.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.474+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=RP7P6M85NLC6 type=del size=2993947 2024-03-09T14:36:21.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.478+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~NWWQPTK6R3Y0.0 size=1048576 2024-03-09T14:36:21.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.478+0000 7f73315818c0 1 < canceled part key=_multipart_P8R5RJI8LGZ9.2~NWWQPTK6R3Y0.1 size=1048576 2024-03-09T14:36:21.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.482+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~NWWQPTK6R3Y0.2 size=245448 2024-03-09T14:36:21.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.486+0000 7f73315818c0 1 < canceled multipart upload key=P8R5RJI8LGZ9 upload=NWWQPTK6R3Y0 tag=NWQ0E8BOGHRI type=write size=3391176 2024-03-09T14:36:21.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.494+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~8VCUAOPU123T.0 size=1048576 2024-03-09T14:36:21.499 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.494+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~8VCUAOPU123T.1 size=1048576 2024-03-09T14:36:21.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.498+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~8VCUAOPU123T.2 size=744067 2024-03-09T14:36:21.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.498+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=8VCUAOPU123T tag=M9HO9U059UDV type=write size=3889795 2024-03-09T14:36:21.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.502+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=QK6XEG4FZQUU type=del 2024-03-09T14:36:21.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.506+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=LELYDDL50Y42 type=del size=2741717 2024-03-09T14:36:21.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.510+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~E4TRZH2U8L4N.0 size=1048576 2024-03-09T14:36:21.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.514+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~E4TRZH2U8L4N.1 size=1048576 2024-03-09T14:36:21.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.518+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~E4TRZH2U8L4N.2 size=339291 2024-03-09T14:36:21.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.522+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=E4TRZH2U8L4N tag=ASKJ7H02PUH6 type=write size=3485019 2024-03-09T14:36:21.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.530+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=3KSFYSQ9B3KX type=del size=2859710 2024-03-09T14:36:21.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.534+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=28EPMT810FPW type=del 2024-03-09T14:36:21.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.542+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~O9U01J0BA30B.0 size=787331 2024-03-09T14:36:21.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.542+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=O9U01J0BA30B tag=IFITVN4XJ2V5 type=write size=1835907 2024-03-09T14:36:21.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.546+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~SRRNZO2MH5AI.0 size=1048576 2024-03-09T14:36:21.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.546+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~SRRNZO2MH5AI.1 size=1048576 2024-03-09T14:36:21.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.550+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~SRRNZO2MH5AI.2 size=317854 2024-03-09T14:36:21.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.554+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=SRRNZO2MH5AI tag=VBH00D3PRCYX type=write size=3463582 2024-03-09T14:36:21.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.558+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=DFT35XI5MYD8 type=del size=3921458 2024-03-09T14:36:21.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.562+0000 7f73315818c0 1 < completed write operation key=XELG1AWATMVL tag=LXRR5SM8M0X2 type=write size=883902 2024-03-09T14:36:21.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.566+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=DLHI9W6VQV08 type=del size=900277 2024-03-09T14:36:21.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.570+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=GUG0A5RXRO6X type=del 2024-03-09T14:36:21.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.574+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=8LF0EOYXS9E2 type=del size=1589438 2024-03-09T14:36:21.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.578+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=SYCHZLL2FY31 type=del 2024-03-09T14:36:21.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.582+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=9GMP42AF538S type=del size=2843354 2024-03-09T14:36:21.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.586+0000 7f73315818c0 1 < completed write operation key=1NYNL8M5DVER tag=Y5BKFYRZ02JH type=write size=668801 2024-03-09T14:36:21.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.590+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~4HMFUWARHAAZ.0 size=1048576 2024-03-09T14:36:21.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.594+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~4HMFUWARHAAZ.1 size=224646 2024-03-09T14:36:21.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.594+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=4HMFUWARHAAZ tag=2N7NWIUV45UA type=write size=2321798 2024-03-09T14:36:21.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.598+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~H2XDNN41ECQJ.0 size=57298 2024-03-09T14:36:21.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.602+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=H2XDNN41ECQJ tag=NBFP9915PSFP type=write size=1105874 2024-03-09T14:36:21.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.606+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=IU1ODIXL7KC5 type=write size=866242 2024-03-09T14:36:21.615 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.610+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~XYOQ18PXW8FU.0 size=1048576 2024-03-09T14:36:21.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.614+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~XYOQ18PXW8FU.1 size=1048576 2024-03-09T14:36:21.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.618+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~XYOQ18PXW8FU.2 size=483315 2024-03-09T14:36:21.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.622+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=XYOQ18PXW8FU tag=0MWERWHE1KN2 type=write size=3629043 2024-03-09T14:36:21.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.630+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~UDZVEWBEMAIC.0 size=1048576 2024-03-09T14:36:21.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.630+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~UDZVEWBEMAIC.1 size=1048576 2024-03-09T14:36:21.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.634+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~UDZVEWBEMAIC.2 size=131225 2024-03-09T14:36:21.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.634+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=UDZVEWBEMAIC tag=OIWE72NO9GA6 type=write size=3276953 2024-03-09T14:36:21.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.638+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=O59SZHBK6YN0 type=del 2024-03-09T14:36:21.644 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.642+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=I3X51BK893MU type=write size=4087 2024-03-09T14:36:21.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.646+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~S5QTBC7ZFS09.0 size=173604 2024-03-09T14:36:21.653 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.650+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=S5QTBC7ZFS09 tag=2JQALYGSON9M type=write size=1222180 2024-03-09T14:36:21.660 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.658+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~X33GKQFAVVA3.0 size=639835 2024-03-09T14:36:21.660 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.658+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=X33GKQFAVVA3 tag=0OA2VZZJ33M0 type=write size=1688411 2024-03-09T14:36:21.664 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.662+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~3MQCHMQL4K3S.0 size=124738 2024-03-09T14:36:21.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.666+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=3MQCHMQL4K3S tag=PVIE13QRYWKE type=write size=1173314 2024-03-09T14:36:21.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.670+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=FZGL0SVIW1KN type=write size=194685 2024-03-09T14:36:21.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.674+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~PKZPBVU36XX3.0 size=1048576 2024-03-09T14:36:21.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.674+0000 7f73315818c0 1 < canceled part key=_multipart_4P89277EEYJ3.2~PKZPBVU36XX3.1 size=1048576 2024-03-09T14:36:21.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.678+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~PKZPBVU36XX3.2 size=25366 2024-03-09T14:36:21.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.682+0000 7f73315818c0 1 < canceled multipart upload key=4P89277EEYJ3 upload=PKZPBVU36XX3 tag=WQV30A90BM6I type=write size=3171094 2024-03-09T14:36:21.689 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.686+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=RYYA90QOX3NR type=del size=1092352 2024-03-09T14:36:21.693 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.690+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=RP7P6M85NLC6 type=del 2024-03-09T14:36:21.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.694+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=X8WYT9E6RNUL type=write size=355273 2024-03-09T14:36:21.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.698+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~8VCUAOPU123T.0 size=1048576 2024-03-09T14:36:21.703 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.698+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~8VCUAOPU123T.1 size=1048576 2024-03-09T14:36:21.705 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.702+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~8VCUAOPU123T.2 size=744067 2024-03-09T14:36:21.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.710+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=8VCUAOPU123T tag=M9HO9U059UDV type=write size=3889795 2024-03-09T14:36:21.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.714+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=MH7IVZ3NXLK5 type=del size=3515410 2024-03-09T14:36:21.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.718+0000 7f73315818c0 1 < canceled operation key=4P89277EEYJ3 tag=LELYDDL50Y42 type=del size=2741717 2024-03-09T14:36:21.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.718+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=YC4H1KB71UWZ type=del size=2978905 2024-03-09T14:36:21.726 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.722+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=3KSFYSQ9B3KX type=del 2024-03-09T14:36:21.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.726+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=IO6WAKPLAQ92 type=del size=1853435 2024-03-09T14:36:21.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.730+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~O9U01J0BA30B.0 size=787331 2024-03-09T14:36:21.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.734+0000 7f73315818c0 1 < completed multipart upload key=LZC7NPF0GRO7 upload=O9U01J0BA30B tag=IFITVN4XJ2V5 type=write size=1835907 2024-03-09T14:36:21.742 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.738+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=LULWWUCSP90I type=del size=2368980 2024-03-09T14:36:21.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.746+0000 7f73315818c0 1 < suggested remove operation key=9B0RAHZZGLYU tag=DFT35XI5MYD8 type=del 2024-03-09T14:36:21.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.754+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~JRH1FQX4S0BI.0 size=688786 2024-03-09T14:36:21.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.754+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=JRH1FQX4S0BI tag=1OM9XQE1609F type=write size=1737362 2024-03-09T14:36:21.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.758+0000 7f73315818c0 1 < suggested update operation key=M1B6O6N8N84T tag=DLHI9W6VQV08 type=del size=900277 2024-03-09T14:36:21.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.762+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=LCT2T99Z51BS type=write size=671042 2024-03-09T14:36:21.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.766+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=8LF0EOYXS9E2 type=del 2024-03-09T14:36:21.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.770+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=INIRTVEY7PDH type=del size=2038198 2024-03-09T14:36:21.775 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.774+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=9GMP42AF538S type=del 2024-03-09T14:36:21.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.774+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=EUHDXNM62BEE type=del size=4040001 2024-03-09T14:36:21.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.774+0000 7f73315818c0 1 < canceled part key=_multipart_4P89277EEYJ3.2~4HMFUWARHAAZ.0 size=1048576 2024-03-09T14:36:21.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.778+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~4HMFUWARHAAZ.1 size=224646 2024-03-09T14:36:21.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.786+0000 7f73315818c0 1 < canceled multipart upload key=4P89277EEYJ3 upload=4HMFUWARHAAZ tag=2N7NWIUV45UA type=write size=2321798 2024-03-09T14:36:21.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.786+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=3EPLGUKPTN6T type=del size=275758 2024-03-09T14:36:21.794 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.790+0000 7f73315818c0 1 < completed write operation key=4E3QMBHEAEHO tag=IU1ODIXL7KC5 type=write size=866242 2024-03-09T14:36:21.798 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.794+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=HTF38711XVP4 type=write size=564660 2024-03-09T14:36:21.801 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.798+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~UDZVEWBEMAIC.0 size=1048576 2024-03-09T14:36:21.801 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.798+0000 7f73315818c0 1 < canceled part key=_multipart_162AOMHO2BQL.2~UDZVEWBEMAIC.1 size=1048576 2024-03-09T14:36:21.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.802+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~UDZVEWBEMAIC.2 size=131225 2024-03-09T14:36:21.809 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.806+0000 7f73315818c0 1 < canceled multipart upload key=162AOMHO2BQL upload=UDZVEWBEMAIC tag=OIWE72NO9GA6 type=write size=3276953 2024-03-09T14:36:21.816 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.814+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~BCY10618SF6G.0 size=1048576 2024-03-09T14:36:21.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.814+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~BCY10618SF6G.1 size=1048576 2024-03-09T14:36:21.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.818+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~BCY10618SF6G.2 size=394876 2024-03-09T14:36:21.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.818+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=BCY10618SF6G tag=F2LMWQUQYQX1 type=write size=3540604 2024-03-09T14:36:21.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.822+0000 7f73315818c0 1 < completed write operation key=S302JFYEDEL7 tag=I3X51BK893MU type=write size=4087 2024-03-09T14:36:21.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.826+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=BESO1RG4L8SC type=write size=901180 2024-03-09T14:36:21.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.826+0000 7f73315818c0 1 < canceled part key=_multipart_162AOMHO2BQL.2~X33GKQFAVVA3.0 size=639835 2024-03-09T14:36:21.834 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.830+0000 7f73315818c0 1 < canceled multipart upload key=162AOMHO2BQL upload=X33GKQFAVVA3 tag=0OA2VZZJ33M0 type=write size=1688411 2024-03-09T14:36:21.841 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.838+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~94RHVPMX05EG.0 size=1048576 2024-03-09T14:36:21.843 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.838+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~94RHVPMX05EG.1 size=847450 2024-03-09T14:36:21.843 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.842+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=94RHVPMX05EG tag=SHC2OPU6UPBH type=write size=2944602 2024-03-09T14:36:21.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.846+0000 7f73315818c0 1 < suggested remove operation key=MSMD3ANK4T7J tag=FZGL0SVIW1KN type=write 2024-03-09T14:36:21.854 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.850+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=2OE4KRI4QWW0 type=del size=1777680 2024-03-09T14:36:21.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.858+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=RYYA90QOX3NR type=del 2024-03-09T14:36:21.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.866+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~3MWMFOXFDN61.0 size=1048576 2024-03-09T14:36:21.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.870+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~3MWMFOXFDN61.1 size=21048 2024-03-09T14:36:21.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.870+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=3MWMFOXFDN61 tag=WNLRVP6IWVE1 type=write size=2118200 2024-03-09T14:36:21.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.874+0000 7f73315818c0 1 < suggested remove operation key=FYFVW9R0M1PX tag=X8WYT9E6RNUL type=write 2024-03-09T14:36:21.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.882+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~3S9Z8VF5W2LS.0 size=1048576 2024-03-09T14:36:21.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.886+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~3S9Z8VF5W2LS.1 size=766006 2024-03-09T14:36:21.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.886+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=3S9Z8VF5W2LS tag=22YYYDTE1FQA type=write size=2863158 2024-03-09T14:36:21.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.886+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=MH7IVZ3NXLK5 type=del 2024-03-09T14:36:21.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.890+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=FO7LV3TYIVUT type=write size=58804 2024-03-09T14:36:21.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.898+0000 7f73315818c0 1 < suggested update operation key=CVAWRZMAX997 tag=YC4H1KB71UWZ type=del size=2978905 2024-03-09T14:36:21.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.902+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=A3EZRHFRHU5A type=del size=1612053 2024-03-09T14:36:21.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.906+0000 7f73315818c0 1 < canceled operation key=PXFZLDSAR1EG tag=IO6WAKPLAQ92 type=del size=1853435 2024-03-09T14:36:21.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.910+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=U0MDJAI02WBP type=del size=3884330 2024-03-09T14:36:21.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.914+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=LULWWUCSP90I type=del 2024-03-09T14:36:21.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.918+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~OUHGEWLFCRYT.0 size=1039812 2024-03-09T14:36:21.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.918+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=OUHGEWLFCRYT tag=FH07V2OKG2H2 type=write size=2088388 2024-03-09T14:36:21.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.922+0000 7f73315818c0 1 < canceled part key=_multipart_9B0RAHZZGLYU.2~JRH1FQX4S0BI.0 size=688786 2024-03-09T14:36:21.928 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.926+0000 7f73315818c0 1 < canceled multipart upload key=9B0RAHZZGLYU upload=JRH1FQX4S0BI tag=1OM9XQE1609F type=write size=1737362 2024-03-09T14:36:21.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.930+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=28VCKTU2MBMC type=write size=758210 2024-03-09T14:36:21.938 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.934+0000 7f73315818c0 1 < suggested remove operation key=9B0RAHZZGLYU tag=LCT2T99Z51BS type=write 2024-03-09T14:36:21.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.942+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~RUQWI57HBOMZ.0 size=191669 2024-03-09T14:36:21.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.942+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=RUQWI57HBOMZ tag=SZ71JCMHYY3P type=write size=1240245 2024-03-09T14:36:21.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.946+0000 7f73315818c0 1 < canceled operation key=DQOY1JJ1YQ2W tag=INIRTVEY7PDH type=del size=2038198 2024-03-09T14:36:21.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.950+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~XNJJ8LG3A4SQ.0 size=1048576 2024-03-09T14:36:21.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.954+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~XNJJ8LG3A4SQ.1 size=1048576 2024-03-09T14:36:21.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.954+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~XNJJ8LG3A4SQ.2 size=557838 2024-03-09T14:36:21.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.954+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=XNJJ8LG3A4SQ tag=3B3713JWGS1W type=write size=3703566 2024-03-09T14:36:21.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.958+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=EUHDXNM62BEE type=del 2024-03-09T14:36:21.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.962+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=YEHPFO8UOZXU type=del size=690514 2024-03-09T14:36:21.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.966+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=3EPLGUKPTN6T type=del 2024-03-09T14:36:21.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.970+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=AA0PGAL1JOQC type=del size=1855939 2024-03-09T14:36:21.977 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.974+0000 7f73315818c0 1 < completed write operation key=0WGGMHTEF2LB tag=HTF38711XVP4 type=write size=564660 2024-03-09T14:36:21.980 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.978+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=AQFSDCAZIIXU type=del size=703809 2024-03-09T14:36:21.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.982+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~BCY10618SF6G.0 size=1048576 2024-03-09T14:36:21.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.982+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~BCY10618SF6G.1 size=1048576 2024-03-09T14:36:21.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.986+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~BCY10618SF6G.2 size=394876 2024-03-09T14:36:21.995 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.994+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=BCY10618SF6G tag=F2LMWQUQYQX1 type=write size=3540604 2024-03-09T14:36:21.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.994+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=63LGHWHV5HE6 type=del size=2087746 2024-03-09T14:36:22.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:21.998+0000 7f73315818c0 1 < completed write operation key=OU0730HGI176 tag=BESO1RG4L8SC type=write size=901180 2024-03-09T14:36:22.010 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.006+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~MN4SMLFZW896.0 size=1048576 2024-03-09T14:36:22.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.010+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~MN4SMLFZW896.1 size=700759 2024-03-09T14:36:22.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.010+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=MN4SMLFZW896 tag=JV7GT2M35PBF type=write size=2797911 2024-03-09T14:36:22.016 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.014+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~94RHVPMX05EG.0 size=1048576 2024-03-09T14:36:22.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.014+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~94RHVPMX05EG.1 size=847450 2024-03-09T14:36:22.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.022+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=94RHVPMX05EG tag=SHC2OPU6UPBH type=write size=2944602 2024-03-09T14:36:22.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.026+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=TH8DE0EE38QJ type=del size=2850892 2024-03-09T14:36:22.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.030+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=2OE4KRI4QWW0 type=del 2024-03-09T14:36:22.036 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.034+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=TGJU6RUAKCPI type=del size=2446433 2024-03-09T14:36:22.040 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.038+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~3MWMFOXFDN61.0 size=1048576 2024-03-09T14:36:22.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.038+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~3MWMFOXFDN61.1 size=21048 2024-03-09T14:36:22.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.046+0000 7f73315818c0 1 < completed multipart upload key=IA0AB7UVLFEJ upload=3MWMFOXFDN61 tag=WNLRVP6IWVE1 type=write size=2118200 2024-03-09T14:36:22.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.046+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=YFM6M8ZC400O type=del size=4021009 2024-03-09T14:36:22.055 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.050+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~3S9Z8VF5W2LS.0 size=1048576 2024-03-09T14:36:22.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.054+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~3S9Z8VF5W2LS.1 size=766006 2024-03-09T14:36:22.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.058+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=3S9Z8VF5W2LS tag=22YYYDTE1FQA type=write size=2863158 2024-03-09T14:36:22.070 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.066+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~RKEYPWULAJIL.0 size=792218 2024-03-09T14:36:22.070 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.066+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=RKEYPWULAJIL tag=7WMLADYVK3YN type=write size=1840794 2024-03-09T14:36:22.074 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.070+0000 7f73315818c0 1 < completed write operation key=MSMD3ANK4T7J tag=FO7LV3TYIVUT type=write size=58804 2024-03-09T14:36:22.080 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.078+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~HYEEWY0G53EK.0 size=1048576 2024-03-09T14:36:22.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.078+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~HYEEWY0G53EK.1 size=934629 2024-03-09T14:36:22.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.078+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=HYEEWY0G53EK tag=81FOP812HRHZ type=write size=3031781 2024-03-09T14:36:22.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.082+0000 7f73315818c0 1 < completed delete operation key=DQOY1JJ1YQ2W tag=A3EZRHFRHU5A type=del 2024-03-09T14:36:22.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.086+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=NH6M9D3OXJPH type=write size=337171 2024-03-09T14:36:22.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.090+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=U0MDJAI02WBP type=del 2024-03-09T14:36:22.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.094+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=RIWHHAOVWBX7 type=del size=1151479 2024-03-09T14:36:22.097 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.094+0000 7f73315818c0 1 < canceled part key=_multipart_MD7FXGYXB823.2~OUHGEWLFCRYT.0 size=1039812 2024-03-09T14:36:22.102 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.098+0000 7f73315818c0 1 < canceled multipart upload key=MD7FXGYXB823 upload=OUHGEWLFCRYT tag=FH07V2OKG2H2 type=write size=2088388 2024-03-09T14:36:22.105 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.102+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=Y812GRZG3CTE type=write size=838331 2024-03-09T14:36:22.109 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.106+0000 7f73315818c0 1 < completed write operation key=M1B6O6N8N84T tag=28VCKTU2MBMC type=write size=758210 2024-03-09T14:36:22.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.110+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=31BYHCS856O3 type=del size=677705 2024-03-09T14:36:22.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.114+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~RUQWI57HBOMZ.0 size=191669 2024-03-09T14:36:22.121 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.118+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=RUQWI57HBOMZ tag=SZ71JCMHYY3P type=write size=1240245 2024-03-09T14:36:22.124 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.122+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=QC0HPI7BFRK0 type=del size=482578 2024-03-09T14:36:22.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.126+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~XNJJ8LG3A4SQ.0 size=1048576 2024-03-09T14:36:22.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.126+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~XNJJ8LG3A4SQ.1 size=1048576 2024-03-09T14:36:22.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.126+0000 7f73315818c0 1 < canceled part key=_multipart_K3JNN1NCKBXN.2~XNJJ8LG3A4SQ.2 size=557838 2024-03-09T14:36:22.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.134+0000 7f73315818c0 1 < canceled multipart upload key=K3JNN1NCKBXN upload=XNJJ8LG3A4SQ tag=3B3713JWGS1W type=write size=3703566 2024-03-09T14:36:22.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.138+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=7Z7F81UNTIVW type=del size=448254 2024-03-09T14:36:22.143 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.138+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=YEHPFO8UOZXU type=del 2024-03-09T14:36:22.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.142+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=J31TUKYCVFIT type=del size=852412 2024-03-09T14:36:22.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.146+0000 7f73315818c0 1 < canceled operation key=92GWY5B0K4SX tag=AA0PGAL1JOQC type=del size=1855939 2024-03-09T14:36:22.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.154+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~2LH5RWV0203E.0 size=1048576 2024-03-09T14:36:22.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.154+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~2LH5RWV0203E.1 size=1048576 2024-03-09T14:36:22.160 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.158+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~2LH5RWV0203E.2 size=384295 2024-03-09T14:36:22.160 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.158+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=2LH5RWV0203E tag=4AF0F1ID8P7K type=write size=3530023 2024-03-09T14:36:22.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.162+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=AQFSDCAZIIXU type=del 2024-03-09T14:36:22.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.162+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=TFEFSHI0N3ZE type=del size=1342600 2024-03-09T14:36:22.171 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.166+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=63LGHWHV5HE6 type=del 2024-03-09T14:36:22.174 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.170+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=ZT4PINTSABN8 type=del size=3868037 2024-03-09T14:36:22.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.174+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~MN4SMLFZW896.0 size=1048576 2024-03-09T14:36:22.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.178+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~MN4SMLFZW896.1 size=700759 2024-03-09T14:36:22.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.186+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=MN4SMLFZW896 tag=JV7GT2M35PBF type=write size=2797911 2024-03-09T14:36:22.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.194+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~MRXK3TVRK0P7.0 size=438499 2024-03-09T14:36:22.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.194+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=MRXK3TVRK0P7 tag=7GPN8U2CYVJZ type=write size=1487075 2024-03-09T14:36:22.207 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.202+0000 7f73315818c0 1 < suggested remove operation key=IA0AB7UVLFEJ tag=TH8DE0EE38QJ type=del 2024-03-09T14:36:22.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.206+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~HYUWJHR4U4MT.0 size=836457 2024-03-09T14:36:22.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.206+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=HYUWJHR4U4MT tag=F06JX2FZZ81T type=write size=1885033 2024-03-09T14:36:22.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.210+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=TGJU6RUAKCPI type=del 2024-03-09T14:36:22.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.214+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=JX2D0OP8RF0Z type=write size=511679 2024-03-09T14:36:22.222 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.218+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=YFM6M8ZC400O type=del 2024-03-09T14:36:22.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.222+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=IH66TCZXQ364 type=write size=99282 2024-03-09T14:36:22.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.222+0000 7f73315818c0 1 < canceled part key=_multipart_0WGGMHTEF2LB.2~RKEYPWULAJIL.0 size=792218 2024-03-09T14:36:22.233 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.230+0000 7f73315818c0 1 < canceled multipart upload key=0WGGMHTEF2LB upload=RKEYPWULAJIL tag=7WMLADYVK3YN type=write size=1840794 2024-03-09T14:36:22.237 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.234+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=AR3BYJP4O0P2 type=del size=3282974 2024-03-09T14:36:22.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.238+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~HYEEWY0G53EK.0 size=1048576 2024-03-09T14:36:22.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.238+0000 7f73315818c0 1 < canceled part key=_multipart_1NYNL8M5DVER.2~HYEEWY0G53EK.1 size=934629 2024-03-09T14:36:22.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.242+0000 7f73315818c0 1 < canceled multipart upload key=1NYNL8M5DVER upload=HYEEWY0G53EK tag=81FOP812HRHZ type=write size=3031781 2024-03-09T14:36:22.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.246+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=68FIXFE9JAKZ type=del size=1818861 2024-03-09T14:36:22.258 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.254+0000 7f73315818c0 1 < suggested update operation key=W6KVP8EQFQ8J tag=NH6M9D3OXJPH type=write size=337171 2024-03-09T14:36:22.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.262+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~B4VVBLXO548H.0 size=48184 2024-03-09T14:36:22.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.262+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=B4VVBLXO548H tag=K9U433G74D40 type=write size=1096760 2024-03-09T14:36:22.271 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.270+0000 7f73315818c0 1 < suggested remove operation key=92GWY5B0K4SX tag=RIWHHAOVWBX7 type=del 2024-03-09T14:36:22.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.270+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=KAZ9W2SDV9AX type=del size=1590134 2024-03-09T14:36:22.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.274+0000 7f73315818c0 1 < completed write operation key=DQOY1JJ1YQ2W tag=Y812GRZG3CTE type=write size=838331 2024-03-09T14:36:22.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.282+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~HC5P3CZC9XDA.0 size=1048576 2024-03-09T14:36:22.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.286+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~HC5P3CZC9XDA.1 size=751175 2024-03-09T14:36:22.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.286+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=HC5P3CZC9XDA tag=6D678VX404A7 type=write size=2848327 2024-03-09T14:36:22.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.290+0000 7f73315818c0 1 < suggested remove operation key=0IB00CZ92VV4 tag=31BYHCS856O3 type=del 2024-03-09T14:36:22.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.298+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~7X9BCU16BP15.0 size=1048576 2024-03-09T14:36:22.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.302+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~7X9BCU16BP15.1 size=1048576 2024-03-09T14:36:22.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.302+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~7X9BCU16BP15.2 size=440228 2024-03-09T14:36:22.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.302+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=7X9BCU16BP15 tag=KJ9VX7KH4GWS type=write size=3585956 2024-03-09T14:36:22.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.310+0000 7f73315818c0 1 < suggested remove operation key=162AOMHO2BQL tag=QC0HPI7BFRK0 type=del 2024-03-09T14:36:22.321 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.318+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~ULDYDLWDEGRP.0 size=780622 2024-03-09T14:36:22.321 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.318+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=ULDYDLWDEGRP tag=PH3Q7RBX9FNF type=write size=1829198 2024-03-09T14:36:22.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.322+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=7Z7F81UNTIVW type=del 2024-03-09T14:36:22.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.326+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=RYJ48IKGAUML type=del size=3048724 2024-03-09T14:36:22.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.330+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=J31TUKYCVFIT type=del 2024-03-09T14:36:22.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.338+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~DQC192YENP2D.0 size=1048576 2024-03-09T14:36:22.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.338+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~DQC192YENP2D.1 size=640849 2024-03-09T14:36:22.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.338+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=DQC192YENP2D tag=R50HH05SMR94 type=write size=2738001 2024-03-09T14:36:22.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.342+0000 7f73315818c0 1 < canceled part key=_multipart_P8R5RJI8LGZ9.2~2LH5RWV0203E.0 size=1048576 2024-03-09T14:36:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.342+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~2LH5RWV0203E.1 size=1048576 2024-03-09T14:36:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.342+0000 7f73315818c0 1 < canceled part key=_multipart_P8R5RJI8LGZ9.2~2LH5RWV0203E.2 size=384295 2024-03-09T14:36:22.352 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.350+0000 7f73315818c0 1 < canceled multipart upload key=P8R5RJI8LGZ9 upload=2LH5RWV0203E tag=4AF0F1ID8P7K type=write size=3530023 2024-03-09T14:36:22.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.350+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=PC99QGCTJTS0 type=del size=1700330 2024-03-09T14:36:22.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.354+0000 7f73315818c0 1 < canceled operation key=3ADG4GWVARC1 tag=TFEFSHI0N3ZE type=del size=1342600 2024-03-09T14:36:22.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.358+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=4PK0CVO2PYIZ type=del size=2362487 2024-03-09T14:36:22.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.362+0000 7f73315818c0 1 < suggested remove operation key=M1B6O6N8N84T tag=ZT4PINTSABN8 type=del 2024-03-09T14:36:22.370 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.366+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=TOSH1D218WXU type=write size=717144 2024-03-09T14:36:22.370 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.366+0000 7f73315818c0 1 < canceled part key=_multipart_UMSNM53HTY1J.2~MRXK3TVRK0P7.0 size=438499 2024-03-09T14:36:22.375 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.370+0000 7f73315818c0 1 < canceled multipart upload key=UMSNM53HTY1J upload=MRXK3TVRK0P7 tag=7GPN8U2CYVJZ type=write size=1487075 2024-03-09T14:36:22.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.374+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=1A6S5LBZRKVT type=del size=3351858 2024-03-09T14:36:22.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.374+0000 7f73315818c0 1 < canceled part key=_multipart_DQOY1JJ1YQ2W.2~HYUWJHR4U4MT.0 size=836457 2024-03-09T14:36:22.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.378+0000 7f73315818c0 1 < canceled multipart upload key=DQOY1JJ1YQ2W upload=HYUWJHR4U4MT tag=F06JX2FZZ81T type=write size=1885033 2024-03-09T14:36:22.389 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.386+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~GF6O286GHY73.0 size=1048576 2024-03-09T14:36:22.391 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.386+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~GF6O286GHY73.1 size=784532 2024-03-09T14:36:22.391 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.386+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=GF6O286GHY73 tag=8WVXFEIEV6X9 type=write size=2881684 2024-03-09T14:36:22.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.390+0000 7f73315818c0 1 < completed write operation key=0LP8KKU53X0A tag=JX2D0OP8RF0Z type=write size=511679 2024-03-09T14:36:22.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.394+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=0RTFMWFSU87J type=del size=382244 2024-03-09T14:36:22.400 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.398+0000 7f73315818c0 1 < completed write operation key=IA0AB7UVLFEJ tag=IH66TCZXQ364 type=write size=99282 2024-03-09T14:36:22.404 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.402+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=RQ7507JDBR8Y type=del size=1962582 2024-03-09T14:36:22.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.402+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=AR3BYJP4O0P2 type=del 2024-03-09T14:36:22.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.406+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=ZPKHOVELOPFW type=del size=1612718 2024-03-09T14:36:22.413 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.410+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=68FIXFE9JAKZ type=del 2024-03-09T14:36:22.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.418+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~FAR27F6AS2AG.0 size=843116 2024-03-09T14:36:22.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.418+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=FAR27F6AS2AG tag=OC4HJ1HDQI9V type=write size=1891692 2024-03-09T14:36:22.423 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.418+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~B4VVBLXO548H.0 size=48184 2024-03-09T14:36:22.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.426+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=B4VVBLXO548H tag=K9U433G74D40 type=write size=1096760 2024-03-09T14:36:22.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.430+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=240L63PX12JA type=del size=3230850 2024-03-09T14:36:22.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.430+0000 7f73315818c0 1 < canceled operation key=8O5OP8C5NX12 tag=KAZ9W2SDV9AX type=del size=1590134 2024-03-09T14:36:22.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.434+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=KE2FXHWZJ57V type=del size=2445606 2024-03-09T14:36:22.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.438+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~HC5P3CZC9XDA.0 size=1048576 2024-03-09T14:36:22.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.442+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~HC5P3CZC9XDA.1 size=751175 2024-03-09T14:36:22.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.446+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=HC5P3CZC9XDA tag=6D678VX404A7 type=write size=2848327 2024-03-09T14:36:22.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.450+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=K1THL490LMXN type=del size=2526132 2024-03-09T14:36:22.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.454+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~7X9BCU16BP15.0 size=1048576 2024-03-09T14:36:22.459 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.458+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~7X9BCU16BP15.1 size=1048576 2024-03-09T14:36:22.462 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.458+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~7X9BCU16BP15.2 size=440228 2024-03-09T14:36:22.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.466+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=7X9BCU16BP15 tag=KJ9VX7KH4GWS type=write size=3585956 2024-03-09T14:36:22.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.470+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~CF0VOX0S239S.0 size=222091 2024-03-09T14:36:22.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.470+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=CF0VOX0S239S tag=Q08IEL0AGDS5 type=write size=1270667 2024-03-09T14:36:22.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.474+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~ULDYDLWDEGRP.0 size=780622 2024-03-09T14:36:22.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.478+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=ULDYDLWDEGRP tag=PH3Q7RBX9FNF type=write size=1829198 2024-03-09T14:36:22.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.482+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=MRYIFQ9FPAWA type=del size=1250141 2024-03-09T14:36:22.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.486+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=RYJ48IKGAUML type=del 2024-03-09T14:36:22.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.490+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=MEAT77ECV72O type=del size=1875783 2024-03-09T14:36:22.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.494+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~DQC192YENP2D.0 size=1048576 2024-03-09T14:36:22.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.494+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~DQC192YENP2D.1 size=640849 2024-03-09T14:36:22.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.502+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=DQC192YENP2D tag=R50HH05SMR94 type=write size=2738001 2024-03-09T14:36:22.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.510+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~O8OD6TR13RRQ.0 size=701446 2024-03-09T14:36:22.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.510+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=O8OD6TR13RRQ tag=KUGTTH0P7Y9V type=write size=1750022 2024-03-09T14:36:22.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.514+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=PC99QGCTJTS0 type=del 2024-03-09T14:36:22.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.518+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=N3BM3QEBEDIS type=del size=1262151 2024-03-09T14:36:22.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.522+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=4PK0CVO2PYIZ type=del 2024-03-09T14:36:22.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.526+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=SBVSZRGO2HS7 type=del size=25398 2024-03-09T14:36:22.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.526+0000 7f73315818c0 1 < completed write operation key=92GWY5B0K4SX tag=TOSH1D218WXU type=write size=717144 2024-03-09T14:36:22.534 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.530+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=R4Y7OM6S5EN7 type=del size=3533504 2024-03-09T14:36:22.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.534+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=1A6S5LBZRKVT type=del 2024-03-09T14:36:22.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.538+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=CCS0L4FLHFC5 type=del size=388213 2024-03-09T14:36:22.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.538+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~GF6O286GHY73.0 size=1048576 2024-03-09T14:36:22.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.542+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~GF6O286GHY73.1 size=784532 2024-03-09T14:36:22.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.546+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=GF6O286GHY73 tag=8WVXFEIEV6X9 type=write size=2881684 2024-03-09T14:36:22.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.550+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~5QQEIB43OGC3.0 size=1048576 2024-03-09T14:36:22.557 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.554+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~5QQEIB43OGC3.1 size=1048576 2024-03-09T14:36:22.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.554+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~5QQEIB43OGC3.2 size=378203 2024-03-09T14:36:22.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.554+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=5QQEIB43OGC3 tag=0L1F7LSVKS3Y type=write size=3523931 2024-03-09T14:36:22.565 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.562+0000 7f73315818c0 1 < suggested update operation key=XELG1AWATMVL tag=0RTFMWFSU87J type=del size=382244 2024-03-09T14:36:22.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.566+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=SLYB3DVH84K1 type=write size=883499 2024-03-09T14:36:22.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.570+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=RQ7507JDBR8Y type=del 2024-03-09T14:36:22.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.574+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~SMYHOCYF2DEU.0 size=1048576 2024-03-09T14:36:22.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.578+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~SMYHOCYF2DEU.1 size=1048576 2024-03-09T14:36:22.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.578+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~SMYHOCYF2DEU.2 size=388503 2024-03-09T14:36:22.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.578+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=SMYHOCYF2DEU tag=RUHU58NDQTQS type=write size=3534231 2024-03-09T14:36:22.585 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.582+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=ZPKHOVELOPFW type=del 2024-03-09T14:36:22.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.586+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=5D395L5M3R6V type=write size=249470 2024-03-09T14:36:22.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.590+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~FAR27F6AS2AG.0 size=843116 2024-03-09T14:36:22.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.594+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=FAR27F6AS2AG tag=OC4HJ1HDQI9V type=write size=1891692 2024-03-09T14:36:22.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.598+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~18PSO0WG494J.0 size=889571 2024-03-09T14:36:22.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.598+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=18PSO0WG494J tag=K2PGZJYW98K3 type=write size=1938147 2024-03-09T14:36:22.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.606+0000 7f73315818c0 1 < suggested update operation key=MD7FXGYXB823 tag=240L63PX12JA type=del size=3230850 2024-03-09T14:36:22.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.610+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~QWQJ30XXQVCY.0 size=1048576 2024-03-09T14:36:22.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.614+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~QWQJ30XXQVCY.1 size=1048576 2024-03-09T14:36:22.618 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.614+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~QWQJ30XXQVCY.2 size=741543 2024-03-09T14:36:22.618 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.614+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=QWQJ30XXQVCY tag=NIOZAHIP5L9T type=write size=3887271 2024-03-09T14:36:22.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.622+0000 7f73315818c0 1 < suggested remove operation key=FYFVW9R0M1PX tag=KE2FXHWZJ57V type=del 2024-03-09T14:36:22.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.626+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~M9L87L1YAL5W.0 size=524467 2024-03-09T14:36:22.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.626+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=M9L87L1YAL5W tag=6SJRD4A9L7YP type=write size=1573043 2024-03-09T14:36:22.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.630+0000 7f73315818c0 1 < canceled operation key=W6KVP8EQFQ8J tag=K1THL490LMXN type=del size=2526132 2024-03-09T14:36:22.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.630+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=S6RQLBCZ6CMT type=write size=188272 2024-03-09T14:36:22.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.634+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~CF0VOX0S239S.0 size=222091 2024-03-09T14:36:22.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.638+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=CF0VOX0S239S tag=Q08IEL0AGDS5 type=write size=1270667 2024-03-09T14:36:22.644 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.642+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=DM8Z9VL3Q48Z type=del size=3449664 2024-03-09T14:36:22.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.646+0000 7f73315818c0 1 < suggested update operation key=0IB00CZ92VV4 tag=MRYIFQ9FPAWA type=del size=1250141 2024-03-09T14:36:22.653 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.650+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=9GQ0NFORBPKT type=del size=1309690 2024-03-09T14:36:22.656 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.654+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=MEAT77ECV72O type=del 2024-03-09T14:36:22.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.658+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~J4KD0C5ZZGT9.0 size=1048576 2024-03-09T14:36:22.664 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.662+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~J4KD0C5ZZGT9.1 size=574438 2024-03-09T14:36:22.664 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.662+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=J4KD0C5ZZGT9 tag=P6BCSFSLNGR2 type=write size=2671590 2024-03-09T14:36:22.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.662+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~O8OD6TR13RRQ.0 size=701446 2024-03-09T14:36:22.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.670+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=O8OD6TR13RRQ tag=KUGTTH0P7Y9V type=write size=1750022 2024-03-09T14:36:22.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.670+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=MQJJT0YDPI6Q type=del size=354664 2024-03-09T14:36:22.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.674+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=N3BM3QEBEDIS type=del 2024-03-09T14:36:22.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.678+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=HIUQBDDSV41G type=write size=878985 2024-03-09T14:36:22.689 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.686+0000 7f73315818c0 1 < suggested remove operation key=162AOMHO2BQL tag=SBVSZRGO2HS7 type=del 2024-03-09T14:36:22.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.690+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=8Z6B92N9SIYI type=del size=3455174 2024-03-09T14:36:22.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.694+0000 7f73315818c0 1 < suggested update operation key=P9LI5ESR3N9G tag=R4Y7OM6S5EN7 type=del size=3533504 2024-03-09T14:36:22.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.698+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=N0T6YND96T86 type=del size=1895419 2024-03-09T14:36:22.703 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.698+0000 7f73315818c0 1 < completed delete operation key=M1B6O6N8N84T tag=CCS0L4FLHFC5 type=del 2024-03-09T14:36:22.706 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.702+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=R0B6ZIG41P13 type=write size=559974 2024-03-09T14:36:22.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.706+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~5QQEIB43OGC3.0 size=1048576 2024-03-09T14:36:22.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.706+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~5QQEIB43OGC3.1 size=1048576 2024-03-09T14:36:22.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.710+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~5QQEIB43OGC3.2 size=378203 2024-03-09T14:36:22.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.714+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=5QQEIB43OGC3 tag=0L1F7LSVKS3Y type=write size=3523931 2024-03-09T14:36:22.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.718+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=3CRQZVW27AI3 type=del size=135244 2024-03-09T14:36:22.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.718+0000 7f73315818c0 -1 < failed to complete operation key=FYFVW9R0M1PX tag=SLYB3DVH84K1 type=write size=883499: (22) Invalid argument 2024-03-09T14:36:22.726 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.722+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=V0FXM0LLZIZU type=del size=2647996 2024-03-09T14:36:22.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.722+0000 7f73315818c0 1 < canceled part key=_multipart_P8R5RJI8LGZ9.2~SMYHOCYF2DEU.0 size=1048576 2024-03-09T14:36:22.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.726+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~SMYHOCYF2DEU.1 size=1048576 2024-03-09T14:36:22.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.730+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~SMYHOCYF2DEU.2 size=388503 2024-03-09T14:36:22.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.734+0000 7f73315818c0 1 < canceled multipart upload key=P8R5RJI8LGZ9 upload=SMYHOCYF2DEU tag=RUHU58NDQTQS type=write size=3534231 2024-03-09T14:36:22.741 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.738+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~3QTT6YX05YPT.0 size=350667 2024-03-09T14:36:22.742 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.738+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=3QTT6YX05YPT tag=K1NVFA7TGG3F type=write size=1399243 2024-03-09T14:36:22.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.742+0000 7f73315818c0 1 < completed write operation key=Y8EQAHMTL66U tag=5D395L5M3R6V type=write size=249470 2024-03-09T14:36:22.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.746+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=Z6ZHU83CYMK3 type=write size=956327 2024-03-09T14:36:22.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.746+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~18PSO0WG494J.0 size=889571 2024-03-09T14:36:22.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.754+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=18PSO0WG494J tag=K2PGZJYW98K3 type=write size=1938147 2024-03-09T14:36:22.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.758+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=7U1JV0Y52K0A type=del size=3779200 2024-03-09T14:36:22.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.758+0000 7f73315818c0 1 < canceled part key=_multipart_8O5OP8C5NX12.2~QWQJ30XXQVCY.0 size=1048576 2024-03-09T14:36:22.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.758+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~QWQJ30XXQVCY.1 size=1048576 2024-03-09T14:36:22.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.762+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~QWQJ30XXQVCY.2 size=741543 2024-03-09T14:36:22.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.770+0000 7f73315818c0 1 < canceled multipart upload key=8O5OP8C5NX12 upload=QWQJ30XXQVCY tag=NIOZAHIP5L9T type=write size=3887271 2024-03-09T14:36:22.775 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.774+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=ZK0JKSW59API type=write size=299734 2024-03-09T14:36:22.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.774+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~M9L87L1YAL5W.0 size=524467 2024-03-09T14:36:22.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.782+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=M9L87L1YAL5W tag=6SJRD4A9L7YP type=write size=1573043 2024-03-09T14:36:22.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.786+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=IR7PVXK7UCAW type=del size=69233 2024-03-09T14:36:22.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.790+0000 7f73315818c0 1 < completed write operation key=UMSNM53HTY1J tag=S6RQLBCZ6CMT type=write size=188272 2024-03-09T14:36:22.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.794+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=XZ8622Q45JMP type=del size=1035803 2024-03-09T14:36:22.803 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.798+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=DM8Z9VL3Q48Z type=del 2024-03-09T14:36:22.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.802+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=Y8H9VC19JP1X type=del size=3560918 2024-03-09T14:36:22.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.806+0000 7f73315818c0 1 < canceled operation key=IA0AB7UVLFEJ tag=9GQ0NFORBPKT type=del size=1309690 2024-03-09T14:36:22.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.810+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=6KSAMTZGYU85 type=write size=503832 2024-03-09T14:36:22.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.814+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~J4KD0C5ZZGT9.0 size=1048576 2024-03-09T14:36:22.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.814+0000 7f73315818c0 1 < canceled part key=_multipart_IA0AB7UVLFEJ.2~J4KD0C5ZZGT9.1 size=574438 2024-03-09T14:36:22.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.818+0000 7f73315818c0 1 < canceled multipart upload key=IA0AB7UVLFEJ upload=J4KD0C5ZZGT9 tag=P6BCSFSLNGR2 type=write size=2671590 2024-03-09T14:36:22.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.822+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=RMSHF4A6FCJY type=del size=3001676 2024-03-09T14:36:22.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.826+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=MQJJT0YDPI6Q type=del 2024-03-09T14:36:22.832 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.830+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=OS018QOCPV0E type=del size=2273657 2024-03-09T14:36:22.835 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.834+0000 7f73315818c0 1 < completed write operation key=LZC7NPF0GRO7 tag=HIUQBDDSV41G type=write size=878985 2024-03-09T14:36:22.839 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.834+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=TV8GIKMQ0I8B type=del size=1700838 2024-03-09T14:36:22.845 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.842+0000 7f73315818c0 1 < suggested update operation key=0LP8KKU53X0A tag=8Z6B92N9SIYI type=del size=3455174 2024-03-09T14:36:22.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.846+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=M0220R54F4L9 type=del size=667149 2024-03-09T14:36:22.855 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.854+0000 7f73315818c0 1 < suggested update operation key=0IB00CZ92VV4 tag=N0T6YND96T86 type=del size=1895419 2024-03-09T14:36:22.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.854+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=A3H6NZH80G30 type=write size=169773 2024-03-09T14:36:22.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.858+0000 7f73315818c0 1 < completed write operation key=4P89277EEYJ3 tag=R0B6ZIG41P13 type=write size=559974 2024-03-09T14:36:22.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.862+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=XV19BTHUNR6J type=del size=2535689 2024-03-09T14:36:22.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.866+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=3CRQZVW27AI3 type=del 2024-03-09T14:36:22.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.874+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~B6HZK7YV60R5.0 size=1048576 2024-03-09T14:36:22.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.874+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~B6HZK7YV60R5.1 size=189784 2024-03-09T14:36:22.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.874+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=B6HZK7YV60R5 tag=6SRCXYYFFKC6 type=write size=2286936 2024-03-09T14:36:22.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.878+0000 7f73315818c0 1 < canceled operation key=XELG1AWATMVL tag=V0FXM0LLZIZU type=del size=2647996 2024-03-09T14:36:22.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.882+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=KT09KD1OUTAJ type=del size=1086095 2024-03-09T14:36:22.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.886+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~3QTT6YX05YPT.0 size=350667 2024-03-09T14:36:22.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.890+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=3QTT6YX05YPT tag=K1NVFA7TGG3F type=write size=1399243 2024-03-09T14:36:22.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.894+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=GZ1ISFJI90A1 type=write size=102572 2024-03-09T14:36:22.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.898+0000 7f73315818c0 1 < suggested remove operation key=3ADG4GWVARC1 tag=Z6ZHU83CYMK3 type=write 2024-03-09T14:36:22.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.906+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~WQQO6Y2NHH3I.0 size=1048576 2024-03-09T14:36:22.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.906+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~WQQO6Y2NHH3I.1 size=1048576 2024-03-09T14:36:22.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.910+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~WQQO6Y2NHH3I.2 size=457115 2024-03-09T14:36:22.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.910+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=WQQO6Y2NHH3I tag=WLDA3CDHQB5W type=write size=3602843 2024-03-09T14:36:22.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.914+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=7U1JV0Y52K0A type=del 2024-03-09T14:36:22.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.918+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~4W007I1QLOHO.0 size=1048576 2024-03-09T14:36:22.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.922+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~4W007I1QLOHO.1 size=286326 2024-03-09T14:36:22.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.922+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=4W007I1QLOHO tag=3DBMNQFOCR2X type=write size=2383478 2024-03-09T14:36:22.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.926+0000 7f73315818c0 1 < suggested update operation key=P9LI5ESR3N9G tag=ZK0JKSW59API type=write size=299734 2024-03-09T14:36:22.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.930+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=WWT610IF0S7Z type=del size=1477628 2024-03-09T14:36:22.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.934+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=IR7PVXK7UCAW type=del 2024-03-09T14:36:22.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.938+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=K03VR9XPNLO3 type=del size=904383 2024-03-09T14:36:22.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.942+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=XZ8622Q45JMP type=del 2024-03-09T14:36:22.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.946+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=7NUZQGN66EXA type=del size=3357379 2024-03-09T14:36:22.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.950+0000 7f73315818c0 1 < canceled operation key=Y8EQAHMTL66U tag=Y8H9VC19JP1X type=del size=3560918 2024-03-09T14:36:22.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.954+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=H7E3JLLHY2HT type=del size=987338 2024-03-09T14:36:22.960 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.958+0000 7f73315818c0 -1 < failed to complete operation key=3ADG4GWVARC1 tag=6KSAMTZGYU85 type=write size=503832: (22) Invalid argument 2024-03-09T14:36:22.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.962+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=Q8DPMASOES0D type=del size=3239654 2024-03-09T14:36:22.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.970+0000 7f73315818c0 1 < suggested remove operation key=P9LI5ESR3N9G tag=RMSHF4A6FCJY type=del 2024-03-09T14:36:22.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.970+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=MK8KJY35D59V type=del size=3218950 2024-03-09T14:36:22.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.974+0000 7f73315818c0 1 < completed delete operation key=M1B6O6N8N84T tag=OS018QOCPV0E type=del 2024-03-09T14:36:22.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.982+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~NA37NVM3L85X.0 size=1048576 2024-03-09T14:36:22.988 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.986+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~NA37NVM3L85X.1 size=315474 2024-03-09T14:36:22.988 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.986+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=NA37NVM3L85X tag=3YLTVAEK8NY8 type=write size=2412626 2024-03-09T14:36:22.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.986+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=TV8GIKMQ0I8B type=del 2024-03-09T14:36:22.996 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.994+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~X0DBR6M3CHP1.0 size=1048576 2024-03-09T14:36:22.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:22.998+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~X0DBR6M3CHP1.1 size=1048576 2024-03-09T14:36:23.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.002+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~X0DBR6M3CHP1.2 size=665694 2024-03-09T14:36:23.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.002+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=X0DBR6M3CHP1 tag=1WT9JQ06ZXZI type=write size=3811422 2024-03-09T14:36:23.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.006+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=M0220R54F4L9 type=del 2024-03-09T14:36:23.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.010+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=0QWHSHZ4EJTG type=del size=105638 2024-03-09T14:36:23.016 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.014+0000 7f73315818c0 1 < completed write operation key=0WGGMHTEF2LB tag=A3H6NZH80G30 type=write size=169773 2024-03-09T14:36:23.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.014+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=M1E3YPFOUCFG type=del size=3677193 2024-03-09T14:36:23.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.022+0000 7f73315818c0 1 < suggested update operation key=MD7FXGYXB823 tag=XV19BTHUNR6J type=del size=2535689 2024-03-09T14:36:23.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.026+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=BIS5LOABKD6O type=del size=2713855 2024-03-09T14:36:23.031 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.030+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~B6HZK7YV60R5.0 size=1048576 2024-03-09T14:36:23.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.030+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~B6HZK7YV60R5.1 size=189784 2024-03-09T14:36:23.039 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.038+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=B6HZK7YV60R5 tag=6SRCXYYFFKC6 type=write size=2286936 2024-03-09T14:36:23.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.042+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~LKPNJCIR3U5P.0 size=811043 2024-03-09T14:36:23.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.042+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=LKPNJCIR3U5P tag=RR1NWVZ1IMLS type=write size=1859619 2024-03-09T14:36:23.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.050+0000 7f73315818c0 1 < suggested update operation key=XELG1AWATMVL tag=KT09KD1OUTAJ type=del size=1086095 2024-03-09T14:36:23.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.054+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~O2W36HE0138K.0 size=1048576 2024-03-09T14:36:23.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.058+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~O2W36HE0138K.1 size=1048576 2024-03-09T14:36:23.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.058+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~O2W36HE0138K.2 size=345964 2024-03-09T14:36:23.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.058+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=O2W36HE0138K tag=T682E7SJSM62 type=write size=3491692 2024-03-09T14:36:23.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.066+0000 7f73315818c0 1 < suggested remove operation key=0IB00CZ92VV4 tag=GZ1ISFJI90A1 type=write 2024-03-09T14:36:23.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.070+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=DN2FVHHH98SZ type=del size=2137762 2024-03-09T14:36:23.075 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.074+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~WQQO6Y2NHH3I.0 size=1048576 2024-03-09T14:36:23.075 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.074+0000 7f73315818c0 1 < canceled part key=_multipart_162AOMHO2BQL.2~WQQO6Y2NHH3I.1 size=1048576 2024-03-09T14:36:23.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.074+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~WQQO6Y2NHH3I.2 size=457115 2024-03-09T14:36:23.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.078+0000 7f73315818c0 1 < canceled multipart upload key=162AOMHO2BQL upload=WQQO6Y2NHH3I tag=WLDA3CDHQB5W type=write size=3602843 2024-03-09T14:36:23.085 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.082+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=ZTWBY8PK450Y type=del size=1177562 2024-03-09T14:36:23.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.086+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~4W007I1QLOHO.0 size=1048576 2024-03-09T14:36:23.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.086+0000 7f73315818c0 1 < canceled part key=_multipart_MD7FXGYXB823.2~4W007I1QLOHO.1 size=286326 2024-03-09T14:36:23.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.094+0000 7f73315818c0 1 < canceled multipart upload key=MD7FXGYXB823 upload=4W007I1QLOHO tag=3DBMNQFOCR2X type=write size=2383478 2024-03-09T14:36:23.100 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.098+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=CH98BV4W6E3C type=write size=1028636 2024-03-09T14:36:23.104 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.102+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=WWT610IF0S7Z type=del 2024-03-09T14:36:23.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.106+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=2WYT3TSGJCV7 type=write size=489653 2024-03-09T14:36:23.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.110+0000 7f73315818c0 1 < suggested remove operation key=QG4O2L526B8E tag=K03VR9XPNLO3 type=del 2024-03-09T14:36:23.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.122+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~2AOD9QVQM2QH.0 size=1048576 2024-03-09T14:36:23.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.126+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~2AOD9QVQM2QH.1 size=1048576 2024-03-09T14:36:23.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.126+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~2AOD9QVQM2QH.2 size=1048271 2024-03-09T14:36:23.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.126+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=2AOD9QVQM2QH tag=OL28CTDBPCB6 type=write size=4193999 2024-03-09T14:36:23.135 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.130+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=7NUZQGN66EXA type=del 2024-03-09T14:36:23.140 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.138+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~4OX31FPNN8M7.0 size=119182 2024-03-09T14:36:23.140 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.138+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=4OX31FPNN8M7 tag=J8BX7341J9OX type=write size=1167758 2024-03-09T14:36:23.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.142+0000 7f73315818c0 1 < suggested update operation key=0IB00CZ92VV4 tag=H7E3JLLHY2HT type=del size=987338 2024-03-09T14:36:23.149 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.146+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=0D5F4B3DT6M2 type=del size=3248270 2024-03-09T14:36:23.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.150+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=Q8DPMASOES0D type=del 2024-03-09T14:36:23.159 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.154+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~8TIL2TSQMQD4.0 size=1048576 2024-03-09T14:36:23.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.158+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~8TIL2TSQMQD4.1 size=822091 2024-03-09T14:36:23.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.158+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=8TIL2TSQMQD4 tag=3VRGLI2GT2H1 type=write size=2919243 2024-03-09T14:36:23.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.162+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=MK8KJY35D59V type=del 2024-03-09T14:36:23.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.166+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=LQ1T9I2JUH4Z type=del size=1089718 2024-03-09T14:36:23.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.166+0000 7f73315818c0 1 < canceled part key=_multipart_M8F9TA80VME6.2~NA37NVM3L85X.0 size=1048576 2024-03-09T14:36:23.171 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.166+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~NA37NVM3L85X.1 size=315474 2024-03-09T14:36:23.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.174+0000 7f73315818c0 1 < canceled multipart upload key=M8F9TA80VME6 upload=NA37NVM3L85X tag=3YLTVAEK8NY8 type=write size=2412626 2024-03-09T14:36:23.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.178+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=Z9X62AN490UJ type=del size=2637802 2024-03-09T14:36:23.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.182+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~X0DBR6M3CHP1.0 size=1048576 2024-03-09T14:36:23.186 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.182+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~X0DBR6M3CHP1.1 size=1048576 2024-03-09T14:36:23.189 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.186+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~X0DBR6M3CHP1.2 size=665694 2024-03-09T14:36:23.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.190+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=X0DBR6M3CHP1 tag=1WT9JQ06ZXZI type=write size=3811422 2024-03-09T14:36:23.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.198+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~RO9PPP8OSFRQ.0 size=1048576 2024-03-09T14:36:23.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.202+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~RO9PPP8OSFRQ.1 size=784241 2024-03-09T14:36:23.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.202+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=RO9PPP8OSFRQ tag=EQZPVIR4JJS7 type=write size=2881393 2024-03-09T14:36:23.207 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.202+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=0QWHSHZ4EJTG type=del 2024-03-09T14:36:23.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.206+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=8AD4XT0IS7M6 type=del size=1501227 2024-03-09T14:36:23.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.210+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=M1E3YPFOUCFG type=del 2024-03-09T14:36:23.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.214+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~B5P98G8FG48T.0 size=278291 2024-03-09T14:36:23.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.218+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=B5P98G8FG48T tag=V64145Q04PVE type=write size=1326867 2024-03-09T14:36:23.222 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.218+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=BIS5LOABKD6O type=del 2024-03-09T14:36:23.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.226+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~BOMF1KUVFY1Z.0 size=1048576 2024-03-09T14:36:23.231 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.226+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~BOMF1KUVFY1Z.1 size=287482 2024-03-09T14:36:23.231 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.226+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=BOMF1KUVFY1Z tag=408GTR335NOI type=write size=2384634 2024-03-09T14:36:23.235 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.230+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~LKPNJCIR3U5P.0 size=811043 2024-03-09T14:36:23.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.238+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=LKPNJCIR3U5P tag=RR1NWVZ1IMLS type=write size=1859619 2024-03-09T14:36:23.244 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.242+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=RI0TIME2DNNN type=write size=216677 2024-03-09T14:36:23.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.246+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~O2W36HE0138K.0 size=1048576 2024-03-09T14:36:23.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.246+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~O2W36HE0138K.1 size=1048576 2024-03-09T14:36:23.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.250+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~O2W36HE0138K.2 size=345964 2024-03-09T14:36:23.258 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.254+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=O2W36HE0138K tag=T682E7SJSM62 type=write size=3491692 2024-03-09T14:36:23.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.258+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=RY0W8QY46C2C type=del size=68549 2024-03-09T14:36:23.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.266+0000 7f73315818c0 1 < suggested update operation key=S302JFYEDEL7 tag=DN2FVHHH98SZ type=del size=2137762 2024-03-09T14:36:23.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.270+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~R4YJEL7PRLQP.0 size=1048576 2024-03-09T14:36:23.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.274+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~R4YJEL7PRLQP.1 size=1048576 2024-03-09T14:36:23.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.278+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~R4YJEL7PRLQP.2 size=783882 2024-03-09T14:36:23.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.278+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=R4YJEL7PRLQP tag=5VIQRI1XGPQK type=write size=3929610 2024-03-09T14:36:23.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.282+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=ZTWBY8PK450Y type=del 2024-03-09T14:36:23.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.282+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=RYXXEP0V55IQ type=del size=2915542 2024-03-09T14:36:23.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.286+0000 7f73315818c0 -1 < failed to complete operation key=0IB00CZ92VV4 tag=CH98BV4W6E3C type=write size=1028636: (22) Invalid argument 2024-03-09T14:36:23.293 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.290+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=C6FVRGM1NDDU type=del size=2818584 2024-03-09T14:36:23.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.298+0000 7f73315818c0 1 < suggested remove operation key=3ADG4GWVARC1 tag=2WYT3TSGJCV7 type=write 2024-03-09T14:36:23.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.298+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=1F75XN1D2ZL1 type=write size=269859 2024-03-09T14:36:23.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.302+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~2AOD9QVQM2QH.0 size=1048576 2024-03-09T14:36:23.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.306+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~2AOD9QVQM2QH.1 size=1048576 2024-03-09T14:36:23.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.310+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~2AOD9QVQM2QH.2 size=1048271 2024-03-09T14:36:23.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.318+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=2AOD9QVQM2QH tag=OL28CTDBPCB6 type=write size=4193999 2024-03-09T14:36:23.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.322+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~3N3FSHBVQ3N9.0 size=1048576 2024-03-09T14:36:23.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.326+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~3N3FSHBVQ3N9.1 size=1048576 2024-03-09T14:36:23.331 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.330+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~3N3FSHBVQ3N9.2 size=244607 2024-03-09T14:36:23.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.330+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=3N3FSHBVQ3N9 tag=RPVNQM0HS9IR type=write size=3390335 2024-03-09T14:36:23.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.330+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~4OX31FPNN8M7.0 size=119182 2024-03-09T14:36:23.341 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.338+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=4OX31FPNN8M7 tag=J8BX7341J9OX type=write size=1167758 2024-03-09T14:36:23.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.342+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~65AKQID1I2O0.0 size=1048576 2024-03-09T14:36:23.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.346+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~65AKQID1I2O0.1 size=1048576 2024-03-09T14:36:23.352 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.350+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~65AKQID1I2O0.2 size=702295 2024-03-09T14:36:23.352 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.350+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=65AKQID1I2O0 tag=SQXF9ORC8PKJ type=write size=3848023 2024-03-09T14:36:23.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.350+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=0D5F4B3DT6M2 type=del 2024-03-09T14:36:23.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.358+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~Q0EA94MVCD4P.0 size=1048576 2024-03-09T14:36:23.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.362+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~Q0EA94MVCD4P.1 size=301725 2024-03-09T14:36:23.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.362+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=Q0EA94MVCD4P tag=T4PFXQ0SH2IC type=write size=2398877 2024-03-09T14:36:23.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.366+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~8TIL2TSQMQD4.0 size=1048576 2024-03-09T14:36:23.370 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.366+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~8TIL2TSQMQD4.1 size=822091 2024-03-09T14:36:23.376 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.374+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=8TIL2TSQMQD4 tag=3VRGLI2GT2H1 type=write size=2919243 2024-03-09T14:36:23.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.378+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=M35JVN5URUSQ type=write size=1020095 2024-03-09T14:36:23.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.378+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=LQ1T9I2JUH4Z type=del 2024-03-09T14:36:23.387 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.382+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=BOMU38I7RBE5 type=del size=1319360 2024-03-09T14:36:23.391 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.386+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=Z9X62AN490UJ type=del 2024-03-09T14:36:23.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.394+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~AVTNCUQD0PO2.0 size=1048576 2024-03-09T14:36:23.400 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.398+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~AVTNCUQD0PO2.1 size=744213 2024-03-09T14:36:23.400 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.398+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=AVTNCUQD0PO2 tag=IN1I4JLJ12OF type=write size=2841365 2024-03-09T14:36:23.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.402+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~RO9PPP8OSFRQ.0 size=1048576 2024-03-09T14:36:23.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.406+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~RO9PPP8OSFRQ.1 size=784241 2024-03-09T14:36:23.414 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.410+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=RO9PPP8OSFRQ tag=EQZPVIR4JJS7 type=write size=2881393 2024-03-09T14:36:23.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.418+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~TKW8YKEMS8G1.0 size=1048576 2024-03-09T14:36:23.423 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.418+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~TKW8YKEMS8G1.1 size=1048576 2024-03-09T14:36:23.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.422+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~TKW8YKEMS8G1.2 size=192351 2024-03-09T14:36:23.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.422+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=TKW8YKEMS8G1 tag=YAHOGJVDF2H2 type=write size=3338079 2024-03-09T14:36:23.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.426+0000 7f73315818c0 -1 < failed to complete operation key=S302JFYEDEL7 tag=8AD4XT0IS7M6 type=del size=1501227: (22) Invalid argument 2024-03-09T14:36:23.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.430+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=H310KV85W1QT type=write size=1042305 2024-03-09T14:36:23.440 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.438+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~B5P98G8FG48T.0 size=278291 2024-03-09T14:36:23.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.442+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=B5P98G8FG48T tag=V64145Q04PVE type=write size=1326867 2024-03-09T14:36:23.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.446+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=M21SJI0V48DE type=write size=667131 2024-03-09T14:36:23.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.446+0000 7f73315818c0 1 < canceled part key=_multipart_FYFVW9R0M1PX.2~BOMF1KUVFY1Z.0 size=1048576 2024-03-09T14:36:23.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.450+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~BOMF1KUVFY1Z.1 size=287482 2024-03-09T14:36:23.460 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.458+0000 7f73315818c0 1 < canceled multipart upload key=FYFVW9R0M1PX upload=BOMF1KUVFY1Z tag=408GTR335NOI type=write size=2384634 2024-03-09T14:36:23.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.458+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=F1PS9RB2BKWE type=write size=813044 2024-03-09T14:36:23.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.462+0000 7f73315818c0 1 < completed write operation key=CVAWRZMAX997 tag=RI0TIME2DNNN type=write size=216677 2024-03-09T14:36:23.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.466+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=K4BVWXUB9UFP type=del size=382108 2024-03-09T14:36:23.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.474+0000 7f73315818c0 1 < suggested update operation key=IA0AB7UVLFEJ tag=RY0W8QY46C2C type=del size=68549 2024-03-09T14:36:23.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.478+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=D2UJOYURUB4B type=del size=1440301 2024-03-09T14:36:23.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.482+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~R4YJEL7PRLQP.0 size=1048576 2024-03-09T14:36:23.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.486+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~R4YJEL7PRLQP.1 size=1048576 2024-03-09T14:36:23.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.486+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~R4YJEL7PRLQP.2 size=783882 2024-03-09T14:36:23.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.494+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=R4YJEL7PRLQP tag=5VIQRI1XGPQK type=write size=3929610 2024-03-09T14:36:23.503 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.498+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~6MWITSEJRKJZ.0 size=697130 2024-03-09T14:36:23.503 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.498+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=6MWITSEJRKJZ tag=ITLNQ2S86U1U type=write size=1745706 2024-03-09T14:36:23.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.502+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=RYXXEP0V55IQ type=del 2024-03-09T14:36:23.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.506+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=22RH3RFAGPDJ type=del size=3029834 2024-03-09T14:36:23.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.514+0000 7f73315818c0 1 < suggested update operation key=CVAWRZMAX997 tag=C6FVRGM1NDDU type=del size=2818584 2024-03-09T14:36:23.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.518+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=EH8ANNVONSS1 type=del size=2322205 2024-03-09T14:36:23.528 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.526+0000 7f73315818c0 1 < suggested update operation key=8O5OP8C5NX12 tag=1F75XN1D2ZL1 type=write size=269859 2024-03-09T14:36:23.534 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.530+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~58BVNC428I2I.0 size=330082 2024-03-09T14:36:23.534 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.530+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=58BVNC428I2I tag=LEECCZ51SRIE type=write size=1378658 2024-03-09T14:36:23.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.534+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~3N3FSHBVQ3N9.0 size=1048576 2024-03-09T14:36:23.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.538+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~3N3FSHBVQ3N9.1 size=1048576 2024-03-09T14:36:23.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.538+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~3N3FSHBVQ3N9.2 size=244607 2024-03-09T14:36:23.549 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.546+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=3N3FSHBVQ3N9 tag=RPVNQM0HS9IR type=write size=3390335 2024-03-09T14:36:23.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.550+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~ZDNP9IQUWW9Y.0 size=941791 2024-03-09T14:36:23.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.554+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=ZDNP9IQUWW9Y tag=SA8G1QZIQYLN type=write size=1990367 2024-03-09T14:36:23.556 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.554+0000 7f73315818c0 1 < canceled part key=_multipart_92GWY5B0K4SX.2~65AKQID1I2O0.0 size=1048576 2024-03-09T14:36:23.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.554+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~65AKQID1I2O0.1 size=1048576 2024-03-09T14:36:23.561 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.558+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~65AKQID1I2O0.2 size=702295 2024-03-09T14:36:23.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.566+0000 7f73315818c0 1 < canceled multipart upload key=92GWY5B0K4SX upload=65AKQID1I2O0 tag=SQXF9ORC8PKJ type=write size=3848023 2024-03-09T14:36:23.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.566+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=JYGQ5CT4MBHV type=del size=1539814 2024-03-09T14:36:23.574 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.570+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~Q0EA94MVCD4P.0 size=1048576 2024-03-09T14:36:23.574 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.570+0000 7f73315818c0 1 < canceled part key=_multipart_MSMD3ANK4T7J.2~Q0EA94MVCD4P.1 size=301725 2024-03-09T14:36:23.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.578+0000 7f73315818c0 1 < canceled multipart upload key=MSMD3ANK4T7J upload=Q0EA94MVCD4P tag=T4PFXQ0SH2IC type=write size=2398877 2024-03-09T14:36:23.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.578+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=P41K24O0NRWA type=write size=47433 2024-03-09T14:36:23.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.582+0000 7f73315818c0 1 < completed write operation key=PXFZLDSAR1EG tag=M35JVN5URUSQ type=write size=1020095 2024-03-09T14:36:23.590 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.586+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=13C6DDX5E2HO type=del size=3394475 2024-03-09T14:36:23.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.590+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=BOMU38I7RBE5 type=del 2024-03-09T14:36:23.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.594+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=5RYL3GXPOPU5 type=del size=1518948 2024-03-09T14:36:23.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.598+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~AVTNCUQD0PO2.0 size=1048576 2024-03-09T14:36:23.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.602+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~AVTNCUQD0PO2.1 size=744213 2024-03-09T14:36:23.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.606+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=AVTNCUQD0PO2 tag=IN1I4JLJ12OF type=write size=2841365 2024-03-09T14:36:23.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.610+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=37NAWGCQ5QQR type=del size=1244889 2024-03-09T14:36:23.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.614+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~TKW8YKEMS8G1.0 size=1048576 2024-03-09T14:36:23.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.614+0000 7f73315818c0 1 < canceled part key=_multipart_Y8EQAHMTL66U.2~TKW8YKEMS8G1.1 size=1048576 2024-03-09T14:36:23.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.614+0000 7f73315818c0 1 < canceled part key=_multipart_Y8EQAHMTL66U.2~TKW8YKEMS8G1.2 size=192351 2024-03-09T14:36:23.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.622+0000 7f73315818c0 1 < canceled multipart upload key=Y8EQAHMTL66U upload=TKW8YKEMS8G1 tag=YAHOGJVDF2H2 type=write size=3338079 2024-03-09T14:36:23.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.622+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=LP8ONQ34S95X type=write size=429221 2024-03-09T14:36:23.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.626+0000 7f73315818c0 1 < completed write operation key=MSMD3ANK4T7J tag=H310KV85W1QT type=write size=1042305 2024-03-09T14:36:23.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.630+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=OHEZE4B1HWI3 type=write size=1033697 2024-03-09T14:36:23.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.634+0000 7f73315818c0 1 < canceled operation key=K3JNN1NCKBXN tag=M21SJI0V48DE type=write size=667131 2024-03-09T14:36:23.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.638+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=QRDP00X75PS4 type=write size=967774 2024-03-09T14:36:23.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.642+0000 7f73315818c0 1 < completed write operation key=4P89277EEYJ3 tag=F1PS9RB2BKWE type=write size=813044 2024-03-09T14:36:23.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.646+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~I40RMMQJBBZW.0 size=359352 2024-03-09T14:36:23.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.646+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=I40RMMQJBBZW tag=1P7GJGUJ72VE type=write size=1407928 2024-03-09T14:36:23.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.650+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=K4BVWXUB9UFP type=del 2024-03-09T14:36:23.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.658+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~NJQ5O938PCID.0 size=731899 2024-03-09T14:36:23.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.658+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=NJQ5O938PCID tag=2HXZNJCGBH6M type=write size=1780475 2024-03-09T14:36:23.664 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.662+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=D2UJOYURUB4B type=del 2024-03-09T14:36:23.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.666+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=D2J5U5UI3VIG type=write size=654433 2024-03-09T14:36:23.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.666+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~6MWITSEJRKJZ.0 size=697130 2024-03-09T14:36:23.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.674+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=6MWITSEJRKJZ tag=ITLNQ2S86U1U type=write size=1745706 2024-03-09T14:36:23.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.678+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=0LVCEXIH1QM8 type=del size=39095 2024-03-09T14:36:23.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.678+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=22RH3RFAGPDJ type=del 2024-03-09T14:36:23.689 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.686+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~N2CC0VHXBJSJ.0 size=669382 2024-03-09T14:36:23.689 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.686+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=N2CC0VHXBJSJ tag=C9XQ7T0W0VHI type=write size=1717958 2024-03-09T14:36:23.693 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.690+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=EH8ANNVONSS1 type=del 2024-03-09T14:36:23.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.694+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~56UK32Z70UQH.0 size=1048576 2024-03-09T14:36:23.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.698+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~56UK32Z70UQH.1 size=33872 2024-03-09T14:36:23.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.698+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=56UK32Z70UQH tag=S03TT5YR8ITU type=write size=2131024 2024-03-09T14:36:23.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.702+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~58BVNC428I2I.0 size=330082 2024-03-09T14:36:23.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.706+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=58BVNC428I2I tag=LEECCZ51SRIE type=write size=1378658 2024-03-09T14:36:23.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.710+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=EL5TBI96IYXQ type=del size=1877842 2024-03-09T14:36:23.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.710+0000 7f73315818c0 1 < canceled part key=_multipart_FYFVW9R0M1PX.2~ZDNP9IQUWW9Y.0 size=941791 2024-03-09T14:36:23.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.714+0000 7f73315818c0 1 < canceled multipart upload key=FYFVW9R0M1PX upload=ZDNP9IQUWW9Y tag=SA8G1QZIQYLN type=write size=1990367 2024-03-09T14:36:23.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.718+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=Q00V5JKBTCLP type=del size=3739152 2024-03-09T14:36:23.725 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.722+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=JYGQ5CT4MBHV type=del 2024-03-09T14:36:23.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.726+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=TV7UQBLBG9BN type=del size=3879664 2024-03-09T14:36:23.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.730+0000 7f73315818c0 1 < completed write operation key=MD7FXGYXB823 tag=P41K24O0NRWA type=write size=47433 2024-03-09T14:36:23.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.730+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=DXTQWESJ5T1G type=del size=454327 2024-03-09T14:36:23.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.734+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=13C6DDX5E2HO type=del 2024-03-09T14:36:23.742 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.738+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=AZTYIW1NR2GP type=write size=596865 2024-03-09T14:36:23.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.742+0000 7f73315818c0 1 < canceled operation key=W6KVP8EQFQ8J tag=5RYL3GXPOPU5 type=del size=1518948 2024-03-09T14:36:23.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.746+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=70AEA3L163SD type=del size=547714 2024-03-09T14:36:23.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.750+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=37NAWGCQ5QQR type=del 2024-03-09T14:36:23.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.754+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=QRQW8TRN879D type=write size=903239 2024-03-09T14:36:23.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.758+0000 7f73315818c0 1 < suggested remove operation key=Y8EQAHMTL66U tag=LP8ONQ34S95X type=write 2024-03-09T14:36:23.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.766+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=Z7FE0RBB153P type=del size=440333 2024-03-09T14:36:23.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.770+0000 7f73315818c0 1 < canceled operation key=XELG1AWATMVL tag=OHEZE4B1HWI3 type=write size=1033697 2024-03-09T14:36:23.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.778+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~U44UMKREOSKO.0 size=1047439 2024-03-09T14:36:23.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.778+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=U44UMKREOSKO tag=ORAG682BNDPE type=write size=2096015 2024-03-09T14:36:23.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.778+0000 7f73315818c0 1 < completed write operation key=DQOY1JJ1YQ2W tag=QRDP00X75PS4 type=write size=967774 2024-03-09T14:36:23.786 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.782+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=DLP8VI4609SS type=del size=1462902 2024-03-09T14:36:23.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.786+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~I40RMMQJBBZW.0 size=359352 2024-03-09T14:36:23.796 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.794+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=I40RMMQJBBZW tag=1P7GJGUJ72VE type=write size=1407928 2024-03-09T14:36:23.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.798+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=TIXFJBMWPBC0 type=del size=2634718 2024-03-09T14:36:23.803 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.802+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~NJQ5O938PCID.0 size=731899 2024-03-09T14:36:23.809 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.806+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=NJQ5O938PCID tag=2HXZNJCGBH6M type=write size=1780475 2024-03-09T14:36:23.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.810+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~8LVLZF42CVUG.0 size=1048576 2024-03-09T14:36:23.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.814+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~8LVLZF42CVUG.1 size=1048576 2024-03-09T14:36:23.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.818+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~8LVLZF42CVUG.2 size=799938 2024-03-09T14:36:23.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.818+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=8LVLZF42CVUG tag=EBLC2TF6B2TY type=write size=3945666 2024-03-09T14:36:23.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.822+0000 7f73315818c0 1 < completed write operation key=MSMD3ANK4T7J tag=D2J5U5UI3VIG type=write size=654433 2024-03-09T14:36:23.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.830+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~UBNU9EVP7MI2.0 size=383855 2024-03-09T14:36:23.832 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.830+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=UBNU9EVP7MI2 tag=60QXBCB1NX2Z type=write size=1432431 2024-03-09T14:36:23.835 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.830+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=0LVCEXIH1QM8 type=del 2024-03-09T14:36:23.838 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.834+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=ZN2ZX5DH0MAF type=del size=328452 2024-03-09T14:36:23.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.838+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~N2CC0VHXBJSJ.0 size=669382 2024-03-09T14:36:23.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.846+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=N2CC0VHXBJSJ tag=C9XQ7T0W0VHI type=write size=1717958 2024-03-09T14:36:23.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.846+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=H6V8414AG54P type=write size=609194 2024-03-09T14:36:23.854 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.850+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~56UK32Z70UQH.0 size=1048576 2024-03-09T14:36:23.857 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.854+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~56UK32Z70UQH.1 size=33872 2024-03-09T14:36:23.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.858+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=56UK32Z70UQH tag=S03TT5YR8ITU type=write size=2131024 2024-03-09T14:36:23.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.866+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~31UL22U8QO99.0 size=1048576 2024-03-09T14:36:23.871 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.870+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~31UL22U8QO99.1 size=401288 2024-03-09T14:36:23.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.870+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=31UL22U8QO99 tag=0WJAV7A6ZRUW type=write size=2498440 2024-03-09T14:36:23.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.870+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=EL5TBI96IYXQ type=del 2024-03-09T14:36:23.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.874+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=ZW9BN7J0UY3R type=del size=1894314 2024-03-09T14:36:23.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.878+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=Q00V5JKBTCLP type=del 2024-03-09T14:36:23.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.882+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=APLJQH77SYMD type=del size=430459 2024-03-09T14:36:23.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.886+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=TV7UQBLBG9BN type=del 2024-03-09T14:36:23.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.890+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=OXHRYRHTK628 type=del size=3010326 2024-03-09T14:36:23.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.894+0000 7f73315818c0 1 < suggested update operation key=DQOY1JJ1YQ2W tag=DXTQWESJ5T1G type=del size=454327 2024-03-09T14:36:23.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.902+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~W3URUH2HEV90.0 size=1048576 2024-03-09T14:36:23.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.906+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~W3URUH2HEV90.1 size=1048576 2024-03-09T14:36:23.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.906+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~W3URUH2HEV90.2 size=254104 2024-03-09T14:36:23.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.906+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=W3URUH2HEV90 tag=CF4UWUZPXV3L type=write size=3399832 2024-03-09T14:36:23.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.910+0000 7f73315818c0 1 < completed write operation key=W6KVP8EQFQ8J tag=AZTYIW1NR2GP type=write size=596865 2024-03-09T14:36:23.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.914+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=DGPKX82IA1KX type=del size=3188483 2024-03-09T14:36:23.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.918+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=70AEA3L163SD type=del 2024-03-09T14:36:23.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.922+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=4G44SSPANTW3 type=del size=2912277 2024-03-09T14:36:23.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.926+0000 7f73315818c0 1 < completed write operation key=9B0RAHZZGLYU tag=QRQW8TRN879D type=write size=903239 2024-03-09T14:36:23.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.930+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=R9TW9LQ3ND1F type=del size=34779 2024-03-09T14:36:23.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.934+0000 7f73315818c0 1 < canceled operation key=1NYNL8M5DVER tag=Z7FE0RBB153P type=del size=440333 2024-03-09T14:36:23.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.938+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~N36N4ZBD21Y2.0 size=1048576 2024-03-09T14:36:23.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.942+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~N36N4ZBD21Y2.1 size=1048576 2024-03-09T14:36:23.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.946+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~N36N4ZBD21Y2.2 size=747435 2024-03-09T14:36:23.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.946+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=N36N4ZBD21Y2 tag=AL59HQYV1NMW type=write size=3893163 2024-03-09T14:36:23.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.950+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~U44UMKREOSKO.0 size=1047439 2024-03-09T14:36:23.958 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.954+0000 7f73315818c0 1 < completed multipart upload key=IA0AB7UVLFEJ upload=U44UMKREOSKO tag=ORAG682BNDPE type=write size=2096015 2024-03-09T14:36:23.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.962+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~9FI0HETG7JH1.0 size=1048576 2024-03-09T14:36:23.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.966+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~9FI0HETG7JH1.1 size=858801 2024-03-09T14:36:23.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.966+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=9FI0HETG7JH1 tag=J3B2HPE0LLAC type=write size=2955953 2024-03-09T14:36:23.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.966+0000 7f73315818c0 -1 < failed to complete operation key=DQOY1JJ1YQ2W tag=DLP8VI4609SS type=del size=1462902: (22) Invalid argument 2024-03-09T14:36:23.977 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.974+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~8RCU7R6D66P7.0 size=1048576 2024-03-09T14:36:23.980 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.978+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~8RCU7R6D66P7.1 size=686946 2024-03-09T14:36:23.980 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.978+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=8RCU7R6D66P7 tag=VMULYGM2RW7O type=write size=2784098 2024-03-09T14:36:23.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.982+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=TIXFJBMWPBC0 type=del 2024-03-09T14:36:23.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.986+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=4I5MK87BYPMS type=del size=1745079 2024-03-09T14:36:23.988 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.986+0000 7f73315818c0 1 < canceled part key=_multipart_4E3QMBHEAEHO.2~8LVLZF42CVUG.0 size=1048576 2024-03-09T14:36:23.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.990+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~8LVLZF42CVUG.1 size=1048576 2024-03-09T14:36:23.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.990+0000 7f73315818c0 1 < canceled part key=_multipart_4E3QMBHEAEHO.2~8LVLZF42CVUG.2 size=799938 2024-03-09T14:36:23.997 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.994+0000 7f73315818c0 1 < canceled multipart upload key=4E3QMBHEAEHO upload=8LVLZF42CVUG tag=EBLC2TF6B2TY type=write size=3945666 2024-03-09T14:36:24.001 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:23.998+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=M61XNP1AZK56 type=del size=1488180 2024-03-09T14:36:24.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.002+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~UBNU9EVP7MI2.0 size=383855 2024-03-09T14:36:24.010 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.006+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=UBNU9EVP7MI2 tag=60QXBCB1NX2Z type=write size=1432431 2024-03-09T14:36:24.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.010+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=0P8V5XXASW2H type=del size=3784234 2024-03-09T14:36:24.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.018+0000 7f73315818c0 1 < suggested update operation key=UMSNM53HTY1J tag=ZN2ZX5DH0MAF type=del size=328452 2024-03-09T14:36:24.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.022+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=FX4Q42SVP8HF type=del size=2133032 2024-03-09T14:36:24.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.026+0000 7f73315818c0 1 < completed write operation key=9B0RAHZZGLYU tag=H6V8414AG54P type=write size=609194 2024-03-09T14:36:24.035 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.030+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~UFH2X0I6GLTI.0 size=1048576 2024-03-09T14:36:24.038 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.034+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~UFH2X0I6GLTI.1 size=1048576 2024-03-09T14:36:24.040 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.038+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~UFH2X0I6GLTI.2 size=494541 2024-03-09T14:36:24.041 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.038+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=UFH2X0I6GLTI tag=4HHH7YBOBWCM type=write size=3640269 2024-03-09T14:36:24.044 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.042+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~31UL22U8QO99.0 size=1048576 2024-03-09T14:36:24.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.042+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~31UL22U8QO99.1 size=401288 2024-03-09T14:36:24.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.050+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=31UL22U8QO99 tag=0WJAV7A6ZRUW type=write size=2498440 2024-03-09T14:36:24.056 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.054+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=IOKMFP4OUJW6 type=del size=2914662 2024-03-09T14:36:24.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.058+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=ZW9BN7J0UY3R type=del 2024-03-09T14:36:24.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.062+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=H2AAW5RYJJRZ type=write size=1009070 2024-03-09T14:36:24.067 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.062+0000 7f73315818c0 -1 < failed to complete operation key=DQOY1JJ1YQ2W tag=APLJQH77SYMD type=del size=430459: (22) Invalid argument 2024-03-09T14:36:24.073 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.070+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~6X3YWWGDPC92.0 size=1048576 2024-03-09T14:36:24.076 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.074+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~6X3YWWGDPC92.1 size=547994 2024-03-09T14:36:24.076 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.074+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=6X3YWWGDPC92 tag=03TQE9B11P2L type=write size=2645146 2024-03-09T14:36:24.080 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.078+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=OXHRYRHTK628 type=del 2024-03-09T14:36:24.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.082+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=BDU8RD3UQ3M4 type=del size=1811608 2024-03-09T14:36:24.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.082+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~W3URUH2HEV90.0 size=1048576 2024-03-09T14:36:24.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.086+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~W3URUH2HEV90.1 size=1048576 2024-03-09T14:36:24.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.090+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~W3URUH2HEV90.2 size=254104 2024-03-09T14:36:24.104 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.102+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=W3URUH2HEV90 tag=CF4UWUZPXV3L type=write size=3399832 2024-03-09T14:36:24.108 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.106+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=0OVHM0AFUSAG type=del size=3567322 2024-03-09T14:36:24.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.110+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=DGPKX82IA1KX type=del 2024-03-09T14:36:24.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.114+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~561Z8FD6AH99.0 size=1048576 2024-03-09T14:36:24.121 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.118+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~561Z8FD6AH99.1 size=898203 2024-03-09T14:36:24.121 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.118+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=561Z8FD6AH99 tag=9S6DTQR0ZMRM type=write size=2995355 2024-03-09T14:36:24.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.126+0000 7f73315818c0 1 < suggested remove operation key=LZC7NPF0GRO7 tag=4G44SSPANTW3 type=del 2024-03-09T14:36:24.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.130+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=VSGK5LF1M8YU type=del size=121356 2024-03-09T14:36:24.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.134+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=R9TW9LQ3ND1F type=del 2024-03-09T14:36:24.143 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.138+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~XR0J4V02QNON.0 size=1048576 2024-03-09T14:36:24.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.142+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~XR0J4V02QNON.1 size=1048576 2024-03-09T14:36:24.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.146+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~XR0J4V02QNON.2 size=508755 2024-03-09T14:36:24.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.146+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=XR0J4V02QNON tag=IQMA0AXGINP0 type=write size=3654483 2024-03-09T14:36:24.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.150+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~N36N4ZBD21Y2.0 size=1048576 2024-03-09T14:36:24.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.154+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~N36N4ZBD21Y2.1 size=1048576 2024-03-09T14:36:24.159 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.158+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~N36N4ZBD21Y2.2 size=747435 2024-03-09T14:36:24.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.162+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=N36N4ZBD21Y2 tag=AL59HQYV1NMW type=write size=3893163 2024-03-09T14:36:24.172 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.170+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~69SPU1KNP26A.0 size=38072 2024-03-09T14:36:24.172 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.170+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=69SPU1KNP26A tag=OUSN5N4OG332 type=write size=1086648 2024-03-09T14:36:24.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.174+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~9FI0HETG7JH1.0 size=1048576 2024-03-09T14:36:24.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.174+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~9FI0HETG7JH1.1 size=858801 2024-03-09T14:36:24.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.182+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=9FI0HETG7JH1 tag=J3B2HPE0LLAC type=write size=2955953 2024-03-09T14:36:24.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.186+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=D40NHB2FDI3J type=del size=1380850 2024-03-09T14:36:24.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.190+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~8RCU7R6D66P7.0 size=1048576 2024-03-09T14:36:24.196 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.190+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~8RCU7R6D66P7.1 size=686946 2024-03-09T14:36:24.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.198+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=8RCU7R6D66P7 tag=VMULYGM2RW7O type=write size=2784098 2024-03-09T14:36:24.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.202+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=DB3I4OVGG3VI type=write size=461861 2024-03-09T14:36:24.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.206+0000 7f73315818c0 1 < suggested update operation key=92GWY5B0K4SX tag=4I5MK87BYPMS type=del size=1745079 2024-03-09T14:36:24.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.210+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=UCVR5E97TXZ5 type=del size=1858546 2024-03-09T14:36:24.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.214+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=M61XNP1AZK56 type=del 2024-03-09T14:36:24.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.218+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=H45CWQAFDFVN type=write size=447913 2024-03-09T14:36:24.225 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.222+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=0P8V5XXASW2H type=del 2024-03-09T14:36:24.232 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.230+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~7K2ZYHT0Y5N1.0 size=1048576 2024-03-09T14:36:24.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.230+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~7K2ZYHT0Y5N1.1 size=759366 2024-03-09T14:36:24.234 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.230+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=7K2ZYHT0Y5N1 tag=CU5PJR9GC11U type=write size=2856518 2024-03-09T14:36:24.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.234+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=FX4Q42SVP8HF type=del 2024-03-09T14:36:24.244 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.242+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~2Q6NUOIT4IUG.0 size=1048576 2024-03-09T14:36:24.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.242+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~2Q6NUOIT4IUG.1 size=413296 2024-03-09T14:36:24.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.242+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=2Q6NUOIT4IUG tag=96WGCKYMP0IO type=write size=2510448 2024-03-09T14:36:24.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.246+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~UFH2X0I6GLTI.0 size=1048576 2024-03-09T14:36:24.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.250+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~UFH2X0I6GLTI.1 size=1048576 2024-03-09T14:36:24.255 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.254+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~UFH2X0I6GLTI.2 size=494541 2024-03-09T14:36:24.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.258+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=UFH2X0I6GLTI tag=4HHH7YBOBWCM type=write size=3640269 2024-03-09T14:36:24.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.262+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=YN3OV4N8HMAF type=del size=2965863 2024-03-09T14:36:24.271 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.266+0000 7f73315818c0 1 < suggested remove operation key=DQOY1JJ1YQ2W tag=IOKMFP4OUJW6 type=del 2024-03-09T14:36:24.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.274+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~1OTILS7313MY.0 size=1048576 2024-03-09T14:36:24.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.278+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~1OTILS7313MY.1 size=1048576 2024-03-09T14:36:24.282 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.278+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~1OTILS7313MY.2 size=257588 2024-03-09T14:36:24.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.278+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=1OTILS7313MY tag=K4RAUB02QW37 type=write size=3403316 2024-03-09T14:36:24.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.282+0000 7f73315818c0 1 < completed write operation key=P9LI5ESR3N9G tag=H2AAW5RYJJRZ type=write size=1009070 2024-03-09T14:36:24.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.286+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=I0JVKN3P1U7G type=del size=989402 2024-03-09T14:36:24.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.290+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~6X3YWWGDPC92.0 size=1048576 2024-03-09T14:36:24.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.290+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~6X3YWWGDPC92.1 size=547994 2024-03-09T14:36:24.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.298+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=6X3YWWGDPC92 tag=03TQE9B11P2L type=write size=2645146 2024-03-09T14:36:24.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.302+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~1BK3OHZXJU3K.0 size=22229 2024-03-09T14:36:24.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.302+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=1BK3OHZXJU3K tag=NDY3EGSR2JN2 type=write size=1070805 2024-03-09T14:36:24.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.306+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=BDU8RD3UQ3M4 type=del 2024-03-09T14:36:24.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.310+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=XMNIPYNP29XZ type=del size=135677 2024-03-09T14:36:24.316 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.314+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=0OVHM0AFUSAG type=del 2024-03-09T14:36:24.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.314+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=29NWWI4Y01X0 type=del size=1619165 2024-03-09T14:36:24.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.318+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~561Z8FD6AH99.0 size=1048576 2024-03-09T14:36:24.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.322+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~561Z8FD6AH99.1 size=898203 2024-03-09T14:36:24.331 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.326+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=561Z8FD6AH99 tag=9S6DTQR0ZMRM type=write size=2995355 2024-03-09T14:36:24.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.330+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=521EBI26OV4Z type=del size=76962 2024-03-09T14:36:24.337 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.334+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=VSGK5LF1M8YU type=del 2024-03-09T14:36:24.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.338+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=PII9ZEKPJHYB type=del size=577346 2024-03-09T14:36:24.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.342+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~XR0J4V02QNON.0 size=1048576 2024-03-09T14:36:24.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.342+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~XR0J4V02QNON.1 size=1048576 2024-03-09T14:36:24.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.346+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~XR0J4V02QNON.2 size=508755 2024-03-09T14:36:24.354 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.350+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=XR0J4V02QNON tag=IQMA0AXGINP0 type=write size=3654483 2024-03-09T14:36:24.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.358+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~09W4223R46U5.0 size=1048576 2024-03-09T14:36:24.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.358+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~09W4223R46U5.1 size=1048576 2024-03-09T14:36:24.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.362+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~09W4223R46U5.2 size=782727 2024-03-09T14:36:24.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.362+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=09W4223R46U5 tag=IZNLH2I4QK7U type=write size=3928455 2024-03-09T14:36:24.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.366+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~69SPU1KNP26A.0 size=38072 2024-03-09T14:36:24.373 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.370+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=69SPU1KNP26A tag=OUSN5N4OG332 type=write size=1086648 2024-03-09T14:36:24.379 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.374+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~3R60MJRNWAGA.0 size=1048576 2024-03-09T14:36:24.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.378+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~3R60MJRNWAGA.1 size=347502 2024-03-09T14:36:24.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.378+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=3R60MJRNWAGA tag=O24VMRX7EZ56 type=write size=2444654 2024-03-09T14:36:24.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.382+0000 7f73315818c0 -1 < failed to complete operation key=DQOY1JJ1YQ2W tag=D40NHB2FDI3J type=del size=1380850: (22) Invalid argument 2024-03-09T14:36:24.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.386+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=RHNCJ8I45248 type=write size=715892 2024-03-09T14:36:24.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.390+0000 7f73315818c0 1 < canceled operation key=LZC7NPF0GRO7 tag=DB3I4OVGG3VI type=write size=461861 2024-03-09T14:36:24.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.390+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=1PUM7874Y8D0 type=del size=1293220 2024-03-09T14:36:24.398 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.394+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=UCVR5E97TXZ5 type=del 2024-03-09T14:36:24.404 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.402+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~8YY1SZEF7AMN.0 size=1048576 2024-03-09T14:36:24.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.402+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~8YY1SZEF7AMN.1 size=581181 2024-03-09T14:36:24.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.402+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=8YY1SZEF7AMN tag=23J9WZO7Y5U3 type=write size=2678333 2024-03-09T14:36:24.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.406+0000 7f73315818c0 1 < completed write operation key=FYFVW9R0M1PX tag=H45CWQAFDFVN type=write size=447913 2024-03-09T14:36:24.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.414+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~OTAYEUQW3F1L.0 size=444857 2024-03-09T14:36:24.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.414+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=OTAYEUQW3F1L tag=IMEIN12XI2FA type=write size=1493433 2024-03-09T14:36:24.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.418+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~7K2ZYHT0Y5N1.0 size=1048576 2024-03-09T14:36:24.424 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.422+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~7K2ZYHT0Y5N1.1 size=759366 2024-03-09T14:36:24.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.426+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=7K2ZYHT0Y5N1 tag=CU5PJR9GC11U type=write size=2856518 2024-03-09T14:36:24.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.430+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=J2UT4X9YH19Q type=del size=1597461 2024-03-09T14:36:24.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.434+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~2Q6NUOIT4IUG.0 size=1048576 2024-03-09T14:36:24.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.434+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~2Q6NUOIT4IUG.1 size=413296 2024-03-09T14:36:24.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.438+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=2Q6NUOIT4IUG tag=96WGCKYMP0IO type=write size=2510448 2024-03-09T14:36:24.449 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.446+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~EQKMGZU9NW5W.0 size=1048576 2024-03-09T14:36:24.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.446+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~EQKMGZU9NW5W.1 size=1048576 2024-03-09T14:36:24.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.450+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~EQKMGZU9NW5W.2 size=611613 2024-03-09T14:36:24.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.450+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=EQKMGZU9NW5W tag=RYZ36DYTNSEA type=write size=3757341 2024-03-09T14:36:24.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.454+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=YN3OV4N8HMAF type=del 2024-03-09T14:36:24.460 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.458+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=O432EO8ZBO1L type=del size=2607350 2024-03-09T14:36:24.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.462+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~1OTILS7313MY.0 size=1048576 2024-03-09T14:36:24.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.462+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~1OTILS7313MY.1 size=1048576 2024-03-09T14:36:24.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.466+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~1OTILS7313MY.2 size=257588 2024-03-09T14:36:24.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.470+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=1OTILS7313MY tag=K4RAUB02QW37 type=write size=3403316 2024-03-09T14:36:24.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.474+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=8PAWBAX51ZKG type=del size=519485 2024-03-09T14:36:24.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.478+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=I0JVKN3P1U7G type=del 2024-03-09T14:36:24.526 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.478+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=DKKKOSS98E9K type=del size=2056015 2024-03-09T14:36:24.526 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.482+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~1BK3OHZXJU3K.0 size=22229 2024-03-09T14:36:24.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.486+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=1BK3OHZXJU3K tag=NDY3EGSR2JN2 type=write size=1070805 2024-03-09T14:36:24.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.494+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~VVAO9CC1YST9.0 size=1026583 2024-03-09T14:36:24.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.494+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=VVAO9CC1YST9 tag=DS2QA2D40W97 type=write size=2075159 2024-03-09T14:36:24.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.498+0000 7f73315818c0 1 < suggested update operation key=MD7FXGYXB823 tag=XMNIPYNP29XZ type=del size=135677 2024-03-09T14:36:24.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.506+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~Z6FUAHHNH4UP.0 size=1048576 2024-03-09T14:36:24.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.506+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~Z6FUAHHNH4UP.1 size=1048576 2024-03-09T14:36:24.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.510+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~Z6FUAHHNH4UP.2 size=310260 2024-03-09T14:36:24.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.510+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=Z6FUAHHNH4UP tag=RIVZOLT0QIAI type=write size=3455988 2024-03-09T14:36:24.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.514+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=29NWWI4Y01X0 type=del 2024-03-09T14:36:24.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.518+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=QLWFYYKTHXRL type=del size=3519487 2024-03-09T14:36:24.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.518+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=521EBI26OV4Z type=del 2024-03-09T14:36:24.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.522+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=3EKQPQKO2I1M type=del size=927435 2024-03-09T14:36:24.528 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.526+0000 7f73315818c0 1 < canceled operation key=PXFZLDSAR1EG tag=PII9ZEKPJHYB type=del size=577346 2024-03-09T14:36:24.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.530+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=2H0HYBJ1YK69 type=del size=2583954 2024-03-09T14:36:24.535 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.530+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~09W4223R46U5.0 size=1048576 2024-03-09T14:36:24.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.534+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~09W4223R46U5.1 size=1048576 2024-03-09T14:36:24.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.538+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~09W4223R46U5.2 size=782727 2024-03-09T14:36:24.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.542+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=09W4223R46U5 tag=IZNLH2I4QK7U type=write size=3928455 2024-03-09T14:36:24.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.546+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=B3OUK3DIDEML type=del size=1734603 2024-03-09T14:36:24.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.550+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~3R60MJRNWAGA.0 size=1048576 2024-03-09T14:36:24.556 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.554+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~3R60MJRNWAGA.1 size=347502 2024-03-09T14:36:24.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.558+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=3R60MJRNWAGA tag=O24VMRX7EZ56 type=write size=2444654 2024-03-09T14:36:24.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.562+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=9RJMXSOQVD0E type=del size=2294402 2024-03-09T14:36:24.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.566+0000 7f73315818c0 1 < completed write operation key=M8F9TA80VME6 tag=RHNCJ8I45248 type=write size=715892 2024-03-09T14:36:24.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.570+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=K7FZFOH1M6DS type=del size=3758499 2024-03-09T14:36:24.581 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.578+0000 7f73315818c0 1 < suggested remove operation key=9B0RAHZZGLYU tag=1PUM7874Y8D0 type=del 2024-03-09T14:36:24.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.582+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~G7QZZDBBTX4V.0 size=1048576 2024-03-09T14:36:24.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.586+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~G7QZZDBBTX4V.1 size=721022 2024-03-09T14:36:24.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.586+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=G7QZZDBBTX4V tag=P2SF720KLZFW type=write size=2818174 2024-03-09T14:36:24.593 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.590+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~8YY1SZEF7AMN.0 size=1048576 2024-03-09T14:36:24.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.594+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~8YY1SZEF7AMN.1 size=581181 2024-03-09T14:36:24.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.598+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=8YY1SZEF7AMN tag=23J9WZO7Y5U3 type=write size=2678333 2024-03-09T14:36:24.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.606+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~KV86AA28XDTG.0 size=1048576 2024-03-09T14:36:24.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.606+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~KV86AA28XDTG.1 size=1048576 2024-03-09T14:36:24.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.610+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~KV86AA28XDTG.2 size=854844 2024-03-09T14:36:24.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.610+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=KV86AA28XDTG tag=RORE8TKJ3MB7 type=write size=4000572 2024-03-09T14:36:24.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.614+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~OTAYEUQW3F1L.0 size=444857 2024-03-09T14:36:24.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.622+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=OTAYEUQW3F1L tag=IMEIN12XI2FA type=write size=1493433 2024-03-09T14:36:24.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.626+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~QHR4HWDXJJA8.0 size=1048576 2024-03-09T14:36:24.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.630+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~QHR4HWDXJJA8.1 size=751283 2024-03-09T14:36:24.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.630+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=QHR4HWDXJJA8 tag=QQENONRL2FE5 type=write size=2848435 2024-03-09T14:36:24.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.634+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=J2UT4X9YH19Q type=del 2024-03-09T14:36:24.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.638+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=V09K30J38RJO type=del size=2731477 2024-03-09T14:36:24.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.638+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~EQKMGZU9NW5W.0 size=1048576 2024-03-09T14:36:24.646 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.642+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~EQKMGZU9NW5W.1 size=1048576 2024-03-09T14:36:24.646 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.642+0000 7f73315818c0 1 < canceled part key=_multipart_0IB00CZ92VV4.2~EQKMGZU9NW5W.2 size=611613 2024-03-09T14:36:24.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.650+0000 7f73315818c0 1 < canceled multipart upload key=0IB00CZ92VV4 upload=EQKMGZU9NW5W tag=RYZ36DYTNSEA type=write size=3757341 2024-03-09T14:36:24.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.654+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~7MZNTOFQPM7Q.0 size=1048576 2024-03-09T14:36:24.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.658+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~7MZNTOFQPM7Q.1 size=958991 2024-03-09T14:36:24.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.658+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=7MZNTOFQPM7Q tag=BHXGI7Z3EX2X type=write size=3056143 2024-03-09T14:36:24.665 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.662+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=O432EO8ZBO1L type=del 2024-03-09T14:36:24.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.666+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~U6U30RLNS4XR.0 size=1048576 2024-03-09T14:36:24.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.670+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~U6U30RLNS4XR.1 size=1048576 2024-03-09T14:36:24.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.674+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~U6U30RLNS4XR.2 size=996171 2024-03-09T14:36:24.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.674+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=U6U30RLNS4XR tag=2DORPBX7QZPZ type=write size=4141899 2024-03-09T14:36:24.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.678+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=8PAWBAX51ZKG type=del 2024-03-09T14:36:24.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.682+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=7V063VHQJ9RD type=del size=238018 2024-03-09T14:36:24.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.686+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=DKKKOSS98E9K type=del 2024-03-09T14:36:24.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.686+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=QTR6ZKQVJU4C type=del size=1423878 2024-03-09T14:36:24.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.690+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~VVAO9CC1YST9.0 size=1026583 2024-03-09T14:36:24.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.698+0000 7f73315818c0 1 < completed multipart upload key=LZC7NPF0GRO7 upload=VVAO9CC1YST9 tag=DS2QA2D40W97 type=write size=2075159 2024-03-09T14:36:24.703 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.698+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=JBDV8SQKUG2S type=write size=508351 2024-03-09T14:36:24.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.702+0000 7f73315818c0 1 < canceled part key=_multipart_0WGGMHTEF2LB.2~Z6FUAHHNH4UP.0 size=1048576 2024-03-09T14:36:24.706 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.702+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~Z6FUAHHNH4UP.1 size=1048576 2024-03-09T14:36:24.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.706+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~Z6FUAHHNH4UP.2 size=310260 2024-03-09T14:36:24.715 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.710+0000 7f73315818c0 1 < canceled multipart upload key=0WGGMHTEF2LB upload=Z6FUAHHNH4UP tag=RIVZOLT0QIAI type=write size=3455988 2024-03-09T14:36:24.725 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.722+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~H6GFN2H1TSLV.0 size=80405 2024-03-09T14:36:24.726 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.722+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=H6GFN2H1TSLV tag=MRYH82WQPMNJ type=write size=1128981 2024-03-09T14:36:24.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.730+0000 7f73315818c0 1 < suggested update operation key=MSMD3ANK4T7J tag=QLWFYYKTHXRL type=del size=3519487 2024-03-09T14:36:24.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.734+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=WFJLOZ14K0K9 type=del size=453949 2024-03-09T14:36:24.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.738+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=3EKQPQKO2I1M type=del 2024-03-09T14:36:24.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.742+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~19KVG9GT2W7P.0 size=1048576 2024-03-09T14:36:24.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.746+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~19KVG9GT2W7P.1 size=604306 2024-03-09T14:36:24.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.746+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=19KVG9GT2W7P tag=HG3AD8TGO424 type=write size=2701458 2024-03-09T14:36:24.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.750+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=2H0HYBJ1YK69 type=del 2024-03-09T14:36:24.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.754+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~0MK3OE03WYPS.0 size=1048576 2024-03-09T14:36:24.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.758+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~0MK3OE03WYPS.1 size=1048576 2024-03-09T14:36:24.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.762+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~0MK3OE03WYPS.2 size=147782 2024-03-09T14:36:24.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.762+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=0MK3OE03WYPS tag=USZXN7WM8P1H type=write size=3293510 2024-03-09T14:36:24.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.766+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=B3OUK3DIDEML type=del 2024-03-09T14:36:24.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.766+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=271RQJ2GBK1F type=del size=1228753 2024-03-09T14:36:24.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.774+0000 7f73315818c0 1 < suggested update operation key=XELG1AWATMVL tag=9RJMXSOQVD0E type=del size=2294402 2024-03-09T14:36:24.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.778+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=QTW9JF8IN4US type=del size=3903291 2024-03-09T14:36:24.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.782+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=K7FZFOH1M6DS type=del 2024-03-09T14:36:24.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.786+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~53CGDI4OPJMR.0 size=440103 2024-03-09T14:36:24.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.786+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=53CGDI4OPJMR tag=CG5XGB24108H type=write size=1488679 2024-03-09T14:36:24.794 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.790+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~G7QZZDBBTX4V.0 size=1048576 2024-03-09T14:36:24.797 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.794+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~G7QZZDBBTX4V.1 size=721022 2024-03-09T14:36:24.803 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.798+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=G7QZZDBBTX4V tag=P2SF720KLZFW type=write size=2818174 2024-03-09T14:36:24.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.802+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=WRNW8LRAWD35 type=del size=1385049 2024-03-09T14:36:24.812 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.810+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~KV86AA28XDTG.0 size=1048576 2024-03-09T14:36:24.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.810+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~KV86AA28XDTG.1 size=1048576 2024-03-09T14:36:24.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.814+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~KV86AA28XDTG.2 size=854844 2024-03-09T14:36:24.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.822+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=KV86AA28XDTG tag=RORE8TKJ3MB7 type=write size=4000572 2024-03-09T14:36:24.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.826+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=AN0JBW7TO9AT type=del size=2849291 2024-03-09T14:36:24.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.826+0000 7f73315818c0 1 < canceled part key=_multipart_4E3QMBHEAEHO.2~QHR4HWDXJJA8.0 size=1048576 2024-03-09T14:36:24.832 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.830+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~QHR4HWDXJJA8.1 size=751283 2024-03-09T14:36:24.838 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.834+0000 7f73315818c0 1 < canceled multipart upload key=4E3QMBHEAEHO upload=QHR4HWDXJJA8 tag=QQENONRL2FE5 type=write size=2848435 2024-03-09T14:36:24.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.838+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=XQ3RKNZ68XHI type=del size=807049 2024-03-09T14:36:24.846 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.842+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=V09K30J38RJO type=del 2024-03-09T14:36:24.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.846+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=VD24X5F1MJ2M type=del size=967567 2024-03-09T14:36:24.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.850+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~7MZNTOFQPM7Q.0 size=1048576 2024-03-09T14:36:24.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.854+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~7MZNTOFQPM7Q.1 size=958991 2024-03-09T14:36:24.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.858+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=7MZNTOFQPM7Q tag=BHXGI7Z3EX2X type=write size=3056143 2024-03-09T14:36:24.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.862+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=MWFZLPIS8ZPM type=del size=147260 2024-03-09T14:36:24.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.866+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~U6U30RLNS4XR.0 size=1048576 2024-03-09T14:36:24.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.870+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~U6U30RLNS4XR.1 size=1048576 2024-03-09T14:36:24.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.874+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~U6U30RLNS4XR.2 size=996171 2024-03-09T14:36:24.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.878+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=U6U30RLNS4XR tag=2DORPBX7QZPZ type=write size=4141899 2024-03-09T14:36:24.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.886+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~IWSNPO3NGYR7.0 size=64333 2024-03-09T14:36:24.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.886+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=IWSNPO3NGYR7 tag=MHY65K2XZ36F type=write size=1112909 2024-03-09T14:36:24.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.890+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=7V063VHQJ9RD type=del 2024-03-09T14:36:24.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.890+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=ML2VPXZIL7SW type=del size=1817203 2024-03-09T14:36:24.898 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.894+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=QTR6ZKQVJU4C type=del 2024-03-09T14:36:24.904 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.902+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~NYPRCGU0P7CT.0 size=1048576 2024-03-09T14:36:24.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.902+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~NYPRCGU0P7CT.1 size=449176 2024-03-09T14:36:24.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.902+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=NYPRCGU0P7CT tag=YTLIZNKRI3DO type=write size=2546328 2024-03-09T14:36:24.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.906+0000 7f73315818c0 1 < completed write operation key=K3JNN1NCKBXN tag=JBDV8SQKUG2S type=write size=508351 2024-03-09T14:36:24.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.910+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=JQXNY7DYCEPX type=del size=2849001 2024-03-09T14:36:24.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.914+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~H6GFN2H1TSLV.0 size=80405 2024-03-09T14:36:24.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.922+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=H6GFN2H1TSLV tag=MRYH82WQPMNJ type=write size=1128981 2024-03-09T14:36:24.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.926+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~PINEW8PFFT1H.0 size=262037 2024-03-09T14:36:24.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.926+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=PINEW8PFFT1H tag=MPRY6T9JY1CU type=write size=1310613 2024-03-09T14:36:24.934 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.930+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=WFJLOZ14K0K9 type=del 2024-03-09T14:36:24.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.938+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~Z9OTFVZK1OII.0 size=1048576 2024-03-09T14:36:24.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.938+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~Z9OTFVZK1OII.1 size=433126 2024-03-09T14:36:24.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.938+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=Z9OTFVZK1OII tag=ZKHDDUHT82A9 type=write size=2530278 2024-03-09T14:36:24.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.942+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~19KVG9GT2W7P.0 size=1048576 2024-03-09T14:36:24.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.946+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~19KVG9GT2W7P.1 size=604306 2024-03-09T14:36:24.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.950+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=19KVG9GT2W7P tag=HG3AD8TGO424 type=write size=2701458 2024-03-09T14:36:24.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.954+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=4ET0TGIMTVA3 type=del size=3416076 2024-03-09T14:36:24.960 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.958+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~0MK3OE03WYPS.0 size=1048576 2024-03-09T14:36:24.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.958+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~0MK3OE03WYPS.1 size=1048576 2024-03-09T14:36:24.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.962+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~0MK3OE03WYPS.2 size=147782 2024-03-09T14:36:24.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.970+0000 7f73315818c0 1 < completed multipart upload key=LZC7NPF0GRO7 upload=0MK3OE03WYPS tag=USZXN7WM8P1H type=write size=3293510 2024-03-09T14:36:24.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.974+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~A6B3HLTLDNRC.0 size=1048576 2024-03-09T14:36:24.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.978+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~A6B3HLTLDNRC.1 size=1048576 2024-03-09T14:36:24.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.982+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~A6B3HLTLDNRC.2 size=789997 2024-03-09T14:36:24.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.982+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=A6B3HLTLDNRC tag=W2KZXDEHUAZ1 type=write size=3935725 2024-03-09T14:36:24.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.986+0000 7f73315818c0 1 < canceled operation key=0WGGMHTEF2LB tag=271RQJ2GBK1F type=del size=1228753 2024-03-09T14:36:24.993 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.990+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=PVJPFEF2JO47 type=del size=4250 2024-03-09T14:36:24.997 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:24.994+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=QTW9JF8IN4US type=del 2024-03-09T14:36:25.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.002+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~M4USJ07YZS6N.0 size=1048576 2024-03-09T14:36:25.007 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.002+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~M4USJ07YZS6N.1 size=409814 2024-03-09T14:36:25.007 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.002+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=M4USJ07YZS6N tag=Y14LWO8TJECV type=write size=2506966 2024-03-09T14:36:25.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.006+0000 7f73315818c0 1 < canceled part key=_multipart_DQOY1JJ1YQ2W.2~53CGDI4OPJMR.0 size=440103 2024-03-09T14:36:25.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.010+0000 7f73315818c0 1 < canceled multipart upload key=DQOY1JJ1YQ2W upload=53CGDI4OPJMR tag=CG5XGB24108H type=write size=1488679 2024-03-09T14:36:25.016 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.014+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=IOFH2P9WJ1EM type=del size=3097864 2024-03-09T14:36:25.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.018+0000 7f73315818c0 1 < suggested remove operation key=XELG1AWATMVL tag=WRNW8LRAWD35 type=del 2024-03-09T14:36:25.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.026+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~APCEQ6ZR9RJ1.0 size=1048576 2024-03-09T14:36:25.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.030+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~APCEQ6ZR9RJ1.1 size=1048576 2024-03-09T14:36:25.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.030+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~APCEQ6ZR9RJ1.2 size=266664 2024-03-09T14:36:25.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.030+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=APCEQ6ZR9RJ1 tag=EF43XRXL83E1 type=write size=3412392 2024-03-09T14:36:25.038 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.034+0000 7f73315818c0 1 < canceled operation key=92GWY5B0K4SX tag=AN0JBW7TO9AT type=del size=2849291 2024-03-09T14:36:25.043 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.042+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~85J292FK3VYP.0 size=1048576 2024-03-09T14:36:25.050 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.046+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~85J292FK3VYP.1 size=166968 2024-03-09T14:36:25.050 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.046+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=85J292FK3VYP tag=JA413HD47WO7 type=write size=2264120 2024-03-09T14:36:25.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.050+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=XQ3RKNZ68XHI type=del 2024-03-09T14:36:25.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.054+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=OHGMRSE3AFMF type=del size=320056 2024-03-09T14:36:25.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.058+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=VD24X5F1MJ2M type=del 2024-03-09T14:36:25.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.062+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~TDYVC8ILWQG1.0 size=288249 2024-03-09T14:36:25.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.062+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=TDYVC8ILWQG1 tag=SVCHWCQFIACK type=write size=1336825 2024-03-09T14:36:25.070 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.066+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=MWFZLPIS8ZPM type=del 2024-03-09T14:36:25.076 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.074+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~V0K6KI378J4J.0 size=1048576 2024-03-09T14:36:25.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.074+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~V0K6KI378J4J.1 size=481215 2024-03-09T14:36:25.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.074+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=V0K6KI378J4J tag=0VXM7813OK0H type=write size=2578367 2024-03-09T14:36:25.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.078+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~IWSNPO3NGYR7.0 size=64333 2024-03-09T14:36:25.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.086+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=IWSNPO3NGYR7 tag=MHY65K2XZ36F type=write size=1112909 2024-03-09T14:36:25.094 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.090+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~SPGOPGET1QZS.0 size=902686 2024-03-09T14:36:25.094 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.090+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=SPGOPGET1QZS tag=9VJVNCV76YUU type=write size=1951262 2024-03-09T14:36:25.097 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.094+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=ML2VPXZIL7SW type=del 2024-03-09T14:36:25.101 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.098+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=ZB7R7L92OLKT type=del size=2480031 2024-03-09T14:36:25.104 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.102+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~NYPRCGU0P7CT.0 size=1048576 2024-03-09T14:36:25.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.102+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~NYPRCGU0P7CT.1 size=449176 2024-03-09T14:36:25.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.110+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=NYPRCGU0P7CT tag=YTLIZNKRI3DO type=write size=2546328 2024-03-09T14:36:25.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.114+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=ABSFC6YH38UE type=del size=3296487 2024-03-09T14:36:25.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.118+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=JQXNY7DYCEPX type=del 2024-03-09T14:36:25.126 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.122+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~G14QNZHWWGXS.0 size=1048576 2024-03-09T14:36:25.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.126+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~G14QNZHWWGXS.1 size=1048576 2024-03-09T14:36:25.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.130+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~G14QNZHWWGXS.2 size=397503 2024-03-09T14:36:25.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.130+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=G14QNZHWWGXS tag=PZPAGG5NX2DM type=write size=3543231 2024-03-09T14:36:25.135 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.134+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~PINEW8PFFT1H.0 size=262037 2024-03-09T14:36:25.141 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.138+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=PINEW8PFFT1H tag=MPRY6T9JY1CU type=write size=1310613 2024-03-09T14:36:25.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.142+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=6WAF7GWYRLV8 type=del size=227331 2024-03-09T14:36:25.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.146+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~Z9OTFVZK1OII.0 size=1048576 2024-03-09T14:36:25.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.146+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~Z9OTFVZK1OII.1 size=433126 2024-03-09T14:36:25.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.154+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=Z9OTFVZK1OII tag=ZKHDDUHT82A9 type=write size=2530278 2024-03-09T14:36:25.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.158+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=TDJKGDYKSHHS type=del size=3639191 2024-03-09T14:36:25.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.162+0000 7f73315818c0 1 < suggested update operation key=IA0AB7UVLFEJ tag=4ET0TGIMTVA3 type=del size=3416076 2024-03-09T14:36:25.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.170+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~4L4W48U05SXJ.0 size=1048576 2024-03-09T14:36:25.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.174+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~4L4W48U05SXJ.1 size=446048 2024-03-09T14:36:25.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.174+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=4L4W48U05SXJ tag=6R1DKSJ0HVGS type=write size=2543200 2024-03-09T14:36:25.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.178+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~A6B3HLTLDNRC.0 size=1048576 2024-03-09T14:36:25.182 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.178+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~A6B3HLTLDNRC.1 size=1048576 2024-03-09T14:36:25.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.182+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~A6B3HLTLDNRC.2 size=789997 2024-03-09T14:36:25.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.186+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=A6B3HLTLDNRC tag=W2KZXDEHUAZ1 type=write size=3935725 2024-03-09T14:36:25.194 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.190+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=3T97KG9689NP type=del size=3106693 2024-03-09T14:36:25.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.194+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=PVJPFEF2JO47 type=del 2024-03-09T14:36:25.205 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.202+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~GLHKPHIXBCAL.0 size=1048576 2024-03-09T14:36:25.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.206+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~GLHKPHIXBCAL.1 size=1048576 2024-03-09T14:36:25.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.206+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~GLHKPHIXBCAL.2 size=685445 2024-03-09T14:36:25.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.206+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=GLHKPHIXBCAL tag=BEAGQ6W9XUAC type=write size=3831173 2024-03-09T14:36:25.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.210+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~M4USJ07YZS6N.0 size=1048576 2024-03-09T14:36:25.217 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.214+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~M4USJ07YZS6N.1 size=409814 2024-03-09T14:36:25.222 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.218+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=M4USJ07YZS6N tag=Y14LWO8TJECV type=write size=2506966 2024-03-09T14:36:25.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.222+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=BXVWVPZ26OEO type=write size=376261 2024-03-09T14:36:25.230 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.226+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=IOFH2P9WJ1EM type=del 2024-03-09T14:36:25.233 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.230+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=WD6KLP4DYK87 type=del size=3714982 2024-03-09T14:36:25.236 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.234+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~APCEQ6ZR9RJ1.0 size=1048576 2024-03-09T14:36:25.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.238+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~APCEQ6ZR9RJ1.1 size=1048576 2024-03-09T14:36:25.242 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.238+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~APCEQ6ZR9RJ1.2 size=266664 2024-03-09T14:36:25.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.246+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=APCEQ6ZR9RJ1 tag=EF43XRXL83E1 type=write size=3412392 2024-03-09T14:36:25.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.250+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~SZMUA8VF1SPI.0 size=1048576 2024-03-09T14:36:25.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.254+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~SZMUA8VF1SPI.1 size=1048576 2024-03-09T14:36:25.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.254+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~SZMUA8VF1SPI.2 size=582700 2024-03-09T14:36:25.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.254+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=SZMUA8VF1SPI tag=XH08DJOX6DMB type=write size=3728428 2024-03-09T14:36:25.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.258+0000 7f73315818c0 1 < canceled part key=_multipart_W6KVP8EQFQ8J.2~85J292FK3VYP.0 size=1048576 2024-03-09T14:36:25.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.258+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~85J292FK3VYP.1 size=166968 2024-03-09T14:36:25.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.266+0000 7f73315818c0 1 < canceled multipart upload key=W6KVP8EQFQ8J upload=85J292FK3VYP tag=JA413HD47WO7 type=write size=2264120 2024-03-09T14:36:25.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.270+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~LM8CNWP0BJK6.0 size=1048576 2024-03-09T14:36:25.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.274+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~LM8CNWP0BJK6.1 size=155684 2024-03-09T14:36:25.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.274+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=LM8CNWP0BJK6 tag=AS6AZGOOC1AD type=write size=2252836 2024-03-09T14:36:25.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.278+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=OHGMRSE3AFMF type=del 2024-03-09T14:36:25.282 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.278+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=I1EW4JYB9V0D type=del size=3455802 2024-03-09T14:36:25.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.282+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~TDYVC8ILWQG1.0 size=288249 2024-03-09T14:36:25.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.290+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=TDYVC8ILWQG1 tag=SVCHWCQFIACK type=write size=1336825 2024-03-09T14:36:25.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.294+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~FBF6VFN6MRKU.0 size=1048576 2024-03-09T14:36:25.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.298+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~FBF6VFN6MRKU.1 size=1048576 2024-03-09T14:36:25.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.302+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~FBF6VFN6MRKU.2 size=456888 2024-03-09T14:36:25.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.302+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=FBF6VFN6MRKU tag=K5Z3T7D76AIH type=write size=3602616 2024-03-09T14:36:25.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.302+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~V0K6KI378J4J.0 size=1048576 2024-03-09T14:36:25.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.306+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~V0K6KI378J4J.1 size=481215 2024-03-09T14:36:25.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.310+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=V0K6KI378J4J tag=0VXM7813OK0H type=write size=2578367 2024-03-09T14:36:25.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.314+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=18VVBRKZ0NWT type=write size=813566 2024-03-09T14:36:25.323 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.318+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~SPGOPGET1QZS.0 size=902686 2024-03-09T14:36:25.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.326+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=SPGOPGET1QZS tag=9VJVNCV76YUU type=write size=1951262 2024-03-09T14:36:25.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.330+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~UO98ODF0A43F.0 size=1048576 2024-03-09T14:36:25.337 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.334+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~UO98ODF0A43F.1 size=1048576 2024-03-09T14:36:25.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.338+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~UO98ODF0A43F.2 size=1014211 2024-03-09T14:36:25.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.338+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=UO98ODF0A43F tag=KYESUIO1UN10 type=write size=4159939 2024-03-09T14:36:25.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.342+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=ZB7R7L92OLKT type=del 2024-03-09T14:36:25.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.342+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=K4XA804RW981 type=del size=208756 2024-03-09T14:36:25.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.346+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=ABSFC6YH38UE type=del 2024-03-09T14:36:25.356 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.354+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~URRMR2QZAW6H.0 size=1048576 2024-03-09T14:36:25.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.358+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~URRMR2QZAW6H.1 size=1048576 2024-03-09T14:36:25.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.362+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~URRMR2QZAW6H.2 size=235003 2024-03-09T14:36:25.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.362+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=URRMR2QZAW6H tag=4G3VM6PFFXN8 type=write size=3380731 2024-03-09T14:36:25.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.366+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~G14QNZHWWGXS.0 size=1048576 2024-03-09T14:36:25.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.366+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~G14QNZHWWGXS.1 size=1048576 2024-03-09T14:36:25.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.370+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~G14QNZHWWGXS.2 size=397503 2024-03-09T14:36:25.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.378+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=G14QNZHWWGXS tag=PZPAGG5NX2DM type=write size=3543231 2024-03-09T14:36:25.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.382+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~FE2XFQLAD5RC.0 size=1048576 2024-03-09T14:36:25.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.386+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~FE2XFQLAD5RC.1 size=466980 2024-03-09T14:36:25.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.386+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=FE2XFQLAD5RC tag=6UEKQ69BC48G type=write size=2564132 2024-03-09T14:36:25.391 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.386+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=6WAF7GWYRLV8 type=del 2024-03-09T14:36:25.394 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.390+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=U1WSGQCHA292 type=del size=186679 2024-03-09T14:36:25.398 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.394+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=TDJKGDYKSHHS type=del 2024-03-09T14:36:25.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.402+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~YX1RJOC3GF73.0 size=958333 2024-03-09T14:36:25.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.402+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=YX1RJOC3GF73 tag=R4K5MOTERMXN type=write size=2006909 2024-03-09T14:36:25.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.406+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~4L4W48U05SXJ.0 size=1048576 2024-03-09T14:36:25.411 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.410+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~4L4W48U05SXJ.1 size=446048 2024-03-09T14:36:25.418 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.414+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=4L4W48U05SXJ tag=6R1DKSJ0HVGS type=write size=2543200 2024-03-09T14:36:25.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.418+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=41YZ7XMTWS3Q type=del size=3230131 2024-03-09T14:36:25.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.426+0000 7f73315818c0 1 < suggested update operation key=9B0RAHZZGLYU tag=3T97KG9689NP type=del size=3106693 2024-03-09T14:36:25.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.434+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~U066YXKVQSO2.0 size=1048576 2024-03-09T14:36:25.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.438+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~U066YXKVQSO2.1 size=793902 2024-03-09T14:36:25.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.438+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=U066YXKVQSO2 tag=1635EGSDGYRH type=write size=2891054 2024-03-09T14:36:25.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.438+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~GLHKPHIXBCAL.0 size=1048576 2024-03-09T14:36:25.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.442+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~GLHKPHIXBCAL.1 size=1048576 2024-03-09T14:36:25.449 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.446+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~GLHKPHIXBCAL.2 size=685445 2024-03-09T14:36:25.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.454+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=GLHKPHIXBCAL tag=BEAGQ6W9XUAC type=write size=3831173 2024-03-09T14:36:25.460 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.458+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=JAQYD3NS57L1 type=del size=662504 2024-03-09T14:36:25.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.462+0000 7f73315818c0 1 < completed write operation key=MD7FXGYXB823 tag=BXVWVPZ26OEO type=write size=376261 2024-03-09T14:36:25.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.466+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=YGQW3H468K79 type=del size=3682434 2024-03-09T14:36:25.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.470+0000 7f73315818c0 1 < canceled operation key=MD7FXGYXB823 tag=WD6KLP4DYK87 type=del size=3714982 2024-03-09T14:36:25.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.474+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~300VMN4C4OQD.0 size=87597 2024-03-09T14:36:25.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.474+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=300VMN4C4OQD tag=CWEM2A9NYX4F type=write size=1136173 2024-03-09T14:36:25.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.478+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~SZMUA8VF1SPI.0 size=1048576 2024-03-09T14:36:25.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.482+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~SZMUA8VF1SPI.1 size=1048576 2024-03-09T14:36:25.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.486+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~SZMUA8VF1SPI.2 size=582700 2024-03-09T14:36:25.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.490+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=SZMUA8VF1SPI tag=XH08DJOX6DMB type=write size=3728428 2024-03-09T14:36:25.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.494+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=NB6LM6FZHYJG type=write size=769343 2024-03-09T14:36:25.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.498+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~LM8CNWP0BJK6.0 size=1048576 2024-03-09T14:36:25.505 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.502+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~LM8CNWP0BJK6.1 size=155684 2024-03-09T14:36:25.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.506+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=LM8CNWP0BJK6 tag=AS6AZGOOC1AD type=write size=2252836 2024-03-09T14:36:25.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.514+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~L3S2UH21X8TZ.0 size=348661 2024-03-09T14:36:25.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.514+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=L3S2UH21X8TZ tag=0MQ7Y2K8Q7DO type=write size=1397237 2024-03-09T14:36:25.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.518+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=I1EW4JYB9V0D type=del 2024-03-09T14:36:25.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.522+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=E9MROBUPXMP2 type=del size=2495336 2024-03-09T14:36:25.528 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.526+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~FBF6VFN6MRKU.0 size=1048576 2024-03-09T14:36:25.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.530+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~FBF6VFN6MRKU.1 size=1048576 2024-03-09T14:36:25.535 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.530+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~FBF6VFN6MRKU.2 size=456888 2024-03-09T14:36:25.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.538+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=FBF6VFN6MRKU tag=K5Z3T7D76AIH type=write size=3602616 2024-03-09T14:36:25.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.542+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=KP8GO394LLE4 type=del size=3651606 2024-03-09T14:36:25.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.546+0000 7f73315818c0 1 < completed write operation key=IA0AB7UVLFEJ tag=18VVBRKZ0NWT type=write size=813566 2024-03-09T14:36:25.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.550+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~ZEY11FMI0KD7.0 size=75066 2024-03-09T14:36:25.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.550+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=ZEY11FMI0KD7 tag=YPKQ9XNEOJ3W type=write size=1123642 2024-03-09T14:36:25.558 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.554+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~UO98ODF0A43F.0 size=1048576 2024-03-09T14:36:25.561 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.558+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~UO98ODF0A43F.1 size=1048576 2024-03-09T14:36:25.564 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.562+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~UO98ODF0A43F.2 size=1014211 2024-03-09T14:36:25.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.566+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=UO98ODF0A43F tag=KYESUIO1UN10 type=write size=4159939 2024-03-09T14:36:25.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.574+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~3FN79B8WDJUV.0 size=1048576 2024-03-09T14:36:25.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.578+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~3FN79B8WDJUV.1 size=277475 2024-03-09T14:36:25.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.578+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=3FN79B8WDJUV tag=AC1YKET60C82 type=write size=2374627 2024-03-09T14:36:25.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.578+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=K4XA804RW981 type=del 2024-03-09T14:36:25.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.582+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=L1K0K8TLD516 type=del size=2875361 2024-03-09T14:36:25.590 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.586+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~URRMR2QZAW6H.0 size=1048576 2024-03-09T14:36:25.593 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.590+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~URRMR2QZAW6H.1 size=1048576 2024-03-09T14:36:25.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.594+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~URRMR2QZAW6H.2 size=235003 2024-03-09T14:36:25.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.598+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=URRMR2QZAW6H tag=4G3VM6PFFXN8 type=write size=3380731 2024-03-09T14:36:25.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.606+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~ZZXAIV6M4RZZ.0 size=1048576 2024-03-09T14:36:25.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.606+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~ZZXAIV6M4RZZ.1 size=884014 2024-03-09T14:36:25.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.606+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=ZZXAIV6M4RZZ tag=U21GMIFO91HN type=write size=2981166 2024-03-09T14:36:25.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.610+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~FE2XFQLAD5RC.0 size=1048576 2024-03-09T14:36:25.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.614+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~FE2XFQLAD5RC.1 size=466980 2024-03-09T14:36:25.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.618+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=FE2XFQLAD5RC tag=6UEKQ69BC48G type=write size=2564132 2024-03-09T14:36:25.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.622+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=7KVELIB71L32 type=del size=1261418 2024-03-09T14:36:25.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.626+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=U1WSGQCHA292 type=del 2024-03-09T14:36:25.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.630+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=5M0A72M994V1 type=del size=1509135 2024-03-09T14:36:25.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.634+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~YX1RJOC3GF73.0 size=958333 2024-03-09T14:36:25.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.638+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=YX1RJOC3GF73 tag=R4K5MOTERMXN type=write size=2006909 2024-03-09T14:36:25.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.642+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=Q6QX5V3Y6T1Z type=del size=3009223 2024-03-09T14:36:25.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.646+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=41YZ7XMTWS3Q type=del 2024-03-09T14:36:25.653 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.650+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=5XDI3HOWTBL0 type=del size=1492257 2024-03-09T14:36:25.656 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.654+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~U066YXKVQSO2.0 size=1048576 2024-03-09T14:36:25.659 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.654+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~U066YXKVQSO2.1 size=793902 2024-03-09T14:36:25.665 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.662+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=U066YXKVQSO2 tag=1635EGSDGYRH type=write size=2891054 2024-03-09T14:36:25.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.666+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=9QSQ6REZ1PBD type=del size=983695 2024-03-09T14:36:25.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.670+0000 7f73315818c0 1 < canceled operation key=XELG1AWATMVL tag=JAQYD3NS57L1 type=del size=662504 2024-03-09T14:36:25.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.674+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~5PT5FKQDKBB6.0 size=1048576 2024-03-09T14:36:25.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.678+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~5PT5FKQDKBB6.1 size=1048576 2024-03-09T14:36:25.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.682+0000 7f73315818c0 1 > prepared part key=_multipart_LZC7NPF0GRO7.2~5PT5FKQDKBB6.2 size=589094 2024-03-09T14:36:25.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.682+0000 7f73315818c0 1 > prepared multipart upload key=LZC7NPF0GRO7 upload=5PT5FKQDKBB6 tag=WAGZLP6G6JRN type=write size=3734822 2024-03-09T14:36:25.693 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.690+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=YGQW3H468K79 type=del 2024-03-09T14:36:25.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.694+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=C799VL4M3KNK type=del size=4023249 2024-03-09T14:36:25.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.694+0000 7f73315818c0 1 < canceled part key=_multipart_P8R5RJI8LGZ9.2~300VMN4C4OQD.0 size=87597 2024-03-09T14:36:25.704 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.702+0000 7f73315818c0 1 < canceled multipart upload key=P8R5RJI8LGZ9 upload=300VMN4C4OQD tag=CWEM2A9NYX4F type=write size=1136173 2024-03-09T14:36:25.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.706+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~4ATA51DUOYOY.0 size=246423 2024-03-09T14:36:25.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.706+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=4ATA51DUOYOY tag=9BHCMYHLJBTU type=write size=1294999 2024-03-09T14:36:25.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.710+0000 7f73315818c0 1 < canceled operation key=FYFVW9R0M1PX tag=NB6LM6FZHYJG type=write size=769343 2024-03-09T14:36:25.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.714+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~SC1BIWL405TN.0 size=936055 2024-03-09T14:36:25.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.714+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=SC1BIWL405TN tag=NJC7OVMN04Z4 type=write size=1984631 2024-03-09T14:36:25.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.718+0000 7f73315818c0 1 < canceled part key=_multipart_4E3QMBHEAEHO.2~L3S2UH21X8TZ.0 size=348661 2024-03-09T14:36:25.725 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.722+0000 7f73315818c0 1 < canceled multipart upload key=4E3QMBHEAEHO upload=L3S2UH21X8TZ tag=0MQ7Y2K8Q7DO type=write size=1397237 2024-03-09T14:36:25.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.730+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~6K1GPVQZM7PQ.0 size=83882 2024-03-09T14:36:25.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.730+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=6K1GPVQZM7PQ tag=TM50W1T9QP08 type=write size=1132458 2024-03-09T14:36:25.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.734+0000 7f73315818c0 1 < canceled operation key=MD7FXGYXB823 tag=E9MROBUPXMP2 type=del size=2495336 2024-03-09T14:36:25.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.734+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=FW2HLWNETM2A type=write size=954168 2024-03-09T14:36:25.743 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.742+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=KP8GO394LLE4 type=del 2024-03-09T14:36:25.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.742+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=ELPLLC2B7UAJ type=del size=4035058 2024-03-09T14:36:25.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.746+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~ZEY11FMI0KD7.0 size=75066 2024-03-09T14:36:25.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.754+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=ZEY11FMI0KD7 tag=YPKQ9XNEOJ3W type=write size=1123642 2024-03-09T14:36:25.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.754+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=QBVVXTT3P3EV type=del size=3170924 2024-03-09T14:36:25.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.758+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~3FN79B8WDJUV.0 size=1048576 2024-03-09T14:36:25.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.762+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~3FN79B8WDJUV.1 size=277475 2024-03-09T14:36:25.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.766+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=3FN79B8WDJUV tag=AC1YKET60C82 type=write size=2374627 2024-03-09T14:36:25.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.770+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=TSL3NPXYRDJY type=del size=1088252 2024-03-09T14:36:25.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.778+0000 7f73315818c0 1 < suggested remove operation key=3ADG4GWVARC1 tag=L1K0K8TLD516 type=del 2024-03-09T14:36:25.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.782+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=4IF18Y0CDBW0 type=del size=2419654 2024-03-09T14:36:25.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.786+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~ZZXAIV6M4RZZ.0 size=1048576 2024-03-09T14:36:25.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.790+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~ZZXAIV6M4RZZ.1 size=884014 2024-03-09T14:36:25.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.794+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=ZZXAIV6M4RZZ tag=U21GMIFO91HN type=write size=2981166 2024-03-09T14:36:25.803 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.798+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=DV92DBPPM5A3 type=del size=1234064 2024-03-09T14:36:25.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.802+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=7KVELIB71L32 type=del 2024-03-09T14:36:25.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.806+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=XMZOKL79AOC5 type=del size=2856123 2024-03-09T14:36:25.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.810+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=5M0A72M994V1 type=del 2024-03-09T14:36:25.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.818+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~LWLOFLUM9XGE.0 size=216937 2024-03-09T14:36:25.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.818+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=LWLOFLUM9XGE tag=8HT6TCP451SL type=write size=1265513 2024-03-09T14:36:25.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.822+0000 7f73315818c0 1 < suggested update operation key=PXFZLDSAR1EG tag=Q6QX5V3Y6T1Z type=del size=3009223 2024-03-09T14:36:25.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.826+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=MM3OQB22M4JA type=del size=695856 2024-03-09T14:36:25.833 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.830+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=5XDI3HOWTBL0 type=del 2024-03-09T14:36:25.837 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.834+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=0BMJDAEG79D9 type=del size=3217833 2024-03-09T14:36:25.840 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.838+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=9QSQ6REZ1PBD type=del 2024-03-09T14:36:25.844 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.842+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=X8VZQKBPCVEV type=del size=1559573 2024-03-09T14:36:25.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.846+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~5PT5FKQDKBB6.0 size=1048576 2024-03-09T14:36:25.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.846+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~5PT5FKQDKBB6.1 size=1048576 2024-03-09T14:36:25.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.850+0000 7f73315818c0 1 < completed part key=_multipart_LZC7NPF0GRO7.2~5PT5FKQDKBB6.2 size=589094 2024-03-09T14:36:25.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.854+0000 7f73315818c0 1 < completed multipart upload key=LZC7NPF0GRO7 upload=5PT5FKQDKBB6 tag=WAGZLP6G6JRN type=write size=3734822 2024-03-09T14:36:25.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.858+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=2W6C776TCTBD type=write size=976556 2024-03-09T14:36:25.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.862+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=C799VL4M3KNK type=del 2024-03-09T14:36:25.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.866+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=HFLUEHPLVY8H type=del size=3308589 2024-03-09T14:36:25.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.870+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~4ATA51DUOYOY.0 size=246423 2024-03-09T14:36:25.879 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.874+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=4ATA51DUOYOY tag=9BHCMYHLJBTU type=write size=1294999 2024-03-09T14:36:25.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.878+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=MXZFN1APVFIS type=del size=2286657 2024-03-09T14:36:25.886 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.882+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~SC1BIWL405TN.0 size=936055 2024-03-09T14:36:25.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.890+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=SC1BIWL405TN tag=NJC7OVMN04Z4 type=write size=1984631 2024-03-09T14:36:25.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.894+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~3SFYC4KC8KWW.0 size=110354 2024-03-09T14:36:25.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.894+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=3SFYC4KC8KWW tag=1WFAU5BRB8H8 type=write size=1158930 2024-03-09T14:36:25.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.898+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~6K1GPVQZM7PQ.0 size=83882 2024-03-09T14:36:25.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.906+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=6K1GPVQZM7PQ tag=TM50W1T9QP08 type=write size=1132458 2024-03-09T14:36:25.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.910+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=7NF86325WLRJ type=del size=2737562 2024-03-09T14:36:25.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.918+0000 7f73315818c0 1 < suggested remove operation key=FYFVW9R0M1PX tag=FW2HLWNETM2A type=write 2024-03-09T14:36:25.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.918+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=6QCUCYV56M7V type=del size=2652827 2024-03-09T14:36:25.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.922+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=ELPLLC2B7UAJ type=del 2024-03-09T14:36:25.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.926+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=XUCAS9ZXGR37 type=del size=1530877 2024-03-09T14:36:25.934 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.930+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=QBVVXTT3P3EV type=del 2024-03-09T14:36:25.938 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.934+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=6QZJ2BBRRTAJ type=del size=1568655 2024-03-09T14:36:25.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.938+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=TSL3NPXYRDJY type=del 2024-03-09T14:36:25.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.942+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=YTHSNJKZC8Z0 type=del size=1595543 2024-03-09T14:36:25.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.946+0000 7f73315818c0 1 < canceled operation key=UMSNM53HTY1J tag=4IF18Y0CDBW0 type=del size=2419654 2024-03-09T14:36:25.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.950+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=57D58OI03ZUM type=del size=3836243 2024-03-09T14:36:25.960 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.958+0000 7f73315818c0 1 < suggested update operation key=0LP8KKU53X0A tag=DV92DBPPM5A3 type=del size=1234064 2024-03-09T14:36:25.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.962+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~07QET0A4SGR6.0 size=829009 2024-03-09T14:36:25.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.962+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=07QET0A4SGR6 tag=T3EQ628B8L1X type=write size=1877585 2024-03-09T14:36:25.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.966+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=XMZOKL79AOC5 type=del 2024-03-09T14:36:25.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.974+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~JKN88M338L64.0 size=997047 2024-03-09T14:36:25.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.974+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=JKN88M338L64 tag=H1CL47ST1YN9 type=write size=2045623 2024-03-09T14:36:25.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.974+0000 7f73315818c0 1 < canceled part key=_multipart_PAJYGD2GXHDT.2~LWLOFLUM9XGE.0 size=216937 2024-03-09T14:36:25.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.982+0000 7f73315818c0 1 < canceled multipart upload key=PAJYGD2GXHDT upload=LWLOFLUM9XGE tag=8HT6TCP451SL type=write size=1265513 2024-03-09T14:36:25.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.986+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~G8HYQ89RE3W4.0 size=1048576 2024-03-09T14:36:25.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.990+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~G8HYQ89RE3W4.1 size=297937 2024-03-09T14:36:25.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.990+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=G8HYQ89RE3W4 tag=GK3XF81XNIEC type=write size=2395089 2024-03-09T14:36:25.995 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.994+0000 7f73315818c0 1 < completed delete operation key=M1B6O6N8N84T tag=MM3OQB22M4JA type=del 2024-03-09T14:36:25.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:25.994+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=9V4S66NLUMO3 type=del size=494160 2024-03-09T14:36:26.007 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.002+0000 7f73315818c0 1 < suggested remove operation key=92GWY5B0K4SX tag=0BMJDAEG79D9 type=del 2024-03-09T14:36:26.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.010+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~XHVPAV1XP3ZI.0 size=295198 2024-03-09T14:36:26.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.010+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=XHVPAV1XP3ZI tag=X23XN2LFKGNF type=write size=1343774 2024-03-09T14:36:26.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.014+0000 7f73315818c0 1 < suggested update operation key=OU0730HGI176 tag=X8VZQKBPCVEV type=del size=1559573 2024-03-09T14:36:26.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.018+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=LGP3420WHB02 type=del size=279848 2024-03-09T14:36:26.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.030+0000 7f73315818c0 1 < suggested update operation key=0LP8KKU53X0A tag=2W6C776TCTBD type=write size=976556 2024-03-09T14:36:26.039 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.038+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~WB1WKL02EAPF.0 size=1048576 2024-03-09T14:36:26.042 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.038+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~WB1WKL02EAPF.1 size=1048576 2024-03-09T14:36:26.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.042+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~WB1WKL02EAPF.2 size=171596 2024-03-09T14:36:26.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.042+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=WB1WKL02EAPF tag=YBUJQ84BQZAN type=write size=3317324 2024-03-09T14:36:26.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.046+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=HFLUEHPLVY8H type=del 2024-03-09T14:36:26.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.050+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=FRDCPPZLT2QA type=write size=861554 2024-03-09T14:36:26.055 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.054+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=MXZFN1APVFIS type=del 2024-03-09T14:36:26.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.054+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=ISXO4UVWHYG4 type=del size=2018099 2024-03-09T14:36:26.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.058+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~3SFYC4KC8KWW.0 size=110354 2024-03-09T14:36:26.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.066+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=3SFYC4KC8KWW tag=1WFAU5BRB8H8 type=write size=1158930 2024-03-09T14:36:26.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.070+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=2ZZQKQ2LWWAZ type=del size=1925652 2024-03-09T14:36:26.079 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.074+0000 7f73315818c0 1 < suggested remove operation key=PAJYGD2GXHDT tag=7NF86325WLRJ type=del 2024-03-09T14:36:26.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.078+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=24OOCOMBGWZJ type=del size=1887359 2024-03-09T14:36:26.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.082+0000 7f73315818c0 1 < canceled operation key=162AOMHO2BQL tag=6QCUCYV56M7V type=del size=2652827 2024-03-09T14:36:26.092 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.090+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~KA4XJTRI4B9F.0 size=1048576 2024-03-09T14:36:26.095 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.090+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~KA4XJTRI4B9F.1 size=1048576 2024-03-09T14:36:26.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.094+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~KA4XJTRI4B9F.2 size=332938 2024-03-09T14:36:26.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.094+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=KA4XJTRI4B9F tag=YIKTHONR9TB9 type=write size=3478666 2024-03-09T14:36:26.105 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.102+0000 7f73315818c0 1 < suggested update operation key=IA0AB7UVLFEJ tag=XUCAS9ZXGR37 type=del size=1530877 2024-03-09T14:36:26.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.106+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~3CO356FT1WI5.0 size=989239 2024-03-09T14:36:26.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.106+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=3CO356FT1WI5 tag=2DYDSDCPE3UD type=write size=2037815 2024-03-09T14:36:26.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.110+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=6QZJ2BBRRTAJ type=del 2024-03-09T14:36:26.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.114+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=1KC0VYMUMCJH type=del size=3296901 2024-03-09T14:36:26.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.122+0000 7f73315818c0 1 < suggested remove operation key=0LP8KKU53X0A tag=YTHSNJKZC8Z0 type=del 2024-03-09T14:36:26.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.126+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~XCP6RETL0VQO.0 size=1048576 2024-03-09T14:36:26.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.130+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~XCP6RETL0VQO.1 size=1048576 2024-03-09T14:36:26.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.134+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~XCP6RETL0VQO.2 size=301794 2024-03-09T14:36:26.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.134+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=XCP6RETL0VQO tag=G4ZM2FL8BF46 type=write size=3447522 2024-03-09T14:36:26.140 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.138+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=57D58OI03ZUM type=del 2024-03-09T14:36:26.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.142+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~Q3KLHTA9F28G.0 size=1032046 2024-03-09T14:36:26.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.142+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=Q3KLHTA9F28G tag=FABM1ME41AYU type=write size=2080622 2024-03-09T14:36:26.149 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.146+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~07QET0A4SGR6.0 size=829009 2024-03-09T14:36:26.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.150+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=07QET0A4SGR6 tag=T3EQ628B8L1X type=write size=1877585 2024-03-09T14:36:26.160 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.158+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~2LMT4C71RXXP.0 size=1048576 2024-03-09T14:36:26.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.158+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~2LMT4C71RXXP.1 size=1048576 2024-03-09T14:36:26.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.162+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~2LMT4C71RXXP.2 size=444492 2024-03-09T14:36:26.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.162+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=2LMT4C71RXXP tag=4CO06DC9CCO5 type=write size=3590220 2024-03-09T14:36:26.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.166+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~JKN88M338L64.0 size=997047 2024-03-09T14:36:26.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.170+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=JKN88M338L64 tag=H1CL47ST1YN9 type=write size=2045623 2024-03-09T14:36:26.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.174+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=KBY5NB2MWHXC type=write size=615324 2024-03-09T14:36:26.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.178+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~G8HYQ89RE3W4.0 size=1048576 2024-03-09T14:36:26.182 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.178+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~G8HYQ89RE3W4.1 size=297937 2024-03-09T14:36:26.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.186+0000 7f73315818c0 1 < completed multipart upload key=0IB00CZ92VV4 upload=G8HYQ89RE3W4 tag=GK3XF81XNIEC type=write size=2395089 2024-03-09T14:36:26.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.190+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~UMC8XTB5W1M6.0 size=1048576 2024-03-09T14:36:26.196 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.194+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~UMC8XTB5W1M6.1 size=1048576 2024-03-09T14:36:26.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.194+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~UMC8XTB5W1M6.2 size=433210 2024-03-09T14:36:26.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.194+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=UMC8XTB5W1M6 tag=2VVA44KAOGCN type=write size=3578938 2024-03-09T14:36:26.205 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.202+0000 7f73315818c0 1 < suggested update operation key=UMSNM53HTY1J tag=9V4S66NLUMO3 type=del size=494160 2024-03-09T14:36:26.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.210+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~2Y7KD4QNCKAU.0 size=405337 2024-03-09T14:36:26.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.210+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=2Y7KD4QNCKAU tag=D5FJMMZ4KRJ8 type=write size=1453913 2024-03-09T14:36:26.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.210+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~XHVPAV1XP3ZI.0 size=295198 2024-03-09T14:36:26.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.218+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=XHVPAV1XP3ZI tag=X23XN2LFKGNF type=write size=1343774 2024-03-09T14:36:26.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.222+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=NLHMVMJEPB5C type=del size=3346477 2024-03-09T14:36:26.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.226+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=LGP3420WHB02 type=del 2024-03-09T14:36:26.232 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.230+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=52OH329DKU44 type=del size=1440868 2024-03-09T14:36:26.235 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.234+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~WB1WKL02EAPF.0 size=1048576 2024-03-09T14:36:26.236 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.234+0000 7f73315818c0 1 < canceled part key=_multipart_XELG1AWATMVL.2~WB1WKL02EAPF.1 size=1048576 2024-03-09T14:36:26.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.234+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~WB1WKL02EAPF.2 size=171596 2024-03-09T14:36:26.244 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.242+0000 7f73315818c0 1 < canceled multipart upload key=XELG1AWATMVL upload=WB1WKL02EAPF tag=YBUJQ84BQZAN type=write size=3317324 2024-03-09T14:36:26.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.246+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~KMJIIAC9DUNU.0 size=914605 2024-03-09T14:36:26.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.246+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=KMJIIAC9DUNU tag=BK4UO7UV596B type=write size=1963181 2024-03-09T14:36:26.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.254+0000 7f73315818c0 1 < suggested remove operation key=0WGGMHTEF2LB tag=FRDCPPZLT2QA type=write 2024-03-09T14:36:26.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.258+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=BOOAN9XCMMBT type=write size=856772 2024-03-09T14:36:26.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.262+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=ISXO4UVWHYG4 type=del 2024-03-09T14:36:26.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.266+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=9K5Z4MH8BJKJ type=del size=993752 2024-03-09T14:36:26.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.274+0000 7f73315818c0 1 < suggested remove operation key=UMSNM53HTY1J tag=2ZZQKQ2LWWAZ type=del 2024-03-09T14:36:26.282 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.278+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~U1Q2WVQE9EJ0.0 size=1048576 2024-03-09T14:36:26.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.282+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~U1Q2WVQE9EJ0.1 size=1048576 2024-03-09T14:36:26.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.286+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~U1Q2WVQE9EJ0.2 size=930241 2024-03-09T14:36:26.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.286+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=U1Q2WVQE9EJ0 tag=03SKX3RHOJH5 type=write size=4075969 2024-03-09T14:36:26.291 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.286+0000 7f73315818c0 1 < completed delete operation key=DQOY1JJ1YQ2W tag=24OOCOMBGWZJ type=del 2024-03-09T14:36:26.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.290+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=1KPEQQ6KY2SV type=del size=3305706 2024-03-09T14:36:26.299 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.294+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~KA4XJTRI4B9F.0 size=1048576 2024-03-09T14:36:26.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.298+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~KA4XJTRI4B9F.1 size=1048576 2024-03-09T14:36:26.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.298+0000 7f73315818c0 1 < canceled part key=_multipart_IA0AB7UVLFEJ.2~KA4XJTRI4B9F.2 size=332938 2024-03-09T14:36:26.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.306+0000 7f73315818c0 1 < canceled multipart upload key=IA0AB7UVLFEJ upload=KA4XJTRI4B9F tag=YIKTHONR9TB9 type=write size=3478666 2024-03-09T14:36:26.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.310+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=G0I0E5NT2DU6 type=del size=1875448 2024-03-09T14:36:26.316 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.314+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~3CO356FT1WI5.0 size=989239 2024-03-09T14:36:26.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.318+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=3CO356FT1WI5 tag=2DYDSDCPE3UD type=write size=2037815 2024-03-09T14:36:26.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.322+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=Z15411CWUZYS type=del size=479072 2024-03-09T14:36:26.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.326+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=1KC0VYMUMCJH type=del 2024-03-09T14:36:26.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.330+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=PNAGJB7AI9ZT type=del size=2110665 2024-03-09T14:36:26.337 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.334+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~XCP6RETL0VQO.0 size=1048576 2024-03-09T14:36:26.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.338+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~XCP6RETL0VQO.1 size=1048576 2024-03-09T14:36:26.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.338+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~XCP6RETL0VQO.2 size=301794 2024-03-09T14:36:26.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.346+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=XCP6RETL0VQO tag=G4ZM2FL8BF46 type=write size=3447522 2024-03-09T14:36:26.356 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.354+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=02Z72O0U5TPL type=del size=1537250 2024-03-09T14:36:26.359 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.358+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~Q3KLHTA9F28G.0 size=1032046 2024-03-09T14:36:26.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.362+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=Q3KLHTA9F28G tag=FABM1ME41AYU type=write size=2080622 2024-03-09T14:36:26.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.370+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~A48OPR5PVF2R.0 size=1048576 2024-03-09T14:36:26.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.370+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~A48OPR5PVF2R.1 size=129107 2024-03-09T14:36:26.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.370+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=A48OPR5PVF2R tag=0OH65JP2QC68 type=write size=2226259 2024-03-09T14:36:26.377 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.374+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~2LMT4C71RXXP.0 size=1048576 2024-03-09T14:36:26.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.378+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~2LMT4C71RXXP.1 size=1048576 2024-03-09T14:36:26.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.378+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~2LMT4C71RXXP.2 size=444492 2024-03-09T14:36:26.389 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.386+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=2LMT4C71RXXP tag=4CO06DC9CCO5 type=write size=3590220 2024-03-09T14:36:26.396 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.394+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~DD18NQF1BD4A.0 size=1048576 2024-03-09T14:36:26.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.394+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~DD18NQF1BD4A.1 size=1048576 2024-03-09T14:36:26.402 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.398+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~DD18NQF1BD4A.2 size=134489 2024-03-09T14:36:26.402 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.398+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=DD18NQF1BD4A tag=C1OPM3E9ZM7B type=write size=3280217 2024-03-09T14:36:26.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.402+0000 7f73315818c0 1 < completed write operation key=P9LI5ESR3N9G tag=KBY5NB2MWHXC type=write size=615324 2024-03-09T14:36:26.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.406+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=YCAS40D2WBL4 type=del size=3569019 2024-03-09T14:36:26.414 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.410+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~UMC8XTB5W1M6.0 size=1048576 2024-03-09T14:36:26.417 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.414+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~UMC8XTB5W1M6.1 size=1048576 2024-03-09T14:36:26.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.418+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~UMC8XTB5W1M6.2 size=433210 2024-03-09T14:36:26.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.422+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=UMC8XTB5W1M6 tag=2VVA44KAOGCN type=write size=3578938 2024-03-09T14:36:26.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.430+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~4TJH71E6V7US.0 size=1048576 2024-03-09T14:36:26.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.434+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~4TJH71E6V7US.1 size=1048576 2024-03-09T14:36:26.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.434+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~4TJH71E6V7US.2 size=188173 2024-03-09T14:36:26.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.434+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=4TJH71E6V7US tag=S536B2TZXZVF type=write size=3333901 2024-03-09T14:36:26.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.438+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~2Y7KD4QNCKAU.0 size=405337 2024-03-09T14:36:26.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.446+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=2Y7KD4QNCKAU tag=D5FJMMZ4KRJ8 type=write size=1453913 2024-03-09T14:36:26.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.446+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=ERW3SAIWWFWA type=del size=30320 2024-03-09T14:36:26.458 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.454+0000 7f73315818c0 1 < suggested remove operation key=CVAWRZMAX997 tag=NLHMVMJEPB5C type=del 2024-03-09T14:36:26.461 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.458+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=CFD9FKLGDY9Q type=write size=433703 2024-03-09T14:36:26.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.462+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=52OH329DKU44 type=del 2024-03-09T14:36:26.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.466+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~FVNIIDMUUDXN.0 size=1048576 2024-03-09T14:36:26.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.470+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~FVNIIDMUUDXN.1 size=951131 2024-03-09T14:36:26.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.470+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=FVNIIDMUUDXN tag=7Z5QHZ86I9JH type=write size=3048283 2024-03-09T14:36:26.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.474+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~KMJIIAC9DUNU.0 size=914605 2024-03-09T14:36:26.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.478+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=KMJIIAC9DUNU tag=BK4UO7UV596B type=write size=1963181 2024-03-09T14:36:26.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.486+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~BVJTA6CKHVU0.0 size=819825 2024-03-09T14:36:26.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.486+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=BVJTA6CKHVU0 tag=9FAUWAA2V937 type=write size=1868401 2024-03-09T14:36:26.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.494+0000 7f73315818c0 1 < suggested remove operation key=K3JNN1NCKBXN tag=BOOAN9XCMMBT type=write 2024-03-09T14:36:26.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.498+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~IVWA1G3N7WGD.0 size=1048576 2024-03-09T14:36:26.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.502+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~IVWA1G3N7WGD.1 size=133363 2024-03-09T14:36:26.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.502+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=IVWA1G3N7WGD tag=Q32S2ZRS8MEU type=write size=2230515 2024-03-09T14:36:26.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.506+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=9K5Z4MH8BJKJ type=del 2024-03-09T14:36:26.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.510+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~Y1LQHHK3O647.0 size=73237 2024-03-09T14:36:26.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.510+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=Y1LQHHK3O647 tag=8OOCM33DCZUY type=write size=1121813 2024-03-09T14:36:26.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.514+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~U1Q2WVQE9EJ0.0 size=1048576 2024-03-09T14:36:26.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.518+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~U1Q2WVQE9EJ0.1 size=1048576 2024-03-09T14:36:26.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.518+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~U1Q2WVQE9EJ0.2 size=930241 2024-03-09T14:36:26.528 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.526+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=U1Q2WVQE9EJ0 tag=03SKX3RHOJH5 type=write size=4075969 2024-03-09T14:36:26.534 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.530+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~2I5TQNWG2657.0 size=1048576 2024-03-09T14:36:26.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.534+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~2I5TQNWG2657.1 size=1048576 2024-03-09T14:36:26.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.538+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~2I5TQNWG2657.2 size=45250 2024-03-09T14:36:26.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.538+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=2I5TQNWG2657 tag=Z8PASZUR4FP1 type=write size=3190978 2024-03-09T14:36:26.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.542+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=1KPEQQ6KY2SV type=del 2024-03-09T14:36:26.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.546+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~D4DUP9AGAOK8.0 size=537280 2024-03-09T14:36:26.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.546+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=D4DUP9AGAOK8 tag=WKLMZVK65MZ4 type=write size=1585856 2024-03-09T14:36:26.557 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.554+0000 7f73315818c0 1 < suggested remove operation key=LZC7NPF0GRO7 tag=G0I0E5NT2DU6 type=del 2024-03-09T14:36:26.560 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.558+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=LP8D49JAI5P6 type=del size=2533959 2024-03-09T14:36:26.564 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.562+0000 7f73315818c0 -1 < failed to complete operation key=K3JNN1NCKBXN tag=Z15411CWUZYS type=del size=479072: (22) Invalid argument 2024-03-09T14:36:26.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.566+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~7L5MTQ3UGM73.0 size=1048576 2024-03-09T14:36:26.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.570+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~7L5MTQ3UGM73.1 size=779490 2024-03-09T14:36:26.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.570+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=7L5MTQ3UGM73 tag=XV8H0PADIHXO type=write size=2876642 2024-03-09T14:36:26.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.574+0000 7f73315818c0 -1 < failed to complete operation key=K3JNN1NCKBXN tag=PNAGJB7AI9ZT type=del size=2110665: (22) Invalid argument 2024-03-09T14:36:26.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.578+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=G7FPG6H7EVZU type=del size=644829 2024-03-09T14:36:26.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.582+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=02Z72O0U5TPL type=del 2024-03-09T14:36:26.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.582+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=FGEN2W53F0MC type=del size=1448884 2024-03-09T14:36:26.591 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.586+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~A48OPR5PVF2R.0 size=1048576 2024-03-09T14:36:26.591 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.586+0000 7f73315818c0 1 < canceled part key=_multipart_9B0RAHZZGLYU.2~A48OPR5PVF2R.1 size=129107 2024-03-09T14:36:26.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.594+0000 7f73315818c0 1 < canceled multipart upload key=9B0RAHZZGLYU upload=A48OPR5PVF2R tag=0OH65JP2QC68 type=write size=2226259 2024-03-09T14:36:26.600 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.598+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=BM1NGWY2FA3S type=del size=1875193 2024-03-09T14:36:26.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.602+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~DD18NQF1BD4A.0 size=1048576 2024-03-09T14:36:26.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.606+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~DD18NQF1BD4A.1 size=1048576 2024-03-09T14:36:26.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.606+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~DD18NQF1BD4A.2 size=134489 2024-03-09T14:36:26.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.614+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=DD18NQF1BD4A tag=C1OPM3E9ZM7B type=write size=3280217 2024-03-09T14:36:26.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.618+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=2J7FD8ZOD12N type=del size=2981405 2024-03-09T14:36:26.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.622+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=YCAS40D2WBL4 type=del 2024-03-09T14:36:26.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.626+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~U2RVP7XF1JAD.0 size=1048576 2024-03-09T14:36:26.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.630+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~U2RVP7XF1JAD.1 size=176906 2024-03-09T14:36:26.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.630+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=U2RVP7XF1JAD tag=XPL5XS6B2BO0 type=write size=2274058 2024-03-09T14:36:26.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.634+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~4TJH71E6V7US.0 size=1048576 2024-03-09T14:36:26.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.638+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~4TJH71E6V7US.1 size=1048576 2024-03-09T14:36:26.644 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.642+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~4TJH71E6V7US.2 size=188173 2024-03-09T14:36:26.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.646+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=4TJH71E6V7US tag=S536B2TZXZVF type=write size=3333901 2024-03-09T14:36:26.656 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.654+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~8ORL1QS1UW8C.0 size=1048576 2024-03-09T14:36:26.659 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.654+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~8ORL1QS1UW8C.1 size=736407 2024-03-09T14:36:26.659 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.654+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=8ORL1QS1UW8C tag=ECXEMW573G5J type=write size=2833559 2024-03-09T14:36:26.665 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.662+0000 7f73315818c0 1 < suggested update operation key=PAJYGD2GXHDT tag=ERW3SAIWWFWA type=del size=30320 2024-03-09T14:36:26.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.666+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=EBLQGXXJTLRY type=write size=326009 2024-03-09T14:36:26.673 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.670+0000 7f73315818c0 1 < completed write operation key=0LP8KKU53X0A tag=CFD9FKLGDY9Q type=write size=433703 2024-03-09T14:36:26.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.678+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=08SKFRI01S4K type=del size=3403180 2024-03-09T14:36:26.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.682+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~FVNIIDMUUDXN.0 size=1048576 2024-03-09T14:36:26.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.682+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~FVNIIDMUUDXN.1 size=951131 2024-03-09T14:36:26.692 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.690+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=FVNIIDMUUDXN tag=7Z5QHZ86I9JH type=write size=3048283 2024-03-09T14:36:26.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.694+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=X0XI3CWZIN3C type=del size=838981 2024-03-09T14:36:26.699 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.698+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~BVJTA6CKHVU0.0 size=819825 2024-03-09T14:36:26.705 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.702+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=BVJTA6CKHVU0 tag=9FAUWAA2V937 type=write size=1868401 2024-03-09T14:36:26.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.706+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=HZZFGYU0EAJ9 type=del size=1826834 2024-03-09T14:36:26.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.706+0000 7f73315818c0 1 < canceled part key=_multipart_OU0730HGI176.2~IVWA1G3N7WGD.0 size=1048576 2024-03-09T14:36:26.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.710+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~IVWA1G3N7WGD.1 size=133363 2024-03-09T14:36:26.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.714+0000 7f73315818c0 1 < canceled multipart upload key=OU0730HGI176 upload=IVWA1G3N7WGD tag=Q32S2ZRS8MEU type=write size=2230515 2024-03-09T14:36:26.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.718+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=HLH6DSL6WQFT type=del size=3676209 2024-03-09T14:36:26.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.722+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~Y1LQHHK3O647.0 size=73237 2024-03-09T14:36:26.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.726+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=Y1LQHHK3O647 tag=8OOCM33DCZUY type=write size=1121813 2024-03-09T14:36:26.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.730+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=XDO1TUO0KL6P type=write size=88406 2024-03-09T14:36:26.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.734+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~2I5TQNWG2657.0 size=1048576 2024-03-09T14:36:26.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.734+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~2I5TQNWG2657.1 size=1048576 2024-03-09T14:36:26.742 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.738+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~2I5TQNWG2657.2 size=45250 2024-03-09T14:36:26.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.746+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=2I5TQNWG2657 tag=Z8PASZUR4FP1 type=write size=3190978 2024-03-09T14:36:26.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.750+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~53YIMQOY1PZZ.0 size=1048576 2024-03-09T14:36:26.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.754+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~53YIMQOY1PZZ.1 size=352503 2024-03-09T14:36:26.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.754+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=53YIMQOY1PZZ tag=02QA0BOMHF4K type=write size=2449655 2024-03-09T14:36:26.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.758+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~D4DUP9AGAOK8.0 size=537280 2024-03-09T14:36:26.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.762+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=D4DUP9AGAOK8 tag=WKLMZVK65MZ4 type=write size=1585856 2024-03-09T14:36:26.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.766+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~3O7DMQL8ZUSP.0 size=1048576 2024-03-09T14:36:26.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.770+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~3O7DMQL8ZUSP.1 size=1048576 2024-03-09T14:36:26.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.774+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~3O7DMQL8ZUSP.2 size=312835 2024-03-09T14:36:26.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.774+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=3O7DMQL8ZUSP tag=T3B4SNRPGXTV type=write size=3458563 2024-03-09T14:36:26.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.778+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=LP8D49JAI5P6 type=del 2024-03-09T14:36:26.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.782+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=4S4RVMBZ8YGZ type=del size=3049935 2024-03-09T14:36:26.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.786+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~7L5MTQ3UGM73.0 size=1048576 2024-03-09T14:36:26.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.790+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~7L5MTQ3UGM73.1 size=779490 2024-03-09T14:36:26.797 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.794+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=7L5MTQ3UGM73 tag=XV8H0PADIHXO type=write size=2876642 2024-03-09T14:36:26.801 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.798+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=44G2TVJH06FO type=del size=1683315 2024-03-09T14:36:26.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.802+0000 7f73315818c0 1 < canceled operation key=3ADG4GWVARC1 tag=G7FPG6H7EVZU type=del size=644829 2024-03-09T14:36:26.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.806+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~6N0YLB45N7MZ.0 size=1048576 2024-03-09T14:36:26.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.810+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~6N0YLB45N7MZ.1 size=1048576 2024-03-09T14:36:26.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.814+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~6N0YLB45N7MZ.2 size=64687 2024-03-09T14:36:26.816 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.814+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=6N0YLB45N7MZ tag=OOPQ9MC6Q2WA type=write size=3210415 2024-03-09T14:36:26.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.814+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=FGEN2W53F0MC type=del 2024-03-09T14:36:26.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.822+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~HNPA9OR2FLXN.0 size=57812 2024-03-09T14:36:26.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.822+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=HNPA9OR2FLXN tag=TB17BMPM4L2T type=write size=1106388 2024-03-09T14:36:26.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.826+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=BM1NGWY2FA3S type=del 2024-03-09T14:36:26.834 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.830+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~OP4C8TNDF6NG.0 size=1048576 2024-03-09T14:36:26.837 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.834+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~OP4C8TNDF6NG.1 size=5624 2024-03-09T14:36:26.837 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.834+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=OP4C8TNDF6NG tag=WXN9CVC08FBQ type=write size=2102776 2024-03-09T14:36:26.841 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.838+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=2J7FD8ZOD12N type=del 2024-03-09T14:36:26.843 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.842+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=2MC9NLJ9U67K type=del size=3240625 2024-03-09T14:36:26.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.842+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~U2RVP7XF1JAD.0 size=1048576 2024-03-09T14:36:26.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.842+0000 7f73315818c0 1 < canceled part key=_multipart_M8F9TA80VME6.2~U2RVP7XF1JAD.1 size=176906 2024-03-09T14:36:26.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.850+0000 7f73315818c0 1 < canceled multipart upload key=M8F9TA80VME6 upload=U2RVP7XF1JAD tag=XPL5XS6B2BO0 type=write size=2274058 2024-03-09T14:36:26.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.854+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=98MG3X03357G type=del size=1703749 2024-03-09T14:36:26.860 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.858+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~8ORL1QS1UW8C.0 size=1048576 2024-03-09T14:36:26.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.858+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~8ORL1QS1UW8C.1 size=736407 2024-03-09T14:36:26.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.866+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=8ORL1QS1UW8C tag=ECXEMW573G5J type=write size=2833559 2024-03-09T14:36:26.871 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.870+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=0CKAOB8VG58Y type=del size=3857826 2024-03-09T14:36:26.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.870+0000 7f73315818c0 1 < completed write operation key=CVAWRZMAX997 tag=EBLQGXXJTLRY type=write size=326009 2024-03-09T14:36:26.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.878+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~EUDZODZ3JKZS.0 size=177329 2024-03-09T14:36:26.881 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.878+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=EUDZODZ3JKZS tag=7LBXTW5I16I5 type=write size=1225905 2024-03-09T14:36:26.884 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.882+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=08SKFRI01S4K type=del 2024-03-09T14:36:26.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.886+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=4TRCO597QW7F type=del size=3795776 2024-03-09T14:36:26.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.886+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=X0XI3CWZIN3C type=del 2024-03-09T14:36:26.897 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.894+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~FRAWT0API2VK.0 size=1048576 2024-03-09T14:36:26.900 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.898+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~FRAWT0API2VK.1 size=1048576 2024-03-09T14:36:26.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.898+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~FRAWT0API2VK.2 size=934520 2024-03-09T14:36:26.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.898+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=FRAWT0API2VK tag=62UGH79GYSHZ type=write size=4080248 2024-03-09T14:36:26.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.902+0000 7f73315818c0 1 < canceled operation key=MD7FXGYXB823 tag=HZZFGYU0EAJ9 type=del size=1826834 2024-03-09T14:36:26.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.906+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=7OEMU97P6GDO type=del size=2814855 2024-03-09T14:36:26.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.910+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=HLH6DSL6WQFT type=del 2024-03-09T14:36:26.917 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.914+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=XS4PYNOFGN0R type=del size=2607458 2024-03-09T14:36:26.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.918+0000 7f73315818c0 1 < completed write operation key=IA0AB7UVLFEJ tag=XDO1TUO0KL6P type=write size=88406 2024-03-09T14:36:26.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.922+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=MYID1L3M9ZL7 type=del size=4170345 2024-03-09T14:36:26.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.926+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~53YIMQOY1PZZ.0 size=1048576 2024-03-09T14:36:26.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.926+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~53YIMQOY1PZZ.1 size=352503 2024-03-09T14:36:26.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.934+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=53YIMQOY1PZZ tag=02QA0BOMHF4K type=write size=2449655 2024-03-09T14:36:26.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.938+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~8UD1LH6JP6V0.0 size=603348 2024-03-09T14:36:26.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.938+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=8UD1LH6JP6V0 tag=RI0JKO0R7AXU type=write size=1651924 2024-03-09T14:36:26.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.942+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~3O7DMQL8ZUSP.0 size=1048576 2024-03-09T14:36:26.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.946+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~3O7DMQL8ZUSP.1 size=1048576 2024-03-09T14:36:26.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.950+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~3O7DMQL8ZUSP.2 size=312835 2024-03-09T14:36:26.958 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.954+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=3O7DMQL8ZUSP tag=T3B4SNRPGXTV type=write size=3458563 2024-03-09T14:36:26.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.958+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=DOPHX1YNL58V type=del size=2551080 2024-03-09T14:36:26.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.962+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=4S4RVMBZ8YGZ type=del 2024-03-09T14:36:26.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.966+0000 7f73315818c0 1 > prepared operation key=M1B6O6N8N84T tag=FW4UEX3E4U6R type=del size=2035500 2024-03-09T14:36:26.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.970+0000 7f73315818c0 1 < completed delete operation key=IA0AB7UVLFEJ tag=44G2TVJH06FO type=del 2024-03-09T14:36:26.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.974+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=SDH0JCQRHL34 type=write size=987582 2024-03-09T14:36:26.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.974+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~6N0YLB45N7MZ.0 size=1048576 2024-03-09T14:36:26.982 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.978+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~6N0YLB45N7MZ.1 size=1048576 2024-03-09T14:36:26.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.982+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~6N0YLB45N7MZ.2 size=64687 2024-03-09T14:36:26.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.986+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=6N0YLB45N7MZ tag=OOPQ9MC6Q2WA type=write size=3210415 2024-03-09T14:36:26.999 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.998+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=EV9BMSFVEQCG type=del size=1166037 2024-03-09T14:36:27.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:26.998+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~HNPA9OR2FLXN.0 size=57812 2024-03-09T14:36:27.009 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.006+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=HNPA9OR2FLXN tag=TB17BMPM4L2T type=write size=1106388 2024-03-09T14:36:27.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.010+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~XXUK9OF8V56S.0 size=1048576 2024-03-09T14:36:27.017 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.014+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~XXUK9OF8V56S.1 size=1048576 2024-03-09T14:36:27.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.018+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~XXUK9OF8V56S.2 size=635174 2024-03-09T14:36:27.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.018+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=XXUK9OF8V56S tag=OC6HMP3JUWJV type=write size=3780902 2024-03-09T14:36:27.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.022+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~OP4C8TNDF6NG.0 size=1048576 2024-03-09T14:36:27.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.022+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~OP4C8TNDF6NG.1 size=5624 2024-03-09T14:36:27.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.030+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=OP4C8TNDF6NG tag=WXN9CVC08FBQ type=write size=2102776 2024-03-09T14:36:27.036 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.034+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=OYEF9GIFIOXQ type=del size=1549512 2024-03-09T14:36:27.040 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.038+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=2MC9NLJ9U67K type=del 2024-03-09T14:36:27.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.042+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=WLDQPI7ECPXK type=del size=1007629 2024-03-09T14:36:27.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.042+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=98MG3X03357G type=del 2024-03-09T14:36:27.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.050+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~L0NL3PYV8IUW.0 size=420385 2024-03-09T14:36:27.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.050+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=L0NL3PYV8IUW tag=FPKA71EWP2IF type=write size=1468961 2024-03-09T14:36:27.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.054+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=0CKAOB8VG58Y type=del 2024-03-09T14:36:27.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.058+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~JNLBCRVL21M5.0 size=910862 2024-03-09T14:36:27.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.058+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=JNLBCRVL21M5 tag=PAGLAG6BOTPL type=write size=1959438 2024-03-09T14:36:27.067 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.062+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~EUDZODZ3JKZS.0 size=177329 2024-03-09T14:36:27.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.070+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=EUDZODZ3JKZS tag=7LBXTW5I16I5 type=write size=1225905 2024-03-09T14:36:27.079 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.074+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~0DJ0QA1WKF8G.0 size=1048576 2024-03-09T14:36:27.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.078+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~0DJ0QA1WKF8G.1 size=266035 2024-03-09T14:36:27.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.078+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=0DJ0QA1WKF8G tag=IZLWUWN4WC7N type=write size=2363187 2024-03-09T14:36:27.084 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.082+0000 7f73315818c0 1 < canceled operation key=LZC7NPF0GRO7 tag=4TRCO597QW7F type=del size=3795776 2024-03-09T14:36:27.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.090+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~823DZ9VYJ2E2.0 size=527882 2024-03-09T14:36:27.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.090+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=823DZ9VYJ2E2 tag=58XOBT2TBUBM type=write size=1576458 2024-03-09T14:36:27.095 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.094+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~FRAWT0API2VK.0 size=1048576 2024-03-09T14:36:27.095 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.094+0000 7f73315818c0 1 < canceled part key=_multipart_9B0RAHZZGLYU.2~FRAWT0API2VK.1 size=1048576 2024-03-09T14:36:27.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.094+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~FRAWT0API2VK.2 size=934520 2024-03-09T14:36:27.104 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.102+0000 7f73315818c0 1 < canceled multipart upload key=9B0RAHZZGLYU upload=FRAWT0API2VK tag=62UGH79GYSHZ type=write size=4080248 2024-03-09T14:36:27.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.106+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=NAZDNOGPB5OA type=del size=3870481 2024-03-09T14:36:27.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.110+0000 7f73315818c0 1 < suggested update operation key=MSMD3ANK4T7J tag=7OEMU97P6GDO type=del size=2814855 2024-03-09T14:36:27.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.114+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=U5T2AO9DZAEX type=del size=1937479 2024-03-09T14:36:27.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.118+0000 7f73315818c0 1 < completed delete operation key=M1B6O6N8N84T tag=XS4PYNOFGN0R type=del 2024-03-09T14:36:27.124 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.122+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=NJ0OETXHZL21 type=write size=453381 2024-03-09T14:36:27.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.126+0000 7f73315818c0 1 < suggested update operation key=K3JNN1NCKBXN tag=MYID1L3M9ZL7 type=del size=4170345 2024-03-09T14:36:27.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.130+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=7QFGX7C8NHNN type=del size=2239096 2024-03-09T14:36:27.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.134+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~8UD1LH6JP6V0.0 size=603348 2024-03-09T14:36:27.142 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.138+0000 7f73315818c0 1 < completed multipart upload key=DQOY1JJ1YQ2W upload=8UD1LH6JP6V0 tag=RI0JKO0R7AXU type=write size=1651924 2024-03-09T14:36:27.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.142+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=YIGBY62V3Z5Z type=del size=2200689 2024-03-09T14:36:27.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.150+0000 7f73315818c0 1 < suggested update operation key=M8F9TA80VME6 tag=DOPHX1YNL58V type=del size=2551080 2024-03-09T14:36:27.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.154+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=GCGY6QT62MC2 type=del size=840210 2024-03-09T14:36:27.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.162+0000 7f73315818c0 1 < suggested remove operation key=M1B6O6N8N84T tag=FW4UEX3E4U6R type=del 2024-03-09T14:36:27.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.162+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=9VOT92F10QPZ type=del size=2316483 2024-03-09T14:36:27.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.170+0000 7f73315818c0 1 < suggested remove operation key=Y8EQAHMTL66U tag=SDH0JCQRHL34 type=write 2024-03-09T14:36:27.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.174+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~HW21YOTE0HLL.0 size=1048576 2024-03-09T14:36:27.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.178+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~HW21YOTE0HLL.1 size=1048576 2024-03-09T14:36:27.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.182+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~HW21YOTE0HLL.2 size=28609 2024-03-09T14:36:27.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.182+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=HW21YOTE0HLL tag=PO21S0OJUXFW type=write size=3174337 2024-03-09T14:36:27.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.182+0000 7f73315818c0 -1 < failed to complete operation key=K3JNN1NCKBXN tag=EV9BMSFVEQCG type=del size=1166037: (22) Invalid argument 2024-03-09T14:36:27.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.190+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~WKWU3J3C1PZU.0 size=1048576 2024-03-09T14:36:27.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.190+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~WKWU3J3C1PZU.1 size=601474 2024-03-09T14:36:27.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.190+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=WKWU3J3C1PZU tag=HK0U6EVKLECB type=write size=2698626 2024-03-09T14:36:27.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.194+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~XXUK9OF8V56S.0 size=1048576 2024-03-09T14:36:27.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.198+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~XXUK9OF8V56S.1 size=1048576 2024-03-09T14:36:27.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.202+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~XXUK9OF8V56S.2 size=635174 2024-03-09T14:36:27.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.206+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=XXUK9OF8V56S tag=OC6HMP3JUWJV type=write size=3780902 2024-03-09T14:36:27.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.214+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~DZ2U338Y9UIC.0 size=1048576 2024-03-09T14:36:27.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.214+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~DZ2U338Y9UIC.1 size=1048576 2024-03-09T14:36:27.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.218+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~DZ2U338Y9UIC.2 size=152910 2024-03-09T14:36:27.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.218+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=DZ2U338Y9UIC tag=SDM47L0GENQ3 type=write size=3298638 2024-03-09T14:36:27.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.226+0000 7f73315818c0 1 < suggested update operation key=S302JFYEDEL7 tag=OYEF9GIFIOXQ type=del size=1549512 2024-03-09T14:36:27.235 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.230+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~3FH5SCYNRO8A.0 size=1003507 2024-03-09T14:36:27.235 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.230+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=3FH5SCYNRO8A tag=QFK6OQY2OBQQ type=write size=2052083 2024-03-09T14:36:27.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.234+0000 7f73315818c0 1 < completed delete operation key=9B0RAHZZGLYU tag=WLDQPI7ECPXK type=del 2024-03-09T14:36:27.243 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.238+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=I9NANFB4RSJB type=write size=323455 2024-03-09T14:36:27.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.246+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~L0NL3PYV8IUW.0 size=420385 2024-03-09T14:36:27.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.250+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=L0NL3PYV8IUW tag=FPKA71EWP2IF type=write size=1468961 2024-03-09T14:36:27.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.254+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=WAD22Q4J7UOV type=del size=1433403 2024-03-09T14:36:27.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.258+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~JNLBCRVL21M5.0 size=910862 2024-03-09T14:36:27.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.262+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=JNLBCRVL21M5 tag=PAGLAG6BOTPL type=write size=1959438 2024-03-09T14:36:27.269 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.266+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=F329YK5N5KWZ type=del size=496560 2024-03-09T14:36:27.272 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.270+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~0DJ0QA1WKF8G.0 size=1048576 2024-03-09T14:36:27.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.270+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~0DJ0QA1WKF8G.1 size=266035 2024-03-09T14:36:27.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.278+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=0DJ0QA1WKF8G tag=IZLWUWN4WC7N type=write size=2363187 2024-03-09T14:36:27.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.286+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~JJPSRE0E3HEH.0 size=1048576 2024-03-09T14:36:27.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.286+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~JJPSRE0E3HEH.1 size=1048576 2024-03-09T14:36:27.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.290+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~JJPSRE0E3HEH.2 size=898063 2024-03-09T14:36:27.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.290+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=JJPSRE0E3HEH tag=ESHOMUSJ6IBM type=write size=4043791 2024-03-09T14:36:27.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.294+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~823DZ9VYJ2E2.0 size=527882 2024-03-09T14:36:27.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.298+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=823DZ9VYJ2E2 tag=58XOBT2TBUBM type=write size=1576458 2024-03-09T14:36:27.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.302+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=IRZD4IL4ZE3W type=del size=1925746 2024-03-09T14:36:27.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.306+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=NAZDNOGPB5OA type=del 2024-03-09T14:36:27.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.310+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=BWE9N18O7U83 type=del size=3705186 2024-03-09T14:36:27.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.318+0000 7f73315818c0 1 < canceled operation key=DQOY1JJ1YQ2W tag=U5T2AO9DZAEX type=del size=1937479 2024-03-09T14:36:27.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.326+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~PRO983NY7ZCQ.0 size=1048576 2024-03-09T14:36:27.331 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.330+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~PRO983NY7ZCQ.1 size=1048576 2024-03-09T14:36:27.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.330+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~PRO983NY7ZCQ.2 size=891773 2024-03-09T14:36:27.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.330+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=PRO983NY7ZCQ tag=OB2YQHFFVUSL type=write size=4037501 2024-03-09T14:36:27.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.334+0000 7f73315818c0 1 < canceled operation key=PXFZLDSAR1EG tag=NJ0OETXHZL21 type=write size=453381 2024-03-09T14:36:27.342 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.338+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=E7T3F5GXTWAE type=write size=508677 2024-03-09T14:36:27.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.342+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=7QFGX7C8NHNN type=del 2024-03-09T14:36:27.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.346+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=UTHLGCP9BHLU type=del size=4107635 2024-03-09T14:36:27.353 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.350+0000 7f73315818c0 1 < completed delete operation key=8O5OP8C5NX12 tag=YIGBY62V3Z5Z type=del 2024-03-09T14:36:27.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.358+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~K2ST49R022ZI.0 size=1048576 2024-03-09T14:36:27.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.362+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~K2ST49R022ZI.1 size=1048576 2024-03-09T14:36:27.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.362+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~K2ST49R022ZI.2 size=275493 2024-03-09T14:36:27.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.362+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=K2ST49R022ZI tag=B7KIBB5W3369 type=write size=3421221 2024-03-09T14:36:27.370 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.366+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=GCGY6QT62MC2 type=del 2024-03-09T14:36:27.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.370+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=CX90UV28RVKR type=del size=1550299 2024-03-09T14:36:27.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.374+0000 7f73315818c0 1 < canceled operation key=LZC7NPF0GRO7 tag=9VOT92F10QPZ type=del size=2316483 2024-03-09T14:36:27.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.382+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~H40FA43IS5PU.0 size=1048576 2024-03-09T14:36:27.387 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.382+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~H40FA43IS5PU.1 size=1048576 2024-03-09T14:36:27.389 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.386+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~H40FA43IS5PU.2 size=641822 2024-03-09T14:36:27.390 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.386+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=H40FA43IS5PU tag=0OYOLH5TU90H type=write size=3787550 2024-03-09T14:36:27.393 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.390+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~HW21YOTE0HLL.0 size=1048576 2024-03-09T14:36:27.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.394+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~HW21YOTE0HLL.1 size=1048576 2024-03-09T14:36:27.400 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.398+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~HW21YOTE0HLL.2 size=28609 2024-03-09T14:36:27.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.402+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=HW21YOTE0HLL tag=PO21S0OJUXFW type=write size=3174337 2024-03-09T14:36:27.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.406+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=J52BP2U9UXCH type=del size=3116876 2024-03-09T14:36:27.414 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.410+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~WKWU3J3C1PZU.0 size=1048576 2024-03-09T14:36:27.417 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.414+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~WKWU3J3C1PZU.1 size=601474 2024-03-09T14:36:27.423 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.418+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=WKWU3J3C1PZU tag=HK0U6EVKLECB type=write size=2698626 2024-03-09T14:36:27.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.426+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~DC7W6PJCHVMH.0 size=1048576 2024-03-09T14:36:27.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.430+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~DC7W6PJCHVMH.1 size=434836 2024-03-09T14:36:27.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.430+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=DC7W6PJCHVMH tag=5OPRH6K9YSIB type=write size=2531988 2024-03-09T14:36:27.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.434+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~DZ2U338Y9UIC.0 size=1048576 2024-03-09T14:36:27.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.434+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~DZ2U338Y9UIC.1 size=1048576 2024-03-09T14:36:27.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.438+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~DZ2U338Y9UIC.2 size=152910 2024-03-09T14:36:27.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.446+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=DZ2U338Y9UIC tag=SDM47L0GENQ3 type=write size=3298638 2024-03-09T14:36:27.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.450+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=NZ126AJBZ1ZH type=write size=728650 2024-03-09T14:36:27.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.450+0000 7f73315818c0 1 < canceled part key=_multipart_Y8EQAHMTL66U.2~3FH5SCYNRO8A.0 size=1003507 2024-03-09T14:36:27.458 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.454+0000 7f73315818c0 1 < canceled multipart upload key=Y8EQAHMTL66U upload=3FH5SCYNRO8A tag=QFK6OQY2OBQQ type=write size=2052083 2024-03-09T14:36:27.461 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.458+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=Y3NKH7JTC608 type=del size=2902831 2024-03-09T14:36:27.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.466+0000 7f73315818c0 1 < suggested remove operation key=W6KVP8EQFQ8J tag=I9NANFB4RSJB type=write 2024-03-09T14:36:27.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.474+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~OYE3NIVLBBNH.0 size=228467 2024-03-09T14:36:27.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.474+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=OYE3NIVLBBNH tag=KZ8LYR2XBDDX type=write size=1277043 2024-03-09T14:36:27.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.482+0000 7f73315818c0 1 < suggested remove operation key=LZC7NPF0GRO7 tag=WAD22Q4J7UOV type=del 2024-03-09T14:36:27.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.482+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=UWFDNNH2DD1O type=del size=307646 2024-03-09T14:36:27.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.486+0000 7f73315818c0 -1 < failed to complete operation key=W6KVP8EQFQ8J tag=F329YK5N5KWZ type=del size=496560: (22) Invalid argument 2024-03-09T14:36:27.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.490+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=5QMR6SE3YX4X type=del size=4024140 2024-03-09T14:36:27.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.494+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~JJPSRE0E3HEH.0 size=1048576 2024-03-09T14:36:27.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.498+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~JJPSRE0E3HEH.1 size=1048576 2024-03-09T14:36:27.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.502+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~JJPSRE0E3HEH.2 size=898063 2024-03-09T14:36:27.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.506+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=JJPSRE0E3HEH tag=ESHOMUSJ6IBM type=write size=4043791 2024-03-09T14:36:27.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.510+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=KG10YZEDD97X type=del size=3956256 2024-03-09T14:36:27.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.518+0000 7f73315818c0 1 < suggested update operation key=4E3QMBHEAEHO tag=IRZD4IL4ZE3W type=del size=1925746 2024-03-09T14:36:27.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.522+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=M0HSSABLDK0J type=del size=3614523 2024-03-09T14:36:27.528 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.526+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=BWE9N18O7U83 type=del 2024-03-09T14:36:27.535 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.530+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~DY3P4J835IOX.0 size=1048576 2024-03-09T14:36:27.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.534+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~DY3P4J835IOX.1 size=1048576 2024-03-09T14:36:27.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.538+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~DY3P4J835IOX.2 size=105184 2024-03-09T14:36:27.541 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.538+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=DY3P4J835IOX tag=BG81RTVTACU0 type=write size=3250912 2024-03-09T14:36:27.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.542+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~PRO983NY7ZCQ.0 size=1048576 2024-03-09T14:36:27.547 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.542+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~PRO983NY7ZCQ.1 size=1048576 2024-03-09T14:36:27.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.546+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~PRO983NY7ZCQ.2 size=891773 2024-03-09T14:36:27.557 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.554+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=PRO983NY7ZCQ tag=OB2YQHFFVUSL type=write size=4037501 2024-03-09T14:36:27.564 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.562+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~MORB8CKUY8PH.0 size=1048576 2024-03-09T14:36:27.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.562+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~MORB8CKUY8PH.1 size=1048576 2024-03-09T14:36:27.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.566+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~MORB8CKUY8PH.2 size=947776 2024-03-09T14:36:27.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.566+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=MORB8CKUY8PH tag=WJ4ATCYNH28B type=write size=4093504 2024-03-09T14:36:27.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.570+0000 7f73315818c0 -1 < failed to complete operation key=4E3QMBHEAEHO tag=E7T3F5GXTWAE type=write size=508677: (22) Invalid argument 2024-03-09T14:36:27.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.574+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=X6LCD3VXI79I type=del size=2500595 2024-03-09T14:36:27.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.578+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=UTHLGCP9BHLU type=del 2024-03-09T14:36:27.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.578+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=EUUETO4X18BP type=del size=2427956 2024-03-09T14:36:27.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.582+0000 7f73315818c0 1 < canceled part key=_multipart_PAJYGD2GXHDT.2~K2ST49R022ZI.0 size=1048576 2024-03-09T14:36:27.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.582+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~K2ST49R022ZI.1 size=1048576 2024-03-09T14:36:27.590 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.586+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~K2ST49R022ZI.2 size=275493 2024-03-09T14:36:27.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.594+0000 7f73315818c0 1 < canceled multipart upload key=PAJYGD2GXHDT upload=K2ST49R022ZI tag=B7KIBB5W3369 type=write size=3421221 2024-03-09T14:36:27.599 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.598+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=WWALUWTXLNZ2 type=write size=466730 2024-03-09T14:36:27.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.598+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=CX90UV28RVKR type=del 2024-03-09T14:36:27.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.606+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~4MB4GZHWC8HH.0 size=409233 2024-03-09T14:36:27.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.606+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=4MB4GZHWC8HH tag=59VHKTC5UOTA type=write size=1457809 2024-03-09T14:36:27.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.610+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~H40FA43IS5PU.0 size=1048576 2024-03-09T14:36:27.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.614+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~H40FA43IS5PU.1 size=1048576 2024-03-09T14:36:27.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.618+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~H40FA43IS5PU.2 size=641822 2024-03-09T14:36:27.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.622+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=H40FA43IS5PU tag=0OYOLH5TU90H type=write size=3787550 2024-03-09T14:36:27.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.630+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~41U3UUMGGBJZ.0 size=243755 2024-03-09T14:36:27.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.630+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=41U3UUMGGBJZ tag=IS1GIO3550RJ type=write size=1292331 2024-03-09T14:36:27.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.634+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=J52BP2U9UXCH type=del 2024-03-09T14:36:27.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.638+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=OJH98XCX2CHS type=del size=1371262 2024-03-09T14:36:27.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.638+0000 7f73315818c0 1 < canceled part key=_multipart_4E3QMBHEAEHO.2~DC7W6PJCHVMH.0 size=1048576 2024-03-09T14:36:27.644 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.642+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~DC7W6PJCHVMH.1 size=434836 2024-03-09T14:36:27.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.646+0000 7f73315818c0 1 < canceled multipart upload key=4E3QMBHEAEHO upload=DC7W6PJCHVMH tag=5OPRH6K9YSIB type=write size=2531988 2024-03-09T14:36:27.660 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.658+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~D4VR7L2X6PV4.0 size=1048576 2024-03-09T14:36:27.663 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.658+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~D4VR7L2X6PV4.1 size=885451 2024-03-09T14:36:27.663 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.658+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=D4VR7L2X6PV4 tag=8Q0A2PA0G65X type=write size=2982603 2024-03-09T14:36:27.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.666+0000 7f73315818c0 1 < suggested remove operation key=162AOMHO2BQL tag=NZ126AJBZ1ZH type=write 2024-03-09T14:36:27.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.670+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=7SCR37PKX9VN type=del size=1203606 2024-03-09T14:36:27.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.674+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=Y3NKH7JTC608 type=del 2024-03-09T14:36:27.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.682+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~ZVOUBP3YE40U.0 size=1048576 2024-03-09T14:36:27.688 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.686+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~ZVOUBP3YE40U.1 size=1048576 2024-03-09T14:36:27.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.690+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~ZVOUBP3YE40U.2 size=848064 2024-03-09T14:36:27.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.690+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=ZVOUBP3YE40U tag=TSV998K0PSUX type=write size=3993792 2024-03-09T14:36:27.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.694+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~OYE3NIVLBBNH.0 size=228467 2024-03-09T14:36:27.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.698+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=OYE3NIVLBBNH tag=KZ8LYR2XBDDX type=write size=1277043 2024-03-09T14:36:27.705 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.702+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=BDW35WIG1JWY type=del size=3867412 2024-03-09T14:36:27.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.706+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=UWFDNNH2DD1O type=del 2024-03-09T14:36:27.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.710+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=KBCFFB4IAENX type=del size=381529 2024-03-09T14:36:27.714 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.710+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=5QMR6SE3YX4X type=del 2024-03-09T14:36:27.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.714+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=24DD8GD4694P type=del size=3482108 2024-03-09T14:36:27.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.718+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=KG10YZEDD97X type=del 2024-03-09T14:36:27.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.726+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~W60MFAS998JR.0 size=1048576 2024-03-09T14:36:27.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.726+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~W60MFAS998JR.1 size=1030480 2024-03-09T14:36:27.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.726+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=W60MFAS998JR tag=0P9KPMIGNIRY type=write size=3127632 2024-03-09T14:36:27.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.734+0000 7f73315818c0 1 < suggested remove operation key=1NYNL8M5DVER tag=M0HSSABLDK0J type=del 2024-03-09T14:36:27.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.738+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=EUVWBQ9VS5VZ type=del size=3505555 2024-03-09T14:36:27.743 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.742+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~DY3P4J835IOX.0 size=1048576 2024-03-09T14:36:27.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.742+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~DY3P4J835IOX.1 size=1048576 2024-03-09T14:36:27.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.746+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~DY3P4J835IOX.2 size=105184 2024-03-09T14:36:27.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.750+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=DY3P4J835IOX tag=BG81RTVTACU0 type=write size=3250912 2024-03-09T14:36:27.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.754+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=VDQ5GIE5ND28 type=del size=376586 2024-03-09T14:36:27.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.758+0000 7f73315818c0 1 < canceled part key=_multipart_1NYNL8M5DVER.2~MORB8CKUY8PH.0 size=1048576 2024-03-09T14:36:27.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.758+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~MORB8CKUY8PH.1 size=1048576 2024-03-09T14:36:27.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.762+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~MORB8CKUY8PH.2 size=947776 2024-03-09T14:36:27.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.766+0000 7f73315818c0 1 < canceled multipart upload key=1NYNL8M5DVER upload=MORB8CKUY8PH tag=WJ4ATCYNH28B type=write size=4093504 2024-03-09T14:36:27.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.770+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=9V9HJFKNYKON type=del size=1593036 2024-03-09T14:36:27.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.774+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=X6LCD3VXI79I type=del 2024-03-09T14:36:27.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.782+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~TD6ZSHGDU2W9.0 size=895784 2024-03-09T14:36:27.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.782+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=TD6ZSHGDU2W9 tag=4WSOPNM7JG1W type=write size=1944360 2024-03-09T14:36:27.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.782+0000 7f73315818c0 1 < canceled operation key=UMSNM53HTY1J tag=EUUETO4X18BP type=del size=2427956 2024-03-09T14:36:27.794 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.790+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~HWYGPU8GTXOJ.0 size=1048576 2024-03-09T14:36:27.796 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.794+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~HWYGPU8GTXOJ.1 size=1048576 2024-03-09T14:36:27.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.798+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~HWYGPU8GTXOJ.2 size=120955 2024-03-09T14:36:27.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.798+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=HWYGPU8GTXOJ tag=Q70RIZD3DN49 type=write size=3266683 2024-03-09T14:36:27.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.802+0000 7f73315818c0 1 < completed write operation key=LZC7NPF0GRO7 tag=WWALUWTXLNZ2 type=write size=466730 2024-03-09T14:36:27.811 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.806+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~X8K08JX7DA20.0 size=165011 2024-03-09T14:36:27.811 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.806+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=X8K08JX7DA20 tag=L73KSWJM0E3B type=write size=1213587 2024-03-09T14:36:27.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.810+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~4MB4GZHWC8HH.0 size=409233 2024-03-09T14:36:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.818+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=4MB4GZHWC8HH tag=59VHKTC5UOTA type=write size=1457809 2024-03-09T14:36:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.822+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=8KIEFQ5773TG type=del size=3986829 2024-03-09T14:36:27.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.826+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~41U3UUMGGBJZ.0 size=243755 2024-03-09T14:36:27.834 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.830+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=41U3UUMGGBJZ tag=IS1GIO3550RJ type=write size=1292331 2024-03-09T14:36:27.840 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.838+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~6ZALEO9PBKWR.0 size=1048576 2024-03-09T14:36:27.843 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.842+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~6ZALEO9PBKWR.1 size=834756 2024-03-09T14:36:27.843 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.842+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=6ZALEO9PBKWR tag=93XGIBD3ROR8 type=write size=2931908 2024-03-09T14:36:27.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.842+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=OJH98XCX2CHS type=del 2024-03-09T14:36:27.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.846+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=BK2PE7A8ANHB type=del size=1592315 2024-03-09T14:36:27.854 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.850+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~D4VR7L2X6PV4.0 size=1048576 2024-03-09T14:36:27.857 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.854+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~D4VR7L2X6PV4.1 size=885451 2024-03-09T14:36:27.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.862+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=D4VR7L2X6PV4 tag=8Q0A2PA0G65X type=write size=2982603 2024-03-09T14:36:27.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.870+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~70OXI6UPK4WK.0 size=1048576 2024-03-09T14:36:27.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.870+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~70OXI6UPK4WK.1 size=433749 2024-03-09T14:36:27.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.870+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=70OXI6UPK4WK tag=XLF92SPBJKT2 type=write size=2530901 2024-03-09T14:36:27.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.874+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=7SCR37PKX9VN type=del 2024-03-09T14:36:27.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.882+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~GPO9NJWA1NA0.0 size=1048576 2024-03-09T14:36:27.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.886+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~GPO9NJWA1NA0.1 size=1048576 2024-03-09T14:36:27.890 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.886+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~GPO9NJWA1NA0.2 size=1039827 2024-03-09T14:36:27.890 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.886+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=GPO9NJWA1NA0 tag=480QM2J7Z8L1 type=write size=4185555 2024-03-09T14:36:27.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.890+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~ZVOUBP3YE40U.0 size=1048576 2024-03-09T14:36:27.897 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.894+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~ZVOUBP3YE40U.1 size=1048576 2024-03-09T14:36:27.900 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.898+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~ZVOUBP3YE40U.2 size=848064 2024-03-09T14:36:27.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.902+0000 7f73315818c0 1 < completed multipart upload key=0IB00CZ92VV4 upload=ZVOUBP3YE40U tag=TSV998K0PSUX type=write size=3993792 2024-03-09T14:36:27.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.906+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=VRE2WXMRUD6I type=del size=168171 2024-03-09T14:36:27.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.910+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=BDW35WIG1JWY type=del 2024-03-09T14:36:27.917 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.914+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=Y0027YD8NNM1 type=del size=2264413 2024-03-09T14:36:27.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.918+0000 7f73315818c0 1 < canceled operation key=W6KVP8EQFQ8J tag=KBCFFB4IAENX type=del size=381529 2024-03-09T14:36:27.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.922+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=IKB0C2X2AKSF type=del size=1715488 2024-03-09T14:36:27.928 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.926+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=24DD8GD4694P type=del 2024-03-09T14:36:27.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.930+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=TJOL0BEGNAJ2 type=write size=638102 2024-03-09T14:36:27.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.934+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~W60MFAS998JR.0 size=1048576 2024-03-09T14:36:27.939 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.934+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~W60MFAS998JR.1 size=1030480 2024-03-09T14:36:27.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.942+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=W60MFAS998JR tag=0P9KPMIGNIRY type=write size=3127632 2024-03-09T14:36:27.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.946+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=W3HFCBODL309 type=del size=1722939 2024-03-09T14:36:27.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.950+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=EUVWBQ9VS5VZ type=del 2024-03-09T14:36:27.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.954+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=6XVXDX650DEA type=del size=2790987 2024-03-09T14:36:27.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.954+0000 7f73315818c0 1 < completed delete operation key=0WGGMHTEF2LB tag=VDQ5GIE5ND28 type=del 2024-03-09T14:36:27.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.962+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~28ZTVDLZLPBH.0 size=440831 2024-03-09T14:36:27.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.962+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=28ZTVDLZLPBH tag=X6DEF127EU2A type=write size=1489407 2024-03-09T14:36:27.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.966+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=9V9HJFKNYKON type=del 2024-03-09T14:36:27.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.970+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=BRDPRV1BK0WD type=del size=3441155 2024-03-09T14:36:27.978 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.974+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~TD6ZSHGDU2W9.0 size=895784 2024-03-09T14:36:27.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.978+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=TD6ZSHGDU2W9 tag=4WSOPNM7JG1W type=write size=1944360 2024-03-09T14:36:27.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.982+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=AO32PMQPO6EX type=del size=3305730 2024-03-09T14:36:27.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.986+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~HWYGPU8GTXOJ.0 size=1048576 2024-03-09T14:36:27.995 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.990+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~HWYGPU8GTXOJ.1 size=1048576 2024-03-09T14:36:27.998 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:27.994+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~HWYGPU8GTXOJ.2 size=120955 2024-03-09T14:36:28.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.002+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=HWYGPU8GTXOJ tag=Q70RIZD3DN49 type=write size=3266683 2024-03-09T14:36:28.007 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.006+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=F0XDU567UWP3 type=del size=2302780 2024-03-09T14:36:28.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.010+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~X8K08JX7DA20.0 size=165011 2024-03-09T14:36:28.017 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.014+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=X8K08JX7DA20 tag=L73KSWJM0E3B type=write size=1213587 2024-03-09T14:36:28.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.018+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=66B38KNX1BBC type=del size=618299 2024-03-09T14:36:28.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.022+0000 7f73315818c0 1 < completed delete operation key=0IB00CZ92VV4 tag=8KIEFQ5773TG type=del 2024-03-09T14:36:28.031 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.026+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~13XGDK4A23GV.0 size=1048576 2024-03-09T14:36:28.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.030+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~13XGDK4A23GV.1 size=1048576 2024-03-09T14:36:28.036 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.034+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~13XGDK4A23GV.2 size=1003294 2024-03-09T14:36:28.036 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.034+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=13XGDK4A23GV tag=G75F2S6BBYPB type=write size=4149022 2024-03-09T14:36:28.037 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.034+0000 7f73315818c0 1 < canceled part key=_multipart_8O5OP8C5NX12.2~6ZALEO9PBKWR.0 size=1048576 2024-03-09T14:36:28.039 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.038+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~6ZALEO9PBKWR.1 size=834756 2024-03-09T14:36:28.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.042+0000 7f73315818c0 1 < canceled multipart upload key=8O5OP8C5NX12 upload=6ZALEO9PBKWR tag=93XGIBD3ROR8 type=write size=2931908 2024-03-09T14:36:28.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.046+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=2URFS2ZSF8AO type=del size=1321674 2024-03-09T14:36:28.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.046+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=BK2PE7A8ANHB type=del 2024-03-09T14:36:28.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.054+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~4OFKMMWG2Z8O.0 size=1048576 2024-03-09T14:36:28.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.058+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~4OFKMMWG2Z8O.1 size=966287 2024-03-09T14:36:28.060 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.058+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=4OFKMMWG2Z8O tag=0OLHI3UYE513 type=write size=3063439 2024-03-09T14:36:28.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.058+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~70OXI6UPK4WK.0 size=1048576 2024-03-09T14:36:28.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.062+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~70OXI6UPK4WK.1 size=433749 2024-03-09T14:36:28.071 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.066+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=70OXI6UPK4WK tag=XLF92SPBJKT2 type=write size=2530901 2024-03-09T14:36:28.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.074+0000 7f73315818c0 1 > prepared part key=_multipart_FYFVW9R0M1PX.2~B2EW1IFSUU74.0 size=298809 2024-03-09T14:36:28.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.074+0000 7f73315818c0 1 > prepared multipart upload key=FYFVW9R0M1PX upload=B2EW1IFSUU74 tag=01P89YPI3H8G type=write size=1347385 2024-03-09T14:36:28.080 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.078+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~GPO9NJWA1NA0.0 size=1048576 2024-03-09T14:36:28.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.078+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~GPO9NJWA1NA0.1 size=1048576 2024-03-09T14:36:28.085 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.082+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~GPO9NJWA1NA0.2 size=1039827 2024-03-09T14:36:28.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.086+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=GPO9NJWA1NA0 tag=480QM2J7Z8L1 type=write size=4185555 2024-03-09T14:36:28.097 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.094+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~1QU10879EBVX.0 size=1048576 2024-03-09T14:36:28.099 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.098+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~1QU10879EBVX.1 size=1048576 2024-03-09T14:36:28.102 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.098+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~1QU10879EBVX.2 size=642052 2024-03-09T14:36:28.102 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.098+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=1QU10879EBVX tag=AS85ISRXDLCW type=write size=3787780 2024-03-09T14:36:28.104 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.102+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=VRE2WXMRUD6I type=del 2024-03-09T14:36:28.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.106+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=S10DNS19T7W1 type=del size=3625367 2024-03-09T14:36:28.110 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.106+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=Y0027YD8NNM1 type=del 2024-03-09T14:36:28.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.110+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=QJDWV2Y8FIZN type=del size=24212 2024-03-09T14:36:28.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.114+0000 7f73315818c0 1 < completed delete operation key=P8R5RJI8LGZ9 tag=IKB0C2X2AKSF type=del 2024-03-09T14:36:28.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.118+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=MQEHSZUBVNDQ type=write size=709600 2024-03-09T14:36:28.122 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.118+0000 7f73315818c0 1 < completed write operation key=FYFVW9R0M1PX tag=TJOL0BEGNAJ2 type=write size=638102 2024-03-09T14:36:28.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.126+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~PA0OFQBTQ6Q8.0 size=1048576 2024-03-09T14:36:28.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.126+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~PA0OFQBTQ6Q8.1 size=1048576 2024-03-09T14:36:28.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.130+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~PA0OFQBTQ6Q8.2 size=68256 2024-03-09T14:36:28.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.130+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=PA0OFQBTQ6Q8 tag=6QKKF3M7NQMB type=write size=3213984 2024-03-09T14:36:28.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.134+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=W3HFCBODL309 type=del 2024-03-09T14:36:28.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.134+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=J1LY6G2ZOS6G type=del size=2268252 2024-03-09T14:36:28.142 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.138+0000 7f73315818c0 1 < completed delete operation key=S302JFYEDEL7 tag=6XVXDX650DEA type=del 2024-03-09T14:36:28.149 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.146+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~XGBVDVWRQ87A.0 size=1048576 2024-03-09T14:36:28.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.146+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~XGBVDVWRQ87A.1 size=191261 2024-03-09T14:36:28.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.146+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=XGBVDVWRQ87A tag=SLTJX0K8JV37 type=write size=2288413 2024-03-09T14:36:28.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.150+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~28ZTVDLZLPBH.0 size=440831 2024-03-09T14:36:28.160 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.158+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=28ZTVDLZLPBH tag=X6DEF127EU2A type=write size=1489407 2024-03-09T14:36:28.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.162+0000 7f73315818c0 1 > prepared part key=_multipart_M1B6O6N8N84T.2~90NARQ90IZA9.0 size=444181 2024-03-09T14:36:28.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.162+0000 7f73315818c0 1 > prepared multipart upload key=M1B6O6N8N84T upload=90NARQ90IZA9 tag=NL1Z86TKCIC8 type=write size=1492757 2024-03-09T14:36:28.169 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.166+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=BRDPRV1BK0WD type=del 2024-03-09T14:36:28.172 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.170+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=TZNA3GIXZMPP type=write size=985237 2024-03-09T14:36:28.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.174+0000 7f73315818c0 1 < suggested remove operation key=K3JNN1NCKBXN tag=AO32PMQPO6EX type=del 2024-03-09T14:36:28.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.178+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=D1I6S93KSTQS type=del size=3937607 2024-03-09T14:36:28.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.182+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=F0XDU567UWP3 type=del 2024-03-09T14:36:28.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.182+0000 7f73315818c0 1 > prepared operation key=9B0RAHZZGLYU tag=OX34QKAILI6O type=write size=565957 2024-03-09T14:36:28.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.186+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=66B38KNX1BBC type=del 2024-03-09T14:36:28.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.190+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=Z1PXOU60F875 type=write size=251494 2024-03-09T14:36:28.196 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.194+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~13XGDK4A23GV.0 size=1048576 2024-03-09T14:36:28.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.194+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~13XGDK4A23GV.1 size=1048576 2024-03-09T14:36:28.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.198+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~13XGDK4A23GV.2 size=1003294 2024-03-09T14:36:28.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.202+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=13XGDK4A23GV tag=G75F2S6BBYPB type=write size=4149022 2024-03-09T14:36:28.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.206+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=HTLWAFIGSHKD type=del size=3672698 2024-03-09T14:36:28.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.210+0000 7f73315818c0 1 < suggested update operation key=PAJYGD2GXHDT tag=2URFS2ZSF8AO type=del size=1321674 2024-03-09T14:36:28.220 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.218+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~9W2F7RRQLPTO.0 size=1048576 2024-03-09T14:36:28.222 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.218+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~9W2F7RRQLPTO.1 size=806926 2024-03-09T14:36:28.222 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.218+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=9W2F7RRQLPTO tag=VDAXI251WNI7 type=write size=2904078 2024-03-09T14:36:28.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.222+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~4OFKMMWG2Z8O.0 size=1048576 2024-03-09T14:36:28.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.226+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~4OFKMMWG2Z8O.1 size=966287 2024-03-09T14:36:28.233 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.230+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=4OFKMMWG2Z8O tag=0OLHI3UYE513 type=write size=3063439 2024-03-09T14:36:28.236 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.234+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=T2HSBYT3I0Z7 type=del size=836735 2024-03-09T14:36:28.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.234+0000 7f73315818c0 1 < completed part key=_multipart_FYFVW9R0M1PX.2~B2EW1IFSUU74.0 size=298809 2024-03-09T14:36:28.243 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.242+0000 7f73315818c0 1 < completed multipart upload key=FYFVW9R0M1PX upload=B2EW1IFSUU74 tag=01P89YPI3H8G type=write size=1347385 2024-03-09T14:36:28.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.242+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=1F5TR102KC91 type=del size=986952 2024-03-09T14:36:28.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.242+0000 7f73315818c0 1 < canceled part key=_multipart_M1B6O6N8N84T.2~1QU10879EBVX.0 size=1048576 2024-03-09T14:36:28.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.246+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~1QU10879EBVX.1 size=1048576 2024-03-09T14:36:28.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.250+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~1QU10879EBVX.2 size=642052 2024-03-09T14:36:28.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.254+0000 7f73315818c0 1 < canceled multipart upload key=M1B6O6N8N84T upload=1QU10879EBVX tag=AS85ISRXDLCW type=write size=3787780 2024-03-09T14:36:28.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.258+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=13UMIHXVV806 type=write size=264564 2024-03-09T14:36:28.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.262+0000 7f73315818c0 1 < canceled operation key=FYFVW9R0M1PX tag=S10DNS19T7W1 type=del size=3625367 2024-03-09T14:36:28.271 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.270+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~L07BIFXBH96X.0 size=1048576 2024-03-09T14:36:28.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.270+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~L07BIFXBH96X.1 size=1048576 2024-03-09T14:36:28.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.274+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~L07BIFXBH96X.2 size=94060 2024-03-09T14:36:28.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.274+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=L07BIFXBH96X tag=R4KMWY5JIQVS type=write size=3239788 2024-03-09T14:36:28.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.278+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=QJDWV2Y8FIZN type=del 2024-03-09T14:36:28.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.278+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=D7LR0Y6831EF type=del size=2077743 2024-03-09T14:36:28.291 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.286+0000 7f73315818c0 1 < completed write operation key=3ADG4GWVARC1 tag=MQEHSZUBVNDQ type=write size=709600 2024-03-09T14:36:28.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.290+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=VJ3RX6C47BA5 type=del size=1663055 2024-03-09T14:36:28.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.294+0000 7f73315818c0 1 < canceled part key=_multipart_UMSNM53HTY1J.2~PA0OFQBTQ6Q8.0 size=1048576 2024-03-09T14:36:28.299 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.294+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~PA0OFQBTQ6Q8.1 size=1048576 2024-03-09T14:36:28.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.298+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~PA0OFQBTQ6Q8.2 size=68256 2024-03-09T14:36:28.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.306+0000 7f73315818c0 1 < canceled multipart upload key=UMSNM53HTY1J upload=PA0OFQBTQ6Q8 tag=6QKKF3M7NQMB type=write size=3213984 2024-03-09T14:36:28.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.310+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=8OTRN607V21W type=del size=1967433 2024-03-09T14:36:28.316 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.314+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=J1LY6G2ZOS6G type=del 2024-03-09T14:36:28.323 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.318+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~1SKYEPBC4OES.0 size=1048576 2024-03-09T14:36:28.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.322+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~1SKYEPBC4OES.1 size=145773 2024-03-09T14:36:28.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.322+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=1SKYEPBC4OES tag=HSO7ICJ50YJT type=write size=2242925 2024-03-09T14:36:28.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.322+0000 7f73315818c0 1 < canceled part key=_multipart_4P89277EEYJ3.2~XGBVDVWRQ87A.0 size=1048576 2024-03-09T14:36:28.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.322+0000 7f73315818c0 1 < canceled part key=_multipart_4P89277EEYJ3.2~XGBVDVWRQ87A.1 size=191261 2024-03-09T14:36:28.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.330+0000 7f73315818c0 1 < canceled multipart upload key=4P89277EEYJ3 upload=XGBVDVWRQ87A tag=SLTJX0K8JV37 type=write size=2288413 2024-03-09T14:36:28.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.334+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=4POGFYLOQCFH type=del size=3267318 2024-03-09T14:36:28.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.338+0000 7f73315818c0 1 < completed part key=_multipart_M1B6O6N8N84T.2~90NARQ90IZA9.0 size=444181 2024-03-09T14:36:28.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.342+0000 7f73315818c0 1 < completed multipart upload key=M1B6O6N8N84T upload=90NARQ90IZA9 tag=NL1Z86TKCIC8 type=write size=1492757 2024-03-09T14:36:28.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.346+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~4UFVPMKGCRAA.0 size=1048576 2024-03-09T14:36:28.352 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.350+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~4UFVPMKGCRAA.1 size=1048576 2024-03-09T14:36:28.356 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.354+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~4UFVPMKGCRAA.2 size=337381 2024-03-09T14:36:28.356 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.354+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=4UFVPMKGCRAA tag=G85TMMX7UO2R type=write size=3483109 2024-03-09T14:36:28.359 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.358+0000 7f73315818c0 1 < completed write operation key=FYFVW9R0M1PX tag=TZNA3GIXZMPP type=write size=985237 2024-03-09T14:36:28.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.362+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~F9WM391NFSCZ.0 size=340173 2024-03-09T14:36:28.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.362+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=F9WM391NFSCZ tag=DO041CX6EGII type=write size=1388749 2024-03-09T14:36:28.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.366+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=D1I6S93KSTQS type=del 2024-03-09T14:36:28.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.366+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=MMVX8VK6OP2C type=del size=3098617 2024-03-09T14:36:28.377 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.374+0000 7f73315818c0 1 < suggested remove operation key=9B0RAHZZGLYU tag=OX34QKAILI6O type=write 2024-03-09T14:36:28.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.378+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~8EUK3SLKUT4P.0 size=1048576 2024-03-09T14:36:28.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.382+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~8EUK3SLKUT4P.1 size=1048576 2024-03-09T14:36:28.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.386+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~8EUK3SLKUT4P.2 size=482205 2024-03-09T14:36:28.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.386+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=8EUK3SLKUT4P tag=HLFPEAFI75HW type=write size=3627933 2024-03-09T14:36:28.393 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.390+0000 7f73315818c0 1 < suggested update operation key=3ADG4GWVARC1 tag=Z1PXOU60F875 type=write size=251494 2024-03-09T14:36:28.398 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.394+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=0EWRX1RBQIH2 type=del size=2777501 2024-03-09T14:36:28.400 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.398+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=HTLWAFIGSHKD type=del 2024-03-09T14:36:28.404 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.402+0000 7f73315818c0 1 > prepared operation key=S302JFYEDEL7 tag=6MW7B1FFZ2PQ type=write size=1045460 2024-03-09T14:36:28.404 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.402+0000 7f73315818c0 1 < canceled part key=_multipart_0LP8KKU53X0A.2~9W2F7RRQLPTO.0 size=1048576 2024-03-09T14:36:28.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.402+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~9W2F7RRQLPTO.1 size=806926 2024-03-09T14:36:28.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.410+0000 7f73315818c0 1 < canceled multipart upload key=0LP8KKU53X0A upload=9W2F7RRQLPTO tag=VDAXI251WNI7 type=write size=2904078 2024-03-09T14:36:28.415 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.410+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=W7QD59UPJU75 type=del size=331012 2024-03-09T14:36:28.418 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.414+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=T2HSBYT3I0Z7 type=del 2024-03-09T14:36:28.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.418+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=HAN7RGWCANTU type=del size=2620234 2024-03-09T14:36:28.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.422+0000 7f73315818c0 1 < canceled operation key=LZC7NPF0GRO7 tag=1F5TR102KC91 type=del size=986952 2024-03-09T14:36:28.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.426+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=98S9VBIXYQIC type=del size=1433308 2024-03-09T14:36:28.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.430+0000 7f73315818c0 1 < suggested update operation key=4E3QMBHEAEHO tag=13UMIHXVV806 type=write size=264564 2024-03-09T14:36:28.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.438+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~TZUCKYNOY2PD.0 size=1048576 2024-03-09T14:36:28.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.438+0000 7f73315818c0 1 > prepared part key=_multipart_3ADG4GWVARC1.2~TZUCKYNOY2PD.1 size=366748 2024-03-09T14:36:28.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.438+0000 7f73315818c0 1 > prepared multipart upload key=3ADG4GWVARC1 upload=TZUCKYNOY2PD tag=YGUVD3C0UGEL type=write size=2463900 2024-03-09T14:36:28.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.442+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~L07BIFXBH96X.0 size=1048576 2024-03-09T14:36:28.449 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.446+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~L07BIFXBH96X.1 size=1048576 2024-03-09T14:36:28.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.450+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~L07BIFXBH96X.2 size=94060 2024-03-09T14:36:28.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.454+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=L07BIFXBH96X tag=R4KMWY5JIQVS type=write size=3239788 2024-03-09T14:36:28.460 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.458+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=8KYU994VX00F type=del size=1303383 2024-03-09T14:36:28.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.458+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=D7LR0Y6831EF type=del 2024-03-09T14:36:28.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.462+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=JV2YXYHBG6GX type=write size=1008820 2024-03-09T14:36:28.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.466+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=VJ3RX6C47BA5 type=del 2024-03-09T14:36:28.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.470+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=CX7YLM1LBNZ8 type=del size=2810236 2024-03-09T14:36:28.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.474+0000 7f73315818c0 -1 < failed to complete operation key=4E3QMBHEAEHO tag=8OTRN607V21W type=del size=1967433: (22) Invalid argument 2024-03-09T14:36:28.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.482+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~T0GX07X5D9ZZ.0 size=26807 2024-03-09T14:36:28.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.482+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=T0GX07X5D9ZZ tag=28OTXTVUTQGD type=write size=1075383 2024-03-09T14:36:28.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.486+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~1SKYEPBC4OES.0 size=1048576 2024-03-09T14:36:28.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.486+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~1SKYEPBC4OES.1 size=145773 2024-03-09T14:36:28.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.494+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=1SKYEPBC4OES tag=HSO7ICJ50YJT type=write size=2242925 2024-03-09T14:36:28.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.498+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~Z623HBE6H3VC.0 size=1048576 2024-03-09T14:36:28.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.502+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~Z623HBE6H3VC.1 size=909854 2024-03-09T14:36:28.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.502+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=Z623HBE6H3VC tag=05IV7V327V49 type=write size=3007006 2024-03-09T14:36:28.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.506+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=4POGFYLOQCFH type=del 2024-03-09T14:36:28.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.510+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=YBADDUAA8T2P type=del size=2412565 2024-03-09T14:36:28.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.510+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~4UFVPMKGCRAA.0 size=1048576 2024-03-09T14:36:28.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.514+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~4UFVPMKGCRAA.1 size=1048576 2024-03-09T14:36:28.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.514+0000 7f73315818c0 1 < canceled part key=_multipart_S302JFYEDEL7.2~4UFVPMKGCRAA.2 size=337381 2024-03-09T14:36:28.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.522+0000 7f73315818c0 1 < canceled multipart upload key=S302JFYEDEL7 upload=4UFVPMKGCRAA tag=G85TMMX7UO2R type=write size=3483109 2024-03-09T14:36:28.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.522+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=49OH5BIM9MWD type=del size=1081836 2024-03-09T14:36:28.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.526+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~F9WM391NFSCZ.0 size=340173 2024-03-09T14:36:28.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.534+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=F9WM391NFSCZ tag=DO041CX6EGII type=write size=1388749 2024-03-09T14:36:28.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.538+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=4L8KGU2LI7II type=del size=2770023 2024-03-09T14:36:28.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.542+0000 7f73315818c0 1 < canceled operation key=LZC7NPF0GRO7 tag=MMVX8VK6OP2C type=del size=3098617 2024-03-09T14:36:28.547 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.542+0000 7f73315818c0 1 > prepared operation key=CVAWRZMAX997 tag=740FIAE31BXP type=del size=2700991 2024-03-09T14:36:28.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.546+0000 7f73315818c0 1 < canceled part key=_multipart_DQOY1JJ1YQ2W.2~8EUK3SLKUT4P.0 size=1048576 2024-03-09T14:36:28.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.546+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~8EUK3SLKUT4P.1 size=1048576 2024-03-09T14:36:28.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.550+0000 7f73315818c0 1 < completed part key=_multipart_DQOY1JJ1YQ2W.2~8EUK3SLKUT4P.2 size=482205 2024-03-09T14:36:28.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.558+0000 7f73315818c0 1 < canceled multipart upload key=DQOY1JJ1YQ2W upload=8EUK3SLKUT4P tag=HLFPEAFI75HW type=write size=3627933 2024-03-09T14:36:28.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.558+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=4WEEQA8W5J8F type=write size=505611 2024-03-09T14:36:28.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.562+0000 7f73315818c0 1 < canceled operation key=LZC7NPF0GRO7 tag=0EWRX1RBQIH2 type=del size=2777501 2024-03-09T14:36:28.573 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.570+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~HZ961NZSIGL5.0 size=1048576 2024-03-09T14:36:28.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.570+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~HZ961NZSIGL5.1 size=408312 2024-03-09T14:36:28.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.570+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=HZ961NZSIGL5 tag=JTQIUWQNUET2 type=write size=2505464 2024-03-09T14:36:28.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.574+0000 7f73315818c0 1 < canceled operation key=S302JFYEDEL7 tag=6MW7B1FFZ2PQ type=write size=1045460 2024-03-09T14:36:28.585 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.582+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~SYGKETY77IQP.0 size=1048576 2024-03-09T14:36:28.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.586+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~SYGKETY77IQP.1 size=215674 2024-03-09T14:36:28.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.586+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=SYGKETY77IQP tag=BR6H6X8GF6Z7 type=write size=2312826 2024-03-09T14:36:28.591 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.586+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=W7QD59UPJU75 type=del 2024-03-09T14:36:28.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.594+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~6QIT1BLU3CMP.0 size=589308 2024-03-09T14:36:28.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.594+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=6QIT1BLU3CMP tag=XXO4KRYFHOU2 type=write size=1637884 2024-03-09T14:36:28.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.598+0000 7f73315818c0 1 < completed delete operation key=4P89277EEYJ3 tag=HAN7RGWCANTU type=del 2024-03-09T14:36:28.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.606+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=7EGEVNQMYZ4W type=del size=3518667 2024-03-09T14:36:28.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.614+0000 7f73315818c0 1 < suggested update operation key=0WGGMHTEF2LB tag=98S9VBIXYQIC type=del size=1433308 2024-03-09T14:36:28.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.618+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~PEXFWPCBU3SR.0 size=926067 2024-03-09T14:36:28.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.618+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=PEXFWPCBU3SR tag=XQC7WKP63UHP type=write size=1974643 2024-03-09T14:36:28.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.622+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~TZUCKYNOY2PD.0 size=1048576 2024-03-09T14:36:28.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.626+0000 7f73315818c0 1 < completed part key=_multipart_3ADG4GWVARC1.2~TZUCKYNOY2PD.1 size=366748 2024-03-09T14:36:28.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.630+0000 7f73315818c0 1 < completed multipart upload key=3ADG4GWVARC1 upload=TZUCKYNOY2PD tag=YGUVD3C0UGEL type=write size=2463900 2024-03-09T14:36:28.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.638+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~IYW9YGPG7EKR.0 size=1048576 2024-03-09T14:36:28.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.638+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~IYW9YGPG7EKR.1 size=1048576 2024-03-09T14:36:28.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.642+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~IYW9YGPG7EKR.2 size=719765 2024-03-09T14:36:28.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.642+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=IYW9YGPG7EKR tag=ZZEXA19FCGWE type=write size=3865493 2024-03-09T14:36:28.652 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.650+0000 7f73315818c0 1 < suggested update operation key=MD7FXGYXB823 tag=8KYU994VX00F type=del size=1303383 2024-03-09T14:36:28.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.654+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~7C2NVTNJSULY.0 size=1048576 2024-03-09T14:36:28.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.658+0000 7f73315818c0 1 > prepared part key=_multipart_M8F9TA80VME6.2~7C2NVTNJSULY.1 size=355772 2024-03-09T14:36:28.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.658+0000 7f73315818c0 1 > prepared multipart upload key=M8F9TA80VME6 upload=7C2NVTNJSULY tag=ALBA1XRPI73S type=write size=2452924 2024-03-09T14:36:28.665 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.662+0000 7f73315818c0 1 < completed write operation key=W6KVP8EQFQ8J tag=JV2YXYHBG6GX type=write size=1008820 2024-03-09T14:36:28.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.666+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=WQE9LE9U24QI type=write size=896914 2024-03-09T14:36:28.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.670+0000 7f73315818c0 1 < canceled operation key=PXFZLDSAR1EG tag=CX7YLM1LBNZ8 type=del size=2810236 2024-03-09T14:36:28.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.674+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~2VLHA3E8ESO4.0 size=1048576 2024-03-09T14:36:28.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.678+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~2VLHA3E8ESO4.1 size=1048576 2024-03-09T14:36:28.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.682+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~2VLHA3E8ESO4.2 size=266054 2024-03-09T14:36:28.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.682+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=2VLHA3E8ESO4 tag=2NAKK02GNZIP type=write size=3411782 2024-03-09T14:36:28.688 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.686+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~T0GX07X5D9ZZ.0 size=26807 2024-03-09T14:36:28.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.690+0000 7f73315818c0 1 < completed multipart upload key=IA0AB7UVLFEJ upload=T0GX07X5D9ZZ tag=28OTXTVUTQGD type=write size=1075383 2024-03-09T14:36:28.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.694+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=SPZJ43NA2YGX type=del size=1071998 2024-03-09T14:36:28.702 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.698+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~Z623HBE6H3VC.0 size=1048576 2024-03-09T14:36:28.705 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.702+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~Z623HBE6H3VC.1 size=909854 2024-03-09T14:36:28.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.710+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=Z623HBE6H3VC tag=05IV7V327V49 type=write size=3007006 2024-03-09T14:36:28.715 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.714+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=3E89T20BRIV3 type=del size=547452 2024-03-09T14:36:28.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.714+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=YBADDUAA8T2P type=del 2024-03-09T14:36:28.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.718+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=R0IKZSIDQ49R type=del size=974015 2024-03-09T14:36:28.726 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.722+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=49OH5BIM9MWD type=del 2024-03-09T14:36:28.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.726+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=1NJW23GXIE0Y type=del size=240548 2024-03-09T14:36:28.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.730+0000 7f73315818c0 1 < canceled operation key=1NYNL8M5DVER tag=4L8KGU2LI7II type=del size=2770023 2024-03-09T14:36:28.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.734+0000 7f73315818c0 1 > prepared operation key=0LP8KKU53X0A tag=RZ8CG6OE0MD8 type=del size=2861411 2024-03-09T14:36:28.741 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.738+0000 7f73315818c0 1 < completed delete operation key=CVAWRZMAX997 tag=740FIAE31BXP type=del 2024-03-09T14:36:28.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.742+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=HLY0BTB6VC8V type=del size=428865 2024-03-09T14:36:28.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.750+0000 7f73315818c0 1 < suggested update operation key=M8F9TA80VME6 tag=4WEEQA8W5J8F type=write size=505611 2024-03-09T14:36:28.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.754+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=ZD514Z7CFTR2 type=del size=810538 2024-03-09T14:36:28.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.758+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~HZ961NZSIGL5.0 size=1048576 2024-03-09T14:36:28.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.758+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~HZ961NZSIGL5.1 size=408312 2024-03-09T14:36:28.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.766+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=HZ961NZSIGL5 tag=JTQIUWQNUET2 type=write size=2505464 2024-03-09T14:36:28.773 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.770+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=50Y9SNB5BMOJ type=del size=1180507 2024-03-09T14:36:28.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.774+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~SYGKETY77IQP.0 size=1048576 2024-03-09T14:36:28.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.778+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~SYGKETY77IQP.1 size=215674 2024-03-09T14:36:28.786 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.782+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=SYGKETY77IQP tag=BR6H6X8GF6Z7 type=write size=2312826 2024-03-09T14:36:28.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.786+0000 7f73315818c0 1 > prepared operation key=MSMD3ANK4T7J tag=FNB1OX8N5P6Z type=del size=3993890 2024-03-09T14:36:28.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.790+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~6QIT1BLU3CMP.0 size=589308 2024-03-09T14:36:28.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.798+0000 7f73315818c0 1 < completed multipart upload key=PXFZLDSAR1EG upload=6QIT1BLU3CMP tag=XXO4KRYFHOU2 type=write size=1637884 2024-03-09T14:36:28.803 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.802+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=9R3XFX48AIV7 type=del size=1132700 2024-03-09T14:36:28.808 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.806+0000 7f73315818c0 1 < completed delete operation key=0LP8KKU53X0A tag=7EGEVNQMYZ4W type=del 2024-03-09T14:36:28.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.810+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~ASAXG3WLQE0R.0 size=1048576 2024-03-09T14:36:28.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.814+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~ASAXG3WLQE0R.1 size=524580 2024-03-09T14:36:28.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.814+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=ASAXG3WLQE0R tag=LH92PVXDW8RA type=write size=2621732 2024-03-09T14:36:28.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.814+0000 7f73315818c0 1 < canceled part key=_multipart_8O5OP8C5NX12.2~PEXFWPCBU3SR.0 size=926067 2024-03-09T14:36:28.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.822+0000 7f73315818c0 1 < canceled multipart upload key=8O5OP8C5NX12 upload=PEXFWPCBU3SR tag=XQC7WKP63UHP type=write size=1974643 2024-03-09T14:36:28.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.822+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=JK6A888NDR0H type=del size=1148198 2024-03-09T14:36:28.831 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.826+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~IYW9YGPG7EKR.0 size=1048576 2024-03-09T14:36:28.834 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.830+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~IYW9YGPG7EKR.1 size=1048576 2024-03-09T14:36:28.836 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.834+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~IYW9YGPG7EKR.2 size=719765 2024-03-09T14:36:28.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.838+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=IYW9YGPG7EKR tag=ZZEXA19FCGWE type=write size=3865493 2024-03-09T14:36:28.846 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.842+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=SBVYVDPS8PYD type=del size=3231443 2024-03-09T14:36:28.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.846+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~7C2NVTNJSULY.0 size=1048576 2024-03-09T14:36:28.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.850+0000 7f73315818c0 1 < completed part key=_multipart_M8F9TA80VME6.2~7C2NVTNJSULY.1 size=355772 2024-03-09T14:36:28.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.854+0000 7f73315818c0 1 < completed multipart upload key=M8F9TA80VME6 upload=7C2NVTNJSULY tag=ALBA1XRPI73S type=write size=2452924 2024-03-09T14:36:28.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.862+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~DNKO5DV3G0E8.0 size=1048576 2024-03-09T14:36:28.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.866+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~DNKO5DV3G0E8.1 size=175397 2024-03-09T14:36:28.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.866+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=DNKO5DV3G0E8 tag=SWGZW52W5KIF type=write size=2272549 2024-03-09T14:36:28.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.870+0000 7f73315818c0 1 < completed write operation key=4P89277EEYJ3 tag=WQE9LE9U24QI type=write size=896914 2024-03-09T14:36:28.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.874+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=W2BR44H4DER6 type=write size=509034 2024-03-09T14:36:28.880 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.878+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~2VLHA3E8ESO4.0 size=1048576 2024-03-09T14:36:28.886 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.882+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~2VLHA3E8ESO4.1 size=1048576 2024-03-09T14:36:28.886 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.882+0000 7f73315818c0 1 < completed part key=_multipart_MD7FXGYXB823.2~2VLHA3E8ESO4.2 size=266054 2024-03-09T14:36:28.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.890+0000 7f73315818c0 1 < completed multipart upload key=MD7FXGYXB823 upload=2VLHA3E8ESO4 tag=2NAKK02GNZIP type=write size=3411782 2024-03-09T14:36:28.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.894+0000 7f73315818c0 1 > prepared operation key=4E3QMBHEAEHO tag=2G6SFUPLVZMY type=del size=3745380 2024-03-09T14:36:28.900 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.898+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=SPZJ43NA2YGX type=del 2024-03-09T14:36:28.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.902+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~HI1R1AQSEUCX.0 size=1048576 2024-03-09T14:36:28.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.906+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~HI1R1AQSEUCX.1 size=1048576 2024-03-09T14:36:28.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.910+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~HI1R1AQSEUCX.2 size=259263 2024-03-09T14:36:28.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.910+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=HI1R1AQSEUCX tag=3ZSJH1RFUTFT type=write size=3404991 2024-03-09T14:36:28.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.914+0000 7f73315818c0 1 < completed delete operation key=P9LI5ESR3N9G tag=3E89T20BRIV3 type=del 2024-03-09T14:36:28.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.918+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=XX395B6ENYNK type=del size=326762 2024-03-09T14:36:28.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.918+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=R0IKZSIDQ49R type=del 2024-03-09T14:36:28.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.926+0000 7f73315818c0 1 > prepared part key=_multipart_XELG1AWATMVL.2~YAVL0F86K3RZ.0 size=169271 2024-03-09T14:36:28.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.926+0000 7f73315818c0 1 > prepared multipart upload key=XELG1AWATMVL upload=YAVL0F86K3RZ tag=BRXCW39741Z6 type=write size=1217847 2024-03-09T14:36:28.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.930+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=1NJW23GXIE0Y type=del 2024-03-09T14:36:28.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.934+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=KOCWHLSTQO6K type=del size=3476422 2024-03-09T14:36:28.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.942+0000 7f73315818c0 1 < suggested update operation key=0LP8KKU53X0A tag=RZ8CG6OE0MD8 type=del size=2861411 2024-03-09T14:36:28.951 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.950+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~RKHPIA1JNWI0.0 size=1048576 2024-03-09T14:36:28.954 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.950+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~RKHPIA1JNWI0.1 size=35260 2024-03-09T14:36:28.954 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.950+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=RKHPIA1JNWI0 tag=EKS3H6ON4WMA type=write size=2132412 2024-03-09T14:36:28.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.958+0000 7f73315818c0 1 < suggested update operation key=LZC7NPF0GRO7 tag=HLY0BTB6VC8V type=del size=428865 2024-03-09T14:36:28.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.962+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=0JUE6ZJ1LOQC type=del size=1266938 2024-03-09T14:36:28.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.966+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=ZD514Z7CFTR2 type=del 2024-03-09T14:36:28.974 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.970+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~VYE9H2G2UHBC.0 size=1048576 2024-03-09T14:36:28.977 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.974+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~VYE9H2G2UHBC.1 size=1048576 2024-03-09T14:36:28.980 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.978+0000 7f73315818c0 1 > prepared part key=_multipart_PXFZLDSAR1EG.2~VYE9H2G2UHBC.2 size=661748 2024-03-09T14:36:28.980 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.978+0000 7f73315818c0 1 > prepared multipart upload key=PXFZLDSAR1EG upload=VYE9H2G2UHBC tag=TX2TJDONV5X2 type=write size=3807476 2024-03-09T14:36:28.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.982+0000 7f73315818c0 1 < suggested update operation key=QG4O2L526B8E tag=50Y9SNB5BMOJ type=del size=1180507 2024-03-09T14:36:28.993 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.990+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~VGGO2CI3Q73A.0 size=1048576 2024-03-09T14:36:28.995 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.994+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~VGGO2CI3Q73A.1 size=1048576 2024-03-09T14:36:28.998 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.994+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~VGGO2CI3Q73A.2 size=850633 2024-03-09T14:36:28.998 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.994+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=VGGO2CI3Q73A tag=I03M81WNHYB4 type=write size=3996361 2024-03-09T14:36:29.001 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:28.998+0000 7f73315818c0 1 < completed delete operation key=MSMD3ANK4T7J tag=FNB1OX8N5P6Z type=del 2024-03-09T14:36:29.005 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.002+0000 7f73315818c0 1 > prepared operation key=4P89277EEYJ3 tag=GA2EWTJR3WBN type=write size=1017664 2024-03-09T14:36:29.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.006+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=9R3XFX48AIV7 type=del 2024-03-09T14:36:29.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.010+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=TQ5LNUW13U02 type=del size=2050615 2024-03-09T14:36:29.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.014+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~ASAXG3WLQE0R.0 size=1048576 2024-03-09T14:36:29.018 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.014+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~ASAXG3WLQE0R.1 size=524580 2024-03-09T14:36:29.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.022+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=ASAXG3WLQE0R tag=LH92PVXDW8RA type=write size=2621732 2024-03-09T14:36:29.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.026+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~L2HPDM3E8KKS.0 size=1048576 2024-03-09T14:36:29.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.030+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~L2HPDM3E8KKS.1 size=1048576 2024-03-09T14:36:29.035 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.034+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~L2HPDM3E8KKS.2 size=169805 2024-03-09T14:36:29.035 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.034+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=L2HPDM3E8KKS tag=08LCAU0WZ9SK type=write size=3315533 2024-03-09T14:36:29.039 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.034+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=JK6A888NDR0H type=del 2024-03-09T14:36:29.046 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.042+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~BJ5GOLIPIRXS.0 size=1048576 2024-03-09T14:36:29.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.046+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~BJ5GOLIPIRXS.1 size=729884 2024-03-09T14:36:29.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.046+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=BJ5GOLIPIRXS tag=IYPA14KTBH2V type=write size=2827036 2024-03-09T14:36:29.052 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.050+0000 7f73315818c0 1 < canceled operation key=MD7FXGYXB823 tag=SBVYVDPS8PYD type=del size=3231443 2024-03-09T14:36:29.055 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.054+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=MXAM484TQVLP type=del size=529994 2024-03-09T14:36:29.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.054+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~DNKO5DV3G0E8.0 size=1048576 2024-03-09T14:36:29.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.058+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~DNKO5DV3G0E8.1 size=175397 2024-03-09T14:36:29.068 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.066+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=DNKO5DV3G0E8 tag=SWGZW52W5KIF type=write size=2272549 2024-03-09T14:36:29.071 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.070+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=EPD66BWSGF9I type=del size=891431 2024-03-09T14:36:29.075 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.074+0000 7f73315818c0 1 < canceled operation key=8O5OP8C5NX12 tag=W2BR44H4DER6 type=write size=509034 2024-03-09T14:36:29.079 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.078+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=E16NADB8TME0 type=write size=499750 2024-03-09T14:36:29.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.082+0000 7f73315818c0 1 < completed delete operation key=4E3QMBHEAEHO tag=2G6SFUPLVZMY type=del 2024-03-09T14:36:29.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.086+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=PHHJAV98BLEQ type=write size=635223 2024-03-09T14:36:29.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.090+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~HI1R1AQSEUCX.0 size=1048576 2024-03-09T14:36:29.094 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.090+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~HI1R1AQSEUCX.1 size=1048576 2024-03-09T14:36:29.097 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.094+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~HI1R1AQSEUCX.2 size=259263 2024-03-09T14:36:29.103 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.098+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=HI1R1AQSEUCX tag=3ZSJH1RFUTFT type=write size=3404991 2024-03-09T14:36:29.106 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.102+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=0DS9TDIYJZK0 type=del size=1122553 2024-03-09T14:36:29.110 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.106+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=XX395B6ENYNK type=del 2024-03-09T14:36:29.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.114+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~28CNCP02T0XK.0 size=869725 2024-03-09T14:36:29.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.114+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=28CNCP02T0XK tag=RBQM4ONSIZV6 type=write size=1918301 2024-03-09T14:36:29.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.118+0000 7f73315818c0 1 < completed part key=_multipart_XELG1AWATMVL.2~YAVL0F86K3RZ.0 size=169271 2024-03-09T14:36:29.126 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.122+0000 7f73315818c0 1 < completed multipart upload key=XELG1AWATMVL upload=YAVL0F86K3RZ tag=BRXCW39741Z6 type=write size=1217847 2024-03-09T14:36:29.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.126+0000 7f73315818c0 1 > prepared operation key=LZC7NPF0GRO7 tag=FQI44FLF2IRE type=del size=361173 2024-03-09T14:36:29.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.130+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=KOCWHLSTQO6K type=del 2024-03-09T14:36:29.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.134+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=T33HNOW8FCH1 type=del size=473040 2024-03-09T14:36:29.140 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.138+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~RKHPIA1JNWI0.0 size=1048576 2024-03-09T14:36:29.143 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.142+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~RKHPIA1JNWI0.1 size=35260 2024-03-09T14:36:29.149 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.146+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=RKHPIA1JNWI0 tag=EKS3H6ON4WMA type=write size=2132412 2024-03-09T14:36:29.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.150+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~JE0TCPHFJMHI.0 size=1048576 2024-03-09T14:36:29.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.154+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~JE0TCPHFJMHI.1 size=471009 2024-03-09T14:36:29.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.154+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=JE0TCPHFJMHI tag=XOTVZLQE2KT5 type=write size=2568161 2024-03-09T14:36:29.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.158+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=0JUE6ZJ1LOQC type=del 2024-03-09T14:36:29.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.162+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~VZFF11AT6HP9.0 size=1048576 2024-03-09T14:36:29.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.166+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~VZFF11AT6HP9.1 size=250673 2024-03-09T14:36:29.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.166+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=VZFF11AT6HP9 tag=S832H17HJJC2 type=write size=2347825 2024-03-09T14:36:29.174 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.170+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~VYE9H2G2UHBC.0 size=1048576 2024-03-09T14:36:29.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.174+0000 7f73315818c0 1 < completed part key=_multipart_PXFZLDSAR1EG.2~VYE9H2G2UHBC.1 size=1048576 2024-03-09T14:36:29.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.174+0000 7f73315818c0 1 < canceled part key=_multipart_PXFZLDSAR1EG.2~VYE9H2G2UHBC.2 size=661748 2024-03-09T14:36:29.182 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.178+0000 7f73315818c0 1 < canceled multipart upload key=PXFZLDSAR1EG upload=VYE9H2G2UHBC tag=TX2TJDONV5X2 type=write size=3807476 2024-03-09T14:36:29.186 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.182+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=9I6N10TIWRKI type=del size=993310 2024-03-09T14:36:29.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.186+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~VGGO2CI3Q73A.0 size=1048576 2024-03-09T14:36:29.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.190+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~VGGO2CI3Q73A.1 size=1048576 2024-03-09T14:36:29.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.194+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~VGGO2CI3Q73A.2 size=850633 2024-03-09T14:36:29.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.198+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=VGGO2CI3Q73A tag=I03M81WNHYB4 type=write size=3996361 2024-03-09T14:36:29.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.202+0000 7f73315818c0 1 > prepared operation key=8O5OP8C5NX12 tag=2EEAO309W2WC type=del size=1278792 2024-03-09T14:36:29.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.210+0000 7f73315818c0 1 < suggested remove operation key=4P89277EEYJ3 tag=GA2EWTJR3WBN type=write 2024-03-09T14:36:29.217 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.214+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=QS3BVN1PJHBJ type=write size=524516 2024-03-09T14:36:29.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.218+0000 7f73315818c0 1 < completed delete operation key=DQOY1JJ1YQ2W tag=TQ5LNUW13U02 type=del 2024-03-09T14:36:29.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.222+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=7UUFFEXB1NTK type=del size=638279 2024-03-09T14:36:29.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.226+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~L2HPDM3E8KKS.0 size=1048576 2024-03-09T14:36:29.232 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.230+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~L2HPDM3E8KKS.1 size=1048576 2024-03-09T14:36:29.235 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.230+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~L2HPDM3E8KKS.2 size=169805 2024-03-09T14:36:29.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.238+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=L2HPDM3E8KKS tag=08LCAU0WZ9SK type=write size=3315533 2024-03-09T14:36:29.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.246+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~DTQ9P8LT3MFG.0 size=423653 2024-03-09T14:36:29.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.246+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=DTQ9P8LT3MFG tag=SGQU4G53U3ZO type=write size=1472229 2024-03-09T14:36:29.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.250+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~BJ5GOLIPIRXS.0 size=1048576 2024-03-09T14:36:29.255 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.250+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~BJ5GOLIPIRXS.1 size=729884 2024-03-09T14:36:29.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.258+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=BJ5GOLIPIRXS tag=IYPA14KTBH2V type=write size=2827036 2024-03-09T14:36:29.267 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.266+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~1R2HGNFN8G2Y.0 size=1048576 2024-03-09T14:36:29.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.270+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~1R2HGNFN8G2Y.1 size=442079 2024-03-09T14:36:29.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.270+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=1R2HGNFN8G2Y tag=XPP6222JGKO6 type=write size=2539231 2024-03-09T14:36:29.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.274+0000 7f73315818c0 1 < canceled operation key=OU0730HGI176 tag=MXAM484TQVLP type=del size=529994 2024-03-09T14:36:29.282 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.278+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=SJLDH4DSB6UD type=del size=394282 2024-03-09T14:36:29.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.282+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=EPD66BWSGF9I type=del 2024-03-09T14:36:29.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.286+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=RDBFLEQRVZ8F type=del size=2753300 2024-03-09T14:36:29.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.294+0000 7f73315818c0 1 < suggested update operation key=PAJYGD2GXHDT tag=E16NADB8TME0 type=write size=499750 2024-03-09T14:36:29.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.298+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=77UI0UF8AGAW type=write size=726119 2024-03-09T14:36:29.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.302+0000 7f73315818c0 1 < completed write operation key=0WGGMHTEF2LB tag=PHHJAV98BLEQ type=write size=635223 2024-03-09T14:36:29.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.306+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=VPKSAHNMPIRZ type=del size=1207714 2024-03-09T14:36:29.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.310+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=0DS9TDIYJZK0 type=del 2024-03-09T14:36:29.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.318+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~YVL2EAVDIQLX.0 size=1048576 2024-03-09T14:36:29.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.318+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~YVL2EAVDIQLX.1 size=809965 2024-03-09T14:36:29.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.318+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=YVL2EAVDIQLX tag=HNMXH42K6EKP type=write size=2907117 2024-03-09T14:36:29.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.322+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~28CNCP02T0XK.0 size=869725 2024-03-09T14:36:29.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.330+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=28CNCP02T0XK tag=RBQM4ONSIZV6 type=write size=1918301 2024-03-09T14:36:29.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.334+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=23SMPS4FDWY9 type=del size=2694961 2024-03-09T14:36:29.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.338+0000 7f73315818c0 1 < completed delete operation key=LZC7NPF0GRO7 tag=FQI44FLF2IRE type=del 2024-03-09T14:36:29.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.342+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=Y88AVXCHLS9R type=write size=538039 2024-03-09T14:36:29.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.346+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=T33HNOW8FCH1 type=del 2024-03-09T14:36:29.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.350+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~773XHTCRDGWM.0 size=1048576 2024-03-09T14:36:29.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.354+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~773XHTCRDGWM.1 size=746199 2024-03-09T14:36:29.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.354+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=773XHTCRDGWM tag=Y8V73BFXWHF8 type=write size=2843351 2024-03-09T14:36:29.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.358+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~JE0TCPHFJMHI.0 size=1048576 2024-03-09T14:36:29.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.362+0000 7f73315818c0 1 < completed part key=_multipart_1NYNL8M5DVER.2~JE0TCPHFJMHI.1 size=471009 2024-03-09T14:36:29.372 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.370+0000 7f73315818c0 1 < completed multipart upload key=1NYNL8M5DVER upload=JE0TCPHFJMHI tag=XOTVZLQE2KT5 type=write size=2568161 2024-03-09T14:36:29.376 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.374+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=KLI0K5LRJ87G type=del size=582387 2024-03-09T14:36:29.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.378+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~VZFF11AT6HP9.0 size=1048576 2024-03-09T14:36:29.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.382+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~VZFF11AT6HP9.1 size=250673 2024-03-09T14:36:29.390 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.386+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=VZFF11AT6HP9 tag=S832H17HJJC2 type=write size=2347825 2024-03-09T14:36:29.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.394+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~X277QT5PRHCU.0 size=1048576 2024-03-09T14:36:29.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.398+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~X277QT5PRHCU.1 size=235903 2024-03-09T14:36:29.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.398+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=X277QT5PRHCU tag=XKVEBK3O1USB type=write size=2333055 2024-03-09T14:36:29.403 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.398+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=9I6N10TIWRKI type=del 2024-03-09T14:36:29.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.406+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~YFPRKQ24PAL9.0 size=1048576 2024-03-09T14:36:29.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.410+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~YFPRKQ24PAL9.1 size=1048576 2024-03-09T14:36:29.415 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.410+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~YFPRKQ24PAL9.2 size=360479 2024-03-09T14:36:29.415 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.410+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=YFPRKQ24PAL9 tag=JZN22Y8EE7PZ type=write size=3506207 2024-03-09T14:36:29.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.418+0000 7f73315818c0 1 < suggested remove operation key=8O5OP8C5NX12 tag=2EEAO309W2WC type=del 2024-03-09T14:36:29.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.422+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=QU8AIMQD310Q type=del size=3679573 2024-03-09T14:36:29.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.426+0000 7f73315818c0 1 < completed write operation key=MD7FXGYXB823 tag=QS3BVN1PJHBJ type=write size=524516 2024-03-09T14:36:29.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.430+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~TMBZGX5RDDVN.0 size=860574 2024-03-09T14:36:29.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.430+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=TMBZGX5RDDVN tag=PHYHVABXV1WN type=write size=1909150 2024-03-09T14:36:29.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.438+0000 7f73315818c0 1 < suggested remove operation key=Y8EQAHMTL66U tag=7UUFFEXB1NTK type=del 2024-03-09T14:36:29.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.446+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~BUJIDM4K8VR2.0 size=1048576 2024-03-09T14:36:29.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.450+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~BUJIDM4K8VR2.1 size=562030 2024-03-09T14:36:29.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.450+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=BUJIDM4K8VR2 tag=4KRDTQN7GM2I type=write size=2659182 2024-03-09T14:36:29.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.454+0000 7f73315818c0 1 < completed part key=_multipart_OU0730HGI176.2~DTQ9P8LT3MFG.0 size=423653 2024-03-09T14:36:29.462 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.458+0000 7f73315818c0 1 < completed multipart upload key=OU0730HGI176 upload=DTQ9P8LT3MFG tag=SGQU4G53U3ZO type=write size=1472229 2024-03-09T14:36:29.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.466+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~S03WM737WET2.0 size=1048576 2024-03-09T14:36:29.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.470+0000 7f73315818c0 1 > prepared part key=_multipart_W6KVP8EQFQ8J.2~S03WM737WET2.1 size=453237 2024-03-09T14:36:29.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.470+0000 7f73315818c0 1 > prepared multipart upload key=W6KVP8EQFQ8J upload=S03WM737WET2 tag=YK26OL8LDI4C type=write size=2550389 2024-03-09T14:36:29.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.470+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~1R2HGNFN8G2Y.0 size=1048576 2024-03-09T14:36:29.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.474+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~1R2HGNFN8G2Y.1 size=442079 2024-03-09T14:36:29.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.482+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=1R2HGNFN8G2Y tag=XPP6222JGKO6 type=write size=2539231 2024-03-09T14:36:29.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.486+0000 7f73315818c0 1 > prepared operation key=FYFVW9R0M1PX tag=S9IJVPDBQQUD type=write size=558213 2024-03-09T14:36:29.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.486+0000 7f73315818c0 1 < completed delete operation key=FYFVW9R0M1PX tag=SJLDH4DSB6UD type=del 2024-03-09T14:36:29.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.494+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~4TC2D0MCEUZZ.0 size=747850 2024-03-09T14:36:29.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.494+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=4TC2D0MCEUZZ tag=ZJDYV0G0XMLA type=write size=1796426 2024-03-09T14:36:29.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.498+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=RDBFLEQRVZ8F type=del 2024-03-09T14:36:29.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.502+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=QT2ZD66PH4JU type=del size=4002166 2024-03-09T14:36:29.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.506+0000 7f73315818c0 1 < completed write operation key=92GWY5B0K4SX tag=77UI0UF8AGAW type=write size=726119 2024-03-09T14:36:29.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.510+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=MJ5DOPSZXJET type=del size=2183782 2024-03-09T14:36:29.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.510+0000 7f73315818c0 1 < completed delete operation key=K3JNN1NCKBXN tag=VPKSAHNMPIRZ type=del 2024-03-09T14:36:29.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.518+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~2NQZO68RCQ4T.0 size=1048576 2024-03-09T14:36:29.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.522+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~2NQZO68RCQ4T.1 size=86881 2024-03-09T14:36:29.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.522+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=2NQZO68RCQ4T tag=42JYP1DMXZY8 type=write size=2184033 2024-03-09T14:36:29.528 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.526+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~YVL2EAVDIQLX.0 size=1048576 2024-03-09T14:36:29.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.526+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~YVL2EAVDIQLX.1 size=809965 2024-03-09T14:36:29.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.534+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=YVL2EAVDIQLX tag=HNMXH42K6EKP type=write size=2907117 2024-03-09T14:36:29.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.538+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=B3QCRK9WQW4L type=del size=2833949 2024-03-09T14:36:29.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.542+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=23SMPS4FDWY9 type=del 2024-03-09T14:36:29.547 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.546+0000 7f73315818c0 1 > prepared operation key=0WGGMHTEF2LB tag=8OH1S1IMAZJO type=del size=1341897 2024-03-09T14:36:29.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.546+0000 7f73315818c0 -1 < failed to complete operation key=Y8EQAHMTL66U tag=Y88AVXCHLS9R type=write size=538039: (22) Invalid argument 2024-03-09T14:36:29.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.550+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=WTVDNV9A2VXQ type=write size=268677 2024-03-09T14:36:29.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.554+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~773XHTCRDGWM.0 size=1048576 2024-03-09T14:36:29.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.558+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~773XHTCRDGWM.1 size=746199 2024-03-09T14:36:29.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.566+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=773XHTCRDGWM tag=Y8V73BFXWHF8 type=write size=2843351 2024-03-09T14:36:29.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.570+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=E07B384SOLAK type=del size=3859755 2024-03-09T14:36:29.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.574+0000 7f73315818c0 1 < suggested remove operation key=PXFZLDSAR1EG tag=KLI0K5LRJ87G type=del 2024-03-09T14:36:29.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.582+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~MAVFBW4JR7X2.0 size=1048576 2024-03-09T14:36:29.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.586+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~MAVFBW4JR7X2.1 size=171945 2024-03-09T14:36:29.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.586+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=MAVFBW4JR7X2 tag=XBHLD6K9S4JE type=write size=2269097 2024-03-09T14:36:29.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.590+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~X277QT5PRHCU.0 size=1048576 2024-03-09T14:36:29.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.590+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~X277QT5PRHCU.1 size=235903 2024-03-09T14:36:29.600 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.598+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=X277QT5PRHCU tag=XKVEBK3O1USB type=write size=2333055 2024-03-09T14:36:29.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.602+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=08H2YDY3CDG4 type=del size=2784444 2024-03-09T14:36:29.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.606+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~YFPRKQ24PAL9.0 size=1048576 2024-03-09T14:36:29.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.606+0000 7f73315818c0 1 < canceled part key=_multipart_PAJYGD2GXHDT.2~YFPRKQ24PAL9.1 size=1048576 2024-03-09T14:36:29.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.606+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~YFPRKQ24PAL9.2 size=360479 2024-03-09T14:36:29.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.618+0000 7f73315818c0 1 < canceled multipart upload key=PAJYGD2GXHDT upload=YFPRKQ24PAL9 tag=JZN22Y8EE7PZ type=write size=3506207 2024-03-09T14:36:29.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.622+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=3T1V0YZ17LRK type=del size=1857708 2024-03-09T14:36:29.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.626+0000 7f73315818c0 1 < completed delete operation key=3ADG4GWVARC1 tag=QU8AIMQD310Q type=del 2024-03-09T14:36:29.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.630+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~4A594KO4JF3E.0 size=1048576 2024-03-09T14:36:29.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.634+0000 7f73315818c0 1 > prepared part key=_multipart_92GWY5B0K4SX.2~4A594KO4JF3E.1 size=361079 2024-03-09T14:36:29.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.634+0000 7f73315818c0 1 > prepared multipart upload key=92GWY5B0K4SX upload=4A594KO4JF3E tag=4GYKXKC5YN7L type=write size=2458231 2024-03-09T14:36:29.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.638+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~TMBZGX5RDDVN.0 size=860574 2024-03-09T14:36:29.647 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.642+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=TMBZGX5RDDVN tag=PHYHVABXV1WN type=write size=1909150 2024-03-09T14:36:29.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.646+0000 7f73315818c0 1 > prepared operation key=MD7FXGYXB823 tag=6ONCYOURTWXN type=del size=3835001 2024-03-09T14:36:29.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.650+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~BUJIDM4K8VR2.0 size=1048576 2024-03-09T14:36:29.658 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.654+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~BUJIDM4K8VR2.1 size=562030 2024-03-09T14:36:29.663 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.658+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=BUJIDM4K8VR2 tag=4KRDTQN7GM2I type=write size=2659182 2024-03-09T14:36:29.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.666+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~VDB2PK8G6AA0.0 size=1048576 2024-03-09T14:36:29.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.670+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~VDB2PK8G6AA0.1 size=1048576 2024-03-09T14:36:29.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.674+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~VDB2PK8G6AA0.2 size=322565 2024-03-09T14:36:29.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.674+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=VDB2PK8G6AA0 tag=2Q9N2PNVVV35 type=write size=3468293 2024-03-09T14:36:29.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.678+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~S03WM737WET2.0 size=1048576 2024-03-09T14:36:29.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.678+0000 7f73315818c0 1 < completed part key=_multipart_W6KVP8EQFQ8J.2~S03WM737WET2.1 size=453237 2024-03-09T14:36:29.689 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.686+0000 7f73315818c0 1 < completed multipart upload key=W6KVP8EQFQ8J upload=S03WM737WET2 tag=YK26OL8LDI4C type=write size=2550389 2024-03-09T14:36:29.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.694+0000 7f73315818c0 1 > prepared part key=_multipart_0WGGMHTEF2LB.2~RQDZM03G98JI.0 size=898357 2024-03-09T14:36:29.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.694+0000 7f73315818c0 1 > prepared multipart upload key=0WGGMHTEF2LB upload=RQDZM03G98JI tag=K41TX2O0FKGW type=write size=1946933 2024-03-09T14:36:29.699 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.698+0000 7f73315818c0 1 < completed write operation key=FYFVW9R0M1PX tag=S9IJVPDBQQUD type=write size=558213 2024-03-09T14:36:29.703 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.698+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=GBKOZE9FUOPR type=del size=3704102 2024-03-09T14:36:29.707 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.702+0000 7f73315818c0 1 < completed part key=_multipart_QG4O2L526B8E.2~4TC2D0MCEUZZ.0 size=747850 2024-03-09T14:36:29.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.710+0000 7f73315818c0 1 < completed multipart upload key=QG4O2L526B8E upload=4TC2D0MCEUZZ tag=ZJDYV0G0XMLA type=write size=1796426 2024-03-09T14:36:29.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.714+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=ALLNU67EYKF7 type=del size=1098982 2024-03-09T14:36:29.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.722+0000 7f73315818c0 1 < suggested update operation key=IA0AB7UVLFEJ tag=QT2ZD66PH4JU type=del size=4002166 2024-03-09T14:36:29.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.726+0000 7f73315818c0 1 > prepared operation key=162AOMHO2BQL tag=ETDMBMT4515P type=del size=1396357 2024-03-09T14:36:29.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.726+0000 7f73315818c0 -1 < failed to complete operation key=PXFZLDSAR1EG tag=MJ5DOPSZXJET type=del size=2183782: (22) Invalid argument 2024-03-09T14:36:29.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.730+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=XPSXIR34MZXX type=del size=3188447 2024-03-09T14:36:29.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.734+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~2NQZO68RCQ4T.0 size=1048576 2024-03-09T14:36:29.741 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.738+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~2NQZO68RCQ4T.1 size=86881 2024-03-09T14:36:29.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.746+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=2NQZO68RCQ4T tag=42JYP1DMXZY8 type=write size=2184033 2024-03-09T14:36:29.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.750+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=DSF9ZXPCC0K0 type=del size=2222361 2024-03-09T14:36:29.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.754+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=B3QCRK9WQW4L type=del 2024-03-09T14:36:29.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.758+0000 7f73315818c0 1 > prepared part key=_multipart_IA0AB7UVLFEJ.2~9DLW6NQT3KWB.0 size=189473 2024-03-09T14:36:29.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.758+0000 7f73315818c0 1 > prepared multipart upload key=IA0AB7UVLFEJ upload=9DLW6NQT3KWB tag=66J1O0KG0JW7 type=write size=1238049 2024-03-09T14:36:29.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.766+0000 7f73315818c0 1 < suggested remove operation key=0WGGMHTEF2LB tag=8OH1S1IMAZJO type=del 2024-03-09T14:36:29.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.770+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=GCWRWBXI11HE type=del size=1908763 2024-03-09T14:36:29.778 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.774+0000 7f73315818c0 -1 < failed to complete operation key=PXFZLDSAR1EG tag=WTVDNV9A2VXQ type=write size=268677: (22) Invalid argument 2024-03-09T14:36:29.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.778+0000 7f73315818c0 1 > prepared operation key=OU0730HGI176 tag=Q0HW4IUVSA3P type=del size=2833392 2024-03-09T14:36:29.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.782+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=E07B384SOLAK type=del 2024-03-09T14:36:29.795 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.790+0000 7f73315818c0 1 > prepared part key=_multipart_S302JFYEDEL7.2~IHIAA7EP6CCT.0 size=382946 2024-03-09T14:36:29.795 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.790+0000 7f73315818c0 1 > prepared multipart upload key=S302JFYEDEL7 upload=IHIAA7EP6CCT tag=4BTHYKJ1OAL6 type=write size=1431522 2024-03-09T14:36:29.796 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.794+0000 7f73315818c0 1 < canceled part key=_multipart_IA0AB7UVLFEJ.2~MAVFBW4JR7X2.0 size=1048576 2024-03-09T14:36:29.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.794+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~MAVFBW4JR7X2.1 size=171945 2024-03-09T14:36:29.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.802+0000 7f73315818c0 1 < canceled multipart upload key=IA0AB7UVLFEJ upload=MAVFBW4JR7X2 tag=XBHLD6K9S4JE type=write size=2269097 2024-03-09T14:36:29.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.806+0000 7f73315818c0 1 > prepared operation key=P9LI5ESR3N9G tag=RFO0366SYVSA type=del size=1617202 2024-03-09T14:36:29.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.810+0000 7f73315818c0 1 < completed delete operation key=PXFZLDSAR1EG tag=08H2YDY3CDG4 type=del 2024-03-09T14:36:29.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.814+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=WOVXNHYLM66K type=del size=1489170 2024-03-09T14:36:29.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.818+0000 7f73315818c0 1 < completed delete operation key=1NYNL8M5DVER tag=3T1V0YZ17LRK type=del 2024-03-09T14:36:29.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.826+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~PPMP11GAQF84.0 size=1048576 2024-03-09T14:36:29.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.826+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~PPMP11GAQF84.1 size=1048576 2024-03-09T14:36:29.833 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.830+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~PPMP11GAQF84.2 size=213179 2024-03-09T14:36:29.834 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.830+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=PPMP11GAQF84 tag=AGVKH3PVL5NS type=write size=3358907 2024-03-09T14:36:29.837 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.834+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~4A594KO4JF3E.0 size=1048576 2024-03-09T14:36:29.841 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.838+0000 7f73315818c0 1 < completed part key=_multipart_92GWY5B0K4SX.2~4A594KO4JF3E.1 size=361079 2024-03-09T14:36:29.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.842+0000 7f73315818c0 1 < completed multipart upload key=92GWY5B0K4SX upload=4A594KO4JF3E tag=4GYKXKC5YN7L type=write size=2458231 2024-03-09T14:36:29.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.850+0000 7f73315818c0 1 > prepared operation key=IA0AB7UVLFEJ tag=877TCSB3ONPF type=del size=753377 2024-03-09T14:36:29.855 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.854+0000 7f73315818c0 1 < completed delete operation key=MD7FXGYXB823 tag=6ONCYOURTWXN type=del 2024-03-09T14:36:29.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.858+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~PUP3TX53ER5U.0 size=1048576 2024-03-09T14:36:29.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.862+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~PUP3TX53ER5U.1 size=1048576 2024-03-09T14:36:29.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.866+0000 7f73315818c0 1 > prepared part key=_multipart_4P89277EEYJ3.2~PUP3TX53ER5U.2 size=759758 2024-03-09T14:36:29.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.866+0000 7f73315818c0 1 > prepared multipart upload key=4P89277EEYJ3 upload=PUP3TX53ER5U tag=HUENU54CNUYT type=write size=3905486 2024-03-09T14:36:29.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.870+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~VDB2PK8G6AA0.0 size=1048576 2024-03-09T14:36:29.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.874+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~VDB2PK8G6AA0.1 size=1048576 2024-03-09T14:36:29.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.874+0000 7f73315818c0 1 < canceled part key=_multipart_PAJYGD2GXHDT.2~VDB2PK8G6AA0.2 size=322565 2024-03-09T14:36:29.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.878+0000 7f73315818c0 1 < canceled multipart upload key=PAJYGD2GXHDT upload=VDB2PK8G6AA0 tag=2Q9N2PNVVV35 type=write size=3468293 2024-03-09T14:36:29.890 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.886+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~ZTA0YTT659GJ.0 size=1048576 2024-03-09T14:36:29.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.890+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~ZTA0YTT659GJ.1 size=1048576 2024-03-09T14:36:29.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.894+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~ZTA0YTT659GJ.2 size=688931 2024-03-09T14:36:29.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.894+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=ZTA0YTT659GJ tag=L9IS5JG73SK4 type=write size=3834659 2024-03-09T14:36:29.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.898+0000 7f73315818c0 1 < completed part key=_multipart_0WGGMHTEF2LB.2~RQDZM03G98JI.0 size=898357 2024-03-09T14:36:29.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.902+0000 7f73315818c0 1 < completed multipart upload key=0WGGMHTEF2LB upload=RQDZM03G98JI tag=K41TX2O0FKGW type=write size=1946933 2024-03-09T14:36:29.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.906+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=GQ4ZEDII65CF type=del size=881135 2024-03-09T14:36:29.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.910+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=GBKOZE9FUOPR type=del 2024-03-09T14:36:29.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.914+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=96D91Y802HUP type=del size=2844939 2024-03-09T14:36:29.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.914+0000 7f73315818c0 1 < completed delete operation key=W6KVP8EQFQ8J tag=ALLNU67EYKF7 type=del 2024-03-09T14:36:29.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.922+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~JU0SXX7P26F7.0 size=1048576 2024-03-09T14:36:29.928 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.926+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~JU0SXX7P26F7.1 size=625790 2024-03-09T14:36:29.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.926+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=JU0SXX7P26F7 tag=1HL6PCGWOZTV type=write size=2722942 2024-03-09T14:36:29.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.930+0000 7f73315818c0 1 < completed delete operation key=162AOMHO2BQL tag=ETDMBMT4515P type=del 2024-03-09T14:36:29.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.934+0000 7f73315818c0 1 > prepared operation key=0IB00CZ92VV4 tag=AK8O94305RFH type=del size=788637 2024-03-09T14:36:29.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.938+0000 7f73315818c0 1 < completed delete operation key=QG4O2L526B8E tag=XPSXIR34MZXX type=del 2024-03-09T14:36:29.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.942+0000 7f73315818c0 1 > prepared part key=_multipart_162AOMHO2BQL.2~T5XFY7I298P7.0 size=522032 2024-03-09T14:36:29.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.942+0000 7f73315818c0 1 > prepared multipart upload key=162AOMHO2BQL upload=T5XFY7I298P7 tag=5CFPZSO1V3A0 type=write size=1570608 2024-03-09T14:36:29.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.950+0000 7f73315818c0 1 < suggested remove operation key=IA0AB7UVLFEJ tag=DSF9ZXPCC0K0 type=del 2024-03-09T14:36:29.958 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.954+0000 7f73315818c0 1 > prepared operation key=PXFZLDSAR1EG tag=QOB7OU4ZX3F7 type=write size=886125 2024-03-09T14:36:29.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.958+0000 7f73315818c0 1 < completed part key=_multipart_IA0AB7UVLFEJ.2~9DLW6NQT3KWB.0 size=189473 2024-03-09T14:36:29.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.966+0000 7f73315818c0 1 < completed multipart upload key=IA0AB7UVLFEJ upload=9DLW6NQT3KWB tag=66J1O0KG0JW7 type=write size=1238049 2024-03-09T14:36:29.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.974+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~OHJYOEHP1JOO.0 size=1048576 2024-03-09T14:36:29.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.974+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~OHJYOEHP1JOO.1 size=471728 2024-03-09T14:36:29.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.974+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=OHJYOEHP1JOO tag=LKK44EJC9ANS type=write size=2568880 2024-03-09T14:36:29.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.978+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=GCWRWBXI11HE type=del 2024-03-09T14:36:29.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.986+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~51X4H7JTDNA4.0 size=1048576 2024-03-09T14:36:29.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.990+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~51X4H7JTDNA4.1 size=1048576 2024-03-09T14:36:29.995 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.990+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~51X4H7JTDNA4.2 size=652179 2024-03-09T14:36:29.995 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.990+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=51X4H7JTDNA4 tag=5UZX40UCILR4 type=write size=3797907 2024-03-09T14:36:29.998 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.994+0000 7f73315818c0 1 < completed delete operation key=OU0730HGI176 tag=Q0HW4IUVSA3P type=del 2024-03-09T14:36:30.001 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:29.998+0000 7f73315818c0 1 > prepared operation key=XELG1AWATMVL tag=EXD0WM04888R type=del size=3486340 2024-03-09T14:36:30.005 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.002+0000 7f73315818c0 1 < completed part key=_multipart_S302JFYEDEL7.2~IHIAA7EP6CCT.0 size=382946 2024-03-09T14:36:30.016 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.006+0000 7f73315818c0 1 < completed multipart upload key=S302JFYEDEL7 upload=IHIAA7EP6CCT tag=4BTHYKJ1OAL6 type=write size=1431522 2024-03-09T14:36:30.016 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.010+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=7PB0EVPOCWW9 type=del size=978951 2024-03-09T14:36:30.017 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.014+0000 7f73315818c0 1 < canceled operation key=P9LI5ESR3N9G tag=RFO0366SYVSA type=del size=1617202 2024-03-09T14:36:30.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.018+0000 7f73315818c0 1 > prepared operation key=M8F9TA80VME6 tag=OGK1WP7MN1EO type=del size=4165830 2024-03-09T14:36:30.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.022+0000 7f73315818c0 1 < suggested remove operation key=K3JNN1NCKBXN tag=WOVXNHYLM66K type=del 2024-03-09T14:36:30.031 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.026+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=RTJTWHX2J79A type=del size=1987398 2024-03-09T14:36:30.035 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.030+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~PPMP11GAQF84.0 size=1048576 2024-03-09T14:36:30.038 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.034+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~PPMP11GAQF84.1 size=1048576 2024-03-09T14:36:30.041 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.038+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~PPMP11GAQF84.2 size=213179 2024-03-09T14:36:30.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.046+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=PPMP11GAQF84 tag=AGVKH3PVL5NS type=write size=3358907 2024-03-09T14:36:30.055 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.054+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~ZO7OXS01M7PU.0 size=1048576 2024-03-09T14:36:30.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.054+0000 7f73315818c0 1 > prepared part key=_multipart_8O5OP8C5NX12.2~ZO7OXS01M7PU.1 size=665309 2024-03-09T14:36:30.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.054+0000 7f73315818c0 1 > prepared multipart upload key=8O5OP8C5NX12 upload=ZO7OXS01M7PU tag=XS8Q7HIE3T33 type=write size=2762461 2024-03-09T14:36:30.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.058+0000 7f73315818c0 -1 < failed to complete operation key=IA0AB7UVLFEJ tag=877TCSB3ONPF type=del size=753377: (22) Invalid argument 2024-03-09T14:36:30.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.066+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~2R7X0EC24JPS.0 size=1048576 2024-03-09T14:36:30.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.070+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~2R7X0EC24JPS.1 size=1048576 2024-03-09T14:36:30.075 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.070+0000 7f73315818c0 1 > prepared part key=_multipart_P8R5RJI8LGZ9.2~2R7X0EC24JPS.2 size=925653 2024-03-09T14:36:30.075 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.070+0000 7f73315818c0 1 > prepared multipart upload key=P8R5RJI8LGZ9 upload=2R7X0EC24JPS tag=WD0O0BWIXDEU type=write size=4071381 2024-03-09T14:36:30.079 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.074+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~PUP3TX53ER5U.0 size=1048576 2024-03-09T14:36:30.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.078+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~PUP3TX53ER5U.1 size=1048576 2024-03-09T14:36:30.085 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.082+0000 7f73315818c0 1 < completed part key=_multipart_4P89277EEYJ3.2~PUP3TX53ER5U.2 size=759758 2024-03-09T14:36:30.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.090+0000 7f73315818c0 1 < completed multipart upload key=4P89277EEYJ3 upload=PUP3TX53ER5U tag=HUENU54CNUYT type=write size=3905486 2024-03-09T14:36:30.095 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.090+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=KYFLUUR2BFGC type=del size=1711265 2024-03-09T14:36:30.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.094+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~ZTA0YTT659GJ.0 size=1048576 2024-03-09T14:36:30.101 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.098+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~ZTA0YTT659GJ.1 size=1048576 2024-03-09T14:36:30.101 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.098+0000 7f73315818c0 1 < canceled part key=_multipart_9B0RAHZZGLYU.2~ZTA0YTT659GJ.2 size=688931 2024-03-09T14:36:30.110 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.102+0000 7f73315818c0 1 < canceled multipart upload key=9B0RAHZZGLYU upload=ZTA0YTT659GJ tag=L9IS5JG73SK4 type=write size=3834659 2024-03-09T14:36:30.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.110+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~O8B3SQOTPOP8.0 size=1048576 2024-03-09T14:36:30.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.114+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~O8B3SQOTPOP8.1 size=178907 2024-03-09T14:36:30.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.114+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=O8B3SQOTPOP8 tag=QNADZOEGVMSN type=write size=2276059 2024-03-09T14:36:30.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.114+0000 7f73315818c0 1 < completed delete operation key=92GWY5B0K4SX tag=GQ4ZEDII65CF type=del 2024-03-09T14:36:30.125 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.122+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~19BFEHLSBBT2.0 size=1048576 2024-03-09T14:36:30.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.122+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~19BFEHLSBBT2.1 size=1048576 2024-03-09T14:36:30.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.126+0000 7f73315818c0 1 > prepared part key=_multipart_PAJYGD2GXHDT.2~19BFEHLSBBT2.2 size=286340 2024-03-09T14:36:30.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.126+0000 7f73315818c0 1 > prepared multipart upload key=PAJYGD2GXHDT upload=19BFEHLSBBT2 tag=0O9A0SCA81OC type=write size=3432068 2024-03-09T14:36:30.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.130+0000 7f73315818c0 1 < completed delete operation key=UMSNM53HTY1J tag=96D91Y802HUP type=del 2024-03-09T14:36:30.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.134+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=Y8X0RRX420DX type=del size=240742 2024-03-09T14:36:30.141 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.138+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~JU0SXX7P26F7.0 size=1048576 2024-03-09T14:36:30.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.142+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~JU0SXX7P26F7.1 size=625790 2024-03-09T14:36:30.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.146+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=JU0SXX7P26F7 tag=1HL6PCGWOZTV type=write size=2722942 2024-03-09T14:36:30.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.154+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~C0EM56AOOGBX.0 size=563524 2024-03-09T14:36:30.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.154+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=C0EM56AOOGBX tag=UG65YROVAHGB type=write size=1612100 2024-03-09T14:36:30.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.162+0000 7f73315818c0 1 < suggested remove operation key=0IB00CZ92VV4 tag=AK8O94305RFH type=del 2024-03-09T14:36:30.174 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.170+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~XI22PEL2UONO.0 size=95427 2024-03-09T14:36:30.174 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.170+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=XI22PEL2UONO tag=JHQY2Y8IAWI3 type=write size=1144003 2024-03-09T14:36:30.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.174+0000 7f73315818c0 1 < completed part key=_multipart_162AOMHO2BQL.2~T5XFY7I298P7.0 size=522032 2024-03-09T14:36:30.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.182+0000 7f73315818c0 1 < completed multipart upload key=162AOMHO2BQL upload=T5XFY7I298P7 tag=5CFPZSO1V3A0 type=write size=1570608 2024-03-09T14:36:30.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.186+0000 7f73315818c0 1 > prepared operation key=Y8EQAHMTL66U tag=QPGI82720FJ8 type=del size=1343061 2024-03-09T14:36:30.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.190+0000 7f73315818c0 1 < suggested remove operation key=PXFZLDSAR1EG tag=QOB7OU4ZX3F7 type=write 2024-03-09T14:36:30.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.198+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~NM46J2BEAL54.0 size=1048576 2024-03-09T14:36:30.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.202+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~NM46J2BEAL54.1 size=1048576 2024-03-09T14:36:30.207 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.202+0000 7f73315818c0 1 > prepared part key=_multipart_K3JNN1NCKBXN.2~NM46J2BEAL54.2 size=662717 2024-03-09T14:36:30.207 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.202+0000 7f73315818c0 1 > prepared multipart upload key=K3JNN1NCKBXN upload=NM46J2BEAL54 tag=WW6PVBFIB7Q1 type=write size=3808445 2024-03-09T14:36:30.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.206+0000 7f73315818c0 1 < canceled part key=_multipart_Y8EQAHMTL66U.2~OHJYOEHP1JOO.0 size=1048576 2024-03-09T14:36:30.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.206+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~OHJYOEHP1JOO.1 size=471728 2024-03-09T14:36:30.216 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.214+0000 7f73315818c0 1 < canceled multipart upload key=Y8EQAHMTL66U upload=OHJYOEHP1JOO tag=LKK44EJC9ANS type=write size=2568880 2024-03-09T14:36:30.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.218+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~FFKO6U7AO2RY.0 size=1048576 2024-03-09T14:36:30.224 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.222+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~FFKO6U7AO2RY.1 size=1048576 2024-03-09T14:36:30.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.222+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~FFKO6U7AO2RY.2 size=413272 2024-03-09T14:36:30.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.222+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=FFKO6U7AO2RY tag=FYXR6OQL5ALO type=write size=3559000 2024-03-09T14:36:30.230 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.226+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~51X4H7JTDNA4.0 size=1048576 2024-03-09T14:36:30.232 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.230+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~51X4H7JTDNA4.1 size=1048576 2024-03-09T14:36:30.236 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.234+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~51X4H7JTDNA4.2 size=652179 2024-03-09T14:36:30.242 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.238+0000 7f73315818c0 1 < completed multipart upload key=9B0RAHZZGLYU upload=51X4H7JTDNA4 tag=5UZX40UCILR4 type=write size=3797907 2024-03-09T14:36:30.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.246+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~OK4SAYMZJT27.0 size=1048576 2024-03-09T14:36:30.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.246+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~OK4SAYMZJT27.1 size=1048576 2024-03-09T14:36:30.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.250+0000 7f73315818c0 1 > prepared part key=_multipart_Y8EQAHMTL66U.2~OK4SAYMZJT27.2 size=23507 2024-03-09T14:36:30.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.250+0000 7f73315818c0 1 > prepared multipart upload key=Y8EQAHMTL66U upload=OK4SAYMZJT27 tag=0AWR46YTTUMM type=write size=3169235 2024-03-09T14:36:30.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.254+0000 7f73315818c0 1 < completed delete operation key=XELG1AWATMVL tag=EXD0WM04888R type=del 2024-03-09T14:36:30.264 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.262+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~SKVNGHXDQ478.0 size=993219 2024-03-09T14:36:30.264 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.262+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=SKVNGHXDQ478 tag=UD017MNTR83R type=write size=2041795 2024-03-09T14:36:30.267 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.262+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=7PB0EVPOCWW9 type=del 2024-03-09T14:36:30.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.270+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~5U7W1V6DETD7.0 size=506600 2024-03-09T14:36:30.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.270+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=5U7W1V6DETD7 tag=8RZS38KCSIXC type=write size=1555176 2024-03-09T14:36:30.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.274+0000 7f73315818c0 1 < completed delete operation key=M8F9TA80VME6 tag=OGK1WP7MN1EO type=del 2024-03-09T14:36:30.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.278+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=JPY58NUI35JR type=write size=509553 2024-03-09T14:36:30.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.282+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=RTJTWHX2J79A type=del 2024-03-09T14:36:30.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.290+0000 7f73315818c0 1 > prepared part key=_multipart_0LP8KKU53X0A.2~AOL2JXG9CBAA.0 size=350982 2024-03-09T14:36:30.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.290+0000 7f73315818c0 1 > prepared multipart upload key=0LP8KKU53X0A upload=AOL2JXG9CBAA tag=5IW1V0M914ME type=write size=1399558 2024-03-09T14:36:30.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.294+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~ZO7OXS01M7PU.0 size=1048576 2024-03-09T14:36:30.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.298+0000 7f73315818c0 1 < completed part key=_multipart_8O5OP8C5NX12.2~ZO7OXS01M7PU.1 size=665309 2024-03-09T14:36:30.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.302+0000 7f73315818c0 1 < completed multipart upload key=8O5OP8C5NX12 upload=ZO7OXS01M7PU tag=XS8Q7HIE3T33 type=write size=2762461 2024-03-09T14:36:30.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.306+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=Z61632XJEM41 type=del size=1678734 2024-03-09T14:36:30.314 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.310+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~2R7X0EC24JPS.0 size=1048576 2024-03-09T14:36:30.316 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.314+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~2R7X0EC24JPS.1 size=1048576 2024-03-09T14:36:30.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.314+0000 7f73315818c0 1 < completed part key=_multipart_P8R5RJI8LGZ9.2~2R7X0EC24JPS.2 size=925653 2024-03-09T14:36:30.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.322+0000 7f73315818c0 1 < completed multipart upload key=P8R5RJI8LGZ9 upload=2R7X0EC24JPS tag=WD0O0BWIXDEU type=write size=4071381 2024-03-09T14:36:30.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.326+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=5Y6TOIE84XYD type=del size=35408 2024-03-09T14:36:30.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.330+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=KYFLUUR2BFGC type=del 2024-03-09T14:36:30.341 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.338+0000 7f73315818c0 1 > prepared part key=_multipart_4E3QMBHEAEHO.2~PL20P1KISM0P.0 size=922450 2024-03-09T14:36:30.341 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.338+0000 7f73315818c0 1 > prepared multipart upload key=4E3QMBHEAEHO upload=PL20P1KISM0P tag=HBS1HMO3QY1J type=write size=1971026 2024-03-09T14:36:30.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.342+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~O8B3SQOTPOP8.0 size=1048576 2024-03-09T14:36:30.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.342+0000 7f73315818c0 1 < completed part key=_multipart_0IB00CZ92VV4.2~O8B3SQOTPOP8.1 size=178907 2024-03-09T14:36:30.352 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.350+0000 7f73315818c0 1 < completed multipart upload key=0IB00CZ92VV4 upload=O8B3SQOTPOP8 tag=QNADZOEGVMSN type=write size=2276059 2024-03-09T14:36:30.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.354+0000 7f73315818c0 1 > prepared operation key=92GWY5B0K4SX tag=HG2KLI97TYZT type=del size=2027495 2024-03-09T14:36:30.359 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.354+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~19BFEHLSBBT2.0 size=1048576 2024-03-09T14:36:30.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.358+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~19BFEHLSBBT2.1 size=1048576 2024-03-09T14:36:30.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.362+0000 7f73315818c0 1 < completed part key=_multipart_PAJYGD2GXHDT.2~19BFEHLSBBT2.2 size=286340 2024-03-09T14:36:30.370 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.366+0000 7f73315818c0 1 < completed multipart upload key=PAJYGD2GXHDT upload=19BFEHLSBBT2 tag=0O9A0SCA81OC type=write size=3432068 2024-03-09T14:36:30.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.370+0000 7f73315818c0 1 > prepared operation key=3ADG4GWVARC1 tag=PARO9FBAJOGJ type=del size=2504498 2024-03-09T14:36:30.381 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.378+0000 7f73315818c0 1 < suggested update operation key=UMSNM53HTY1J tag=Y8X0RRX420DX type=del size=240742 2024-03-09T14:36:30.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.386+0000 7f73315818c0 1 > prepared part key=_multipart_QG4O2L526B8E.2~XTWMYP0W0KTA.0 size=695965 2024-03-09T14:36:30.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.386+0000 7f73315818c0 1 > prepared multipart upload key=QG4O2L526B8E upload=XTWMYP0W0KTA tag=4HE4NLTQ82AS type=write size=1744541 2024-03-09T14:36:30.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.390+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~C0EM56AOOGBX.0 size=563524 2024-03-09T14:36:30.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.394+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=C0EM56AOOGBX tag=UG65YROVAHGB type=write size=1612100 2024-03-09T14:36:30.400 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.398+0000 7f73315818c0 1 > prepared operation key=DQOY1JJ1YQ2W tag=R1WSE6TOQN0Y type=write size=427025 2024-03-09T14:36:30.403 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.398+0000 7f73315818c0 1 < completed part key=_multipart_MSMD3ANK4T7J.2~XI22PEL2UONO.0 size=95427 2024-03-09T14:36:30.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.406+0000 7f73315818c0 1 < completed multipart upload key=MSMD3ANK4T7J upload=XI22PEL2UONO tag=JHQY2Y8IAWI3 type=write size=1144003 2024-03-09T14:36:30.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.406+0000 7f73315818c0 1 > prepared operation key=UMSNM53HTY1J tag=725MY88VZLSU type=del size=3912832 2024-03-09T14:36:30.414 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.410+0000 7f73315818c0 1 < completed delete operation key=Y8EQAHMTL66U tag=QPGI82720FJ8 type=del 2024-03-09T14:36:30.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.418+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~QOAYVS2IP6DH.0 size=1048576 2024-03-09T14:36:30.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.418+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~QOAYVS2IP6DH.1 size=1048576 2024-03-09T14:36:30.424 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.422+0000 7f73315818c0 1 > prepared part key=_multipart_P9LI5ESR3N9G.2~QOAYVS2IP6DH.2 size=953643 2024-03-09T14:36:30.424 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.422+0000 7f73315818c0 1 > prepared multipart upload key=P9LI5ESR3N9G upload=QOAYVS2IP6DH tag=0GSSWOR7WC74 type=write size=4099371 2024-03-09T14:36:30.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.426+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~NM46J2BEAL54.0 size=1048576 2024-03-09T14:36:30.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.426+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~NM46J2BEAL54.1 size=1048576 2024-03-09T14:36:30.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.430+0000 7f73315818c0 1 < completed part key=_multipart_K3JNN1NCKBXN.2~NM46J2BEAL54.2 size=662717 2024-03-09T14:36:30.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.434+0000 7f73315818c0 1 < completed multipart upload key=K3JNN1NCKBXN upload=NM46J2BEAL54 tag=WW6PVBFIB7Q1 type=write size=3808445 2024-03-09T14:36:30.440 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.438+0000 7f73315818c0 1 > prepared operation key=P8R5RJI8LGZ9 tag=G32PO5SQ557X type=write size=147630 2024-03-09T14:36:30.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.442+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~FFKO6U7AO2RY.0 size=1048576 2024-03-09T14:36:30.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.442+0000 7f73315818c0 1 < canceled part key=_multipart_9B0RAHZZGLYU.2~FFKO6U7AO2RY.1 size=1048576 2024-03-09T14:36:30.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.442+0000 7f73315818c0 1 < completed part key=_multipart_9B0RAHZZGLYU.2~FFKO6U7AO2RY.2 size=413272 2024-03-09T14:36:30.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.446+0000 7f73315818c0 1 < canceled multipart upload key=9B0RAHZZGLYU upload=FFKO6U7AO2RY tag=FYXR6OQL5ALO type=write size=3559000 2024-03-09T14:36:30.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.450+0000 7f73315818c0 1 > prepared operation key=1NYNL8M5DVER tag=FBLJ959BXR3O type=del size=3870237 2024-03-09T14:36:30.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.454+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~OK4SAYMZJT27.0 size=1048576 2024-03-09T14:36:30.460 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.458+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~OK4SAYMZJT27.1 size=1048576 2024-03-09T14:36:30.462 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.458+0000 7f73315818c0 1 < completed part key=_multipart_Y8EQAHMTL66U.2~OK4SAYMZJT27.2 size=23507 2024-03-09T14:36:30.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.462+0000 7f73315818c0 1 < completed multipart upload key=Y8EQAHMTL66U upload=OK4SAYMZJT27 tag=0AWR46YTTUMM type=write size=3169235 2024-03-09T14:36:30.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.470+0000 7f73315818c0 1 > prepared part key=_multipart_CVAWRZMAX997.2~0KT3U82VZ90Y.0 size=741692 2024-03-09T14:36:30.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.470+0000 7f73315818c0 1 > prepared multipart upload key=CVAWRZMAX997 upload=0KT3U82VZ90Y tag=UCPY9XX96NVH type=write size=1790268 2024-03-09T14:36:30.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.470+0000 7f73315818c0 1 < completed part key=_multipart_UMSNM53HTY1J.2~SKVNGHXDQ478.0 size=993219 2024-03-09T14:36:30.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.478+0000 7f73315818c0 1 < completed multipart upload key=UMSNM53HTY1J upload=SKVNGHXDQ478 tag=UD017MNTR83R type=write size=2041795 2024-03-09T14:36:30.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.482+0000 7f73315818c0 1 > prepared part key=_multipart_MD7FXGYXB823.2~N1MN2NFMFLMX.0 size=824964 2024-03-09T14:36:30.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.482+0000 7f73315818c0 1 > prepared multipart upload key=MD7FXGYXB823 upload=N1MN2NFMFLMX tag=A13ORMG8TWLT type=write size=1873540 2024-03-09T14:36:30.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.486+0000 7f73315818c0 1 < completed part key=_multipart_CVAWRZMAX997.2~5U7W1V6DETD7.0 size=506600 2024-03-09T14:36:30.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.490+0000 7f73315818c0 1 < completed multipart upload key=CVAWRZMAX997 upload=5U7W1V6DETD7 tag=8RZS38KCSIXC type=write size=1555176 2024-03-09T14:36:30.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.494+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~2LMTK8E1MFE4.0 size=1048576 2024-03-09T14:36:30.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.498+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~2LMTK8E1MFE4.1 size=1048576 2024-03-09T14:36:30.503 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.498+0000 7f73315818c0 1 > prepared part key=_multipart_1NYNL8M5DVER.2~2LMTK8E1MFE4.2 size=781593 2024-03-09T14:36:30.503 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.498+0000 7f73315818c0 1 > prepared multipart upload key=1NYNL8M5DVER upload=2LMTK8E1MFE4 tag=FITOW8AFIZGQ type=write size=3927321 2024-03-09T14:36:30.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.502+0000 7f73315818c0 1 < completed write operation key=K3JNN1NCKBXN tag=JPY58NUI35JR type=write size=509553 2024-03-09T14:36:30.509 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.506+0000 7f73315818c0 1 > prepared operation key=PAJYGD2GXHDT tag=O5EV8ZWXTW4J type=del size=3009904 2024-03-09T14:36:30.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.510+0000 7f73315818c0 1 < completed part key=_multipart_0LP8KKU53X0A.2~AOL2JXG9CBAA.0 size=350982 2024-03-09T14:36:30.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.514+0000 7f73315818c0 1 < completed multipart upload key=0LP8KKU53X0A upload=AOL2JXG9CBAA tag=5IW1V0M914ME type=write size=1399558 2024-03-09T14:36:30.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.522+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~R4VY5ZXJRP2C.0 size=1048576 2024-03-09T14:36:30.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.522+0000 7f73315818c0 1 > prepared part key=_multipart_DQOY1JJ1YQ2W.2~R4VY5ZXJRP2C.1 size=742172 2024-03-09T14:36:30.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.522+0000 7f73315818c0 1 > prepared multipart upload key=DQOY1JJ1YQ2W upload=R4VY5ZXJRP2C tag=QD0SM8WAA039 type=write size=2839324 2024-03-09T14:36:30.534 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.530+0000 7f73315818c0 1 < suggested remove operation key=UMSNM53HTY1J tag=Z61632XJEM41 type=del 2024-03-09T14:36:30.539 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.538+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~L5N70D637OU9.0 size=1048576 2024-03-09T14:36:30.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.538+0000 7f73315818c0 1 > prepared part key=_multipart_OU0730HGI176.2~L5N70D637OU9.1 size=131096 2024-03-09T14:36:30.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.538+0000 7f73315818c0 1 > prepared multipart upload key=OU0730HGI176 upload=L5N70D637OU9 tag=MRG57LBG7Y7R type=write size=2228248 2024-03-09T14:36:30.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.542+0000 7f73315818c0 1 < completed delete operation key=PAJYGD2GXHDT tag=5Y6TOIE84XYD type=del 2024-03-09T14:36:30.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.546+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~D2AVW4LZQ6KN.0 size=1048576 2024-03-09T14:36:30.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.550+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~D2AVW4LZQ6KN.1 size=774046 2024-03-09T14:36:30.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.550+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=D2AVW4LZQ6KN tag=2GI3Z2SZEB8P type=write size=2871198 2024-03-09T14:36:30.556 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.554+0000 7f73315818c0 1 < completed part key=_multipart_4E3QMBHEAEHO.2~PL20P1KISM0P.0 size=922450 2024-03-09T14:36:30.561 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.558+0000 7f73315818c0 1 < completed multipart upload key=4E3QMBHEAEHO upload=PL20P1KISM0P tag=HBS1HMO3QY1J type=write size=1971026 2024-03-09T14:36:30.565 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.562+0000 7f73315818c0 1 > prepared operation key=QG4O2L526B8E tag=74CP0ZT3CLGV type=del size=905722 2024-03-09T14:36:30.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.570+0000 7f73315818c0 1 < suggested remove operation key=92GWY5B0K4SX tag=HG2KLI97TYZT type=del 2024-03-09T14:36:30.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.574+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~T8TRYOKC9U8K.0 size=1048576 2024-03-09T14:36:30.581 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.578+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~T8TRYOKC9U8K.1 size=1048576 2024-03-09T14:36:30.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.582+0000 7f73315818c0 1 > prepared part key=_multipart_MSMD3ANK4T7J.2~T8TRYOKC9U8K.2 size=625468 2024-03-09T14:36:30.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.582+0000 7f73315818c0 1 > prepared multipart upload key=MSMD3ANK4T7J upload=T8TRYOKC9U8K tag=917TNKQNX9D1 type=write size=3771196 2024-03-09T14:36:30.593 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.590+0000 7f73315818c0 1 < canceled operation key=3ADG4GWVARC1 tag=PARO9FBAJOGJ type=del size=2504498 2024-03-09T14:36:30.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.594+0000 7f73315818c0 1 > prepared operation key=K3JNN1NCKBXN tag=385LDOSVJ3OZ type=del size=272802 2024-03-09T14:36:30.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.594+0000 7f73315818c0 1 < canceled part key=_multipart_QG4O2L526B8E.2~XTWMYP0W0KTA.0 size=695965 2024-03-09T14:36:30.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.602+0000 7f73315818c0 1 < canceled multipart upload key=QG4O2L526B8E upload=XTWMYP0W0KTA tag=4HE4NLTQ82AS type=write size=1744541 2024-03-09T14:36:30.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.606+0000 7f73315818c0 1 > prepared part key=_multipart_0IB00CZ92VV4.2~TZMOAD2125BN.0 size=300895 2024-03-09T14:36:30.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.606+0000 7f73315818c0 1 > prepared multipart upload key=0IB00CZ92VV4 upload=TZMOAD2125BN tag=I2RJSFEU45H5 type=write size=1349471 2024-03-09T14:36:30.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.610+0000 7f73315818c0 1 < completed write operation key=DQOY1JJ1YQ2W tag=R1WSE6TOQN0Y type=write size=427025 2024-03-09T14:36:30.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.618+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~4AWVULEM6EDA.0 size=1048576 2024-03-09T14:36:30.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.618+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~4AWVULEM6EDA.1 size=1048576 2024-03-09T14:36:30.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.622+0000 7f73315818c0 1 > prepared part key=_multipart_UMSNM53HTY1J.2~4AWVULEM6EDA.2 size=800523 2024-03-09T14:36:30.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.622+0000 7f73315818c0 1 > prepared multipart upload key=UMSNM53HTY1J upload=4AWVULEM6EDA tag=ZJJCD2WCMLB2 type=write size=3946251 2024-03-09T14:36:30.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.622+0000 7f73315818c0 -1 < no bi entry to suggest for operation key=UMSNM53HTY1J tag=725MY88VZLSU type=del size=3912832: (2) No such file or directory 2024-03-09T14:36:30.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.626+0000 7f73315818c0 1 > prepared operation key=W6KVP8EQFQ8J tag=AJI78EV3QF7B type=del size=3538540 2024-03-09T14:36:30.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.630+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~QOAYVS2IP6DH.0 size=1048576 2024-03-09T14:36:30.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.630+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~QOAYVS2IP6DH.1 size=1048576 2024-03-09T14:36:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.634+0000 7f73315818c0 1 < completed part key=_multipart_P9LI5ESR3N9G.2~QOAYVS2IP6DH.2 size=953643 2024-03-09T14:36:30.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.638+0000 7f73315818c0 1 < completed multipart upload key=P9LI5ESR3N9G upload=QOAYVS2IP6DH tag=0GSSWOR7WC74 type=write size=4099371 2024-03-09T14:36:30.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.646+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~3E9UXLW4QINQ.0 size=1048576 2024-03-09T14:36:30.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.646+0000 7f73315818c0 1 > prepared part key=_multipart_9B0RAHZZGLYU.2~3E9UXLW4QINQ.1 size=710760 2024-03-09T14:36:30.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:36:30.646+0000 7f73315818c0 1 > prepared multipart upload key=9B0RAHZZGLYU upload=3E9UXLW4QINQ tag=L54JC506MCEX type=write size=2807912 2024-03-09T14:36:30.652 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] cls_rgw_stats.simulate (26081 ms) 2024-03-09T14:36:30.652 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 1 test from cls_rgw_stats (26081 ms total) 2024-03-09T14:36:30.652 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:36:30.652 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:36:31.626 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 1 test from 1 test suite ran. (28942 ms total) 2024-03-09T14:36:31.627 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 1 test. 2024-03-09T14:36:31.637 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:36:31.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:36:31.693 INFO:tasks.workunit:Running workunits matching cls/test_cls_cmpomap.sh on client.0... 2024-03-09T14:36:31.694 INFO:tasks.workunit:Running workunit cls/test_cls_cmpomap.sh... 2024-03-09T14:36:31.695 DEBUG:teuthology.orchestra.run.smithi001:workunit test cls/test_cls_cmpomap.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_cmpomap.sh 2024-03-09T14:36:31.758 INFO:tasks.workunit.client.0.smithi001.stdout:Running main() from gmock_main.cc 2024-03-09T14:36:31.758 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 31 tests from 1 test suite. 2024-03-09T14:36:31.758 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:36:33.661 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 31 tests from CmpOmap 2024-03-09T14:36:33.661 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_str 2024-03-09T14:36:33.677 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_vals_noexist_str (16 ms) 2024-03-09T14:36:33.677 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_str_default 2024-03-09T14:36:33.695 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_vals_noexist_str_default (18 ms) 2024-03-09T14:36:33.695 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_u64 2024-03-09T14:36:33.701 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_vals_noexist_u64 (7 ms) 2024-03-09T14:36:33.701 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_u64_default 2024-03-09T14:36:33.707 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_vals_noexist_u64_default (5 ms) 2024-03-09T14:36:33.707 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_vals_str 2024-03-09T14:36:33.731 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_vals_str (24 ms) 2024-03-09T14:36:33.731 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_vals_u64 2024-03-09T14:36:33.742 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_vals_u64 (11 ms) 2024-03-09T14:36:33.742 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_input 2024-03-09T14:36:33.750 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_input (8 ms) 2024-03-09T14:36:33.750 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_default 2024-03-09T14:36:33.760 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_default (10 ms) 2024-03-09T14:36:33.760 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_vals_u64_empty_default 2024-03-09T14:36:33.770 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_vals_u64_empty_default (10 ms) 2024-03-09T14:36:33.770 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_value 2024-03-09T14:36:33.782 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_value (12 ms) 2024-03-09T14:36:33.782 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_vals_at_max_keys 2024-03-09T14:36:33.784 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_vals_at_max_keys (2 ms) 2024-03-09T14:36:33.784 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_vals_over_max_keys 2024-03-09T14:36:33.786 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_vals_over_max_keys (2 ms) 2024-03-09T14:36:33.786 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_str 2024-03-09T14:36:33.802 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_str (17 ms) 2024-03-09T14:36:33.802 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_str_default 2024-03-09T14:36:33.822 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_str_default (19 ms) 2024-03-09T14:36:33.822 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_u64 2024-03-09T14:36:33.840 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_u64 (18 ms) 2024-03-09T14:36:33.840 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_u64_default 2024-03-09T14:36:33.857 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_u64_default (17 ms) 2024-03-09T14:36:33.857 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_set_vals_str 2024-03-09T14:36:33.882 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_set_vals_str (26 ms) 2024-03-09T14:36:33.883 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64 2024-03-09T14:36:33.904 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_set_vals_u64 (22 ms) 2024-03-09T14:36:33.905 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64_einval 2024-03-09T14:36:33.912 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_set_vals_u64_einval (7 ms) 2024-03-09T14:36:33.912 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64_eio 2024-03-09T14:36:33.918 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_set_vals_u64_eio (7 ms) 2024-03-09T14:36:33.918 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_set_vals_at_max_keys 2024-03-09T14:36:33.921 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_set_vals_at_max_keys (3 ms) 2024-03-09T14:36:33.921 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_set_vals_over_max_keys 2024-03-09T14:36:33.923 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_set_vals_over_max_keys (2 ms) 2024-03-09T14:36:33.924 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_rm_keys_noexist_str 2024-03-09T14:36:33.941 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_rm_keys_noexist_str (17 ms) 2024-03-09T14:36:33.941 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_rm_keys_noexist_u64 2024-03-09T14:36:33.959 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_rm_keys_noexist_u64 (18 ms) 2024-03-09T14:36:33.959 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_rm_keys_str 2024-03-09T14:36:33.979 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_rm_keys_str (20 ms) 2024-03-09T14:36:33.979 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64 2024-03-09T14:36:34.000 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64 (22 ms) 2024-03-09T14:36:34.001 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_einval 2024-03-09T14:36:34.006 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_einval (6 ms) 2024-03-09T14:36:34.006 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_eio 2024-03-09T14:36:34.013 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_eio (6 ms) 2024-03-09T14:36:34.013 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_rm_keys_at_max_keys 2024-03-09T14:36:34.016 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_rm_keys_at_max_keys (3 ms) 2024-03-09T14:36:34.016 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_rm_keys_over_max_keys 2024-03-09T14:36:34.018 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_rm_keys_over_max_keys (3 ms) 2024-03-09T14:36:34.018 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_empty 2024-03-09T14:36:34.041 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_empty (22 ms) 2024-03-09T14:36:34.041 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 31 tests from CmpOmap (380 ms total) 2024-03-09T14:36:34.041 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:36:34.041 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:36:34.659 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 31 tests from 1 test suite ran. (2901 ms total) 2024-03-09T14:36:34.659 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 31 tests. 2024-03-09T14:36:34.663 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:36:34.663 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:36:34.720 INFO:tasks.workunit:Running workunits matching cls/test_cls_2pc_queue.sh on client.0... 2024-03-09T14:36:34.722 INFO:tasks.workunit:Running workunit cls/test_cls_2pc_queue.sh... 2024-03-09T14:36:34.722 DEBUG:teuthology.orchestra.run.smithi001:workunit test cls/test_cls_2pc_queue.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2024-03-09T14:36:34.796 INFO:tasks.workunit.client.0.smithi001.stdout:Running main() from gmock_main.cc 2024-03-09T14:36:34.796 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 24 tests from 1 test suite. 2024-03-09T14:36:34.796 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:36:34.796 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 24 tests from TestCls2PCQueue 2024-03-09T14:36:34.796 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.GetCapacity 2024-03-09T14:36:37.670 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.GetCapacity (2876 ms) 2024-03-09T14:36:37.670 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.AsyncGetCapacity 2024-03-09T14:36:40.719 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.AsyncGetCapacity (3048 ms) 2024-03-09T14:36:40.720 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.Reserve 2024-03-09T14:36:43.754 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.Reserve (3036 ms) 2024-03-09T14:36:43.754 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.AsyncReserve 2024-03-09T14:36:46.814 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.AsyncReserve (3060 ms) 2024-03-09T14:36:46.815 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.Commit 2024-03-09T14:36:50.854 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.Commit (4041 ms) 2024-03-09T14:36:50.855 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.Stats 2024-03-09T14:36:54.896 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.Stats (4041 ms) 2024-03-09T14:36:54.896 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.UpgradeFromReef 2024-03-09T14:36:58.951 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.UpgradeFromReef (4056 ms) 2024-03-09T14:36:58.952 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.Abort 2024-03-09T14:37:01.954 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.Abort (3003 ms) 2024-03-09T14:37:01.955 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.ReserveError 2024-03-09T14:37:05.010 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.ReserveError (3054 ms) 2024-03-09T14:37:05.011 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.CommitError 2024-03-09T14:37:08.081 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.CommitError (3072 ms) 2024-03-09T14:37:08.081 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.AbortError 2024-03-09T14:37:11.091 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.AbortError (3010 ms) 2024-03-09T14:37:11.091 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.MultiReserve 2024-03-09T14:37:14.136 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.MultiReserve (3045 ms) 2024-03-09T14:37:14.136 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.MultiCommit 2024-03-09T14:37:17.152 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.MultiCommit (3016 ms) 2024-03-09T14:37:17.152 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.MultiAbort 2024-03-09T14:37:20.207 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.MultiAbort (3052 ms) 2024-03-09T14:37:20.208 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.ReserveCommit 2024-03-09T14:37:23.230 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.ReserveCommit (3026 ms) 2024-03-09T14:37:23.230 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.ReserveAbort 2024-03-09T14:37:26.259 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.ReserveAbort (3030 ms) 2024-03-09T14:37:26.260 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.ManualCleanup 2024-03-09T14:37:29.299 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.ManualCleanup (3035 ms) 2024-03-09T14:37:29.300 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.Cleanup 2024-03-09T14:37:32.365 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.Cleanup (3070 ms) 2024-03-09T14:37:32.365 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.MultiProducer 2024-03-09T14:37:45.417 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.MultiProducer (13052 ms) 2024-03-09T14:37:45.417 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.AsyncConsumer 2024-03-09T14:37:50.532 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.AsyncConsumer (5115 ms) 2024-03-09T14:37:50.532 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.MultiProducerConsumer 2024-03-09T14:38:03.603 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.MultiProducerConsumer (13069 ms) 2024-03-09T14:38:03.604 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.ReserveSpillover 2024-03-09T14:38:09.708 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.ReserveSpillover (6107 ms) 2024-03-09T14:38:09.709 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.CommitSpillover 2024-03-09T14:38:20.028 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.CommitSpillover (10319 ms) 2024-03-09T14:38:20.028 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestCls2PCQueue.AbortSpillover 2024-03-09T14:38:30.094 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestCls2PCQueue.AbortSpillover (10066 ms) 2024-03-09T14:38:30.095 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 24 tests from TestCls2PCQueue (115300 ms total) 2024-03-09T14:38:30.095 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:30.095 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:30.095 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 24 tests from 1 test suite ran. (115300 ms total) 2024-03-09T14:38:30.095 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 24 tests. 2024-03-09T14:38:30.099 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:38:30.099 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:38:30.157 INFO:tasks.workunit:Running workunits matching cls/test_cls_user.sh on client.0... 2024-03-09T14:38:30.158 INFO:tasks.workunit:Running workunit cls/test_cls_user.sh... 2024-03-09T14:38:30.158 DEBUG:teuthology.orchestra.run.smithi001:workunit test cls/test_cls_user.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_user.sh 2024-03-09T14:38:30.233 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:30.234 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:30.234 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 4 tests from 1 test suite. 2024-03-09T14:38:30.234 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:32.136 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 4 tests from ClsAccount 2024-03-09T14:38:32.136 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsAccount.add 2024-03-09T14:38:32.158 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsAccount.add (22 ms) 2024-03-09T14:38:32.158 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsAccount.get 2024-03-09T14:38:32.168 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsAccount.get (10 ms) 2024-03-09T14:38:32.168 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsAccount.rm 2024-03-09T14:38:32.240 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsAccount.rm (72 ms) 2024-03-09T14:38:32.240 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] ClsAccount.list 2024-03-09T14:38:32.285 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] ClsAccount.list (45 ms) 2024-03-09T14:38:32.285 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 4 tests from ClsAccount (149 ms total) 2024-03-09T14:38:32.285 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:32.285 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:33.114 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 4 tests from 1 test suite ran. (2881 ms total) 2024-03-09T14:38:33.114 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 4 tests. 2024-03-09T14:38:33.119 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:38:33.120 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:38:33.177 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_gc_log.sh on client.0... 2024-03-09T14:38:33.178 INFO:tasks.workunit:Running workunit rgw/test_rgw_gc_log.sh... 2024-03-09T14:38:33.178 DEBUG:teuthology.orchestra.run.smithi001:workunit test rgw/test_rgw_gc_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_gc_log.sh 2024-03-09T14:38:33.254 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:33.254 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:33.254 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 6 tests from 1 test suite. 2024-03-09T14:38:33.254 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:35.160 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 6 tests from rgw_gc_log 2024-03-09T14:38:35.160 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] rgw_gc_log.init_existing_queue 2024-03-09T14:38:35.181 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] rgw_gc_log.init_existing_queue (21 ms) 2024-03-09T14:38:35.181 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] rgw_gc_log.init_existing_omap 2024-03-09T14:38:35.186 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] rgw_gc_log.init_existing_omap (5 ms) 2024-03-09T14:38:35.186 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] rgw_gc_log.enqueue1_after_init 2024-03-09T14:38:35.193 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] rgw_gc_log.enqueue1_after_init (7 ms) 2024-03-09T14:38:35.193 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] rgw_gc_log.enqueue2_before_init 2024-03-09T14:38:35.195 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] rgw_gc_log.enqueue2_before_init (2 ms) 2024-03-09T14:38:35.196 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] rgw_gc_log.defer1_after_init 2024-03-09T14:38:35.201 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] rgw_gc_log.defer1_after_init (6 ms) 2024-03-09T14:38:35.201 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] rgw_gc_log.defer2_before_init 2024-03-09T14:38:35.204 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] rgw_gc_log.defer2_before_init (3 ms) 2024-03-09T14:38:35.204 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 6 tests from rgw_gc_log (44 ms total) 2024-03-09T14:38:35.204 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:35.204 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:36.158 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 6 tests from 1 test suite ran. (2904 ms total) 2024-03-09T14:38:36.159 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 6 tests. 2024-03-09T14:38:36.165 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:38:36.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:38:36.223 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_obj.sh on client.0... 2024-03-09T14:38:36.224 INFO:tasks.workunit:Running workunit rgw/test_rgw_obj.sh... 2024-03-09T14:38:36.225 DEBUG:teuthology.orchestra.run.smithi001:workunit test rgw/test_rgw_obj.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_obj.sh 2024-03-09T14:38:36.297 INFO:tasks.workunit.client.0.smithi001.stdout:Running main() from gmock_main.cc 2024-03-09T14:38:36.297 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 4 tests from 1 test suite. 2024-03-09T14:38:36.297 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 4 tests from TestRGWObj 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestRGWObj.underscore 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: _obj ns= i= 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: _obj ns= i= 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: _obj ns= i= 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout: "obj1": { 2024-03-09T14:38:36.298 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.299 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.299 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.299 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.299 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.299 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.299 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.299 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.299 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: "obj2": { 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.300 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.301 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.301 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.301 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.301 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.301 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.301 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.301 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.301 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.301 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.301 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.302 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.302 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.302 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.302 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3": { 2024-03-09T14:38:36.302 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.302 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.302 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.302 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.302 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.302 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.302 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.303 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.303 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.303 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.303 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.303 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.303 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.303 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.303 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.303 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.303 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.303 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3-2": { 2024-03-09T14:38:36.303 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.304 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.304 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.304 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.304 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.304 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.304 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.304 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.304 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.304 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.304 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.304 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.304 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.305 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.305 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.305 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.305 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.305 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.305 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj 2024-03-09T14:38:36.305 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj 2024-03-09T14:38:36.305 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: _obj ns=ns i= 2024-03-09T14:38:36.305 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj 2024-03-09T14:38:36.305 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj 2024-03-09T14:38:36.306 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: _obj ns=ns i= 2024-03-09T14:38:36.306 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj 2024-03-09T14:38:36.306 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj 2024-03-09T14:38:36.306 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: _obj ns=ns i= 2024-03-09T14:38:36.306 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.306 INFO:tasks.workunit.client.0.smithi001.stdout: "obj1": { 2024-03-09T14:38:36.306 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.306 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.306 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.307 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.307 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.307 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.307 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.307 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.307 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.307 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: "obj2": { 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.308 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3": { 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.309 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.310 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3-2": { 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.311 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj[v1] 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj[v1] 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: _obj ns= i=v1 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj[v1] 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj[v1] 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: _obj ns= i=v1 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj[v1] 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj[v1] 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: _obj ns= i=v1 2024-03-09T14:38:36.312 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "obj1": { 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.313 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "v1", 2024-03-09T14:38:36.314 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.314 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.314 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.314 INFO:tasks.workunit.client.0.smithi001.stdout: "obj2": { 2024-03-09T14:38:36.314 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.314 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.314 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.314 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.314 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.314 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "v1", 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3": { 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.315 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.316 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.316 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.316 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.316 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.316 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.316 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.316 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.316 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.316 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.316 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.316 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.316 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.317 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.317 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.317 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3-2": { 2024-03-09T14:38:36.317 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.317 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.317 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.317 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.317 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.317 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.317 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.317 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "v1", 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj[v1] 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj[v1] 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: _obj ns=ns i=v1 2024-03-09T14:38:36.318 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj[v1] 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj[v1] 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: _obj ns=ns i=v1 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj[v1] 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: _obj[v1] 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: _obj ns=ns i=v1 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout: "obj1": { 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.319 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "v1", 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: "obj2": { 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.320 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "v1", 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.321 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3": { 2024-03-09T14:38:36.322 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.322 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.322 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.322 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.322 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.322 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.322 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.322 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.322 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.322 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.322 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.322 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3-2": { 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.323 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_obj", 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "v1", 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestRGWObj.underscore (1 ms) 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestRGWObj.no_underscore 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj 2024-03-09T14:38:36.324 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: obj ns= i= 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: obj ns= i= 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: obj ns= i= 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout: "obj1": { 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.325 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.326 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.326 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.326 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.326 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.326 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.326 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.326 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.326 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.326 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.326 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.326 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "obj2": { 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.327 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3": { 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.328 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3-2": { 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.329 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: obj ns=ns i= 2024-03-09T14:38:36.330 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: obj ns=ns i= 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: obj ns=ns i= 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout: "obj1": { 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.331 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: "obj2": { 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.332 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3": { 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.333 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.334 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.334 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.334 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.334 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.334 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.334 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.334 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.334 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.334 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.334 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.334 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.334 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.335 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.335 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.335 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3-2": { 2024-03-09T14:38:36.335 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.335 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.335 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.335 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.335 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.335 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.335 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.335 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.335 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj[v1] 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj[v1] 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: obj ns= i=v1 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj[v1] 2024-03-09T14:38:36.336 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj[v1] 2024-03-09T14:38:36.337 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: obj ns= i=v1 2024-03-09T14:38:36.337 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj[v1] 2024-03-09T14:38:36.337 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj[v1] 2024-03-09T14:38:36.337 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: obj ns= i=v1 2024-03-09T14:38:36.337 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.337 INFO:tasks.workunit.client.0.smithi001.stdout: "obj1": { 2024-03-09T14:38:36.337 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.337 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.337 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.337 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.337 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.338 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.338 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.338 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.338 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.338 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.338 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.338 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.338 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.338 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "v1", 2024-03-09T14:38:36.338 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.338 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.338 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: "obj2": { 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.339 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.340 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.340 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "v1", 2024-03-09T14:38:36.340 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.340 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.340 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.340 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3": { 2024-03-09T14:38:36.340 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.340 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.340 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.340 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.340 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.341 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3-2": { 2024-03-09T14:38:36.342 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.342 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.342 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.342 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.342 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.342 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.342 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.342 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.342 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.343 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.343 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.343 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.343 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.343 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "v1", 2024-03-09T14:38:36.343 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.343 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.343 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.343 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.343 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj[v1] 2024-03-09T14:38:36.344 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj[v1] 2024-03-09T14:38:36.344 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: obj ns=ns i=v1 2024-03-09T14:38:36.344 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj[v1] 2024-03-09T14:38:36.344 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj[v1] 2024-03-09T14:38:36.344 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: obj ns=ns i=v1 2024-03-09T14:38:36.344 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj[v1] 2024-03-09T14:38:36.344 INFO:tasks.workunit.client.0.smithi001.stdout:parsed: obj[v1] 2024-03-09T14:38:36.344 INFO:tasks.workunit.client.0.smithi001.stdout:stripped: obj ns=ns i=v1 2024-03-09T14:38:36.344 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.344 INFO:tasks.workunit.client.0.smithi001.stdout: "obj1": { 2024-03-09T14:38:36.344 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.345 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.345 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.345 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.345 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.345 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.345 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.345 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.345 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.345 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.345 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.346 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.346 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.346 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "v1", 2024-03-09T14:38:36.346 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.346 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.346 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.346 INFO:tasks.workunit.client.0.smithi001.stdout: "obj2": { 2024-03-09T14:38:36.346 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.346 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "v1", 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.347 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3": { 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.348 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: "obj3-2": { 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.349 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "obj", 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "v1", 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestRGWObj.no_underscore (0 ms) 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestRGWObj.obj_to_raw 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.350 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.351 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.352 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": "default-placement.data", 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker_myobj", 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.353 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.354 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.355 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.356 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker_myobj", 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.357 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.358 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.359 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.360 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": "default-placement.data", 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns_myobj", 2024-03-09T14:38:36.361 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.362 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.363 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.363 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.363 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.363 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.363 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.363 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.363 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.363 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.364 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.365 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.365 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.365 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.365 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.365 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.365 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.365 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.365 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.365 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.365 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.365 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.365 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns_myobj", 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.366 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.367 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.368 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.369 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": "default-placement.data", 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__:inst_myobj", 2024-03-09T14:38:36.370 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.371 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.372 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.373 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.373 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.373 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.373 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.373 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.373 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.373 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.373 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.373 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.373 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.373 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.373 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.374 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.374 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.374 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.374 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.374 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.374 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.374 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.374 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.374 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.374 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__:inst_myobj", 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.375 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.376 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.376 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.376 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.376 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.376 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.376 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.376 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.376 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.376 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.376 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.376 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.376 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.377 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.377 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.377 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.377 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.377 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.377 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.377 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.377 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.377 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.377 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.378 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.379 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.379 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.379 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.379 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.379 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.379 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.379 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.379 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.379 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.379 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.379 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": "default-placement.data", 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns:inst_myobj", 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.380 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.381 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.381 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.381 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.381 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.381 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.381 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.381 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.381 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.381 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.381 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.381 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.382 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.382 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.382 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.382 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.382 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.382 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.382 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.382 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.382 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.382 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.382 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.383 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.383 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.383 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.383 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.383 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.383 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.383 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.383 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.383 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.383 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.383 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.383 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.384 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.384 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.384 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.384 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.384 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.384 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.384 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.384 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.384 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.384 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.385 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.385 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.385 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.385 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.385 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.385 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.385 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.385 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.385 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns:inst_myobj", 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.386 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.387 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.388 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.389 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": "default-placement.data", 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker___myobj", 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj" 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.390 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.391 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.392 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.393 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker___myobj", 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj" 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.394 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.395 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.396 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.397 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": "default-placement.data", 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns__myobj", 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.398 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.399 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.400 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.401 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns__myobj", 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.402 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.403 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.404 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.405 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": "default-placement.data", 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__:inst__myobj", 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj" 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.406 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.407 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.408 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.409 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.410 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__:inst__myobj", 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj" 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.411 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.412 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.413 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.413 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.413 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.413 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.413 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.413 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.413 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.413 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.413 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.413 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.413 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.414 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.415 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.415 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.415 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.415 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.415 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.415 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.415 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.415 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.415 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": "default-placement.data", 2024-03-09T14:38:36.415 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns:inst__myobj", 2024-03-09T14:38:36.415 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.415 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.416 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.417 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.418 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.418 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.418 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.418 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.418 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.418 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.418 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.418 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.418 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.418 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.418 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.419 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.420 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.420 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.420 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.420 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.420 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.420 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.420 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.420 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns:inst__myobj", 2024-03-09T14:38:36.420 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.420 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.420 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.420 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.421 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.421 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.421 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.421 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.421 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.421 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.421 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.421 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.421 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.421 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.421 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.421 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.422 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.423 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.423 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.423 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.423 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.423 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.423 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.423 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.423 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.423 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.423 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.424 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.424 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.424 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.424 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.424 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.424 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.424 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.424 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.424 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.424 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.424 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": "default-placement.data", 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker___myobj_", 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj_" 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.425 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.426 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.426 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.426 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.426 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.426 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.426 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.426 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.426 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.426 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.426 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.426 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.427 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.427 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.427 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.427 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.427 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.427 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.427 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.427 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.427 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.428 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.429 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker___myobj_", 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj_" 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.430 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.431 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.432 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.433 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": "default-placement.data", 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns__myobj_", 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.434 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.435 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.436 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.437 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.438 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns__myobj_", 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.439 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.440 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.441 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.442 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": "default-placement.data", 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__:inst__myobj_", 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj_" 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.443 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.444 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.445 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.446 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__:inst__myobj_", 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj_" 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.447 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.448 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.449 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.450 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": "default-placement.data", 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns:inst__myobj_", 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "test", 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.451 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": "", 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": "" 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.452 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.453 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.454 INFO:tasks.workunit.client.0.smithi001.stdout: "obj": { 2024-03-09T14:38:36.454 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.454 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.454 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.454 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.454 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.454 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.454 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.454 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.454 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.454 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.454 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.455 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.455 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.455 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.455 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.455 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.455 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.455 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.455 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.455 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.455 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.455 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns:inst__myobj_", 2024-03-09T14:38:36.456 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.456 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.456 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.456 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.456 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.456 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.456 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.456 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.456 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.456 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.456 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.456 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.457 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.457 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.457 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.457 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.457 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.457 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.457 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.457 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.457 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.457 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestRGWObj.obj_to_raw (0 ms) 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] TestRGWObj.old_to_raw 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker_myobj", 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.458 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.459 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.459 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.459 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.459 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.459 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.459 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.459 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.459 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.459 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.459 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout:raw=.data-pool:marker_myobj 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj2": { 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker_myobj", 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.460 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.461 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.461 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.461 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj2": { 2024-03-09T14:38:36.461 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.461 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.461 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.461 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.461 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.461 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.461 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.461 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.461 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.462 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.462 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.462 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.462 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.462 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.462 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.462 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.462 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.462 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.462 INFO:tasks.workunit.client.0.smithi001.stdout:raw2=.data-pool:marker_myobj 2024-03-09T14:38:36.462 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.463 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.463 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.463 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns_myobj", 2024-03-09T14:38:36.463 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.463 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.463 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.463 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.463 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.463 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.464 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.464 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.464 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.464 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.464 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.464 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.464 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.465 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.465 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.465 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.465 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.465 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.465 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.465 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.465 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.466 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.466 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.466 INFO:tasks.workunit.client.0.smithi001.stdout:raw=.data-pool:marker__ns_myobj 2024-03-09T14:38:36.466 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.466 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj2": { 2024-03-09T14:38:36.466 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.466 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns_myobj", 2024-03-09T14:38:36.466 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.466 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.467 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.467 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.467 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj2": { 2024-03-09T14:38:36.467 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.467 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.467 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.467 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.467 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.468 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout:raw2=.data-pool:marker__ns_myobj 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__:inst_myobj", 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.469 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.470 INFO:tasks.workunit.client.0.smithi001.stdout:raw=.data-pool:marker__:inst_myobj 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj2": { 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__:inst_myobj", 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj2": { 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.471 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.472 INFO:tasks.workunit.client.0.smithi001.stdout:raw2=.data-pool:marker__:inst_myobj 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns:inst_myobj", 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.473 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout:raw=.data-pool:marker__ns:inst_myobj 2024-03-09T14:38:36.474 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj2": { 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns:inst_myobj", 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj2": { 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.475 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "myobj", 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout:raw2=.data-pool:marker__ns:inst_myobj 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.476 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker___myobj", 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj" 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.477 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout:raw=.data-pool:marker___myobj 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj2": { 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.478 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker___myobj", 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj" 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj2": { 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.479 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout:raw2=.data-pool:marker___myobj 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns__myobj", 2024-03-09T14:38:36.480 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.481 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout:raw=.data-pool:marker__ns__myobj 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj2": { 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns__myobj", 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.482 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj2": { 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.483 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout:raw2=.data-pool:marker__ns__myobj 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__:inst__myobj", 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj" 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.484 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.485 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout:raw=.data-pool:marker__:inst__myobj 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj2": { 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__:inst__myobj", 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj" 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.486 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj2": { 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.487 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout:raw2=.data-pool:marker__:inst__myobj 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns:inst__myobj", 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.488 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.489 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout:raw=.data-pool:marker__ns:inst__myobj 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj2": { 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns:inst__myobj", 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj2": { 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.490 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj", 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.491 INFO:tasks.workunit.client.0.smithi001.stdout:raw2=.data-pool:marker__ns:inst__myobj 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker___myobj_", 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj_" 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.492 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout:raw=.data-pool:marker___myobj_ 2024-03-09T14:38:36.493 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj2": { 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker___myobj_", 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj_" 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj2": { 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.494 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.495 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.495 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.495 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.495 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.495 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.495 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.495 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.495 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.495 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout:raw2=.data-pool:marker___myobj_ 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns__myobj_", 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.496 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.497 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.498 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.498 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.498 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.498 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.498 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.498 INFO:tasks.workunit.client.0.smithi001.stdout:raw=.data-pool:marker__ns__myobj_ 2024-03-09T14:38:36.498 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.498 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj2": { 2024-03-09T14:38:36.498 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.499 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns__myobj_", 2024-03-09T14:38:36.499 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.499 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.499 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.499 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.499 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj2": { 2024-03-09T14:38:36.499 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.499 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.499 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.499 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.499 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.499 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.500 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.500 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.500 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.500 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.500 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.500 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.500 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.500 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "", 2024-03-09T14:38:36.500 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.500 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.500 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.501 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.501 INFO:tasks.workunit.client.0.smithi001.stdout:raw2=.data-pool:marker__ns__myobj_ 2024-03-09T14:38:36.501 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.501 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.501 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.501 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__:inst__myobj_", 2024-03-09T14:38:36.501 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj_" 2024-03-09T14:38:36.501 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.501 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.501 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.502 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.502 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.502 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.502 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.502 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.502 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.502 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.502 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.502 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.502 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.502 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.503 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.503 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.503 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.503 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.503 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.503 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.503 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.503 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.503 INFO:tasks.workunit.client.0.smithi001.stdout:raw=.data-pool:marker__:inst__myobj_ 2024-03-09T14:38:36.504 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.504 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj2": { 2024-03-09T14:38:36.504 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.504 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__:inst__myobj_", 2024-03-09T14:38:36.504 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "marker__myobj_" 2024-03-09T14:38:36.504 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.504 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.504 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.504 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj2": { 2024-03-09T14:38:36.504 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.505 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.505 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.505 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.505 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.505 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.505 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.505 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.505 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.505 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.505 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.505 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.506 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.506 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.506 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "" 2024-03-09T14:38:36.506 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.506 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.506 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.506 INFO:tasks.workunit.client.0.smithi001.stdout:raw2=.data-pool:marker__:inst__myobj_ 2024-03-09T14:38:36.506 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.506 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj": { 2024-03-09T14:38:36.507 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.507 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns:inst__myobj_", 2024-03-09T14:38:36.507 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.507 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.507 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.507 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.507 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj": { 2024-03-09T14:38:36.507 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.507 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.507 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.507 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.508 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout:raw=.data-pool:marker__ns:inst__myobj_ 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout: "raw_obj2": { 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout: "pool": ".data-pool", 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout: "oid": "marker__ns:inst__myobj_", 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout: "loc": "" 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout: "new_obj2": { 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket": { 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "ebtest", 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout: "marker": "marker", 2024-03-09T14:38:36.509 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_id": "bucket-id", 2024-03-09T14:38:36.510 INFO:tasks.workunit.client.0.smithi001.stdout: "tenant": "", 2024-03-09T14:38:36.510 INFO:tasks.workunit.client.0.smithi001.stdout: "explicit_placement": { 2024-03-09T14:38:36.510 INFO:tasks.workunit.client.0.smithi001.stdout: "data_pool": ".data-pool", 2024-03-09T14:38:36.510 INFO:tasks.workunit.client.0.smithi001.stdout: "data_extra_pool": "", 2024-03-09T14:38:36.510 INFO:tasks.workunit.client.0.smithi001.stdout: "index_pool": ".index-pool" 2024-03-09T14:38:36.510 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.510 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.510 INFO:tasks.workunit.client.0.smithi001.stdout: "key": { 2024-03-09T14:38:36.510 INFO:tasks.workunit.client.0.smithi001.stdout: "name": "_myobj_", 2024-03-09T14:38:36.510 INFO:tasks.workunit.client.0.smithi001.stdout: "instance": "inst", 2024-03-09T14:38:36.510 INFO:tasks.workunit.client.0.smithi001.stdout: "ns": "ns" 2024-03-09T14:38:36.510 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.511 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.511 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.511 INFO:tasks.workunit.client.0.smithi001.stdout:raw2=.data-pool:marker__ns:inst__myobj_ 2024-03-09T14:38:36.511 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] TestRGWObj.old_to_raw (1 ms) 2024-03-09T14:38:36.511 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 4 tests from TestRGWObj (2 ms total) 2024-03-09T14:38:36.511 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:36.511 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:36.511 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 4 tests from 1 test suite ran. (2 ms total) 2024-03-09T14:38:36.511 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 4 tests. 2024-03-09T14:38:36.511 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:38:36.512 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:38:36.526 INFO:tasks.workunit:Running workunits matching rgw/test_librgw_file.sh on client.0... 2024-03-09T14:38:36.528 INFO:tasks.workunit:Running workunit rgw/test_librgw_file.sh... 2024-03-09T14:38:36.528 DEBUG:teuthology.orchestra.run.smithi001:workunit test rgw/test_librgw_file.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh 2024-03-09T14:38:36.635 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:36.635 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:36.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.693+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.695 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.697+0000 7f5a1368b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:36.695 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.697+0000 7f5a1368b980 20 realm 2024-03-09T14:38:36.695 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.697+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.697+0000 7f5a1368b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:36.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.697+0000 7f5a1368b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:38:36.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.697+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.697+0000 7f5a1368b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:36.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.697+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.697+0000 7f5a1368b980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:38:36.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.697+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.699 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.701+0000 7f5a1368b980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:38:36.699 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.701+0000 7f5a1368b980 20 searching for the correct realm 2024-03-09T14:38:36.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.717+0000 7f5a119a4640 0 -- 172.21.15.1:0/3295470820 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55841f7cb770 msgr2=0x55841f7cdbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:38:36.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.729+0000 7f5a111a3640 0 -- 172.21.15.1:0/3295470820 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f5a0801a420 msgr2=0x7f5a0801a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:38:36.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:38:36.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:38:36.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:38:36.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:38:36.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:38:36.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:38:36.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:36.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:38:36.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:38:36.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 zone default found 2024-03-09T14:38:36.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 4 Realm: () 2024-03-09T14:38:36.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:38:36.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:38:36.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:38:36.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 zonegroup default 2024-03-09T14:38:36.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:36.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.737+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.741+0000 7f5a1368b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:36.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.741+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.741+0000 7f5a1368b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:36.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.741+0000 7f5a1368b980 20 started sync module instance, tier type = 2024-03-09T14:38:36.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.741+0000 7f5a1368b980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:38:36.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.757+0000 7f5a1368b980 20 add_watcher() i=0 2024-03-09T14:38:36.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.757+0000 7f5a1368b980 20 add_watcher() i=1 2024-03-09T14:38:36.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.757+0000 7f5a1368b980 20 add_watcher() i=2 2024-03-09T14:38:36.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.757+0000 7f5a1368b980 20 add_watcher() i=3 2024-03-09T14:38:36.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.757+0000 7f5a1368b980 20 add_watcher() i=4 2024-03-09T14:38:36.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.757+0000 7f5a1368b980 20 add_watcher() i=5 2024-03-09T14:38:36.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.757+0000 7f5a1368b980 20 add_watcher() i=6 2024-03-09T14:38:36.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.757+0000 7f5a1368b980 20 add_watcher() i=7 2024-03-09T14:38:36.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.757+0000 7f5a1368b980 2 all 8 watchers are set, enabling cache 2024-03-09T14:38:36.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f58c77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:38:36.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5a1368b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:38:36.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5a1368b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:38:36.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5a1368b980 5 note: GC not initialized 2024-03-09T14:38:36.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f58c67e4640 20 reqs_thread_entry: start 2024-03-09T14:38:36.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5a1368b980 20 init_complete bucket index max shards: 1 2024-03-09T14:38:36.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5a1368b980 20 Filter name: none 2024-03-09T14:38:36.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5837fff640 20 reqs_thread_entry: start 2024-03-09T14:38:36.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5a1368b980 10 cache get: name=default.rgw.meta+users.uid+ceph-test-librgw-file : miss 2024-03-09T14:38:36.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5a1368b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:36.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5a1368b980 10 cache put: name=default.rgw.meta+users.uid+ceph-test-librgw-file info.flags=0x0 2024-03-09T14:38:36.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5a1368b980 10 adding default.rgw.meta+users.uid+ceph-test-librgw-file to cache LRU end 2024-03-09T14:38:36.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5a1368b980 10 cache get: name=default.rgw.meta+users.email+librgw@example.com : miss 2024-03-09T14:38:36.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.761+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:36.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 10 cache put: name=default.rgw.meta+users.email+librgw@example.com info.flags=0x0 2024-03-09T14:38:36.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 10 adding default.rgw.meta+users.email+librgw@example.com to cache LRU end 2024-03-09T14:38:36.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 10 cache get: name=default.rgw.meta+users.keys+2FcVxeu2moKry6lWqeI6EPre4bA= : miss 2024-03-09T14:38:36.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 20 rados->read ofs=0 len=0 2024-03-09T14:38:36.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:36.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 10 cache put: name=default.rgw.meta+users.keys+2FcVxeu2moKry6lWqeI6EPre4bA= info.flags=0x0 2024-03-09T14:38:36.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 10 adding default.rgw.meta+users.keys+2FcVxeu2moKry6lWqeI6EPre4bA= to cache LRU end 2024-03-09T14:38:36.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 10 cache get: name=default.rgw.meta+users.keys+2FcVxeu2moKry6lWqeI6EPre4bA= : hit (negative entry) 2024-03-09T14:38:36.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 10 cache get: name=default.rgw.meta+users.keys+2FcVxeu2moKry6lWqeI6EPre4bA= : hit (negative entry) 2024-03-09T14:38:36.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 10 cache put: name=default.rgw.meta+users.uid+ceph-test-librgw-file info.flags=0x17 2024-03-09T14:38:36.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 10 moving default.rgw.meta+users.uid+ceph-test-librgw-file to cache LRU end 2024-03-09T14:38:36.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.765+0000 7f5a1368b980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:ceph-test-librgw-file, ofs0, ns] 2024-03-09T14:38:36.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.769+0000 7f58f4ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1060856922113 cookie 94025952317184 notifier 5556 bl.length()=697 2024-03-09T14:38:36.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.769+0000 7f58f4ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ceph-test-librgw-file info.flags=0x17 2024-03-09T14:38:36.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.769+0000 7f58f4ff9640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ceph-test-librgw-file to cache LRU end 2024-03-09T14:38:36.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.773+0000 7f5a1368b980 10 cache put: name=default.rgw.meta+users.email+librgw@example.com info.flags=0x7 2024-03-09T14:38:36.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.773+0000 7f5a1368b980 10 moving default.rgw.meta+users.email+librgw@example.com to cache LRU end 2024-03-09T14:38:36.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.773+0000 7f5a1368b980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:librgw@example.com, ofs0, ns] 2024-03-09T14:38:36.773 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.773+0000 7f58f4ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1060856922112 cookie 94025952362480 notifier 5556 bl.length()=194 2024-03-09T14:38:36.773 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.773+0000 7f58f4ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+librgw@example.com info.flags=0x7 2024-03-09T14:38:36.773 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.773+0000 7f58f4ff9640 10 rgw watcher librados: moving default.rgw.meta+users.email+librgw@example.com to cache LRU end 2024-03-09T14:38:36.779 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.781+0000 7f5a1368b980 10 cache put: name=default.rgw.meta+users.keys+2FcVxeu2moKry6lWqeI6EPre4bA= info.flags=0x7 2024-03-09T14:38:36.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.781+0000 7f5a1368b980 10 moving default.rgw.meta+users.keys+2FcVxeu2moKry6lWqeI6EPre4bA= to cache LRU end 2024-03-09T14:38:36.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.781+0000 7f5a1368b980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.keys:2FcVxeu2moKry6lWqeI6EPre4bA=, ofs0, ns] 2024-03-09T14:38:36.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.781+0000 7f58f4ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1060856922116 cookie 94025952370880 notifier 5556 bl.length()=203 2024-03-09T14:38:36.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.781+0000 7f58f4ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+2FcVxeu2moKry6lWqeI6EPre4bA= info.flags=0x7 2024-03-09T14:38:36.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.781+0000 7f58f4ff9640 10 rgw watcher librados: moving default.rgw.meta+users.keys+2FcVxeu2moKry6lWqeI6EPre4bA= to cache LRU end 2024-03-09T14:38:36.782 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: "user_id": "ceph-test-librgw-file", 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: "display_name": "librgw test user", 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: "email": "librgw@example.com", 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: "suspended": 0, 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: "max_buckets": 1000, 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: "subusers": [], 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: "keys": [ 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: { 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: "user": "ceph-test-librgw-file", 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: "access_key": "2FcVxeu2moKry6lWqeI6EPre4bA=", 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: "secret_key": "ghDi3kHoBX+yPoyQq1YLK9wdLP+ij/ry1YnpNjJPyg277B/VoEEB0w==", 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: "active": true, 2024-03-09T14:38:36.783 INFO:tasks.workunit.client.0.smithi001.stdout: "create_date": "2024-03-09T14:38:36.766895Z" 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: ], 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: "swift_keys": [], 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: "caps": [], 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: "default_placement": "", 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: "default_storage_class": "", 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: "placement_tags": [], 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_quota": { 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: "enabled": false, 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: "check_on_raw": false, 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: "max_size": -1, 2024-03-09T14:38:36.784 INFO:tasks.workunit.client.0.smithi001.stdout: "max_size_kb": 0, 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: "max_objects": -1 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: "user_quota": { 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: "enabled": false, 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: "check_on_raw": false, 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: "max_size": -1, 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: "max_size_kb": 0, 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: "max_objects": -1 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: "temp_url_keys": [], 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: "type": "rgw", 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: "mfa_ids": [], 2024-03-09T14:38:36.785 INFO:tasks.workunit.client.0.smithi001.stdout: "account_id": "", 2024-03-09T14:38:36.786 INFO:tasks.workunit.client.0.smithi001.stdout: "path": "/", 2024-03-09T14:38:36.786 INFO:tasks.workunit.client.0.smithi001.stdout: "create_date": "2024-03-09T14:38:36.766882Z", 2024-03-09T14:38:36.786 INFO:tasks.workunit.client.0.smithi001.stdout: "tags": [], 2024-03-09T14:38:36.786 INFO:tasks.workunit.client.0.smithi001.stdout: "group_ids": [] 2024-03-09T14:38:36.786 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:36.786 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:36.786 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.785+0000 7f58c77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:38:36.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.789+0000 7f5a1368b980 20 remove_watcher() i=0 2024-03-09T14:38:36.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.789+0000 7f5a1368b980 2 removed watcher, disabling cache 2024-03-09T14:38:36.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.789+0000 7f5a1368b980 20 remove_watcher() i=1 2024-03-09T14:38:36.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.793+0000 7f5a1368b980 20 remove_watcher() i=2 2024-03-09T14:38:36.795 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.797+0000 7f5a1368b980 20 remove_watcher() i=3 2024-03-09T14:38:36.797 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.797+0000 7f5a1368b980 20 remove_watcher() i=4 2024-03-09T14:38:36.798 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.797+0000 7f5a1368b980 20 remove_watcher() i=5 2024-03-09T14:38:36.800 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.801+0000 7f5a1368b980 20 remove_watcher() i=6 2024-03-09T14:38:36.801 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:36.801+0000 7f5a1368b980 20 remove_watcher() i=7 2024-03-09T14:38:36.814 INFO:tasks.workunit.client.0.smithi001.stdout:phase 1.1 2024-03-09T14:38:36.833 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 24 tests from 1 test suite. 2024-03-09T14:38:36.833 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:36.833 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 24 tests from LibRGW 2024-03-09T14:38:36.833 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.INIT 2024-03-09T14:38:36.843 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:36.843 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:37.142 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.INIT (311 ms) 2024-03-09T14:38:37.142 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT_NOROOT 2024-03-09T14:38:37.144 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT_NOROOT (2 ms) 2024-03-09T14:38:37.144 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT 2024-03-09T14:38:37.144 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2024-03-09T14:38:37.144 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SETUP_HIER1 2024-03-09T14:38:37.161 INFO:tasks.workunit.client.0.smithi001.stdout:creating: nfsroot:foo/bar/baz/quux 2024-03-09T14:38:37.167 INFO:tasks.workunit.client.0.smithi001.stdout:creating: nfsroot:foo/f1 2024-03-09T14:38:37.170 INFO:tasks.workunit.client.0.smithi001.stdout:creating: nfsroot:foo/f2 2024-03-09T14:38:37.174 INFO:tasks.workunit.client.0.smithi001.stdout:creating: nfsroot:foo/bar/f1 2024-03-09T14:38:37.179 INFO:tasks.workunit.client.0.smithi001.stdout:creating: nfsroot:foo/bar/d1/ 2024-03-09T14:38:37.183 INFO:tasks.workunit.client.0.smithi001.stdout:creating: nfsroot:foo/bar/baz/hungry 2024-03-09T14:38:37.187 INFO:tasks.workunit.client.0.smithi001.stdout:creating: nfsroot:foo/bar/baz/hungry/ 2024-03-09T14:38:37.190 INFO:tasks.workunit.client.0.smithi001.stdout:creating: nfsroot:foo/bar/baz/momma 2024-03-09T14:38:37.194 INFO:tasks.workunit.client.0.smithi001.stdout:creating: nfsroot:foo/bar/baz/bear/ 2024-03-09T14:38:37.198 INFO:tasks.workunit.client.0.smithi001.stdout:creating: nfsroot:foo/bar/baz/sasquatch 2024-03-09T14:38:37.204 INFO:tasks.workunit.client.0.smithi001.stdout:creating: nfsroot:foo/bar/baz/sasquatch/ 2024-03-09T14:38:37.207 INFO:tasks.workunit.client.0.smithi001.stdout:creating: nfsroot:foo/bar/baz/frobozz 2024-03-09T14:38:37.211 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SETUP_HIER1 (67 ms) 2024-03-09T14:38:37.211 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SETUP_DIRS1 2024-03-09T14:38:37.428 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SETUP_DIRS1 (217 ms) 2024-03-09T14:38:37.429 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SETATTR 2024-03-09T14:38:37.429 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SETATTR (0 ms) 2024-03-09T14:38:37.429 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RGW_CREATE_DIRS1 2024-03-09T14:38:37.469 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RGW_CREATE_DIRS1 (41 ms) 2024-03-09T14:38:37.469 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RGW_SETUP_RENAME1 2024-03-09T14:38:37.569 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RGW_SETUP_RENAME1 (99 ms) 2024-03-09T14:38:37.569 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RGW_INTRABUCKET_RENAME1 2024-03-09T14:38:37.569 INFO:tasks.workunit.client.0.smithi001.stdout:rename file rfile_0 to rfile_r0 (bucket brename0) 2024-03-09T14:38:37.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RGW_INTRABUCKET_RENAME1 (16 ms) 2024-03-09T14:38:37.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RGW_CROSSBUCKET_RENAME1 2024-03-09T14:38:37.585 INFO:tasks.workunit.client.0.smithi001.stdout:rename file rfile_1 (bucket brename0) to rfile_rhilldog (bucket brename1) 2024-03-09T14:38:37.606 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RGW_CROSSBUCKET_RENAME1 (22 ms) 2024-03-09T14:38:37.606 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.GETATTR_DIRS1 2024-03-09T14:38:37.606 INFO:tasks.workunit.client.0.smithi001.stdout:scanning objects in /bdirs1/dir_0 2024-03-09T14:38:37.606 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_0/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_0/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_0/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_0/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout:scanning objects in /bdirs1/dir_1 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_1/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_1/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_1/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_1/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout:scanning objects in /bdirs1/dir_2 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_2/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_2/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_2/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_2/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2024-03-09T14:38:37.607 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.GETATTR_DIRS1 (0 ms) 2024-03-09T14:38:37.608 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READ_DIRS1 2024-03-09T14:38:37.608 INFO:tasks.workunit.client.0.smithi001.stdout:read back objects in /bdirs1/dir_0 2024-03-09T14:38:37.608 INFO:tasks.workunit.client.0.smithi001.stdout:reading 0,256 dir_0/sfile_0 2024-03-09T14:38:37.608 INFO:tasks.workunit.client.0.smithi001.stdout: read back from sfile_0 : "data for sfile_0" 2024-03-09T14:38:37.609 INFO:tasks.workunit.client.0.smithi001.stdout:reading 0,256 dir_0/sfile_1 2024-03-09T14:38:37.611 INFO:tasks.workunit.client.0.smithi001.stdout: read back from sfile_1 : "data for sfile_1" 2024-03-09T14:38:37.611 INFO:tasks.workunit.client.0.smithi001.stdout:read back objects in /bdirs1/dir_1 2024-03-09T14:38:37.611 INFO:tasks.workunit.client.0.smithi001.stdout:reading 0,256 dir_1/sfile_0 2024-03-09T14:38:37.613 INFO:tasks.workunit.client.0.smithi001.stdout: read back from sfile_0 : "data for sfile_0" 2024-03-09T14:38:37.614 INFO:tasks.workunit.client.0.smithi001.stdout:reading 0,256 dir_1/sfile_1 2024-03-09T14:38:37.615 INFO:tasks.workunit.client.0.smithi001.stdout: read back from sfile_1 : "data for sfile_1" 2024-03-09T14:38:37.615 INFO:tasks.workunit.client.0.smithi001.stdout:read back objects in /bdirs1/dir_2 2024-03-09T14:38:37.615 INFO:tasks.workunit.client.0.smithi001.stdout:reading 0,256 dir_2/sfile_0 2024-03-09T14:38:37.617 INFO:tasks.workunit.client.0.smithi001.stdout: read back from sfile_0 : "data for sfile_0" 2024-03-09T14:38:37.617 INFO:tasks.workunit.client.0.smithi001.stdout:reading 0,256 dir_2/sfile_1 2024-03-09T14:38:37.619 INFO:tasks.workunit.client.0.smithi001.stdout: read back from sfile_1 : "data for sfile_1" 2024-03-09T14:38:37.619 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READ_DIRS1 (13 ms) 2024-03-09T14:38:37.619 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READF_SETUP1 2024-03-09T14:38:37.840 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READF_SETUP1 (220 ms) 2024-03-09T14:38:37.840 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READF_DIRS1 2024-03-09T14:38:37.840 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READF_DIRS1 (0 ms) 2024-03-09T14:38:37.840 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.WRITEF_DIRS1 2024-03-09T14:38:37.840 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.WRITEF_DIRS1 (0 ms) 2024-03-09T14:38:37.840 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RELEASE_DIRS1 2024-03-09T14:38:37.840 INFO:tasks.workunit.client.0.smithi001.stdout:release sdir_0 type: DIR refs: 2 2024-03-09T14:38:37.840 INFO:tasks.workunit.client.0.smithi001.stdout:release sfile_0 type: FILE refs: 2 2024-03-09T14:38:37.840 INFO:tasks.workunit.client.0.smithi001.stdout:release sdir_1 type: DIR refs: 2 2024-03-09T14:38:37.840 INFO:tasks.workunit.client.0.smithi001.stdout:release sfile_1 type: FILE refs: 2 2024-03-09T14:38:37.840 INFO:tasks.workunit.client.0.smithi001.stdout:release sdir_0 type: DIR refs: 2 2024-03-09T14:38:37.841 INFO:tasks.workunit.client.0.smithi001.stdout:release sfile_0 type: FILE refs: 2 2024-03-09T14:38:37.841 INFO:tasks.workunit.client.0.smithi001.stdout:release sdir_1 type: DIR refs: 2 2024-03-09T14:38:37.841 INFO:tasks.workunit.client.0.smithi001.stdout:release sfile_1 type: FILE refs: 2 2024-03-09T14:38:37.841 INFO:tasks.workunit.client.0.smithi001.stdout:release sdir_0 type: DIR refs: 2 2024-03-09T14:38:37.841 INFO:tasks.workunit.client.0.smithi001.stdout:release sfile_0 type: FILE refs: 2 2024-03-09T14:38:37.841 INFO:tasks.workunit.client.0.smithi001.stdout:release sdir_1 type: DIR refs: 2 2024-03-09T14:38:37.841 INFO:tasks.workunit.client.0.smithi001.stdout:release sfile_1 type: FILE refs: 2 2024-03-09T14:38:37.841 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RELEASE_DIRS1 (0 ms) 2024-03-09T14:38:37.841 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.HIER1 2024-03-09T14:38:37.855 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/f2 (f2): FILE 2024-03-09T14:38:37.856 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/f1 (f1): FILE 2024-03-09T14:38:37.862 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/f1 (f1): FILE 2024-03-09T14:38:37.865 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/d1 (d1): DIR 2024-03-09T14:38:37.871 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2024-03-09T14:38:37.872 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2024-03-09T14:38:37.873 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/quux (quux): FILE 2024-03-09T14:38:37.875 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/momma (momma): FILE 2024-03-09T14:38:37.876 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2024-03-09T14:38:37.877 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2024-03-09T14:38:37.878 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/frobozz (frobozz): FILE 2024-03-09T14:38:37.881 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/bear (bear): DIR 2024-03-09T14:38:37.881 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz (baz): DIR 2024-03-09T14:38:37.881 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar (bar): DIR 2024-03-09T14:38:37.881 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo (foo): DIR 2024-03-09T14:38:37.881 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot (nfsroot): DIR 2024-03-09T14:38:37.881 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.HIER1 (41 ms) 2024-03-09T14:38:37.881 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2024-03-09T14:38:37.881 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (0 ms) 2024-03-09T14:38:37.882 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2024-03-09T14:38:37.882 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2024-03-09T14:38:37.882 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2024-03-09T14:38:37.882 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2024-03-09T14:38:37.882 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2024-03-09T14:38:37.882 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2024-03-09T14:38:37.882 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CLEANUP 2024-03-09T14:38:37.882 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2024-03-09T14:38:37.882 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.UMOUNT 2024-03-09T14:38:37.882 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2024-03-09T14:38:37.883 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SHUTDOWN 2024-03-09T14:38:37.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:37.881+0000 7fec78b49a80 -1 shutting down 2024-03-09T14:38:37.884 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:37.885+0000 7feb546bc640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 55396, si_uid : 1000, si_addr0x3e80000d864, si_status0 2024-03-09T14:38:37.974 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SHUTDOWN (93 ms) 2024-03-09T14:38:37.974 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 24 tests from LibRGW (1143 ms total) 2024-03-09T14:38:37.974 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:37.975 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:37.975 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 24 tests from 1 test suite ran. (1143 ms total) 2024-03-09T14:38:37.975 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 24 tests. 2024-03-09T14:38:37.996 INFO:tasks.workunit.client.0.smithi001.stdout:phase 1.2 2024-03-09T14:38:38.019 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 24 tests from 1 test suite. 2024-03-09T14:38:38.019 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:38.019 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 24 tests from LibRGW 2024-03-09T14:38:38.019 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.INIT 2024-03-09T14:38:38.029 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:38.029 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:38.581 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.INIT (563 ms) 2024-03-09T14:38:38.581 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT_NOROOT 2024-03-09T14:38:38.584 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT_NOROOT (2 ms) 2024-03-09T14:38:38.584 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT 2024-03-09T14:38:38.584 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2024-03-09T14:38:38.584 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SETUP_HIER1 2024-03-09T14:38:38.584 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SETUP_HIER1 (0 ms) 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SETUP_DIRS1 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SETUP_DIRS1 (0 ms) 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SETATTR 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SETATTR (0 ms) 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RGW_CREATE_DIRS1 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RGW_CREATE_DIRS1 (0 ms) 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RGW_SETUP_RENAME1 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RGW_SETUP_RENAME1 (0 ms) 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RGW_INTRABUCKET_RENAME1 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RGW_INTRABUCKET_RENAME1 (0 ms) 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RGW_CROSSBUCKET_RENAME1 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RGW_CROSSBUCKET_RENAME1 (0 ms) 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.GETATTR_DIRS1 2024-03-09T14:38:38.585 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.GETATTR_DIRS1 (0 ms) 2024-03-09T14:38:38.586 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READ_DIRS1 2024-03-09T14:38:38.586 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READ_DIRS1 (0 ms) 2024-03-09T14:38:38.586 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READF_SETUP1 2024-03-09T14:38:38.586 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READF_SETUP1 (0 ms) 2024-03-09T14:38:38.586 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READF_DIRS1 2024-03-09T14:38:38.586 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READF_DIRS1 (0 ms) 2024-03-09T14:38:38.586 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.WRITEF_DIRS1 2024-03-09T14:38:38.586 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.WRITEF_DIRS1 (0 ms) 2024-03-09T14:38:38.586 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RELEASE_DIRS1 2024-03-09T14:38:38.586 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RELEASE_DIRS1 (0 ms) 2024-03-09T14:38:38.586 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.HIER1 2024-03-09T14:38:38.586 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.HIER1 (0 ms) 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (0 ms) 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CLEANUP 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.UMOUNT 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2024-03-09T14:38:38.587 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SHUTDOWN 2024-03-09T14:38:38.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:38.585+0000 7f33ff98ea80 -1 shutting down 2024-03-09T14:38:38.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:38.585+0000 7f32db501640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 56039, si_uid : 1000, si_addr0x3e80000dae7, si_status0 2024-03-09T14:38:38.731 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SHUTDOWN (148 ms) 2024-03-09T14:38:38.732 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 24 tests from LibRGW (713 ms total) 2024-03-09T14:38:38.732 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:38.732 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:38.732 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 24 tests from 1 test suite ran. (713 ms total) 2024-03-09T14:38:38.732 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 24 tests. 2024-03-09T14:38:38.746 INFO:tasks.workunit.client.0.smithi001.stdout:phase 1.3 2024-03-09T14:38:38.769 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 24 tests from 1 test suite. 2024-03-09T14:38:38.769 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:38.769 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 24 tests from LibRGW 2024-03-09T14:38:38.769 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.INIT 2024-03-09T14:38:38.781 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:38.781 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:39.059 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.INIT (291 ms) 2024-03-09T14:38:39.059 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT_NOROOT 2024-03-09T14:38:39.062 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT_NOROOT (2 ms) 2024-03-09T14:38:39.062 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT 2024-03-09T14:38:39.062 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2024-03-09T14:38:39.062 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SETUP_HIER1 2024-03-09T14:38:39.070 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SETUP_HIER1 (8 ms) 2024-03-09T14:38:39.070 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SETUP_DIRS1 2024-03-09T14:38:39.196 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SETUP_DIRS1 (127 ms) 2024-03-09T14:38:39.196 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SETATTR 2024-03-09T14:38:39.196 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SETATTR (0 ms) 2024-03-09T14:38:39.196 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RGW_CREATE_DIRS1 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RGW_CREATE_DIRS1 (0 ms) 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RGW_SETUP_RENAME1 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RGW_SETUP_RENAME1 (0 ms) 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RGW_INTRABUCKET_RENAME1 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RGW_INTRABUCKET_RENAME1 (0 ms) 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RGW_CROSSBUCKET_RENAME1 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RGW_CROSSBUCKET_RENAME1 (0 ms) 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.GETATTR_DIRS1 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout:scanning objects in /bdirs1/dir_0 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_0/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_0/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_0/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_0/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2024-03-09T14:38:39.197 INFO:tasks.workunit.client.0.smithi001.stdout:scanning objects in /bdirs1/dir_1 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_1/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_1/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_1/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_1/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout:scanning objects in /bdirs1/dir_2 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_2/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_2/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_2/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout: /bdirs1/dir_2/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.GETATTR_DIRS1 (0 ms) 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READ_DIRS1 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout:read back objects in /bdirs1/dir_0 2024-03-09T14:38:39.198 INFO:tasks.workunit.client.0.smithi001.stdout:reading 0,256 dir_0/sfile_0 2024-03-09T14:38:39.199 INFO:tasks.workunit.client.0.smithi001.stdout: read back from sfile_0 : "data for sfile_0" 2024-03-09T14:38:39.199 INFO:tasks.workunit.client.0.smithi001.stdout:reading 0,256 dir_0/sfile_1 2024-03-09T14:38:39.201 INFO:tasks.workunit.client.0.smithi001.stdout: read back from sfile_1 : "data for sfile_1" 2024-03-09T14:38:39.201 INFO:tasks.workunit.client.0.smithi001.stdout:read back objects in /bdirs1/dir_1 2024-03-09T14:38:39.201 INFO:tasks.workunit.client.0.smithi001.stdout:reading 0,256 dir_1/sfile_0 2024-03-09T14:38:39.203 INFO:tasks.workunit.client.0.smithi001.stdout: read back from sfile_0 : "data for sfile_0" 2024-03-09T14:38:39.204 INFO:tasks.workunit.client.0.smithi001.stdout:reading 0,256 dir_1/sfile_1 2024-03-09T14:38:39.206 INFO:tasks.workunit.client.0.smithi001.stdout: read back from sfile_1 : "data for sfile_1" 2024-03-09T14:38:39.206 INFO:tasks.workunit.client.0.smithi001.stdout:read back objects in /bdirs1/dir_2 2024-03-09T14:38:39.206 INFO:tasks.workunit.client.0.smithi001.stdout:reading 0,256 dir_2/sfile_0 2024-03-09T14:38:39.209 INFO:tasks.workunit.client.0.smithi001.stdout: read back from sfile_0 : "data for sfile_0" 2024-03-09T14:38:39.209 INFO:tasks.workunit.client.0.smithi001.stdout:reading 0,256 dir_2/sfile_1 2024-03-09T14:38:39.211 INFO:tasks.workunit.client.0.smithi001.stdout: read back from sfile_1 : "data for sfile_1" 2024-03-09T14:38:39.211 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READ_DIRS1 (15 ms) 2024-03-09T14:38:39.211 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READF_SETUP1 2024-03-09T14:38:39.211 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READF_SETUP1 (0 ms) 2024-03-09T14:38:39.211 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READF_DIRS1 2024-03-09T14:38:39.211 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READF_DIRS1 (0 ms) 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.WRITEF_DIRS1 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.WRITEF_DIRS1 (0 ms) 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.RELEASE_DIRS1 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:release sdir_0 type: DIR refs: 2 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:release sfile_0 type: FILE refs: 2 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:release sdir_1 type: DIR refs: 2 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:release sfile_1 type: FILE refs: 2 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:release sdir_0 type: DIR refs: 2 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:release sfile_0 type: FILE refs: 2 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:release sdir_1 type: DIR refs: 2 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:release sfile_1 type: FILE refs: 2 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:release sdir_0 type: DIR refs: 2 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:release sfile_0 type: FILE refs: 2 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:release sdir_1 type: DIR refs: 2 2024-03-09T14:38:39.212 INFO:tasks.workunit.client.0.smithi001.stdout:release sfile_1 type: FILE refs: 2 2024-03-09T14:38:39.213 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.RELEASE_DIRS1 (0 ms) 2024-03-09T14:38:39.213 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.HIER1 2024-03-09T14:38:39.226 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/f2 (f2): FILE 2024-03-09T14:38:39.227 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/f1 (f1): FILE 2024-03-09T14:38:39.232 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/f1 (f1): FILE 2024-03-09T14:38:39.234 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/d1 (d1): DIR 2024-03-09T14:38:39.239 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2024-03-09T14:38:39.241 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2024-03-09T14:38:39.242 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/quux (quux): FILE 2024-03-09T14:38:39.243 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/momma (momma): FILE 2024-03-09T14:38:39.245 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2024-03-09T14:38:39.246 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2024-03-09T14:38:39.247 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/frobozz (frobozz): FILE 2024-03-09T14:38:39.249 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz/bear (bear): DIR 2024-03-09T14:38:39.249 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar/baz (baz): DIR 2024-03-09T14:38:39.249 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo/bar (bar): DIR 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot/foo (foo): DIR 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:/nfsroot (nfsroot): DIR 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.HIER1 (38 ms) 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (0 ms) 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CLEANUP 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2024-03-09T14:38:39.250 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.UMOUNT 2024-03-09T14:38:39.251 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2024-03-09T14:38:39.251 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SHUTDOWN 2024-03-09T14:38:39.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:39.249+0000 7f9a45b67a80 -1 shutting down 2024-03-09T14:38:39.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:39.253+0000 7f99216da640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 56682, si_uid : 1000, si_addr0x3e80000dd6a, si_status0 2024-03-09T14:38:39.428 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SHUTDOWN (179 ms) 2024-03-09T14:38:39.428 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 24 tests from LibRGW (660 ms total) 2024-03-09T14:38:39.428 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:39.429 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:39.429 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 24 tests from 1 test suite ran. (660 ms total) 2024-03-09T14:38:39.429 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 24 tests. 2024-03-09T14:38:39.442 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2.1 2024-03-09T14:38:39.465 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 9 tests from 1 test suite. 2024-03-09T14:38:39.465 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:39.465 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 9 tests from LibRGW 2024-03-09T14:38:39.465 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.INIT 2024-03-09T14:38:39.476 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:39.476 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:39.763 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.INIT (299 ms) 2024-03-09T14:38:39.763 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT 2024-03-09T14:38:39.765 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT (2 ms) 2024-03-09T14:38:39.765 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2024-03-09T14:38:39.782 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CREATE_BUCKET (16 ms) 2024-03-09T14:38:39.782 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2024-03-09T14:38:39.782 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.DELETE_BUCKET (0 ms) 2024-03-09T14:38:39.782 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CREATE_BUCKET_MULTI 2024-03-09T14:38:39.797 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave0 2024-03-09T14:38:39.814 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave1 2024-03-09T14:38:39.830 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave2 2024-03-09T14:38:39.846 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave3 2024-03-09T14:38:39.866 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave4 2024-03-09T14:38:39.885 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave5 2024-03-09T14:38:39.905 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave6 2024-03-09T14:38:39.928 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave7 2024-03-09T14:38:39.947 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave8 2024-03-09T14:38:39.966 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave9 2024-03-09T14:38:39.967 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CREATE_BUCKET_MULTI (185 ms) 2024-03-09T14:38:39.967 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.DELETE_BUCKET_MULTI 2024-03-09T14:38:40.403 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.DELETE_BUCKET_MULTI (437 ms) 2024-03-09T14:38:40.403 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CLEANUP 2024-03-09T14:38:40.403 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2024-03-09T14:38:40.403 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.UMOUNT 2024-03-09T14:38:40.404 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2024-03-09T14:38:40.404 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SHUTDOWN 2024-03-09T14:38:40.404 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:40.405+0000 7ff9aae50a80 -1 shutting down 2024-03-09T14:38:40.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:40.405+0000 7ff8869c3640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 57325, si_uid : 1000, si_addr0x3e80000dfed, si_status0 2024-03-09T14:38:40.520 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SHUTDOWN (116 ms) 2024-03-09T14:38:40.520 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 9 tests from LibRGW (1055 ms total) 2024-03-09T14:38:40.520 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:40.520 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:40.520 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 9 tests from 1 test suite ran. (1055 ms total) 2024-03-09T14:38:40.520 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 9 tests. 2024-03-09T14:38:40.534 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2.2 2024-03-09T14:38:40.558 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 9 tests from 1 test suite. 2024-03-09T14:38:40.558 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:40.558 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 9 tests from LibRGW 2024-03-09T14:38:40.558 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.INIT 2024-03-09T14:38:40.569 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:40.569 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:41.138 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.INIT (580 ms) 2024-03-09T14:38:41.138 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT 2024-03-09T14:38:41.139 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT (2 ms) 2024-03-09T14:38:41.139 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2024-03-09T14:38:41.139 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CREATE_BUCKET (0 ms) 2024-03-09T14:38:41.140 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2024-03-09T14:38:41.172 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.DELETE_BUCKET (33 ms) 2024-03-09T14:38:41.172 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CREATE_BUCKET_MULTI 2024-03-09T14:38:41.187 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave0 2024-03-09T14:38:41.205 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave1 2024-03-09T14:38:41.222 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave2 2024-03-09T14:38:41.240 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave3 2024-03-09T14:38:41.260 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave4 2024-03-09T14:38:41.281 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave5 2024-03-09T14:38:41.305 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave6 2024-03-09T14:38:41.328 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave7 2024-03-09T14:38:41.352 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave8 2024-03-09T14:38:41.374 INFO:tasks.workunit.client.0.smithi001.stdout:created: sorrydave9 2024-03-09T14:38:41.374 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CREATE_BUCKET_MULTI (201 ms) 2024-03-09T14:38:41.374 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.DELETE_BUCKET_MULTI 2024-03-09T14:38:41.876 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.DELETE_BUCKET_MULTI (503 ms) 2024-03-09T14:38:41.877 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CLEANUP 2024-03-09T14:38:41.877 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2024-03-09T14:38:41.877 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.UMOUNT 2024-03-09T14:38:41.877 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2024-03-09T14:38:41.877 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SHUTDOWN 2024-03-09T14:38:41.877 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:41.877+0000 7fe154f01a80 -1 shutting down 2024-03-09T14:38:41.879 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:41.881+0000 7fe030a74640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 57968, si_uid : 1000, si_addr0x3e80000e270, si_status0 2024-03-09T14:38:41.984 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SHUTDOWN (107 ms) 2024-03-09T14:38:41.984 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 9 tests from LibRGW (1426 ms total) 2024-03-09T14:38:41.984 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:41.984 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:41.984 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 9 tests from 1 test suite ran. (1426 ms total) 2024-03-09T14:38:41.984 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 9 tests. 2024-03-09T14:38:41.996 INFO:tasks.workunit.client.0.smithi001.stdout:phase 3.1 2024-03-09T14:38:42.343 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 13 tests from 1 test suite. 2024-03-09T14:38:42.343 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:42.343 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 13 tests from LibRGW 2024-03-09T14:38:42.343 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.INIT 2024-03-09T14:38:42.352 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:42.352 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:42.899 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.INIT (556 ms) 2024-03-09T14:38:42.899 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT 2024-03-09T14:38:42.901 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT (2 ms) 2024-03-09T14:38:42.901 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2024-03-09T14:38:42.918 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CREATE_BUCKET (16 ms) 2024-03-09T14:38:42.918 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2024-03-09T14:38:42.924 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (7 ms) 2024-03-09T14:38:42.924 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2024-03-09T14:38:42.926 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (2 ms) 2024-03-09T14:38:42.926 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LARGE1 2024-03-09T14:38:43.005 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LARGE1 (78 ms) 2024-03-09T14:38:43.005 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LARGE2 2024-03-09T14:38:43.074 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LARGE2 (69 ms) 2024-03-09T14:38:43.074 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.STAT_OBJECT 2024-03-09T14:38:43.074 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.STAT_OBJECT (1 ms) 2024-03-09T14:38:43.074 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2024-03-09T14:38:43.075 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.DELETE_OBJECT (0 ms) 2024-03-09T14:38:43.075 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2024-03-09T14:38:43.075 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.DELETE_BUCKET (0 ms) 2024-03-09T14:38:43.075 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CLEANUP 2024-03-09T14:38:43.075 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2024-03-09T14:38:43.075 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.UMOUNT 2024-03-09T14:38:43.075 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2024-03-09T14:38:43.075 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SHUTDOWN 2024-03-09T14:38:43.075 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:43.073+0000 7efe178bfa80 -1 shutting down 2024-03-09T14:38:43.078 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:43.077+0000 7efcf3432640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 58613, si_uid : 1000, si_addr0x3e80000e4f5, si_status0 2024-03-09T14:38:43.263 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SHUTDOWN (188 ms) 2024-03-09T14:38:43.263 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 13 tests from LibRGW (920 ms total) 2024-03-09T14:38:43.263 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:43.263 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:43.263 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 13 tests from 1 test suite ran. (920 ms total) 2024-03-09T14:38:43.263 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 13 tests. 2024-03-09T14:38:43.276 INFO:tasks.workunit.client.0.smithi001.stdout:phase 3.2 2024-03-09T14:38:43.621 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 13 tests from 1 test suite. 2024-03-09T14:38:43.621 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:43.621 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 13 tests from LibRGW 2024-03-09T14:38:43.621 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.INIT 2024-03-09T14:38:43.629 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:43.630 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:43.928 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.INIT (307 ms) 2024-03-09T14:38:43.928 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT 2024-03-09T14:38:43.930 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT (2 ms) 2024-03-09T14:38:43.930 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2024-03-09T14:38:43.930 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CREATE_BUCKET (0 ms) 2024-03-09T14:38:43.930 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2024-03-09T14:38:43.939 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (10 ms) 2024-03-09T14:38:43.939 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2024-03-09T14:38:43.945 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (6 ms) 2024-03-09T14:38:43.946 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LARGE1 2024-03-09T14:38:44.026 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LARGE1 (80 ms) 2024-03-09T14:38:44.026 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LARGE2 2024-03-09T14:38:44.026 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LARGE2 (0 ms) 2024-03-09T14:38:44.026 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.STAT_OBJECT 2024-03-09T14:38:44.026 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2024-03-09T14:38:44.026 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2024-03-09T14:38:44.036 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.DELETE_OBJECT (11 ms) 2024-03-09T14:38:44.036 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2024-03-09T14:38:44.078 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.DELETE_BUCKET (41 ms) 2024-03-09T14:38:44.078 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CLEANUP 2024-03-09T14:38:44.078 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2024-03-09T14:38:44.078 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.UMOUNT 2024-03-09T14:38:44.078 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2024-03-09T14:38:44.078 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SHUTDOWN 2024-03-09T14:38:44.079 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:44.077+0000 7f89cab00a80 -1 shutting down 2024-03-09T14:38:44.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:44.081+0000 7f88a6673640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 59256, si_uid : 1000, si_addr0x3e80000e778, si_status0 2024-03-09T14:38:44.169 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SHUTDOWN (91 ms) 2024-03-09T14:38:44.170 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 13 tests from LibRGW (549 ms total) 2024-03-09T14:38:44.170 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:44.170 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:44.170 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 13 tests from 1 test suite ran. (549 ms total) 2024-03-09T14:38:44.170 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 13 tests. 2024-03-09T14:38:44.185 INFO:tasks.workunit.client.0.smithi001.stdout:phase 4.1 2024-03-09T14:38:44.207 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 11 tests from 1 test suite. 2024-03-09T14:38:44.207 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:44.207 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 11 tests from LibRGW 2024-03-09T14:38:44.207 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.TVAR 2024-03-09T14:38:44.207 INFO:tasks.workunit.client.0.smithi001.stdout:read i1: 64001 2024-03-09T14:38:44.207 INFO:tasks.workunit.client.0.smithi001.stdout:read s1: blunderbuss 2024-03-09T14:38:44.207 INFO:tasks.workunit.client.0.smithi001.stdout:read s3: [ OK ] LibRGW.TVAR (0 ms) 2024-03-09T14:38:44.207 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.INIT 2024-03-09T14:38:44.220 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:44.221 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:44.790 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.INIT (582 ms) 2024-03-09T14:38:44.790 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT 2024-03-09T14:38:44.791 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT (2 ms) 2024-03-09T14:38:44.791 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2024-03-09T14:38:44.822 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (30 ms) 2024-03-09T14:38:44.822 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2024-03-09T14:38:46.286 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_0 2024-03-09T14:38:46.287 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_1 2024-03-09T14:38:46.287 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_2 2024-03-09T14:38:46.287 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_3 2024-03-09T14:38:46.287 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_4 2024-03-09T14:38:46.287 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_5 2024-03-09T14:38:46.287 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_6 2024-03-09T14:38:46.287 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_7 2024-03-09T14:38:46.287 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_8 2024-03-09T14:38:46.287 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_9 2024-03-09T14:38:46.287 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_10 2024-03-09T14:38:46.287 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_11 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_12 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_13 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_14 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_15 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_16 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_17 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_18 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_19 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_20 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_21 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_22 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_23 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_24 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_25 2024-03-09T14:38:46.288 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_26 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_27 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_28 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_29 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_30 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_31 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_32 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_33 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_34 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_35 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_36 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_37 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_38 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_39 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_40 2024-03-09T14:38:46.289 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_41 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_42 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_43 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_44 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_45 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_46 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_47 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_48 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_49 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_50 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_51 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_52 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_53 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_54 2024-03-09T14:38:46.290 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_55 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_56 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_57 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_58 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_59 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_60 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_61 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_62 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_63 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_64 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_65 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_66 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_67 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_68 2024-03-09T14:38:46.291 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_69 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_70 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_71 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_72 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_73 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_74 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_75 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_76 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_77 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_78 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_79 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_80 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_81 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_82 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_83 2024-03-09T14:38:46.292 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_84 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_85 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_86 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_87 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_88 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_89 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_90 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_91 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_92 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_93 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_94 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_95 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_96 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_97 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_98 2024-03-09T14:38:46.293 INFO:tasks.workunit.client.0.smithi001.stdout:created: dmarker:f_99 2024-03-09T14:38:46.294 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (1465 ms) 2024-03-09T14:38:46.294 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2024-03-09T14:38:46.298 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=. flags=4 2024-03-09T14:38:46.298 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=.. flags=4 2024-03-09T14:38:46.298 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_0 flags=8 2024-03-09T14:38:46.298 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_1 flags=8 2024-03-09T14:38:46.298 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_10 flags=8 2024-03-09T14:38:46.298 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_11 flags=8 2024-03-09T14:38:46.298 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_12 flags=8 2024-03-09T14:38:46.298 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_13 flags=8 2024-03-09T14:38:46.298 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_14 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_15 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_16 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_17 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_18 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_19 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_2 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_20 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_21 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_22 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_23 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_24 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_25 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_26 flags=8 2024-03-09T14:38:46.299 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_27 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_28 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_29 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_3 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_30 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_31 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_32 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_33 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_34 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_35 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_36 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_37 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_38 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_39 flags=8 2024-03-09T14:38:46.300 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_4 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_40 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_41 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_42 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_43 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_44 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_45 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_46 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_47 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_48 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_49 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_5 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_50 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_51 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_52 flags=8 2024-03-09T14:38:46.301 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_53 flags=8 2024-03-09T14:38:46.302 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_54 flags=8 2024-03-09T14:38:46.302 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_55 flags=8 2024-03-09T14:38:46.302 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_56 flags=8 2024-03-09T14:38:46.302 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_57 flags=8 2024-03-09T14:38:46.302 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_58 flags=8 2024-03-09T14:38:46.302 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_59 flags=8 2024-03-09T14:38:46.302 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_6 flags=8 2024-03-09T14:38:46.302 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_60 flags=8 2024-03-09T14:38:46.302 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_61 flags=8 2024-03-09T14:38:46.302 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_62 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_63 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_64 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_65 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_66 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_67 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_68 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_69 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_7 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_70 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_71 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_72 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_73 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_74 flags=8 2024-03-09T14:38:46.303 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_75 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_76 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_77 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_78 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_79 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_8 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_80 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_81 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_82 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_83 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_84 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_85 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_86 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_87 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_88 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_89 flags=8 2024-03-09T14:38:46.304 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_9 flags=8 2024-03-09T14:38:46.305 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_90 flags=8 2024-03-09T14:38:46.305 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_91 flags=8 2024-03-09T14:38:46.305 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_92 flags=8 2024-03-09T14:38:46.305 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_93 flags=8 2024-03-09T14:38:46.305 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_94 flags=8 2024-03-09T14:38:46.305 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_95 flags=8 2024-03-09T14:38:46.305 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_96 flags=8 2024-03-09T14:38:46.305 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_97 flags=8 2024-03-09T14:38:46.305 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_98 flags=8 2024-03-09T14:38:46.305 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_99 flags=8 2024-03-09T14:38:46.305 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_READDIR (12 ms) 2024-03-09T14:38:46.305 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER2_READDIR 2024-03-09T14:38:46.306 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_0 flags=8 2024-03-09T14:38:46.306 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_1 flags=8 2024-03-09T14:38:46.306 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_10 flags=8 2024-03-09T14:38:46.306 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_11 flags=8 2024-03-09T14:38:46.306 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_12 flags=8 2024-03-09T14:38:46.306 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_13 flags=8 2024-03-09T14:38:46.306 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_14 flags=8 2024-03-09T14:38:46.306 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_15 flags=8 2024-03-09T14:38:46.306 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_16 flags=8 2024-03-09T14:38:46.306 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_17 flags=8 2024-03-09T14:38:46.306 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_18 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_19 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_2 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_20 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_21 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_22 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_23 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_24 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_25 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_26 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_27 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_28 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_29 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_3 flags=8 2024-03-09T14:38:46.307 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_30 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_31 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_32 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_33 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_34 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_35 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_36 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_37 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_38 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_39 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_4 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_40 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_41 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_42 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_43 flags=8 2024-03-09T14:38:46.308 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_44 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_45 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_46 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_47 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_48 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_49 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_5 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_50 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_51 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_52 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_53 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_54 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_55 flags=8 2024-03-09T14:38:46.309 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_56 flags=8 2024-03-09T14:38:46.310 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_57 flags=8 2024-03-09T14:38:46.310 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_58 flags=8 2024-03-09T14:38:46.310 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_59 flags=8 2024-03-09T14:38:46.310 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_6 flags=8 2024-03-09T14:38:46.310 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_60 flags=8 2024-03-09T14:38:46.310 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_61 flags=8 2024-03-09T14:38:46.310 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_62 flags=8 2024-03-09T14:38:46.310 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_63 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_64 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_65 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_66 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_67 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_68 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_69 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_7 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_70 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_71 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_72 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_73 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_74 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_75 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_76 flags=8 2024-03-09T14:38:46.311 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_77 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_78 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_79 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_8 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_80 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_81 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_82 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_83 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_84 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_85 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_86 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_87 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_88 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_89 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_9 flags=8 2024-03-09T14:38:46.312 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_90 flags=8 2024-03-09T14:38:46.313 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_91 flags=8 2024-03-09T14:38:46.313 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_92 flags=8 2024-03-09T14:38:46.313 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_93 flags=8 2024-03-09T14:38:46.313 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_94 flags=8 2024-03-09T14:38:46.313 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_95 flags=8 2024-03-09T14:38:46.313 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_96 flags=8 2024-03-09T14:38:46.313 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_97 flags=8 2024-03-09T14:38:46.313 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_98 flags=8 2024-03-09T14:38:46.313 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_99 flags=8 2024-03-09T14:38:46.313 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER2_READDIR (8 ms) 2024-03-09T14:38:46.313 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2024-03-09T14:38:46.313 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2024-03-09T14:38:46.314 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CLEANUP 2024-03-09T14:38:46.314 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2024-03-09T14:38:46.314 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.UMOUNT 2024-03-09T14:38:46.314 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.UMOUNT (1 ms) 2024-03-09T14:38:46.314 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SHUTDOWN 2024-03-09T14:38:46.314 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:46.309+0000 7f99ab418a80 -1 shutting down 2024-03-09T14:38:46.314 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:46.309+0000 7f9886f8b640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 59899, si_uid : 1000, si_addr0x3e80000e9fb, si_status0 2024-03-09T14:38:46.447 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SHUTDOWN (140 ms) 2024-03-09T14:38:46.447 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 11 tests from LibRGW (2240 ms total) 2024-03-09T14:38:46.447 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:46.448 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:46.448 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 11 tests from 1 test suite ran. (2240 ms total) 2024-03-09T14:38:46.448 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 11 tests. 2024-03-09T14:38:46.462 INFO:tasks.workunit.client.0.smithi001.stdout:phase 4.2 2024-03-09T14:38:46.485 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 11 tests from 1 test suite. 2024-03-09T14:38:46.485 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:46.485 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 11 tests from LibRGW 2024-03-09T14:38:46.485 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.TVAR 2024-03-09T14:38:46.485 INFO:tasks.workunit.client.0.smithi001.stdout:read i1: 64001 2024-03-09T14:38:46.485 INFO:tasks.workunit.client.0.smithi001.stdout:read s1: blunderbuss 2024-03-09T14:38:46.485 INFO:tasks.workunit.client.0.smithi001.stdout:read s3: [ OK ] LibRGW.TVAR (0 ms) 2024-03-09T14:38:46.485 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.INIT 2024-03-09T14:38:46.495 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:46.495 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:46.854 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.INIT (368 ms) 2024-03-09T14:38:46.854 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT 2024-03-09T14:38:46.855 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT (3 ms) 2024-03-09T14:38:46.855 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2024-03-09T14:38:46.872 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (17 ms) 2024-03-09T14:38:46.872 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2024-03-09T14:38:46.873 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2024-03-09T14:38:46.873 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2024-03-09T14:38:46.873 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2024-03-09T14:38:46.873 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER2_READDIR 2024-03-09T14:38:46.879 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_0 flags=8 2024-03-09T14:38:46.879 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_1 flags=8 2024-03-09T14:38:46.879 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_10 flags=8 2024-03-09T14:38:46.879 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_11 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_12 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_13 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_14 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_15 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_16 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_17 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_18 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_19 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_2 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_20 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_21 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_22 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_23 flags=8 2024-03-09T14:38:46.880 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_24 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_25 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_26 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_27 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_28 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_29 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_3 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_30 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_31 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_32 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_33 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_34 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_35 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_36 flags=8 2024-03-09T14:38:46.881 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_37 flags=8 2024-03-09T14:38:46.882 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_38 flags=8 2024-03-09T14:38:46.882 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_39 flags=8 2024-03-09T14:38:46.882 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_4 flags=8 2024-03-09T14:38:46.882 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_40 flags=8 2024-03-09T14:38:46.882 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_41 flags=8 2024-03-09T14:38:46.882 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_42 flags=8 2024-03-09T14:38:46.882 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_43 flags=8 2024-03-09T14:38:46.882 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_44 flags=8 2024-03-09T14:38:46.882 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_45 flags=8 2024-03-09T14:38:46.882 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_46 flags=8 2024-03-09T14:38:46.882 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_47 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_48 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_49 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_5 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_50 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_51 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_52 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_53 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_54 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_55 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_56 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_57 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_58 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_59 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_6 flags=8 2024-03-09T14:38:46.883 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_60 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_61 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_62 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_63 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_64 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_65 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_66 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_67 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_68 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_69 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_7 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_70 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_71 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_72 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_73 flags=8 2024-03-09T14:38:46.884 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_74 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_75 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_76 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_77 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_78 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_79 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_8 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_80 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_81 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_82 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_83 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_84 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_85 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_86 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_87 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_88 flags=8 2024-03-09T14:38:46.885 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_89 flags=8 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_9 flags=8 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_90 flags=8 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_91 flags=8 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_92 flags=8 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_93 flags=8 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_94 flags=8 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_95 flags=8 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_96 flags=8 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_97 flags=8 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_98 flags=8 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_99 flags=8 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER2_READDIR (7 ms) 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2024-03-09T14:38:46.886 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2024-03-09T14:38:46.887 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CLEANUP 2024-03-09T14:38:46.887 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2024-03-09T14:38:46.887 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.UMOUNT 2024-03-09T14:38:46.887 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2024-03-09T14:38:46.887 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SHUTDOWN 2024-03-09T14:38:46.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:46.881+0000 7f2e18dc6a80 -1 shutting down 2024-03-09T14:38:46.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:46.881+0000 7f2cf4939640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 60542, si_uid : 1000, si_addr0x3e80000ec7e, si_status0 2024-03-09T14:38:46.967 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SHUTDOWN (88 ms) 2024-03-09T14:38:46.967 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 11 tests from LibRGW (483 ms total) 2024-03-09T14:38:46.967 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:46.968 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:46.968 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 11 tests from 1 test suite ran. (483 ms total) 2024-03-09T14:38:46.968 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 11 tests. 2024-03-09T14:38:46.976 INFO:tasks.workunit.client.0.smithi001.stdout:phase 5.1 2024-03-09T14:38:47.010 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 19 tests from 1 test suite. 2024-03-09T14:38:47.010 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:47.010 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 19 tests from LibRGW 2024-03-09T14:38:47.010 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.INIT 2024-03-09T14:38:47.021 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:47.022 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:47.306 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.INIT (296 ms) 2024-03-09T14:38:47.306 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT 2024-03-09T14:38:47.307 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2024-03-09T14:38:47.308 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2024-03-09T14:38:47.323 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CREATE_BUCKET (16 ms) 2024-03-09T14:38:47.323 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2024-03-09T14:38:47.329 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (5 ms) 2024-03-09T14:38:47.329 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LIST_OBJECTS 2024-03-09T14:38:47.329 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LIST_OBJECTS (0 ms) 2024-03-09T14:38:47.329 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2024-03-09T14:38:47.330 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (2 ms) 2024-03-09T14:38:47.331 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.OBJ_OPEN 2024-03-09T14:38:47.331 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.OBJ_OPEN (0 ms) 2024-03-09T14:38:47.331 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.PUT_OBJECT 2024-03-09T14:38:47.338 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.PUT_OBJECT (8 ms) 2024-03-09T14:38:47.338 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.GET_OBJECT 2024-03-09T14:38:47.341 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.GET_OBJECT (2 ms) 2024-03-09T14:38:47.341 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.STAT_OBJECT 2024-03-09T14:38:47.341 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2024-03-09T14:38:47.341 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.WRITE_READ_VERIFY 2024-03-09T14:38:47.341 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.WRITE_READ_VERIFY (0 ms) 2024-03-09T14:38:47.341 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.WRITEV 2024-03-09T14:38:47.341 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.WRITEV (0 ms) 2024-03-09T14:38:47.342 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READV 2024-03-09T14:38:47.342 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READV (0 ms) 2024-03-09T14:38:47.342 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READV_AFTER_WRITEV 2024-03-09T14:38:47.342 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READV_AFTER_WRITEV (0 ms) 2024-03-09T14:38:47.342 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2024-03-09T14:38:47.342 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.DELETE_OBJECT (0 ms) 2024-03-09T14:38:47.342 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2024-03-09T14:38:47.342 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.DELETE_BUCKET (0 ms) 2024-03-09T14:38:47.342 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CLEANUP 2024-03-09T14:38:47.343 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2024-03-09T14:38:47.343 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.UMOUNT 2024-03-09T14:38:47.343 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2024-03-09T14:38:47.343 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SHUTDOWN 2024-03-09T14:38:47.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:47.341+0000 7f2d0aa9ba80 -1 shutting down 2024-03-09T14:38:47.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:47.345+0000 7f2be660e640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 61185, si_uid : 1000, si_addr0x3e80000ef01, si_status0 2024-03-09T14:38:47.465 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SHUTDOWN (125 ms) 2024-03-09T14:38:47.466 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 19 tests from LibRGW (455 ms total) 2024-03-09T14:38:47.466 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:47.466 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:47.466 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 19 tests from 1 test suite ran. (455 ms total) 2024-03-09T14:38:47.466 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 19 tests. 2024-03-09T14:38:47.479 INFO:tasks.workunit.client.0.smithi001.stdout:phase 5.2 2024-03-09T14:38:47.514 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] Running 19 tests from 1 test suite. 2024-03-09T14:38:47.514 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment set-up. 2024-03-09T14:38:47.514 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 19 tests from LibRGW 2024-03-09T14:38:47.515 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.INIT 2024-03-09T14:38:47.523 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:47.523 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:48.081 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.INIT (566 ms) 2024-03-09T14:38:48.081 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.MOUNT 2024-03-09T14:38:48.086 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.MOUNT (3 ms) 2024-03-09T14:38:48.086 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2024-03-09T14:38:48.086 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CREATE_BUCKET (0 ms) 2024-03-09T14:38:48.086 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2024-03-09T14:38:48.092 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (8 ms) 2024-03-09T14:38:48.092 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LIST_OBJECTS 2024-03-09T14:38:48.092 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LIST_OBJECTS (0 ms) 2024-03-09T14:38:48.092 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2024-03-09T14:38:48.092 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (0 ms) 2024-03-09T14:38:48.092 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.OBJ_OPEN 2024-03-09T14:38:48.092 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.OBJ_OPEN (0 ms) 2024-03-09T14:38:48.092 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.PUT_OBJECT 2024-03-09T14:38:48.092 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.PUT_OBJECT (0 ms) 2024-03-09T14:38:48.092 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.GET_OBJECT 2024-03-09T14:38:48.092 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.GET_OBJECT (0 ms) 2024-03-09T14:38:48.092 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.STAT_OBJECT 2024-03-09T14:38:48.093 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2024-03-09T14:38:48.093 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.WRITE_READ_VERIFY 2024-03-09T14:38:48.093 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.WRITE_READ_VERIFY (0 ms) 2024-03-09T14:38:48.093 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.WRITEV 2024-03-09T14:38:48.093 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.WRITEV (0 ms) 2024-03-09T14:38:48.093 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READV 2024-03-09T14:38:48.093 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READV (0 ms) 2024-03-09T14:38:48.093 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.READV_AFTER_WRITEV 2024-03-09T14:38:48.093 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.READV_AFTER_WRITEV (0 ms) 2024-03-09T14:38:48.093 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2024-03-09T14:38:48.102 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.DELETE_OBJECT (11 ms) 2024-03-09T14:38:48.102 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2024-03-09T14:38:48.131 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.DELETE_BUCKET (29 ms) 2024-03-09T14:38:48.131 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.CLEANUP 2024-03-09T14:38:48.131 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2024-03-09T14:38:48.131 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.UMOUNT 2024-03-09T14:38:48.131 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2024-03-09T14:38:48.131 INFO:tasks.workunit.client.0.smithi001.stdout:[ RUN ] LibRGW.SHUTDOWN 2024-03-09T14:38:48.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.133+0000 7fcc01507a80 -1 shutting down 2024-03-09T14:38:48.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.133+0000 7fcadd07a640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 61828, si_uid : 1000, si_addr0x3e80000f184, si_status0 2024-03-09T14:38:48.222 INFO:tasks.workunit.client.0.smithi001.stdout:[ OK ] LibRGW.SHUTDOWN (91 ms) 2024-03-09T14:38:48.222 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] 19 tests from LibRGW (708 ms total) 2024-03-09T14:38:48.222 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:48.222 INFO:tasks.workunit.client.0.smithi001.stdout:[----------] Global test environment tear-down 2024-03-09T14:38:48.222 INFO:tasks.workunit.client.0.smithi001.stdout:[==========] 19 tests from 1 test suite ran. (708 ms total) 2024-03-09T14:38:48.223 INFO:tasks.workunit.client.0.smithi001.stdout:[ PASSED ] 19 tests. 2024-03-09T14:38:48.230 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:38:48.231 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:38:48.246 INFO:tasks.workunit:Running workunits matching rgw/test_awssdkv4_sig.sh on client.0... 2024-03-09T14:38:48.247 INFO:tasks.workunit:Running workunit rgw/test_awssdkv4_sig.sh... 2024-03-09T14:38:48.248 DEBUG:teuthology.orchestra.run.smithi001:workunit test rgw/test_awssdkv4_sig.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_awssdkv4_sig.sh 2024-03-09T14:38:48.299 INFO:tasks.workunit.client.0.smithi001.stderr:+ [ -z ] 2024-03-09T14:38:48.299 INFO:tasks.workunit.client.0.smithi001.stderr:+ export AWS_ACCESS_KEY_ID=lNCnR47C2g+ZidCWBAUuwfSAA7Q= 2024-03-09T14:38:48.300 INFO:tasks.workunit.client.0.smithi001.stderr:+ export AWS_SECRET_ACCESS_KEY=tYuA2Y+Uu1ow2l9Xe59tWKVml3gMuVfyhUjjJwfwEI0vFFONIcqf4g== 2024-03-09T14:38:48.300 INFO:tasks.workunit.client.0.smithi001.stderr:+ radosgw-admin user create --uid ceph-test-maven --access-key lNCnR47C2g+ZidCWBAUuwfSAA7Q= --secret tYuA2Y+Uu1ow2l9Xe59tWKVml3gMuVfyhUjjJwfwEI0vFFONIcqf4g== --display-name maven test user --email sigv4@example.com 2024-03-09T14:38:48.336 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:38:48.336 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:38:48.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.381+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.385+0000 7fd89af84980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:48.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.385+0000 7fd89af84980 20 realm 2024-03-09T14:38:48.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.385+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.385+0000 7fd89af84980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:48.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.385+0000 7fd89af84980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:38:48.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.385+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.385+0000 7fd89af84980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:48.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.385+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.385+0000 7fd89af84980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:38:48.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.385+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.385+0000 7fd89af84980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:38:48.387 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.385+0000 7fd89af84980 20 searching for the correct realm 2024-03-09T14:38:48.393 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.393+0000 7fd89929d640 0 -- 172.21.15.1:0/3188685055 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x561f8d61d4b0 msgr2=0x561f8d61f8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:38:48.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.401+0000 7fd89829b640 0 -- 172.21.15.1:0/3188685055 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fd88802e070 msgr2=0x7fd88803f480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:38:48.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:38:48.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:38:48.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:38:48.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:38:48.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:38:48.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:38:48.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:48.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:38:48.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:38:48.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 zone default found 2024-03-09T14:38:48.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 4 Realm: () 2024-03-09T14:38:48.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:38:48.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:38:48.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:38:48.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 zonegroup default 2024-03-09T14:38:48.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:48.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:48.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:48.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 started sync module instance, tier type = 2024-03-09T14:38:48.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.409+0000 7fd89af84980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:38:48.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.425+0000 7fd89af84980 20 add_watcher() i=0 2024-03-09T14:38:48.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.425+0000 7fd89af84980 20 add_watcher() i=1 2024-03-09T14:38:48.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.425+0000 7fd89af84980 20 add_watcher() i=2 2024-03-09T14:38:48.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.425+0000 7fd89af84980 20 add_watcher() i=3 2024-03-09T14:38:48.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.425+0000 7fd89af84980 20 add_watcher() i=4 2024-03-09T14:38:48.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.425+0000 7fd89af84980 20 add_watcher() i=5 2024-03-09T14:38:48.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.425+0000 7fd89af84980 20 add_watcher() i=6 2024-03-09T14:38:48.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.425+0000 7fd89af84980 20 add_watcher() i=7 2024-03-09T14:38:48.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.425+0000 7fd89af84980 2 all 8 watchers are set, enabling cache 2024-03-09T14:38:48.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.429+0000 7fd74f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:38:48.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.429+0000 7fd89af84980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:38:48.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.429+0000 7fd89af84980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:38:48.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.429+0000 7fd89af84980 5 note: GC not initialized 2024-03-09T14:38:48.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.429+0000 7fd74e7e4640 20 reqs_thread_entry: start 2024-03-09T14:38:48.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.429+0000 7fd89af84980 20 init_complete bucket index max shards: 1 2024-03-09T14:38:48.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.429+0000 7fd89af84980 20 Filter name: none 2024-03-09T14:38:48.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.429+0000 7fd7477fe640 20 reqs_thread_entry: start 2024-03-09T14:38:48.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.429+0000 7fd89af84980 10 cache get: name=default.rgw.meta+users.uid+ceph-test-maven : miss 2024-03-09T14:38:48.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.429+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:48.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 cache put: name=default.rgw.meta+users.uid+ceph-test-maven info.flags=0x0 2024-03-09T14:38:48.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 adding default.rgw.meta+users.uid+ceph-test-maven to cache LRU end 2024-03-09T14:38:48.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 cache get: name=default.rgw.meta+users.email+sigv4@example.com : miss 2024-03-09T14:38:48.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:48.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 cache put: name=default.rgw.meta+users.email+sigv4@example.com info.flags=0x0 2024-03-09T14:38:48.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 adding default.rgw.meta+users.email+sigv4@example.com to cache LRU end 2024-03-09T14:38:48.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 cache get: name=default.rgw.meta+users.keys+lNCnR47C2g+ZidCWBAUuwfSAA7Q= : miss 2024-03-09T14:38:48.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 20 rados->read ofs=0 len=0 2024-03-09T14:38:48.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:38:48.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 cache put: name=default.rgw.meta+users.keys+lNCnR47C2g+ZidCWBAUuwfSAA7Q= info.flags=0x0 2024-03-09T14:38:48.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 adding default.rgw.meta+users.keys+lNCnR47C2g+ZidCWBAUuwfSAA7Q= to cache LRU end 2024-03-09T14:38:48.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 cache get: name=default.rgw.meta+users.keys+lNCnR47C2g+ZidCWBAUuwfSAA7Q= : hit (negative entry) 2024-03-09T14:38:48.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 cache get: name=default.rgw.meta+users.keys+lNCnR47C2g+ZidCWBAUuwfSAA7Q= : hit (negative entry) 2024-03-09T14:38:48.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 cache put: name=default.rgw.meta+users.uid+ceph-test-maven info.flags=0x17 2024-03-09T14:38:48.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 moving default.rgw.meta+users.uid+ceph-test-maven to cache LRU end 2024-03-09T14:38:48.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd89af84980 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:ceph-test-maven, ofs0, ns] 2024-03-09T14:38:48.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd76ffff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1060856922125 cookie 94693516034288 notifier 5456 bl.length()=677 2024-03-09T14:38:48.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd76ffff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ceph-test-maven info.flags=0x17 2024-03-09T14:38:48.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.433+0000 7fd76ffff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ceph-test-maven to cache LRU end 2024-03-09T14:38:48.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.437+0000 7fd89af84980 10 cache put: name=default.rgw.meta+users.email+sigv4@example.com info.flags=0x7 2024-03-09T14:38:48.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.437+0000 7fd89af84980 10 moving default.rgw.meta+users.email+sigv4@example.com to cache LRU end 2024-03-09T14:38:48.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.437+0000 7fd89af84980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.email:sigv4@example.com, ofs0, ns] 2024-03-09T14:38:48.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.437+0000 7fd76f7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1060856922165 cookie 94693516029280 notifier 5456 bl.length()=187 2024-03-09T14:38:48.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.437+0000 7fd76f7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+sigv4@example.com info.flags=0x7 2024-03-09T14:38:48.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.437+0000 7fd76f7fe640 10 rgw watcher librados: moving default.rgw.meta+users.email+sigv4@example.com to cache LRU end 2024-03-09T14:38:48.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.441+0000 7fd89af84980 10 cache put: name=default.rgw.meta+users.keys+lNCnR47C2g+ZidCWBAUuwfSAA7Q= info.flags=0x7 2024-03-09T14:38:48.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.441+0000 7fd89af84980 10 moving default.rgw.meta+users.keys+lNCnR47C2g+ZidCWBAUuwfSAA7Q= to cache LRU end 2024-03-09T14:38:48.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.441+0000 7fd89af84980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:lNCnR47C2g+ZidCWBAUuwfSAA7Q=, ofs0, ns] 2024-03-09T14:38:48.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.445+0000 7fd76ffff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1060856922166 cookie 94693516029280 notifier 5456 bl.length()=197 2024-03-09T14:38:48.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.445+0000 7fd76ffff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+lNCnR47C2g+ZidCWBAUuwfSAA7Q= info.flags=0x7 2024-03-09T14:38:48.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.445+0000 7fd76ffff640 10 rgw watcher librados: moving default.rgw.meta+users.keys+lNCnR47C2g+ZidCWBAUuwfSAA7Q= to cache LRU end 2024-03-09T14:38:48.446 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:38:48.446 INFO:tasks.workunit.client.0.smithi001.stdout: "user_id": "ceph-test-maven", 2024-03-09T14:38:48.447 INFO:tasks.workunit.client.0.smithi001.stdout: "display_name": "maven test user", 2024-03-09T14:38:48.447 INFO:tasks.workunit.client.0.smithi001.stdout: "email": "sigv4@example.com", 2024-03-09T14:38:48.447 INFO:tasks.workunit.client.0.smithi001.stdout: "suspended": 0, 2024-03-09T14:38:48.447 INFO:tasks.workunit.client.0.smithi001.stdout: "max_buckets": 1000, 2024-03-09T14:38:48.447 INFO:tasks.workunit.client.0.smithi001.stdout: "subusers": [], 2024-03-09T14:38:48.447 INFO:tasks.workunit.client.0.smithi001.stdout: "keys": [ 2024-03-09T14:38:48.447 INFO:tasks.workunit.client.0.smithi001.stdout: { 2024-03-09T14:38:48.447 INFO:tasks.workunit.client.0.smithi001.stdout: "user": "ceph-test-maven", 2024-03-09T14:38:48.447 INFO:tasks.workunit.client.0.smithi001.stdout: "access_key": "lNCnR47C2g+ZidCWBAUuwfSAA7Q=", 2024-03-09T14:38:48.447 INFO:tasks.workunit.client.0.smithi001.stdout: "secret_key": "tYuA2Y+Uu1ow2l9Xe59tWKVml3gMuVfyhUjjJwfwEI0vFFONIcqf4g==", 2024-03-09T14:38:48.448 INFO:tasks.workunit.client.0.smithi001.stdout: "active": true, 2024-03-09T14:38:48.448 INFO:tasks.workunit.client.0.smithi001.stdout: "create_date": "2024-03-09T14:38:48.434703Z" 2024-03-09T14:38:48.448 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:38:48.448 INFO:tasks.workunit.client.0.smithi001.stdout: ], 2024-03-09T14:38:48.448 INFO:tasks.workunit.client.0.smithi001.stdout: "swift_keys": [], 2024-03-09T14:38:48.448 INFO:tasks.workunit.client.0.smithi001.stdout: "caps": [], 2024-03-09T14:38:48.448 INFO:tasks.workunit.client.0.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T14:38:48.448 INFO:tasks.workunit.client.0.smithi001.stdout: "default_placement": "", 2024-03-09T14:38:48.448 INFO:tasks.workunit.client.0.smithi001.stdout: "default_storage_class": "", 2024-03-09T14:38:48.448 INFO:tasks.workunit.client.0.smithi001.stdout: "placement_tags": [], 2024-03-09T14:38:48.448 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_quota": { 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: "enabled": false, 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: "check_on_raw": false, 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: "max_size": -1, 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: "max_size_kb": 0, 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: "max_objects": -1 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: "user_quota": { 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: "enabled": false, 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: "check_on_raw": false, 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: "max_size": -1, 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: "max_size_kb": 0, 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: "max_objects": -1 2024-03-09T14:38:48.449 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:38:48.450 INFO:tasks.workunit.client.0.smithi001.stdout: "temp_url_keys": [], 2024-03-09T14:38:48.450 INFO:tasks.workunit.client.0.smithi001.stdout: "type": "rgw", 2024-03-09T14:38:48.450 INFO:tasks.workunit.client.0.smithi001.stdout: "mfa_ids": [], 2024-03-09T14:38:48.450 INFO:tasks.workunit.client.0.smithi001.stdout: "account_id": "", 2024-03-09T14:38:48.450 INFO:tasks.workunit.client.0.smithi001.stdout: "path": "/", 2024-03-09T14:38:48.450 INFO:tasks.workunit.client.0.smithi001.stdout: "create_date": "2024-03-09T14:38:48.434681Z", 2024-03-09T14:38:48.450 INFO:tasks.workunit.client.0.smithi001.stdout: "tags": [], 2024-03-09T14:38:48.450 INFO:tasks.workunit.client.0.smithi001.stdout: "group_ids": [] 2024-03-09T14:38:48.450 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:38:48.450 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:38:48.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.445+0000 7fd74f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:38:48.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.453+0000 7fd89af84980 20 remove_watcher() i=0 2024-03-09T14:38:48.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.453+0000 7fd89af84980 2 removed watcher, disabling cache 2024-03-09T14:38:48.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.453+0000 7fd89af84980 20 remove_watcher() i=1 2024-03-09T14:38:48.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.457+0000 7fd89af84980 20 remove_watcher() i=2 2024-03-09T14:38:48.458 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.457+0000 7fd89af84980 20 remove_watcher() i=3 2024-03-09T14:38:48.461 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.461+0000 7fd89af84980 20 remove_watcher() i=4 2024-03-09T14:38:48.462 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.461+0000 7fd89af84980 20 remove_watcher() i=5 2024-03-09T14:38:48.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.465+0000 7fd89af84980 20 remove_watcher() i=6 2024-03-09T14:38:48.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:38:48.465+0000 7fd89af84980 20 remove_watcher() i=7 2024-03-09T14:38:48.479 INFO:tasks.workunit.client.0.smithi001.stderr:+ [ -z ] 2024-03-09T14:38:48.479 INFO:tasks.workunit.client.0.smithi001.stderr:+ [ -z /home/ubuntu/cephtest ] 2024-03-09T14:38:48.480 INFO:tasks.workunit.client.0.smithi001.stderr:+ cat /home/ubuntu/cephtest/url_file 2024-03-09T14:38:48.480 INFO:tasks.workunit.client.0.smithi001.stderr:+ export RGW_HTTP_ENDPOINT_URL=https://smithi001.front.sepia.ceph.com:443 2024-03-09T14:38:48.480 INFO:tasks.workunit.client.0.smithi001.stderr:+ [ -z /home/ubuntu/cephtest/clone.client.0 ] 2024-03-09T14:38:48.481 INFO:tasks.workunit.client.0.smithi001.stderr:+ cd /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum 2024-03-09T14:38:48.481 INFO:tasks.workunit.client.0.smithi001.stderr:+ ./mvnw clean package 2024-03-09T14:38:48.486 INFO:tasks.workunit.client.0.smithi001.stdout:Warning: JAVA_HOME environment variable is not set. 2024-03-09T14:38:51.280 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Scanning for projects... 2024-03-09T14:38:51.412 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/junit-bom/5.10.1/junit-bom-5.10.1.pom 2024-03-09T14:38:51.530 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.4/5.6 kB Progress (1): 2.8/5.6 kB Progress (1): 4.1/5.6 kB Progress (1): 5.5/5.6 kB Progress (1): 5.6 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/junit-bom/5.10.1/junit-bom-5.10.1.pom (5.6 kB at 46 kB/s) 2024-03-09T14:38:51.547 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/bom/2.20.43/bom-2.20.43.pom 2024-03-09T14:38:51.589 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.4/83 kB Progress (1): 2.8/83 kB Progress (1): 4.1/83 kB Progress (1): 5.5/83 kB Progress (1): 6.9/83 kB Progress (1): 8.3/83 kB Progress (1): 9.7/83 kB Progress (1): 11/83 kB Progress (1): 12/83 kB Progress (1): 14/83 kB Progress (1): 15/83 kB Progress (1): 16/83 kB Progress (1): 32/83 kB Progress (1): 49/83 kB Progress (1): 65/83 kB Progress (1): 81/83 kB Progress (1): 83 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/bom/2.20.43/bom-2.20.43.pom (83 kB at 1.9 MB/s) 2024-03-09T14:38:51.611 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-sdk-java-pom/2.20.43/aws-sdk-java-pom-2.20.43.pom 2024-03-09T14:38:51.638 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/77 kB Progress (1): 33/77 kB Progress (1): 49/77 kB Progress (1): 66/77 kB Progress (1): 77 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-sdk-java-pom/2.20.43/aws-sdk-java-pom-2.20.43.pom (77 kB at 2.8 MB/s) 2024-03-09T14:38:51.677 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:38:51.677 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] ---------------------------< io.ceph:jcksum >--------------------------- 2024-03-09T14:38:51.677 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Building jcksum 1.0 2024-03-09T14:38:51.677 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] from pom.xml 2024-03-09T14:38:51.677 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --------------------------------[ jar ]--------------------------------- 2024-03-09T14:38:51.681 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/3.2.0/maven-clean-plugin-3.2.0.pom 2024-03-09T14:38:51.701 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.3 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/3.2.0/maven-clean-plugin-3.2.0.pom (5.3 kB at 265 kB/s) 2024-03-09T14:38:51.712 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/35/maven-plugins-35.pom 2024-03-09T14:38:51.731 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 9.9 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/35/maven-plugins-35.pom (9.9 kB at 495 kB/s) 2024-03-09T14:38:51.736 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/35/maven-parent-35.pom 2024-03-09T14:38:51.757 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/45 kB Progress (1): 33/45 kB Progress (1): 45 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/35/maven-parent-35.pom (45 kB at 2.2 MB/s) 2024-03-09T14:38:51.765 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/apache/25/apache-25.pom 2024-03-09T14:38:51.786 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/21 kB Progress (1): 21 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/apache/25/apache-25.pom (21 kB at 1.0 MB/s) 2024-03-09T14:38:51.805 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/3.2.0/maven-clean-plugin-3.2.0.jar 2024-03-09T14:38:51.826 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/36 kB Progress (1): 33/36 kB Progress (1): 36 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/3.2.0/maven-clean-plugin-3.2.0.jar (36 kB at 1.6 MB/s) 2024-03-09T14:38:51.838 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/3.3.1/maven-resources-plugin-3.3.1.pom 2024-03-09T14:38:51.858 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 8.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/3.3.1/maven-resources-plugin-3.3.1.pom (8.2 kB at 388 kB/s) 2024-03-09T14:38:51.864 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/39/maven-plugins-39.pom 2024-03-09T14:38:51.884 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 8.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/39/maven-plugins-39.pom (8.1 kB at 405 kB/s) 2024-03-09T14:38:51.888 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/39/maven-parent-39.pom 2024-03-09T14:38:51.910 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/48 kB Progress (1): 23/48 kB Progress (1): 39/48 kB Progress (1): 48 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/39/maven-parent-39.pom (48 kB at 2.2 MB/s) 2024-03-09T14:38:51.917 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/apache/29/apache-29.pom 2024-03-09T14:38:51.938 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/21 kB Progress (1): 21 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/apache/29/apache-29.pom (21 kB at 987 kB/s) 2024-03-09T14:38:51.949 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/3.3.1/maven-resources-plugin-3.3.1.jar 2024-03-09T14:38:51.970 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/31 kB Progress (1): 31 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/3.3.1/maven-resources-plugin-3.3.1.jar (31 kB at 1.5 MB/s) 2024-03-09T14:38:51.981 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-compiler-plugin/3.11.0/maven-compiler-plugin-3.11.0.pom 2024-03-09T14:38:52.001 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 9.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-compiler-plugin/3.11.0/maven-compiler-plugin-3.11.0.pom (9.8 kB at 467 kB/s) 2024-03-09T14:38:52.013 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-compiler-plugin/3.11.0/maven-compiler-plugin-3.11.0.jar 2024-03-09T14:38:52.051 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/66 kB Progress (1): 33/66 kB Progress (1): 49/66 kB Progress (1): 66/66 kB Progress (1): 66 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-compiler-plugin/3.11.0/maven-compiler-plugin-3.11.0.jar (66 kB at 1.7 MB/s) 2024-03-09T14:38:52.068 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/3.2.3/maven-surefire-plugin-3.2.3.pom 2024-03-09T14:38:52.088 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.3 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/3.2.3/maven-surefire-plugin-3.2.3.pom (5.3 kB at 253 kB/s) 2024-03-09T14:38:52.093 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire/3.2.3/surefire-3.2.3.pom 2024-03-09T14:38:52.114 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/22 kB Progress (1): 22 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire/3.2.3/surefire-3.2.3.pom (22 kB at 1.0 MB/s) 2024-03-09T14:38:52.121 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/41/maven-parent-41.pom 2024-03-09T14:38:52.144 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/50 kB Progress (1): 33/50 kB Progress (1): 49/50 kB Progress (1): 50 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/41/maven-parent-41.pom (50 kB at 2.2 MB/s) 2024-03-09T14:38:52.155 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/apache/31/apache-31.pom 2024-03-09T14:38:52.177 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/24 kB Progress (1): 24 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/apache/31/apache-31.pom (24 kB at 1.0 MB/s) 2024-03-09T14:38:52.193 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/junit-bom/5.9.3/junit-bom-5.9.3.pom 2024-03-09T14:38:52.213 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.6 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/junit-bom/5.9.3/junit-bom-5.9.3.pom (5.6 kB at 268 kB/s) 2024-03-09T14:38:52.220 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/3.2.3/maven-surefire-plugin-3.2.3.jar 2024-03-09T14:38:52.241 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/45 kB Progress (1): 33/45 kB Progress (1): 45 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/3.2.3/maven-surefire-plugin-3.2.3.jar (45 kB at 2.1 MB/s) 2024-03-09T14:38:52.252 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/3.3.0/maven-jar-plugin-3.3.0.pom 2024-03-09T14:38:52.272 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/3.3.0/maven-jar-plugin-3.3.0.pom (6.8 kB at 338 kB/s) 2024-03-09T14:38:52.278 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/37/maven-plugins-37.pom 2024-03-09T14:38:52.299 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 9.9 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/37/maven-plugins-37.pom (9.9 kB at 450 kB/s) 2024-03-09T14:38:52.306 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/37/maven-parent-37.pom 2024-03-09T14:38:52.328 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/46 kB Progress (1): 33/46 kB Progress (1): 46 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/37/maven-parent-37.pom (46 kB at 2.0 MB/s) 2024-03-09T14:38:52.336 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/apache/27/apache-27.pom 2024-03-09T14:38:52.356 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/20 kB Progress (1): 20 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/apache/27/apache-27.pom (20 kB at 1.1 MB/s) 2024-03-09T14:38:52.364 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/3.3.0/maven-jar-plugin-3.3.0.jar 2024-03-09T14:38:52.386 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/27 kB Progress (1): 27 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/3.3.0/maven-jar-plugin-3.3.0.jar (27 kB at 1.2 MB/s) 2024-03-09T14:38:52.417 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/s3/2.20.43/s3-2.20.43.pom 2024-03-09T14:38:52.436 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/s3/2.20.43/s3-2.20.43.pom (6.1 kB at 320 kB/s) 2024-03-09T14:38:52.439 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/services/2.20.43/services-2.20.43.pom 2024-03-09T14:38:52.458 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/21 kB Progress (1): 21 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/services/2.20.43/services-2.20.43.pom (21 kB at 1.1 MB/s) 2024-03-09T14:38:52.467 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/bom-internal/2.20.43/bom-internal-2.20.43.pom 2024-03-09T14:38:52.488 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/20 kB Progress (1): 20 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/bom-internal/2.20.43/bom-internal-2.20.43.pom (20 kB at 953 kB/s) 2024-03-09T14:38:52.504 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-xml-protocol/2.20.43/aws-xml-protocol-2.20.43.pom 2024-03-09T14:38:52.524 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-xml-protocol/2.20.43/aws-xml-protocol-2.20.43.pom (4.0 kB at 200 kB/s) 2024-03-09T14:38:52.527 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/protocols/2.20.43/protocols-2.20.43.pom 2024-03-09T14:38:52.546 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.6 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/protocols/2.20.43/protocols-2.20.43.pom (1.6 kB at 82 kB/s) 2024-03-09T14:38:52.549 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/core/2.20.43/core-2.20.43.pom 2024-03-09T14:38:52.567 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/core/2.20.43/core-2.20.43.pom (2.2 kB at 124 kB/s) 2024-03-09T14:38:52.577 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-query-protocol/2.20.43/aws-query-protocol-2.20.43.pom 2024-03-09T14:38:52.597 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.6 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-query-protocol/2.20.43/aws-query-protocol-2.20.43.pom (3.6 kB at 173 kB/s) 2024-03-09T14:38:52.607 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/protocol-core/2.20.43/protocol-core-2.20.43.pom 2024-03-09T14:38:52.627 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/protocol-core/2.20.43/protocol-core-2.20.43.pom (3.2 kB at 161 kB/s) 2024-03-09T14:38:52.635 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/sdk-core/2.20.43/sdk-core-2.20.43.pom 2024-03-09T14:38:52.654 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 11 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/sdk-core/2.20.43/sdk-core-2.20.43.pom (11 kB at 577 kB/s) 2024-03-09T14:38:52.660 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/annotations/2.20.43/annotations-2.20.43.pom 2024-03-09T14:38:52.679 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.7 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/annotations/2.20.43/annotations-2.20.43.pom (1.7 kB at 89 kB/s) 2024-03-09T14:38:52.685 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/http-client-spi/2.20.43/http-client-spi-2.20.43.pom 2024-03-09T14:38:52.704 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/http-client-spi/2.20.43/http-client-spi-2.20.43.pom (3.8 kB at 201 kB/s) 2024-03-09T14:38:52.712 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/utils/2.20.43/utils-2.20.43.pom 2024-03-09T14:38:52.731 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.9 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/utils/2.20.43/utils-2.20.43.pom (4.9 kB at 246 kB/s) 2024-03-09T14:38:52.741 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/reactivestreams/reactive-streams/1.0.3/reactive-streams-1.0.3.pom 2024-03-09T14:38:52.761 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/reactivestreams/reactive-streams/1.0.3/reactive-streams-1.0.3.pom (1.2 kB at 58 kB/s) 2024-03-09T14:38:52.767 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.30/slf4j-api-1.7.30.pom 2024-03-09T14:38:52.787 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.30/slf4j-api-1.7.30.pom (3.8 kB at 192 kB/s) 2024-03-09T14:38:52.792 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.30/slf4j-parent-1.7.30.pom 2024-03-09T14:38:52.812 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 14 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.30/slf4j-parent-1.7.30.pom (14 kB at 657 kB/s) 2024-03-09T14:38:52.819 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/metrics-spi/2.20.43/metrics-spi-2.20.43.pom 2024-03-09T14:38:52.844 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.7 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/metrics-spi/2.20.43/metrics-spi-2.20.43.pom (3.7 kB at 147 kB/s) 2024-03-09T14:38:52.853 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/endpoints-spi/2.20.43/endpoints-spi-2.20.43.pom 2024-03-09T14:38:52.873 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.5 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/endpoints-spi/2.20.43/endpoints-spi-2.20.43.pom (2.5 kB at 122 kB/s) 2024-03-09T14:38:52.881 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/profiles/2.20.43/profiles-2.20.43.pom 2024-03-09T14:38:52.904 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.9 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/profiles/2.20.43/profiles-2.20.43.pom (2.9 kB at 121 kB/s) 2024-03-09T14:38:52.913 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-core/2.20.43/aws-core-2.20.43.pom 2024-03-09T14:38:52.935 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 7.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-core/2.20.43/aws-core-2.20.43.pom (7.4 kB at 353 kB/s) 2024-03-09T14:38:52.943 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/regions/2.20.43/regions-2.20.43.pom 2024-03-09T14:38:52.963 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.5 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/regions/2.20.43/regions-2.20.43.pom (5.5 kB at 273 kB/s) 2024-03-09T14:38:52.970 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/json-utils/2.20.43/json-utils-2.20.43.pom 2024-03-09T14:38:52.993 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.5 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/json-utils/2.20.43/json-utils-2.20.43.pom (3.5 kB at 145 kB/s) 2024-03-09T14:38:53.000 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/third-party-jackson-core/2.20.43/third-party-jackson-core-2.20.43.pom 2024-03-09T14:38:53.024 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/third-party-jackson-core/2.20.43/third-party-jackson-core-2.20.43.pom (4.4 kB at 176 kB/s) 2024-03-09T14:38:53.028 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/third-party/2.20.43/third-party-2.20.43.pom 2024-03-09T14:38:53.055 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/third-party/2.20.43/third-party-2.20.43.pom (2.4 kB at 86 kB/s) 2024-03-09T14:38:53.061 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/auth/2.20.43/auth-2.20.43.pom 2024-03-09T14:38:53.089 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/auth/2.20.43/auth-2.20.43.pom (6.4 kB at 230 kB/s) 2024-03-09T14:38:53.096 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/eventstream/eventstream/1.0.1/eventstream-1.0.1.pom 2024-03-09T14:38:53.115 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/eventstream/eventstream/1.0.1/eventstream-1.0.1.pom (6.1 kB at 303 kB/s) 2024-03-09T14:38:53.124 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/arns/2.20.43/arns-2.20.43.pom 2024-03-09T14:38:53.162 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/arns/2.20.43/arns-2.20.43.pom (3.0 kB at 76 kB/s) 2024-03-09T14:38:53.172 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/crt-core/2.20.43/crt-core-2.20.43.pom 2024-03-09T14:38:53.197 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.5 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/crt-core/2.20.43/crt-core-2.20.43.pom (3.5 kB at 139 kB/s) 2024-03-09T14:38:53.209 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/apache-client/2.20.43/apache-client-2.20.43.pom 2024-03-09T14:38:53.238 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/apache-client/2.20.43/apache-client-2.20.43.pom (4.1 kB at 136 kB/s) 2024-03-09T14:38:53.246 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/http-clients/2.20.43/http-clients-2.20.43.pom 2024-03-09T14:38:53.297 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/http-clients/2.20.43/http-clients-2.20.43.pom (1.8 kB at 34 kB/s) 2024-03-09T14:38:53.307 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5.13/httpclient-4.5.13.pom 2024-03-09T14:38:53.329 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.6 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5.13/httpclient-4.5.13.pom (6.6 kB at 300 kB/s) 2024-03-09T14:38:53.334 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-client/4.5.13/httpcomponents-client-4.5.13.pom 2024-03-09T14:38:53.355 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-client/4.5.13/httpcomponents-client-4.5.13.pom (16 kB at 819 kB/s) 2024-03-09T14:38:53.360 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-parent/11/httpcomponents-parent-11.pom 2024-03-09T14:38:53.384 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/35 kB Progress (1): 33/35 kB Progress (1): 35 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-parent/11/httpcomponents-parent-11.pom (35 kB at 1.4 MB/s) 2024-03-09T14:38:53.392 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/apache/21/apache-21.pom 2024-03-09T14:38:53.446 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/17 kB Progress (1): 17 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/apache/21/apache-21.pom (17 kB at 311 kB/s) 2024-03-09T14:38:53.455 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.13/httpcore-4.4.13.pom 2024-03-09T14:38:53.475 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.13/httpcore-4.4.13.pom (5.0 kB at 237 kB/s) 2024-03-09T14:38:53.480 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-core/4.4.13/httpcomponents-core-4.4.13.pom 2024-03-09T14:38:53.525 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 13 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-core/4.4.13/httpcomponents-core-4.4.13.pom (13 kB at 293 kB/s) 2024-03-09T14:38:53.534 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.11/commons-codec-1.11.pom 2024-03-09T14:38:53.573 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 14 kB Downloaded from central: https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.11/commons-codec-1.11.pom (14 kB at 368 kB/s) 2024-03-09T14:38:53.578 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/42/commons-parent-42.pom 2024-03-09T14:38:53.625 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/68 kB Progress (1): 33/68 kB Progress (1): 49/68 kB Progress (1): 66/68 kB Progress (1): 68 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/42/commons-parent-42.pom (68 kB at 1.5 MB/s) 2024-03-09T14:38:53.650 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom 2024-03-09T14:38:53.690 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom (16 kB at 392 kB/s) 2024-03-09T14:38:53.701 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.15/commons-codec-1.15.pom 2024-03-09T14:38:53.755 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 15 kB Downloaded from central: https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.15/commons-codec-1.15.pom (15 kB at 281 kB/s) 2024-03-09T14:38:53.760 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/52/commons-parent-52.pom 2024-03-09T14:38:53.814 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/79 kB Progress (1): 33/79 kB Progress (1): 49/79 kB Progress (1): 65/79 kB Progress (1): 79 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/52/commons-parent-52.pom (79 kB at 1.5 MB/s) 2024-03-09T14:38:53.824 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/apache/23/apache-23.pom 2024-03-09T14:38:53.885 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/18 kB Progress (1): 18 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/apache/23/apache-23.pom (18 kB at 307 kB/s) 2024-03-09T14:38:53.894 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.15.1/commons-io-2.15.1.pom 2024-03-09T14:38:53.922 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/20 kB Progress (1): 20 kB Downloaded from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.15.1/commons-io-2.15.1.pom (20 kB at 689 kB/s) 2024-03-09T14:38:53.928 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/65/commons-parent-65.pom 2024-03-09T14:38:53.967 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/78 kB Progress (1): 33/78 kB Progress (1): 49/78 kB Progress (1): 66/78 kB Progress (1): 78 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/65/commons-parent-65.pom (78 kB at 2.0 MB/s) 2024-03-09T14:38:53.981 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.28/slf4j-api-1.7.28.pom 2024-03-09T14:38:54.021 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.28/slf4j-api-1.7.28.pom (3.8 kB at 98 kB/s) 2024-03-09T14:38:54.025 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.28/slf4j-parent-1.7.28.pom 2024-03-09T14:38:54.047 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 14 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.28/slf4j-parent-1.7.28.pom (14 kB at 657 kB/s) 2024-03-09T14:38:54.053 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter/5.10.1/junit-jupiter-5.10.1.pom 2024-03-09T14:38:54.090 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter/5.10.1/junit-jupiter-5.10.1.pom (3.2 kB at 87 kB/s) 2024-03-09T14:38:54.096 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter-api/5.10.1/junit-jupiter-api-5.10.1.pom 2024-03-09T14:38:54.142 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter-api/5.10.1/junit-jupiter-api-5.10.1.pom (3.2 kB at 68 kB/s) 2024-03-09T14:38:54.147 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/opentest4j/opentest4j/1.3.0/opentest4j-1.3.0.pom 2024-03-09T14:38:54.210 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/opentest4j/opentest4j/1.3.0/opentest4j-1.3.0.pom (2.0 kB at 32 kB/s) 2024-03-09T14:38:54.217 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-commons/1.10.1/junit-platform-commons-1.10.1.pom 2024-03-09T14:38:54.237 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-commons/1.10.1/junit-platform-commons-1.10.1.pom (2.8 kB at 129 kB/s) 2024-03-09T14:38:54.243 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apiguardian/apiguardian-api/1.1.2/apiguardian-api-1.1.2.pom 2024-03-09T14:38:54.273 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.5 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apiguardian/apiguardian-api/1.1.2/apiguardian-api-1.1.2.pom (1.5 kB at 49 kB/s) 2024-03-09T14:38:54.279 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter-params/5.10.1/junit-jupiter-params-5.10.1.pom 2024-03-09T14:38:54.321 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter-params/5.10.1/junit-jupiter-params-5.10.1.pom (3.0 kB at 70 kB/s) 2024-03-09T14:38:54.327 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter-engine/5.10.1/junit-jupiter-engine-5.10.1.pom 2024-03-09T14:38:54.371 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter-engine/5.10.1/junit-jupiter-engine-5.10.1.pom (3.2 kB at 71 kB/s) 2024-03-09T14:38:54.376 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-engine/1.10.1/junit-platform-engine-1.10.1.pom 2024-03-09T14:38:54.441 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-engine/1.10.1/junit-platform-engine-1.10.1.pom (3.2 kB at 49 kB/s) 2024-03-09T14:38:54.448 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-simple/1.7.28/slf4j-simple-1.7.28.pom 2024-03-09T14:38:54.496 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-simple/1.7.28/slf4j-simple-1.7.28.pom (1.0 kB at 21 kB/s) 2024-03-09T14:38:54.506 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.7.28/jcl-over-slf4j-1.7.28.pom 2024-03-09T14:38:54.536 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 964 B Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.7.28/jcl-over-slf4j-1.7.28.pom (964 B at 31 kB/s) 2024-03-09T14:38:54.543 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite/1.10.1/junit-platform-suite-1.10.1.pom 2024-03-09T14:38:54.563 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite/1.10.1/junit-platform-suite-1.10.1.pom (3.1 kB at 145 kB/s) 2024-03-09T14:38:54.569 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite-api/1.10.1/junit-platform-suite-api-1.10.1.pom 2024-03-09T14:38:54.588 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite-api/1.10.1/junit-platform-suite-api-1.10.1.pom (3.0 kB at 152 kB/s) 2024-03-09T14:38:54.594 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite-engine/1.10.1/junit-platform-suite-engine-1.10.1.pom 2024-03-09T14:38:54.616 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite-engine/1.10.1/junit-platform-suite-engine-1.10.1.pom (3.4 kB at 149 kB/s) 2024-03-09T14:38:54.621 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite-commons/1.10.1/junit-platform-suite-commons-1.10.1.pom 2024-03-09T14:38:54.653 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite-commons/1.10.1/junit-platform-suite-commons-1.10.1.pom (3.4 kB at 107 kB/s) 2024-03-09T14:38:54.659 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-launcher/1.10.1/junit-platform-launcher-1.10.1.pom 2024-03-09T14:38:54.682 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-launcher/1.10.1/junit-platform-launcher-1.10.1.pom (3.0 kB at 126 kB/s) 2024-03-09T14:38:54.731 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/s3/2.20.43/s3-2.20.43.jar 2024-03-09T14:38:55.105 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 0/3.3 MB Progress (1): 0/3.3 MB Progress (1): 0/3.3 MB Progress (1): 0.1/3.3 MB Progress (1): 0.1/3.3 MB Progress (1): 0.1/3.3 MB Progress (1): 0.1/3.3 MB Progress (1): 0.1/3.3 MB Progress (1): 0.1/3.3 MB Progress (1): 0.2/3.3 MB Progress (1): 0.2/3.3 MB Progress (1): 0.2/3.3 MB Progress (1): 0.2/3.3 MB Progress (1): 0.2/3.3 MB Progress (1): 0.2/3.3 MB Progress (1): 0.3/3.3 MB Progress (1): 0.3/3.3 MB Progress (1): 0.3/3.3 MB Progress (1): 0.3/3.3 MB Progress (1): 0.3/3.3 MB Progress (1): 0.3/3.3 MB Progress (1): 0.4/3.3 MB Progress (1): 0.4/3.3 MB Progress (1): 0.4/3.3 MB Progress (1): 0.4/3.3 MB Progress (1): 0.4/3.3 MB Progress (1): 0.4/3.3 MB Progress (1): 0.5/3.3 MB Progress (1): 0.5/3.3 MB Progress (1): 0.5/3.3 MB Progress (1): 0.5/3.3 MB Progress (1): 0.5/3.3 MB Progress (1): 0.5/3.3 MB Progress (1): 0.6/3.3 MB Progress (1): 0.6/3.3 MB Progress (1): 0.6/3.3 MB Progress (1): 0.6/3.3 MB Progress (1): 0.6/3.3 MB Progress (1): 0.6/3.3 MB Progress (1): 0.7/3.3 MB Progress (1): 0.7/3.3 MB Progress (1): 0.7/3.3 MB Progress (1): 0.7/3.3 MB Progress (1): 0.7/3.3 MB Progress (1): 0.7/3.3 MB Progress (1): 0.7/3.3 MB Progress (1): 0.8/3.3 MB Progress (1): 0.8/3.3 MB Progress (1): 0.8/3.3 MB Progress (1): 0.8/3.3 MB Progress (1): 0.8/3.3 MB Progress (1): 0.8/3.3 MB Progress (1): 0.9/3.3 MB Progress (1): 0.9/3.3 MB Progress (1): 0.9/3.3 MB Progress (1): 0.9/3.3 MB Progress (1): 0.9/3.3 MB Progress (1): 0.9/3.3 MB Progress (1): 1.0/3.3 MB Progress (1): 1.0/3.3 MB Progress (1): 1.0/3.3 MB Progress (1): 1.0/3.3 MB Progress (1): 1.0/3.3 MB Progress (1): 1.0/3.3 MB Progress (1): 1.1/3.3 MB Progress (1): 1.1/3.3 MB Progress (1): 1.1/3.3 MB Progress (1): 1.1/3.3 MB Progress (1): 1.1/3.3 MB Progress (1): 1.1/3.3 MB Progress (1): 1.2/3.3 MB Progress (1): 1.2/3.3 MB Progress (1): 1.2/3.3 MB Progress (1): 1.2/3.3 MB Progress (1): 1.2/3.3 MB Progress (1): 1.2/3.3 MB Progress (1): 1.3/3.3 MB Progress (1): 1.3/3.3 MB Progress (1): 1.3/3.3 MB Progress (1): 1.3/3.3 MB Progress (1): 1.3/3.3 MB Progress (1): 1.3/3.3 MB Progress (1): 1.3/3.3 MB Progress (1): 1.4/3.3 MB Progress (1): 1.4/3.3 MB Progress (1): 1.4/3.3 MB Progress (1): 1.4/3.3 MB Progress (1): 1.4/3.3 MB Progress (1): 1.4/3.3 MB Progress (1): 1.5/3.3 MB Progress (1): 1.5/3.3 MB Progress (1): 1.5/3.3 MB Progress (1): 1.5/3.3 MB Progress (1): 1.5/3.3 MB Progress (1): 1.5/3.3 MB Progress (1): 1.6/3.3 MB Progress (1): 1.6/3.3 MB Progress (1): 1.6/3.3 MB Progress (1): 1.6/3.3 MB Progress (1): 1.6/3.3 MB Progress (1): 1.6/3.3 MB Progress (1): 1.7/3.3 MB Progress (1): 1.7/3.3 MB Progress (1): 1.7/3.3 MB Progress (1): 1.7/3.3 MB Progress (1): 1.7/3.3 MB Progress (1): 1.7/3.3 MB Progress (1): 1.8/3.3 MB Progress (1): 1.8/3.3 MB Progress (1): 1.8/3.3 MB Progress (1): 1.8/3.3 MB Progress (1): 1.8/3.3 MB Progress (1): 1.8/3.3 MB Progress (1): 1.9/3.3 MB Progress (1): 1.9/3.3 MB Progress (1): 1.9/3.3 MB Progress (1): 1.9/3.3 MB Progress (1): 1.9/3.3 MB Progress (1): 1.9/3.3 MB Progress (1): 2.0/3.3 MB Progress (1): 2.0/3.3 MB Progress (1): 2.0/3.3 MB Progress (1): 2.0/3.3 MB Progress (1): 2.0/3.3 MB Progress (1): 2.0/3.3 MB Progress (1): 2.0/3.3 MB Progress (1): 2.1/3.3 MB Progress (1): 2.1/3.3 MB Progress (1): 2.1/3.3 MB Progress (1): 2.1/3.3 MB Progress (1): 2.1/3.3 MB Progress (1): 2.1/3.3 MB Progress (1): 2.2/3.3 MB Progress (1): 2.2/3.3 MB Progress (1): 2.2/3.3 MB Progress (1): 2.2/3.3 MB Progress (1): 2.2/3.3 MB Progress (1): 2.2/3.3 MB Progress (1): 2.3/3.3 MB Progress (1): 2.3/3.3 MB Progress (1): 2.3/3.3 MB Progress (1): 2.3/3.3 MB Progress (1): 2.3/3.3 MB Progress (1): 2.3/3.3 MB Progress (1): 2.4/3.3 MB Progress (1): 2.4/3.3 MB Progress (1): 2.4/3.3 MB Progress (1): 2.4/3.3 MB Progress (1): 2.4/3.3 MB Progress (1): 2.4/3.3 MB Progress (1): 2.5/3.3 MB Progress (1): 2.5/3.3 MB Progress (1): 2.5/3.3 MB Progress (1): 2.5/3.3 MB Progress (1): 2.5/3.3 MB Progress (1): 2.5/3.3 MB Progress (1): 2.6/3.3 MB Progress (1): 2.6/3.3 MB Progress (1): 2.6/3.3 MB Progress (1): 2.6/3.3 MB Progress (1): 2.6/3.3 MB Progress (1): 2.6/3.3 MB Progress (1): 2.7/3.3 MB Progress (1): 2.7/3.3 MB Progress (1): 2.7/3.3 MB Progress (1): 2.7/3.3 MB Progress (1): 2.7/3.3 MB Progress (1): 2.7/3.3 MB Progress (1): 2.7/3.3 MB Progress (1): 2.8/3.3 MB Progress (1): 2.8/3.3 MB Progress (1): 2.8/3.3 MB Progress (1): 2.8/3.3 MB Progress (1): 2.8/3.3 MB Progress (1): 2.8/3.3 MB Progress (1): 2.9/3.3 MB Progress (1): 2.9/3.3 MB Progress (1): 2.9/3.3 MB Progress (1): 2.9/3.3 MB Progress (1): 2.9/3.3 MB Progress (1): 2.9/3.3 MB Progress (1): 3.0/3.3 MB Progress (1): 3.0/3.3 MB Progress (1): 3.0/3.3 MB Progress (1): 3.0/3.3 MB Progress (1): 3.0/3.3 MB Progress (1): 3.0/3.3 MB Progress (1): 3.1/3.3 MB Progress (1): 3.1/3.3 MB Progress (1): 3.1/3.3 MB Progress (1): 3.1/3.3 MB Progress (1): 3.1/3.3 MB Progress (1): 3.1/3.3 MB Progress (1): 3.2/3.3 MB Progress (1): 3.2/3.3 MB Progress (1): 3.2/3.3 MB Progress (1): 3.2/3.3 MB Progress (1): 3.2/3.3 MB Progress (1): 3.2/3.3 MB Progress (1): 3.3/3.3 MB Progress (1): 3.3/3.3 MB Progress (1): 3.3/3.3 MB Progress (1): 3.3 MB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/s3/2.20.43/s3-2.20.43.jar (3.3 MB at 8.8 MB/s) 2024-03-09T14:38:55.109 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-xml-protocol/2.20.43/aws-xml-protocol-2.20.43.jar 2024-03-09T14:38:55.109 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-query-protocol/2.20.43/aws-query-protocol-2.20.43.jar 2024-03-09T14:38:55.110 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/protocol-core/2.20.43/protocol-core-2.20.43.jar 2024-03-09T14:38:55.111 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/arns/2.20.43/arns-2.20.43.jar 2024-03-09T14:38:55.111 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/profiles/2.20.43/profiles-2.20.43.jar 2024-03-09T14:38:55.138 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/98 kB Progress (1): 33/98 kB Progress (1): 49/98 kB Progress (1): 66/98 kB Progress (1): 82/98 kB Progress (1): 98 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-xml-protocol/2.20.43/aws-xml-protocol-2.20.43.jar (98 kB at 3.0 MB/s) 2024-03-09T14:38:55.139 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/crt-core/2.20.43/crt-core-2.20.43.jar 2024-03-09T14:38:55.164 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/crt-core/2.20.43/crt-core-2.20.43.jar (16 kB at 303 kB/s) 2024-03-09T14:38:55.164 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/sdk-core/2.20.43/sdk-core-2.20.43.jar 2024-03-09T14:38:55.189 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.4/35 kB Progress (1): 2.8/35 kB Progress (2): 2.8/35 kB | 16/71 kB Progress (2): 4.1/35 kB | 16/71 kB Progress (2): 5.5/35 kB | 16/71 kB Progress (2): 6.9/35 kB | 16/71 kB Progress (2): 8.3/35 kB | 16/71 kB Progress (2): 9.7/35 kB | 16/71 kB Progress (2): 11/35 kB | 16/71 kB Progress (2): 12/35 kB | 16/71 kB Progress (2): 14/35 kB | 16/71 kB Progress (2): 15/35 kB | 16/71 kB Progress (2): 17/35 kB | 16/71 kB Progress (2): 18/35 kB | 16/71 kB Progress (2): 19/35 kB | 16/71 kB Progress (2): 21/35 kB | 16/71 kB Progress (2): 22/35 kB | 16/71 kB Progress (2): 23/35 kB | 16/71 kB Progress (2): 25/35 kB | 16/71 kB Progress (2): 26/35 kB | 16/71 kB Progress (2): 28/35 kB | 16/71 kB Progress (2): 29/35 kB | 16/71 kB Progress (2): 29/35 kB | 33/71 kB Progress (2): 30/35 kB | 33/71 kB Progress (2): 32/35 kB | 33/71 kB Progress (2): 33/35 kB | 33/71 kB Progress (2): 34/35 kB | 33/71 kB Progress (2): 35 kB | 33/71 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/protocol-core/2.20.43/protocol-core-2.20.43.jar (35 kB at 440 kB/s) 2024-03-09T14:38:55.189 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/reactivestreams/reactive-streams/1.0.3/reactive-streams-1.0.3.jar 2024-03-09T14:38:55.213 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 33/71 kB | 16/782 kB Progress (2): 33/71 kB | 32/782 kB Progress (2): 33/71 kB | 49/782 kB Progress (2): 33/71 kB | 65/782 kB Progress (2): 33/71 kB | 81/782 kB Progress (2): 33/71 kB | 98/782 kB Progress (2): 33/71 kB | 114/782 kB Progress (2): 33/71 kB | 130/782 kB Progress (2): 33/71 kB | 147/782 kB Progress (2): 33/71 kB | 163/782 kB Progress (2): 33/71 kB | 180/782 kB Progress (2): 33/71 kB | 196/782 kB Progress (2): 33/71 kB | 212/782 kB Progress (2): 33/71 kB | 229/782 kB Progress (2): 33/71 kB | 245/782 kB Progress (2): 49/71 kB | 245/782 kB Progress (2): 49/71 kB | 262/782 kB Progress (2): 49/71 kB | 278/782 kB Progress (3): 49/71 kB | 278/782 kB | 1.4/16 kB Progress (3): 49/71 kB | 278/782 kB | 2.8/16 kB Progress (3): 49/71 kB | 278/782 kB | 4.1/16 kB Progress (3): 49/71 kB | 294/782 kB | 4.1/16 kB Progress (3): 49/71 kB | 294/782 kB | 5.5/16 kB Progress (3): 49/71 kB | 294/782 kB | 6.9/16 kB Progress (3): 49/71 kB | 294/782 kB | 8.3/16 kB Progress (4): 49/71 kB | 294/782 kB | 8.3/16 kB | 1.4/49 kB Progress (4): 49/71 kB | 294/782 kB | 9.7/16 kB | 1.4/49 kB Progress (4): 49/71 kB | 294/782 kB | 9.7/16 kB | 2.8/49 kB Progress (4): 49/71 kB | 294/782 kB | 11/16 kB | 2.8/49 kB Progress (4): 49/71 kB | 294/782 kB | 11/16 kB | 4.1/49 kB Progress (4): 49/71 kB | 294/782 kB | 12/16 kB | 4.1/49 kB Progress (4): 49/71 kB | 294/782 kB | 12/16 kB | 5.5/49 kB Progress (4): 49/71 kB | 311/782 kB | 12/16 kB | 5.5/49 kB Progress (4): 49/71 kB | 311/782 kB | 14/16 kB | 5.5/49 kB Progress (4): 49/71 kB | 311/782 kB | 14/16 kB | 6.9/49 kB Progress (4): 49/71 kB | 311/782 kB | 15/16 kB | 6.9/49 kB Progress (4): 49/71 kB | 311/782 kB | 15/16 kB | 8.3/49 kB Progress (4): 49/71 kB | 311/782 kB | 16 kB | 8.3/49 kB Progress (4): 66/71 kB | 311/782 kB | 16 kB | 8.3/49 kB Progress (4): 66/71 kB | 311/782 kB | 16 kB | 9.7/49 kB Progress (4): 71 kB | 311/782 kB | 16 kB | 9.7/49 kB Progress (4): 71 kB | 311/782 kB | 16 kB | 11/49 kB Progress (4): 71 kB | 311/782 kB | 16 kB | 12/49 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/arns/2.20.43/arns-2.20.43.jar (16 kB at 153 kB/s) 2024-03-09T14:38:55.213 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 71 kB | 311/782 kB | 14/49 kB Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/auth/2.20.43/auth-2.20.43.jar 2024-03-09T14:38:55.215 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 71 kB | 311/782 kB | 15/49 kB Progress (3): 71 kB | 311/782 kB | 16/49 kB Progress (3): 71 kB | 311/782 kB | 17/49 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-query-protocol/2.20.43/aws-query-protocol-2.20.43.jar (71 kB at 665 kB/s) 2024-03-09T14:38:55.215 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 311/782 kB | 19/49 kB Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/eventstream/eventstream/1.0.1/eventstream-1.0.1.jar 2024-03-09T14:38:55.225 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 311/782 kB | 20/49 kB Progress (2): 311/782 kB | 21/49 kB Progress (2): 311/782 kB | 23/49 kB Progress (2): 311/782 kB | 24/49 kB Progress (2): 311/782 kB | 25/49 kB Progress (2): 311/782 kB | 27/49 kB Progress (2): 311/782 kB | 28/49 kB Progress (2): 311/782 kB | 30/49 kB Progress (2): 311/782 kB | 31/49 kB Progress (2): 311/782 kB | 32/49 kB Progress (2): 311/782 kB | 34/49 kB Progress (2): 311/782 kB | 35/49 kB Progress (2): 311/782 kB | 36/49 kB Progress (2): 327/782 kB | 36/49 kB Progress (2): 327/782 kB | 38/49 kB Progress (2): 327/782 kB | 39/49 kB Progress (2): 343/782 kB | 39/49 kB Progress (3): 343/782 kB | 39/49 kB | 1.4/11 kB Progress (3): 343/782 kB | 39/49 kB | 2.8/11 kB Progress (3): 360/782 kB | 39/49 kB | 2.8/11 kB Progress (3): 360/782 kB | 39/49 kB | 4.1/11 kB Progress (3): 360/782 kB | 39/49 kB | 5.5/11 kB Progress (3): 360/782 kB | 39/49 kB | 6.9/11 kB Progress (3): 360/782 kB | 39/49 kB | 8.3/11 kB Progress (3): 360/782 kB | 39/49 kB | 9.7/11 kB Progress (3): 360/782 kB | 39/49 kB | 11/11 kB Progress (3): 360/782 kB | 39/49 kB | 11 kB Progress (3): 376/782 kB | 39/49 kB | 11 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/reactivestreams/reactive-streams/1.0.3/reactive-streams-1.0.3.jar (11 kB at 101 kB/s) 2024-03-09T14:38:55.225 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/http-client-spi/2.20.43/http-client-spi-2.20.43.jar 2024-03-09T14:38:55.248 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 393/782 kB | 39/49 kB Progress (2): 402/782 kB | 39/49 kB Progress (2): 418/782 kB | 39/49 kB Progress (2): 435/782 kB | 39/49 kB Progress (2): 451/782 kB | 39/49 kB Progress (2): 468/782 kB | 39/49 kB Progress (2): 484/782 kB | 39/49 kB Progress (2): 500/782 kB | 39/49 kB Progress (2): 517/782 kB | 39/49 kB Progress (2): 533/782 kB | 39/49 kB Progress (2): 549/782 kB | 39/49 kB Progress (2): 566/782 kB | 39/49 kB Progress (2): 582/782 kB | 39/49 kB Progress (2): 599/782 kB | 39/49 kB Progress (2): 615/782 kB | 39/49 kB Progress (2): 615/782 kB | 41/49 kB Progress (2): 631/782 kB | 41/49 kB Progress (2): 631/782 kB | 42/49 kB Progress (2): 631/782 kB | 43/49 kB Progress (2): 631/782 kB | 45/49 kB Progress (2): 631/782 kB | 46/49 kB Progress (2): 631/782 kB | 48/49 kB Progress (2): 631/782 kB | 49/49 kB Progress (2): 648/782 kB | 49/49 kB Progress (2): 648/782 kB | 49 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/profiles/2.20.43/profiles-2.20.43.jar (49 kB at 362 kB/s) 2024-03-09T14:38:55.248 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/regions/2.20.43/regions-2.20.43.jar 2024-03-09T14:38:55.257 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 648/782 kB | 1.4/30 kB Progress (2): 648/782 kB | 2.8/30 kB Progress (2): 664/782 kB | 2.8/30 kB Progress (3): 664/782 kB | 2.8/30 kB | 16/205 kB Progress (3): 664/782 kB | 4.1/30 kB | 16/205 kB Progress (3): 664/782 kB | 5.5/30 kB | 16/205 kB Progress (3): 681/782 kB | 5.5/30 kB | 16/205 kB Progress (3): 681/782 kB | 6.9/30 kB | 16/205 kB Progress (3): 681/782 kB | 6.9/30 kB | 33/205 kB Progress (3): 681/782 kB | 8.3/30 kB | 33/205 kB Progress (3): 697/782 kB | 8.3/30 kB | 33/205 kB Progress (3): 697/782 kB | 9.7/30 kB | 33/205 kB Progress (3): 697/782 kB | 11/30 kB | 33/205 kB Progress (3): 697/782 kB | 12/30 kB | 33/205 kB Progress (3): 697/782 kB | 12/30 kB | 49/205 kB Progress (3): 713/782 kB | 12/30 kB | 49/205 kB Progress (3): 713/782 kB | 14/30 kB | 49/205 kB Progress (3): 713/782 kB | 15/30 kB | 49/205 kB Progress (3): 730/782 kB | 15/30 kB | 49/205 kB Progress (3): 730/782 kB | 17/30 kB | 49/205 kB Progress (3): 730/782 kB | 18/30 kB | 49/205 kB Progress (3): 746/782 kB | 18/30 kB | 49/205 kB Progress (3): 746/782 kB | 19/30 kB | 49/205 kB Progress (3): 746/782 kB | 21/30 kB | 49/205 kB Progress (3): 762/782 kB | 21/30 kB | 49/205 kB Progress (3): 762/782 kB | 22/30 kB | 49/205 kB Progress (3): 762/782 kB | 22/30 kB | 66/205 kB Progress (3): 762/782 kB | 23/30 kB | 66/205 kB Progress (4): 762/782 kB | 23/30 kB | 66/205 kB | 16/81 kB Progress (4): 779/782 kB | 23/30 kB | 66/205 kB | 16/81 kB Progress (4): 779/782 kB | 25/30 kB | 66/205 kB | 16/81 kB Progress (4): 782 kB | 25/30 kB | 66/205 kB | 16/81 kB Progress (4): 782 kB | 26/30 kB | 66/205 kB | 16/81 kB Progress (4): 782 kB | 26/30 kB | 82/205 kB | 16/81 kB Progress (4): 782 kB | 28/30 kB | 82/205 kB | 16/81 kB Progress (4): 782 kB | 28/30 kB | 82/205 kB | 33/81 kB Progress (4): 782 kB | 29/30 kB | 82/205 kB | 33/81 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/sdk-core/2.20.43/sdk-core-2.20.43.jar (782 kB at 5.4 MB/s) 2024-03-09T14:38:55.257 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 30 kB | 82/205 kB | 33/81 kB Progress (3): 30 kB | 98/205 kB | 33/81 kB Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/annotations/2.20.43/annotations-2.20.43.jar 2024-03-09T14:38:55.258 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 30 kB | 98/205 kB | 49/81 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/eventstream/eventstream/1.0.1/eventstream-1.0.1.jar (30 kB at 208 kB/s) 2024-03-09T14:38:55.258 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/utils/2.20.43/utils-2.20.43.jar 2024-03-09T14:38:55.273 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 115/205 kB | 49/81 kB Progress (2): 115/205 kB | 66/81 kB Progress (2): 131/205 kB | 66/81 kB Progress (2): 131/205 kB | 81 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/http-client-spi/2.20.43/http-client-spi-2.20.43.jar (81 kB at 511 kB/s) 2024-03-09T14:38:55.273 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 147/205 kB Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-core/2.20.43/aws-core-2.20.43.jar 2024-03-09T14:38:55.277 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 147/205 kB | 1.4/13 kB Progress (2): 147/205 kB | 2.8/13 kB Progress (2): 147/205 kB | 4.1/13 kB Progress (2): 147/205 kB | 5.5/13 kB Progress (2): 147/205 kB | 6.9/13 kB Progress (2): 147/205 kB | 8.3/13 kB Progress (2): 164/205 kB | 8.3/13 kB Progress (2): 164/205 kB | 9.7/13 kB Progress (2): 164/205 kB | 11/13 kB Progress (2): 164/205 kB | 12/13 kB Progress (2): 164/205 kB | 13 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/annotations/2.20.43/annotations-2.20.43.jar (13 kB at 81 kB/s) 2024-03-09T14:38:55.277 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/metrics-spi/2.20.43/metrics-spi-2.20.43.jar 2024-03-09T14:38:55.283 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 164/205 kB | 16/182 kB Progress (2): 164/205 kB | 33/182 kB Progress (2): 180/205 kB | 33/182 kB Progress (2): 180/205 kB | 49/182 kB Progress (2): 180/205 kB | 66/182 kB Progress (2): 180/205 kB | 82/182 kB Progress (2): 197/205 kB | 82/182 kB Progress (2): 205 kB | 82/182 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/auth/2.20.43/auth-2.20.43.jar (205 kB at 1.2 MB/s) 2024-03-09T14:38:55.283 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/json-utils/2.20.43/json-utils-2.20.43.jar 2024-03-09T14:38:55.289 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 98/182 kB Progress (2): 98/182 kB | 16/798 kB Progress (2): 115/182 kB | 16/798 kB Progress (2): 131/182 kB | 16/798 kB Progress (2): 147/182 kB | 16/798 kB Progress (2): 164/182 kB | 16/798 kB Progress (2): 180/182 kB | 16/798 kB Progress (2): 182 kB | 16/798 kB Progress (2): 182 kB | 33/798 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/utils/2.20.43/utils-2.20.43.jar (182 kB at 1.0 MB/s) 2024-03-09T14:38:55.289 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/third-party-jackson-core/2.20.43/third-party-jackson-core-2.20.43.jar 2024-03-09T14:38:55.299 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 33/798 kB | 16/151 kB Progress (2): 33/798 kB | 32/151 kB Progress (2): 33/798 kB | 49/151 kB Progress (2): 33/798 kB | 65/151 kB Progress (2): 49/798 kB | 65/151 kB Progress (2): 49/798 kB | 81/151 kB Progress (3): 49/798 kB | 81/151 kB | 16/27 kB Progress (3): 49/798 kB | 98/151 kB | 16/27 kB Progress (3): 49/798 kB | 98/151 kB | 27 kB Progress (3): 49/798 kB | 114/151 kB | 27 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/metrics-spi/2.20.43/metrics-spi-2.20.43.jar (27 kB at 148 kB/s) 2024-03-09T14:38:55.299 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 49/798 kB | 130/151 kB Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/endpoints-spi/2.20.43/endpoints-spi-2.20.43.jar 2024-03-09T14:38:55.303 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 66/798 kB | 130/151 kB Progress (3): 66/798 kB | 130/151 kB | 16/30 kB Progress (3): 66/798 kB | 130/151 kB | 30 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/json-utils/2.20.43/json-utils-2.20.43.jar (30 kB at 160 kB/s) 2024-03-09T14:38:55.303 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/apache-client/2.20.43/apache-client-2.20.43.jar 2024-03-09T14:38:55.307 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 82/798 kB | 130/151 kB Progress (2): 82/798 kB | 147/151 kB Progress (2): 82/798 kB | 151 kB Progress (2): 98/798 kB | 151 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/aws-core/2.20.43/aws-core-2.20.43.jar (151 kB at 777 kB/s) 2024-03-09T14:38:55.307 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5.13/httpclient-4.5.13.jar 2024-03-09T14:38:55.320 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 115/798 kB Progress (2): 115/798 kB | 16/385 kB Progress (2): 115/798 kB | 32/385 kB Progress (2): 115/798 kB | 49/385 kB Progress (2): 115/798 kB | 65/385 kB Progress (2): 131/798 kB | 65/385 kB Progress (2): 131/798 kB | 81/385 kB Progress (2): 131/798 kB | 98/385 kB Progress (2): 131/798 kB | 114/385 kB Progress (2): 131/798 kB | 130/385 kB Progress (2): 131/798 kB | 147/385 kB Progress (2): 131/798 kB | 163/385 kB Progress (2): 147/798 kB | 163/385 kB Progress (3): 147/798 kB | 163/385 kB | 13 kB Progress (3): 147/798 kB | 180/385 kB | 13 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/endpoints-spi/2.20.43/endpoints-spi-2.20.43.jar (13 kB at 64 kB/s) 2024-03-09T14:38:55.320 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 147/798 kB | 196/385 kB Downloading from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.13/httpcore-4.4.13.jar 2024-03-09T14:38:55.341 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 147/798 kB | 196/385 kB | 16/75 kB Progress (3): 147/798 kB | 212/385 kB | 16/75 kB Progress (3): 147/798 kB | 229/385 kB | 16/75 kB Progress (3): 164/798 kB | 229/385 kB | 16/75 kB Progress (3): 164/798 kB | 229/385 kB | 33/75 kB Progress (3): 164/798 kB | 245/385 kB | 33/75 kB Progress (3): 164/798 kB | 245/385 kB | 49/75 kB Progress (3): 180/798 kB | 245/385 kB | 49/75 kB Progress (3): 180/798 kB | 245/385 kB | 66/75 kB Progress (3): 180/798 kB | 262/385 kB | 66/75 kB Progress (3): 180/798 kB | 262/385 kB | 75 kB Progress (3): 180/798 kB | 278/385 kB | 75 kB Progress (3): 197/798 kB | 278/385 kB | 75 kB Progress (4): 197/798 kB | 278/385 kB | 75 kB | 16/780 kB Progress (4): 197/798 kB | 294/385 kB | 75 kB | 16/780 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/apache-client/2.20.43/apache-client-2.20.43.jar (75 kB at 331 kB/s) 2024-03-09T14:38:55.342 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.15.1/commons-io-2.15.1.jar 2024-03-09T14:38:55.343 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 197/798 kB | 294/385 kB | 33/780 kB Progress (3): 197/798 kB | 311/385 kB | 33/780 kB Progress (3): 213/798 kB | 311/385 kB | 33/780 kB Progress (3): 213/798 kB | 311/385 kB | 49/780 kB Progress (3): 213/798 kB | 327/385 kB | 49/780 kB Progress (3): 213/798 kB | 327/385 kB | 66/780 kB Progress (3): 213/798 kB | 343/385 kB | 66/780 kB Progress (3): 213/798 kB | 343/385 kB | 82/780 kB Progress (4): 213/798 kB | 343/385 kB | 82/780 kB | 16/329 kB Progress (4): 229/798 kB | 343/385 kB | 82/780 kB | 16/329 kB Progress (4): 229/798 kB | 360/385 kB | 82/780 kB | 16/329 kB Progress (4): 229/798 kB | 360/385 kB | 98/780 kB | 16/329 kB Progress (4): 229/798 kB | 376/385 kB | 98/780 kB | 16/329 kB Progress (4): 229/798 kB | 376/385 kB | 115/780 kB | 16/329 kB Progress (4): 229/798 kB | 385 kB | 115/780 kB | 16/329 kB Progress (4): 246/798 kB | 385 kB | 115/780 kB | 16/329 kB Progress (4): 246/798 kB | 385 kB | 131/780 kB | 16/329 kB Progress (4): 246/798 kB | 385 kB | 147/780 kB | 16/329 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/third-party-jackson-core/2.20.43/third-party-jackson-core-2.20.43.jar (385 kB at 1.7 MB/s) 2024-03-09T14:38:55.343 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 246/798 kB | 147/780 kB | 32/329 kB Progress (3): 256/798 kB | 147/780 kB | 32/329 kB Downloading from central: https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.15/commons-codec-1.15.jar 2024-03-09T14:38:55.370 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 256/798 kB | 164/780 kB | 32/329 kB Progress (3): 256/798 kB | 180/780 kB | 32/329 kB Progress (3): 272/798 kB | 180/780 kB | 32/329 kB Progress (3): 272/798 kB | 197/780 kB | 32/329 kB Progress (3): 272/798 kB | 213/780 kB | 32/329 kB Progress (3): 289/798 kB | 213/780 kB | 32/329 kB Progress (3): 289/798 kB | 229/780 kB | 32/329 kB Progress (3): 289/798 kB | 229/780 kB | 49/329 kB Progress (3): 289/798 kB | 246/780 kB | 49/329 kB Progress (3): 289/798 kB | 256/780 kB | 49/329 kB Progress (3): 289/798 kB | 272/780 kB | 49/329 kB Progress (3): 289/798 kB | 289/780 kB | 49/329 kB Progress (3): 289/798 kB | 305/780 kB | 49/329 kB Progress (3): 289/798 kB | 305/780 kB | 65/329 kB Progress (3): 289/798 kB | 321/780 kB | 65/329 kB Progress (3): 289/798 kB | 338/780 kB | 65/329 kB Progress (3): 289/798 kB | 354/780 kB | 65/329 kB Progress (3): 289/798 kB | 370/780 kB | 65/329 kB Progress (3): 289/798 kB | 387/780 kB | 65/329 kB Progress (3): 289/798 kB | 403/780 kB | 65/329 kB Progress (3): 289/798 kB | 420/780 kB | 65/329 kB Progress (3): 289/798 kB | 436/780 kB | 65/329 kB Progress (3): 289/798 kB | 452/780 kB | 65/329 kB Progress (3): 289/798 kB | 469/780 kB | 65/329 kB Progress (3): 289/798 kB | 485/780 kB | 65/329 kB Progress (3): 289/798 kB | 502/780 kB | 65/329 kB Progress (3): 289/798 kB | 518/780 kB | 65/329 kB Progress (3): 289/798 kB | 534/780 kB | 65/329 kB Progress (3): 289/798 kB | 551/780 kB | 65/329 kB Progress (3): 289/798 kB | 567/780 kB | 65/329 kB Progress (3): 289/798 kB | 567/780 kB | 81/329 kB Progress (3): 305/798 kB | 567/780 kB | 81/329 kB Progress (3): 305/798 kB | 567/780 kB | 98/329 kB Progress (3): 305/798 kB | 583/780 kB | 98/329 kB Progress (3): 321/798 kB | 583/780 kB | 98/329 kB Progress (3): 321/798 kB | 600/780 kB | 98/329 kB Progress (3): 321/798 kB | 600/780 kB | 114/329 kB Progress (3): 321/798 kB | 616/780 kB | 114/329 kB Progress (4): 321/798 kB | 616/780 kB | 114/329 kB | 16/501 kB Progress (4): 321/798 kB | 616/780 kB | 130/329 kB | 16/501 kB Progress (4): 338/798 kB | 616/780 kB | 130/329 kB | 16/501 kB Progress (4): 338/798 kB | 633/780 kB | 130/329 kB | 16/501 kB Progress (4): 338/798 kB | 633/780 kB | 147/329 kB | 16/501 kB Progress (4): 338/798 kB | 649/780 kB | 147/329 kB | 16/501 kB Progress (4): 338/798 kB | 649/780 kB | 147/329 kB | 33/501 kB Progress (5): 338/798 kB | 649/780 kB | 147/329 kB | 33/501 kB | 16/354 kB Progress (5): 338/798 kB | 649/780 kB | 147/329 kB | 33/501 kB | 33/354 kB Progress (5): 338/798 kB | 665/780 kB | 147/329 kB | 33/501 kB | 33/354 kB Progress (5): 338/798 kB | 665/780 kB | 163/329 kB | 33/501 kB | 33/354 kB Progress (5): 338/798 kB | 665/780 kB | 163/329 kB | 33/501 kB | 49/354 kB Progress (5): 354/798 kB | 665/780 kB | 163/329 kB | 33/501 kB | 49/354 kB Progress (5): 354/798 kB | 682/780 kB | 163/329 kB | 33/501 kB | 49/354 kB Progress (5): 354/798 kB | 682/780 kB | 163/329 kB | 33/501 kB | 66/354 kB Progress (5): 354/798 kB | 682/780 kB | 180/329 kB | 33/501 kB | 66/354 kB Progress (5): 354/798 kB | 682/780 kB | 180/329 kB | 49/501 kB | 66/354 kB Progress (5): 354/798 kB | 682/780 kB | 180/329 kB | 49/501 kB | 82/354 kB Progress (5): 354/798 kB | 698/780 kB | 180/329 kB | 49/501 kB | 82/354 kB Progress (5): 354/798 kB | 698/780 kB | 180/329 kB | 49/501 kB | 98/354 kB Progress (5): 370/798 kB | 698/780 kB | 180/329 kB | 49/501 kB | 98/354 kB Progress (5): 370/798 kB | 698/780 kB | 180/329 kB | 49/501 kB | 115/354 kB Progress (5): 370/798 kB | 715/780 kB | 180/329 kB | 49/501 kB | 115/354 kB Progress (5): 370/798 kB | 715/780 kB | 180/329 kB | 66/501 kB | 115/354 kB Progress (5): 370/798 kB | 715/780 kB | 180/329 kB | 66/501 kB | 131/354 kB Progress (5): 370/798 kB | 731/780 kB | 180/329 kB | 66/501 kB | 131/354 kB Progress (5): 370/798 kB | 731/780 kB | 180/329 kB | 66/501 kB | 147/354 kB Progress (5): 370/798 kB | 747/780 kB | 180/329 kB | 66/501 kB | 147/354 kB Progress (5): 370/798 kB | 747/780 kB | 180/329 kB | 66/501 kB | 164/354 kB Progress (5): 370/798 kB | 747/780 kB | 180/329 kB | 82/501 kB | 164/354 kB Progress (5): 370/798 kB | 747/780 kB | 180/329 kB | 82/501 kB | 180/354 kB Progress (5): 370/798 kB | 764/780 kB | 180/329 kB | 82/501 kB | 180/354 kB Progress (5): 370/798 kB | 764/780 kB | 180/329 kB | 82/501 kB | 197/354 kB Progress (5): 370/798 kB | 780/780 kB | 180/329 kB | 82/501 kB | 197/354 kB Progress (5): 370/798 kB | 780 kB | 180/329 kB | 82/501 kB | 197/354 kB Progress (5): 370/798 kB | 780 kB | 180/329 kB | 98/501 kB | 197/354 kB Progress (5): 370/798 kB | 780 kB | 180/329 kB | 98/501 kB | 213/354 kB Progress (5): 370/798 kB | 780 kB | 180/329 kB | 98/501 kB | 229/354 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5.13/httpclient-4.5.13.jar (780 kB at 3.1 MB/s) 2024-03-09T14:38:55.370 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (4): 370/798 kB | 196/329 kB | 98/501 kB | 229/354 kB Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.28/slf4j-api-1.7.28.jar 2024-03-09T14:38:55.375 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (4): 387/798 kB | 196/329 kB | 98/501 kB | 229/354 kB Progress (4): 387/798 kB | 196/329 kB | 115/501 kB | 229/354 kB Progress (4): 387/798 kB | 212/329 kB | 115/501 kB | 229/354 kB Progress (4): 387/798 kB | 229/329 kB | 115/501 kB | 229/354 kB Progress (4): 387/798 kB | 229/329 kB | 115/501 kB | 246/354 kB Progress (4): 387/798 kB | 229/329 kB | 115/501 kB | 256/354 kB Progress (4): 403/798 kB | 229/329 kB | 115/501 kB | 256/354 kB Progress (4): 403/798 kB | 245/329 kB | 115/501 kB | 256/354 kB Progress (4): 403/798 kB | 245/329 kB | 115/501 kB | 272/354 kB Progress (4): 403/798 kB | 245/329 kB | 115/501 kB | 289/354 kB Progress (4): 403/798 kB | 262/329 kB | 115/501 kB | 289/354 kB Progress (4): 403/798 kB | 262/329 kB | 115/501 kB | 305/354 kB Progress (4): 403/798 kB | 278/329 kB | 115/501 kB | 305/354 kB Progress (4): 403/798 kB | 278/329 kB | 115/501 kB | 321/354 kB Progress (4): 403/798 kB | 278/329 kB | 115/501 kB | 338/354 kB Progress (4): 403/798 kB | 294/329 kB | 115/501 kB | 338/354 kB Progress (4): 420/798 kB | 294/329 kB | 115/501 kB | 338/354 kB Progress (4): 420/798 kB | 294/329 kB | 115/501 kB | 354 kB Progress (4): 420/798 kB | 311/329 kB | 115/501 kB | 354 kB Progress (4): 420/798 kB | 311/329 kB | 131/501 kB | 354 kB Progress (4): 420/798 kB | 327/329 kB | 131/501 kB | 354 kB Downloaded from central: https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.15/commons-codec-1.15.jar (354 kB at 1.4 MB/s) 2024-03-09T14:38:55.375 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 420/798 kB | 329 kB | 131/501 kB Downloading from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter/5.10.1/junit-jupiter-5.10.1.jar 2024-03-09T14:38:55.377 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 420/798 kB | 329 kB | 147/501 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.13/httpcore-4.4.13.jar (329 kB at 1.3 MB/s) 2024-03-09T14:38:55.377 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter-params/5.10.1/junit-jupiter-params-5.10.1.jar 2024-03-09T14:38:55.389 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 436/798 kB | 147/501 kB Progress (2): 436/798 kB | 164/501 kB Progress (2): 452/798 kB | 164/501 kB Progress (2): 452/798 kB | 180/501 kB Progress (2): 452/798 kB | 197/501 kB Progress (2): 452/798 kB | 213/501 kB Progress (2): 469/798 kB | 213/501 kB Progress (2): 469/798 kB | 229/501 kB Progress (2): 485/798 kB | 229/501 kB Progress (3): 485/798 kB | 229/501 kB | 16/41 kB Progress (3): 485/798 kB | 229/501 kB | 33/41 kB Progress (3): 485/798 kB | 229/501 kB | 41 kB Progress (3): 485/798 kB | 246/501 kB | 41 kB Progress (3): 502/798 kB | 246/501 kB | 41 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.28/slf4j-api-1.7.28.jar (41 kB at 150 kB/s) 2024-03-09T14:38:55.389 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter-api/5.10.1/junit-jupiter-api-5.10.1.jar 2024-03-09T14:38:55.393 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 502/798 kB | 256/501 kB Progress (2): 518/798 kB | 256/501 kB Progress (2): 518/798 kB | 272/501 kB Progress (2): 518/798 kB | 289/501 kB Progress (3): 518/798 kB | 289/501 kB | 6.4 kB Progress (3): 534/798 kB | 289/501 kB | 6.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter/5.10.1/junit-jupiter-5.10.1.jar (6.4 kB at 23 kB/s) 2024-03-09T14:38:55.393 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/opentest4j/opentest4j/1.3.0/opentest4j-1.3.0.jar 2024-03-09T14:38:55.417 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 534/798 kB | 305/501 kB Progress (2): 534/798 kB | 321/501 kB Progress (2): 534/798 kB | 338/501 kB Progress (2): 551/798 kB | 338/501 kB Progress (2): 551/798 kB | 354/501 kB Progress (2): 567/798 kB | 354/501 kB Progress (3): 567/798 kB | 354/501 kB | 16/586 kB Progress (3): 567/798 kB | 354/501 kB | 33/586 kB Progress (3): 583/798 kB | 354/501 kB | 33/586 kB Progress (3): 583/798 kB | 354/501 kB | 49/586 kB Progress (3): 583/798 kB | 354/501 kB | 66/586 kB Progress (3): 583/798 kB | 370/501 kB | 66/586 kB Progress (3): 583/798 kB | 370/501 kB | 82/586 kB Progress (3): 600/798 kB | 370/501 kB | 82/586 kB Progress (3): 600/798 kB | 370/501 kB | 98/586 kB Progress (3): 600/798 kB | 387/501 kB | 98/586 kB Progress (3): 600/798 kB | 387/501 kB | 115/586 kB Progress (3): 600/798 kB | 387/501 kB | 131/586 kB Progress (3): 600/798 kB | 387/501 kB | 147/586 kB Progress (3): 600/798 kB | 403/501 kB | 147/586 kB Progress (3): 600/798 kB | 403/501 kB | 164/586 kB Progress (3): 600/798 kB | 403/501 kB | 180/586 kB Progress (4): 600/798 kB | 403/501 kB | 180/586 kB | 16/211 kB Progress (4): 600/798 kB | 403/501 kB | 197/586 kB | 16/211 kB Progress (4): 616/798 kB | 403/501 kB | 197/586 kB | 16/211 kB Progress (4): 616/798 kB | 403/501 kB | 197/586 kB | 33/211 kB Progress (4): 616/798 kB | 420/501 kB | 197/586 kB | 33/211 kB Progress (4): 616/798 kB | 420/501 kB | 213/586 kB | 33/211 kB Progress (4): 616/798 kB | 420/501 kB | 213/586 kB | 49/211 kB Progress (4): 633/798 kB | 420/501 kB | 213/586 kB | 49/211 kB Progress (4): 633/798 kB | 420/501 kB | 229/586 kB | 49/211 kB Progress (4): 633/798 kB | 420/501 kB | 229/586 kB | 66/211 kB Progress (4): 633/798 kB | 436/501 kB | 229/586 kB | 66/211 kB Progress (4): 633/798 kB | 436/501 kB | 246/586 kB | 66/211 kB Progress (4): 633/798 kB | 436/501 kB | 246/586 kB | 82/211 kB Progress (4): 633/798 kB | 436/501 kB | 256/586 kB | 82/211 kB Progress (4): 633/798 kB | 452/501 kB | 256/586 kB | 82/211 kB Progress (4): 633/798 kB | 452/501 kB | 256/586 kB | 98/211 kB Progress (4): 633/798 kB | 452/501 kB | 272/586 kB | 98/211 kB Progress (4): 633/798 kB | 452/501 kB | 289/586 kB | 98/211 kB Progress (4): 633/798 kB | 452/501 kB | 289/586 kB | 115/211 kB Progress (4): 633/798 kB | 469/501 kB | 289/586 kB | 115/211 kB Progress (4): 633/798 kB | 469/501 kB | 305/586 kB | 115/211 kB Progress (4): 633/798 kB | 469/501 kB | 305/586 kB | 131/211 kB Progress (4): 633/798 kB | 469/501 kB | 321/586 kB | 131/211 kB Progress (4): 649/798 kB | 469/501 kB | 321/586 kB | 131/211 kB Progress (4): 649/798 kB | 469/501 kB | 321/586 kB | 147/211 kB Progress (4): 649/798 kB | 469/501 kB | 338/586 kB | 147/211 kB Progress (4): 649/798 kB | 485/501 kB | 338/586 kB | 147/211 kB Progress (4): 649/798 kB | 485/501 kB | 338/586 kB | 164/211 kB Progress (4): 649/798 kB | 485/501 kB | 354/586 kB | 164/211 kB Progress (4): 649/798 kB | 485/501 kB | 354/586 kB | 180/211 kB Progress (4): 665/798 kB | 485/501 kB | 354/586 kB | 180/211 kB Progress (4): 665/798 kB | 485/501 kB | 370/586 kB | 180/211 kB Progress (5): 665/798 kB | 485/501 kB | 370/586 kB | 180/211 kB | 14 kB Progress (5): 665/798 kB | 485/501 kB | 370/586 kB | 197/211 kB | 14 kB Progress (5): 665/798 kB | 485/501 kB | 370/586 kB | 211 kB | 14 kB Progress (5): 665/798 kB | 485/501 kB | 387/586 kB | 211 kB | 14 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/opentest4j/opentest4j/1.3.0/opentest4j-1.3.0.jar (14 kB at 48 kB/s) 2024-03-09T14:38:55.417 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-commons/1.10.1/junit-platform-commons-1.10.1.jar 2024-03-09T14:38:55.418 INFO:tasks.workunit.client.0.smithi001.stdout:Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter-api/5.10.1/junit-jupiter-api-5.10.1.jar (211 kB at 701 kB/s) 2024-03-09T14:38:55.418 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apiguardian/apiguardian-api/1.1.2/apiguardian-api-1.1.2.jar 2024-03-09T14:38:55.420 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 665/798 kB | 485/501 kB | 403/586 kB Progress (3): 682/798 kB | 485/501 kB | 403/586 kB Progress (3): 682/798 kB | 485/501 kB | 420/586 kB Progress (3): 682/798 kB | 501 kB | 420/586 kB Progress (3): 682/798 kB | 501 kB | 436/586 kB Downloaded from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.15.1/commons-io-2.15.1.jar (501 kB at 1.6 MB/s) 2024-03-09T14:38:55.420 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 698/798 kB | 436/586 kB Downloading from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter-engine/5.10.1/junit-jupiter-engine-5.10.1.jar 2024-03-09T14:38:55.426 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 698/798 kB | 452/586 kB Progress (2): 698/798 kB | 469/586 kB Progress (2): 698/798 kB | 485/586 kB Progress (2): 698/798 kB | 502/586 kB Progress (2): 698/798 kB | 518/586 kB Progress (2): 715/798 kB | 518/586 kB Progress (2): 715/798 kB | 534/586 kB Progress (2): 715/798 kB | 551/586 kB Progress (2): 715/798 kB | 567/586 kB Progress (2): 731/798 kB | 567/586 kB Progress (2): 731/798 kB | 583/586 kB Progress (2): 731/798 kB | 586 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter-params/5.10.1/junit-jupiter-params-5.10.1.jar (586 kB at 1.9 MB/s) 2024-03-09T14:38:55.426 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-engine/1.10.1/junit-platform-engine-1.10.1.jar 2024-03-09T14:38:55.436 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 747/798 kB Progress (1): 764/798 kB Progress (1): 780/798 kB Progress (1): 796/798 kB Progress (1): 798 kB Downloaded from central: https://repo.maven.apache.org/maven2/software/amazon/awssdk/regions/2.20.43/regions-2.20.43.jar (798 kB at 2.5 MB/s) 2024-03-09T14:38:55.436 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-simple/1.7.28/slf4j-simple-1.7.28.jar 2024-03-09T14:38:55.439 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/106 kB Progress (1): 33/106 kB Progress (1): 49/106 kB Progress (1): 66/106 kB Progress (1): 82/106 kB Progress (1): 98/106 kB Progress (1): 106 kB Progress (2): 106 kB | 16/245 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-commons/1.10.1/junit-platform-commons-1.10.1.jar (106 kB at 329 kB/s) 2024-03-09T14:38:55.440 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.7.28/jcl-over-slf4j-1.7.28.jar 2024-03-09T14:38:55.451 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 33/245 kB Progress (1): 49/245 kB Progress (1): 66/245 kB Progress (1): 82/245 kB Progress (2): 82/245 kB | 16/205 kB Progress (2): 82/245 kB | 32/205 kB Progress (2): 82/245 kB | 49/205 kB Progress (2): 82/245 kB | 65/205 kB Progress (2): 98/245 kB | 65/205 kB Progress (2): 98/245 kB | 81/205 kB Progress (2): 98/245 kB | 98/205 kB Progress (2): 98/245 kB | 114/205 kB Progress (2): 115/245 kB | 114/205 kB Progress (2): 115/245 kB | 130/205 kB Progress (2): 115/245 kB | 147/205 kB Progress (2): 115/245 kB | 163/205 kB Progress (2): 131/245 kB | 163/205 kB Progress (2): 131/245 kB | 180/205 kB Progress (2): 131/245 kB | 196/205 kB Progress (2): 131/245 kB | 205 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-engine/1.10.1/junit-platform-engine-1.10.1.jar (205 kB at 613 kB/s) 2024-03-09T14:38:55.451 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite/1.10.1/junit-platform-suite-1.10.1.jar 2024-03-09T14:38:55.453 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 131/245 kB | 6.8 kB Progress (3): 131/245 kB | 6.8 kB | 15 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apiguardian/apiguardian-api/1.1.2/apiguardian-api-1.1.2.jar (6.8 kB at 20 kB/s) 2024-03-09T14:38:55.453 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite-api/1.10.1/junit-platform-suite-api-1.10.1.jar 2024-03-09T14:38:55.453 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 147/245 kB | 15 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-simple/1.7.28/slf4j-simple-1.7.28.jar (15 kB at 45 kB/s) 2024-03-09T14:38:55.454 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite-engine/1.10.1/junit-platform-suite-engine-1.10.1.jar 2024-03-09T14:38:55.457 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 164/245 kB Progress (1): 180/245 kB Progress (2): 180/245 kB | 16/16 kB Progress (2): 180/245 kB | 16 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.7.28/jcl-over-slf4j-1.7.28.jar (16 kB at 48 kB/s) 2024-03-09T14:38:55.458 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 197/245 kB Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite-commons/1.10.1/junit-platform-suite-commons-1.10.1.jar 2024-03-09T14:38:55.462 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 213/245 kB Progress (1): 229/245 kB Progress (1): 245 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/jupiter/junit-jupiter-engine/5.10.1/junit-jupiter-engine-5.10.1.jar (245 kB at 709 kB/s) 2024-03-09T14:38:55.462 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-launcher/1.10.1/junit-platform-launcher-1.10.1.jar 2024-03-09T14:38:55.469 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite/1.10.1/junit-platform-suite-1.10.1.jar (6.4 kB at 18 kB/s) 2024-03-09T14:38:55.473 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/23 kB Progress (2): 16/23 kB | 16/24 kB Progress (2): 23 kB | 16/24 kB Progress (2): 23 kB | 24 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite-api/1.10.1/junit-platform-suite-api-1.10.1.jar (23 kB at 64 kB/s) 2024-03-09T14:38:55.473 INFO:tasks.workunit.client.0.smithi001.stdout:Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite-engine/1.10.1/junit-platform-suite-engine-1.10.1.jar (24 kB at 68 kB/s) 2024-03-09T14:38:55.475 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/17 kB Progress (1): 17 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-suite-commons/1.10.1/junit-platform-suite-commons-1.10.1.jar (17 kB at 49 kB/s) 2024-03-09T14:38:55.497 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/184 kB Progress (1): 33/184 kB Progress (1): 49/184 kB Progress (1): 66/184 kB Progress (1): 82/184 kB Progress (1): 98/184 kB Progress (1): 115/184 kB Progress (1): 131/184 kB Progress (1): 147/184 kB Progress (1): 164/184 kB Progress (1): 180/184 kB Progress (1): 184 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-launcher/1.10.1/junit-platform-launcher-1.10.1.jar (184 kB at 485 kB/s) 2024-03-09T14:38:55.542 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:38:55.542 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --- clean:3.2.0:clean (default-clean) @ jcksum --- 2024-03-09T14:38:55.556 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/3.3.4/maven-shared-utils-3.3.4.pom 2024-03-09T14:38:55.575 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/3.3.4/maven-shared-utils-3.3.4.pom (5.8 kB at 307 kB/s) 2024-03-09T14:38:55.578 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-components/34/maven-shared-components-34.pom 2024-03-09T14:38:55.596 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-components/34/maven-shared-components-34.pom (5.1 kB at 283 kB/s) 2024-03-09T14:38:55.599 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/34/maven-parent-34.pom 2024-03-09T14:38:55.619 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/43 kB Progress (1): 33/43 kB Progress (1): 43 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/34/maven-parent-34.pom (43 kB at 2.1 MB/s) 2024-03-09T14:38:55.627 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.6/commons-io-2.6.pom 2024-03-09T14:38:55.646 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 14 kB Downloaded from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.6/commons-io-2.6.pom (14 kB at 792 kB/s) 2024-03-09T14:38:55.656 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/3.3.4/maven-shared-utils-3.3.4.jar 2024-03-09T14:38:55.685 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/153 kB Progress (1): 32/153 kB Progress (1): 49/153 kB Progress (1): 65/153 kB Progress (1): 81/153 kB Progress (1): 98/153 kB Progress (1): 114/153 kB Progress (1): 130/153 kB Progress (1): 147/153 kB Progress (1): 153 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/3.3.4/maven-shared-utils-3.3.4.jar (153 kB at 5.1 MB/s) 2024-03-09T14:38:55.686 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.6/commons-io-2.6.jar 2024-03-09T14:38:55.723 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/215 kB Progress (1): 33/215 kB Progress (1): 49/215 kB Progress (1): 66/215 kB Progress (1): 82/215 kB Progress (1): 98/215 kB Progress (1): 115/215 kB Progress (1): 131/215 kB Progress (1): 147/215 kB Progress (1): 164/215 kB Progress (1): 180/215 kB Progress (1): 197/215 kB Progress (1): 213/215 kB Progress (1): 215 kB Downloaded from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.6/commons-io-2.6.jar (215 kB at 5.7 MB/s) 2024-03-09T14:38:55.838 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:38:55.838 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --- resources:3.3.1:resources (default-resources) @ jcksum --- 2024-03-09T14:38:55.845 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.pom 2024-03-09T14:38:55.864 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.7 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.pom (2.7 kB at 140 kB/s) 2024-03-09T14:38:55.867 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/5.1/plexus-5.1.pom 2024-03-09T14:38:55.886 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/23 kB Progress (1): 23 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/5.1/plexus-5.1.pom (23 kB at 1.2 MB/s) 2024-03-09T14:38:55.892 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.5.1/plexus-utils-3.5.1.pom 2024-03-09T14:38:55.910 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 8.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.5.1/plexus-utils-3.5.1.pom (8.8 kB at 487 kB/s) 2024-03-09T14:38:55.914 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/10/plexus-10.pom 2024-03-09T14:38:55.933 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/25 kB Progress (1): 25 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/10/plexus-10.pom (25 kB at 1.3 MB/s) 2024-03-09T14:38:55.939 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-filtering/3.3.1/maven-filtering-3.3.1.pom 2024-03-09T14:38:55.957 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-filtering/3.3.1/maven-filtering-3.3.1.pom (6.0 kB at 335 kB/s) 2024-03-09T14:38:55.961 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-components/39/maven-shared-components-39.pom 2024-03-09T14:38:55.979 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-components/39/maven-shared-components-39.pom (3.2 kB at 179 kB/s) 2024-03-09T14:38:55.986 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/javax/inject/javax.inject/1/javax.inject-1.pom 2024-03-09T14:38:56.004 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 612 B Downloaded from central: https://repo.maven.apache.org/maven2/javax/inject/javax.inject/1/javax.inject-1.pom (612 B at 32 kB/s) 2024-03-09T14:38:56.008 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.pom 2024-03-09T14:38:56.025 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.7 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.pom (2.7 kB at 152 kB/s) 2024-03-09T14:38:56.029 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.36/slf4j-parent-1.7.36.pom 2024-03-09T14:38:56.048 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 14 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.36/slf4j-parent-1.7.36.pom (14 kB at 742 kB/s) 2024-03-09T14:38:56.053 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-build-api/0.0.7/plexus-build-api-0.0.7.pom 2024-03-09T14:38:56.072 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-build-api/0.0.7/plexus-build-api-0.0.7.pom (3.2 kB at 169 kB/s) 2024-03-09T14:38:56.075 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/15/spice-parent-15.pom 2024-03-09T14:38:56.094 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 8.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/15/spice-parent-15.pom (8.4 kB at 440 kB/s) 2024-03-09T14:38:56.098 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/5/forge-parent-5.pom 2024-03-09T14:38:56.117 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 8.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/5/forge-parent-5.pom (8.4 kB at 465 kB/s) 2024-03-09T14:38:56.122 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.5.0/plexus-utils-3.5.0.pom 2024-03-09T14:38:56.141 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 8.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.5.0/plexus-utils-3.5.0.pom (8.0 kB at 445 kB/s) 2024-03-09T14:38:56.147 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.11.0/commons-io-2.11.0.pom 2024-03-09T14:38:56.166 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/20 kB Progress (1): 20 kB Downloaded from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.11.0/commons-io-2.11.0.pom (20 kB at 1.0 MB/s) 2024-03-09T14:38:56.177 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/junit-bom/5.7.2/junit-bom-5.7.2.pom 2024-03-09T14:38:56.196 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/junit-bom/5.7.2/junit-bom-5.7.2.pom (5.1 kB at 268 kB/s) 2024-03-09T14:38:56.203 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.pom 2024-03-09T14:38:56.223 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/31 kB Progress (1): 31 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.pom (31 kB at 1.5 MB/s) 2024-03-09T14:38:56.238 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/junit-bom/5.7.1/junit-bom-5.7.1.pom 2024-03-09T14:38:56.257 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/junit-bom/5.7.1/junit-bom-5.7.1.pom (5.1 kB at 268 kB/s) 2024-03-09T14:38:56.266 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.jar 2024-03-09T14:38:56.290 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/85 kB Progress (1): 32/85 kB Progress (1): 49/85 kB Progress (1): 65/85 kB Progress (1): 81/85 kB Progress (1): 85 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.jar (85 kB at 3.6 MB/s) 2024-03-09T14:38:56.291 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.5.1/plexus-utils-3.5.1.jar 2024-03-09T14:38:56.291 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-filtering/3.3.1/maven-filtering-3.3.1.jar 2024-03-09T14:38:56.292 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/javax/inject/javax.inject/1/javax.inject-1.jar 2024-03-09T14:38:56.293 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar 2024-03-09T14:38:56.293 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-build-api/0.0.7/plexus-build-api-0.0.7.jar 2024-03-09T14:38:56.309 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.5 kB Downloaded from central: https://repo.maven.apache.org/maven2/javax/inject/javax.inject/1/javax.inject-1.jar (2.5 kB at 139 kB/s) 2024-03-09T14:38:56.309 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar 2024-03-09T14:38:56.313 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/55 kB Progress (1): 33/55 kB Progress (2): 33/55 kB | 16/269 kB Progress (3): 33/55 kB | 16/269 kB | 16/41 kB Progress (3): 49/55 kB | 16/269 kB | 16/41 kB Progress (3): 49/55 kB | 33/269 kB | 16/41 kB Progress (4): 49/55 kB | 33/269 kB | 16/41 kB | 8.5 kB Progress (4): 49/55 kB | 49/269 kB | 16/41 kB | 8.5 kB Progress (4): 49/55 kB | 49/269 kB | 33/41 kB | 8.5 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-build-api/0.0.7/plexus-build-api-0.0.7.jar (8.5 kB at 446 kB/s) 2024-03-09T14:38:56.313 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar 2024-03-09T14:38:56.314 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 49/55 kB | 49/269 kB | 41 kB Progress (3): 55 kB | 49/269 kB | 41 kB Progress (3): 55 kB | 66/269 kB | 41 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar (41 kB at 1.8 MB/s) 2024-03-09T14:38:56.314 INFO:tasks.workunit.client.0.smithi001.stdout:Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-filtering/3.3.1/maven-filtering-3.3.1.jar (55 kB at 2.4 MB/s) 2024-03-09T14:38:56.334 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 82/269 kB Progress (1): 98/269 kB Progress (1): 115/269 kB Progress (1): 131/269 kB Progress (1): 147/269 kB Progress (1): 164/269 kB Progress (1): 180/269 kB Progress (1): 197/269 kB Progress (1): 213/269 kB Progress (1): 229/269 kB Progress (2): 229/269 kB | 16/327 kB Progress (2): 229/269 kB | 33/327 kB Progress (2): 229/269 kB | 49/327 kB Progress (2): 229/269 kB | 66/327 kB Progress (2): 229/269 kB | 82/327 kB Progress (2): 229/269 kB | 98/327 kB Progress (3): 229/269 kB | 98/327 kB | 4.9/587 kB Progress (3): 229/269 kB | 115/327 kB | 4.9/587 kB Progress (3): 229/269 kB | 131/327 kB | 4.9/587 kB Progress (3): 229/269 kB | 147/327 kB | 4.9/587 kB Progress (3): 229/269 kB | 164/327 kB | 4.9/587 kB Progress (3): 229/269 kB | 180/327 kB | 4.9/587 kB Progress (3): 229/269 kB | 180/327 kB | 21/587 kB Progress (3): 229/269 kB | 197/327 kB | 21/587 kB Progress (3): 229/269 kB | 213/327 kB | 21/587 kB Progress (3): 229/269 kB | 229/327 kB | 21/587 kB Progress (3): 229/269 kB | 246/327 kB | 21/587 kB Progress (3): 229/269 kB | 262/327 kB | 21/587 kB Progress (3): 229/269 kB | 262/327 kB | 38/587 kB Progress (3): 229/269 kB | 279/327 kB | 38/587 kB Progress (3): 229/269 kB | 295/327 kB | 38/587 kB Progress (3): 229/269 kB | 311/327 kB | 38/587 kB Progress (3): 246/269 kB | 311/327 kB | 38/587 kB Progress (3): 246/269 kB | 327 kB | 38/587 kB Progress (3): 262/269 kB | 327 kB | 38/587 kB Downloaded from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar (327 kB at 8.0 MB/s) 2024-03-09T14:38:56.335 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 269 kB | 38/587 kB Progress (2): 269 kB | 54/587 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.5.1/plexus-utils-3.5.1.jar (269 kB at 6.0 MB/s) 2024-03-09T14:38:56.404 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 66/587 kB Progress (1): 83/587 kB Progress (1): 99/587 kB Progress (1): 116/587 kB Progress (1): 132/587 kB Progress (1): 148/587 kB Progress (1): 165/587 kB Progress (1): 181/587 kB Progress (1): 197/587 kB Progress (1): 214/587 kB Progress (1): 230/587 kB Progress (1): 247/587 kB Progress (1): 263/587 kB Progress (1): 279/587 kB Progress (1): 296/587 kB Progress (1): 312/587 kB Progress (1): 329/587 kB Progress (1): 345/587 kB Progress (1): 361/587 kB Progress (1): 378/587 kB Progress (1): 394/587 kB Progress (1): 410/587 kB Progress (1): 427/587 kB Progress (1): 443/587 kB Progress (1): 460/587 kB Progress (1): 476/587 kB Progress (1): 492/587 kB Progress (1): 509/587 kB Progress (1): 525/587 kB Progress (1): 541/587 kB Progress (1): 558/587 kB Progress (1): 574/587 kB Progress (1): 587 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar (587 kB at 5.3 MB/s) 2024-03-09T14:38:56.466 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] skip non existing resourceDirectory /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/main/resources 2024-03-09T14:38:56.467 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:38:56.467 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --- compiler:3.11.0:compile (default-compile) @ jcksum --- 2024-03-09T14:38:56.476 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-incremental/1.1/maven-shared-incremental-1.1.pom 2024-03-09T14:38:56.493 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.7 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-incremental/1.1/maven-shared-incremental-1.1.pom (4.7 kB at 263 kB/s) 2024-03-09T14:38:56.495 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-components/19/maven-shared-components-19.pom 2024-03-09T14:38:56.513 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-components/19/maven-shared-components-19.pom (6.4 kB at 353 kB/s) 2024-03-09T14:38:56.515 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom 2024-03-09T14:38:56.535 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/33 kB Progress (1): 33 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom (33 kB at 1.7 MB/s) 2024-03-09T14:38:56.537 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom 2024-03-09T14:38:56.555 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 14 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 kB at 776 kB/s) 2024-03-09T14:38:56.559 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.pom 2024-03-09T14:38:56.576 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 815 B Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.pom (815 B at 48 kB/s) 2024-03-09T14:38:56.578 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-containers/1.5.5/plexus-containers-1.5.5.pom 2024-03-09T14:38:56.596 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-containers/1.5.5/plexus-containers-1.5.5.pom (4.2 kB at 236 kB/s) 2024-03-09T14:38:56.598 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/2.0.7/plexus-2.0.7.pom 2024-03-09T14:38:56.616 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/17 kB Progress (1): 17 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/2.0.7/plexus-2.0.7.pom (17 kB at 961 kB/s) 2024-03-09T14:38:56.620 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-java/1.1.2/plexus-java-1.1.2.pom 2024-03-09T14:38:56.640 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-java/1.1.2/plexus-java-1.1.2.pom (5.0 kB at 249 kB/s) 2024-03-09T14:38:56.643 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-languages/1.1.2/plexus-languages-1.1.2.pom 2024-03-09T14:38:56.660 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-languages/1.1.2/plexus-languages-1.1.2.pom (4.1 kB at 244 kB/s) 2024-03-09T14:38:56.664 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/ow2/asm/asm/9.4/asm-9.4.pom 2024-03-09T14:38:56.681 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/ow2/asm/asm/9.4/asm-9.4.pom (2.4 kB at 139 kB/s) 2024-03-09T14:38:56.683 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/ow2/ow2/1.5.1/ow2-1.5.1.pom 2024-03-09T14:38:56.701 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 11 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/ow2/ow2/1.5.1/ow2-1.5.1.pom (11 kB at 627 kB/s) 2024-03-09T14:38:56.704 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/com/thoughtworks/qdox/qdox/2.0.3/qdox-2.0.3.pom 2024-03-09T14:38:56.722 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/17 kB Progress (1): 17 kB Downloaded from central: https://repo.maven.apache.org/maven2/com/thoughtworks/qdox/qdox/2.0.3/qdox-2.0.3.pom (17 kB at 958 kB/s) 2024-03-09T14:38:56.725 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom 2024-03-09T14:38:56.742 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.6 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom (6.6 kB at 365 kB/s) 2024-03-09T14:38:56.746 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-api/2.13.0/plexus-compiler-api-2.13.0.pom 2024-03-09T14:38:56.764 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-api/2.13.0/plexus-compiler-api-2.13.0.pom (1.1 kB at 56 kB/s) 2024-03-09T14:38:56.766 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler/2.13.0/plexus-compiler-2.13.0.pom 2024-03-09T14:38:56.783 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 8.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler/2.13.0/plexus-compiler-2.13.0.pom (8.4 kB at 493 kB/s) 2024-03-09T14:38:56.786 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-components/10.0/plexus-components-10.0.pom 2024-03-09T14:38:56.804 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.7 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-components/10.0/plexus-components-10.0.pom (2.7 kB at 158 kB/s) 2024-03-09T14:38:56.807 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/junit-bom/5.9.1/junit-bom-5.9.1.pom 2024-03-09T14:38:56.825 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.6 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/junit-bom/5.9.1/junit-bom-5.9.1.pom (5.6 kB at 313 kB/s) 2024-03-09T14:38:56.829 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-manager/2.13.0/plexus-compiler-manager-2.13.0.pom 2024-03-09T14:38:56.846 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-manager/2.13.0/plexus-compiler-manager-2.13.0.pom (1.1 kB at 63 kB/s) 2024-03-09T14:38:56.851 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-component-annotations/2.1.1/plexus-component-annotations-2.1.1.pom 2024-03-09T14:38:56.869 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 770 B Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-component-annotations/2.1.1/plexus-component-annotations-2.1.1.pom (770 B at 43 kB/s) 2024-03-09T14:38:56.871 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-containers/2.1.1/plexus-containers-2.1.1.pom 2024-03-09T14:38:56.894 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-containers/2.1.1/plexus-containers-2.1.1.pom (6.0 kB at 262 kB/s) 2024-03-09T14:38:56.896 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/6.5/plexus-6.5.pom 2024-03-09T14:38:56.915 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/26 kB Progress (1): 26 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/6.5/plexus-6.5.pom (26 kB at 1.4 MB/s) 2024-03-09T14:38:56.919 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-javac/2.13.0/plexus-compiler-javac-2.13.0.pom 2024-03-09T14:38:56.936 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-javac/2.13.0/plexus-compiler-javac-2.13.0.pom (1.2 kB at 71 kB/s) 2024-03-09T14:38:56.938 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compilers/2.13.0/plexus-compilers-2.13.0.pom 2024-03-09T14:38:56.959 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.3 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compilers/2.13.0/plexus-compilers-2.13.0.pom (1.3 kB at 63 kB/s) 2024-03-09T14:38:56.973 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-incremental/1.1/maven-shared-incremental-1.1.jar 2024-03-09T14:38:56.994 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 14 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-incremental/1.1/maven-shared-incremental-1.1.jar (14 kB at 645 kB/s) 2024-03-09T14:38:56.995 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.jar 2024-03-09T14:38:56.995 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-java/1.1.2/plexus-java-1.1.2.jar 2024-03-09T14:38:56.995 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/ow2/asm/asm/9.4/asm-9.4.jar 2024-03-09T14:38:56.996 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/com/thoughtworks/qdox/qdox/2.0.3/qdox-2.0.3.jar 2024-03-09T14:38:56.996 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-api/2.13.0/plexus-compiler-api-2.13.0.jar 2024-03-09T14:38:57.015 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.2 kB Progress (2): 4.2 kB | 16/122 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.jar (4.2 kB at 201 kB/s) 2024-03-09T14:38:57.015 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.5.0/plexus-utils-3.5.0.jar 2024-03-09T14:38:57.019 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 33/122 kB Progress (2): 33/122 kB | 16/55 kB Progress (3): 33/122 kB | 16/55 kB | 16/27 kB Progress (4): 33/122 kB | 16/55 kB | 16/27 kB | 16/334 kB Progress (4): 33/122 kB | 16/55 kB | 27 kB | 16/334 kB Progress (4): 33/122 kB | 33/55 kB | 27 kB | 16/334 kB Progress (4): 49/122 kB | 33/55 kB | 27 kB | 16/334 kB Progress (4): 49/122 kB | 33/55 kB | 27 kB | 33/334 kB Progress (4): 66/122 kB | 33/55 kB | 27 kB | 33/334 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-api/2.13.0/plexus-compiler-api-2.13.0.jar (27 kB at 1.3 MB/s) 2024-03-09T14:38:57.019 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 66/122 kB | 33/55 kB | 49/334 kB Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-manager/2.13.0/plexus-compiler-manager-2.13.0.jar 2024-03-09T14:38:57.020 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 82/122 kB | 33/55 kB | 49/334 kB Progress (3): 82/122 kB | 49/55 kB | 49/334 kB Progress (3): 82/122 kB | 49/55 kB | 66/334 kB Progress (3): 82/122 kB | 55 kB | 66/334 kB Progress (3): 98/122 kB | 55 kB | 66/334 kB Progress (3): 98/122 kB | 55 kB | 82/334 kB Progress (3): 115/122 kB | 55 kB | 82/334 kB Progress (3): 115/122 kB | 55 kB | 98/334 kB Progress (3): 122 kB | 55 kB | 98/334 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-java/1.1.2/plexus-java-1.1.2.jar (55 kB at 2.2 MB/s) 2024-03-09T14:38:57.020 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-javac/2.13.0/plexus-compiler-javac-2.13.0.jar 2024-03-09T14:38:57.020 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 122 kB | 115/334 kB Progress (2): 122 kB | 131/334 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/ow2/asm/asm/9.4/asm-9.4.jar (122 kB at 4.9 MB/s) 2024-03-09T14:38:57.029 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 147/334 kB Progress (1): 164/334 kB Progress (1): 180/334 kB Progress (1): 197/334 kB Progress (1): 213/334 kB Progress (1): 229/334 kB Progress (1): 246/334 kB Progress (1): 256/334 kB Progress (1): 272/334 kB Progress (1): 289/334 kB Progress (1): 305/334 kB Progress (1): 321/334 kB Progress (1): 334 kB Downloaded from central: https://repo.maven.apache.org/maven2/com/thoughtworks/qdox/qdox/2.0.3/qdox-2.0.3.jar (334 kB at 9.8 MB/s) 2024-03-09T14:38:57.037 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/267 kB Progress (2): 16/267 kB | 4.7 kB Progress (2): 33/267 kB | 4.7 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-manager/2.13.0/plexus-compiler-manager-2.13.0.jar (4.7 kB at 114 kB/s) 2024-03-09T14:38:57.038 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 33/267 kB | 16/23 kB Progress (2): 33/267 kB | 23 kB Progress (2): 49/267 kB | 23 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-javac/2.13.0/plexus-compiler-javac-2.13.0.jar (23 kB at 543 kB/s) 2024-03-09T14:38:57.063 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 66/267 kB Progress (1): 82/267 kB Progress (1): 98/267 kB Progress (1): 115/267 kB Progress (1): 131/267 kB Progress (1): 147/267 kB Progress (1): 164/267 kB Progress (1): 180/267 kB Progress (1): 197/267 kB Progress (1): 213/267 kB Progress (1): 229/267 kB Progress (1): 246/267 kB Progress (1): 256/267 kB Progress (1): 267 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.5.0/plexus-utils-3.5.0.jar (267 kB at 4.0 MB/s) 2024-03-09T14:38:57.196 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Changes detected - recompiling the module! :source 2024-03-09T14:38:57.199 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Compiling 1 source file with javac [debug target 1.9] to target/classes 2024-03-09T14:38:58.414 INFO:tasks.workunit.client.0.smithi001.stdout:[WARNING] bootstrap class path not set in conjunction with -source 9 2024-03-09T14:38:58.414 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:38:58.414 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --- resources:3.3.1:testResources (default-testResources) @ jcksum --- 2024-03-09T14:38:58.417 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] skip non existing resourceDirectory /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/test/resources 2024-03-09T14:38:58.417 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:38:58.418 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --- compiler:3.11.0:testCompile (default-testCompile) @ jcksum --- 2024-03-09T14:38:58.423 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Changes detected - recompiling the module! :dependency 2024-03-09T14:38:58.423 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Compiling 1 source file with javac [debug target 1.9] to target/test-classes 2024-03-09T14:38:58.598 INFO:tasks.workunit.client.0.smithi001.stdout:[WARNING] bootstrap class path not set in conjunction with -source 9 2024-03-09T14:38:58.598 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:38:58.598 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --- surefire:3.2.3:test (default-test) @ jcksum --- 2024-03-09T14:38:58.602 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/3.2.3/maven-surefire-common-3.2.3.pom 2024-03-09T14:38:58.619 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/3.2.3/maven-surefire-common-3.2.3.pom (6.2 kB at 367 kB/s) 2024-03-09T14:38:58.623 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/3.2.3/surefire-api-3.2.3.pom 2024-03-09T14:38:58.640 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.5 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/3.2.3/surefire-api-3.2.3.pom (3.5 kB at 208 kB/s) 2024-03-09T14:38:58.644 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-logger-api/3.2.3/surefire-logger-api-3.2.3.pom 2024-03-09T14:38:58.661 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.3 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-logger-api/3.2.3/surefire-logger-api-3.2.3.pom (3.3 kB at 192 kB/s) 2024-03-09T14:38:58.665 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-shared-utils/3.2.3/surefire-shared-utils-3.2.3.pom 2024-03-09T14:38:58.682 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-shared-utils/3.2.3/surefire-shared-utils-3.2.3.pom (4.1 kB at 239 kB/s) 2024-03-09T14:38:58.686 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-extensions-api/3.2.3/surefire-extensions-api-3.2.3.pom 2024-03-09T14:38:58.703 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.3 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-extensions-api/3.2.3/surefire-extensions-api-3.2.3.pom (3.3 kB at 194 kB/s) 2024-03-09T14:38:58.706 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/3.2.3/surefire-booter-3.2.3.pom 2024-03-09T14:38:58.724 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.5 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/3.2.3/surefire-booter-3.2.3.pom (4.5 kB at 262 kB/s) 2024-03-09T14:38:58.727 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-extensions-spi/3.2.3/surefire-extensions-spi-3.2.3.pom 2024-03-09T14:38:58.744 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-extensions-spi/3.2.3/surefire-extensions-spi-3.2.3.pom (1.8 kB at 104 kB/s) 2024-03-09T14:38:58.747 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-util/1.0.0.v20140518/aether-util-1.0.0.v20140518.pom 2024-03-09T14:38:58.765 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-util/1.0.0.v20140518/aether-util-1.0.0.v20140518.pom (2.2 kB at 129 kB/s) 2024-03-09T14:38:58.766 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether/1.0.0.v20140518/aether-1.0.0.v20140518.pom 2024-03-09T14:38:58.785 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/30 kB Progress (1): 30 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether/1.0.0.v20140518/aether-1.0.0.v20140518.pom (30 kB at 1.7 MB/s) 2024-03-09T14:38:58.789 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/1.0.0.v20140518/aether-api-1.0.0.v20140518.pom 2024-03-09T14:38:58.807 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.9 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/1.0.0.v20140518/aether-api-1.0.0.v20140518.pom (1.9 kB at 105 kB/s) 2024-03-09T14:38:58.812 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-common-artifact-filters/3.1.1/maven-common-artifact-filters-3.1.1.pom 2024-03-09T14:38:58.830 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-common-artifact-filters/3.1.1/maven-common-artifact-filters-3.1.1.pom (5.8 kB at 305 kB/s) 2024-03-09T14:38:58.836 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/3.2.5/maven-artifact-3.2.5.pom 2024-03-09T14:38:58.854 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.3 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/3.2.5/maven-artifact-3.2.5.pom (2.3 kB at 124 kB/s) 2024-03-09T14:38:58.858 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.5/maven-3.2.5.pom 2024-03-09T14:38:58.877 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/22 kB Progress (1): 22 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.5/maven-3.2.5.pom (22 kB at 1.1 MB/s) 2024-03-09T14:38:58.882 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/25/maven-parent-25.pom 2024-03-09T14:38:58.901 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/37 kB Progress (1): 33/37 kB Progress (1): 37 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/25/maven-parent-25.pom (37 kB at 1.9 MB/s) 2024-03-09T14:38:58.906 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/apache/15/apache-15.pom 2024-03-09T14:38:58.924 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 15 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/apache/15/apache-15.pom (15 kB at 846 kB/s) 2024-03-09T14:38:58.930 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/4.0.0/plexus-utils-4.0.0.pom 2024-03-09T14:38:58.948 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 8.7 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/4.0.0/plexus-utils-4.0.0.pom (8.7 kB at 456 kB/s) 2024-03-09T14:38:58.951 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/13/plexus-13.pom 2024-03-09T14:38:58.971 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/27 kB Progress (1): 27 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/13/plexus-13.pom (27 kB at 1.4 MB/s) 2024-03-09T14:38:58.977 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/3.2.5/maven-core-3.2.5.pom 2024-03-09T14:38:58.995 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 8.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/3.2.5/maven-core-3.2.5.pom (8.1 kB at 424 kB/s) 2024-03-09T14:38:59.002 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/3.2.5/maven-settings-3.2.5.pom 2024-03-09T14:38:59.020 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/3.2.5/maven-settings-3.2.5.pom (2.2 kB at 114 kB/s) 2024-03-09T14:38:59.026 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings-builder/3.2.5/maven-settings-builder-3.2.5.pom 2024-03-09T14:38:59.044 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.6 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings-builder/3.2.5/maven-settings-builder-3.2.5.pom (2.6 kB at 144 kB/s) 2024-03-09T14:38:59.050 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.21/plexus-interpolation-1.21.pom 2024-03-09T14:38:59.068 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.5 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.21/plexus-interpolation-1.21.pom (1.5 kB at 86 kB/s) 2024-03-09T14:38:59.071 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-components/1.3.1/plexus-components-1.3.1.pom 2024-03-09T14:38:59.088 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-components/1.3.1/plexus-components-1.3.1.pom (3.1 kB at 170 kB/s) 2024-03-09T14:38:59.091 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom 2024-03-09T14:38:59.110 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/20 kB Progress (1): 20 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom (20 kB at 1.1 MB/s) 2024-03-09T14:38:59.114 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom 2024-03-09T14:38:59.133 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom (6.8 kB at 356 kB/s) 2024-03-09T14:38:59.137 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom 2024-03-09T14:38:59.159 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 14 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom (14 kB at 616 kB/s) 2024-03-09T14:38:59.165 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.pom 2024-03-09T14:38:59.193 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.pom (3.0 kB at 102 kB/s) 2024-03-09T14:38:59.197 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/12/spice-parent-12.pom 2024-03-09T14:38:59.215 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/12/spice-parent-12.pom (6.8 kB at 378 kB/s) 2024-03-09T14:38:59.219 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/4/forge-parent-4.pom 2024-03-09T14:38:59.237 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 8.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/4/forge-parent-4.pom (8.4 kB at 442 kB/s) 2024-03-09T14:38:59.244 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.pom 2024-03-09T14:38:59.262 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.pom (2.1 kB at 109 kB/s) 2024-03-09T14:38:59.268 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/3.2.5/maven-repository-metadata-3.2.5.pom 2024-03-09T14:38:59.285 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/3.2.5/maven-repository-metadata-3.2.5.pom (2.2 kB at 124 kB/s) 2024-03-09T14:38:59.292 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.5/maven-plugin-api-3.2.5.pom 2024-03-09T14:38:59.310 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.5/maven-plugin-api-3.2.5.pom (3.0 kB at 168 kB/s) 2024-03-09T14:38:59.320 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/sisu/org.eclipse.sisu.plexus/0.9.0.M2/org.eclipse.sisu.plexus-0.9.0.M2.pom 2024-03-09T14:38:59.341 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 15 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/sisu/org.eclipse.sisu.plexus/0.9.0.M2/org.eclipse.sisu.plexus-0.9.0.M2.pom (15 kB at 713 kB/s) 2024-03-09T14:38:59.347 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/sisu/sisu-plexus/0.9.0.M2/sisu-plexus-0.9.0.M2.pom 2024-03-09T14:38:59.367 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 15 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/sisu/sisu-plexus/0.9.0.M2/sisu-plexus-0.9.0.M2.pom (15 kB at 763 kB/s) 2024-03-09T14:38:59.374 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.pom 2024-03-09T14:38:59.394 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 13 kB Downloaded from central: https://repo.maven.apache.org/maven2/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.pom (13 kB at 672 kB/s) 2024-03-09T14:38:59.398 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/net/java/jvnet-parent/3/jvnet-parent-3.pom 2024-03-09T14:38:59.416 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/net/java/jvnet-parent/3/jvnet-parent-3.pom (4.8 kB at 252 kB/s) 2024-03-09T14:38:59.421 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/javax/enterprise/cdi-api/1.2/cdi-api-1.2.pom 2024-03-09T14:38:59.439 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.3 kB Downloaded from central: https://repo.maven.apache.org/maven2/javax/enterprise/cdi-api/1.2/cdi-api-1.2.pom (6.3 kB at 349 kB/s) 2024-03-09T14:38:59.443 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/jboss/weld/weld-parent/26/weld-parent-26.pom 2024-03-09T14:38:59.463 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/32 kB Progress (1): 32 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/jboss/weld/weld-parent/26/weld-parent-26.pom (32 kB at 1.6 MB/s) 2024-03-09T14:38:59.470 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/sisu/org.eclipse.sisu.inject/0.9.0.M2/org.eclipse.sisu.inject-0.9.0.M2.pom 2024-03-09T14:38:59.488 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/17 kB Progress (1): 17 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/sisu/org.eclipse.sisu.inject/0.9.0.M2/org.eclipse.sisu.inject-0.9.0.M2.pom (17 kB at 913 kB/s) 2024-03-09T14:38:59.493 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/sisu/sisu-inject/0.9.0.M2/sisu-inject-0.9.0.M2.pom 2024-03-09T14:38:59.512 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 15 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/sisu/sisu-inject/0.9.0.M2/sisu-inject-0.9.0.M2.pom (15 kB at 779 kB/s) 2024-03-09T14:38:59.518 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-component-annotations/2.1.0/plexus-component-annotations-2.1.0.pom 2024-03-09T14:38:59.536 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 750 B Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-component-annotations/2.1.0/plexus-component-annotations-2.1.0.pom (750 B at 42 kB/s) 2024-03-09T14:38:59.539 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-containers/2.1.0/plexus-containers-2.1.0.pom 2024-03-09T14:38:59.558 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-containers/2.1.0/plexus-containers-2.1.0.pom (4.8 kB at 253 kB/s) 2024-03-09T14:38:59.563 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.6.0/plexus-classworlds-2.6.0.pom 2024-03-09T14:38:59.581 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 7.9 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.6.0/plexus-classworlds-2.6.0.pom (7.9 kB at 440 kB/s) 2024-03-09T14:38:59.587 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-model-builder/3.2.5/maven-model-builder-3.2.5.pom 2024-03-09T14:38:59.605 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-model-builder/3.2.5/maven-model-builder-3.2.5.pom (3.0 kB at 158 kB/s) 2024-03-09T14:38:59.612 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-aether-provider/3.2.5/maven-aether-provider-3.2.5.pom 2024-03-09T14:38:59.631 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-aether-provider/3.2.5/maven-aether-provider-3.2.5.pom (4.2 kB at 224 kB/s) 2024-03-09T14:38:59.638 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-spi/1.0.0.v20140518/aether-spi-1.0.0.v20140518.pom 2024-03-09T14:38:59.657 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-spi/1.0.0.v20140518/aether-spi-1.0.0.v20140518.pom (2.1 kB at 114 kB/s) 2024-03-09T14:38:59.663 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-impl/1.0.0.v20140518/aether-impl-1.0.0.v20140518.pom 2024-03-09T14:38:59.680 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.5 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-impl/1.0.0.v20140518/aether-impl-1.0.0.v20140518.pom (3.5 kB at 193 kB/s) 2024-03-09T14:38:59.686 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/3.2.3/sisu-guice-3.2.3.pom 2024-03-09T14:38:59.705 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 11 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/3.2.3/sisu-guice-3.2.3.pom (11 kB at 576 kB/s) 2024-03-09T14:38:59.709 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-parent/3.2.3/guice-parent-3.2.3.pom 2024-03-09T14:38:59.728 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 13 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-parent/3.2.3/guice-parent-3.2.3.pom (13 kB at 709 kB/s) 2024-03-09T14:38:59.732 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/38/forge-parent-38.pom 2024-03-09T14:38:59.752 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/19 kB Progress (1): 19 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/38/forge-parent-38.pom (19 kB at 986 kB/s) 2024-03-09T14:38:59.758 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/aopalliance/aopalliance/1.0/aopalliance-1.0.pom 2024-03-09T14:38:59.776 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 363 B Downloaded from central: https://repo.maven.apache.org/maven2/aopalliance/aopalliance/1.0/aopalliance-1.0.pom (363 B at 19 kB/s) 2024-03-09T14:38:59.780 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/com/google/guava/guava/16.0.1/guava-16.0.1.pom 2024-03-09T14:38:59.799 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.1 kB Downloaded from central: https://repo.maven.apache.org/maven2/com/google/guava/guava/16.0.1/guava-16.0.1.pom (6.1 kB at 321 kB/s) 2024-03-09T14:38:59.803 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/16.0.1/guava-parent-16.0.1.pom 2024-03-09T14:38:59.821 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 7.3 kB Downloaded from central: https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/16.0.1/guava-parent-16.0.1.pom (7.3 kB at 408 kB/s) 2024-03-09T14:38:59.825 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/7/oss-parent-7.pom 2024-03-09T14:38:59.843 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/7/oss-parent-7.pom (4.8 kB at 254 kB/s) 2024-03-09T14:38:59.849 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.5.2/plexus-classworlds-2.5.2.pom 2024-03-09T14:38:59.868 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 7.3 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.5.2/plexus-classworlds-2.5.2.pom (7.3 kB at 365 kB/s) 2024-03-09T14:38:59.874 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.15.0/commons-io-2.15.0.pom 2024-03-09T14:38:59.894 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/22 kB Progress (1): 22 kB Downloaded from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.15.0/commons-io-2.15.0.pom (22 kB at 1.1 MB/s) 2024-03-09T14:38:59.898 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/64/commons-parent-64.pom 2024-03-09T14:38:59.922 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/78 kB Progress (1): 33/78 kB Progress (1): 49/78 kB Progress (1): 66/78 kB Progress (1): 78 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/64/commons-parent-64.pom (78 kB at 3.2 MB/s) 2024-03-09T14:38:59.932 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/apache/30/apache-30.pom 2024-03-09T14:38:59.952 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/23 kB Progress (1): 23 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/apache/30/apache-30.pom (23 kB at 1.2 MB/s) 2024-03-09T14:38:59.959 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/junit-bom/5.10.0/junit-bom-5.10.0.pom 2024-03-09T14:38:59.978 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.6 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/junit-bom/5.10.0/junit-bom-5.10.0.pom (5.6 kB at 314 kB/s) 2024-03-09T14:38:59.982 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-java/1.2.0/plexus-java-1.2.0.pom 2024-03-09T14:39:00.001 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.3 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-java/1.2.0/plexus-java-1.2.0.pom (4.3 kB at 237 kB/s) 2024-03-09T14:39:00.004 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-languages/1.2.0/plexus-languages-1.2.0.pom 2024-03-09T14:39:00.022 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-languages/1.2.0/plexus-languages-1.2.0.pom (3.2 kB at 177 kB/s) 2024-03-09T14:39:00.026 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/15/plexus-15.pom 2024-03-09T14:39:00.045 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/28 kB Progress (1): 28 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/15/plexus-15.pom (28 kB at 1.5 MB/s) 2024-03-09T14:39:00.052 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/ow2/asm/asm/9.6/asm-9.6.pom 2024-03-09T14:39:00.070 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.4 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/ow2/asm/asm/9.6/asm-9.6.pom (2.4 kB at 139 kB/s) 2024-03-09T14:39:00.087 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/3.2.3/maven-surefire-common-3.2.3.jar 2024-03-09T14:39:00.137 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/308 kB Progress (1): 33/308 kB Progress (1): 49/308 kB Progress (1): 66/308 kB Progress (1): 82/308 kB Progress (1): 98/308 kB Progress (1): 115/308 kB Progress (1): 131/308 kB Progress (1): 147/308 kB Progress (1): 164/308 kB Progress (1): 180/308 kB Progress (1): 197/308 kB Progress (1): 213/308 kB Progress (1): 229/308 kB Progress (1): 246/308 kB Progress (1): 256/308 kB Progress (1): 272/308 kB Progress (1): 289/308 kB Progress (1): 305/308 kB Progress (1): 308 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/3.2.3/maven-surefire-common-3.2.3.jar (308 kB at 6.0 MB/s) 2024-03-09T14:39:00.138 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/3.2.3/surefire-api-3.2.3.jar 2024-03-09T14:39:00.138 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-logger-api/3.2.3/surefire-logger-api-3.2.3.jar 2024-03-09T14:39:00.139 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-extensions-api/3.2.3/surefire-extensions-api-3.2.3.jar 2024-03-09T14:39:00.139 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/3.2.3/surefire-booter-3.2.3.jar 2024-03-09T14:39:00.139 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-extensions-spi/3.2.3/surefire-extensions-spi-3.2.3.jar 2024-03-09T14:39:00.160 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/171 kB Progress (2): 16/171 kB | 14 kB Progress (2): 33/171 kB | 14 kB Progress (3): 33/171 kB | 14 kB | 8.2 kB Progress (3): 49/171 kB | 14 kB | 8.2 kB Progress (4): 49/171 kB | 14 kB | 8.2 kB | 16/118 kB Progress (4): 66/171 kB | 14 kB | 8.2 kB | 16/118 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-extensions-spi/3.2.3/surefire-extensions-spi-3.2.3.jar (8.2 kB at 409 kB/s) 2024-03-09T14:39:00.160 INFO:tasks.workunit.client.0.smithi001.stdout:Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-logger-api/3.2.3/surefire-logger-api-3.2.3.jar (14 kB at 618 kB/s) 2024-03-09T14:39:00.160 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 66/171 kB | 33/118 kB Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/1.0.0.v20140518/aether-api-1.0.0.v20140518.jar 2024-03-09T14:39:00.160 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-util/1.0.0.v20140518/aether-util-1.0.0.v20140518.jar 2024-03-09T14:39:00.162 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 66/171 kB | 33/118 kB | 16/26 kB Progress (3): 82/171 kB | 33/118 kB | 16/26 kB Progress (3): 82/171 kB | 33/118 kB | 26 kB Progress (3): 98/171 kB | 33/118 kB | 26 kB Progress (3): 98/171 kB | 49/118 kB | 26 kB Progress (3): 115/171 kB | 49/118 kB | 26 kB Progress (3): 131/171 kB | 49/118 kB | 26 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-extensions-api/3.2.3/surefire-extensions-api-3.2.3.jar (26 kB at 1.1 MB/s) 2024-03-09T14:39:00.162 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 131/171 kB | 66/118 kB Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-common-artifact-filters/3.1.1/maven-common-artifact-filters-3.1.1.jar 2024-03-09T14:39:00.163 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 147/171 kB | 66/118 kB Progress (2): 147/171 kB | 82/118 kB Progress (2): 147/171 kB | 98/118 kB Progress (2): 164/171 kB | 98/118 kB Progress (2): 171 kB | 98/118 kB Progress (2): 171 kB | 115/118 kB Progress (2): 171 kB | 118 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/3.2.3/surefire-api-3.2.3.jar (171 kB at 6.6 MB/s) 2024-03-09T14:39:00.163 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.15.0/commons-io-2.15.0.jar 2024-03-09T14:39:00.163 INFO:tasks.workunit.client.0.smithi001.stdout:Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/3.2.3/surefire-booter-3.2.3.jar (118 kB at 4.7 MB/s) 2024-03-09T14:39:00.163 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-java/1.2.0/plexus-java-1.2.0.jar 2024-03-09T14:39:00.182 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/146 kB Progress (1): 32/146 kB Progress (2): 32/146 kB | 16/136 kB Progress (2): 49/146 kB | 16/136 kB Progress (2): 65/146 kB | 16/136 kB Progress (2): 65/146 kB | 33/136 kB Progress (2): 81/146 kB | 33/136 kB Progress (3): 81/146 kB | 33/136 kB | 16/61 kB Progress (3): 98/146 kB | 33/136 kB | 16/61 kB Progress (3): 98/146 kB | 49/136 kB | 16/61 kB Progress (3): 114/146 kB | 49/136 kB | 16/61 kB Progress (4): 114/146 kB | 49/136 kB | 16/61 kB | 16/58 kB Progress (4): 114/146 kB | 49/136 kB | 16/61 kB | 33/58 kB Progress (4): 114/146 kB | 49/136 kB | 32/61 kB | 33/58 kB Progress (4): 131/146 kB | 49/136 kB | 32/61 kB | 33/58 kB Progress (4): 131/146 kB | 49/136 kB | 32/61 kB | 49/58 kB Progress (4): 131/146 kB | 49/136 kB | 49/61 kB | 49/58 kB Progress (4): 146 kB | 49/136 kB | 49/61 kB | 49/58 kB Progress (4): 146 kB | 49/136 kB | 49/61 kB | 58 kB Progress (5): 146 kB | 49/136 kB | 49/61 kB | 58 kB | 16/501 kB Progress (5): 146 kB | 66/136 kB | 49/61 kB | 58 kB | 16/501 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-util/1.0.0.v20140518/aether-util-1.0.0.v20140518.jar (146 kB at 3.4 MB/s) 2024-03-09T14:39:00.183 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (4): 66/136 kB | 61 kB | 58 kB | 16/501 kB Downloading from central: https://repo.maven.apache.org/maven2/org/ow2/asm/asm/9.6/asm-9.6.jar 2024-03-09T14:39:00.183 INFO:tasks.workunit.client.0.smithi001.stdout:Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-java/1.2.0/plexus-java-1.2.0.jar (58 kB at 1.4 MB/s) 2024-03-09T14:39:00.183 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-shared-utils/3.2.3/surefire-shared-utils-3.2.3.jar 2024-03-09T14:39:00.183 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 66/136 kB | 61 kB | 33/501 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-common-artifact-filters/3.1.1/maven-common-artifact-filters-3.1.1.jar (61 kB at 1.4 MB/s) 2024-03-09T14:39:00.187 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 82/136 kB | 33/501 kB Progress (2): 82/136 kB | 49/501 kB Progress (2): 82/136 kB | 66/501 kB Progress (2): 98/136 kB | 66/501 kB Progress (2): 98/136 kB | 82/501 kB Progress (2): 98/136 kB | 98/501 kB Progress (2): 115/136 kB | 98/501 kB Progress (2): 115/136 kB | 115/501 kB Progress (2): 115/136 kB | 131/501 kB Progress (2): 131/136 kB | 131/501 kB Progress (2): 136 kB | 131/501 kB Progress (2): 136 kB | 147/501 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/1.0.0.v20140518/aether-api-1.0.0.v20140518.jar (136 kB at 2.8 MB/s) 2024-03-09T14:39:00.206 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 164/501 kB Progress (1): 180/501 kB Progress (1): 197/501 kB Progress (1): 213/501 kB Progress (1): 229/501 kB Progress (1): 246/501 kB Progress (1): 256/501 kB Progress (1): 272/501 kB Progress (1): 289/501 kB Progress (1): 305/501 kB Progress (1): 321/501 kB Progress (1): 338/501 kB Progress (1): 354/501 kB Progress (1): 370/501 kB Progress (1): 387/501 kB Progress (2): 387/501 kB | 16/124 kB Progress (2): 403/501 kB | 16/124 kB Progress (2): 403/501 kB | 33/124 kB Progress (2): 420/501 kB | 33/124 kB Progress (3): 420/501 kB | 33/124 kB | 0/2.4 MB Progress (3): 420/501 kB | 49/124 kB | 0/2.4 MB Progress (3): 420/501 kB | 49/124 kB | 0/2.4 MB Progress (3): 420/501 kB | 49/124 kB | 0/2.4 MB Progress (3): 436/501 kB | 49/124 kB | 0/2.4 MB Progress (3): 436/501 kB | 49/124 kB | 0.1/2.4 MB Progress (3): 452/501 kB | 49/124 kB | 0.1/2.4 MB Progress (3): 452/501 kB | 49/124 kB | 0.1/2.4 MB Progress (3): 452/501 kB | 66/124 kB | 0.1/2.4 MB Progress (3): 452/501 kB | 66/124 kB | 0.1/2.4 MB Progress (3): 469/501 kB | 66/124 kB | 0.1/2.4 MB Progress (3): 469/501 kB | 66/124 kB | 0.1/2.4 MB Progress (3): 469/501 kB | 66/124 kB | 0.1/2.4 MB Progress (3): 469/501 kB | 82/124 kB | 0.1/2.4 MB Progress (3): 485/501 kB | 82/124 kB | 0.1/2.4 MB Progress (3): 485/501 kB | 82/124 kB | 0.1/2.4 MB Progress (3): 501 kB | 82/124 kB | 0.1/2.4 MB Progress (3): 501 kB | 82/124 kB | 0.2/2.4 MB Progress (3): 501 kB | 82/124 kB | 0.2/2.4 MB Downloaded from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.15.0/commons-io-2.15.0.jar (501 kB at 7.5 MB/s) 2024-03-09T14:39:00.208 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 82/124 kB | 0.2/2.4 MB Progress (2): 98/124 kB | 0.2/2.4 MB Progress (2): 98/124 kB | 0.2/2.4 MB Progress (2): 98/124 kB | 0.2/2.4 MB Progress (2): 98/124 kB | 0.2/2.4 MB Progress (2): 115/124 kB | 0.2/2.4 MB Progress (2): 115/124 kB | 0.3/2.4 MB Progress (2): 115/124 kB | 0.3/2.4 MB Progress (2): 124 kB | 0.3/2.4 MB Progress (2): 124 kB | 0.3/2.4 MB Downloaded from central: https://repo.maven.apache.org/maven2/org/ow2/asm/asm/9.6/asm-9.6.jar (124 kB at 1.8 MB/s) 2024-03-09T14:39:00.253 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 0.3/2.4 MB Progress (1): 0.3/2.4 MB Progress (1): 0.3/2.4 MB Progress (1): 0.4/2.4 MB Progress (1): 0.4/2.4 MB Progress (1): 0.4/2.4 MB Progress (1): 0.4/2.4 MB Progress (1): 0.4/2.4 MB Progress (1): 0.4/2.4 MB Progress (1): 0.5/2.4 MB Progress (1): 0.5/2.4 MB Progress (1): 0.5/2.4 MB Progress (1): 0.5/2.4 MB Progress (1): 0.5/2.4 MB Progress (1): 0.5/2.4 MB Progress (1): 0.6/2.4 MB Progress (1): 0.6/2.4 MB Progress (1): 0.6/2.4 MB Progress (1): 0.6/2.4 MB Progress (1): 0.6/2.4 MB Progress (1): 0.6/2.4 MB Progress (1): 0.6/2.4 MB Progress (1): 0.7/2.4 MB Progress (1): 0.7/2.4 MB Progress (1): 0.7/2.4 MB Progress (1): 0.7/2.4 MB Progress (1): 0.7/2.4 MB Progress (1): 0.7/2.4 MB Progress (1): 0.8/2.4 MB Progress (1): 0.8/2.4 MB Progress (1): 0.8/2.4 MB Progress (1): 0.8/2.4 MB Progress (1): 0.8/2.4 MB Progress (1): 0.8/2.4 MB Progress (1): 0.9/2.4 MB Progress (1): 0.9/2.4 MB Progress (1): 0.9/2.4 MB Progress (1): 0.9/2.4 MB Progress (1): 0.9/2.4 MB Progress (1): 0.9/2.4 MB Progress (1): 1.0/2.4 MB Progress (1): 1.0/2.4 MB Progress (1): 1.0/2.4 MB Progress (1): 1.0/2.4 MB Progress (1): 1.0/2.4 MB Progress (1): 1.0/2.4 MB Progress (1): 1.1/2.4 MB Progress (1): 1.1/2.4 MB Progress (1): 1.1/2.4 MB Progress (1): 1.1/2.4 MB Progress (1): 1.1/2.4 MB Progress (1): 1.1/2.4 MB Progress (1): 1.2/2.4 MB Progress (1): 1.2/2.4 MB Progress (1): 1.2/2.4 MB Progress (1): 1.2/2.4 MB Progress (1): 1.2/2.4 MB Progress (1): 1.2/2.4 MB Progress (1): 1.3/2.4 MB Progress (1): 1.3/2.4 MB Progress (1): 1.3/2.4 MB Progress (1): 1.3/2.4 MB Progress (1): 1.3/2.4 MB Progress (1): 1.3/2.4 MB Progress (1): 1.3/2.4 MB Progress (1): 1.4/2.4 MB Progress (1): 1.4/2.4 MB Progress (1): 1.4/2.4 MB Progress (1): 1.4/2.4 MB Progress (1): 1.4/2.4 MB Progress (1): 1.4/2.4 MB Progress (1): 1.5/2.4 MB Progress (1): 1.5/2.4 MB Progress (1): 1.5/2.4 MB Progress (1): 1.5/2.4 MB Progress (1): 1.5/2.4 MB Progress (1): 1.5/2.4 MB Progress (1): 1.6/2.4 MB Progress (1): 1.6/2.4 MB Progress (1): 1.6/2.4 MB Progress (1): 1.6/2.4 MB Progress (1): 1.6/2.4 MB Progress (1): 1.6/2.4 MB Progress (1): 1.7/2.4 MB Progress (1): 1.7/2.4 MB Progress (1): 1.7/2.4 MB Progress (1): 1.7/2.4 MB Progress (1): 1.7/2.4 MB Progress (1): 1.7/2.4 MB Progress (1): 1.8/2.4 MB Progress (1): 1.8/2.4 MB Progress (1): 1.8/2.4 MB Progress (1): 1.8/2.4 MB Progress (1): 1.8/2.4 MB Progress (1): 1.8/2.4 MB Progress (1): 1.9/2.4 MB Progress (1): 1.9/2.4 MB Progress (1): 1.9/2.4 MB Progress (1): 1.9/2.4 MB Progress (1): 1.9/2.4 MB Progress (1): 1.9/2.4 MB Progress (1): 1.9/2.4 MB Progress (1): 2.0/2.4 MB Progress (1): 2.0/2.4 MB Progress (1): 2.0/2.4 MB Progress (1): 2.0/2.4 MB Progress (1): 2.0/2.4 MB Progress (1): 2.0/2.4 MB Progress (1): 2.1/2.4 MB Progress (1): 2.1/2.4 MB Progress (1): 2.1/2.4 MB Progress (1): 2.1/2.4 MB Progress (1): 2.1/2.4 MB Progress (1): 2.1/2.4 MB Progress (1): 2.2/2.4 MB Progress (1): 2.2/2.4 MB Progress (1): 2.2/2.4 MB Progress (1): 2.2/2.4 MB Progress (1): 2.2/2.4 MB Progress (1): 2.2/2.4 MB Progress (1): 2.3/2.4 MB Progress (1): 2.3/2.4 MB Progress (1): 2.3/2.4 MB Progress (1): 2.3/2.4 MB Progress (1): 2.3/2.4 MB Progress (1): 2.3/2.4 MB Progress (1): 2.4/2.4 MB Progress (1): 2.4/2.4 MB Progress (1): 2.4/2.4 MB Progress (1): 2.4/2.4 MB Progress (1): 2.4/2.4 MB Progress (1): 2.4/2.4 MB Progress (1): 2.4 MB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-shared-utils/3.2.3/surefire-shared-utils-3.2.3.jar (2.4 MB at 22 MB/s) 2024-03-09T14:39:00.355 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:39:00.355 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --- jar:3.3.0:jar (default-jar) @ jcksum --- 2024-03-09T14:39:00.358 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/file-management/3.1.0/file-management-3.1.0.pom 2024-03-09T14:39:00.375 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.5 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/file-management/3.1.0/file-management-3.1.0.pom (4.5 kB at 264 kB/s) 2024-03-09T14:39:00.377 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-components/36/maven-shared-components-36.pom 2024-03-09T14:39:00.394 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 4.9 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-components/36/maven-shared-components-36.pom (4.9 kB at 288 kB/s) 2024-03-09T14:39:00.396 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/36/maven-parent-36.pom 2024-03-09T14:39:00.413 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/45 kB Progress (1): 33/45 kB Progress (1): 45 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/36/maven-parent-36.pom (45 kB at 2.7 MB/s) 2024-03-09T14:39:00.416 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/apache/26/apache-26.pom 2024-03-09T14:39:00.433 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/21 kB Progress (1): 21 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/apache/26/apache-26.pom (21 kB at 1.2 MB/s) 2024-03-09T14:39:00.436 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.3.0/plexus-utils-3.3.0.pom 2024-03-09T14:39:00.454 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 5.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.3.0/plexus-utils-3.3.0.pom (5.2 kB at 305 kB/s) 2024-03-09T14:39:00.456 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-archiver/3.6.0/maven-archiver-3.6.0.pom 2024-03-09T14:39:00.473 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 3.9 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-archiver/3.6.0/maven-archiver-3.6.0.pom (3.9 kB at 231 kB/s) 2024-03-09T14:39:00.476 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-io/3.4.0/plexus-io-3.4.0.pom 2024-03-09T14:39:00.493 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-io/3.4.0/plexus-io-3.4.0.pom (6.0 kB at 354 kB/s) 2024-03-09T14:39:00.496 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/4.4.0/plexus-archiver-4.4.0.pom 2024-03-09T14:39:00.513 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 6.3 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/4.4.0/plexus-archiver-4.4.0.pom (6.3 kB at 368 kB/s) 2024-03-09T14:39:00.517 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-compress/1.21/commons-compress-1.21.pom 2024-03-09T14:39:00.534 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/20 kB Progress (1): 20 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-compress/1.21/commons-compress-1.21.pom (20 kB at 1.2 MB/s) 2024-03-09T14:39:00.538 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/iq80/snappy/snappy/0.4/snappy-0.4.pom 2024-03-09T14:39:00.555 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 15 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/iq80/snappy/snappy/0.4/snappy-0.4.pom (15 kB at 855 kB/s) 2024-03-09T14:39:00.558 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/tukaani/xz/1.9/xz-1.9.pom 2024-03-09T14:39:00.575 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 2.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/tukaani/xz/1.9/xz-1.9.pom (2.0 kB at 120 kB/s) 2024-03-09T14:39:00.577 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.4.2/plexus-utils-3.4.2.pom 2024-03-09T14:39:00.593 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 8.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.4.2/plexus-utils-3.4.2.pom (8.2 kB at 514 kB/s) 2024-03-09T14:39:00.596 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/8/plexus-8.pom 2024-03-09T14:39:00.613 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/25 kB Progress (1): 25 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/8/plexus-8.pom (25 kB at 1.5 MB/s) 2024-03-09T14:39:00.620 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/file-management/3.1.0/file-management-3.1.0.jar 2024-03-09T14:39:00.637 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/36 kB Progress (1): 33/36 kB Progress (1): 36 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/file-management/3.1.0/file-management-3.1.0.jar (36 kB at 2.0 MB/s) 2024-03-09T14:39:00.638 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-archiver/3.6.0/maven-archiver-3.6.0.jar 2024-03-09T14:39:00.638 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-io/3.4.0/plexus-io-3.4.0.jar 2024-03-09T14:39:00.638 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/4.4.0/plexus-archiver-4.4.0.jar 2024-03-09T14:39:00.638 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-compress/1.21/commons-compress-1.21.jar 2024-03-09T14:39:00.639 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/iq80/snappy/snappy/0.4/snappy-0.4.jar 2024-03-09T14:39:00.657 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/79 kB Progress (2): 16/79 kB | 16/26 kB Progress (3): 16/79 kB | 16/26 kB | 16/211 kB Progress (3): 16/79 kB | 26 kB | 16/211 kB Progress (3): 16/79 kB | 26 kB | 33/211 kB Progress (3): 32/79 kB | 26 kB | 33/211 kB Progress (4): 32/79 kB | 26 kB | 33/211 kB | 16/58 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-archiver/3.6.0/maven-archiver-3.6.0.jar (26 kB at 1.3 MB/s) 2024-03-09T14:39:00.657 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/tukaani/xz/1.9/xz-1.9.jar 2024-03-09T14:39:00.659 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (3): 32/79 kB | 49/211 kB | 16/58 kB Progress (3): 32/79 kB | 49/211 kB | 33/58 kB Progress (3): 49/79 kB | 49/211 kB | 33/58 kB Progress (3): 49/79 kB | 66/211 kB | 33/58 kB Progress (3): 49/79 kB | 66/211 kB | 49/58 kB Progress (3): 49/79 kB | 66/211 kB | 58 kB Progress (3): 49/79 kB | 82/211 kB | 58 kB Progress (3): 65/79 kB | 82/211 kB | 58 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/iq80/snappy/snappy/0.4/snappy-0.4.jar (58 kB at 2.8 MB/s) 2024-03-09T14:39:00.659 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 65/79 kB | 98/211 kB Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.4.2/plexus-utils-3.4.2.jar 2024-03-09T14:39:00.660 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 79 kB | 98/211 kB Progress (2): 79 kB | 115/211 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-io/3.4.0/plexus-io-3.4.0.jar (79 kB at 3.4 MB/s) 2024-03-09T14:39:00.664 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 131/211 kB Progress (2): 131/211 kB | 0/1.0 MB Progress (2): 147/211 kB | 0/1.0 MB Progress (2): 147/211 kB | 0/1.0 MB Progress (2): 164/211 kB | 0/1.0 MB Progress (2): 164/211 kB | 0/1.0 MB Progress (2): 180/211 kB | 0/1.0 MB Progress (2): 197/211 kB | 0/1.0 MB Progress (2): 211 kB | 0/1.0 MB Progress (2): 211 kB | 0.1/1.0 MB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/4.4.0/plexus-archiver-4.4.0.jar (211 kB at 8.1 MB/s) 2024-03-09T14:39:00.678 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 0.1/1.0 MB Progress (1): 0.1/1.0 MB Progress (1): 0.1/1.0 MB Progress (1): 0.1/1.0 MB Progress (1): 0.1/1.0 MB Progress (1): 0.2/1.0 MB Progress (2): 0.2/1.0 MB | 16/116 kB Progress (2): 0.2/1.0 MB | 32/116 kB Progress (2): 0.2/1.0 MB | 32/116 kB Progress (2): 0.2/1.0 MB | 49/116 kB Progress (2): 0.2/1.0 MB | 65/116 kB Progress (2): 0.2/1.0 MB | 81/116 kB Progress (2): 0.2/1.0 MB | 81/116 kB Progress (2): 0.2/1.0 MB | 98/116 kB Progress (2): 0.2/1.0 MB | 114/116 kB Progress (3): 0.2/1.0 MB | 114/116 kB | 16/267 kB Progress (3): 0.2/1.0 MB | 116 kB | 16/267 kB Progress (3): 0.2/1.0 MB | 116 kB | 16/267 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/tukaani/xz/1.9/xz-1.9.jar (116 kB at 3.0 MB/s) 2024-03-09T14:39:00.696 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 0.2/1.0 MB | 32/267 kB Progress (2): 0.2/1.0 MB | 32/267 kB Progress (2): 0.2/1.0 MB | 49/267 kB Progress (2): 0.2/1.0 MB | 49/267 kB Progress (2): 0.2/1.0 MB | 65/267 kB Progress (2): 0.3/1.0 MB | 65/267 kB Progress (2): 0.3/1.0 MB | 65/267 kB Progress (2): 0.3/1.0 MB | 81/267 kB Progress (2): 0.3/1.0 MB | 98/267 kB Progress (2): 0.3/1.0 MB | 98/267 kB Progress (2): 0.3/1.0 MB | 98/267 kB Progress (2): 0.3/1.0 MB | 114/267 kB Progress (2): 0.3/1.0 MB | 130/267 kB Progress (2): 0.3/1.0 MB | 130/267 kB Progress (2): 0.3/1.0 MB | 147/267 kB Progress (2): 0.3/1.0 MB | 147/267 kB Progress (2): 0.3/1.0 MB | 163/267 kB Progress (2): 0.3/1.0 MB | 180/267 kB Progress (2): 0.3/1.0 MB | 180/267 kB Progress (2): 0.4/1.0 MB | 180/267 kB Progress (2): 0.4/1.0 MB | 180/267 kB Progress (2): 0.4/1.0 MB | 196/267 kB Progress (2): 0.4/1.0 MB | 212/267 kB Progress (2): 0.4/1.0 MB | 212/267 kB Progress (2): 0.4/1.0 MB | 229/267 kB Progress (2): 0.4/1.0 MB | 229/267 kB Progress (2): 0.4/1.0 MB | 245/267 kB Progress (2): 0.4/1.0 MB | 245/267 kB Progress (2): 0.4/1.0 MB | 262/267 kB Progress (2): 0.4/1.0 MB | 267 kB Progress (2): 0.4/1.0 MB | 267 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.4.2/plexus-utils-3.4.2.jar (267 kB at 4.6 MB/s) 2024-03-09T14:39:00.745 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 0.5/1.0 MB Progress (1): 0.5/1.0 MB Progress (1): 0.5/1.0 MB Progress (1): 0.5/1.0 MB Progress (1): 0.5/1.0 MB Progress (1): 0.5/1.0 MB Progress (1): 0.6/1.0 MB Progress (1): 0.6/1.0 MB Progress (1): 0.6/1.0 MB Progress (1): 0.6/1.0 MB Progress (1): 0.6/1.0 MB Progress (1): 0.6/1.0 MB Progress (1): 0.6/1.0 MB Progress (1): 0.7/1.0 MB Progress (1): 0.7/1.0 MB Progress (1): 0.7/1.0 MB Progress (1): 0.7/1.0 MB Progress (1): 0.7/1.0 MB Progress (1): 0.7/1.0 MB Progress (1): 0.8/1.0 MB Progress (1): 0.8/1.0 MB Progress (1): 0.8/1.0 MB Progress (1): 0.8/1.0 MB Progress (1): 0.8/1.0 MB Progress (1): 0.8/1.0 MB Progress (1): 0.9/1.0 MB Progress (1): 0.9/1.0 MB Progress (1): 0.9/1.0 MB Progress (1): 0.9/1.0 MB Progress (1): 0.9/1.0 MB Progress (1): 0.9/1.0 MB Progress (1): 1.0/1.0 MB Progress (1): 1.0/1.0 MB Progress (1): 1.0/1.0 MB Progress (1): 1.0/1.0 MB Progress (1): 1.0 MB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-compress/1.21/commons-compress-1.21.jar (1.0 MB at 9.5 MB/s) 2024-03-09T14:39:00.926 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Building jar: /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/target/jcksum-1.0.jar 2024-03-09T14:39:00.956 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] ------------------------------------------------------------------------ 2024-03-09T14:39:00.956 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] BUILD SUCCESS 2024-03-09T14:39:00.956 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] ------------------------------------------------------------------------ 2024-03-09T14:39:00.957 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Total time: 9.726 s 2024-03-09T14:39:00.957 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Finished at: 2024-03-09T14:39:00Z 2024-03-09T14:39:00.957 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] ------------------------------------------------------------------------ 2024-03-09T14:39:00.988 INFO:tasks.workunit.client.0.smithi001.stderr:+ ./mvnw test -Dtest=PutObjects 2024-03-09T14:39:00.993 INFO:tasks.workunit.client.0.smithi001.stdout:Warning: JAVA_HOME environment variable is not set. 2024-03-09T14:39:02.309 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Scanning for projects... 2024-03-09T14:39:02.417 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:39:02.417 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] ---------------------------< io.ceph:jcksum >--------------------------- 2024-03-09T14:39:02.417 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Building jcksum 1.0 2024-03-09T14:39:02.417 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] from pom.xml 2024-03-09T14:39:02.417 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --------------------------------[ jar ]--------------------------------- 2024-03-09T14:39:02.720 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:39:02.720 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --- resources:3.3.1:resources (default-resources) @ jcksum --- 2024-03-09T14:39:02.856 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] skip non existing resourceDirectory /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/main/resources 2024-03-09T14:39:02.857 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:39:02.857 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --- compiler:3.11.0:compile (default-compile) @ jcksum --- 2024-03-09T14:39:02.955 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Nothing to compile - all classes are up to date 2024-03-09T14:39:02.955 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:39:02.955 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --- resources:3.3.1:testResources (default-testResources) @ jcksum --- 2024-03-09T14:39:02.957 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] skip non existing resourceDirectory /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/test/resources 2024-03-09T14:39:02.957 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:39:02.957 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --- compiler:3.11.0:testCompile (default-testCompile) @ jcksum --- 2024-03-09T14:39:02.962 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Nothing to compile - all classes are up to date 2024-03-09T14:39:02.962 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:39:02.962 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] --- surefire:3.2.3:test (default-test) @ jcksum --- 2024-03-09T14:39:03.170 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider 2024-03-09T14:39:03.425 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit-platform/3.2.3/surefire-junit-platform-3.2.3.pom 2024-03-09T14:39:03.749 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.4/4.7 kB Progress (1): 2.8/4.7 kB Progress (1): 4.1/4.7 kB Progress (1): 4.7 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit-platform/3.2.3/surefire-junit-platform-3.2.3.pom (4.7 kB at 14 kB/s) 2024-03-09T14:39:03.763 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-providers/3.2.3/surefire-providers-3.2.3.pom 2024-03-09T14:39:03.805 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.4/2.6 kB Progress (1): 2.6 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-providers/3.2.3/surefire-providers-3.2.3.pom (2.6 kB at 62 kB/s) 2024-03-09T14:39:03.820 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-java5/3.2.3/common-java5-3.2.3.pom 2024-03-09T14:39:03.846 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.4/2.8 kB Progress (1): 2.8/2.8 kB Progress (1): 2.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-java5/3.2.3/common-java5-3.2.3.pom (2.8 kB at 104 kB/s) 2024-03-09T14:39:03.859 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-launcher/1.9.3/junit-platform-launcher-1.9.3.pom 2024-03-09T14:39:03.884 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.4/3.0 kB Progress (1): 2.8/3.0 kB Progress (1): 3.0 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-launcher/1.9.3/junit-platform-launcher-1.9.3.pom (3.0 kB at 116 kB/s) 2024-03-09T14:39:03.891 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-engine/1.9.3/junit-platform-engine-1.9.3.pom 2024-03-09T14:39:03.913 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.4/3.2 kB Progress (1): 2.8/3.2 kB Progress (1): 3.2 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-engine/1.9.3/junit-platform-engine-1.9.3.pom (3.2 kB at 139 kB/s) 2024-03-09T14:39:03.918 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.pom 2024-03-09T14:39:03.942 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.4/1.7 kB Progress (1): 1.7 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.pom (1.7 kB at 67 kB/s) 2024-03-09T14:39:03.946 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-commons/1.9.3/junit-platform-commons-1.9.3.pom 2024-03-09T14:39:03.968 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.4/2.8 kB Progress (1): 2.8/2.8 kB Progress (1): 2.8 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-commons/1.9.3/junit-platform-commons-1.9.3.pom (2.8 kB at 129 kB/s) 2024-03-09T14:39:03.978 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit-platform/3.2.3/surefire-junit-platform-3.2.3.jar 2024-03-09T14:39:04.010 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.4/27 kB Progress (1): 2.8/27 kB Progress (1): 4.1/27 kB Progress (1): 5.5/27 kB Progress (1): 6.9/27 kB Progress (1): 8.3/27 kB Progress (1): 9.7/27 kB Progress (1): 11/27 kB Progress (1): 12/27 kB Progress (1): 14/27 kB Progress (1): 15/27 kB Progress (1): 17/27 kB Progress (1): 18/27 kB Progress (1): 19/27 kB Progress (1): 21/27 kB Progress (1): 22/27 kB Progress (1): 23/27 kB Progress (1): 25/27 kB Progress (1): 26/27 kB Progress (1): 27 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit-platform/3.2.3/surefire-junit-platform-3.2.3.jar (27 kB at 823 kB/s) 2024-03-09T14:39:04.013 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-java5/3.2.3/common-java5-3.2.3.jar 2024-03-09T14:39:04.013 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-launcher/1.9.3/junit-platform-launcher-1.9.3.jar 2024-03-09T14:39:04.014 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-engine/1.9.3/junit-platform-engine-1.9.3.jar 2024-03-09T14:39:04.018 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.jar 2024-03-09T14:39:04.018 INFO:tasks.workunit.client.0.smithi001.stdout:Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-commons/1.9.3/junit-platform-commons-1.9.3.jar 2024-03-09T14:39:04.049 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 16/18 kB Progress (1): 18 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-java5/3.2.3/common-java5-3.2.3.jar (18 kB at 462 kB/s) 2024-03-09T14:39:04.095 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 1.4/7.7 kB Progress (1): 2.8/7.7 kB Progress (2): 2.8/7.7 kB | 1.4/103 kB Progress (2): 4.1/7.7 kB | 1.4/103 kB Progress (2): 4.1/7.7 kB | 2.8/103 kB Progress (2): 5.5/7.7 kB | 2.8/103 kB Progress (2): 5.5/7.7 kB | 4.1/103 kB Progress (2): 6.9/7.7 kB | 4.1/103 kB Progress (2): 6.9/7.7 kB | 5.5/103 kB Progress (2): 7.7 kB | 5.5/103 kB Progress (2): 7.7 kB | 6.9/103 kB Progress (2): 7.7 kB | 8.3/103 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.jar (7.7 kB at 94 kB/s) 2024-03-09T14:39:04.131 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (1): 9.7/103 kB Progress (1): 11/103 kB Progress (1): 12/103 kB Progress (2): 12/103 kB | 16/189 kB Progress (2): 14/103 kB | 16/189 kB Progress (3): 14/103 kB | 16/189 kB | 16/169 kB Progress (3): 15/103 kB | 16/189 kB | 16/169 kB Progress (3): 16/103 kB | 16/189 kB | 16/169 kB Progress (3): 16/103 kB | 33/189 kB | 16/169 kB Progress (3): 16/103 kB | 33/189 kB | 33/169 kB Progress (3): 32/103 kB | 33/189 kB | 33/169 kB Progress (3): 32/103 kB | 49/189 kB | 33/169 kB Progress (3): 49/103 kB | 49/189 kB | 33/169 kB Progress (3): 49/103 kB | 49/189 kB | 49/169 kB Progress (3): 49/103 kB | 65/189 kB | 49/169 kB Progress (3): 65/103 kB | 65/189 kB | 49/169 kB Progress (3): 65/103 kB | 65/189 kB | 66/169 kB Progress (3): 65/103 kB | 82/189 kB | 66/169 kB Progress (3): 81/103 kB | 82/189 kB | 66/169 kB Progress (3): 81/103 kB | 82/189 kB | 82/169 kB Progress (3): 98/103 kB | 82/189 kB | 82/169 kB Progress (3): 98/103 kB | 82/189 kB | 98/169 kB Progress (3): 98/103 kB | 98/189 kB | 98/169 kB Progress (3): 103 kB | 98/189 kB | 98/169 kB Progress (3): 103 kB | 98/189 kB | 115/169 kB Progress (3): 103 kB | 114/189 kB | 115/169 kB Progress (3): 103 kB | 114/189 kB | 131/169 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-commons/1.9.3/junit-platform-commons-1.9.3.jar (103 kB at 888 kB/s) 2024-03-09T14:39:04.146 INFO:tasks.workunit.client.0.smithi001.stdout:Progress (2): 131/189 kB | 131/169 kB Progress (2): 147/189 kB | 131/169 kB Progress (2): 164/189 kB | 131/169 kB Progress (2): 164/189 kB | 147/169 kB Progress (2): 180/189 kB | 147/169 kB Progress (2): 180/189 kB | 164/169 kB Progress (2): 180/189 kB | 169 kB Progress (2): 189 kB | 169 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-engine/1.9.3/junit-platform-engine-1.9.3.jar (189 kB at 1.4 MB/s) 2024-03-09T14:39:04.147 INFO:tasks.workunit.client.0.smithi001.stdout:Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-launcher/1.9.3/junit-platform-launcher-1.9.3.jar (169 kB at 1.3 MB/s) 2024-03-09T14:39:04.175 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:39:04.175 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] ------------------------------------------------------- 2024-03-09T14:39:04.175 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] T E S T S 2024-03-09T14:39:04.175 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] ------------------------------------------------------- 2024-03-09T14:39:04.771 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Running io.ceph.jcksum.PutObjects 2024-03-09T14:39:04.851 INFO:tasks.workunit.client.0.smithi001.stdout:PutObjects.java: starting test run: 2024-03-09T14:39:04.851 INFO:tasks.workunit.client.0.smithi001.stdout: AccessKey=lNCnR47C2g+ZidCWBAUuwfSAA7Q= 2024-03-09T14:39:04.851 INFO:tasks.workunit.client.0.smithi001.stdout: SecretKey=tYuA2Y+Uu1ow2l9Xe59tWKVml3gMuVfyhUjjJwfwEI0vFFONIcqf4g== 2024-03-09T14:39:04.851 INFO:tasks.workunit.client.0.smithi001.stdout: EndpointUrl=https://smithi001.front.sepia.ceph.com:443 2024-03-09T14:39:04.851 INFO:tasks.workunit.client.0.smithi001.stderr:[main] WARN software.amazon.awssdk.http.apache.ApacheHttpClient - SSL Certificate verification is disabled. This is not a safe setting and should only be used for testing. 2024-03-09T14:39:05.049 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: Environment Variables Read 2024-03-09T14:39:05.350 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: S3 Client Initialized 2024-03-09T14:39:05.350 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: Generating File 2024-03-09T14:39:05.350 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: File Generated 2024-03-09T14:39:05.350 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: Generating File 2024-03-09T14:39:05.350 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: File Generated 2024-03-09T14:39:05.350 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: Generating File 2024-03-09T14:39:05.450 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: File Generated 2024-03-09T14:39:05.450 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: Generating File 2024-03-09T14:39:05.450 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: File Generated 2024-03-09T14:39:05.450 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: Generating File 2024-03-09T14:39:05.451 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: File Generated 2024-03-09T14:39:05.451 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: Generating File 2024-03-09T14:39:05.451 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: File Generated 2024-03-09T14:39:05.451 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: Generated Big Files 2024-03-09T14:39:06.052 INFO:tasks.workunit.client.0.smithi001.stdout:HeadBucketResponse() 2024-03-09T14:39:06.052 INFO:tasks.workunit.client.0.smithi001.stdout:sheik is ready 2024-03-09T14:39:06.052 INFO:tasks.workunit.client.0.smithi001.stdout:DEBUG: Test Bucket Created 2024-03-09T14:39:06.151 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileNoCksum called with file-0b 2024-03-09T14:39:06.452 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileNoCksum called with file-8b 2024-03-09T14:39:06.853 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileNoCksum called with file-200b 2024-03-09T14:39:07.154 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileNoCksum called with file-21983b 2024-03-09T14:39:07.454 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileNoCksum called with file-5519b 2024-03-09T14:39:07.755 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileNoCksum called with file-204329b 2024-03-09T14:39:08.257 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileNoCksum called with file-256k 2024-03-09T14:39:08.758 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileNoCksum called with file-1m 2024-03-09T14:39:09.860 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileNoCksum called with file-1038757b 2024-03-09T14:39:11.163 INFO:tasks.workunit.client.0.smithi001.stdout:mpuObjectFromFileNoCksum called with file-5m 2024-03-09T14:39:18.179 INFO:tasks.workunit.client.0.smithi001.stdout:mpuObjectFromFileNoCksum called with file-10m 2024-03-09T14:39:25.699 INFO:tasks.workunit.client.0.smithi001.stdout:mpuObjectFromFileNoCksum called with file-100m 2024-03-09T14:40:35.951 INFO:tasks.workunit.client.0.smithi001.stdout:arg: file-0b 2024-03-09T14:40:35.952 INFO:tasks.workunit.client.0.smithi001.stdout:arg: file-8b 2024-03-09T14:40:35.952 INFO:tasks.workunit.client.0.smithi001.stdout:arg: file-200b 2024-03-09T14:40:35.954 INFO:tasks.workunit.client.0.smithi001.stdout:arg: file-21983b 2024-03-09T14:40:35.955 INFO:tasks.workunit.client.0.smithi001.stdout:arg: file-5519b 2024-03-09T14:40:35.956 INFO:tasks.workunit.client.0.smithi001.stdout:arg: file-204329b 2024-03-09T14:40:35.957 INFO:tasks.workunit.client.0.smithi001.stdout:arg: file-256k 2024-03-09T14:40:35.958 INFO:tasks.workunit.client.0.smithi001.stdout:arg: file-1m 2024-03-09T14:40:35.959 INFO:tasks.workunit.client.0.smithi001.stdout:arg: file-1038757b 2024-03-09T14:40:36.000 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileCksum called with file-0b 2024-03-09T14:40:36.200 INFO:tasks.workunit.client.0.smithi001.stdout:PutObjectResponse 2024-03-09T14:40:36.300 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileCksum called with file-8b 2024-03-09T14:40:36.500 INFO:tasks.workunit.client.0.smithi001.stdout:PutObjectResponse 2024-03-09T14:40:36.601 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileCksum called with file-200b 2024-03-09T14:40:36.801 INFO:tasks.workunit.client.0.smithi001.stdout:PutObjectResponse 2024-03-09T14:40:36.901 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileCksum called with file-21983b 2024-03-09T14:40:37.101 INFO:tasks.workunit.client.0.smithi001.stdout:PutObjectResponse 2024-03-09T14:40:37.202 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileCksum called with file-5519b 2024-03-09T14:40:37.402 INFO:tasks.workunit.client.0.smithi001.stdout:PutObjectResponse 2024-03-09T14:40:37.602 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileCksum called with file-204329b 2024-03-09T14:40:37.902 INFO:tasks.workunit.client.0.smithi001.stdout:PutObjectResponse 2024-03-09T14:40:38.103 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileCksum called with file-256k 2024-03-09T14:40:38.403 INFO:tasks.workunit.client.0.smithi001.stdout:PutObjectResponse 2024-03-09T14:40:38.604 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileCksum called with file-1m 2024-03-09T14:40:39.505 INFO:tasks.workunit.client.0.smithi001.stdout:PutObjectResponse 2024-03-09T14:40:39.806 INFO:tasks.workunit.client.0.smithi001.stdout:putObjectFromFileCksum called with file-1038757b 2024-03-09T14:40:40.706 INFO:tasks.workunit.client.0.smithi001.stdout:PutObjectResponse 2024-03-09T14:40:41.007 INFO:tasks.workunit.client.0.smithi001.stdout:mpuObjectFromFileCksum called with file-5m 2024-03-09T14:40:46.414 INFO:tasks.workunit.client.0.smithi001.stdout:mpuObjectFromFileCksum called with file-10m 2024-03-09T14:40:58.330 INFO:tasks.workunit.client.0.smithi001.stdout:mpuObjectFromFileCksum called with file-100m 2024-03-09T14:42:29.565 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 204.8 s -- in io.ceph.jcksum.PutObjects 2024-03-09T14:42:29.591 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:42:29.591 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Results: 2024-03-09T14:42:29.591 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:42:29.591 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0 2024-03-09T14:42:29.591 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] 2024-03-09T14:42:29.593 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] ------------------------------------------------------------------------ 2024-03-09T14:42:29.593 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] BUILD SUCCESS 2024-03-09T14:42:29.593 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] ------------------------------------------------------------------------ 2024-03-09T14:42:29.594 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Total time: 03:27 min 2024-03-09T14:42:29.594 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] Finished at: 2024-03-09T14:42:29Z 2024-03-09T14:42:29.594 INFO:tasks.workunit.client.0.smithi001.stdout:[INFO] ------------------------------------------------------------------------ 2024-03-09T14:42:29.615 INFO:tasks.workunit.client.0.smithi001.stderr:+ exit 2024-03-09T14:42:29.615 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:42:29.615 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:42:29.675 INFO:tasks.workunit:Stopping ['cls/test_cls_lock.sh', 'cls/test_cls_log.sh', 'cls/test_cls_refcount.sh', 'cls/test_cls_rgw.sh', 'cls/test_cls_rgw_gc.sh', 'cls/test_cls_rgw_stats.sh', 'cls/test_cls_cmpomap.sh', 'cls/test_cls_2pc_queue.sh', 'cls/test_cls_user.sh', 'rgw/test_rgw_gc_log.sh', 'rgw/test_rgw_obj.sh', 'rgw/test_librgw_file.sh', 'rgw/test_awssdkv4_sig.sh'] on client.0... 2024-03-09T14:42:29.675 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-09T14:42:30.050 DEBUG:teuthology.parallel:result is None 2024-03-09T14:42:30.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:42:30.062 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:42:30.062 DEBUG:teuthology.orchestra.run.smithi001:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-09T14:42:30.107 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:42:30.107 INFO:teuthology.run_tasks:Running task workunit... 2024-03-09T14:42:30.116 INFO:tasks.workunit:Pulling workunits from ref c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:42:30.117 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-09T14:42:30.117 DEBUG:teuthology.orchestra.run.smithi001:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-09T14:42:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:42:30.157 INFO:teuthology.orchestra.run.smithi001.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-09T14:42:30.158 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-09T14:42:30.204 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-09T14:42:30.205 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-09T14:42:30.252 INFO:tasks.workunit:timeout=3h 2024-03-09T14:42:30.252 INFO:tasks.workunit:cleanup=True 2024-03-09T14:42:30.253 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:42:30.302 INFO:tasks.workunit.client.0.smithi001.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-09T14:43:43.550 INFO:tasks.workunit.client.0.smithi001.stderr:Updating files: 94% (11638/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-09T14:43:44.155 INFO:tasks.workunit.client.0.smithi001.stderr:Note: switching to 'c00d9c609fb42a2a25f1c196bed064ca8bb02413'. 2024-03-09T14:43:44.155 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:43:44.155 INFO:tasks.workunit.client.0.smithi001.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-09T14:43:44.155 INFO:tasks.workunit.client.0.smithi001.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-09T14:43:44.155 INFO:tasks.workunit.client.0.smithi001.stderr:state without impacting any branches by switching back to a branch. 2024-03-09T14:43:44.155 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:43:44.155 INFO:tasks.workunit.client.0.smithi001.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-09T14:43:44.155 INFO:tasks.workunit.client.0.smithi001.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-09T14:43:44.155 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:43:44.156 INFO:tasks.workunit.client.0.smithi001.stderr: git switch -c 2024-03-09T14:43:44.156 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:43:44.156 INFO:tasks.workunit.client.0.smithi001.stderr:Or undo this operation with: 2024-03-09T14:43:44.156 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:43:44.156 INFO:tasks.workunit.client.0.smithi001.stderr: git switch - 2024-03-09T14:43:44.156 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:43:44.156 INFO:tasks.workunit.client.0.smithi001.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-09T14:43:44.156 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:43:44.156 INFO:tasks.workunit.client.0.smithi001.stderr:HEAD is now at c00d9c609fb DNM: point s3tests at branch cbodley:wip-iam-user-apis 2024-03-09T14:43:44.161 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-09T14:43:44.209 INFO:tasks.workunit.client.0.smithi001.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-09T14:43:44.212 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-09T14:43:44.212 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-09T14:43:44.264 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-09T14:43:44.326 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-09T14:43:44.372 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-09T14:43:44.373 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-09T14:43:44.373 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-09T14:43:44.414 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-09T14:43:44.418 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:43:44.418 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-09T14:43:44.465 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_s3_mp_reupload.sh on client.0... 2024-03-09T14:43:44.466 INFO:tasks.workunit:Running workunit rgw/test_rgw_s3_mp_reupload.sh... 2024-03-09T14:43:44.467 DEBUG:teuthology.orchestra.run.smithi001:workunit test rgw/test_rgw_s3_mp_reupload.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_s3_mp_reupload.sh 2024-03-09T14:43:44.519 INFO:tasks.workunit.client.0.smithi001.stdout:INFO: fully qualified domain name: smithi001.front.sepia.ceph.com 2024-03-09T14:43:44.519 INFO:tasks.workunit.client.0.smithi001.stdout:== TESTING *COMPLETING* MULTIPART UPLOAD WITH RE-UPLOADS == 2024-03-09T14:43:44.519 INFO:tasks.workunit.client.0.smithi001.stdout:== TESTING NON-VERSIONED BUCKET == 2024-03-09T14:43:47.884 INFO:tasks.workunit.client.0.smithi001.stdout:Requirement already satisfied: pip in /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages (22.0.2) 2024-03-09T14:43:48.093 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting pip 2024-03-09T14:43:48.100 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached pip-24.0-py3-none-any.whl (2.1 MB) 2024-03-09T14:43:48.138 INFO:tasks.workunit.client.0.smithi001.stdout:Installing collected packages: pip 2024-03-09T14:43:48.138 INFO:tasks.workunit.client.0.smithi001.stdout: Attempting uninstall: pip 2024-03-09T14:43:48.139 INFO:tasks.workunit.client.0.smithi001.stdout: Found existing installation: pip 22.0.2 2024-03-09T14:43:48.413 INFO:tasks.workunit.client.0.smithi001.stdout: Uninstalling pip-22.0.2: 2024-03-09T14:43:48.419 INFO:tasks.workunit.client.0.smithi001.stdout: Successfully uninstalled pip-22.0.2 2024-03-09T14:43:49.400 INFO:tasks.workunit.client.0.smithi001.stdout:Successfully installed pip-24.0 2024-03-09T14:43:50.434 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting boto3 2024-03-09T14:43:50.435 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached boto3-1.34.59-py3-none-any.whl.metadata (6.6 kB) 2024-03-09T14:43:51.020 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting botocore<1.35.0,>=1.34.59 (from boto3) 2024-03-09T14:43:51.021 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached botocore-1.34.59-py3-none-any.whl.metadata (5.7 kB) 2024-03-09T14:43:51.096 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting jmespath<2.0.0,>=0.7.1 (from boto3) 2024-03-09T14:43:51.097 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB) 2024-03-09T14:43:51.169 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting s3transfer<0.11.0,>=0.10.0 (from boto3) 2024-03-09T14:43:51.170 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached s3transfer-0.10.0-py3-none-any.whl.metadata (1.7 kB) 2024-03-09T14:43:51.289 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.35.0,>=1.34.59->boto3) 2024-03-09T14:43:51.291 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB) 2024-03-09T14:43:51.449 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting urllib3<2.1,>=1.25.4 (from botocore<1.35.0,>=1.34.59->boto3) 2024-03-09T14:43:51.451 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached urllib3-2.0.7-py3-none-any.whl.metadata (6.6 kB) 2024-03-09T14:43:51.740 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.35.0,>=1.34.59->boto3) 2024-03-09T14:43:51.741 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB) 2024-03-09T14:43:51.755 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached boto3-1.34.59-py3-none-any.whl (139 kB) 2024-03-09T14:43:51.756 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached botocore-1.34.59-py3-none-any.whl (12.0 MB) 2024-03-09T14:43:51.778 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached jmespath-1.0.1-py3-none-any.whl (20 kB) 2024-03-09T14:43:51.779 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached s3transfer-0.10.0-py3-none-any.whl (82 kB) 2024-03-09T14:43:51.781 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) 2024-03-09T14:43:51.782 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached urllib3-2.0.7-py3-none-any.whl (124 kB) 2024-03-09T14:43:51.784 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) 2024-03-09T14:43:51.880 INFO:tasks.workunit.client.0.smithi001.stdout:Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3 2024-03-09T14:43:53.360 INFO:tasks.workunit.client.0.smithi001.stdout:Successfully installed boto3-1.34.59 botocore-1.34.59 jmespath-1.0.1 python-dateutil-2.9.0.post0 s3transfer-0.10.0 six-1.16.0 urllib3-2.0.7 2024-03-09T14:43:53.678 INFO:tasks.workunit.client.0.smithi001.stdout:INFO: creating user 2024-03-09T14:43:54.257 INFO:tasks.workunit.client.0.smithi001.stdout:{ 2024-03-09T14:43:54.257 INFO:tasks.workunit.client.0.smithi001.stdout: "user_id": "testid", 2024-03-09T14:43:54.257 INFO:tasks.workunit.client.0.smithi001.stdout: "display_name": "M. Tester", 2024-03-09T14:43:54.258 INFO:tasks.workunit.client.0.smithi001.stdout: "email": "tester@ceph.com", 2024-03-09T14:43:54.258 INFO:tasks.workunit.client.0.smithi001.stdout: "suspended": 0, 2024-03-09T14:43:54.258 INFO:tasks.workunit.client.0.smithi001.stdout: "max_buckets": 1000, 2024-03-09T14:43:54.258 INFO:tasks.workunit.client.0.smithi001.stdout: "subusers": [], 2024-03-09T14:43:54.258 INFO:tasks.workunit.client.0.smithi001.stdout: "keys": [ 2024-03-09T14:43:54.258 INFO:tasks.workunit.client.0.smithi001.stdout: { 2024-03-09T14:43:54.258 INFO:tasks.workunit.client.0.smithi001.stdout: "user": "testid", 2024-03-09T14:43:54.258 INFO:tasks.workunit.client.0.smithi001.stdout: "access_key": "0555b35654ad1656d804", 2024-03-09T14:43:54.258 INFO:tasks.workunit.client.0.smithi001.stdout: "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==", 2024-03-09T14:43:54.258 INFO:tasks.workunit.client.0.smithi001.stdout: "active": true, 2024-03-09T14:43:54.258 INFO:tasks.workunit.client.0.smithi001.stdout: "create_date": "2024-03-09T14:43:54.241844Z" 2024-03-09T14:43:54.258 INFO:tasks.workunit.client.0.smithi001.stdout: } 2024-03-09T14:43:54.259 INFO:tasks.workunit.client.0.smithi001.stdout: ], 2024-03-09T14:43:54.259 INFO:tasks.workunit.client.0.smithi001.stdout: "swift_keys": [], 2024-03-09T14:43:54.259 INFO:tasks.workunit.client.0.smithi001.stdout: "caps": [], 2024-03-09T14:43:54.259 INFO:tasks.workunit.client.0.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T14:43:54.259 INFO:tasks.workunit.client.0.smithi001.stdout: "default_placement": "", 2024-03-09T14:43:54.259 INFO:tasks.workunit.client.0.smithi001.stdout: "default_storage_class": "", 2024-03-09T14:43:54.259 INFO:tasks.workunit.client.0.smithi001.stdout: "placement_tags": [], 2024-03-09T14:43:54.259 INFO:tasks.workunit.client.0.smithi001.stdout: "bucket_quota": { 2024-03-09T14:43:54.259 INFO:tasks.workunit.client.0.smithi001.stdout: "enabled": false, 2024-03-09T14:43:54.259 INFO:tasks.workunit.client.0.smithi001.stdout: "check_on_raw": false, 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: "max_size": -1, 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: "max_size_kb": 0, 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: "max_objects": -1 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: "user_quota": { 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: "enabled": false, 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: "check_on_raw": false, 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: "max_size": -1, 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: "max_size_kb": 0, 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: "max_objects": -1 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: }, 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: "temp_url_keys": [], 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: "type": "rgw", 2024-03-09T14:43:54.260 INFO:tasks.workunit.client.0.smithi001.stdout: "mfa_ids": [], 2024-03-09T14:43:54.261 INFO:tasks.workunit.client.0.smithi001.stdout: "account_id": "", 2024-03-09T14:43:54.261 INFO:tasks.workunit.client.0.smithi001.stdout: "path": "/", 2024-03-09T14:43:54.261 INFO:tasks.workunit.client.0.smithi001.stdout: "create_date": "2024-03-09T14:43:54.241815Z", 2024-03-09T14:43:54.261 INFO:tasks.workunit.client.0.smithi001.stdout: "tags": [], 2024-03-09T14:43:54.261 INFO:tasks.workunit.client.0.smithi001.stdout: "group_ids": [] 2024-03-09T14:43:54.261 INFO:tasks.workunit.client.0.smithi001.stdout:} 2024-03-09T14:43:54.261 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-03-09T14:44:00.046 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:00.047 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:00.186 INFO:tasks.workunit.client.0.smithi001.stderr:5+0 records in 2024-03-09T14:44:00.187 INFO:tasks.workunit.client.0.smithi001.stderr:5+0 records out 2024-03-09T14:44:00.187 INFO:tasks.workunit.client.0.smithi001.stderr:5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0220552 s, 238 MB/s 2024-03-09T14:44:00.191 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:00.191 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:00.314 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:00.314 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:04.809 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:04.809 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:09.366 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:09.367 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:14.044 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:14.044 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:18.695 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:18.695 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:23.346 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:23.346 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:27.949 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:27.949 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:32.473 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:32.474 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:37.091 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:37.091 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:41.750 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:41.750 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:46.356 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:46.356 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:50.973 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:50.974 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:44:55.623 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:44:55.624 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:45:00.218 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:45:00.218 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:45:00.483 INFO:tasks.workunit.client.0.smithi001.stdout:bucket nams is reupload-bkt-120375 2024-03-09T14:45:00.483 INFO:tasks.workunit.client.0.smithi001.stdout:endpoint is https://smithi001.front.sepia.ceph.com:443 2024-03-09T14:45:00.483 INFO:tasks.workunit.client.0.smithi001.stdout:start UploadId=2~XYKVyRRJH7hS1iRC4ffkQPbLYR8QEue 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 1 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 1 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 2} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:phase 2 uploaded part {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1} 2024-03-09T14:45:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:completing multipart upload, parts=[{'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 1}, {'ETag': '"b82360d99aaac21e943c893fc13ba8a0"', 'PartNumber': 2}] 2024-03-09T14:45:02.275 INFO:tasks.workunit.client.0.smithi001.stdout:radosgw-admin bucket radoslist: 2024-03-09T14:45:02.276 INFO:tasks.workunit.client.0.smithi001.stdout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.3_mpu_test4 2024-03-09T14:45:02.276 INFO:tasks.workunit.client.0.smithi001.stdout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.3__multipart_mpu_test4.2~XYKVyRRJH7hS1iRC4ffkQPbLYR8QEue.2 2024-03-09T14:45:02.276 INFO:tasks.workunit.client.0.smithi001.stdout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.3__multipart_mpu_test4.m1kKNp9eV3dB2giqWGNZqJ2BfbV10ZHl.1 2024-03-09T14:45:02.365 INFO:tasks.workunit.client.0.smithi001.stdout:rados ls: 2024-03-09T14:45:02.365 INFO:tasks.workunit.client.0.smithi001.stdout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.3_mpu_test4 2024-03-09T14:45:02.365 INFO:tasks.workunit.client.0.smithi001.stdout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.3__multipart_mpu_test4.2~XYKVyRRJH7hS1iRC4ffkQPbLYR8QEue.2 2024-03-09T14:45:02.365 INFO:tasks.workunit.client.0.smithi001.stdout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.3__multipart_mpu_test4.m1kKNp9eV3dB2giqWGNZqJ2BfbV10ZHl.1 2024-03-09T14:45:02.385 INFO:tasks.workunit.client.0.smithi001.stdout:== PASSED == 2024-03-09T14:45:02.385 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:45:02.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:45:02.438 INFO:tasks.workunit:Stopping ['rgw/test_rgw_s3_mp_reupload.sh'] on client.0... 2024-03-09T14:45:02.439 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-09T14:45:02.857 DEBUG:teuthology.parallel:result is None 2024-03-09T14:45:02.858 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:45:02.869 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:45:02.870 DEBUG:teuthology.orchestra.run.smithi001:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-09T14:45:02.916 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:45:02.916 INFO:teuthology.run_tasks:Running task ragweed... 2024-03-09T14:45:02.929 DEBUG:tasks.ragweed:ragweed config is {'client.0': {'default-branch': 'ceph-master', 'rgw_server': 'client.0', 'stages': 'prepare,check'}} 2024-03-09T14:45:02.929 INFO:tasks.ragweed:Downloading ragweed... 2024-03-09T14:45:02.929 INFO:tasks.ragweed:Using branch 'wip-rgw-account-v3' for ragweed 2024-03-09T14:45:02.929 DEBUG:teuthology.orchestra.run.smithi001:> git clone -b wip-rgw-account-v3 https://git.ceph.com/ragweed.git /home/ubuntu/cephtest/ragweed.client.0 2024-03-09T14:45:02.970 INFO:teuthology.orchestra.run.smithi001.stderr:Cloning into '/home/ubuntu/cephtest/ragweed.client.0'... 2024-03-09T14:45:03.027 INFO:teuthology.orchestra.run.smithi001.stderr:fatal: Remote branch wip-rgw-account-v3 not found in upstream origin 2024-03-09T14:45:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 128 2024-03-09T14:45:03.030 INFO:tasks.ragweed:Using branch 'ceph-master' for ragweed 2024-03-09T14:45:03.030 DEBUG:teuthology.orchestra.run.smithi001:> git clone -b ceph-master https://git.ceph.com/ragweed.git /home/ubuntu/cephtest/ragweed.client.0 2024-03-09T14:45:03.076 INFO:teuthology.orchestra.run.smithi001.stderr:Cloning into '/home/ubuntu/cephtest/ragweed.client.0'... 2024-03-09T14:45:03.256 INFO:tasks.ragweed:Creating rgw users... 2024-03-09T14:45:03.257 DEBUG:tasks.ragweed:Creating user ragweed.client.0 on client.0 2024-03-09T14:45:03.257 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid ragweed.client.0 --display-name 'Mr. ragweed.client.0' --access-key NAAISEVWFNNKSHKYHQIO --secret qMn2T2dO+CQdWAYbiOozqBUCtle0Pad/1kJ/9bTKcKoe3XDLq0c5wQ== --email ragweed.client.0+test@test.test --system false 2024-03-09T14:45:03.348 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:45:03.349 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:45:03.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.387+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.387+0000 7f96f926c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.387+0000 7f96f926c980 20 realm 2024-03-09T14:45:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.387+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.387+0000 7f96f926c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.387+0000 7f96f926c980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:45:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.387+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.387+0000 7f96f926c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.387+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.391+0000 7f96f926c980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:45:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.391+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.391+0000 7f96f926c980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:45:03.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.391+0000 7f96f926c980 20 searching for the correct realm 2024-03-09T14:45:03.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.395+0000 7f96f7585640 0 -- 172.21.15.1:0/1682230100 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x558339212eb0 msgr2=0x558339215340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:45:03.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.403+0000 7f96f6583640 0 -- 172.21.15.1:0/1682230100 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f96e80e1040 msgr2=0x7f96e81190b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:45:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:45:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:45:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:45:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:45:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:45:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:45:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:45:03.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:45:03.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 zone default found 2024-03-09T14:45:03.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 4 Realm: () 2024-03-09T14:45:03.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:45:03.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:45:03.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:45:03.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 zonegroup default 2024-03-09T14:45:03.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.411+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.415+0000 7f96f926c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.415+0000 7f96f926c980 20 started sync module instance, tier type = 2024-03-09T14:45:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.415+0000 7f96f926c980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:45:03.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.427+0000 7f96f926c980 20 add_watcher() i=0 2024-03-09T14:45:03.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.427+0000 7f96f926c980 20 add_watcher() i=1 2024-03-09T14:45:03.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.427+0000 7f96f926c980 20 add_watcher() i=2 2024-03-09T14:45:03.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.427+0000 7f96f926c980 20 add_watcher() i=3 2024-03-09T14:45:03.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.427+0000 7f96f926c980 20 add_watcher() i=4 2024-03-09T14:45:03.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.427+0000 7f96f926c980 20 add_watcher() i=5 2024-03-09T14:45:03.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.427+0000 7f96f926c980 20 add_watcher() i=6 2024-03-09T14:45:03.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.427+0000 7f96f926c980 20 add_watcher() i=7 2024-03-09T14:45:03.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.427+0000 7f96f926c980 2 all 8 watchers are set, enabling cache 2024-03-09T14:45:03.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.431+0000 7f95adfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:45:03.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.431+0000 7f96f926c980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:45:03.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.431+0000 7f96f926c980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:45:03.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.431+0000 7f96f926c980 5 note: GC not initialized 2024-03-09T14:45:03.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.431+0000 7f95acfe1640 20 reqs_thread_entry: start 2024-03-09T14:45:03.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.431+0000 7f96f926c980 20 init_complete bucket index max shards: 1 2024-03-09T14:45:03.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.431+0000 7f96f926c980 20 Filter name: none 2024-03-09T14:45:03.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.431+0000 7f95a5ffb640 20 reqs_thread_entry: start 2024-03-09T14:45:03.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.431+0000 7f96f926c980 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : miss 2024-03-09T14:45:03.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.431+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x0 2024-03-09T14:45:03.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 10 adding default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2024-03-09T14:45:03.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 10 cache get: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test : miss 2024-03-09T14:45:03.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x0 2024-03-09T14:45:03.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 10 adding default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2024-03-09T14:45:03.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 10 cache get: name=default.rgw.meta+users.keys+NAAISEVWFNNKSHKYHQIO : miss 2024-03-09T14:45:03.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 10 cache put: name=default.rgw.meta+users.keys+NAAISEVWFNNKSHKYHQIO info.flags=0x0 2024-03-09T14:45:03.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 10 adding default.rgw.meta+users.keys+NAAISEVWFNNKSHKYHQIO to cache LRU end 2024-03-09T14:45:03.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 10 cache get: name=default.rgw.meta+users.keys+NAAISEVWFNNKSHKYHQIO : hit (negative entry) 2024-03-09T14:45:03.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.435+0000 7f96f926c980 10 cache get: name=default.rgw.meta+users.keys+NAAISEVWFNNKSHKYHQIO : hit (negative entry) 2024-03-09T14:45:03.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.439+0000 7f96f926c980 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x17 2024-03-09T14:45:03.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.439+0000 7f96f926c980 10 moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2024-03-09T14:45:03.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.439+0000 7f96f926c980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ragweed.client.0, ofs0, ns] 2024-03-09T14:45:03.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.439+0000 7f95da7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1073741824018 cookie 94022087615792 notifier 5895 bl.length()=675 2024-03-09T14:45:03.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.439+0000 7f95da7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x17 2024-03-09T14:45:03.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.439+0000 7f95da7fc640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2024-03-09T14:45:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.443+0000 7f96f926c980 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x7 2024-03-09T14:45:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.443+0000 7f96f926c980 10 moving default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2024-03-09T14:45:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.443+0000 7f96f926c980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.email:ragweed.client.0+test@test.test, ofs0, ns] 2024-03-09T14:45:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.447+0000 7f95da7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1073741824055 cookie 94022087613088 notifier 5895 bl.length()=202 2024-03-09T14:45:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.447+0000 7f95da7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x7 2024-03-09T14:45:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.447+0000 7f95da7fc640 10 rgw watcher librados: moving default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2024-03-09T14:45:03.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.451+0000 7f96f926c980 10 cache put: name=default.rgw.meta+users.keys+NAAISEVWFNNKSHKYHQIO info.flags=0x7 2024-03-09T14:45:03.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.451+0000 7f96f926c980 10 moving default.rgw.meta+users.keys+NAAISEVWFNNKSHKYHQIO to cache LRU end 2024-03-09T14:45:03.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.451+0000 7f96f926c980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:NAAISEVWFNNKSHKYHQIO, ofs0, ns] 2024-03-09T14:45:03.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.451+0000 7f95d9ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1073741824043 cookie 94022087609536 notifier 5895 bl.length()=190 2024-03-09T14:45:03.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.451+0000 7f95d9ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+NAAISEVWFNNKSHKYHQIO info.flags=0x7 2024-03-09T14:45:03.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.451+0000 7f95d9ffb640 10 rgw watcher librados: moving default.rgw.meta+users.keys+NAAISEVWFNNKSHKYHQIO to cache LRU end 2024-03-09T14:45:03.456 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T14:45:03.456 INFO:teuthology.orchestra.run.smithi001.stdout: "user_id": "ragweed.client.0", 2024-03-09T14:45:03.456 INFO:teuthology.orchestra.run.smithi001.stdout: "display_name": "Mr. ragweed.client.0", 2024-03-09T14:45:03.456 INFO:teuthology.orchestra.run.smithi001.stdout: "email": "ragweed.client.0+test@test.test", 2024-03-09T14:45:03.456 INFO:teuthology.orchestra.run.smithi001.stdout: "suspended": 0, 2024-03-09T14:45:03.456 INFO:teuthology.orchestra.run.smithi001.stdout: "max_buckets": 1000, 2024-03-09T14:45:03.456 INFO:teuthology.orchestra.run.smithi001.stdout: "subusers": [], 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: "keys": [ 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: "user": "ragweed.client.0", 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: "access_key": "NAAISEVWFNNKSHKYHQIO", 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: "secret_key": "qMn2T2dO+CQdWAYbiOozqBUCtle0Pad/1kJ/9bTKcKoe3XDLq0c5wQ==", 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: "active": true, 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T14:45:03.439533Z" 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: "swift_keys": [], 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": [], 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: "default_placement": "", 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: "default_storage_class": "", 2024-03-09T14:45:03.457 INFO:teuthology.orchestra.run.smithi001.stdout: "placement_tags": [], 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: "bucket_quota": { 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: "user_quota": { 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T14:45:03.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T14:45:03.459 INFO:teuthology.orchestra.run.smithi001.stdout: "temp_url_keys": [], 2024-03-09T14:45:03.459 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "rgw", 2024-03-09T14:45:03.459 INFO:teuthology.orchestra.run.smithi001.stdout: "mfa_ids": [], 2024-03-09T14:45:03.459 INFO:teuthology.orchestra.run.smithi001.stdout: "account_id": "", 2024-03-09T14:45:03.459 INFO:teuthology.orchestra.run.smithi001.stdout: "path": "/", 2024-03-09T14:45:03.459 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T14:45:03.439518Z", 2024-03-09T14:45:03.459 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-03-09T14:45:03.459 INFO:teuthology.orchestra.run.smithi001.stdout: "group_ids": [] 2024-03-09T14:45:03.459 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T14:45:03.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:45:03.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.455+0000 7f95adfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:45:03.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.459+0000 7f96f926c980 20 remove_watcher() i=0 2024-03-09T14:45:03.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.459+0000 7f96f926c980 2 removed watcher, disabling cache 2024-03-09T14:45:03.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.463+0000 7f96f926c980 20 remove_watcher() i=1 2024-03-09T14:45:03.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.463+0000 7f96f926c980 20 remove_watcher() i=2 2024-03-09T14:45:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.467+0000 7f96f926c980 20 remove_watcher() i=3 2024-03-09T14:45:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.467+0000 7f96f926c980 20 remove_watcher() i=4 2024-03-09T14:45:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.471+0000 7f96f926c980 20 remove_watcher() i=5 2024-03-09T14:45:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.471+0000 7f96f926c980 20 remove_watcher() i=6 2024-03-09T14:45:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.471+0000 7f96f926c980 20 remove_watcher() i=7 2024-03-09T14:45:03.487 DEBUG:tasks.ragweed:Creating user sysuser.client.0 on client.0 2024-03-09T14:45:03.487 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid sysuser.client.0 --display-name 'Mr. sysuser.client.0' --access-key QGUAXOPOXEVLUOTENWHO --secret 80c3tPCD9xLPx5YuaKeCT3Fp7JnForKwHx5GdeayXqGaWHsEhLzTsw== --email sysuser.client.0+test@test.test --system true 2024-03-09T14:45:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:45:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:45:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.559+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.563+0000 7fc514694980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.563+0000 7fc514694980 20 realm 2024-03-09T14:45:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.563+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.563+0000 7fc514694980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.563+0000 7fc514694980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:45:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.563+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.563+0000 7fc514694980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.563+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.563+0000 7fc514694980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:45:03.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.563+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:03.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.563+0000 7fc514694980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:45:03.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.563+0000 7fc514694980 20 searching for the correct realm 2024-03-09T14:45:03.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.567+0000 7fc5119ab640 0 -- 172.21.15.1:0/3878074849 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fc504053040 msgr2=0x7fc504034460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:45:03.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.783+0000 7fc5121ac640 0 -- 172.21.15.1:0/3878074849 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fc50808c3a0 msgr2=0x7fc5080190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:45:04.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.999+0000 7fc514694980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:45:04.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.999+0000 7fc514694980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:45:04.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.999+0000 7fc514694980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:45:04.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.999+0000 7fc514694980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:45:04.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.999+0000 7fc514694980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:45:04.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.999+0000 7fc514694980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:45:04.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.999+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:04.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.999+0000 7fc514694980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:04.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:03.999+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:04.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:45:04.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:04.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:45:04.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 zone default found 2024-03-09T14:45:04.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 4 Realm: () 2024-03-09T14:45:04.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:45:04.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:45:04.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:45:04.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 zonegroup default 2024-03-09T14:45:04.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:04.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:04.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:04.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:04.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 started sync module instance, tier type = 2024-03-09T14:45:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.003+0000 7fc514694980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:45:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.027+0000 7fc514694980 20 add_watcher() i=0 2024-03-09T14:45:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.027+0000 7fc514694980 20 add_watcher() i=1 2024-03-09T14:45:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.027+0000 7fc514694980 20 add_watcher() i=2 2024-03-09T14:45:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.027+0000 7fc514694980 20 add_watcher() i=3 2024-03-09T14:45:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.027+0000 7fc514694980 20 add_watcher() i=4 2024-03-09T14:45:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.027+0000 7fc514694980 20 add_watcher() i=5 2024-03-09T14:45:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.027+0000 7fc514694980 20 add_watcher() i=6 2024-03-09T14:45:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.027+0000 7fc514694980 20 add_watcher() i=7 2024-03-09T14:45:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.027+0000 7fc514694980 2 all 8 watchers are set, enabling cache 2024-03-09T14:45:04.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.031+0000 7fc3c8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:45:04.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.031+0000 7fc514694980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:45:04.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.031+0000 7fc514694980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:45:04.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.031+0000 7fc514694980 5 note: GC not initialized 2024-03-09T14:45:04.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.031+0000 7fc3437fe640 20 reqs_thread_entry: start 2024-03-09T14:45:04.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.031+0000 7fc514694980 20 init_complete bucket index max shards: 1 2024-03-09T14:45:04.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.031+0000 7fc514694980 20 Filter name: none 2024-03-09T14:45:04.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc3417fa640 20 reqs_thread_entry: start 2024-03-09T14:45:04.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : miss 2024-03-09T14:45:04.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:04.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:04.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x0 2024-03-09T14:45:04.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 10 adding default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2024-03-09T14:45:04.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 10 cache get: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test : miss 2024-03-09T14:45:04.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:04.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:04.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x0 2024-03-09T14:45:04.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 10 adding default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2024-03-09T14:45:04.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 10 cache get: name=default.rgw.meta+users.keys+QGUAXOPOXEVLUOTENWHO : miss 2024-03-09T14:45:04.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 20 rados->read ofs=0 len=0 2024-03-09T14:45:04.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:45:04.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 10 cache put: name=default.rgw.meta+users.keys+QGUAXOPOXEVLUOTENWHO info.flags=0x0 2024-03-09T14:45:04.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 10 adding default.rgw.meta+users.keys+QGUAXOPOXEVLUOTENWHO to cache LRU end 2024-03-09T14:45:04.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 10 cache get: name=default.rgw.meta+users.keys+QGUAXOPOXEVLUOTENWHO : hit (negative entry) 2024-03-09T14:45:04.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.035+0000 7fc514694980 10 cache get: name=default.rgw.meta+users.keys+QGUAXOPOXEVLUOTENWHO : hit (negative entry) 2024-03-09T14:45:04.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.039+0000 7fc514694980 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x17 2024-03-09T14:45:04.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.039+0000 7fc514694980 10 moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2024-03-09T14:45:04.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.039+0000 7fc514694980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:sysuser.client.0, ofs0, ns] 2024-03-09T14:45:04.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.039+0000 7fc3f5ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1073741824056 cookie 93943981827072 notifier 5579 bl.length()=675 2024-03-09T14:45:04.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.039+0000 7fc3f5ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x17 2024-03-09T14:45:04.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.039+0000 7fc3f5ffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2024-03-09T14:45:04.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.043+0000 7fc514694980 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x7 2024-03-09T14:45:04.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.043+0000 7fc514694980 10 moving default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2024-03-09T14:45:04.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.043+0000 7fc514694980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.email:sysuser.client.0+test@test.test, ofs0, ns] 2024-03-09T14:45:04.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.043+0000 7fc3f5ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1073741824019 cookie 93943981840496 notifier 5579 bl.length()=202 2024-03-09T14:45:04.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.043+0000 7fc3f5ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x7 2024-03-09T14:45:04.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.043+0000 7fc3f5ffb640 10 rgw watcher librados: moving default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2024-03-09T14:45:04.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.051+0000 7fc514694980 10 cache put: name=default.rgw.meta+users.keys+QGUAXOPOXEVLUOTENWHO info.flags=0x7 2024-03-09T14:45:04.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.051+0000 7fc514694980 10 moving default.rgw.meta+users.keys+QGUAXOPOXEVLUOTENWHO to cache LRU end 2024-03-09T14:45:04.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.051+0000 7fc514694980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.keys:QGUAXOPOXEVLUOTENWHO, ofs0, ns] 2024-03-09T14:45:04.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.051+0000 7fc3f57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1073741824020 cookie 93943981829008 notifier 5579 bl.length()=190 2024-03-09T14:45:04.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.051+0000 7fc3f57fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+QGUAXOPOXEVLUOTENWHO info.flags=0x7 2024-03-09T14:45:04.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.051+0000 7fc3f57fa640 10 rgw watcher librados: moving default.rgw.meta+users.keys+QGUAXOPOXEVLUOTENWHO to cache LRU end 2024-03-09T14:45:04.055 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T14:45:04.055 INFO:teuthology.orchestra.run.smithi001.stdout: "user_id": "sysuser.client.0", 2024-03-09T14:45:04.055 INFO:teuthology.orchestra.run.smithi001.stdout: "display_name": "Mr. sysuser.client.0", 2024-03-09T14:45:04.055 INFO:teuthology.orchestra.run.smithi001.stdout: "email": "sysuser.client.0+test@test.test", 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: "suspended": 0, 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: "max_buckets": 1000, 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: "subusers": [], 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: "keys": [ 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: "user": "sysuser.client.0", 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: "access_key": "QGUAXOPOXEVLUOTENWHO", 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: "secret_key": "80c3tPCD9xLPx5YuaKeCT3Fp7JnForKwHx5GdeayXqGaWHsEhLzTsw==", 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: "active": true, 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T14:45:04.040105Z" 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: "swift_keys": [], 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": [], 2024-03-09T14:45:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "system": true, 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "default_placement": "", 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "default_storage_class": "", 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "placement_tags": [], 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "bucket_quota": { 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "user_quota": { 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T14:45:04.057 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout: "temp_url_keys": [], 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "rgw", 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout: "mfa_ids": [], 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout: "account_id": "", 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout: "path": "/", 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T14:45:04.040088Z", 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout: "group_ids": [] 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T14:45:04.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:45:04.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.055+0000 7fc3c8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:45:04.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.059+0000 7fc514694980 20 remove_watcher() i=0 2024-03-09T14:45:04.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.059+0000 7fc514694980 2 removed watcher, disabling cache 2024-03-09T14:45:04.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.063+0000 7fc514694980 20 remove_watcher() i=1 2024-03-09T14:45:04.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.063+0000 7fc514694980 20 remove_watcher() i=2 2024-03-09T14:45:04.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.067+0000 7fc514694980 20 remove_watcher() i=3 2024-03-09T14:45:04.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.067+0000 7fc514694980 20 remove_watcher() i=4 2024-03-09T14:45:04.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.071+0000 7fc514694980 20 remove_watcher() i=5 2024-03-09T14:45:04.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.071+0000 7fc514694980 20 remove_watcher() i=6 2024-03-09T14:45:04.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:45:04.075+0000 7fc514694980 20 remove_watcher() i=7 2024-03-09T14:45:04.092 INFO:tasks.ragweed:Configuring ragweed... 2024-03-09T14:45:04.092 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:45:04.092 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/archive/ragweed.client.0.conf 2024-03-09T14:45:04.141 INFO:tasks.ragweed:Configuring boto... 2024-03-09T14:45:04.141 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:45:04.141 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/boto.cfg 2024-03-09T14:45:04.188 DEBUG:teuthology.orchestra.run.smithi001:ragweed tests against rgw> source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/ragweed.client.0 && RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg tox --sitepackages -- -v -m 'not fails_on_rgw' 2024-03-09T14:45:04.372 INFO:teuthology.orchestra.run.smithi001.stdout:GLOB sdist-make: /home/ubuntu/cephtest/ragweed.client.0/setup.py 2024-03-09T14:45:04.566 INFO:teuthology.orchestra.run.smithi001.stdout:py create: /home/ubuntu/cephtest/ragweed.client.0/.tox/py 2024-03-09T14:45:05.117 INFO:teuthology.orchestra.run.smithi001.stdout:py installdeps: -rrequirements.txt 2024-03-09T14:45:08.226 INFO:teuthology.orchestra.run.smithi001.stdout:py inst: /home/ubuntu/cephtest/ragweed.client.0/.tox/.tmp/package/1/ragweed-0.0.1.zip 2024-03-09T14:45:10.942 INFO:teuthology.orchestra.run.smithi001.stdout:py installed: appdirs==1.4.4,asyncssh==2.5.0,attrs==21.2.0,Babel==2.8.0,bcrypt==3.2.0,beautifulsoup4==4.10.0,beniget==0.4.1,blinker==1.4,boto==2.49.0,Brotli==1.0.9,cachetools==5.0.0,ceph==1.0.0,ceph-volume==1.0.0,cephfs==2.0.0,certifi==2020.6.20,chardet==4.0.0,cheroot==8.5.2+ds1,CherryPy==18.6.1,command-not-found==0.3,configobj==5.0.6,cryptography==3.4.8,cycler==0.11.0,dbus-python==1.2.18,decorator==4.4.2,distlib==0.3.4,distro==1.7.0,distro-info===1.1build1,docopt==0.6.2,execnet==1.9.0,filelock==3.6.0,fonttools==4.29.1,fs==2.4.12,gast==0.5.2,gevent==21.8.0,google-auth==1.5.1,greenlet==1.1.2,html5lib==1.1,httplib2==0.20.2,idna==3.3,importlib-metadata==4.6.4,iniconfig==1.1.1,isodate==0.6.1,jaraco.classes==3.2.1,jaraco.collections==3.4.0,jaraco.functools==3.4.0,jaraco.text==3.6.0,jeepney==0.7.1,Jinja2==3.0.3,joblib==0.17.0,keyring==23.5.0,kiwisolver==1.3.2,kubernetes==12.0.1,language-selector==0.1,launchpadlib==1.10.16,lazr.restfulclient==0.14.4,lazr.uri==1.0.6,logutils==0.3.3,lxml==4.8.0,lz4==3.1.3+dfsg,Mako==1.1.3,MarkupSafe==2.0.1,matplotlib==3.5.1,more-itertools==8.10.0,mpmath==0.0.0,munch==4.0.0,natsort==8.0.2,netifaces==0.11.0,nose==1.3.7,numpy==1.21.5,oauthlib==3.2.0,olefile==0.46,packaging==21.3,Paste==3.5.0,PasteDeploy==2.1.1,PasteScript==2.0.2,pecan==1.3.3,Pillow==9.0.1,platformdirs==2.5.1,pluggy==0.13.0,ply==3.11,portend==3.0.0,prettytable==2.5.0,psutil==5.9.0,py==1.10.0,pyasn1==0.4.8,pyasn1-modules==0.2.1,Pygments==2.11.2,PyGObject==3.42.1,pyinotify==0.9.6,PyJWT==2.3.0,pymacaroons==0.13.0,PyNaCl==1.5.0,pyOpenSSL==21.0.0,pyparsing==2.4.7,pytest==6.2.5,python-apt==2.3.0+ubuntu2.1,python-dateutil==2.8.1,python-magic==0.4.24,pythran==0.10.0,pytz==2022.1,PyYAML==5.4.1,rados==2.0.0,ragweed @ file:///home/ubuntu/cephtest/ragweed.client.0/.tox/.tmp/package/1/ragweed-0.0.1.zip#sha256=fa30d8d200022cc17c626a386d91b2b02d3cd16c869011b18f8efb1c20576715,rbd==2.0.0,registries-conf-ctl @ git+https://github.com/sebastian-philipp/registries-conf-ctl@0e9bf2dc202aa9be9b0bf8e8265aac3f2308f9db,remoto==1.2.1,repoze.lru==0.7,requests==2.25.1,requests-oauthlib==1.3.0,rgw==2.0.0,Routes==2.5.1,rsa==4.8,s3cmd==2.2.0,scikit-learn==0.23.2,scipy==1.8.0,SecretStorage==3.3.1,simplegeneric==0.8.1,simplejson==3.17.6,six==1.16.0,soupsieve==2.3.1,ssh-import-id==5.11,sympy==1.9,Tempita==0.5.2,tempora==4.1.2,threadpoolctl==3.1.0,toml==0.10.2,ubuntu-advantage-tools==27.12,ufoLib2==0.13.1,ufw==0.36.1,unattended-upgrades==0.1,unicodedata2==14.0.0,urllib3==1.26.5,virtualenv==20.13.0+ds,wadllib==1.3.6,waitress==1.4.4,wcwidth==0.2.5,webencodings==0.5.1,WebOb==1.8.6,websocket-client==1.2.3,WebTest==2.0.35,Werkzeug==2.0.2,zc.lockfile==2.0,zipp==1.0.0,zope.event==4.4,zope.interface==5.4.0 2024-03-09T14:45:10.942 INFO:teuthology.orchestra.run.smithi001.stdout:py run-test-pre: PYTHONHASHSEED='1892879687' 2024-03-09T14:45:10.942 INFO:teuthology.orchestra.run.smithi001.stdout:py run-test: commands[0] | python -m pytest -v -m 'not fails_on_rgw' 2024-03-09T14:45:11.111 INFO:teuthology.orchestra.run.smithi001.stdout:============================= test session starts ============================== 2024-03-09T14:45:11.111 INFO:teuthology.orchestra.run.smithi001.stdout:platform linux -- Python 3.10.6, pytest-6.2.5, py-1.10.0, pluggy-0.13.0 -- /home/ubuntu/cephtest/ragweed.client.0/.tox/py/bin/python 2024-03-09T14:45:11.111 INFO:teuthology.orchestra.run.smithi001.stdout:cachedir: .tox/py/.pytest_cache 2024-03-09T14:45:11.111 INFO:teuthology.orchestra.run.smithi001.stdout:rootdir: /home/ubuntu/cephtest/ragweed.client.0, configfile: pytest.ini 2024-03-09T14:45:11.280 INFO:teuthology.orchestra.run.smithi001.stdout:collecting ... collected 9 items 2024-03-09T14:45:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:45:23.460 INFO:teuthology.orchestra.run.smithi001.stdout:ragweed/tests/tests.py::r_test_small_obj_data::test PASSED [ 11%] 2024-03-09T14:45:37.016 INFO:teuthology.orchestra.run.smithi001.stdout:ragweed/tests/tests.py::r_test_multipart_simple::test PASSED [ 22%] 2024-03-09T14:45:48.126 INFO:teuthology.orchestra.run.smithi001.stdout:ragweed/tests/tests.py::r_test_multipart_index_versioning::test PASSED [ 33%] 2024-03-09T14:46:03.309 INFO:teuthology.orchestra.run.smithi001.stdout:ragweed/tests/tests.py::r_test_multipart_defer_complete::test PASSED [ 44%] 2024-03-09T14:46:16.840 INFO:teuthology.orchestra.run.smithi001.stdout:ragweed/tests/tests.py::r_test_multipart_defer_update_complete::test PASSED [ 55%] 2024-03-09T14:46:44.510 INFO:teuthology.orchestra.run.smithi001.stdout:ragweed/tests/tests.py::r_test_obj_storage_class::test PASSED [ 66%] 2024-03-09T14:47:10.883 INFO:teuthology.orchestra.run.smithi001.stdout:ragweed/tests/tests.py::r_test_obj_storage_class_multipart::test PASSED [ 77%] 2024-03-09T14:47:34.973 INFO:teuthology.orchestra.run.smithi001.stdout:ragweed/tests/tests.py::r_test_obj_storage_class_copy::test PASSED [ 88%] 2024-03-09T14:48:22.546 INFO:teuthology.orchestra.run.smithi001.stdout:ragweed/tests/tests.py::r_test_obj_storage_class_multipart_copy::test PASSED [100%] 2024-03-09T14:48:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:48:22.546 INFO:teuthology.orchestra.run.smithi001.stdout:=============================== warnings summary =============================== 2024-03-09T14:48:22.546 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/boto/plugin.py:40 2024-03-09T14:48:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/ragweed.client.0/.tox/py/lib/python3.10/site-packages/boto/plugin.py:40: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses 2024-03-09T14:48:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: import imp 2024-03-09T14:48:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:48:22.547 INFO:teuthology.orchestra.run.smithi001.stdout:-- Docs: https://docs.pytest.org/en/stable/warnings.html 2024-03-09T14:48:22.547 INFO:teuthology.orchestra.run.smithi001.stdout:=================== 9 passed, 1 warning in 191.43s (0:03:11) =================== 2024-03-09T14:48:22.601 INFO:teuthology.orchestra.run.smithi001.stdout:___________________________________ summary ____________________________________ 2024-03-09T14:48:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: py: commands succeeded 2024-03-09T14:48:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: congratulations :) 2024-03-09T14:48:22.622 INFO:tasks.ragweed:Cleaning up boto... 2024-03-09T14:48:22.622 DEBUG:teuthology.orchestra.run.smithi001:> rm -f /home/ubuntu/cephtest/boto.cfg 2024-03-09T14:48:22.628 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid ragweed.client.0 --purge-data 2024-03-09T14:48:22.707 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:48:22.707 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:48:22.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.745+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.745+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:22.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.745+0000 7ff050e07980 20 realm 2024-03-09T14:48:22.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.745+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.749+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:22.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.749+0000 7ff050e07980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:48:22.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.749+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.749+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.749+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.749+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:48:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.749+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.749+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:48:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.749+0000 7ff050e07980 20 searching for the correct realm 2024-03-09T14:48:22.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.757+0000 7ff04f120640 0 -- 172.21.15.1:0/375142006 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x564c57dd6230 msgr2=0x564c57dd86f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:48:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.765+0000 7ff04e11e640 0 -- 172.21.15.1:0/375142006 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7ff04000d390 msgr2=0x7ff04006d880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:48:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.769+0000 7ff050e07980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:48:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.769+0000 7ff050e07980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:48:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.769+0000 7ff050e07980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:48:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.769+0000 7ff050e07980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:48:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.769+0000 7ff050e07980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:48:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.769+0000 7ff050e07980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:48:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.769+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.769+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.769+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:48:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:48:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 zone default found 2024-03-09T14:48:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 4 Realm: () 2024-03-09T14:48:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:48:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:48:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:48:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 zonegroup default 2024-03-09T14:48:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:22.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 started sync module instance, tier type = 2024-03-09T14:48:22.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.773+0000 7ff050e07980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:48:22.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.789+0000 7ff050e07980 20 add_watcher() i=0 2024-03-09T14:48:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.789+0000 7ff050e07980 20 add_watcher() i=1 2024-03-09T14:48:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.789+0000 7ff050e07980 20 add_watcher() i=2 2024-03-09T14:48:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.789+0000 7ff050e07980 20 add_watcher() i=3 2024-03-09T14:48:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.789+0000 7ff050e07980 20 add_watcher() i=4 2024-03-09T14:48:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.789+0000 7ff050e07980 20 add_watcher() i=5 2024-03-09T14:48:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.789+0000 7ff050e07980 20 add_watcher() i=6 2024-03-09T14:48:22.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.789+0000 7ff050e07980 20 add_watcher() i=7 2024-03-09T14:48:22.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.789+0000 7ff050e07980 2 all 8 watchers are set, enabling cache 2024-03-09T14:48:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.793+0000 7fef05fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:48:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.793+0000 7ff050e07980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:48:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.793+0000 7ff050e07980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:48:22.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.861+0000 7fef04fe1640 20 reqs_thread_entry: start 2024-03-09T14:48:22.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 20 init_complete bucket index max shards: 1 2024-03-09T14:48:22.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 20 Filter name: none 2024-03-09T14:48:22.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7feefe7fc640 20 reqs_thread_entry: start 2024-03-09T14:48:22.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : miss 2024-03-09T14:48:22.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=486 2024-03-09T14:48:22.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x13 2024-03-09T14:48:22.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 10 adding default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2024-03-09T14:48:22.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4 : miss 2024-03-09T14:48:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=418 2024-03-09T14:48:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4 info.flags=0x17 2024-03-09T14:48:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-conf:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4 to cache LRU end 2024-03-09T14:48:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 10 updating xattr: name=user.rgw.acl bl.length()=209 2024-03-09T14:48:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.865+0000 7ff050e07980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-conf:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4 2024-03-09T14:48:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 10 cls_bucket_list_unordered: got tests/r_test_multipart_defer_complete 2024-03-09T14:48:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 10 cls_bucket_list_unordered: got tests/r_test_multipart_defer_update_complete 2024-03-09T14:48:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 10 cls_bucket_list_unordered: got tests/r_test_multipart_index_versioning 2024-03-09T14:48:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 10 cls_bucket_list_unordered: got tests/r_test_multipart_simple 2024-03-09T14:48:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 10 cls_bucket_list_unordered: got tests/r_test_obj_storage_class 2024-03-09T14:48:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 10 cls_bucket_list_unordered: got tests/r_test_obj_storage_class_copy 2024-03-09T14:48:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 10 cls_bucket_list_unordered: got tests/r_test_obj_storage_class_multipart 2024-03-09T14:48:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 10 cls_bucket_list_unordered: got tests/r_test_obj_storage_class_multipart_copy 2024-03-09T14:48:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 10 cls_bucket_list_unordered: got tests/r_test_small_obj_data 2024-03-09T14:48:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044003e10 obj=test-client.0-conf:tests/r_test_multipart_defer_complete state=0x564c57e8bb58 s->prefetch_data=0 2024-03-09T14:48:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 10 manifest: total_size = 7636 2024-03-09T14:48:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2694798983853814564 2024-03-09T14:48:22.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044003e10 obj=test-client.0-conf:tests/r_test_multipart_defer_complete state=0x564c57e8bb58 s->prefetch_data=0 2024-03-09T14:48:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e8bb58 2024-03-09T14:48:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.869+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.4.0 2024-03-09T14:48:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.873+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.873+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=7636 stripe_ofs=7636 part_ofs=0 rule->part_size=0 2024-03-09T14:48:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.873+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2694798983853814564 2024-03-09T14:48:22.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.873+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:22.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.873+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.GagB3SqRbMBKp1DUcA8oirsIulWiX6m_0 2024-03-09T14:48:22.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.873+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2024-03-09T14:48:22.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.873+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:22.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.873+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.19tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2694798983853814564 2024-03-09T14:48:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.877+0000 7fef327fc640 20 handle_completion(): completion ok for obj=tests/r_test_multipart_defer_complete 2024-03-09T14:48:22.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.877+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044003e10 obj=test-client.0-conf:tests/r_test_multipart_defer_update_complete state=0x564c57e11168 s->prefetch_data=0 2024-03-09T14:48:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.877+0000 7ff050e07980 10 manifest: total_size = 5188 2024-03-09T14:48:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.877+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14667829419773088597 2024-03-09T14:48:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.877+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044003e10 obj=test-client.0-conf:tests/r_test_multipart_defer_update_complete state=0x564c57e11168 s->prefetch_data=0 2024-03-09T14:48:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.877+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e11168 2024-03-09T14:48:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.877+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.4.0 2024-03-09T14:48:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.885+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.885+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5188 stripe_ofs=5188 part_ofs=0 rule->part_size=0 2024-03-09T14:48:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.885+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14667829419773088597 2024-03-09T14:48:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.885+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:22.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.885+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.cd606BUFCfHHbFBKTfYGBhZGXx0OSuq_0 2024-03-09T14:48:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.885+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2024-03-09T14:48:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.885+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.885+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.3tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14667829419773088597 2024-03-09T14:48:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.889+0000 7ff050e07980 20 get_obj_state: octx=0x564c57deff50 obj=test-client.0-conf:tests/r_test_multipart_index_versioning state=0x564c57e11168 s->prefetch_data=0 2024-03-09T14:48:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.889+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=tests/r_test_multipart_defer_update_complete 2024-03-09T14:48:22.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.889+0000 7ff050e07980 10 manifest: total_size = 7383 2024-03-09T14:48:22.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.889+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2270428550249880474 2024-03-09T14:48:22.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.889+0000 7ff050e07980 20 get_obj_state: octx=0x564c57deff50 obj=test-client.0-conf:tests/r_test_multipart_index_versioning state=0x564c57e11168 s->prefetch_data=0 2024-03-09T14:48:22.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.889+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e11168 2024-03-09T14:48:22.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.889+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.4.0 2024-03-09T14:48:22.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.893+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:22.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.893+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=7383 stripe_ofs=7383 part_ofs=0 rule->part_size=0 2024-03-09T14:48:22.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.893+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2270428550249880474 2024-03-09T14:48:22.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.893+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:22.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.893+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.2u5tvqMBdf_LqBM7do0Az_QQ__vd4rE_0 2024-03-09T14:48:22.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.893+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2024-03-09T14:48:22.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.893+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:22.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.893+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.9tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2270428550249880474 2024-03-09T14:48:22.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.897+0000 7fef327fc640 20 handle_completion(): completion ok for obj=tests/r_test_multipart_index_versioning 2024-03-09T14:48:22.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.897+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e166f0 obj=test-client.0-conf:tests/r_test_multipart_simple state=0x564c57e16c88 s->prefetch_data=0 2024-03-09T14:48:22.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.897+0000 7ff050e07980 10 manifest: total_size = 4860 2024-03-09T14:48:22.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.897+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.17513661504900508715 2024-03-09T14:48:22.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.897+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e166f0 obj=test-client.0-conf:tests/r_test_multipart_simple state=0x564c57e16c88 s->prefetch_data=0 2024-03-09T14:48:22.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.897+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16c88 2024-03-09T14:48:22.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.897+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.4.0 2024-03-09T14:48:22.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.905+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:22.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.905+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=4860 stripe_ofs=4860 part_ofs=0 rule->part_size=0 2024-03-09T14:48:22.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.905+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.17513661504900508715 2024-03-09T14:48:22.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.905+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:22.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.905+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.ewxgXrsmdbuUM7dEAoL48fSKuywN4v6_0 2024-03-09T14:48:22.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.905+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2024-03-09T14:48:22.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.905+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:22.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.905+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.6tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.17513661504900508715 2024-03-09T14:48:22.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.905+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=tests/r_test_multipart_simple 2024-03-09T14:48:22.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.905+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044003e10 obj=test-client.0-conf:tests/r_test_obj_storage_class state=0x564c57e14268 s->prefetch_data=0 2024-03-09T14:48:22.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.909+0000 7ff050e07980 10 manifest: total_size = 5158 2024-03-09T14:48:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.909+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.12030105678604887599 2024-03-09T14:48:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.909+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044003e10 obj=test-client.0-conf:tests/r_test_obj_storage_class state=0x564c57e14268 s->prefetch_data=0 2024-03-09T14:48:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.909+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e14268 2024-03-09T14:48:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.909+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.4.0 2024-03-09T14:48:22.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.913+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:22.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.913+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5158 stripe_ofs=5158 part_ofs=0 rule->part_size=0 2024-03-09T14:48:22.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.913+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.12030105678604887599 2024-03-09T14:48:22.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.913+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:22.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.913+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.ucvhn6aR6ALNwPvWkCpM8s3tr-FpxGF_0 2024-03-09T14:48:22.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.913+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2024-03-09T14:48:22.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.913+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:22.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.913+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.8tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.12030105678604887599 2024-03-09T14:48:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.917+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044003e10 obj=test-client.0-conf:tests/r_test_obj_storage_class_copy state=0x564c57e14268 s->prefetch_data=0 2024-03-09T14:48:22.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.917+0000 7fef327fc640 20 handle_completion(): completion ok for obj=tests/r_test_obj_storage_class 2024-03-09T14:48:22.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.917+0000 7ff050e07980 10 manifest: total_size = 15881 2024-03-09T14:48:22.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.917+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.3728002694601589019 2024-03-09T14:48:22.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.917+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044003e10 obj=test-client.0-conf:tests/r_test_obj_storage_class_copy state=0x564c57e14268 s->prefetch_data=0 2024-03-09T14:48:22.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.917+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e14268 2024-03-09T14:48:22.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.917+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.4.0 2024-03-09T14:48:22.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.921+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:22.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.921+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=15881 stripe_ofs=15881 part_ofs=0 rule->part_size=0 2024-03-09T14:48:22.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.921+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.3728002694601589019 2024-03-09T14:48:22.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.921+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:22.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.921+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.QWVRSSM-4q-ufjt8IUHezXzGBNgxdwA_0 2024-03-09T14:48:22.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.921+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2024-03-09T14:48:22.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.921+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:22.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.921+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.17tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.3728002694601589019 2024-03-09T14:48:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.925+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048034bf0 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart state=0x564c57e14268 s->prefetch_data=0 2024-03-09T14:48:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.925+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=tests/r_test_obj_storage_class_copy 2024-03-09T14:48:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.925+0000 7ff050e07980 10 manifest: total_size = 5360 2024-03-09T14:48:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.925+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.1986235792109021368 2024-03-09T14:48:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.925+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048034bf0 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart state=0x564c57e14268 s->prefetch_data=0 2024-03-09T14:48:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.925+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e14268 2024-03-09T14:48:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.925+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.4.0 2024-03-09T14:48:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.929+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.929+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5360 stripe_ofs=5360 part_ofs=0 rule->part_size=0 2024-03-09T14:48:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.929+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.1986235792109021368 2024-03-09T14:48:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.929+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.929+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.0566Emuqp471bh3hg3aCRaQD7K6bALR_0 2024-03-09T14:48:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.929+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2024-03-09T14:48:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.929+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:22.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.929+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.9tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.1986235792109021368 2024-03-09T14:48:22.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.933+0000 7fef327fc640 20 handle_completion(): completion ok for obj=tests/r_test_obj_storage_class_multipart 2024-03-09T14:48:22.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.933+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04000d2a0 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart_copy state=0x564c57e14268 s->prefetch_data=0 2024-03-09T14:48:22.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.933+0000 7ff050e07980 10 manifest: total_size = 7709 2024-03-09T14:48:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.933+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14106023454034700089 2024-03-09T14:48:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.933+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04000d2a0 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart_copy state=0x564c57e14268 s->prefetch_data=0 2024-03-09T14:48:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.933+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e14268 2024-03-09T14:48:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.933+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.4.0 2024-03-09T14:48:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.941+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.941+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=7709 stripe_ofs=7709 part_ofs=0 rule->part_size=0 2024-03-09T14:48:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.941+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14106023454034700089 2024-03-09T14:48:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.941+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.941+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.FuRQbdMmReskxAGawS4Bsu2hFfu8rns_0 2024-03-09T14:48:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.941+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2024-03-09T14:48:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.941+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.941+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.16tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14106023454034700089 2024-03-09T14:48:22.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.945+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=tests/r_test_obj_storage_class_multipart_copy 2024-03-09T14:48:22.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.945+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044003e10 obj=test-client.0-conf:tests/r_test_small_obj_data state=0x564c57e14218 s->prefetch_data=0 2024-03-09T14:48:22.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.945+0000 7ff050e07980 10 manifest: total_size = 5410 2024-03-09T14:48:22.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.945+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2996683048297604930 2024-03-09T14:48:22.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.945+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044003e10 obj=test-client.0-conf:tests/r_test_small_obj_data state=0x564c57e14218 s->prefetch_data=0 2024-03-09T14:48:22.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.945+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e14218 2024-03-09T14:48:22.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.945+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.4.0 2024-03-09T14:48:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5410 stripe_ofs=5410 part_ofs=0 rule->part_size=0 2024-03-09T14:48:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2996683048297604930 2024-03-09T14:48:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.QBQcztc3-0WgNyDnoVJLPQAotXq7zZa_0 2024-03-09T14:48:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2024-03-09T14:48:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.13tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2996683048297604930 2024-03-09T14:48:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:48:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:48:22.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7fef327fc640 20 handle_completion(): completion ok for obj=tests/r_test_small_obj_data 2024-03-09T14:48:22.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:48:22.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:48:22.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:48:22.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:48:22.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.953+0000 7ff050e07980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-conf[8c209108-e1cd-4d45-9608-6f491550b24c.4644.4])"; uploads.size()=0; is_truncated=0 2024-03-09T14:48:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.957+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+test-client.0-conf : miss 2024-03-09T14:48:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.957+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.961+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=204 2024-03-09T14:48:22.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.961+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+test-client.0-conf info.flags=0x11 2024-03-09T14:48:22.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.961+0000 7ff050e07980 10 adding default.rgw.meta+root+test-client.0-conf to cache LRU end 2024-03-09T14:48:22.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.961+0000 7ff050e07980 10 removing default.rgw.meta+root+test-client.0-conf from cache 2024-03-09T14:48:22.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.961+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-conf, ofs0, ns] 2024-03-09T14:48:22.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.961+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562398 cookie 94885891637440 notifier 5946 bl.length()=162 2024-03-09T14:48:22.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.969+0000 7ff050e07980 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-conf:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4 from cache 2024-03-09T14:48:22.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.969+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-conf:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4, ofs0, ns] 2024-03-09T14:48:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.969+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562419 cookie 94885891680208 notifier 5946 bl.length()=219 2024-03-09T14:48:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.977+0000 7ff050e07980 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-conf/8c209108-e1cd-4d45-9608-6f491550b24c.4644.4 : miss 2024-03-09T14:48:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.977+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.977+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:22.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.977+0000 7ff050e07980 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-conf/8c209108-e1cd-4d45-9608-6f491550b24c.4644.4 info.flags=0x0 2024-03-09T14:48:22.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.977+0000 7ff050e07980 10 adding default.rgw.log++pubsub..bucket.test-client.0-conf/8c209108-e1cd-4d45-9608-6f491550b24c.4644.4 to cache LRU end 2024-03-09T14:48:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.981+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.11 : miss 2024-03-09T14:48:22.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.981+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=449 2024-03-09T14:48:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.11 info.flags=0x17 2024-03-09T14:48:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.11 to cache LRU end 2024-03-09T14:48:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 10 updating xattr: name=user.rgw.acl bl.length()=209 2024-03-09T14:48:22.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.11 2024-03-09T14:48:22.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo 2024-03-09T14:48:22.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044125f50 obj=test-client.0-r-test-multipart-defer-complete.1:foo state=0x564c57e0f5f8 s->prefetch_data=0 2024-03-09T14:48:22.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 10 manifest: total_size = 26082 2024-03-09T14:48:22.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4002354929203554414 2024-03-09T14:48:22.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044125f50 obj=test-client.0-r-test-multipart-defer-complete.1:foo state=0x564c57e0f5f8 s->prefetch_data=0 2024-03-09T14:48:22.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e0f5f8 2024-03-09T14:48:22.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.985+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.11.0 2024-03-09T14:48:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=7671 rules.size()=3 2024-03-09T14:48:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=7671 2024-03-09T14:48:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=7671 stripe_ofs=7671 part_ofs=7671 rule->part_size=7671 2024-03-09T14:48:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=7671 rules.size()=3 2024-03-09T14:48:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6299127 part_ofs=7671 rule->part_size=7671 2024-03-09T14:48:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=15342 stripe_ofs=15342 part_ofs=15342 rule->part_size=7673 2024-03-09T14:48:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=7673 rules.size()=3 2024-03-09T14:48:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6306798 part_ofs=15342 rule->part_size=7673 2024-03-09T14:48:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=23015 stripe_ofs=23015 part_ofs=23015 rule->part_size=3067 2024-03-09T14:48:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=3067 rules.size()=3 2024-03-09T14:48:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6314471 part_ofs=23015 rule->part_size=3067 2024-03-09T14:48:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=26082 stripe_ofs=26082 part_ofs=26082 rule->part_size=3067 2024-03-09T14:48:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4002354929203554414 2024-03-09T14:48:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11__multipart_foo.2~6QP15A_WX6Wb-S4ld-DKSzM7SOVvjTQ.1 2024-03-09T14:48:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 346 2024-03-09T14:48:22.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11__multipart_foo.2~6QP15A_WX6Wb-S4ld-DKSzM7SOVvjTQ.2 2024-03-09T14:48:22.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 592 2024-03-09T14:48:22.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11__multipart_foo.2~6QP15A_WX6Wb-S4ld-DKSzM7SOVvjTQ.3 2024-03-09T14:48:22.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 838 2024-03-09T14:48:22.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11__multipart_foo.2~6QP15A_WX6Wb-S4ld-DKSzM7SOVvjTQ.4 2024-03-09T14:48:22.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1084 2024-03-09T14:48:22.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:22.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.19tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4002354929203554414 2024-03-09T14:48:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.993+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo 2024-03-09T14:48:22.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.997+0000 7ff050e07980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:48:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.997+0000 7ff050e07980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:48:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.997+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:48:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.997+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:48:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.997+0000 7ff050e07980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:48:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.997+0000 7ff050e07980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:48:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:22.997+0000 7ff050e07980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-defer-complete.1[8c209108-e1cd-4d45-9608-6f491550b24c.4644.11])"; uploads.size()=0; is_truncated=0 2024-03-09T14:48:23.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.001+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 : miss 2024-03-09T14:48:23.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.001+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.001+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=235 2024-03-09T14:48:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.001+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 info.flags=0x11 2024-03-09T14:48:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.001+0000 7ff050e07980 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 to cache LRU end 2024-03-09T14:48:23.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.005+0000 7ff050e07980 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 from cache 2024-03-09T14:48:23.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.005+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-multipart-defer-complete.1, ofs0, ns] 2024-03-09T14:48:23.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.005+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562420 cookie 94885891680208 notifier 5946 bl.length()=191 2024-03-09T14:48:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.013+0000 7ff050e07980 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.11 from cache 2024-03-09T14:48:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.013+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.11, ofs0, ns] 2024-03-09T14:48:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.013+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562379 cookie 94885891693184 notifier 5946 bl.length()=249 2024-03-09T14:48:23.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.017+0000 7ff050e07980 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.11 : miss 2024-03-09T14:48:23.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.017+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.021+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:23.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.021+0000 7ff050e07980 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.11 info.flags=0x0 2024-03-09T14:48:23.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.021+0000 7ff050e07980 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.11 to cache LRU end 2024-03-09T14:48:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.12 : miss 2024-03-09T14:48:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=456 2024-03-09T14:48:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.12 info.flags=0x17 2024-03-09T14:48:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.12 to cache LRU end 2024-03-09T14:48:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 10 updating xattr: name=user.rgw.acl bl.length()=209 2024-03-09T14:48:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.12 2024-03-09T14:48:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo 2024-03-09T14:48:23.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04411bd20 obj=test-client.0-r-test-multipart-defer-update-complete.1:foo state=0x564c57e1b998 s->prefetch_data=0 2024-03-09T14:48:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 10 manifest: total_size = 1667 2024-03-09T14:48:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16367796950388320495 2024-03-09T14:48:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04411bd20 obj=test-client.0-r-test-multipart-defer-update-complete.1:foo state=0x564c57e1b998 s->prefetch_data=0 2024-03-09T14:48:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1b998 2024-03-09T14:48:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.025+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.12.0 2024-03-09T14:48:23.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=516 rules.size()=2 2024-03-09T14:48:23.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=516 2024-03-09T14:48:23.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=516 stripe_ofs=516 part_ofs=516 rule->part_size=516 2024-03-09T14:48:23.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=516 rules.size()=2 2024-03-09T14:48:23.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291972 part_ofs=516 rule->part_size=516 2024-03-09T14:48:23.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=1032 stripe_ofs=1032 part_ofs=1032 rule->part_size=516 2024-03-09T14:48:23.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=516 rules.size()=2 2024-03-09T14:48:23.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6292488 part_ofs=1032 rule->part_size=516 2024-03-09T14:48:23.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=1548 stripe_ofs=1548 part_ofs=1548 rule->part_size=119 2024-03-09T14:48:23.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=119 rules.size()=2 2024-03-09T14:48:23.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6293004 part_ofs=1548 rule->part_size=119 2024-03-09T14:48:23.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=1667 stripe_ofs=1667 part_ofs=1667 rule->part_size=119 2024-03-09T14:48:23.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16367796950388320495 2024-03-09T14:48:23.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.12__multipart_foo.2~YojN6LMtFT97TQR8YUGgGk0T-dv6dC2.1 2024-03-09T14:48:23.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 347 2024-03-09T14:48:23.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.12__multipart_foo.2~YojN6LMtFT97TQR8YUGgGk0T-dv6dC2.2 2024-03-09T14:48:23.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 593 2024-03-09T14:48:23.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.12__multipart_foo.2~YojN6LMtFT97TQR8YUGgGk0T-dv6dC2.3 2024-03-09T14:48:23.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 839 2024-03-09T14:48:23.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.12__multipart_foo.2~YojN6LMtFT97TQR8YUGgGk0T-dv6dC2.4 2024-03-09T14:48:23.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1085 2024-03-09T14:48:23.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.033+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16367796950388320495 2024-03-09T14:48:23.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.037+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo 2024-03-09T14:48:23.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.037+0000 7ff050e07980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:48:23.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.037+0000 7ff050e07980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:48:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.037+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:48:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.037+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:48:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.037+0000 7ff050e07980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:48:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.037+0000 7ff050e07980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:48:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.037+0000 7ff050e07980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-defer-update-complete.1[8c209108-e1cd-4d45-9608-6f491550b24c.4644.12])"; uploads.size()=0; is_truncated=0 2024-03-09T14:48:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.041+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 : miss 2024-03-09T14:48:23.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.041+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.041+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=242 2024-03-09T14:48:23.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.041+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 info.flags=0x11 2024-03-09T14:48:23.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.041+0000 7ff050e07980 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 to cache LRU end 2024-03-09T14:48:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.045+0000 7ff050e07980 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 from cache 2024-03-09T14:48:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.045+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-multipart-defer-update-complete.1, ofs0, ns] 2024-03-09T14:48:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.045+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562431 cookie 94885891684208 notifier 5946 bl.length()=198 2024-03-09T14:48:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.053+0000 7ff050e07980 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.12 from cache 2024-03-09T14:48:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.053+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.12, ofs0, ns] 2024-03-09T14:48:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.053+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562432 cookie 94885891684208 notifier 5946 bl.length()=256 2024-03-09T14:48:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.061+0000 7ff050e07980 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.12 : miss 2024-03-09T14:48:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.061+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.061+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.061+0000 7ff050e07980 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.12 info.flags=0x0 2024-03-09T14:48:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.061+0000 7ff050e07980 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.12 to cache LRU end 2024-03-09T14:48:23.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.065+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.10 : miss 2024-03-09T14:48:23.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.065+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.069+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=451 2024-03-09T14:48:23.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.069+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.10 info.flags=0x17 2024-03-09T14:48:23.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.069+0000 7ff050e07980 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.10 to cache LRU end 2024-03-09T14:48:23.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.069+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:23.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.069+0000 7ff050e07980 10 updating xattr: name=user.rgw.acl bl.length()=209 2024-03-09T14:48:23.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.069+0000 7ff050e07980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.10 2024-03-09T14:48:23.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.069+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo[1T0I794WkVw.n.hXUCqfnF.kFjmf8QV] 2024-03-09T14:48:23.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.069+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.10.0 2024-03-09T14:48:23.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.069+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-multipart-index-versioning.1:foo state=0x564c57e13fc8 s->prefetch_data=0 2024-03-09T14:48:23.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.069+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 2bkducz21zczuauvy7n6ayeirzsd349t 2024-03-09T14:48:23.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.069+0000 7ff050e07980 20 get_obj_state_impl: setting s->olh_tag to uqdv3ovi6tuiksixvx0t92v20bpbis3n 2024-03-09T14:48:23.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.069+0000 7ff050e07980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=uqdv3ovi6tuiksixvx0t92v20bpbis3n 2024-03-09T14:48:23.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.073+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.10.0 2024-03-09T14:48:23.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.077+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.10.0 2024-03-09T14:48:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.077+0000 7ff050e07980 20 olh_log_entry: epoch=3 op=2 key=foo[] 2024-03-09T14:48:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.077+0000 7ff050e07980 20 olh_log_entry: epoch=3 op=3 key=foo[1T0I794WkVw.n.hXUCqfnF.kFjmf8QV] 2024-03-09T14:48:23.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.077+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-multipart-index-versioning.1:_:1T0I794WkVw.n.hXUCqfnF.kFjmf8QV_foo state=0x564c57e21658 s->prefetch_data=0 2024-03-09T14:48:23.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.077+0000 7ff050e07980 10 manifest: total_size = 886297 2024-03-09T14:48:23.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.077+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.7664845607759348143 2024-03-09T14:48:23.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.077+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-multipart-index-versioning.1:_:1T0I794WkVw.n.hXUCqfnF.kFjmf8QV_foo state=0x564c57e21658 s->prefetch_data=0 2024-03-09T14:48:23.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.077+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e21658 2024-03-09T14:48:23.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.077+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.10.0 2024-03-09T14:48:23.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2024-03-09T14:48:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=246247 2024-03-09T14:48:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=246247 stripe_ofs=246247 part_ofs=246247 rule->part_size=246247 2024-03-09T14:48:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2024-03-09T14:48:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6537703 part_ofs=246247 rule->part_size=246247 2024-03-09T14:48:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=492494 stripe_ofs=492494 part_ofs=492494 rule->part_size=246247 2024-03-09T14:48:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2024-03-09T14:48:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6783950 part_ofs=492494 rule->part_size=246247 2024-03-09T14:48:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=738741 stripe_ofs=738741 part_ofs=738741 rule->part_size=147556 2024-03-09T14:48:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=147556 rules.size()=2 2024-03-09T14:48:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=7030197 part_ofs=738741 rule->part_size=147556 2024-03-09T14:48:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=886297 stripe_ofs=886297 part_ofs=886297 rule->part_size=147556 2024-03-09T14:48:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.7664845607759348143 2024-03-09T14:48:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.10__multipart_foo.2~MSsVaeq9Oxxzh-JaYXXBj6-l5LjwqhK.1 2024-03-09T14:48:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 346 2024-03-09T14:48:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.10__multipart_foo.2~MSsVaeq9Oxxzh-JaYXXBj6-l5LjwqhK.2 2024-03-09T14:48:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 592 2024-03-09T14:48:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.10__multipart_foo.2~MSsVaeq9Oxxzh-JaYXXBj6-l5LjwqhK.3 2024-03-09T14:48:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 838 2024-03-09T14:48:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.10__multipart_foo.2~MSsVaeq9Oxxzh-JaYXXBj6-l5LjwqhK.4 2024-03-09T14:48:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1084 2024-03-09T14:48:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.31tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.7664845607759348143 2024-03-09T14:48:23.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.085+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo[1T0I794WkVw.n.hXUCqfnF.kFjmf8QV] 2024-03-09T14:48:23.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.089+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-multipart-index-versioning.1:foo state=0x564c57e13fc8 s->prefetch_data=0 2024-03-09T14:48:23.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.093+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.10.0 2024-03-09T14:48:23.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.097+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.10.0 2024-03-09T14:48:23.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.097+0000 7ff050e07980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:48:23.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.097+0000 7ff050e07980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:48:23.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.097+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:48:23.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.097+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:48:23.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.097+0000 7ff050e07980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:48:23.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.097+0000 7ff050e07980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:48:23.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.097+0000 7ff050e07980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-index-versioning.1[8c209108-e1cd-4d45-9608-6f491550b24c.4644.10])"; uploads.size()=0; is_truncated=0 2024-03-09T14:48:23.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.101+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 : miss 2024-03-09T14:48:23.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.101+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.101+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=237 2024-03-09T14:48:23.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.101+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 info.flags=0x11 2024-03-09T14:48:23.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.101+0000 7ff050e07980 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 to cache LRU end 2024-03-09T14:48:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.105+0000 7ff050e07980 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 from cache 2024-03-09T14:48:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.105+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-multipart-index-versioning.1, ofs0, ns] 2024-03-09T14:48:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.105+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562433 cookie 94885891684208 notifier 5946 bl.length()=193 2024-03-09T14:48:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.109+0000 7ff050e07980 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.10 from cache 2024-03-09T14:48:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.109+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.10, ofs0, ns] 2024-03-09T14:48:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.113+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562399 cookie 94885891637440 notifier 5946 bl.length()=251 2024-03-09T14:48:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.117+0000 7ff050e07980 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.10 : miss 2024-03-09T14:48:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.117+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.117+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:23.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.117+0000 7ff050e07980 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.10 info.flags=0x0 2024-03-09T14:48:23.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.117+0000 7ff050e07980 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.10 to cache LRU end 2024-03-09T14:48:23.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.9 : miss 2024-03-09T14:48:23.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=439 2024-03-09T14:48:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.9 info.flags=0x17 2024-03-09T14:48:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.9 to cache LRU end 2024-03-09T14:48:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 10 updating xattr: name=user.rgw.acl bl.length()=209 2024-03-09T14:48:23.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.9 2024-03-09T14:48:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo 2024-03-09T14:48:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 20 get_obj_state: octx=0x564c57df0100 obj=test-client.0-r-test-multipart-simple.1:foo state=0x564c57e13d48 s->prefetch_data=0 2024-03-09T14:48:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 10 manifest: total_size = 74198 2024-03-09T14:48:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9024160014172828040 2024-03-09T14:48:23.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 20 get_obj_state: octx=0x564c57df0100 obj=test-client.0-r-test-multipart-simple.1:foo state=0x564c57e13d48 s->prefetch_data=0 2024-03-09T14:48:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13d48 2024-03-09T14:48:23.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.125+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.9.0 2024-03-09T14:48:23.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2024-03-09T14:48:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=20613 2024-03-09T14:48:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=20613 stripe_ofs=20613 part_ofs=20613 rule->part_size=20613 2024-03-09T14:48:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2024-03-09T14:48:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6312069 part_ofs=20613 rule->part_size=20613 2024-03-09T14:48:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=41226 stripe_ofs=41226 part_ofs=41226 rule->part_size=20613 2024-03-09T14:48:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2024-03-09T14:48:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6332682 part_ofs=41226 rule->part_size=20613 2024-03-09T14:48:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=61839 stripe_ofs=61839 part_ofs=61839 rule->part_size=12359 2024-03-09T14:48:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=12359 rules.size()=2 2024-03-09T14:48:23.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6353295 part_ofs=61839 rule->part_size=12359 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=74198 stripe_ofs=74198 part_ofs=74198 rule->part_size=12359 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9024160014172828040 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9__multipart_foo.2~JkrYhrO3HjDn-TJHm0koE5L_sheBWIE.1 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 344 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9__multipart_foo.2~JkrYhrO3HjDn-TJHm0koE5L_sheBWIE.2 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 588 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9__multipart_foo.2~JkrYhrO3HjDn-TJHm0koE5L_sheBWIE.3 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 832 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9__multipart_foo.2~JkrYhrO3HjDn-TJHm0koE5L_sheBWIE.4 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1076 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.1tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9024160014172828040 2024-03-09T14:48:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.133+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo 2024-03-09T14:48:23.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.137+0000 7ff050e07980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:48:23.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.137+0000 7ff050e07980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:48:23.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.137+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:48:23.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.137+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:48:23.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.137+0000 7ff050e07980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:48:23.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.137+0000 7ff050e07980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:48:23.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.137+0000 7ff050e07980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-simple.1[8c209108-e1cd-4d45-9608-6f491550b24c.4644.9])"; uploads.size()=0; is_truncated=0 2024-03-09T14:48:23.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.141+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 : miss 2024-03-09T14:48:23.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.141+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.141+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=225 2024-03-09T14:48:23.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.141+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 info.flags=0x11 2024-03-09T14:48:23.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.141+0000 7ff050e07980 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 to cache LRU end 2024-03-09T14:48:23.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.145+0000 7ff050e07980 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 from cache 2024-03-09T14:48:23.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.145+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-multipart-simple.1, ofs0, ns] 2024-03-09T14:48:23.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.145+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562400 cookie 94885891690560 notifier 5946 bl.length()=183 2024-03-09T14:48:23.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.153+0000 7ff050e07980 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.9 from cache 2024-03-09T14:48:23.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.153+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-simple.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.9, ofs0, ns] 2024-03-09T14:48:23.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.153+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562384 cookie 94885891691616 notifier 5946 bl.length()=240 2024-03-09T14:48:23.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.157+0000 7ff050e07980 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-simple.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.9 : miss 2024-03-09T14:48:23.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.157+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.161+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:23.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.161+0000 7ff050e07980 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-simple.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.9 info.flags=0x0 2024-03-09T14:48:23.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.161+0000 7ff050e07980 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-simple.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.9 to cache LRU end 2024-03-09T14:48:23.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.165+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15 : miss 2024-03-09T14:48:23.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.165+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.165+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=447 2024-03-09T14:48:23.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.165+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15 info.flags=0x17 2024-03-09T14:48:23.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.165+0000 7ff050e07980 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15 to cache LRU end 2024-03-09T14:48:23.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.165+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:23.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.165+0000 7ff050e07980 10 updating xattr: name=user.rgw.acl bl.length()=209 2024-03-09T14:48:23.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.165+0000 7ff050e07980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15 2024-03-09T14:48:23.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-FROZEN 2024-03-09T14:48:23.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-FROZEN-FROZEN 2024-03-09T14:48:23.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-FROZEN-LUKEWARM 2024-03-09T14:48:23.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-FROZEN-STANDARD 2024-03-09T14:48:23.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-LUKEWARM 2024-03-09T14:48:23.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-LUKEWARM-FROZEN 2024-03-09T14:48:23.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-LUKEWARM-LUKEWARM 2024-03-09T14:48:23.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-LUKEWARM-STANDARD 2024-03-09T14:48:23.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-STANDARD 2024-03-09T14:48:23.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-STANDARD-FROZEN 2024-03-09T14:48:23.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-STANDARD-LUKEWARM 2024-03-09T14:48:23.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-STANDARD-STANDARD 2024-03-09T14:48:23.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e14d50 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 10 manifest: total_size = 5242880 2024-03-09T14:48:23.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.5773990313830998531 2024-03-09T14:48:23.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e14d50 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13cd8 2024-03-09T14:48:23.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.169+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.15.0 2024-03-09T14:48:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.177+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.177+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.177+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.5773990313830998531 2024-03-09T14:48:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.177+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.177+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.ehEWaBBPx_0LAoLFpfGo1Zeknu9diW8_0 2024-03-09T14:48:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.177+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 337 2024-03-09T14:48:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.177+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.177+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.22tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.5773990313830998531 2024-03-09T14:48:23.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.177+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-FROZEN 2024-03-09T14:48:23.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.181+0000 7ff050e07980 20 get_obj_state: octx=0x7ff040082f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-FROZEN state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.181+0000 7ff050e07980 10 manifest: total_size = 5242880 2024-03-09T14:48:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.181+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.181+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.181+0000 7ff050e07980 10 generate_fake_tag new tag=8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.ehEWaBBPx_0LAoLFpfGo1Zeknu9diW8_1_2646a65bd3b80e7732a4f9cee15a21d7 2024-03-09T14:48:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.181+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.ehEWaBBPx_0LAoLFpfGo1Zeknu9diW8_1_2646a65bd3b80e7732a4f9cee15a21d7 2024-03-09T14:48:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.181+0000 7ff050e07980 20 get_obj_state: octx=0x7ff040082f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-FROZEN state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.181+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13cd8 2024-03-09T14:48:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.181+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.15.0 2024-03-09T14:48:23.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.185+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.185+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.185+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.ehEWaBBPx_0LAoLFpfGo1Zeknu9diW8_1_2646a65bd3b80e7732a4f9cee15a21d7 2024-03-09T14:48:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.185+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.185+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.ehEWaBBPx_0LAoLFpfGo1Zeknu9diW8_0 2024-03-09T14:48:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.185+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 396 2024-03-09T14:48:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.185+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.185+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.22tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.ehEWaBBPx_0LAoLFpfGo1Zeknu9diW8_1_2646a65bd3b80e7732a4f9cee15a21d7 2024-03-09T14:48:23.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.189+0000 7ff050e07980 20 get_obj_state: octx=0x7ff040082f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-LUKEWARM state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.189+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-FROZEN-FROZEN 2024-03-09T14:48:23.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.189+0000 7ff050e07980 10 manifest: total_size = 5242880 2024-03-09T14:48:23.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.189+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to _6zE_D3DnOOxg9xH_ZkKf4Ye-a6z6J31 2024-03-09T14:48:23.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.189+0000 7ff050e07980 20 get_obj_state: octx=0x7ff040082f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-LUKEWARM state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.189+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13cd8 2024-03-09T14:48:23.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.189+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.15.0 2024-03-09T14:48:23.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.197+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.197+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.197+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: _6zE_D3DnOOxg9xH_ZkKf4Ye-a6z6J31 2024-03-09T14:48:23.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.197+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.197+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.FMUX1JBrs0pSWsFvs13oOPxkmxO24KS_0 2024-03-09T14:48:23.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.197+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 310 2024-03-09T14:48:23.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.197+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.197+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: _6zE_D3DnOOxg9xH_ZkKf4Ye-a6z6J31 2024-03-09T14:48:23.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.201+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-FROZEN-LUKEWARM 2024-03-09T14:48:23.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.201+0000 7ff050e07980 20 get_obj_state: octx=0x7ff040032490 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-STANDARD state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.201+0000 7ff050e07980 10 manifest: total_size = 5242880 2024-03-09T14:48:23.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.201+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to _2WVWqnDFZT-Z4WQWlJGi7YMQOma_Vhb 2024-03-09T14:48:23.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.201+0000 7ff050e07980 20 get_obj_state: octx=0x7ff040032490 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-STANDARD state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.201+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13cd8 2024-03-09T14:48:23.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.201+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.15.0 2024-03-09T14:48:23.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.205+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.205+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.205+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: _2WVWqnDFZT-Z4WQWlJGi7YMQOma_Vhb 2024-03-09T14:48:23.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.205+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.205+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.K2rC165_M7WeAiZsE61r3gabQ8KbHlD_0 2024-03-09T14:48:23.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.205+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 301 2024-03-09T14:48:23.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.205+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.205+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.27tag is: _2WVWqnDFZT-Z4WQWlJGi7YMQOma_Vhb 2024-03-09T14:48:23.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.209+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-FROZEN-STANDARD 2024-03-09T14:48:23.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.209+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04000d990 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.209+0000 7ff050e07980 10 manifest: total_size = 5242880 2024-03-09T14:48:23.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.209+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15211718419860944605 2024-03-09T14:48:23.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.209+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04000d990 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.209+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13cd8 2024-03-09T14:48:23.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.209+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.15.0 2024-03-09T14:48:23.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.217+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.217+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.217+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15211718419860944605 2024-03-09T14:48:23.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.217+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.217+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.JeUygVC1yQfDqC_IPNqgVkOubTC1qT-_0 2024-03-09T14:48:23.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.217+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 340 2024-03-09T14:48:23.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.217+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.217+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.14tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15211718419860944605 2024-03-09T14:48:23.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.221+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-LUKEWARM 2024-03-09T14:48:23.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.221+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044125f50 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-FROZEN state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.221+0000 7ff050e07980 10 manifest: total_size = 5242880 2024-03-09T14:48:23.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.221+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to _iPOmxjM6KaE4nLwN33Mt9luBFK7J3Bb 2024-03-09T14:48:23.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.221+0000 7ff050e07980 20 get_obj_state: octx=0x7ff044125f50 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-FROZEN state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.221+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13cd8 2024-03-09T14:48:23.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.221+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.15.0 2024-03-09T14:48:23.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.229+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.229+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.229+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: _iPOmxjM6KaE4nLwN33Mt9luBFK7J3Bb 2024-03-09T14:48:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.229+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.229+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.am4nZkfyda-CXJgV6DAEI9-rA5g8prB_0 2024-03-09T14:48:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.229+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 308 2024-03-09T14:48:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.229+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.229+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.27tag is: _iPOmxjM6KaE4nLwN33Mt9luBFK7J3Bb 2024-03-09T14:48:23.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.233+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-FROZEN 2024-03-09T14:48:23.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.233+0000 7ff050e07980 20 get_obj_state: octx=0x7ff0440ef470 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-LUKEWARM state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.233+0000 7ff050e07980 10 manifest: total_size = 5242880 2024-03-09T14:48:23.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.233+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.233+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.233+0000 7ff050e07980 10 generate_fake_tag new tag=8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.JeUygVC1yQfDqC_IPNqgVkOubTC1qT-_1_91928a36ca750d9150dccd2ff36d26b8 2024-03-09T14:48:23.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.233+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.JeUygVC1yQfDqC_IPNqgVkOubTC1qT-_1_91928a36ca750d9150dccd2ff36d26b8 2024-03-09T14:48:23.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.233+0000 7ff050e07980 20 get_obj_state: octx=0x7ff0440ef470 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-LUKEWARM state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.233+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13cd8 2024-03-09T14:48:23.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.233+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.15.0 2024-03-09T14:48:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.241+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.241+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.241+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.JeUygVC1yQfDqC_IPNqgVkOubTC1qT-_1_91928a36ca750d9150dccd2ff36d26b8 2024-03-09T14:48:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.241+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.241+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.JeUygVC1yQfDqC_IPNqgVkOubTC1qT-_0 2024-03-09T14:48:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.241+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 398 2024-03-09T14:48:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.241+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.241+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.26tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.JeUygVC1yQfDqC_IPNqgVkOubTC1qT-_1_91928a36ca750d9150dccd2ff36d26b8 2024-03-09T14:48:23.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.245+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-LUKEWARM 2024-03-09T14:48:23.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.245+0000 7ff050e07980 20 get_obj_state: octx=0x7ff0440ef470 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-STANDARD state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.245+0000 7ff050e07980 10 manifest: total_size = 5242880 2024-03-09T14:48:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.245+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to _cAoOva1YdXWJE2zH4WiIwMcC3vGTdVU 2024-03-09T14:48:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.245+0000 7ff050e07980 20 get_obj_state: octx=0x7ff0440ef470 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-STANDARD state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.245+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13cd8 2024-03-09T14:48:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.245+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.15.0 2024-03-09T14:48:23.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.249+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.249+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.249+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: _cAoOva1YdXWJE2zH4WiIwMcC3vGTdVU 2024-03-09T14:48:23.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.249+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.249+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.R9hHXYlSeqza3oyBXe5Q55Jvr-ebV4a_0 2024-03-09T14:48:23.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.249+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 301 2024-03-09T14:48:23.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.249+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.253+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.27tag is: _cAoOva1YdXWJE2zH4WiIwMcC3vGTdVU 2024-03-09T14:48:23.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.257+0000 7ff050e07980 20 get_obj_state: octx=0x7ff0440ef470 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.257+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-STANDARD 2024-03-09T14:48:23.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.257+0000 7ff050e07980 10 manifest: total_size = 22651 2024-03-09T14:48:23.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.257+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11763471558483991125 2024-03-09T14:48:23.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.257+0000 7ff050e07980 20 get_obj_state: octx=0x7ff0440ef470 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.257+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13cd8 2024-03-09T14:48:23.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.257+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.15.0 2024-03-09T14:48:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.261+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.261+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=22651 stripe_ofs=22651 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.261+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11763471558483991125 2024-03-09T14:48:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.261+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.261+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.K7A44frPkoX30Kh2NETJKndjf8Z7j22_0 2024-03-09T14:48:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.261+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 331 2024-03-09T14:48:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.261+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.261+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11763471558483991125 2024-03-09T14:48:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.265+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-STANDARD 2024-03-09T14:48:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.265+0000 7ff050e07980 20 get_obj_state: octx=0x7ff040082f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-FROZEN state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.265+0000 7ff050e07980 10 manifest: total_size = 22651 2024-03-09T14:48:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.265+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to _lKU5NSfyeT97Uul9hdQgAJuOZM-VvN2 2024-03-09T14:48:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.265+0000 7ff050e07980 20 get_obj_state: octx=0x7ff040082f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-FROZEN state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.265+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13cd8 2024-03-09T14:48:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.265+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.15.0 2024-03-09T14:48:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.273+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.273+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=22651 stripe_ofs=22651 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.273+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: _lKU5NSfyeT97Uul9hdQgAJuOZM-VvN2 2024-03-09T14:48:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.273+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.273+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.hQfaoR55zzlk6TfGoc_SdciF006tP75_0 2024-03-09T14:48:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.273+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 308 2024-03-09T14:48:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.273+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.273+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.12tag is: _lKU5NSfyeT97Uul9hdQgAJuOZM-VvN2 2024-03-09T14:48:23.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.277+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-STANDARD-FROZEN 2024-03-09T14:48:23.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.277+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04000d990 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-LUKEWARM state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.277+0000 7ff050e07980 10 manifest: total_size = 22651 2024-03-09T14:48:23.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.277+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to _JE_5-O7EiDLbAYq56EPirG7OZCTG0Pk 2024-03-09T14:48:23.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.277+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04000d990 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-LUKEWARM state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.277+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13cd8 2024-03-09T14:48:23.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.277+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.15.0 2024-03-09T14:48:23.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.281+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.281+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=22651 stripe_ofs=22651 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.281+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: _JE_5-O7EiDLbAYq56EPirG7OZCTG0Pk 2024-03-09T14:48:23.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.281+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.281+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.zQ7D3ZyR40Q77MUVa_M_FDgOaqpc006_0 2024-03-09T14:48:23.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.281+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 310 2024-03-09T14:48:23.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.281+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.281+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: _JE_5-O7EiDLbAYq56EPirG7OZCTG0Pk 2024-03-09T14:48:23.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.285+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-STANDARD-LUKEWARM 2024-03-09T14:48:23.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.285+0000 7ff050e07980 20 get_obj_state: octx=0x7ff040082f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-STANDARD state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.285+0000 7ff050e07980 10 manifest: total_size = 22651 2024-03-09T14:48:23.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.285+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.285+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=22651 stripe_ofs=22651 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.285+0000 7ff050e07980 10 generate_fake_tag new tag=8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.K7A44frPkoX30Kh2NETJKndjf8Z7j22_1_492a66082cf69c835c3e6112cb23f544 2024-03-09T14:48:23.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.285+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.K7A44frPkoX30Kh2NETJKndjf8Z7j22_1_492a66082cf69c835c3e6112cb23f544 2024-03-09T14:48:23.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.285+0000 7ff050e07980 20 get_obj_state: octx=0x7ff040082f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-STANDARD state=0x564c57e13cd8 s->prefetch_data=0 2024-03-09T14:48:23.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.285+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e13cd8 2024-03-09T14:48:23.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.285+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.15.0 2024-03-09T14:48:23.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.293+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.293+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=22651 stripe_ofs=22651 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.293+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.K7A44frPkoX30Kh2NETJKndjf8Z7j22_1_492a66082cf69c835c3e6112cb23f544 2024-03-09T14:48:23.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.293+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.293+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.K7A44frPkoX30Kh2NETJKndjf8Z7j22_0 2024-03-09T14:48:23.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.293+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 389 2024-03-09T14:48:23.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.293+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.293+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.31tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.K7A44frPkoX30Kh2NETJKndjf8Z7j22_1_492a66082cf69c835c3e6112cb23f544 2024-03-09T14:48:23.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.297+0000 7ff050e07980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:48:23.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.297+0000 7ff050e07980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:48:23.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.297+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-STANDARD-STANDARD 2024-03-09T14:48:23.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.297+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:48:23.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.297+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:48:23.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.297+0000 7ff050e07980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:48:23.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.297+0000 7ff050e07980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:48:23.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.297+0000 7ff050e07980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-copy.1[8c209108-e1cd-4d45-9608-6f491550b24c.4644.15])"; uploads.size()=0; is_truncated=0 2024-03-09T14:48:23.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.301+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 : miss 2024-03-09T14:48:23.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.301+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.301+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=233 2024-03-09T14:48:23.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.301+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 info.flags=0x11 2024-03-09T14:48:23.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.301+0000 7ff050e07980 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 to cache LRU end 2024-03-09T14:48:23.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.305+0000 7ff050e07980 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 from cache 2024-03-09T14:48:23.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.305+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-obj-storage-class-copy.1, ofs0, ns] 2024-03-09T14:48:23.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.305+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562434 cookie 94885891684208 notifier 5946 bl.length()=189 2024-03-09T14:48:23.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.313+0000 7ff050e07980 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15 from cache 2024-03-09T14:48:23.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.313+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15, ofs0, ns] 2024-03-09T14:48:23.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.313+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562401 cookie 94885891637440 notifier 5946 bl.length()=247 2024-03-09T14:48:23.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.321+0000 7ff050e07980 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.15 : miss 2024-03-09T14:48:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.321+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.321+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.321+0000 7ff050e07980 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.15 info.flags=0x0 2024-03-09T14:48:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.321+0000 7ff050e07980 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.15 to cache LRU end 2024-03-09T14:48:23.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16 : miss 2024-03-09T14:48:23.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=457 2024-03-09T14:48:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16 info.flags=0x17 2024-03-09T14:48:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16 to cache LRU end 2024-03-09T14:48:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 updating xattr: name=user.rgw.acl bl.length()=209 2024-03-09T14:48:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-FROZEN 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-FROZEN-FROZEN 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-FROZEN-LUKEWARM 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-FROZEN-STANDARD 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-LUKEWARM 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-LUKEWARM-FROZEN 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-LUKEWARM-LUKEWARM 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-LUKEWARM-STANDARD 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-STANDARD 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-STANDARD-FROZEN 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-STANDARD-LUKEWARM 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-STANDARD-STANDARD 2024-03-09T14:48:23.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.329+0000 7ff050e07980 20 get_obj_state: octx=0x564c57de1d30 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.333+0000 7ff050e07980 10 manifest: total_size = 11534336 2024-03-09T14:48:23.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.333+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9475011269421850760 2024-03-09T14:48:23.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.333+0000 7ff050e07980 20 get_obj_state: octx=0x564c57de1d30 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.333+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16958 2024-03-09T14:48:23.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.333+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.16.0 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9475011269421850760 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.1 2024-03-09T14:48:23.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 367 2024-03-09T14:48:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.2 2024-03-09T14:48:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 634 2024-03-09T14:48:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.3 2024-03-09T14:48:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 901 2024-03-09T14:48:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.337+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.24tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9475011269421850760 2024-03-09T14:48:23.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.341+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04411bd20 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-FROZEN state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.341+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-FROZEN 2024-03-09T14:48:23.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.341+0000 7ff050e07980 10 manifest: total_size = 11534336 2024-03-09T14:48:23.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.341+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.341+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2024-03-09T14:48:23.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.341+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.341+0000 7ff050e07980 10 generate_fake_tag new tag=8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.2_49163a9c25627249aa24353e2c2b13b6 2024-03-09T14:48:23.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.341+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.2_49163a9c25627249aa24353e2c2b13b6 2024-03-09T14:48:23.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.341+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04411bd20 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-FROZEN state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.341+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16958 2024-03-09T14:48:23.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.341+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.16.0 2024-03-09T14:48:23.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2024-03-09T14:48:23.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2024-03-09T14:48:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2024-03-09T14:48:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2024-03-09T14:48:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2024-03-09T14:48:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.2_49163a9c25627249aa24353e2c2b13b6 2024-03-09T14:48:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.1 2024-03-09T14:48:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 441 2024-03-09T14:48:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.2 2024-03-09T14:48:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 708 2024-03-09T14:48:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.3 2024-03-09T14:48:23.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 975 2024-03-09T14:48:23.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.349+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.3tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.2_49163a9c25627249aa24353e2c2b13b6 2024-03-09T14:48:23.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.353+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-FROZEN-FROZEN 2024-03-09T14:48:23.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.353+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04005bab0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-LUKEWARM state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.353+0000 7ff050e07980 10 manifest: total_size = 11534336 2024-03-09T14:48:23.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.353+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to _XgO7GyB48tqkeyPWpuw2F4lss2fWUMv 2024-03-09T14:48:23.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.353+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04005bab0 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-LUKEWARM state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.353+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16958 2024-03-09T14:48:23.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.353+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.16.0 2024-03-09T14:48:23.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.361+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.361+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.361+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.361+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.361+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: _XgO7GyB48tqkeyPWpuw2F4lss2fWUMv 2024-03-09T14:48:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.361+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.361+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.HhU-pMKO4BONFlReICRhhHT3RHFhB4s_0 2024-03-09T14:48:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.361+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 310 2024-03-09T14:48:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.361+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.HhU-pMKO4BONFlReICRhhHT3RHFhB4s_1 2024-03-09T14:48:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.361+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 549 2024-03-09T14:48:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.361+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.361+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.15tag is: _XgO7GyB48tqkeyPWpuw2F4lss2fWUMv 2024-03-09T14:48:23.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.365+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-FROZEN-LUKEWARM 2024-03-09T14:48:23.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.365+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-STANDARD state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.365+0000 7ff050e07980 10 manifest: total_size = 11534336 2024-03-09T14:48:23.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.365+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to _w4wX0ZjCwcErA_sEIOl6KEdhp2gNRK8 2024-03-09T14:48:23.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.365+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-STANDARD state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.365+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16958 2024-03-09T14:48:23.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.365+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.16.0 2024-03-09T14:48:23.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: _w4wX0ZjCwcErA_sEIOl6KEdhp2gNRK8 2024-03-09T14:48:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.PRJj2nODkvBDgmQUCXF8EPXaGcYcZ_I_0 2024-03-09T14:48:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 301 2024-03-09T14:48:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.PRJj2nODkvBDgmQUCXF8EPXaGcYcZ_I_1 2024-03-09T14:48:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 531 2024-03-09T14:48:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.19tag is: _w4wX0ZjCwcErA_sEIOl6KEdhp2gNRK8 2024-03-09T14:48:23.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.373+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-FROZEN-STANDARD 2024-03-09T14:48:23.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.377+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.377+0000 7ff050e07980 10 manifest: total_size = 11534336 2024-03-09T14:48:23.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.377+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2317020279843432629 2024-03-09T14:48:23.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.377+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.377+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16958 2024-03-09T14:48:23.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.377+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.16.0 2024-03-09T14:48:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2024-03-09T14:48:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2317020279843432629 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.1 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 373 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.2 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 646 2024-03-09T14:48:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.3 2024-03-09T14:48:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 919 2024-03-09T14:48:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.381+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.3tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2317020279843432629 2024-03-09T14:48:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.385+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-LUKEWARM 2024-03-09T14:48:23.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.385+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e1c110 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-FROZEN state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.389+0000 7ff050e07980 10 manifest: total_size = 11534336 2024-03-09T14:48:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.389+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to _Vg3UWQoIHbBn5LughNpupS2b_VFdF8O 2024-03-09T14:48:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.389+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e1c110 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-FROZEN state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.389+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16958 2024-03-09T14:48:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.389+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.16.0 2024-03-09T14:48:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.393+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.393+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.393+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.393+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.393+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: _Vg3UWQoIHbBn5LughNpupS2b_VFdF8O 2024-03-09T14:48:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.393+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.393+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.TgcdGeKwhJZuhCE33iNomtBXECVBLz6_0 2024-03-09T14:48:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.393+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 308 2024-03-09T14:48:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.393+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.TgcdGeKwhJZuhCE33iNomtBXECVBLz6_1 2024-03-09T14:48:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.393+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 545 2024-03-09T14:48:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.393+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.393+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.9tag is: _Vg3UWQoIHbBn5LughNpupS2b_VFdF8O 2024-03-09T14:48:23.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.397+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-FROZEN 2024-03-09T14:48:23.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.397+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04411bd20 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-LUKEWARM state=0x564c57e1cfb8 s->prefetch_data=0 2024-03-09T14:48:23.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.401+0000 7ff050e07980 10 manifest: total_size = 11534336 2024-03-09T14:48:23.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.401+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.401+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2024-03-09T14:48:23.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.401+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.401+0000 7ff050e07980 10 generate_fake_tag new tag=8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.2_d15e3a387bdcc02287b7f0a248c8905b 2024-03-09T14:48:23.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.401+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.2_d15e3a387bdcc02287b7f0a248c8905b 2024-03-09T14:48:23.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.401+0000 7ff050e07980 20 get_obj_state: octx=0x7ff04411bd20 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-LUKEWARM state=0x564c57e1cfb8 s->prefetch_data=0 2024-03-09T14:48:23.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.401+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cfb8 2024-03-09T14:48:23.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.401+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.16.0 2024-03-09T14:48:23.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.2_d15e3a387bdcc02287b7f0a248c8905b 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.1 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 449 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.2 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 722 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.3 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 995 2024-03-09T14:48:23.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.405+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.19tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.2_d15e3a387bdcc02287b7f0a248c8905b 2024-03-09T14:48:23.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.409+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-LUKEWARM 2024-03-09T14:48:23.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.409+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-STANDARD state=0x564c57e1cfb8 s->prefetch_data=0 2024-03-09T14:48:23.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.409+0000 7ff050e07980 10 manifest: total_size = 11534336 2024-03-09T14:48:23.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.409+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to _uTDLojna3U_-eEF3ss3mW3Np2WJxPHv 2024-03-09T14:48:23.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.409+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-STANDARD state=0x564c57e1cfb8 s->prefetch_data=0 2024-03-09T14:48:23.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.409+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cfb8 2024-03-09T14:48:23.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.413+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.16.0 2024-03-09T14:48:23.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.417+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.417+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.417+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.417+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.417+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: _uTDLojna3U_-eEF3ss3mW3Np2WJxPHv 2024-03-09T14:48:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.417+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.417+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.8UCPaMPgVOnLdRqT4AuTvGiwrv9b3TG_0 2024-03-09T14:48:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.417+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 301 2024-03-09T14:48:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.417+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.8UCPaMPgVOnLdRqT4AuTvGiwrv9b3TG_1 2024-03-09T14:48:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.417+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 531 2024-03-09T14:48:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.417+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.417+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.30tag is: _uTDLojna3U_-eEF3ss3mW3Np2WJxPHv 2024-03-09T14:48:23.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.421+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e1e340 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD state=0x564c57e1cfb8 s->prefetch_data=0 2024-03-09T14:48:23.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.421+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-STANDARD 2024-03-09T14:48:23.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.421+0000 7ff050e07980 10 manifest: total_size = 16383 2024-03-09T14:48:23.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.421+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15357775618702116300 2024-03-09T14:48:23.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.421+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e1e340 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD state=0x564c57e1cfb8 s->prefetch_data=0 2024-03-09T14:48:23.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.421+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cfb8 2024-03-09T14:48:23.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.421+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.16.0 2024-03-09T14:48:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=7674 rules.size()=2 2024-03-09T14:48:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=7674 2024-03-09T14:48:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=7674 stripe_ofs=7674 part_ofs=7674 rule->part_size=7674 2024-03-09T14:48:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=7674 rules.size()=2 2024-03-09T14:48:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6299130 part_ofs=7674 rule->part_size=7674 2024-03-09T14:48:23.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=15348 stripe_ofs=15348 part_ofs=15348 rule->part_size=1035 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=1035 rules.size()=2 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6306804 part_ofs=15348 rule->part_size=1035 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=16383 stripe_ofs=16383 part_ofs=16383 rule->part_size=1035 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15357775618702116300 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.1 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 365 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.2 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 629 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.3 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 893 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.429+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.22tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15357775618702116300 2024-03-09T14:48:23.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.433+0000 7ff050e07980 20 get_obj_state: octx=0x564c57c0aa70 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-FROZEN state=0x564c57e1cfb8 s->prefetch_data=0 2024-03-09T14:48:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.433+0000 7ff050e07980 10 manifest: total_size = 16383 2024-03-09T14:48:23.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.433+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to _ncer9XPVEkGlCSGLfui_KFDOQmvzhpr 2024-03-09T14:48:23.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.433+0000 7ff050e07980 20 get_obj_state: octx=0x564c57c0aa70 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-FROZEN state=0x564c57e1cfb8 s->prefetch_data=0 2024-03-09T14:48:23.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.433+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cfb8 2024-03-09T14:48:23.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.433+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.16.0 2024-03-09T14:48:23.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.437+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-STANDARD 2024-03-09T14:48:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.441+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.441+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=16383 stripe_ofs=16383 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.441+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: _ncer9XPVEkGlCSGLfui_KFDOQmvzhpr 2024-03-09T14:48:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.441+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.441+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.uZ8j_bwWmyYzcQhApZhEFEZJ-scwUQv_0 2024-03-09T14:48:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.441+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 308 2024-03-09T14:48:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.441+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.441+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.16tag is: _ncer9XPVEkGlCSGLfui_KFDOQmvzhpr 2024-03-09T14:48:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.445+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-LUKEWARM state=0x564c57e1cfb8 s->prefetch_data=0 2024-03-09T14:48:23.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.445+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-STANDARD-FROZEN 2024-03-09T14:48:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.445+0000 7ff050e07980 10 manifest: total_size = 16383 2024-03-09T14:48:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.445+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to _wOUf_3YSVi9eBYX04y7Qtu7Lvc6pOF7 2024-03-09T14:48:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.445+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-LUKEWARM state=0x564c57e1cfb8 s->prefetch_data=0 2024-03-09T14:48:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.445+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cfb8 2024-03-09T14:48:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.445+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.16.0 2024-03-09T14:48:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.453+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.453+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=16383 stripe_ofs=16383 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.453+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: _wOUf_3YSVi9eBYX04y7Qtu7Lvc6pOF7 2024-03-09T14:48:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.453+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.453+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.K19KTQwDT_7HP7rCPnwozXs0sIJ_JGe_0 2024-03-09T14:48:23.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.453+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 310 2024-03-09T14:48:23.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.453+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.453+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.5tag is: _wOUf_3YSVi9eBYX04y7Qtu7Lvc6pOF7 2024-03-09T14:48:23.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.453+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-STANDARD-LUKEWARM 2024-03-09T14:48:23.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.453+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-STANDARD state=0x564c57e1cfb8 s->prefetch_data=0 2024-03-09T14:48:23.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.457+0000 7ff050e07980 10 manifest: total_size = 16383 2024-03-09T14:48:23.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.457+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=7674 rules.size()=2 2024-03-09T14:48:23.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.457+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=7674 2024-03-09T14:48:23.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.457+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=7674 stripe_ofs=7674 part_ofs=7674 rule->part_size=7674 2024-03-09T14:48:23.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.457+0000 7ff050e07980 10 generate_fake_tag new tag=8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.2_642486375c9191b85f4f634810046761 2024-03-09T14:48:23.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.457+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.2_642486375c9191b85f4f634810046761 2024-03-09T14:48:23.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.457+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-STANDARD state=0x564c57e1cfb8 s->prefetch_data=0 2024-03-09T14:48:23.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.457+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cfb8 2024-03-09T14:48:23.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.457+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.16.0 2024-03-09T14:48:23.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=7674 rules.size()=2 2024-03-09T14:48:23.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=7674 2024-03-09T14:48:23.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=7674 stripe_ofs=7674 part_ofs=7674 rule->part_size=7674 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=7674 rules.size()=2 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6299130 part_ofs=7674 rule->part_size=7674 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=15348 stripe_ofs=15348 part_ofs=15348 rule->part_size=1035 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=1035 rules.size()=2 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6306804 part_ofs=15348 rule->part_size=1035 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=16383 stripe_ofs=16383 part_ofs=16383 rule->part_size=1035 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.2_642486375c9191b85f4f634810046761 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.1 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 440 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.2 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 704 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.3 2024-03-09T14:48:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 968 2024-03-09T14:48:23.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.461+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.14tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.2_642486375c9191b85f4f634810046761 2024-03-09T14:48:23.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.465+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-STANDARD-STANDARD 2024-03-09T14:48:23.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.465+0000 7ff050e07980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:48:23.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.465+0000 7ff050e07980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:48:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.469+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:48:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.469+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:48:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.469+0000 7ff050e07980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:48:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.469+0000 7ff050e07980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:48:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.469+0000 7ff050e07980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-multipart-copy.1[8c209108-e1cd-4d45-9608-6f491550b24c.4644.16])"; uploads.size()=0; is_truncated=0 2024-03-09T14:48:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.473+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 : miss 2024-03-09T14:48:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.473+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.473+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=243 2024-03-09T14:48:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.473+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 info.flags=0x11 2024-03-09T14:48:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.473+0000 7ff050e07980 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 to cache LRU end 2024-03-09T14:48:23.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.477+0000 7ff050e07980 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 from cache 2024-03-09T14:48:23.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.477+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart-copy.1, ofs0, ns] 2024-03-09T14:48:23.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.477+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562385 cookie 94885891686912 notifier 5946 bl.length()=199 2024-03-09T14:48:23.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.485+0000 7ff050e07980 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16 from cache 2024-03-09T14:48:23.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.485+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16, ofs0, ns] 2024-03-09T14:48:23.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.485+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562386 cookie 94885891686912 notifier 5946 bl.length()=257 2024-03-09T14:48:23.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.493+0000 7ff050e07980 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.16 : miss 2024-03-09T14:48:23.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.493+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.493+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:23.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.493+0000 7ff050e07980 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.16 info.flags=0x0 2024-03-09T14:48:23.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.493+0000 7ff050e07980 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.16 to cache LRU end 2024-03-09T14:48:23.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.501+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14 : miss 2024-03-09T14:48:23.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.501+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.501+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=452 2024-03-09T14:48:23.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.501+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14 info.flags=0x17 2024-03-09T14:48:23.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.501+0000 7ff050e07980 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14 to cache LRU end 2024-03-09T14:48:23.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.501+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:23.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.501+0000 7ff050e07980 10 updating xattr: name=user.rgw.acl bl.length()=209 2024-03-09T14:48:23.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.501+0000 7ff050e07980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14 2024-03-09T14:48:23.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.501+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-FROZEN 2024-03-09T14:48:23.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.501+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-LUKEWARM 2024-03-09T14:48:23.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.501+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-STANDARD 2024-03-09T14:48:23.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.501+0000 7ff050e07980 20 get_obj_state: octx=0x564c57c0c590 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-FROZEN state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.505+0000 7ff050e07980 10 manifest: total_size = 11534336 2024-03-09T14:48:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.505+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.6477235068231068944 2024-03-09T14:48:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.505+0000 7ff050e07980 20 get_obj_state: octx=0x564c57c0c590 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-FROZEN state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.505+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16958 2024-03-09T14:48:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.505+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.14.0 2024-03-09T14:48:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.6477235068231068944 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-FROZEN.2~Ab5Ciq80sRaRAuyeLQrsIvc_YNUaFe2.1 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 367 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-FROZEN.2~Ab5Ciq80sRaRAuyeLQrsIvc_YNUaFe2.2 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 634 2024-03-09T14:48:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-FROZEN.2~Ab5Ciq80sRaRAuyeLQrsIvc_YNUaFe2.3 2024-03-09T14:48:23.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 901 2024-03-09T14:48:23.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.509+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.3tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.6477235068231068944 2024-03-09T14:48:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.513+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-FROZEN 2024-03-09T14:48:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.513+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e1ebf0 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-LUKEWARM state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.513+0000 7ff050e07980 10 manifest: total_size = 11534336 2024-03-09T14:48:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.513+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11339083919106427599 2024-03-09T14:48:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.513+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e1ebf0 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-LUKEWARM state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.513+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16958 2024-03-09T14:48:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.513+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.14.0 2024-03-09T14:48:23.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=5242880 2024-03-09T14:48:23.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2024-03-09T14:48:23.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=11534336 part_ofs=5242880 rule->part_size=5242880 2024-03-09T14:48:23.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=10485760 rule->part_size=1048576 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11339083919106427599 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-LUKEWARM.2~BjQJOEOSmNwAKFK8SlNi5IzXPjPVv6d.1 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 374 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-LUKEWARM.2~BjQJOEOSmNwAKFK8SlNi5IzXPjPVv6d.2 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 647 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-LUKEWARM.2~BjQJOEOSmNwAKFK8SlNi5IzXPjPVv6d.3 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 920 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.521+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.15tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11339083919106427599 2024-03-09T14:48:23.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.525+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-LUKEWARM 2024-03-09T14:48:23.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.525+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e1ed10 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-STANDARD state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.525+0000 7ff050e07980 10 manifest: total_size = 45360 2024-03-09T14:48:23.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.525+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9921280845339081383 2024-03-09T14:48:23.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.525+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e1ed10 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-STANDARD state=0x564c57e16958 s->prefetch_data=0 2024-03-09T14:48:23.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.525+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16958 2024-03-09T14:48:23.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.525+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.14.0 2024-03-09T14:48:23.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2024-03-09T14:48:23.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6291456 part_ofs=0 rule->part_size=20613 2024-03-09T14:48:23.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=20613 stripe_ofs=20613 part_ofs=20613 rule->part_size=20613 2024-03-09T14:48:23.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2024-03-09T14:48:23.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6312069 part_ofs=20613 rule->part_size=20613 2024-03-09T14:48:23.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=41226 stripe_ofs=41226 part_ofs=41226 rule->part_size=4134 2024-03-09T14:48:23.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=4134 rules.size()=2 2024-03-09T14:48:23.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 RGWObjManifest::operator++(): stripe_ofs=6332682 part_ofs=41226 rule->part_size=4134 2024-03-09T14:48:23.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=45360 stripe_ofs=45360 part_ofs=45360 rule->part_size=4134 2024-03-09T14:48:23.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9921280845339081383 2024-03-09T14:48:23.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-STANDARD.2~LR-QRm1F_-76gwLwW4Kv2FQGX0mscth.1 2024-03-09T14:48:23.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 364 2024-03-09T14:48:23.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-STANDARD.2~LR-QRm1F_-76gwLwW4Kv2FQGX0mscth.2 2024-03-09T14:48:23.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 628 2024-03-09T14:48:23.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-STANDARD.2~LR-QRm1F_-76gwLwW4Kv2FQGX0mscth.3 2024-03-09T14:48:23.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 892 2024-03-09T14:48:23.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.533+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.9921280845339081383 2024-03-09T14:48:23.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.537+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-STANDARD 2024-03-09T14:48:23.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.537+0000 7ff050e07980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:48:23.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.537+0000 7ff050e07980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:48:23.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.537+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:48:23.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.537+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:48:23.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.537+0000 7ff050e07980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:48:23.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.537+0000 7ff050e07980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:48:23.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.537+0000 7ff050e07980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-multipart.1[8c209108-e1cd-4d45-9608-6f491550b24c.4644.14])"; uploads.size()=0; is_truncated=0 2024-03-09T14:48:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.541+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 : miss 2024-03-09T14:48:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.541+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.545+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=238 2024-03-09T14:48:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.545+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 info.flags=0x11 2024-03-09T14:48:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.545+0000 7ff050e07980 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 to cache LRU end 2024-03-09T14:48:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.545+0000 7ff050e07980 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 from cache 2024-03-09T14:48:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.545+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart.1, ofs0, ns] 2024-03-09T14:48:23.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.549+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562380 cookie 94885891693184 notifier 5946 bl.length()=194 2024-03-09T14:48:23.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.553+0000 7ff050e07980 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14 from cache 2024-03-09T14:48:23.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.553+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14, ofs0, ns] 2024-03-09T14:48:23.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.557+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562387 cookie 94885891686912 notifier 5946 bl.length()=252 2024-03-09T14:48:23.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.561+0000 7ff050e07980 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.14 : miss 2024-03-09T14:48:23.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.561+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.561+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:23.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.561+0000 7ff050e07980 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.14 info.flags=0x0 2024-03-09T14:48:23.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.561+0000 7ff050e07980 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.14 to cache LRU end 2024-03-09T14:48:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.569+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.13 : miss 2024-03-09T14:48:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.569+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.569+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=442 2024-03-09T14:48:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.569+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.13 info.flags=0x17 2024-03-09T14:48:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.569+0000 7ff050e07980 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.13 to cache LRU end 2024-03-09T14:48:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.569+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.569+0000 7ff050e07980 10 updating xattr: name=user.rgw.acl bl.length()=209 2024-03-09T14:48:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.569+0000 7ff050e07980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.13 2024-03-09T14:48:23.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.573+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-FROZEN 2024-03-09T14:48:23.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.573+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-LUKEWARM 2024-03-09T14:48:23.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.573+0000 7ff050e07980 10 cls_bucket_list_unordered: got foo-STANDARD 2024-03-09T14:48:23.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.573+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e154a0 obj=test-client.0-r-test-obj-storage-class.1:foo-FROZEN state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.573+0000 7ff050e07980 10 manifest: total_size = 10485760 2024-03-09T14:48:23.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.573+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13686339807024696114 2024-03-09T14:48:23.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.573+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e154a0 obj=test-client.0-r-test-obj-storage-class.1:foo-FROZEN state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.573+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cf48 2024-03-09T14:48:23.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.573+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.13.0 2024-03-09T14:48:23.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.581+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.581+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.581+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.581+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.581+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13686339807024696114 2024-03-09T14:48:23.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.581+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.581+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13__shadow_.RrZhGDK-UQZ8M1do_JumiygN_CGx8V-_0 2024-03-09T14:48:23.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.581+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 338 2024-03-09T14:48:23.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.581+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13__shadow_.RrZhGDK-UQZ8M1do_JumiygN_CGx8V-_1 2024-03-09T14:48:23.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.581+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 575 2024-03-09T14:48:23.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.581+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.581+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.29tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13686339807024696114 2024-03-09T14:48:23.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.585+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-FROZEN 2024-03-09T14:48:23.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.585+0000 7ff050e07980 20 get_obj_state: octx=0x564c57debda0 obj=test-client.0-r-test-obj-storage-class.1:foo-LUKEWARM state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.585+0000 7ff050e07980 10 manifest: total_size = 10485760 2024-03-09T14:48:23.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.585+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13707277977245231977 2024-03-09T14:48:23.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.585+0000 7ff050e07980 20 get_obj_state: octx=0x564c57debda0 obj=test-client.0-r-test-obj-storage-class.1:foo-LUKEWARM state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.585+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cf48 2024-03-09T14:48:23.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.585+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.13.0 2024-03-09T14:48:23.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.589+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.589+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.589+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.589+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.589+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13707277977245231977 2024-03-09T14:48:23.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.589+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.589+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13__shadow_._FUyh-tATS0YCp0K2D4uUZEqn73H_PD_0 2024-03-09T14:48:23.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.589+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 340 2024-03-09T14:48:23.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.589+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13__shadow_._FUyh-tATS0YCp0K2D4uUZEqn73H_PD_1 2024-03-09T14:48:23.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.589+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 579 2024-03-09T14:48:23.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.589+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.589+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.1tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13707277977245231977 2024-03-09T14:48:23.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.593+0000 7fef327fc640 20 handle_completion(): completion ok for obj=foo-LUKEWARM 2024-03-09T14:48:23.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.593+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e1b5a0 obj=test-client.0-r-test-obj-storage-class.1:foo-STANDARD state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.597+0000 7ff050e07980 10 manifest: total_size = 10485760 2024-03-09T14:48:23.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.597+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15213637999648483998 2024-03-09T14:48:23.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.597+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e1b5a0 obj=test-client.0-r-test-obj-storage-class.1:foo-STANDARD state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.597+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cf48 2024-03-09T14:48:23.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.597+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.13.0 2024-03-09T14:48:23.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.601+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.601+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=6291456 stripe_ofs=6291456 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.601+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.601+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.601+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15213637999648483998 2024-03-09T14:48:23.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.601+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.601+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13__shadow_.ZxKOD4EmqQaLYg9Sguk6rie-mmGxCts_0 2024-03-09T14:48:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.601+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 331 2024-03-09T14:48:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.601+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13__shadow_.ZxKOD4EmqQaLYg9Sguk6rie-mmGxCts_1 2024-03-09T14:48:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.601+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 561 2024-03-09T14:48:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.601+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.601+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15213637999648483998 2024-03-09T14:48:23.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.605+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=foo-STANDARD 2024-03-09T14:48:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.605+0000 7ff050e07980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:48:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.605+0000 7ff050e07980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:48:23.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.609+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:48:23.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.609+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:48:23.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.609+0000 7ff050e07980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:48:23.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.609+0000 7ff050e07980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:48:23.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.609+0000 7ff050e07980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class.1[8c209108-e1cd-4d45-9608-6f491550b24c.4644.13])"; uploads.size()=0; is_truncated=0 2024-03-09T14:48:23.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.609+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 : miss 2024-03-09T14:48:23.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.609+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.613+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=228 2024-03-09T14:48:23.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.613+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 info.flags=0x11 2024-03-09T14:48:23.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.613+0000 7ff050e07980 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 to cache LRU end 2024-03-09T14:48:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.613+0000 7ff050e07980 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 from cache 2024-03-09T14:48:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.613+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-obj-storage-class.1, ofs0, ns] 2024-03-09T14:48:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.617+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562435 cookie 94885891684208 notifier 5946 bl.length()=184 2024-03-09T14:48:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.621+0000 7ff050e07980 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.13 from cache 2024-03-09T14:48:23.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.621+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.13, ofs0, ns] 2024-03-09T14:48:23.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.625+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562436 cookie 94885891684208 notifier 5946 bl.length()=242 2024-03-09T14:48:23.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.629+0000 7ff050e07980 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.13 : miss 2024-03-09T14:48:23.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.629+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.629+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:23.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.629+0000 7ff050e07980 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.13 info.flags=0x0 2024-03-09T14:48:23.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.629+0000 7ff050e07980 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.13 to cache LRU end 2024-03-09T14:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.637+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.6 : miss 2024-03-09T14:48:23.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.637+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.637+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=437 2024-03-09T14:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.637+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.6 info.flags=0x17 2024-03-09T14:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.637+0000 7ff050e07980 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.6 to cache LRU end 2024-03-09T14:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.637+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.637+0000 7ff050e07980 10 updating xattr: name=user.rgw.acl bl.length()=209 2024-03-09T14:48:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.637+0000 7ff050e07980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.6 2024-03-09T14:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.641+0000 7ff050e07980 10 cls_bucket_list_unordered: got __ 2024-03-09T14:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.641+0000 7ff050e07980 10 cls_bucket_list_unordered: got __ _ 2024-03-09T14:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.641+0000 7ff050e07980 10 cls_bucket_list_unordered: got ___ 2024-03-09T14:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.641+0000 7ff050e07980 10 cls_bucket_list_unordered: got obj 2024-03-09T14:48:23.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.641+0000 7ff050e07980 20 get_obj_state: octx=0x564c57df0540 obj=test-client.0-r-test-small-obj-data.1:__ state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.641+0000 7ff050e07980 10 manifest: total_size = 0 2024-03-09T14:48:23.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.641+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.2190656368832613650 2024-03-09T14:48:23.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.641+0000 7ff050e07980 20 get_obj_state: octx=0x564c57df0540 obj=test-client.0-r-test-small-obj-data.1:__ state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.641+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cf48 2024-03-09T14:48:23.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.641+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.6.0 2024-03-09T14:48:23.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.645+0000 7ff050e07980 20 get_obj_state: octx=0x564c57df0540 obj=test-client.0-r-test-small-obj-data.1:__ _ state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.649+0000 7ff050e07980 10 manifest: total_size = 0 2024-03-09T14:48:23.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.649+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13280623556649372571 2024-03-09T14:48:23.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.649+0000 7ff050e07980 20 get_obj_state: octx=0x564c57df0540 obj=test-client.0-r-test-small-obj-data.1:__ _ state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.649+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cf48 2024-03-09T14:48:23.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.649+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.6.0 2024-03-09T14:48:23.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.649+0000 7fef327fc640 20 handle_completion(): completion ok for obj=__ 2024-03-09T14:48:23.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.653+0000 7ff050e07980 20 get_obj_state: octx=0x7ff0440ef470 obj=test-client.0-r-test-small-obj-data.1:___ state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.657+0000 7ff050e07980 10 manifest: total_size = 0 2024-03-09T14:48:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.657+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.164480006075975116 2024-03-09T14:48:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.657+0000 7ff050e07980 20 get_obj_state: octx=0x7ff0440ef470 obj=test-client.0-r-test-small-obj-data.1:___ state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.657+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cf48 2024-03-09T14:48:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.657+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.6.0 2024-03-09T14:48:23.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.657+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=__ _ 2024-03-09T14:48:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.661+0000 7ff050e07980 20 get_obj_state: octx=0x7ff0440ef470 obj=test-client.0-r-test-small-obj-data.1:obj state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.661+0000 7ff050e07980 10 manifest: total_size = 0 2024-03-09T14:48:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.661+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.6774832539232880715 2024-03-09T14:48:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.661+0000 7ff050e07980 20 get_obj_state: octx=0x7ff0440ef470 obj=test-client.0-r-test-small-obj-data.1:obj state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.661+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cf48 2024-03-09T14:48:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.661+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.6.0 2024-03-09T14:48:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.665+0000 7fef327fc640 20 handle_completion(): completion ok for obj=___ 2024-03-09T14:48:23.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.669+0000 7ff050e07980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:48:23.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.669+0000 7ff050e07980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:48:23.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.673+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=obj 2024-03-09T14:48:23.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.673+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:48:23.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.673+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:48:23.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.673+0000 7ff050e07980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:48:23.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.673+0000 7ff050e07980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:48:23.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.673+0000 7ff050e07980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.1[8c209108-e1cd-4d45-9608-6f491550b24c.4644.6])"; uploads.size()=0; is_truncated=0 2024-03-09T14:48:23.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.677+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 : miss 2024-03-09T14:48:23.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.677+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.677+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=223 2024-03-09T14:48:23.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.677+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 info.flags=0x11 2024-03-09T14:48:23.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.677+0000 7ff050e07980 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 to cache LRU end 2024-03-09T14:48:23.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.681+0000 7ff050e07980 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 from cache 2024-03-09T14:48:23.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.681+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-small-obj-data.1, ofs0, ns] 2024-03-09T14:48:23.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.681+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562421 cookie 94885891680208 notifier 5946 bl.length()=181 2024-03-09T14:48:23.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.689+0000 7ff050e07980 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.6 from cache 2024-03-09T14:48:23.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.689+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.1:8c209108-e1cd-4d45-9608-6f491550b24c.4644.6, ofs0, ns] 2024-03-09T14:48:23.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.689+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562402 cookie 94885891690560 notifier 5946 bl.length()=238 2024-03-09T14:48:23.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.697+0000 7ff050e07980 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.6 : miss 2024-03-09T14:48:23.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.697+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.697+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:23.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.697+0000 7ff050e07980 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.6 info.flags=0x0 2024-03-09T14:48:23.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.697+0000 7ff050e07980 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.1/8c209108-e1cd-4d45-9608-6f491550b24c.4644.6 to cache LRU end 2024-03-09T14:48:23.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:8c209108-e1cd-4d45-9608-6f491550b24c.4644.7 : miss 2024-03-09T14:48:23.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=437 2024-03-09T14:48:23.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:8c209108-e1cd-4d45-9608-6f491550b24c.4644.7 info.flags=0x17 2024-03-09T14:48:23.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:8c209108-e1cd-4d45-9608-6f491550b24c.4644.7 to cache LRU end 2024-03-09T14:48:23.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:23.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 10 updating xattr: name=user.rgw.acl bl.length()=209 2024-03-09T14:48:23.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:8c209108-e1cd-4d45-9608-6f491550b24c.4644.7 2024-03-09T14:48:23.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 10 cls_bucket_list_unordered: got __ 2024-03-09T14:48:23.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 10 cls_bucket_list_unordered: got __ _ 2024-03-09T14:48:23.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 10 cls_bucket_list_unordered: got ___ 2024-03-09T14:48:23.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 10 cls_bucket_list_unordered: got obj 2024-03-09T14:48:23.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e0f5b0 obj=test-client.0-r-test-small-obj-data.2:__ state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 10 manifest: total_size = 2078 2024-03-09T14:48:23.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11924295039160912840 2024-03-09T14:48:23.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e0f5b0 obj=test-client.0-r-test-small-obj-data.2:__ state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16bc8 2024-03-09T14:48:23.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.705+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.7.0 2024-03-09T14:48:23.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.713+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.713+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=2078 stripe_ofs=2078 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.713+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11924295039160912840 2024-03-09T14:48:23.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.713+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.713+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.7__shadow_.06poVVlwvJuNBLPkHB6EV9G0wwCl5F9_0 2024-03-09T14:48:23.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.713+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2024-03-09T14:48:23.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.713+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.713+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.20tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.11924295039160912840 2024-03-09T14:48:23.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.717+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=__ 2024-03-09T14:48:23.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.717+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e0f5b0 obj=test-client.0-r-test-small-obj-data.2:__ _ state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.717+0000 7ff050e07980 10 manifest: total_size = 527 2024-03-09T14:48:23.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.717+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14750674554078757919 2024-03-09T14:48:23.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.717+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e0f5b0 obj=test-client.0-r-test-small-obj-data.2:__ _ state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.717+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16bc8 2024-03-09T14:48:23.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.717+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.7.0 2024-03-09T14:48:23.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.725+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.725+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=527 stripe_ofs=527 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.725+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14750674554078757919 2024-03-09T14:48:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.725+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.725+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.7__shadow_.1ge7qY3D8o3sXVXQcIcHbF3ZSkNtPkB_0 2024-03-09T14:48:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.725+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2024-03-09T14:48:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.725+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.725+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.26tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.14750674554078757919 2024-03-09T14:48:23.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.729+0000 7fef327fc640 20 handle_completion(): completion ok for obj=__ _ 2024-03-09T14:48:23.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.729+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e0f5b0 obj=test-client.0-r-test-small-obj-data.2:___ state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.729+0000 7ff050e07980 10 manifest: total_size = 2078 2024-03-09T14:48:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.729+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13340010106827674520 2024-03-09T14:48:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.729+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e0f5b0 obj=test-client.0-r-test-small-obj-data.2:___ state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.729+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16bc8 2024-03-09T14:48:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.729+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.7.0 2024-03-09T14:48:23.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.733+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.733+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=2078 stripe_ofs=2078 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.733+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13340010106827674520 2024-03-09T14:48:23.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.733+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.733+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.7__shadow_.jla_Quf_BuLresvfBWPO24jxvMXfYr2_0 2024-03-09T14:48:23.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.733+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2024-03-09T14:48:23.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.733+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.733+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.30tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.13340010106827674520 2024-03-09T14:48:23.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.737+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-small-obj-data.2:obj state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.737+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=___ 2024-03-09T14:48:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.737+0000 7ff050e07980 10 manifest: total_size = 524288 2024-03-09T14:48:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.737+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.1348032474127964003 2024-03-09T14:48:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.741+0000 7ff050e07980 20 get_obj_state: octx=0x7ff048016f50 obj=test-client.0-r-test-small-obj-data.2:obj state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.741+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16bc8 2024-03-09T14:48:23.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.741+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.7.0 2024-03-09T14:48:23.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.745+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.745+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=524288 stripe_ofs=524288 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.745+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.1348032474127964003 2024-03-09T14:48:23.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.745+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.745+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.7__shadow_.S27DIO7XxNaZS-YZ9bxyhqZZJNdEVGL_0 2024-03-09T14:48:23.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.745+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2024-03-09T14:48:23.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.745+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.745+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.1348032474127964003 2024-03-09T14:48:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.749+0000 7fef327fc640 20 handle_completion(): completion ok for obj=obj 2024-03-09T14:48:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.749+0000 7ff050e07980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:48:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.749+0000 7ff050e07980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:48:23.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.749+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:48:23.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.749+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:48:23.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.749+0000 7ff050e07980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:48:23.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.749+0000 7ff050e07980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:48:23.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.749+0000 7ff050e07980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.2[8c209108-e1cd-4d45-9608-6f491550b24c.4644.7])"; uploads.size()=0; is_truncated=0 2024-03-09T14:48:23.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.753+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 : miss 2024-03-09T14:48:23.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.753+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.753+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=223 2024-03-09T14:48:23.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.753+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 info.flags=0x11 2024-03-09T14:48:23.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.753+0000 7ff050e07980 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 to cache LRU end 2024-03-09T14:48:23.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.757+0000 7ff050e07980 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 from cache 2024-03-09T14:48:23.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.757+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-small-obj-data.2, ofs0, ns] 2024-03-09T14:48:23.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.757+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562422 cookie 94885891680208 notifier 5946 bl.length()=181 2024-03-09T14:48:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.765+0000 7ff050e07980 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:8c209108-e1cd-4d45-9608-6f491550b24c.4644.7 from cache 2024-03-09T14:48:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.765+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.2:8c209108-e1cd-4d45-9608-6f491550b24c.4644.7, ofs0, ns] 2024-03-09T14:48:23.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.769+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562403 cookie 94885891690560 notifier 5946 bl.length()=238 2024-03-09T14:48:23.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.773+0000 7ff050e07980 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.2/8c209108-e1cd-4d45-9608-6f491550b24c.4644.7 : miss 2024-03-09T14:48:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.773+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.773+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.773+0000 7ff050e07980 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.2/8c209108-e1cd-4d45-9608-6f491550b24c.4644.7 info.flags=0x0 2024-03-09T14:48:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.773+0000 7ff050e07980 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.2/8c209108-e1cd-4d45-9608-6f491550b24c.4644.7 to cache LRU end 2024-03-09T14:48:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:8c209108-e1cd-4d45-9608-6f491550b24c.4644.8 : miss 2024-03-09T14:48:23.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=437 2024-03-09T14:48:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:8c209108-e1cd-4d45-9608-6f491550b24c.4644.8 info.flags=0x17 2024-03-09T14:48:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:8c209108-e1cd-4d45-9608-6f491550b24c.4644.8 to cache LRU end 2024-03-09T14:48:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 10 updating xattr: name=user.rgw.acl bl.length()=209 2024-03-09T14:48:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:8c209108-e1cd-4d45-9608-6f491550b24c.4644.8 2024-03-09T14:48:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 10 cls_bucket_list_unordered: got __ 2024-03-09T14:48:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 10 cls_bucket_list_unordered: got __ _ 2024-03-09T14:48:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 10 cls_bucket_list_unordered: got ___ 2024-03-09T14:48:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 10 cls_bucket_list_unordered: got obj 2024-03-09T14:48:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.781+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e155b0 obj=test-client.0-r-test-small-obj-data.3:__ state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.785+0000 7ff050e07980 10 manifest: total_size = 1048576 2024-03-09T14:48:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.785+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.12764550389043283490 2024-03-09T14:48:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.785+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e155b0 obj=test-client.0-r-test-small-obj-data.3:__ state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.785+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16bc8 2024-03-09T14:48:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.785+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.8.0 2024-03-09T14:48:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.789+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.789+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=1048576 stripe_ofs=1048576 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.789+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.12764550389043283490 2024-03-09T14:48:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.789+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.789+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.8__shadow_.Pev7LwXAuL1O0-LElHQp_ngbPn4VO3F_0 2024-03-09T14:48:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.789+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2024-03-09T14:48:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.789+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.789+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.22tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.12764550389043283490 2024-03-09T14:48:23.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.793+0000 7fef327fc640 20 handle_completion(): completion ok for obj=__ 2024-03-09T14:48:23.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.793+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e155b0 obj=test-client.0-r-test-small-obj-data.3:__ _ state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.793+0000 7ff050e07980 10 manifest: total_size = 4134 2024-03-09T14:48:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.793+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.5914922296680739730 2024-03-09T14:48:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.793+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e155b0 obj=test-client.0-r-test-small-obj-data.3:__ _ state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.793+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16bc8 2024-03-09T14:48:23.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.793+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.8.0 2024-03-09T14:48:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.801+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.801+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=4134 stripe_ofs=4134 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.801+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.5914922296680739730 2024-03-09T14:48:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.801+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.801+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.8__shadow_.Kbi0c8qCk-Xqmtvgp8RB1wsa-6e1A6U_0 2024-03-09T14:48:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.801+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2024-03-09T14:48:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.801+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.801+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.6tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.5914922296680739730 2024-03-09T14:48:23.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.805+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=__ _ 2024-03-09T14:48:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.805+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e155b0 obj=test-client.0-r-test-small-obj-data.3:___ state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.805+0000 7ff050e07980 10 manifest: total_size = 55 2024-03-09T14:48:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.805+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15940397791667813442 2024-03-09T14:48:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.805+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e155b0 obj=test-client.0-r-test-small-obj-data.3:___ state=0x564c57e16bc8 s->prefetch_data=0 2024-03-09T14:48:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.805+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e16bc8 2024-03-09T14:48:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.805+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.8.0 2024-03-09T14:48:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.813+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.813+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=55 stripe_ofs=55 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.813+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15940397791667813442 2024-03-09T14:48:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.813+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.813+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.8__shadow_.5adlYQ7CWuxeRTCex7tU7N8s5keeJWY_0 2024-03-09T14:48:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.813+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2024-03-09T14:48:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.813+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.813+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.22tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.15940397791667813442 2024-03-09T14:48:23.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.817+0000 7fef327fc640 20 handle_completion(): completion ok for obj=___ 2024-03-09T14:48:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.817+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e155b0 obj=test-client.0-r-test-small-obj-data.3:obj state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.817+0000 7ff050e07980 10 manifest: total_size = 1048576 2024-03-09T14:48:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.817+0000 7ff050e07980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.7494590672314839681 2024-03-09T14:48:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.817+0000 7ff050e07980 20 get_obj_state: octx=0x564c57e155b0 obj=test-client.0-r-test-small-obj-data.3:obj state=0x564c57e1cf48 s->prefetch_data=0 2024-03-09T14:48:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.817+0000 7ff050e07980 20 prepare_atomic_modification: state is not atomic. state=0x564c57e1cf48 2024-03-09T14:48:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.817+0000 7ff050e07980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.8.0 2024-03-09T14:48:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.825+0000 7ff050e07980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:48:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.825+0000 7ff050e07980 20 RGWObjManifest::operator++(): result: ofs=1048576 stripe_ofs=1048576 part_ofs=0 rule->part_size=0 2024-03-09T14:48:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.825+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.7494590672314839681 2024-03-09T14:48:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.825+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2024-03-09T14:48:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.825+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.8__shadow_.MT1sPPgci5eFBZW8SgfgB6jRilmn49r_0 2024-03-09T14:48:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.825+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2024-03-09T14:48:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.825+0000 7ff050e07980 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2024-03-09T14:48:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.825+0000 7ff050e07980 20 garbage collection: RGWGC::send_chain - on object name: gc.4tag is: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.7494590672314839681 2024-03-09T14:48:23.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.829+0000 7fef31ffb640 20 handle_completion(): completion ok for obj=obj 2024-03-09T14:48:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.829+0000 7ff050e07980 20 list_objects_ordered: starting attempt 1 2024-03-09T14:48:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.829+0000 7ff050e07980 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2024-03-09T14:48:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.829+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-03-09T14:48:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.829+0000 7ff050e07980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-03-09T14:48:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.829+0000 7ff050e07980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-03-09T14:48:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.829+0000 7ff050e07980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-03-09T14:48:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.829+0000 7ff050e07980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.3[8c209108-e1cd-4d45-9608-6f491550b24c.4644.8])"; uploads.size()=0; is_truncated=0 2024-03-09T14:48:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.833+0000 7ff050e07980 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 : miss 2024-03-09T14:48:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.833+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.833+0000 7ff050e07980 20 rados_obj.operate() r=0 bl.length=223 2024-03-09T14:48:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.833+0000 7ff050e07980 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 info.flags=0x11 2024-03-09T14:48:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.833+0000 7ff050e07980 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 to cache LRU end 2024-03-09T14:48:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.837+0000 7ff050e07980 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 from cache 2024-03-09T14:48:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.837+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-small-obj-data.3, ofs0, ns] 2024-03-09T14:48:23.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.837+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562423 cookie 94885891680208 notifier 5946 bl.length()=181 2024-03-09T14:48:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.845+0000 7ff050e07980 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:8c209108-e1cd-4d45-9608-6f491550b24c.4644.8 from cache 2024-03-09T14:48:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.845+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.3:8c209108-e1cd-4d45-9608-6f491550b24c.4644.8, ofs0, ns] 2024-03-09T14:48:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.845+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562404 cookie 94885891690560 notifier 5946 bl.length()=238 2024-03-09T14:48:23.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.853+0000 7ff050e07980 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.3/8c209108-e1cd-4d45-9608-6f491550b24c.4644.8 : miss 2024-03-09T14:48:23.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.853+0000 7ff050e07980 20 rados->read ofs=0 len=0 2024-03-09T14:48:23.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.853+0000 7ff050e07980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:23.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.853+0000 7ff050e07980 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.3/8c209108-e1cd-4d45-9608-6f491550b24c.4644.8 info.flags=0x0 2024-03-09T14:48:23.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.853+0000 7ff050e07980 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.3/8c209108-e1cd-4d45-9608-6f491550b24c.4644.8 to cache LRU end 2024-03-09T14:48:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.861+0000 7ff050e07980 10 removing key index: NAAISEVWFNNKSHKYHQIO 2024-03-09T14:48:23.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.861+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:NAAISEVWFNNKSHKYHQIO, ofs0, ns] 2024-03-09T14:48:23.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.865+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562424 cookie 94885891680208 notifier 5946 bl.length()=170 2024-03-09T14:48:23.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.869+0000 7ff050e07980 10 removing email index: ragweed.client.0+test@test.test 2024-03-09T14:48:23.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.869+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.email:ragweed.client.0+test@test.test, ofs0, ns] 2024-03-09T14:48:23.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.873+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562437 cookie 94885891684208 notifier 5946 bl.length()=182 2024-03-09T14:48:23.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.873+0000 7ff050e07980 10 removing user buckets index 2024-03-09T14:48:23.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.877+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:ragweed.client.0.buckets, ofs0, ns] 2024-03-09T14:48:23.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.877+0000 7fef327fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562381 cookie 94885891693184 notifier 5946 bl.length()=173 2024-03-09T14:48:23.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.881+0000 7ff050e07980 10 removing user index: ragweed.client.0 2024-03-09T14:48:23.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.885+0000 7ff050e07980 10 removing default.rgw.meta+users.uid+ragweed.client.0 from cache 2024-03-09T14:48:23.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.885+0000 7ff050e07980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.uid:ragweed.client.0, ofs0, ns] 2024-03-09T14:48:23.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.885+0000 7fef31ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562405 cookie 94885891690560 notifier 5946 bl.length()=165 2024-03-09T14:48:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.897+0000 7fef05fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:48:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.901+0000 7ff050e07980 20 remove_watcher() i=0 2024-03-09T14:48:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.901+0000 7ff050e07980 2 removed watcher, disabling cache 2024-03-09T14:48:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.905+0000 7ff050e07980 20 remove_watcher() i=1 2024-03-09T14:48:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.909+0000 7ff050e07980 20 remove_watcher() i=2 2024-03-09T14:48:23.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.909+0000 7ff050e07980 20 remove_watcher() i=3 2024-03-09T14:48:23.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.913+0000 7ff050e07980 20 remove_watcher() i=4 2024-03-09T14:48:23.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.913+0000 7ff050e07980 20 remove_watcher() i=5 2024-03-09T14:48:23.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.917+0000 7ff050e07980 20 remove_watcher() i=6 2024-03-09T14:48:23.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:23.917+0000 7ff050e07980 20 remove_watcher() i=7 2024-03-09T14:48:23.939 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid sysuser.client.0 --purge-data 2024-03-09T14:48:24.022 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:48:24.023 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:48:24.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.065+0000 7fe374f81980 20 rados->read ofs=0 len=0 2024-03-09T14:48:24.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.065+0000 7fe374f81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:24.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.065+0000 7fe374f81980 20 realm 2024-03-09T14:48:24.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.065+0000 7fe374f81980 20 rados->read ofs=0 len=0 2024-03-09T14:48:24.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.065+0000 7fe374f81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:24.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.065+0000 7fe374f81980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:48:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.065+0000 7fe374f81980 20 rados->read ofs=0 len=0 2024-03-09T14:48:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.065+0000 7fe374f81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.065+0000 7fe374f81980 20 rados->read ofs=0 len=0 2024-03-09T14:48:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.069+0000 7fe374f81980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:48:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.069+0000 7fe374f81980 20 rados->read ofs=0 len=0 2024-03-09T14:48:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.069+0000 7fe374f81980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:48:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.069+0000 7fe374f81980 20 searching for the correct realm 2024-03-09T14:48:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.077+0000 7fe372298640 0 -- 172.21.15.1:0/4223984292 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x557be78ed250 msgr2=0x557be78ef710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:48:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.081+0000 7fe372a99640 0 -- 172.21.15.1:0/4223984292 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fe3681059a0 msgr2=0x7fe3680efcc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:48:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.089+0000 7fe374f81980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:48:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.089+0000 7fe374f81980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:48:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.089+0000 7fe374f81980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:48:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.089+0000 7fe374f81980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:48:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.089+0000 7fe374f81980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:48:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.089+0000 7fe374f81980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:48:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.089+0000 7fe374f81980 20 rados->read ofs=0 len=0 2024-03-09T14:48:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.089+0000 7fe374f81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.089+0000 7fe374f81980 20 rados->read ofs=0 len=0 2024-03-09T14:48:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:48:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 rados->read ofs=0 len=0 2024-03-09T14:48:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:48:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 zone default found 2024-03-09T14:48:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 4 Realm: () 2024-03-09T14:48:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:48:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:48:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:48:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 zonegroup default 2024-03-09T14:48:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 rados->read ofs=0 len=0 2024-03-09T14:48:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 rados->read ofs=0 len=0 2024-03-09T14:48:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 rados->read ofs=0 len=0 2024-03-09T14:48:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:48:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 started sync module instance, tier type = 2024-03-09T14:48:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.093+0000 7fe374f81980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:48:24.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.109+0000 7fe374f81980 20 add_watcher() i=0 2024-03-09T14:48:24.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.109+0000 7fe374f81980 20 add_watcher() i=1 2024-03-09T14:48:24.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.109+0000 7fe374f81980 20 add_watcher() i=2 2024-03-09T14:48:24.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.109+0000 7fe374f81980 20 add_watcher() i=3 2024-03-09T14:48:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.109+0000 7fe374f81980 20 add_watcher() i=4 2024-03-09T14:48:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.109+0000 7fe374f81980 20 add_watcher() i=5 2024-03-09T14:48:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.109+0000 7fe374f81980 20 add_watcher() i=6 2024-03-09T14:48:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.113+0000 7fe374f81980 20 add_watcher() i=7 2024-03-09T14:48:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.113+0000 7fe374f81980 2 all 8 watchers are set, enabling cache 2024-03-09T14:48:24.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.113+0000 7fe2297e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:48:24.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.113+0000 7fe374f81980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:48:24.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.113+0000 7fe374f81980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:48:24.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.197+0000 7fe1a3fff640 20 reqs_thread_entry: start 2024-03-09T14:48:24.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.197+0000 7fe374f81980 20 init_complete bucket index max shards: 1 2024-03-09T14:48:24.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.197+0000 7fe374f81980 20 Filter name: none 2024-03-09T14:48:24.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.197+0000 7fe1a1ffb640 20 reqs_thread_entry: start 2024-03-09T14:48:24.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.197+0000 7fe374f81980 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : miss 2024-03-09T14:48:24.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.197+0000 7fe374f81980 20 rados->read ofs=0 len=0 2024-03-09T14:48:24.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.197+0000 7fe374f81980 20 rados_obj.operate() r=0 bl.length=486 2024-03-09T14:48:24.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.197+0000 7fe374f81980 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x13 2024-03-09T14:48:24.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.197+0000 7fe374f81980 10 adding default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2024-03-09T14:48:24.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.197+0000 7fe374f81980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:48:24.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.197+0000 7fe374f81980 10 removing key index: QGUAXOPOXEVLUOTENWHO 2024-03-09T14:48:24.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.201+0000 7fe374f81980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.keys:QGUAXOPOXEVLUOTENWHO, ofs0, ns] 2024-03-09T14:48:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.201+0000 7fe2567fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562406 cookie 93990654240016 notifier 5621 bl.length()=170 2024-03-09T14:48:24.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.205+0000 7fe374f81980 10 removing email index: sysuser.client.0+test@test.test 2024-03-09T14:48:24.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.209+0000 7fe374f81980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.email:sysuser.client.0+test@test.test, ofs0, ns] 2024-03-09T14:48:24.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.209+0000 7fe2567fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562407 cookie 93990654255536 notifier 5621 bl.length()=182 2024-03-09T14:48:24.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.213+0000 7fe374f81980 10 removing user buckets index 2024-03-09T14:48:24.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.213+0000 7fe374f81980 10 removing user index: sysuser.client.0 2024-03-09T14:48:24.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.217+0000 7fe374f81980 10 removing default.rgw.meta+users.uid+sysuser.client.0 from cache 2024-03-09T14:48:24.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.217+0000 7fe374f81980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:sysuser.client.0, ofs0, ns] 2024-03-09T14:48:24.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.217+0000 7fe255ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562438 cookie 93990654251872 notifier 5621 bl.length()=165 2024-03-09T14:48:24.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.221+0000 7fe2297e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:48:24.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.225+0000 7fe374f81980 20 remove_watcher() i=0 2024-03-09T14:48:24.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.225+0000 7fe374f81980 2 removed watcher, disabling cache 2024-03-09T14:48:24.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.229+0000 7fe374f81980 20 remove_watcher() i=1 2024-03-09T14:48:24.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.229+0000 7fe374f81980 20 remove_watcher() i=2 2024-03-09T14:48:24.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.233+0000 7fe374f81980 20 remove_watcher() i=3 2024-03-09T14:48:24.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.237+0000 7fe374f81980 20 remove_watcher() i=4 2024-03-09T14:48:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.237+0000 7fe374f81980 20 remove_watcher() i=5 2024-03-09T14:48:24.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.237+0000 7fe374f81980 20 remove_watcher() i=6 2024-03-09T14:48:24.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:48:24.241+0000 7fe374f81980 20 remove_watcher() i=7 2024-03-09T14:48:24.258 INFO:tasks.ragweed:Removing ragweed... 2024-03-09T14:48:24.259 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/ragweed.client.0 2024-03-09T14:48:24.353 INFO:teuthology.run_tasks:Running task workunit... 2024-03-09T14:48:24.364 INFO:tasks.workunit:Pulling workunits from ref c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:48:24.364 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-09T14:48:24.365 DEBUG:teuthology.orchestra.run.smithi001:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-09T14:48:24.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:48:24.373 INFO:teuthology.orchestra.run.smithi001.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-09T14:48:24.373 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-09T14:48:24.421 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-09T14:48:24.422 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-09T14:48:24.469 INFO:tasks.workunit:timeout=3h 2024-03-09T14:48:24.469 INFO:tasks.workunit:cleanup=True 2024-03-09T14:48:24.469 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c00d9c609fb42a2a25f1c196bed064ca8bb02413 2024-03-09T14:48:24.519 INFO:tasks.workunit.client.0.smithi001.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-09T14:49:39.894 INFO:tasks.workunit.client.0.smithi001.stderr:Updating files: 96% (11801/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-09T14:49:40.644 INFO:tasks.workunit.client.0.smithi001.stderr:Note: switching to 'c00d9c609fb42a2a25f1c196bed064ca8bb02413'. 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr:state without impacting any branches by switching back to a branch. 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr: git switch -c 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr:Or undo this operation with: 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr: git switch - 2024-03-09T14:49:40.645 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:49:40.646 INFO:tasks.workunit.client.0.smithi001.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-09T14:49:40.646 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:49:40.646 INFO:tasks.workunit.client.0.smithi001.stderr:HEAD is now at c00d9c609fb DNM: point s3tests at branch cbodley:wip-iam-user-apis 2024-03-09T14:49:40.650 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-09T14:49:40.698 INFO:tasks.workunit.client.0.smithi001.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-09T14:49:40.700 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-09T14:49:40.700 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-09T14:49:40.756 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-09T14:49:40.814 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-09T14:49:40.860 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-09T14:49:40.861 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-09T14:49:40.861 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-09T14:49:40.911 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-09T14:49:40.915 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:49:40.915 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-09T14:49:40.960 INFO:tasks.workunit:Running workunits matching rgw/run-reshard.sh on client.0... 2024-03-09T14:49:40.962 INFO:tasks.workunit:Running workunit rgw/run-reshard.sh... 2024-03-09T14:49:40.962 DEBUG:teuthology.orchestra.run.smithi001:workunit test rgw/run-reshard.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c00d9c609fb42a2a25f1c196bed064ca8bb02413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh 2024-03-09T14:49:41.013 INFO:tasks.workunit.client.0.smithi001.stderr:+ ulimit -c 0 2024-03-09T14:49:41.014 INFO:tasks.workunit.client.0.smithi001.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh 2024-03-09T14:49:41.014 INFO:tasks.workunit.client.0.smithi001.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2024-03-09T14:49:41.014 INFO:tasks.workunit.client.0.smithi001.stderr:+ python3 -m venv /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2024-03-09T14:49:44.015 INFO:tasks.workunit.client.0.smithi001.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/activate 2024-03-09T14:49:44.015 INFO:tasks.workunit.client.0.smithi001.stderr:++ deactivate nondestructive 2024-03-09T14:49:44.015 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -n '' ']' 2024-03-09T14:49:44.015 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -n '' ']' 2024-03-09T14:49:44.015 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2024-03-09T14:49:44.015 INFO:tasks.workunit.client.0.smithi001.stderr:++ hash -r 2024-03-09T14:49:44.015 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -n '' ']' 2024-03-09T14:49:44.015 INFO:tasks.workunit.client.0.smithi001.stderr:++ unset VIRTUAL_ENV 2024-03-09T14:49:44.015 INFO:tasks.workunit.client.0.smithi001.stderr:++ unset VIRTUAL_ENV_PROMPT 2024-03-09T14:49:44.015 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' '!' nondestructive = nondestructive ']' 2024-03-09T14:49:44.015 INFO:tasks.workunit.client.0.smithi001.stderr:++ VIRTUAL_ENV=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ export VIRTUAL_ENV 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ _OLD_VIRTUAL_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ PATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ export PATH 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -n '' ']' 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -z '' ']' 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ _OLD_VIRTUAL_PS1= 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ PS1='(rgw) ' 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ export PS1 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ VIRTUAL_ENV_PROMPT='(rgw) ' 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ export VIRTUAL_ENV_PROMPT 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:++ hash -r 2024-03-09T14:49:44.016 INFO:tasks.workunit.client.0.smithi001.stderr:+ pip install pip --upgrade 2024-03-09T14:49:44.403 INFO:tasks.workunit.client.0.smithi001.stdout:Requirement already satisfied: pip in /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages (22.0.2) 2024-03-09T14:49:44.630 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting pip 2024-03-09T14:49:44.637 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached pip-24.0-py3-none-any.whl (2.1 MB) 2024-03-09T14:49:44.675 INFO:tasks.workunit.client.0.smithi001.stdout:Installing collected packages: pip 2024-03-09T14:49:44.676 INFO:tasks.workunit.client.0.smithi001.stdout: Attempting uninstall: pip 2024-03-09T14:49:44.676 INFO:tasks.workunit.client.0.smithi001.stdout: Found existing installation: pip 22.0.2 2024-03-09T14:49:44.949 INFO:tasks.workunit.client.0.smithi001.stdout: Uninstalling pip-22.0.2: 2024-03-09T14:49:44.955 INFO:tasks.workunit.client.0.smithi001.stdout: Successfully uninstalled pip-22.0.2 2024-03-09T14:49:45.921 INFO:tasks.workunit.client.0.smithi001.stdout:Successfully installed pip-24.0 2024-03-09T14:49:45.984 INFO:tasks.workunit.client.0.smithi001.stderr:+ pip install boto3 2024-03-09T14:49:46.973 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting boto3 2024-03-09T14:49:46.974 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached boto3-1.34.59-py3-none-any.whl.metadata (6.6 kB) 2024-03-09T14:49:47.501 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting botocore<1.35.0,>=1.34.59 (from boto3) 2024-03-09T14:49:47.502 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached botocore-1.34.59-py3-none-any.whl.metadata (5.7 kB) 2024-03-09T14:49:47.560 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting jmespath<2.0.0,>=0.7.1 (from boto3) 2024-03-09T14:49:47.561 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB) 2024-03-09T14:49:47.627 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting s3transfer<0.11.0,>=0.10.0 (from boto3) 2024-03-09T14:49:47.628 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached s3transfer-0.10.0-py3-none-any.whl.metadata (1.7 kB) 2024-03-09T14:49:47.706 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.35.0,>=1.34.59->boto3) 2024-03-09T14:49:47.708 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB) 2024-03-09T14:49:47.831 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting urllib3<2.1,>=1.25.4 (from botocore<1.35.0,>=1.34.59->boto3) 2024-03-09T14:49:47.833 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached urllib3-2.0.7-py3-none-any.whl.metadata (6.6 kB) 2024-03-09T14:49:48.006 INFO:tasks.workunit.client.0.smithi001.stdout:Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.35.0,>=1.34.59->boto3) 2024-03-09T14:49:48.007 INFO:tasks.workunit.client.0.smithi001.stdout: Using cached six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB) 2024-03-09T14:49:48.025 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached boto3-1.34.59-py3-none-any.whl (139 kB) 2024-03-09T14:49:48.025 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached botocore-1.34.59-py3-none-any.whl (12.0 MB) 2024-03-09T14:49:48.053 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached jmespath-1.0.1-py3-none-any.whl (20 kB) 2024-03-09T14:49:48.054 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached s3transfer-0.10.0-py3-none-any.whl (82 kB) 2024-03-09T14:49:48.056 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) 2024-03-09T14:49:48.057 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached urllib3-2.0.7-py3-none-any.whl (124 kB) 2024-03-09T14:49:48.059 INFO:tasks.workunit.client.0.smithi001.stdout:Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) 2024-03-09T14:49:48.162 INFO:tasks.workunit.client.0.smithi001.stdout:Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3 2024-03-09T14:49:49.280 INFO:tasks.workunit.client.0.smithi001.stdout:Successfully installed boto3-1.34.59 botocore-1.34.59 jmespath-1.0.1 python-dateutil-2.9.0.post0 s3transfer-0.10.0 six-1.16.0 urllib3-2.0.7 2024-03-09T14:49:49.454 INFO:tasks.workunit.client.0.smithi001.stderr:+ /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_reshard.py 2024-03-09T14:49:49.599 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin user create --uid tester --display-name "Testing" --access-key NX5QOQKC6BH2IDN8HC7A --secret LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn 2024-03-09T14:49:49.639 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:49:49.640 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:49:49.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.680+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.680+0000 7fa4573f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:49:49.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.680+0000 7fa4573f7980 20 realm 2024-03-09T14:49:49.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.680+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.684+0000 7fa4573f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:49:49.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.684+0000 7fa4573f7980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:49:49.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.684+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.684+0000 7fa4573f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:49:49.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.684+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.684+0000 7fa4573f7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:49:49.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.684+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.684+0000 7fa4573f7980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:49:49.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.684+0000 7fa4573f7980 20 searching for the correct realm 2024-03-09T14:49:49.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.692+0000 7fa45470e640 0 -- 172.21.15.1:0/672940750 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fa444082070 msgr2=0x7fa4440547f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:49:49.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.700+0000 7fa45470e640 0 -- 172.21.15.1:0/672940750 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fa320075e60 msgr2=0x7fa320078290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:49:49.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:49:49.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:49:49.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:49:49.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:49:49.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:49:49.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:49:49.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:49:49.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:49:49.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:49:49.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 zone default found 2024-03-09T14:49:49.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 4 Realm: () 2024-03-09T14:49:49.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:49:49.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:49:49.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:49:49.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 zonegroup default 2024-03-09T14:49:49.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:49:49.711 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:49:49.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:49:49.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 started sync module instance, tier type = 2024-03-09T14:49:49.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.708+0000 7fa4573f7980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:49:49.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.724+0000 7fa4573f7980 20 add_watcher() i=0 2024-03-09T14:49:49.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.724+0000 7fa4573f7980 20 add_watcher() i=1 2024-03-09T14:49:49.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.724+0000 7fa4573f7980 20 add_watcher() i=2 2024-03-09T14:49:49.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.724+0000 7fa4573f7980 20 add_watcher() i=3 2024-03-09T14:49:49.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.724+0000 7fa4573f7980 20 add_watcher() i=4 2024-03-09T14:49:49.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.724+0000 7fa4573f7980 20 add_watcher() i=5 2024-03-09T14:49:49.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.724+0000 7fa4573f7980 20 add_watcher() i=6 2024-03-09T14:49:49.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.728+0000 7fa4573f7980 20 add_watcher() i=7 2024-03-09T14:49:49.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.728+0000 7fa4573f7980 2 all 8 watchers are set, enabling cache 2024-03-09T14:49:49.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.728+0000 7fa30afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:49:49.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.728+0000 7fa4573f7980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:49:49.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.728+0000 7fa4573f7980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:49:49.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.728+0000 7fa4573f7980 5 note: GC not initialized 2024-03-09T14:49:49.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.728+0000 7fa309fe3640 20 reqs_thread_entry: start 2024-03-09T14:49:49.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.728+0000 7fa4573f7980 20 init_complete bucket index max shards: 1 2024-03-09T14:49:49.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 20 Filter name: none 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa27b7fe640 20 reqs_thread_entry: start 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 10 cache get: name=default.rgw.meta+users.uid+tester : miss 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 10 cache put: name=default.rgw.meta+users.uid+tester info.flags=0x0 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 10 adding default.rgw.meta+users.uid+tester to cache LRU end 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : miss 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 20 rados->read ofs=0 len=0 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 10 cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x0 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 10 adding default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : hit (negative entry) 2024-03-09T14:49:49.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : hit (negative entry) 2024-03-09T14:49:49.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 10 cache put: name=default.rgw.meta+users.uid+tester info.flags=0x17 2024-03-09T14:49:49.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 10 moving default.rgw.meta+users.uid+tester to cache LRU end 2024-03-09T14:49:49.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa4573f7980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.uid:tester, ofs0, ns] 2024-03-09T14:49:49.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.732+0000 7fa32bfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562382 cookie 94461464688064 notifier 5639 bl.length()=569 2024-03-09T14:49:49.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.736+0000 7fa32bfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+tester info.flags=0x17 2024-03-09T14:49:49.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.736+0000 7fa32bfff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+tester to cache LRU end 2024-03-09T14:49:49.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.736+0000 7fa4573f7980 10 cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2024-03-09T14:49:49.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.736+0000 7fa4573f7980 10 moving default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2024-03-09T14:49:49.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.736+0000 7fa4573f7980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:NX5QOQKC6BH2IDN8HC7A, ofs0, ns] 2024-03-09T14:49:49.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.736+0000 7fa32b7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562383 cookie 94461464685568 notifier 5639 bl.length()=180 2024-03-09T14:49:49.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.736+0000 7fa32b7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2024-03-09T14:49:49.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.736+0000 7fa32b7fe640 10 rgw watcher librados: moving default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2024-03-09T14:49:49.742 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.740+0000 7fa30afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:49:49.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.744+0000 7fa4573f7980 20 remove_watcher() i=0 2024-03-09T14:49:49.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.744+0000 7fa4573f7980 2 removed watcher, disabling cache 2024-03-09T14:49:49.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.748+0000 7fa4573f7980 20 remove_watcher() i=1 2024-03-09T14:49:49.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.752+0000 7fa4573f7980 20 remove_watcher() i=2 2024-03-09T14:49:49.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.752+0000 7fa4573f7980 20 remove_watcher() i=3 2024-03-09T14:49:49.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.756+0000 7fa4573f7980 20 remove_watcher() i=4 2024-03-09T14:49:49.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.756+0000 7fa4573f7980 20 remove_watcher() i=5 2024-03-09T14:49:49.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.756+0000 7fa4573f7980 20 remove_watcher() i=6 2024-03-09T14:49:49.760 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:49:49.756+0000 7fa4573f7980 20 remove_watcher() i=7 2024-03-09T14:50:07.923 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:07.923 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:08.000 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:08.000 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:08.113 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:08.113 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:08.230 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:08.230 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:08.309 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:08.310 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:08.408 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:08.408 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:08.494 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard add 2024-03-09T14:50:08.494 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:08.494 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket a-bucket 2024-03-09T14:50:08.529 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:08.529 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:08.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.571+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.575+0000 7f26fa897980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.575+0000 7f26fa897980 20 realm 2024-03-09T14:50:08.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.575+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.575+0000 7f26fa897980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.575+0000 7f26fa897980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:08.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.575+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.575+0000 7f26fa897980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.575+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.579+0000 7f26fa897980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:08.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.579+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.581 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.579+0000 7f26fa897980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:08.581 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.579+0000 7f26fa897980 20 searching for the correct realm 2024-03-09T14:50:08.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.583+0000 7f26f8bb0640 0 -- 172.21.15.1:0/2261247820 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x5580c9cbbea0 msgr2=0x5580c9cbe330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:08.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.591+0000 7f26f7bae640 0 -- 172.21.15.1:0/2261247820 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f26e8004050 msgr2=0x7f26e807e3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:08.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:08.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:08.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:08.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:08.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:08.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:08.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:08.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:08.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 zone default found 2024-03-09T14:50:08.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 4 Realm: () 2024-03-09T14:50:08.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:08.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:08.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:08.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 zonegroup default 2024-03-09T14:50:08.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.599+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.603+0000 7f26fa897980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.603+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.603+0000 7f26fa897980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.603+0000 7f26fa897980 20 started sync module instance, tier type = 2024-03-09T14:50:08.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.603+0000 7f26fa897980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:08.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.619+0000 7f26fa897980 20 add_watcher() i=0 2024-03-09T14:50:08.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.619+0000 7f26fa897980 20 add_watcher() i=1 2024-03-09T14:50:08.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.619+0000 7f26fa897980 20 add_watcher() i=2 2024-03-09T14:50:08.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.619+0000 7f26fa897980 20 add_watcher() i=3 2024-03-09T14:50:08.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.619+0000 7f26fa897980 20 add_watcher() i=4 2024-03-09T14:50:08.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.619+0000 7f26fa897980 20 add_watcher() i=5 2024-03-09T14:50:08.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.619+0000 7f26fa897980 20 add_watcher() i=6 2024-03-09T14:50:08.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.619+0000 7f26fa897980 20 add_watcher() i=7 2024-03-09T14:50:08.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.619+0000 7f26fa897980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:08.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.623+0000 7f25aefdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:08.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.623+0000 7f26fa897980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:08.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.623+0000 7f26fa897980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:08.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.623+0000 7f26fa897980 5 note: GC not initialized 2024-03-09T14:50:08.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.623+0000 7f25adfdb640 20 reqs_thread_entry: start 2024-03-09T14:50:08.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.623+0000 7f26fa897980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:08.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.623+0000 7f26fa897980 20 Filter name: none 2024-03-09T14:50:08.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.623+0000 7f251f7fe640 20 reqs_thread_entry: start 2024-03-09T14:50:08.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.623+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.623+0000 7f26fa897980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:50:08.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.623+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.627+0000 7f26fa897980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:50:08.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.627+0000 7f26fa897980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:08.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.627+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.627+0000 7f26fa897980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:50:08.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.627+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.627+0000 7f26fa897980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:50:08.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.627+0000 7f26fa897980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:08.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.631+0000 7f26fa897980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.631+0000 7f26fa897980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.631+0000 7f25aefdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:08.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.635+0000 7f26fa897980 20 remove_watcher() i=0 2024-03-09T14:50:08.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.635+0000 7f26fa897980 2 removed watcher, disabling cache 2024-03-09T14:50:08.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.639+0000 7f26fa897980 20 remove_watcher() i=1 2024-03-09T14:50:08.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.639+0000 7f26fa897980 20 remove_watcher() i=2 2024-03-09T14:50:08.644 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.643+0000 7f26fa897980 20 remove_watcher() i=3 2024-03-09T14:50:08.646 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.643+0000 7f26fa897980 20 remove_watcher() i=4 2024-03-09T14:50:08.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.643+0000 7f26fa897980 20 remove_watcher() i=5 2024-03-09T14:50:08.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.647+0000 7f26fa897980 20 remove_watcher() i=6 2024-03-09T14:50:08.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.647+0000 7f26fa897980 20 remove_watcher() i=7 2024-03-09T14:50:08.666 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:08.666 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:50:08.666 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: a-bucket id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 num_objects: 0 size_kb: 0 num_shards: 1 2024-03-09T14:50:08.666 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:08.666 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin reshard add --bucket a-bucket --num-shards 2 2024-03-09T14:50:08.694 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:08.694 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:08.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.735+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.735+0000 7ff2e7fbc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.735+0000 7ff2e7fbc980 20 realm 2024-03-09T14:50:08.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.735+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.735+0000 7ff2e7fbc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.735+0000 7ff2e7fbc980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:08.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.735+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.741 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.739+0000 7ff2e7fbc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.741 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.739+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.742 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.739+0000 7ff2e7fbc980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:08.742 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.739+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.743 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.739+0000 7ff2e7fbc980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:08.743 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.739+0000 7ff2e7fbc980 20 searching for the correct realm 2024-03-09T14:50:08.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.747+0000 7ff2e52d3640 0 -- 172.21.15.1:0/912691865 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x56383b66d1f0 msgr2=0x56383b66f6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:08.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.751+0000 7ff2e5ad4640 0 -- 172.21.15.1:0/912691865 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7ff2dc0473e0 msgr2=0x7ff2dc047860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:08.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.759+0000 7ff2e7fbc980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:08.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.759+0000 7ff2e7fbc980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:08.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.759+0000 7ff2e7fbc980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:08.767 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.759+0000 7ff2e7fbc980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:08.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.759+0000 7ff2e7fbc980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:08.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.759+0000 7ff2e7fbc980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:08.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.759+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 zone default found 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 4 Realm: () 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 zonegroup default 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.769 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 started sync module instance, tier type = 2024-03-09T14:50:08.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.763+0000 7ff2e7fbc980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:08.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.779+0000 7ff2e7fbc980 20 add_watcher() i=0 2024-03-09T14:50:08.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.779+0000 7ff2e7fbc980 20 add_watcher() i=1 2024-03-09T14:50:08.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.779+0000 7ff2e7fbc980 20 add_watcher() i=2 2024-03-09T14:50:08.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.779+0000 7ff2e7fbc980 20 add_watcher() i=3 2024-03-09T14:50:08.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.779+0000 7ff2e7fbc980 20 add_watcher() i=4 2024-03-09T14:50:08.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.779+0000 7ff2e7fbc980 20 add_watcher() i=5 2024-03-09T14:50:08.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.779+0000 7ff2e7fbc980 20 add_watcher() i=6 2024-03-09T14:50:08.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.779+0000 7ff2e7fbc980 20 add_watcher() i=7 2024-03-09T14:50:08.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.779+0000 7ff2e7fbc980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:08.786 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.783+0000 7ff19bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:08.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.783+0000 7ff2e7fbc980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:08.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.783+0000 7ff2e7fbc980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:08.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.783+0000 7ff2e7fbc980 5 note: GC not initialized 2024-03-09T14:50:08.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.783+0000 7ff19afe5640 20 reqs_thread_entry: start 2024-03-09T14:50:08.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.783+0000 7ff2e7fbc980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:08.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.783+0000 7ff2e7fbc980 20 Filter name: none 2024-03-09T14:50:08.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.783+0000 7ff198fe1640 20 reqs_thread_entry: start 2024-03-09T14:50:08.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.783+0000 7ff2e7fbc980 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2024-03-09T14:50:08.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.783+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.787+0000 7ff2e7fbc980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:50:08.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.787+0000 7ff2e7fbc980 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x11 2024-03-09T14:50:08.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.787+0000 7ff2e7fbc980 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2024-03-09T14:50:08.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.787+0000 7ff2e7fbc980 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 : miss 2024-03-09T14:50:08.789 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.787+0000 7ff2e7fbc980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.787+0000 7ff2e7fbc980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:50:08.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.787+0000 7ff2e7fbc980 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:50:08.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.787+0000 7ff2e7fbc980 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:50:08.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.787+0000 7ff2e7fbc980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:50:08.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.787+0000 7ff2e7fbc980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:50:08.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.787+0000 7ff2e7fbc980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 2024-03-09T14:50:08.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.791+0000 7ff19bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:08.798 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.795+0000 7ff2e7fbc980 20 remove_watcher() i=0 2024-03-09T14:50:08.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.795+0000 7ff2e7fbc980 2 removed watcher, disabling cache 2024-03-09T14:50:08.800 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.799+0000 7ff2e7fbc980 20 remove_watcher() i=1 2024-03-09T14:50:08.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.799+0000 7ff2e7fbc980 20 remove_watcher() i=2 2024-03-09T14:50:08.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.803+0000 7ff2e7fbc980 20 remove_watcher() i=3 2024-03-09T14:50:08.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.803+0000 7ff2e7fbc980 20 remove_watcher() i=4 2024-03-09T14:50:08.808 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.803+0000 7ff2e7fbc980 20 remove_watcher() i=5 2024-03-09T14:50:08.809 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.807+0000 7ff2e7fbc980 20 remove_watcher() i=6 2024-03-09T14:50:08.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.807+0000 7ff2e7fbc980 20 remove_watcher() i=7 2024-03-09T14:50:08.824 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin reshard list 2024-03-09T14:50:08.847 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:08.848 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:08.890 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.887+0000 7f243feeb980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.891+0000 7f243feeb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.891+0000 7f243feeb980 20 realm 2024-03-09T14:50:08.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.891+0000 7f243feeb980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.891+0000 7f243feeb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.891+0000 7f243feeb980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:08.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.891+0000 7f243feeb980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.891+0000 7f243feeb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:08.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.891+0000 7f243feeb980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.891+0000 7f243feeb980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:08.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.891+0000 7f243feeb980 20 rados->read ofs=0 len=0 2024-03-09T14:50:08.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.891+0000 7f243feeb980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:08.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.891+0000 7f243feeb980 20 searching for the correct realm 2024-03-09T14:50:08.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:08.899+0000 7f243e204640 0 -- 172.21.15.1:0/3299860457 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f2434051450 msgr2=0x7f24340518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:50:09.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.115+0000 7f243da03640 0 -- 172.21.15.1:0/3299860457 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x561e54a5d3d0 msgr2=0x561e54a5f860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:09.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.123+0000 7f243feeb980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:09.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.123+0000 7f243feeb980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:09.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.123+0000 7f243feeb980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:09.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.123+0000 7f243feeb980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:09.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.123+0000 7f243feeb980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:09.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.123+0000 7f243feeb980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:09.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.123+0000 7f243feeb980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:09.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:09.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:09.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 20 zone default found 2024-03-09T14:50:09.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 4 Realm: () 2024-03-09T14:50:09.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:09.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:09.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:09.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 20 zonegroup default 2024-03-09T14:50:09.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:09.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.127+0000 7f243feeb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:09.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.131+0000 7f243feeb980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.131+0000 7f243feeb980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:09.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.131+0000 7f243feeb980 20 started sync module instance, tier type = 2024-03-09T14:50:09.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.131+0000 7f243feeb980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:09.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.155+0000 7f243feeb980 20 add_watcher() i=0 2024-03-09T14:50:09.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.155+0000 7f243feeb980 20 add_watcher() i=1 2024-03-09T14:50:09.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.155+0000 7f243feeb980 20 add_watcher() i=2 2024-03-09T14:50:09.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.155+0000 7f243feeb980 20 add_watcher() i=3 2024-03-09T14:50:09.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.155+0000 7f243feeb980 20 add_watcher() i=4 2024-03-09T14:50:09.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.155+0000 7f243feeb980 20 add_watcher() i=5 2024-03-09T14:50:09.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.155+0000 7f243feeb980 20 add_watcher() i=6 2024-03-09T14:50:09.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.155+0000 7f243feeb980 20 add_watcher() i=7 2024-03-09T14:50:09.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.155+0000 7f243feeb980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:09.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.159+0000 7f22ef7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:09.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.163+0000 7f243feeb980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:09.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.163+0000 7f243feeb980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:09.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.163+0000 7f243feeb980 5 note: GC not initialized 2024-03-09T14:50:09.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.163+0000 7f22ee7fc640 20 reqs_thread_entry: start 2024-03-09T14:50:09.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.163+0000 7f243feeb980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:09.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.163+0000 7f243feeb980 20 Filter name: none 2024-03-09T14:50:09.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.163+0000 7f225ffff640 20 reqs_thread_entry: start 2024-03-09T14:50:09.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.179+0000 7f22ef7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:09.186 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.183+0000 7f243feeb980 20 remove_watcher() i=0 2024-03-09T14:50:09.186 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.183+0000 7f243feeb980 2 removed watcher, disabling cache 2024-03-09T14:50:09.189 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.187+0000 7f243feeb980 20 remove_watcher() i=1 2024-03-09T14:50:09.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.191+0000 7f243feeb980 20 remove_watcher() i=2 2024-03-09T14:50:09.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.191+0000 7f243feeb980 20 remove_watcher() i=3 2024-03-09T14:50:09.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.195+0000 7f243feeb980 20 remove_watcher() i=4 2024-03-09T14:50:09.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.195+0000 7f243feeb980 20 remove_watcher() i=5 2024-03-09T14:50:09.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.199+0000 7f243feeb980 20 remove_watcher() i=6 2024-03-09T14:50:09.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.199+0000 7f243feeb980 20 remove_watcher() i=7 2024-03-09T14:50:09.221 INFO:tasks.workunit.client.0.smithi001.stderr:bucket name a-bucket 2024-03-09T14:50:09.221 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard process 2024-03-09T14:50:09.221 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:09.221 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin reshard process 2024-03-09T14:50:09.246 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:09.246 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:09.291 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.287+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.291+0000 7f9daa6c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:09.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.291+0000 7f9daa6c8980 20 realm 2024-03-09T14:50:09.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.291+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.293 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.291+0000 7f9daa6c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:09.293 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.291+0000 7f9daa6c8980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:09.293 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.291+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.293 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.291+0000 7f9daa6c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:09.293 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.291+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.291+0000 7f9daa6c8980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:09.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.291+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.291+0000 7f9daa6c8980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:09.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.291+0000 7f9daa6c8980 20 searching for the correct realm 2024-03-09T14:50:09.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.299+0000 7f9da79df640 0 -- 172.21.15.1:0/3059498679 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f9d98008050 msgr2=0x7f9d9808da50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:09.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.307+0000 7f9da89e1640 0 -- 172.21.15.1:0/3059498679 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f9d9c008040 msgr2=0x7f9d9c07b470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:09.316 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:09.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:09.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:09.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:09.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:09.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:09.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:09.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:09.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:09.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 zone default found 2024-03-09T14:50:09.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 4 Realm: () 2024-03-09T14:50:09.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:09.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:09.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:09.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 zonegroup default 2024-03-09T14:50:09.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:09.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:09.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:09.321 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 started sync module instance, tier type = 2024-03-09T14:50:09.321 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.315+0000 7f9daa6c8980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:09.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.331+0000 7f9daa6c8980 20 add_watcher() i=0 2024-03-09T14:50:09.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.331+0000 7f9daa6c8980 20 add_watcher() i=1 2024-03-09T14:50:09.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.331+0000 7f9daa6c8980 20 add_watcher() i=2 2024-03-09T14:50:09.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.331+0000 7f9daa6c8980 20 add_watcher() i=3 2024-03-09T14:50:09.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.331+0000 7f9daa6c8980 20 add_watcher() i=4 2024-03-09T14:50:09.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.331+0000 7f9daa6c8980 20 add_watcher() i=5 2024-03-09T14:50:09.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.331+0000 7f9daa6c8980 20 add_watcher() i=6 2024-03-09T14:50:09.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.335+0000 7f9daa6c8980 20 add_watcher() i=7 2024-03-09T14:50:09.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.335+0000 7f9daa6c8980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:09.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.335+0000 7f9c5f7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:09.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.335+0000 7f9daa6c8980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:09.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.335+0000 7f9daa6c8980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:09.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.335+0000 7f9daa6c8980 5 note: GC not initialized 2024-03-09T14:50:09.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.335+0000 7f9c5e7dc640 20 reqs_thread_entry: start 2024-03-09T14:50:09.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.335+0000 7f9daa6c8980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:09.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.335+0000 7f9daa6c8980 20 Filter name: none 2024-03-09T14:50:09.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.335+0000 7f9c577fe640 20 reqs_thread_entry: start 2024-03-09T14:50:09.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.335+0000 7f9daa6c8980 20 processing logshard = reshard.0000000000 2024-03-09T14:50:09.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.343+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000000 , ret = 0 2024-03-09T14:50:09.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.343+0000 7f9daa6c8980 20 processing logshard = reshard.0000000001 2024-03-09T14:50:09.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.347+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000001 , ret = 0 2024-03-09T14:50:09.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.347+0000 7f9daa6c8980 20 processing logshard = reshard.0000000002 2024-03-09T14:50:09.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.351+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000002 , ret = 0 2024-03-09T14:50:09.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.351+0000 7f9daa6c8980 20 processing logshard = reshard.0000000003 2024-03-09T14:50:09.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.359+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000003 , ret = 0 2024-03-09T14:50:09.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.359+0000 7f9daa6c8980 20 processing logshard = reshard.0000000004 2024-03-09T14:50:09.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.363+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000004 , ret = 0 2024-03-09T14:50:09.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.363+0000 7f9daa6c8980 20 processing logshard = reshard.0000000005 2024-03-09T14:50:09.372 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.367+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000005 , ret = 0 2024-03-09T14:50:09.372 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.367+0000 7f9daa6c8980 20 processing logshard = reshard.0000000006 2024-03-09T14:50:09.375 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.371+0000 7f9daa6c8980 20 process_entry resharding a-bucket 2024-03-09T14:50:09.375 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.371+0000 7f9daa6c8980 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2024-03-09T14:50:09.375 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.371+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.376 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.371+0000 7f9daa6c8980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:50:09.376 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.371+0000 7f9daa6c8980 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x1 2024-03-09T14:50:09.376 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.371+0000 7f9daa6c8980 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2024-03-09T14:50:09.376 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.371+0000 7f9daa6c8980 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 : miss 2024-03-09T14:50:09.376 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.371+0000 7f9daa6c8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:09.377 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.375+0000 7f9daa6c8980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:50:09.377 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.375+0000 7f9daa6c8980 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:50:09.377 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.375+0000 7f9daa6c8980 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:50:09.377 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.375+0000 7f9daa6c8980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:50:09.377 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.375+0000 7f9daa6c8980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:50:09.377 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.375+0000 7f9daa6c8980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 2024-03-09T14:50:09.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.383+0000 7f9daa6c8980 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:50:09.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.383+0000 7f9daa6c8980 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:50:09.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.383+0000 7f9daa6c8980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:50:09.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.383+0000 7f9daa6c8980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17, ofs0, ns] 2024-03-09T14:50:09.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.383+0000 7f9c7f7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562387 cookie 94459168909584 notifier 5708 bl.length()=819 2024-03-09T14:50:09.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.383+0000 7f9c7f7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:50:09.387 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.383+0000 7f9c7f7fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:50:09.387 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.383+0000 7f9c7f7fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:50:09.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.387+0000 7f9daa6c8980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): orig_sources= new_sources= 2024-03-09T14:50:09.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.387+0000 7f9daa6c8980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): potential sources added=[] removed=[] 2024-03-09T14:50:09.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.387+0000 7f9daa6c8980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): orig_dests= new_dests= 2024-03-09T14:50:09.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.387+0000 7f9daa6c8980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): potential dests added=[] removed=[] 2024-03-09T14:50:09.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.387+0000 7f9daa6c8980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.1.0 2024-03-09T14:50:09.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.387+0000 7f9daa6c8980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.1.0 2024-03-09T14:50:09.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.387+0000 7f9daa6c8980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.1.1 2024-03-09T14:50:09.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.387+0000 7f9daa6c8980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.1.1 2024-03-09T14:50:09.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.387+0000 7f9daa6c8980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.0 2024-03-09T14:50:09.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.391+0000 7f9daa6c8980 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 : hit (requested=0x17, cached=0x17) 2024-03-09T14:50:09.392 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.391+0000 7f9daa6c8980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 2024-03-09T14:50:09.394 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.391+0000 7f9daa6c8980 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:50:09.394 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.391+0000 7f9daa6c8980 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:50:09.394 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.391+0000 7f9daa6c8980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:50:09.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.391+0000 7f9daa6c8980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17, ofs0, ns] 2024-03-09T14:50:09.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.391+0000 7f9c7effd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562388 cookie 94459168909584 notifier 5708 bl.length()=787 2024-03-09T14:50:09.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.391+0000 7f9c7effd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:50:09.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.391+0000 7f9c7effd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:50:09.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.391+0000 7f9c7effd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:50:09.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.395+0000 7f9daa6c8980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): orig_sources= new_sources= 2024-03-09T14:50:09.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.395+0000 7f9daa6c8980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): potential sources added=[] removed=[] 2024-03-09T14:50:09.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.395+0000 7f9daa6c8980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): orig_dests= new_dests= 2024-03-09T14:50:09.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.395+0000 7f9daa6c8980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): potential dests added=[] removed=[] 2024-03-09T14:50:09.400 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.395+0000 7f9daa6c8980 1 execute INFO: reshard of bucket "a-bucket" completed successfully 2024-03-09T14:50:09.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.399+0000 7f9daa6c8980 20 process_entry removing reshard queue entry for bucket a-bucket 2024-03-09T14:50:09.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.403+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000006 , ret = 0 2024-03-09T14:50:09.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.403+0000 7f9daa6c8980 20 processing logshard = reshard.0000000007 2024-03-09T14:50:09.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.407+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000007 , ret = 0 2024-03-09T14:50:09.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.407+0000 7f9daa6c8980 20 processing logshard = reshard.0000000008 2024-03-09T14:50:09.415 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.411+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000008 , ret = 0 2024-03-09T14:50:09.416 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.411+0000 7f9daa6c8980 20 processing logshard = reshard.0000000009 2024-03-09T14:50:09.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.419+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000009 , ret = 0 2024-03-09T14:50:09.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.419+0000 7f9daa6c8980 20 processing logshard = reshard.0000000010 2024-03-09T14:50:09.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.423+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000010 , ret = 0 2024-03-09T14:50:09.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.423+0000 7f9daa6c8980 20 processing logshard = reshard.0000000011 2024-03-09T14:50:09.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.427+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000011 , ret = 0 2024-03-09T14:50:09.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.427+0000 7f9daa6c8980 20 processing logshard = reshard.0000000012 2024-03-09T14:50:09.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.435+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000012 , ret = 0 2024-03-09T14:50:09.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.435+0000 7f9daa6c8980 20 processing logshard = reshard.0000000013 2024-03-09T14:50:09.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.439+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000013 , ret = 0 2024-03-09T14:50:09.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.439+0000 7f9daa6c8980 20 processing logshard = reshard.0000000014 2024-03-09T14:50:09.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.443+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000014 , ret = 0 2024-03-09T14:50:09.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.443+0000 7f9daa6c8980 20 processing logshard = reshard.0000000015 2024-03-09T14:50:09.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.447+0000 7f9daa6c8980 20 finish processing logshard = reshard.0000000015 , ret = 0 2024-03-09T14:50:09.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.451+0000 7f9c5f7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:09.458 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.455+0000 7f9daa6c8980 20 remove_watcher() i=0 2024-03-09T14:50:09.458 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.455+0000 7f9daa6c8980 2 removed watcher, disabling cache 2024-03-09T14:50:09.460 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.459+0000 7f9daa6c8980 20 remove_watcher() i=1 2024-03-09T14:50:09.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.459+0000 7f9daa6c8980 20 remove_watcher() i=2 2024-03-09T14:50:09.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.463+0000 7f9daa6c8980 20 remove_watcher() i=3 2024-03-09T14:50:09.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.467+0000 7f9daa6c8980 20 remove_watcher() i=4 2024-03-09T14:50:09.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.467+0000 7f9daa6c8980 20 remove_watcher() i=5 2024-03-09T14:50:09.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.467+0000 7f9daa6c8980 20 remove_watcher() i=6 2024-03-09T14:50:09.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:09.471+0000 7f9daa6c8980 20 remove_watcher() i=7 2024-03-09T14:50:14.497 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket a-bucket 2024-03-09T14:50:14.531 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:14.531 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:14.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.575+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.575+0000 7f95e029f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:14.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.575+0000 7f95e029f980 20 realm 2024-03-09T14:50:14.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.575+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.579+0000 7f95e029f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:14.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.579+0000 7f95e029f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:14.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.579+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.579+0000 7f95e029f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:14.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.579+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.579+0000 7f95e029f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:14.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.579+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.579+0000 7f95e029f980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:14.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.579+0000 7f95e029f980 20 searching for the correct realm 2024-03-09T14:50:14.590 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.587+0000 7f95de5b8640 0 -- 172.21.15.1:0/4235644398 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f95c8008970 msgr2=0x7f95c8054410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:50:14.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.803+0000 7f95dddb7640 0 -- 172.21.15.1:0/4235644398 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x55e769a72a30 msgr2=0x55e769a74e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:14.814 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.811+0000 7f95e029f980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:14.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.811+0000 7f95e029f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:14.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.811+0000 7f95e029f980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:14.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.811+0000 7f95e029f980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:14.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.811+0000 7f95e029f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:14.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.811+0000 7f95e029f980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:14.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.811+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.811+0000 7f95e029f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:14.815 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.811+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.811+0000 7f95e029f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:14.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.811+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:14.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 20 zone default found 2024-03-09T14:50:14.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 4 Realm: () 2024-03-09T14:50:14.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:14.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:14.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:14.818 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 20 zonegroup default 2024-03-09T14:50:14.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:14.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:14.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:14.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 20 started sync module instance, tier type = 2024-03-09T14:50:14.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.815+0000 7f95e029f980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:14.841 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.839+0000 7f95e029f980 20 add_watcher() i=0 2024-03-09T14:50:14.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.839+0000 7f95e029f980 20 add_watcher() i=1 2024-03-09T14:50:14.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.839+0000 7f95e029f980 20 add_watcher() i=2 2024-03-09T14:50:14.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.839+0000 7f95e029f980 20 add_watcher() i=3 2024-03-09T14:50:14.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.839+0000 7f95e029f980 20 add_watcher() i=4 2024-03-09T14:50:14.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.839+0000 7f95e029f980 20 add_watcher() i=5 2024-03-09T14:50:14.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.839+0000 7f95e029f980 20 add_watcher() i=6 2024-03-09T14:50:14.843 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.839+0000 7f95e029f980 20 add_watcher() i=7 2024-03-09T14:50:14.843 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.839+0000 7f95e029f980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:14.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.843+0000 7f9493fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:14.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.843+0000 7f95e029f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:14.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.843+0000 7f95e029f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:14.847 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.843+0000 7f95e029f980 5 note: GC not initialized 2024-03-09T14:50:14.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.843+0000 7f9492fe5640 20 reqs_thread_entry: start 2024-03-09T14:50:14.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.847+0000 7f95e029f980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:14.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.847+0000 7f95e029f980 20 Filter name: none 2024-03-09T14:50:14.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.847+0000 7f9490fe1640 20 reqs_thread_entry: start 2024-03-09T14:50:14.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.847+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.847+0000 7f95e029f980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:50:14.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.847+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.847+0000 7f95e029f980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:50:14.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.847+0000 7f95e029f980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:14.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.847+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.852 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.851+0000 7f95e029f980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:50:14.852 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.851+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.851+0000 7f95e029f980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:50:14.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.851+0000 7f95e029f980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:14.854 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.851+0000 7f95e029f980 20 rados->read ofs=0 len=0 2024-03-09T14:50:14.855 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.851+0000 7f95e029f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:14.856 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.855+0000 7f9493fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:14.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.859+0000 7f95e029f980 20 remove_watcher() i=0 2024-03-09T14:50:14.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.859+0000 7f95e029f980 2 removed watcher, disabling cache 2024-03-09T14:50:14.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.859+0000 7f95e029f980 20 remove_watcher() i=1 2024-03-09T14:50:14.866 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.863+0000 7f95e029f980 20 remove_watcher() i=2 2024-03-09T14:50:14.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.867+0000 7f95e029f980 20 remove_watcher() i=3 2024-03-09T14:50:14.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.867+0000 7f95e029f980 20 remove_watcher() i=4 2024-03-09T14:50:14.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.871+0000 7f95e029f980 20 remove_watcher() i=5 2024-03-09T14:50:14.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.871+0000 7f95e029f980 20 remove_watcher() i=6 2024-03-09T14:50:14.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:14.871+0000 7f95e029f980 20 remove_watcher() i=7 2024-03-09T14:50:14.892 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:14.893 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:50:14.893 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: a-bucket id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 num_objects: 0 size_kb: 0 num_shards: 2 2024-03-09T14:50:14.893 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:14.893 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard add non empty bucket 2024-03-09T14:50:14.893 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:14.897 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:14.898 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:14.961 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:14.961 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:15.028 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:15.029 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:15.123 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:15.123 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:15.220 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:15.221 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:15.315 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:15.316 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:15.416 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:15.416 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:15.515 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:15.515 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:15.602 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket a-bucket 2024-03-09T14:50:15.637 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:15.637 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:15.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.679+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.683+0000 7f799af0c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:15.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.683+0000 7f799af0c980 20 realm 2024-03-09T14:50:15.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.683+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.683+0000 7f799af0c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:15.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.683+0000 7f799af0c980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:15.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.683+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.683+0000 7f799af0c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:15.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.683+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.683+0000 7f799af0c980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:15.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.683+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.683+0000 7f799af0c980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:15.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.683+0000 7f799af0c980 20 searching for the correct realm 2024-03-09T14:50:15.693 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.691+0000 7f7998223640 0 -- 172.21.15.1:0/411751881 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x56523a1f7880 msgr2=0x56523a1f9d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:50:15.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.699+0000 7f7998a24640 0 -- 172.21.15.1:0/411751881 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f79800fa7f0 msgr2=0x7f79800f89c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:15.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:15.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:15.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:15.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:15.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:15.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:15.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:15.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:15.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:15.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 zone default found 2024-03-09T14:50:15.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 4 Realm: () 2024-03-09T14:50:15.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:15.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:15.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:15.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 zonegroup default 2024-03-09T14:50:15.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:15.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.919+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.923+0000 7f799af0c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:15.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.923+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.923+0000 7f799af0c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:15.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.923+0000 7f799af0c980 20 started sync module instance, tier type = 2024-03-09T14:50:15.927 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.923+0000 7f799af0c980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:15.951 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.947+0000 7f799af0c980 20 add_watcher() i=0 2024-03-09T14:50:15.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.947+0000 7f799af0c980 20 add_watcher() i=1 2024-03-09T14:50:15.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.947+0000 7f799af0c980 20 add_watcher() i=2 2024-03-09T14:50:15.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.947+0000 7f799af0c980 20 add_watcher() i=3 2024-03-09T14:50:15.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.947+0000 7f799af0c980 20 add_watcher() i=4 2024-03-09T14:50:15.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.947+0000 7f799af0c980 20 add_watcher() i=5 2024-03-09T14:50:15.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.947+0000 7f799af0c980 20 add_watcher() i=6 2024-03-09T14:50:15.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.951+0000 7f799af0c980 20 add_watcher() i=7 2024-03-09T14:50:15.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.951+0000 7f799af0c980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:15.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.955+0000 7f784f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:15.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.955+0000 7f799af0c980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:15.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.955+0000 7f799af0c980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:15.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.955+0000 7f799af0c980 5 note: GC not initialized 2024-03-09T14:50:15.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.955+0000 7f784e7e4640 20 reqs_thread_entry: start 2024-03-09T14:50:15.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.955+0000 7f799af0c980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:15.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.955+0000 7f799af0c980 20 Filter name: none 2024-03-09T14:50:15.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.955+0000 7f78477fe640 20 reqs_thread_entry: start 2024-03-09T14:50:15.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.955+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.959+0000 7f799af0c980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:50:15.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.959+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.959+0000 7f799af0c980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:50:15.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.959+0000 7f799af0c980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:15.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.959+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.959+0000 7f799af0c980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:50:15.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.959+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.963+0000 7f799af0c980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:50:15.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.963+0000 7f799af0c980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:15.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.963+0000 7f799af0c980 20 rados->read ofs=0 len=0 2024-03-09T14:50:15.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.963+0000 7f799af0c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:15.967 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.963+0000 7f784f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:15.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.971+0000 7f799af0c980 20 remove_watcher() i=0 2024-03-09T14:50:15.973 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.971+0000 7f799af0c980 2 removed watcher, disabling cache 2024-03-09T14:50:15.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.971+0000 7f799af0c980 20 remove_watcher() i=1 2024-03-09T14:50:15.977 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.975+0000 7f799af0c980 20 remove_watcher() i=2 2024-03-09T14:50:15.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.975+0000 7f799af0c980 20 remove_watcher() i=3 2024-03-09T14:50:15.982 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.979+0000 7f799af0c980 20 remove_watcher() i=4 2024-03-09T14:50:15.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.979+0000 7f799af0c980 20 remove_watcher() i=5 2024-03-09T14:50:15.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.983+0000 7f799af0c980 20 remove_watcher() i=6 2024-03-09T14:50:15.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:15.983+0000 7f799af0c980 20 remove_watcher() i=7 2024-03-09T14:50:16.005 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:16.005 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:50:16.005 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: a-bucket id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 num_objects: 8 size_kb: 1 num_shards: 2 2024-03-09T14:50:16.005 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:16.006 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin reshard add --bucket a-bucket --num-shards 3 2024-03-09T14:50:16.039 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:16.039 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:16.084 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.079+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.083+0000 7f588c55e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.083+0000 7f588c55e980 20 realm 2024-03-09T14:50:16.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.083+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.083+0000 7f588c55e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.083+0000 7f588c55e980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:16.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.083+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.086 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.083+0000 7f588c55e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.087 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.083+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.087+0000 7f588c55e980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:16.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.087+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.087+0000 7f588c55e980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:16.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.087+0000 7f588c55e980 20 searching for the correct realm 2024-03-09T14:50:16.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.091+0000 7f588a877640 0 -- 172.21.15.1:0/3176268013 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55bd0a4071c0 msgr2=0x55bd0a409620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:16.102 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.099+0000 7f5889875640 0 -- 172.21.15.1:0/3176268013 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f587c0f1070 msgr2=0x7f587c0d4970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:16.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 zone default found 2024-03-09T14:50:16.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 4 Realm: () 2024-03-09T14:50:16.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:16.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:16.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:16.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 zonegroup default 2024-03-09T14:50:16.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.107+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.111+0000 7f588c55e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.111+0000 7f588c55e980 20 started sync module instance, tier type = 2024-03-09T14:50:16.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.111+0000 7f588c55e980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:16.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.127+0000 7f588c55e980 20 add_watcher() i=0 2024-03-09T14:50:16.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.127+0000 7f588c55e980 20 add_watcher() i=1 2024-03-09T14:50:16.128 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.127+0000 7f588c55e980 20 add_watcher() i=2 2024-03-09T14:50:16.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.127+0000 7f588c55e980 20 add_watcher() i=3 2024-03-09T14:50:16.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.127+0000 7f588c55e980 20 add_watcher() i=4 2024-03-09T14:50:16.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.127+0000 7f588c55e980 20 add_watcher() i=5 2024-03-09T14:50:16.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.127+0000 7f588c55e980 20 add_watcher() i=6 2024-03-09T14:50:16.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.127+0000 7f588c55e980 20 add_watcher() i=7 2024-03-09T14:50:16.129 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.127+0000 7f588c55e980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:16.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.127+0000 7f5740fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:16.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:16.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:16.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 5 note: GC not initialized 2024-03-09T14:50:16.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f56bb7fe640 20 reqs_thread_entry: start 2024-03-09T14:50:16.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:16.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 20 Filter name: none 2024-03-09T14:50:16.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f56b97fa640 20 reqs_thread_entry: start 2024-03-09T14:50:16.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2024-03-09T14:50:16.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:50:16.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x11 2024-03-09T14:50:16.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2024-03-09T14:50:16.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 : miss 2024-03-09T14:50:16.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:50:16.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:50:16.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:50:16.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:50:16.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:50:16.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.131+0000 7f588c55e980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 2024-03-09T14:50:16.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.135+0000 7f5740fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:16.142 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.139+0000 7f588c55e980 20 remove_watcher() i=0 2024-03-09T14:50:16.142 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.139+0000 7f588c55e980 2 removed watcher, disabling cache 2024-03-09T14:50:16.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.143+0000 7f588c55e980 20 remove_watcher() i=1 2024-03-09T14:50:16.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.143+0000 7f588c55e980 20 remove_watcher() i=2 2024-03-09T14:50:16.149 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.147+0000 7f588c55e980 20 remove_watcher() i=3 2024-03-09T14:50:16.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.147+0000 7f588c55e980 20 remove_watcher() i=4 2024-03-09T14:50:16.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.147+0000 7f588c55e980 20 remove_watcher() i=5 2024-03-09T14:50:16.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.151+0000 7f588c55e980 20 remove_watcher() i=6 2024-03-09T14:50:16.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.151+0000 7f588c55e980 20 remove_watcher() i=7 2024-03-09T14:50:16.168 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin reshard list 2024-03-09T14:50:16.192 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:16.193 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:16.236 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.231+0000 7fb169ef2980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.235+0000 7fb169ef2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.235+0000 7fb169ef2980 20 realm 2024-03-09T14:50:16.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.235+0000 7fb169ef2980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.235+0000 7fb169ef2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.235+0000 7fb169ef2980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:16.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.235+0000 7fb169ef2980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.235+0000 7fb169ef2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.235+0000 7fb169ef2980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.235+0000 7fb169ef2980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:16.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.235+0000 7fb169ef2980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.239+0000 7fb169ef2980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:16.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.239+0000 7fb169ef2980 20 searching for the correct realm 2024-03-09T14:50:16.244 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.239+0000 7fb167a0a640 0 -- 172.21.15.1:0/1282574771 >> [v2:172.21.15.1:6824/1666056338,v1:172.21.15.1:6825/1666056338] conn(0x7fb150003be0 msgr2=0x7fb160057240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:16.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.247+0000 7fb16820b640 0 -- 172.21.15.1:0/1282574771 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7fb15c0d2b20 msgr2=0x7fb15c0d1460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:16.264 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 zone default found 2024-03-09T14:50:16.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 4 Realm: () 2024-03-09T14:50:16.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:16.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:16.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:16.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 zonegroup default 2024-03-09T14:50:16.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 started sync module instance, tier type = 2024-03-09T14:50:16.266 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.259+0000 7fb169ef2980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:16.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.275+0000 7fb169ef2980 20 add_watcher() i=0 2024-03-09T14:50:16.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.275+0000 7fb169ef2980 20 add_watcher() i=1 2024-03-09T14:50:16.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.275+0000 7fb169ef2980 20 add_watcher() i=2 2024-03-09T14:50:16.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.275+0000 7fb169ef2980 20 add_watcher() i=3 2024-03-09T14:50:16.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.275+0000 7fb169ef2980 20 add_watcher() i=4 2024-03-09T14:50:16.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.275+0000 7fb169ef2980 20 add_watcher() i=5 2024-03-09T14:50:16.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.275+0000 7fb169ef2980 20 add_watcher() i=6 2024-03-09T14:50:16.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.279+0000 7fb169ef2980 20 add_watcher() i=7 2024-03-09T14:50:16.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.279+0000 7fb169ef2980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:16.282 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.279+0000 7fb01dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:16.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.279+0000 7fb169ef2980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:16.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.279+0000 7fb169ef2980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:16.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.279+0000 7fb169ef2980 5 note: GC not initialized 2024-03-09T14:50:16.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.279+0000 7fb01cfe1640 20 reqs_thread_entry: start 2024-03-09T14:50:16.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.279+0000 7fb169ef2980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:16.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.279+0000 7fb169ef2980 20 Filter name: none 2024-03-09T14:50:16.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.279+0000 7faf967fc640 20 reqs_thread_entry: start 2024-03-09T14:50:16.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.291+0000 7fb01dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:16.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.299+0000 7fb169ef2980 20 remove_watcher() i=0 2024-03-09T14:50:16.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.299+0000 7fb169ef2980 2 removed watcher, disabling cache 2024-03-09T14:50:16.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.299+0000 7fb169ef2980 20 remove_watcher() i=1 2024-03-09T14:50:16.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.303+0000 7fb169ef2980 20 remove_watcher() i=2 2024-03-09T14:50:16.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.303+0000 7fb169ef2980 20 remove_watcher() i=3 2024-03-09T14:50:16.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.307+0000 7fb169ef2980 20 remove_watcher() i=4 2024-03-09T14:50:16.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.307+0000 7fb169ef2980 20 remove_watcher() i=5 2024-03-09T14:50:16.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.307+0000 7fb169ef2980 20 remove_watcher() i=6 2024-03-09T14:50:16.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.311+0000 7fb169ef2980 20 remove_watcher() i=7 2024-03-09T14:50:16.328 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard process non empty bucket 2024-03-09T14:50:16.328 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:16.328 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin reshard process 2024-03-09T14:50:16.351 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:16.351 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:16.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.391+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.395+0000 7f1d26239980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.395+0000 7f1d26239980 20 realm 2024-03-09T14:50:16.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.395+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.395+0000 7f1d26239980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.395+0000 7f1d26239980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:16.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.395+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.395+0000 7f1d26239980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.395+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.398 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.395+0000 7f1d26239980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:16.398 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.395+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.395+0000 7f1d26239980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:16.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.395+0000 7f1d26239980 20 searching for the correct realm 2024-03-09T14:50:16.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.403+0000 7f1d23550640 0 -- 172.21.15.1:0/3144411896 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f1d14074c20 msgr2=0x7f1d1405aa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:16.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.411+0000 7f1d24552640 0 -- 172.21.15.1:0/3144411896 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f1d180cd760 msgr2=0x7f1d180e8960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:16.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.415+0000 7f1d26239980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:16.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.415+0000 7f1d26239980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:16.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.415+0000 7f1d26239980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:16.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.415+0000 7f1d26239980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:16.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.415+0000 7f1d26239980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:16.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.415+0000 7f1d26239980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:16.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.415+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.415+0000 7f1d26239980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.415+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.415+0000 7f1d26239980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:16.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.415+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:16.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 20 zone default found 2024-03-09T14:50:16.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 4 Realm: () 2024-03-09T14:50:16.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:16.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:16.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:16.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 20 zonegroup default 2024-03-09T14:50:16.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 20 started sync module instance, tier type = 2024-03-09T14:50:16.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.419+0000 7f1d26239980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:16.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.435+0000 7f1d26239980 20 add_watcher() i=0 2024-03-09T14:50:16.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.435+0000 7f1d26239980 20 add_watcher() i=1 2024-03-09T14:50:16.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.435+0000 7f1d26239980 20 add_watcher() i=2 2024-03-09T14:50:16.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.435+0000 7f1d26239980 20 add_watcher() i=3 2024-03-09T14:50:16.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.435+0000 7f1d26239980 20 add_watcher() i=4 2024-03-09T14:50:16.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.435+0000 7f1d26239980 20 add_watcher() i=5 2024-03-09T14:50:16.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.435+0000 7f1d26239980 20 add_watcher() i=6 2024-03-09T14:50:16.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.435+0000 7f1d26239980 20 add_watcher() i=7 2024-03-09T14:50:16.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.435+0000 7f1d26239980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:16.441 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.439+0000 7f1bda7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:16.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.439+0000 7f1d26239980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:16.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.439+0000 7f1d26239980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:16.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.439+0000 7f1d26239980 5 note: GC not initialized 2024-03-09T14:50:16.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.439+0000 7f1bd97e2640 20 reqs_thread_entry: start 2024-03-09T14:50:16.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.439+0000 7f1d26239980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:16.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.439+0000 7f1d26239980 20 Filter name: none 2024-03-09T14:50:16.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.439+0000 7f1bd27fc640 20 reqs_thread_entry: start 2024-03-09T14:50:16.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.439+0000 7f1d26239980 20 processing logshard = reshard.0000000000 2024-03-09T14:50:16.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.447+0000 7f1d26239980 20 finish processing logshard = reshard.0000000000 , ret = 0 2024-03-09T14:50:16.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.447+0000 7f1d26239980 20 processing logshard = reshard.0000000001 2024-03-09T14:50:16.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.451+0000 7f1d26239980 20 finish processing logshard = reshard.0000000001 , ret = 0 2024-03-09T14:50:16.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.451+0000 7f1d26239980 20 processing logshard = reshard.0000000002 2024-03-09T14:50:16.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.455+0000 7f1d26239980 20 finish processing logshard = reshard.0000000002 , ret = 0 2024-03-09T14:50:16.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.455+0000 7f1d26239980 20 processing logshard = reshard.0000000003 2024-03-09T14:50:16.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.459+0000 7f1d26239980 20 finish processing logshard = reshard.0000000003 , ret = 0 2024-03-09T14:50:16.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.459+0000 7f1d26239980 20 processing logshard = reshard.0000000004 2024-03-09T14:50:16.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.467+0000 7f1d26239980 20 finish processing logshard = reshard.0000000004 , ret = 0 2024-03-09T14:50:16.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.467+0000 7f1d26239980 20 processing logshard = reshard.0000000005 2024-03-09T14:50:16.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.471+0000 7f1d26239980 20 finish processing logshard = reshard.0000000005 , ret = 0 2024-03-09T14:50:16.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.471+0000 7f1d26239980 20 processing logshard = reshard.0000000006 2024-03-09T14:50:16.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 20 process_entry resharding a-bucket 2024-03-09T14:50:16.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2024-03-09T14:50:16.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:50:16.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x1 2024-03-09T14:50:16.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2024-03-09T14:50:16.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 : miss 2024-03-09T14:50:16.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:50:16.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:50:16.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:50:16.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:50:16.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:50:16.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.475+0000 7f1d26239980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 2024-03-09T14:50:16.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.487+0000 7f1d26239980 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:50:16.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.487+0000 7f1d26239980 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:50:16.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.487+0000 7f1d26239980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:50:16.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.487+0000 7f1d26239980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17, ofs0, ns] 2024-03-09T14:50:16.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.487+0000 7f1c027fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562389 cookie 93947651983616 notifier 6159 bl.length()=819 2024-03-09T14:50:16.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.487+0000 7f1c027fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:50:16.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.487+0000 7f1c027fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:50:16.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.487+0000 7f1c027fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:50:16.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.487+0000 7f1d26239980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): orig_sources= new_sources= 2024-03-09T14:50:16.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.487+0000 7f1d26239980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): potential sources added=[] removed=[] 2024-03-09T14:50:16.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.487+0000 7f1d26239980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): orig_dests= new_dests= 2024-03-09T14:50:16.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.487+0000 7f1d26239980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): potential dests added=[] removed=[] 2024-03-09T14:50:16.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.491+0000 7f1d26239980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.0 2024-03-09T14:50:16.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.491+0000 7f1d26239980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.0 2024-03-09T14:50:16.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.491+0000 7f1d26239980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.1 2024-03-09T14:50:16.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.491+0000 7f1d26239980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.1 2024-03-09T14:50:16.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.491+0000 7f1d26239980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.2 2024-03-09T14:50:16.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.491+0000 7f1d26239980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.2 2024-03-09T14:50:16.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.491+0000 7f1d26239980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.1.0 2024-03-09T14:50:16.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.491+0000 7f1d26239980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.1.0 2024-03-09T14:50:16.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.491+0000 7f1d26239980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.1.1 2024-03-09T14:50:16.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.491+0000 7f1d26239980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.1.1 2024-03-09T14:50:16.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.495+0000 7f1d26239980 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 : hit (requested=0x17, cached=0x17) 2024-03-09T14:50:16.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.495+0000 7f1d26239980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 2024-03-09T14:50:16.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.499+0000 7f1d26239980 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:50:16.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.499+0000 7f1d26239980 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:50:16.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.499+0000 7f1d26239980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:50:16.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.499+0000 7f1d26239980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17, ofs0, ns] 2024-03-09T14:50:16.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.499+0000 7f1c027fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562390 cookie 93947651983616 notifier 6159 bl.length()=787 2024-03-09T14:50:16.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.499+0000 7f1c027fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:50:16.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.499+0000 7f1c027fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:50:16.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.499+0000 7f1c027fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:50:16.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.499+0000 7f1d26239980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): orig_sources= new_sources= 2024-03-09T14:50:16.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.499+0000 7f1d26239980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): potential sources added=[] removed=[] 2024-03-09T14:50:16.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.499+0000 7f1d26239980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): orig_dests= new_dests= 2024-03-09T14:50:16.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.499+0000 7f1d26239980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): potential dests added=[] removed=[] 2024-03-09T14:50:16.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.503+0000 7f1d26239980 1 execute INFO: reshard of bucket "a-bucket" completed successfully 2024-03-09T14:50:16.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.503+0000 7f1d26239980 20 process_entry removing reshard queue entry for bucket a-bucket 2024-03-09T14:50:16.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.507+0000 7f1d26239980 20 finish processing logshard = reshard.0000000006 , ret = 0 2024-03-09T14:50:16.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.507+0000 7f1d26239980 20 processing logshard = reshard.0000000007 2024-03-09T14:50:16.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.511+0000 7f1d26239980 20 finish processing logshard = reshard.0000000007 , ret = 0 2024-03-09T14:50:16.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.511+0000 7f1d26239980 20 processing logshard = reshard.0000000008 2024-03-09T14:50:16.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.519+0000 7f1d26239980 20 finish processing logshard = reshard.0000000008 , ret = 0 2024-03-09T14:50:16.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.519+0000 7f1d26239980 20 processing logshard = reshard.0000000009 2024-03-09T14:50:16.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.527+0000 7f1d26239980 20 finish processing logshard = reshard.0000000009 , ret = 0 2024-03-09T14:50:16.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.527+0000 7f1d26239980 20 processing logshard = reshard.0000000010 2024-03-09T14:50:16.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.531+0000 7f1d26239980 20 finish processing logshard = reshard.0000000010 , ret = 0 2024-03-09T14:50:16.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.531+0000 7f1d26239980 20 processing logshard = reshard.0000000011 2024-03-09T14:50:16.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.535+0000 7f1d26239980 20 finish processing logshard = reshard.0000000011 , ret = 0 2024-03-09T14:50:16.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.535+0000 7f1d26239980 20 processing logshard = reshard.0000000012 2024-03-09T14:50:16.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.543+0000 7f1d26239980 20 finish processing logshard = reshard.0000000012 , ret = 0 2024-03-09T14:50:16.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.543+0000 7f1d26239980 20 processing logshard = reshard.0000000013 2024-03-09T14:50:16.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.547+0000 7f1d26239980 20 finish processing logshard = reshard.0000000013 , ret = 0 2024-03-09T14:50:16.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.547+0000 7f1d26239980 20 processing logshard = reshard.0000000014 2024-03-09T14:50:16.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.551+0000 7f1d26239980 20 finish processing logshard = reshard.0000000014 , ret = 0 2024-03-09T14:50:16.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.551+0000 7f1d26239980 20 processing logshard = reshard.0000000015 2024-03-09T14:50:16.561 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.559+0000 7f1d26239980 20 finish processing logshard = reshard.0000000015 , ret = 0 2024-03-09T14:50:16.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.559+0000 7f1bda7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:16.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.563+0000 7f1d26239980 20 remove_watcher() i=0 2024-03-09T14:50:16.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.563+0000 7f1d26239980 2 removed watcher, disabling cache 2024-03-09T14:50:16.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.567+0000 7f1d26239980 20 remove_watcher() i=1 2024-03-09T14:50:16.572 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.571+0000 7f1d26239980 20 remove_watcher() i=2 2024-03-09T14:50:16.574 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.571+0000 7f1d26239980 20 remove_watcher() i=3 2024-03-09T14:50:16.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.575+0000 7f1d26239980 20 remove_watcher() i=4 2024-03-09T14:50:16.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.575+0000 7f1d26239980 20 remove_watcher() i=5 2024-03-09T14:50:16.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.579+0000 7f1d26239980 20 remove_watcher() i=6 2024-03-09T14:50:16.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.579+0000 7f1d26239980 20 remove_watcher() i=7 2024-03-09T14:50:16.599 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket a-bucket 2024-03-09T14:50:16.630 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:16.630 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:16.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.671+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.675+0000 7f628ff00980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.675+0000 7f628ff00980 20 realm 2024-03-09T14:50:16.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.675+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.675+0000 7f628ff00980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.675+0000 7f628ff00980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:16.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.675+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.675+0000 7f628ff00980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.675+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.675+0000 7f628ff00980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:16.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.675+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.675+0000 7f628ff00980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:16.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.675+0000 7f628ff00980 20 searching for the correct realm 2024-03-09T14:50:16.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.683+0000 7f628e219640 0 -- 172.21.15.1:0/3735630421 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f6284008040 msgr2=0x7f628403a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:16.900 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.899+0000 7f628e219640 0 -- 172.21.15.1:0/3735630421 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f6284008040 msgr2=0x7f628403a400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:16.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.907+0000 7f628ff00980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:16.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.907+0000 7f628ff00980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:16.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.907+0000 7f628ff00980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:16.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.907+0000 7f628ff00980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:16.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.907+0000 7f628ff00980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:16.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.907+0000 7f628ff00980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:16.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.907+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.907+0000 7f628ff00980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.907+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:16.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:16.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 20 zone default found 2024-03-09T14:50:16.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 4 Realm: () 2024-03-09T14:50:16.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:16.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:16.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:16.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 20 zonegroup default 2024-03-09T14:50:16.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.911+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.915+0000 7f628ff00980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.915+0000 7f628ff00980 20 started sync module instance, tier type = 2024-03-09T14:50:16.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.915+0000 7f628ff00980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:16.938 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.935+0000 7f628ff00980 20 add_watcher() i=0 2024-03-09T14:50:16.938 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.935+0000 7f628ff00980 20 add_watcher() i=1 2024-03-09T14:50:16.938 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.935+0000 7f628ff00980 20 add_watcher() i=2 2024-03-09T14:50:16.939 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.935+0000 7f628ff00980 20 add_watcher() i=3 2024-03-09T14:50:16.939 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.935+0000 7f628ff00980 20 add_watcher() i=4 2024-03-09T14:50:16.939 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.935+0000 7f628ff00980 20 add_watcher() i=5 2024-03-09T14:50:16.939 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.935+0000 7f628ff00980 20 add_watcher() i=6 2024-03-09T14:50:16.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.939+0000 7f628ff00980 20 add_watcher() i=7 2024-03-09T14:50:16.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.939+0000 7f628ff00980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:16.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.943+0000 7f613f7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:16.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.943+0000 7f628ff00980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:16.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.943+0000 7f628ff00980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:16.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.943+0000 7f628ff00980 5 note: GC not initialized 2024-03-09T14:50:16.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.943+0000 7f613e7fc640 20 reqs_thread_entry: start 2024-03-09T14:50:16.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.943+0000 7f628ff00980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:16.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.943+0000 7f628ff00980 20 Filter name: none 2024-03-09T14:50:16.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.943+0000 7f60affff640 20 reqs_thread_entry: start 2024-03-09T14:50:16.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.943+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.947+0000 7f628ff00980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:50:16.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.947+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.947+0000 7f628ff00980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:50:16.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.947+0000 7f628ff00980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:16.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.947+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.947+0000 7f628ff00980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:50:16.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.947+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.947+0000 7f628ff00980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:50:16.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.947+0000 7f628ff00980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:16.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.951+0000 7f628ff00980 20 rados->read ofs=0 len=0 2024-03-09T14:50:16.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.951+0000 7f628ff00980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:16.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.951+0000 7f613f7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:16.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.959+0000 7f628ff00980 20 remove_watcher() i=0 2024-03-09T14:50:16.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.959+0000 7f628ff00980 2 removed watcher, disabling cache 2024-03-09T14:50:16.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.959+0000 7f628ff00980 20 remove_watcher() i=1 2024-03-09T14:50:16.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.963+0000 7f628ff00980 20 remove_watcher() i=2 2024-03-09T14:50:16.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.967+0000 7f628ff00980 20 remove_watcher() i=3 2024-03-09T14:50:16.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.967+0000 7f628ff00980 20 remove_watcher() i=4 2024-03-09T14:50:16.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.971+0000 7f628ff00980 20 remove_watcher() i=5 2024-03-09T14:50:16.974 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.971+0000 7f628ff00980 20 remove_watcher() i=6 2024-03-09T14:50:16.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:16.971+0000 7f628ff00980 20 remove_watcher() i=7 2024-03-09T14:50:16.991 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:16.991 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:50:16.991 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: a-bucket id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 num_objects: 8 size_kb: 1 num_shards: 3 2024-03-09T14:50:16.991 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:16.991 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard bucket with EIO injected at set_target_layout 2024-03-09T14:50:16.992 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:16.994 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:16.994 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:17.066 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:17.067 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:17.132 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:17.132 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:17.232 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:17.232 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:17.333 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:17.333 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:17.409 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:17.409 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:17.514 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:17.514 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:17.615 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:17.615 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:17.707 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:17.708 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:17.803 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:17.803 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:17.902 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:17.902 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:18.002 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:18.002 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:18.103 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:18.103 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:18.204 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:18.204 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:18.302 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:18.302 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:18.407 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:18.407 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:18.503 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:18.504 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:18.568 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:18.568 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:18.667 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:18.667 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:18.767 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:18.767 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:18.865 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:18.865 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:18.962 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:18.962 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:19.055 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2024-03-09T14:50:19.088 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:19.088 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:19.135 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.131+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.135+0000 7f5ea35b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.135+0000 7f5ea35b8980 20 realm 2024-03-09T14:50:19.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.135+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.135+0000 7f5ea35b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.135+0000 7f5ea35b8980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:19.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.135+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.135+0000 7f5ea35b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.135+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.135+0000 7f5ea35b8980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:19.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.135+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.135+0000 7f5ea35b8980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:19.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.135+0000 7f5ea35b8980 20 searching for the correct realm 2024-03-09T14:50:19.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.143+0000 7f5ea18d1640 0 -- 172.21.15.1:0/180560296 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55559ebaa740 msgr2=0x55559ebacbb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:19.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.151+0000 7f5ea08cf640 0 -- 172.21.15.1:0/180560296 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f5e9402d070 msgr2=0x7f5e9403b900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:19.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:19.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:19.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:19.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:19.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:19.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:19.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:19.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:19.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 zone default found 2024-03-09T14:50:19.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 4 Realm: () 2024-03-09T14:50:19.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:19.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:19.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:19.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 zonegroup default 2024-03-09T14:50:19.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 started sync module instance, tier type = 2024-03-09T14:50:19.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.159+0000 7f5ea35b8980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:19.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.175+0000 7f5ea35b8980 20 add_watcher() i=0 2024-03-09T14:50:19.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.175+0000 7f5ea35b8980 20 add_watcher() i=1 2024-03-09T14:50:19.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.175+0000 7f5ea35b8980 20 add_watcher() i=2 2024-03-09T14:50:19.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.175+0000 7f5ea35b8980 20 add_watcher() i=3 2024-03-09T14:50:19.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.175+0000 7f5ea35b8980 20 add_watcher() i=4 2024-03-09T14:50:19.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.175+0000 7f5ea35b8980 20 add_watcher() i=5 2024-03-09T14:50:19.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.175+0000 7f5ea35b8980 20 add_watcher() i=6 2024-03-09T14:50:19.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.179+0000 7f5ea35b8980 20 add_watcher() i=7 2024-03-09T14:50:19.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.179+0000 7f5ea35b8980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:19.183 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.179+0000 7f5d57fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:19.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:19.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:19.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 5 note: GC not initialized 2024-03-09T14:50:19.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5d56fe5640 20 reqs_thread_entry: start 2024-03-09T14:50:19.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:19.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 20 Filter name: none 2024-03-09T14:50:19.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5d54fe1640 20 reqs_thread_entry: start 2024-03-09T14:50:19.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.186 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:19.186 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:19.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:19.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:19.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.183+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.187+0000 7f5ea35b8980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:19.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.187+0000 7f5ea35b8980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:19.189 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.187+0000 7f5ea35b8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.187+0000 7f5ea35b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.187+0000 7f5d57fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:19.196 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.195+0000 7f5ea35b8980 20 remove_watcher() i=0 2024-03-09T14:50:19.196 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.195+0000 7f5ea35b8980 2 removed watcher, disabling cache 2024-03-09T14:50:19.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.195+0000 7f5ea35b8980 20 remove_watcher() i=1 2024-03-09T14:50:19.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.199+0000 7f5ea35b8980 20 remove_watcher() i=2 2024-03-09T14:50:19.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.199+0000 7f5ea35b8980 20 remove_watcher() i=3 2024-03-09T14:50:19.205 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.203+0000 7f5ea35b8980 20 remove_watcher() i=4 2024-03-09T14:50:19.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.203+0000 7f5ea35b8980 20 remove_watcher() i=5 2024-03-09T14:50:19.207 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.203+0000 7f5ea35b8980 20 remove_watcher() i=6 2024-03-09T14:50:19.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.207+0000 7f5ea35b8980 20 remove_watcher() i=7 2024-03-09T14:50:19.225 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:19.225 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:50:19.225 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-set-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:50:19.225 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:19.225 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket error-at-set-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-error-at set_target_layout 2024-03-09T14:50:19.257 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:19.257 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:19.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.299+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.299+0000 7facbc5cd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.299+0000 7facbc5cd980 20 realm 2024-03-09T14:50:19.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.299+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.299+0000 7facbc5cd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.299+0000 7facbc5cd980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:19.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.299+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.299+0000 7facbc5cd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.299+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.299+0000 7facbc5cd980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:19.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.299+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.303+0000 7facbc5cd980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:19.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.303+0000 7facbc5cd980 20 searching for the correct realm 2024-03-09T14:50:19.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.307+0000 7facba8e6640 0 -- 172.21.15.1:0/3933289480 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55ee1ba952b0 msgr2=0x55ee1ba976c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:19.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.315+0000 7facba0e5640 0 -- 172.21.15.1:0/3933289480 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7faca406ab60 msgr2=0x7faca400f730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:19.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:19.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:19.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:19.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:19.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:19.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:19.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:19.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:19.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 zone default found 2024-03-09T14:50:19.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 4 Realm: () 2024-03-09T14:50:19.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:19.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:19.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:19.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 zonegroup default 2024-03-09T14:50:19.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.327 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 started sync module instance, tier type = 2024-03-09T14:50:19.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.323+0000 7facbc5cd980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:19.343 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.339+0000 7facbc5cd980 20 add_watcher() i=0 2024-03-09T14:50:19.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.339+0000 7facbc5cd980 20 add_watcher() i=1 2024-03-09T14:50:19.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.339+0000 7facbc5cd980 20 add_watcher() i=2 2024-03-09T14:50:19.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.339+0000 7facbc5cd980 20 add_watcher() i=3 2024-03-09T14:50:19.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.339+0000 7facbc5cd980 20 add_watcher() i=4 2024-03-09T14:50:19.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.339+0000 7facbc5cd980 20 add_watcher() i=5 2024-03-09T14:50:19.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.339+0000 7facbc5cd980 20 add_watcher() i=6 2024-03-09T14:50:19.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.343+0000 7facbc5cd980 20 add_watcher() i=7 2024-03-09T14:50:19.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.343+0000 7facbc5cd980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:19.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.343+0000 7fab70fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:19.348 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.343+0000 7facbc5cd980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:19.348 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.343+0000 7facbc5cd980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:19.348 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.343+0000 7facbc5cd980 5 note: GC not initialized 2024-03-09T14:50:19.348 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7faaeb7fe640 20 reqs_thread_entry: start 2024-03-09T14:50:19.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:19.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 20 Filter name: none 2024-03-09T14:50:19.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7faae97fa640 20 reqs_thread_entry: start 2024-03-09T14:50:19.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 10 cache get: name=default.rgw.meta+root+error-at-set-target-layout : miss 2024-03-09T14:50:19.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:19.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 10 cache put: name=default.rgw.meta+root+error-at-set-target-layout info.flags=0x11 2024-03-09T14:50:19.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 10 adding default.rgw.meta+root+error-at-set-target-layout to cache LRU end 2024-03-09T14:50:19.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 : miss 2024-03-09T14:50:19.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:19.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 info.flags=0x17 2024-03-09T14:50:19.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 10 adding default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 to cache LRU end 2024-03-09T14:50:19.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:50:19.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:19.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.347+0000 7facbc5cd980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 2024-03-09T14:50:19.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.351+0000 7facbc5cd980 -1 Injecting error=-5 at location=set_target_layout 2024-03-09T14:50:19.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.351+0000 7facbc5cd980 0 ERROR: init_target_layout failed to write target index layout to bucket info: (5) Input/output error 2024-03-09T14:50:19.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.359+0000 7fab70fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:19.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.363+0000 7facbc5cd980 20 remove_watcher() i=0 2024-03-09T14:50:19.366 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.363+0000 7facbc5cd980 2 removed watcher, disabling cache 2024-03-09T14:50:19.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.363+0000 7facbc5cd980 20 remove_watcher() i=1 2024-03-09T14:50:19.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.367+0000 7facbc5cd980 20 remove_watcher() i=2 2024-03-09T14:50:19.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.367+0000 7facbc5cd980 20 remove_watcher() i=3 2024-03-09T14:50:19.373 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.371+0000 7facbc5cd980 20 remove_watcher() i=4 2024-03-09T14:50:19.376 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.371+0000 7facbc5cd980 20 remove_watcher() i=5 2024-03-09T14:50:19.377 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.375+0000 7facbc5cd980 20 remove_watcher() i=6 2024-03-09T14:50:19.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.375+0000 7facbc5cd980 20 remove_watcher() i=7 2024-03-09T14:50:19.395 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2024-03-09T14:50:19.419 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:19.419 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:19.461 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.459+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.459+0000 7f1b1b502980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.459+0000 7f1b1b502980 20 realm 2024-03-09T14:50:19.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.459+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.459+0000 7f1b1b502980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.459+0000 7f1b1b502980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:19.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.459+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.459+0000 7f1b1b502980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.459+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.463+0000 7f1b1b502980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:19.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.463+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.463+0000 7f1b1b502980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:19.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.463+0000 7f1b1b502980 20 searching for the correct realm 2024-03-09T14:50:19.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.471+0000 7f1b18819640 0 -- 172.21.15.1:0/1495876754 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f1b0c03e9a0 msgr2=0x7f1b0c03d460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:19.480 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.475+0000 7f1b1981b640 0 -- 172.21.15.1:0/1495876754 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f1b10114480 msgr2=0x7f1b10110000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:50:19.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.483+0000 7f1b1b502980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:19.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.483+0000 7f1b1b502980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:19.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.483+0000 7f1b1b502980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:19.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.483+0000 7f1b1b502980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:19.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.483+0000 7f1b1b502980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:19.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.483+0000 7f1b1b502980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:19.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.483+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.483+0000 7f1b1b502980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.483+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:19.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:19.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 zone default found 2024-03-09T14:50:19.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 4 Realm: () 2024-03-09T14:50:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 zonegroup default 2024-03-09T14:50:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 started sync module instance, tier type = 2024-03-09T14:50:19.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.487+0000 7f1b1b502980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:19.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.503+0000 7f1b1b502980 20 add_watcher() i=0 2024-03-09T14:50:19.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.503+0000 7f1b1b502980 20 add_watcher() i=1 2024-03-09T14:50:19.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.503+0000 7f1b1b502980 20 add_watcher() i=2 2024-03-09T14:50:19.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.503+0000 7f1b1b502980 20 add_watcher() i=3 2024-03-09T14:50:19.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.503+0000 7f1b1b502980 20 add_watcher() i=4 2024-03-09T14:50:19.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.503+0000 7f1b1b502980 20 add_watcher() i=5 2024-03-09T14:50:19.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.503+0000 7f1b1b502980 20 add_watcher() i=6 2024-03-09T14:50:19.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.503+0000 7f1b1b502980 20 add_watcher() i=7 2024-03-09T14:50:19.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.503+0000 7f1b1b502980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:19.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.507+0000 7f19cf7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:19.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.507+0000 7f1b1b502980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:19.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.507+0000 7f1b1b502980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:19.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.507+0000 7f1b1b502980 5 note: GC not initialized 2024-03-09T14:50:19.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.507+0000 7f19ce7e4640 20 reqs_thread_entry: start 2024-03-09T14:50:19.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.507+0000 7f1b1b502980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:19.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.507+0000 7f1b1b502980 20 Filter name: none 2024-03-09T14:50:19.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.507+0000 7f193ffff640 20 reqs_thread_entry: start 2024-03-09T14:50:19.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.507+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.507+0000 7f1b1b502980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:19.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.507+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.511+0000 7f1b1b502980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:19.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.511+0000 7f1b1b502980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:19.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.511+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.511+0000 7f1b1b502980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:19.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.511+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.511+0000 7f1b1b502980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:19.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.511+0000 7f1b1b502980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:19.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.511+0000 7f1b1b502980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.511+0000 7f1b1b502980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.515+0000 7f19cf7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:19.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.519+0000 7f1b1b502980 20 remove_watcher() i=0 2024-03-09T14:50:19.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.519+0000 7f1b1b502980 2 removed watcher, disabling cache 2024-03-09T14:50:19.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.519+0000 7f1b1b502980 20 remove_watcher() i=1 2024-03-09T14:50:19.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.523+0000 7f1b1b502980 20 remove_watcher() i=2 2024-03-09T14:50:19.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.523+0000 7f1b1b502980 20 remove_watcher() i=3 2024-03-09T14:50:19.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.527+0000 7f1b1b502980 20 remove_watcher() i=4 2024-03-09T14:50:19.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.527+0000 7f1b1b502980 20 remove_watcher() i=5 2024-03-09T14:50:19.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.531+0000 7f1b1b502980 20 remove_watcher() i=6 2024-03-09T14:50:19.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.531+0000 7f1b1b502980 20 remove_watcher() i=7 2024-03-09T14:50:19.547 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:19.547 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:50:19.548 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-set-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:50:19.548 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:19.549 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:19.549 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:19.592 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:19.592 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:19.654 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket error-at-set-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2024-03-09T14:50:19.689 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:19.689 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:19.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.731+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.731+0000 7f0538db8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.731+0000 7f0538db8980 20 realm 2024-03-09T14:50:19.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.731+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.731+0000 7f0538db8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.731+0000 7f0538db8980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:19.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.731+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.731+0000 7f0538db8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.731+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.731+0000 7f0538db8980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:19.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.731+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.735+0000 7f0538db8980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:19.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.735+0000 7f0538db8980 20 searching for the correct realm 2024-03-09T14:50:19.743 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.739+0000 7f05360cf640 0 -- 172.21.15.1:0/3047138671 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f05280e1040 msgr2=0x7f05280c4400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:19.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.747+0000 7f05360cf640 0 -- 172.21.15.1:0/3047138671 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f0400075e90 msgr2=0x7f04000782c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:19.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:19.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:19.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:19.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:19.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:19.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:19.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 zone default found 2024-03-09T14:50:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 4 Realm: () 2024-03-09T14:50:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 zonegroup default 2024-03-09T14:50:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 started sync module instance, tier type = 2024-03-09T14:50:19.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.755+0000 7f0538db8980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:19.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.775+0000 7f0538db8980 20 add_watcher() i=0 2024-03-09T14:50:19.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.775+0000 7f0538db8980 20 add_watcher() i=1 2024-03-09T14:50:19.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.775+0000 7f0538db8980 20 add_watcher() i=2 2024-03-09T14:50:19.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.775+0000 7f0538db8980 20 add_watcher() i=3 2024-03-09T14:50:19.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.775+0000 7f0538db8980 20 add_watcher() i=4 2024-03-09T14:50:19.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.775+0000 7f0538db8980 20 add_watcher() i=5 2024-03-09T14:50:19.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.775+0000 7f0538db8980 20 add_watcher() i=6 2024-03-09T14:50:19.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.775+0000 7f0538db8980 20 add_watcher() i=7 2024-03-09T14:50:19.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.775+0000 7f0538db8980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:19.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f03ed7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:19.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:19.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:19.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 5 note: GC not initialized 2024-03-09T14:50:19.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0367fff640 20 reqs_thread_entry: start 2024-03-09T14:50:19.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:19.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 20 Filter name: none 2024-03-09T14:50:19.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0365ffb640 20 reqs_thread_entry: start 2024-03-09T14:50:19.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 10 cache get: name=default.rgw.meta+root+error-at-set-target-layout : miss 2024-03-09T14:50:19.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:19.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 10 cache put: name=default.rgw.meta+root+error-at-set-target-layout info.flags=0x11 2024-03-09T14:50:19.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 10 adding default.rgw.meta+root+error-at-set-target-layout to cache LRU end 2024-03-09T14:50:19.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 : miss 2024-03-09T14:50:19.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:19.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 info.flags=0x17 2024-03-09T14:50:19.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 10 adding default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 to cache LRU end 2024-03-09T14:50:19.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:50:19.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:19.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.779+0000 7f0538db8980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 2024-03-09T14:50:19.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.787+0000 7f0538db8980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 info.flags=0x17 2024-03-09T14:50:19.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.787+0000 7f0538db8980 10 moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 to cache LRU end 2024-03-09T14:50:19.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.787+0000 7f0538db8980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:19.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.787+0000 7f0538db8980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19, ofs0, ns] 2024-03-09T14:50:19.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.787+0000 7f041a7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562392 cookie 94707493517168 notifier 5882 bl.length()=917 2024-03-09T14:50:19.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.787+0000 7f041a7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 info.flags=0x17 2024-03-09T14:50:19.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.787+0000 7f041a7fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 to cache LRU end 2024-03-09T14:50:19.791 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.787+0000 7f041a7fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:19.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.791+0000 7f0538db8980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.19]): orig_sources= new_sources= 2024-03-09T14:50:19.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.791+0000 7f0538db8980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.19]): potential sources added=[] removed=[] 2024-03-09T14:50:19.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.791+0000 7f0538db8980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.19]): orig_dests= new_dests= 2024-03-09T14:50:19.793 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.791+0000 7f0538db8980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.19]): potential dests added=[] removed=[] 2024-03-09T14:50:19.795 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.791+0000 7f0538db8980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.19.1.0 2024-03-09T14:50:19.795 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.791+0000 7f0538db8980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.19.1.0 2024-03-09T14:50:19.795 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.791+0000 7f0538db8980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.19.1.1 2024-03-09T14:50:19.795 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.791+0000 7f0538db8980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.19.1.1 2024-03-09T14:50:19.796 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.791+0000 7f0538db8980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.19.0 2024-03-09T14:50:19.800 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.799+0000 7f0538db8980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 : hit (requested=0x17, cached=0x17) 2024-03-09T14:50:19.800 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.799+0000 7f0538db8980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 2024-03-09T14:50:19.803 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.799+0000 7f0538db8980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 info.flags=0x17 2024-03-09T14:50:19.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.799+0000 7f0538db8980 10 moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 to cache LRU end 2024-03-09T14:50:19.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.799+0000 7f0538db8980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:19.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.799+0000 7f0538db8980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19, ofs0, ns] 2024-03-09T14:50:19.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.803+0000 7f0419ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562393 cookie 94707493517168 notifier 5882 bl.length()=885 2024-03-09T14:50:19.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.803+0000 7f0419ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 info.flags=0x17 2024-03-09T14:50:19.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.803+0000 7f0419ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 to cache LRU end 2024-03-09T14:50:19.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.803+0000 7f0419ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:19.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.803+0000 7f0538db8980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.19]): orig_sources= new_sources= 2024-03-09T14:50:19.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.803+0000 7f0538db8980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.19]): potential sources added=[] removed=[] 2024-03-09T14:50:19.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.803+0000 7f0538db8980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.19]): orig_dests= new_dests= 2024-03-09T14:50:19.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.803+0000 7f0538db8980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.19]): potential dests added=[] removed=[] 2024-03-09T14:50:19.808 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.807+0000 7f0538db8980 1 execute INFO: reshard of bucket "error-at-set-target-layout" completed successfully 2024-03-09T14:50:19.812 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.807+0000 7f03ed7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:19.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.815+0000 7f0538db8980 20 remove_watcher() i=0 2024-03-09T14:50:19.817 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.815+0000 7f0538db8980 2 removed watcher, disabling cache 2024-03-09T14:50:19.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.815+0000 7f0538db8980 20 remove_watcher() i=1 2024-03-09T14:50:19.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.819+0000 7f0538db8980 20 remove_watcher() i=2 2024-03-09T14:50:19.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.819+0000 7f0538db8980 20 remove_watcher() i=3 2024-03-09T14:50:19.826 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.823+0000 7f0538db8980 20 remove_watcher() i=4 2024-03-09T14:50:19.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.823+0000 7f0538db8980 20 remove_watcher() i=5 2024-03-09T14:50:19.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.827+0000 7f0538db8980 20 remove_watcher() i=6 2024-03-09T14:50:19.830 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.827+0000 7f0538db8980 20 remove_watcher() i=7 2024-03-09T14:50:19.846 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2024-03-09T14:50:19.876 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:19.877 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:19.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.915+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.915+0000 7f27de536980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.915+0000 7f27de536980 20 realm 2024-03-09T14:50:19.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.915+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.915+0000 7f27de536980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.915+0000 7f27de536980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:19.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.915+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.915+0000 7f27de536980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.915+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.919+0000 7f27de536980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:19.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.919+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.919+0000 7f27de536980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:19.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.919+0000 7f27de536980 20 searching for the correct realm 2024-03-09T14:50:19.930 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.927+0000 7f27db84d640 0 -- 172.21.15.1:0/3760956293 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f27cc008050 msgr2=0x7f27cc054c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:19.937 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.935+0000 7f27dc84f640 0 -- 172.21.15.1:0/3760956293 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f27d0035070 msgr2=0x7f27d0056920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:19.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.939+0000 7f27de536980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:19.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.939+0000 7f27de536980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:19.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.939+0000 7f27de536980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:19.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.939+0000 7f27de536980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:19.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.939+0000 7f27de536980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:19.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.939+0000 7f27de536980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:19.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.939+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.939+0000 7f27de536980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.939+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:19.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:19.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 zone default found 2024-03-09T14:50:19.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 4 Realm: () 2024-03-09T14:50:19.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:19.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:19.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:19.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 zonegroup default 2024-03-09T14:50:19.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 started sync module instance, tier type = 2024-03-09T14:50:19.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.943+0000 7f27de536980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:19.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.959+0000 7f27de536980 20 add_watcher() i=0 2024-03-09T14:50:19.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.959+0000 7f27de536980 20 add_watcher() i=1 2024-03-09T14:50:19.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.959+0000 7f27de536980 20 add_watcher() i=2 2024-03-09T14:50:19.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.959+0000 7f27de536980 20 add_watcher() i=3 2024-03-09T14:50:19.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.959+0000 7f27de536980 20 add_watcher() i=4 2024-03-09T14:50:19.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.959+0000 7f27de536980 20 add_watcher() i=5 2024-03-09T14:50:19.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.959+0000 7f27de536980 20 add_watcher() i=6 2024-03-09T14:50:19.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.959+0000 7f27de536980 20 add_watcher() i=7 2024-03-09T14:50:19.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.959+0000 7f27de536980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:19.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.963+0000 7f26937de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:19.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.963+0000 7f27de536980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:19.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.963+0000 7f27de536980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:19.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.963+0000 7f27de536980 5 note: GC not initialized 2024-03-09T14:50:19.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.963+0000 7f26927dc640 20 reqs_thread_entry: start 2024-03-09T14:50:19.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.963+0000 7f27de536980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:19.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.963+0000 7f27de536980 20 Filter name: none 2024-03-09T14:50:19.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.963+0000 7f268b7fe640 20 reqs_thread_entry: start 2024-03-09T14:50:19.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.963+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.967+0000 7f27de536980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:19.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.967+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.967+0000 7f27de536980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:19.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.967+0000 7f27de536980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:19.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.967+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.967+0000 7f27de536980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:19.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.967+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.967+0000 7f27de536980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:19.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.967+0000 7f27de536980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:19.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.971+0000 7f27de536980 20 rados->read ofs=0 len=0 2024-03-09T14:50:19.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.971+0000 7f27de536980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:19.974 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.971+0000 7f26937de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:19.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.975+0000 7f27de536980 20 remove_watcher() i=0 2024-03-09T14:50:19.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.975+0000 7f27de536980 2 removed watcher, disabling cache 2024-03-09T14:50:19.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.979+0000 7f27de536980 20 remove_watcher() i=1 2024-03-09T14:50:19.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.979+0000 7f27de536980 20 remove_watcher() i=2 2024-03-09T14:50:19.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.983+0000 7f27de536980 20 remove_watcher() i=3 2024-03-09T14:50:19.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.983+0000 7f27de536980 20 remove_watcher() i=4 2024-03-09T14:50:19.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.987+0000 7f27de536980 20 remove_watcher() i=5 2024-03-09T14:50:19.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.987+0000 7f27de536980 20 remove_watcher() i=6 2024-03-09T14:50:19.991 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:19.987+0000 7f27de536980 20 remove_watcher() i=7 2024-03-09T14:50:20.006 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:20.006 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:50:20.006 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-set-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.19 num_objects: 19 size_kb: 1 num_shards: 2 2024-03-09T14:50:20.006 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:20.007 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:20.007 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:20.031 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:20.031 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:20.637 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:20.637 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:20.832 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard bucket with ECANCELED injected at set_target_layout 2024-03-09T14:50:20.832 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:20.834 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:20.834 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:20.963 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:20.963 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:21.055 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:21.056 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:21.157 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:21.158 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:21.258 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:21.258 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:21.363 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:21.363 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:21.464 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:21.464 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:21.561 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:21.561 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:21.679 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:21.680 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:21.780 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:21.780 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:21.884 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:21.884 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:21.978 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:21.978 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:22.079 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:22.079 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:22.180 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:22.180 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:22.280 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:22.280 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:22.381 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:22.381 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:22.464 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:22.464 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:22.567 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:22.568 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:22.666 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:22.667 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:22.763 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:22.763 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:22.863 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:22.864 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:22.949 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:22.949 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:23.047 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2024-03-09T14:50:23.082 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:23.082 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:23.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.123+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.127+0000 7f456a4be980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.127+0000 7f456a4be980 20 realm 2024-03-09T14:50:23.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.127+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.127+0000 7f456a4be980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.127+0000 7f456a4be980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:23.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.127+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.127+0000 7f456a4be980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.130 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.127+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.127+0000 7f456a4be980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:23.131 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.127+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.131+0000 7f456a4be980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:23.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.131+0000 7f456a4be980 20 searching for the correct realm 2024-03-09T14:50:23.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.135+0000 7f45687d7640 0 -- 172.21.15.1:0/940996244 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55715b094ad0 msgr2=0x55715b096f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:23.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.143+0000 7f4567fd6640 0 -- 172.21.15.1:0/940996244 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f455c0793d0 msgr2=0x7f455c082210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:23.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:23.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:23.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:23.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:23.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:23.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:23.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:23.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:23.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 zone default found 2024-03-09T14:50:23.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 4 Realm: () 2024-03-09T14:50:23.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:23.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:23.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:23.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 zonegroup default 2024-03-09T14:50:23.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 started sync module instance, tier type = 2024-03-09T14:50:23.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.151+0000 7f456a4be980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:23.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.167+0000 7f456a4be980 20 add_watcher() i=0 2024-03-09T14:50:23.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.167+0000 7f456a4be980 20 add_watcher() i=1 2024-03-09T14:50:23.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.167+0000 7f456a4be980 20 add_watcher() i=2 2024-03-09T14:50:23.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.167+0000 7f456a4be980 20 add_watcher() i=3 2024-03-09T14:50:23.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.167+0000 7f456a4be980 20 add_watcher() i=4 2024-03-09T14:50:23.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.167+0000 7f456a4be980 20 add_watcher() i=5 2024-03-09T14:50:23.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.167+0000 7f456a4be980 20 add_watcher() i=6 2024-03-09T14:50:23.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.171+0000 7f456a4be980 20 add_watcher() i=7 2024-03-09T14:50:23.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.171+0000 7f456a4be980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:23.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.171+0000 7f441e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:23.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.171+0000 7f456a4be980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:23.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.171+0000 7f456a4be980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:23.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.171+0000 7f456a4be980 5 note: GC not initialized 2024-03-09T14:50:23.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f441d7e2640 20 reqs_thread_entry: start 2024-03-09T14:50:23.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:23.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 Filter name: none 2024-03-09T14:50:23.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f44167fc640 20 reqs_thread_entry: start 2024-03-09T14:50:23.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:23.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:23.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:23.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:23.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:23.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:23.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.179 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.175+0000 7f456a4be980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.179+0000 7f441e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:23.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.183+0000 7f456a4be980 20 remove_watcher() i=0 2024-03-09T14:50:23.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.183+0000 7f456a4be980 2 removed watcher, disabling cache 2024-03-09T14:50:23.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.183+0000 7f456a4be980 20 remove_watcher() i=1 2024-03-09T14:50:23.190 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.187+0000 7f456a4be980 20 remove_watcher() i=2 2024-03-09T14:50:23.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.191+0000 7f456a4be980 20 remove_watcher() i=3 2024-03-09T14:50:23.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.191+0000 7f456a4be980 20 remove_watcher() i=4 2024-03-09T14:50:23.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.195+0000 7f456a4be980 20 remove_watcher() i=5 2024-03-09T14:50:23.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.195+0000 7f456a4be980 20 remove_watcher() i=6 2024-03-09T14:50:23.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.195+0000 7f456a4be980 20 remove_watcher() i=7 2024-03-09T14:50:23.214 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:23.214 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:50:23.214 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-set-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:50:23.214 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:23.214 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket error-at-set-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-error-at set_target_layout --inject-error-code 125 2024-03-09T14:50:23.248 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:23.248 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:23.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.287+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.293 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.291+0000 7f2c660c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.291+0000 7f2c660c1980 20 realm 2024-03-09T14:50:23.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.291+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.291+0000 7f2c660c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.291+0000 7f2c660c1980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:23.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.291+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.291+0000 7f2c660c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.291+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.291+0000 7f2c660c1980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:23.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.291+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.295+0000 7f2c660c1980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:23.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.295+0000 7f2c660c1980 20 searching for the correct realm 2024-03-09T14:50:23.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.299+0000 7f2c643da640 0 -- 172.21.15.1:0/551958234 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x558f2454dbe0 msgr2=0x558f24550020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:23.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.511+0000 7f2c63bd9640 0 -- 172.21.15.1:0/551958234 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f2c58075c30 msgr2=0x7f2c58065960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:23.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.727+0000 7f2c660c1980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:23.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.727+0000 7f2c660c1980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:23.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.727+0000 7f2c660c1980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:23.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.727+0000 7f2c660c1980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:23.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.727+0000 7f2c660c1980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:23.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.727+0000 7f2c660c1980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:23.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.727+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:23.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:23.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 zone default found 2024-03-09T14:50:23.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 4 Realm: () 2024-03-09T14:50:23.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:23.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:23.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:23.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 zonegroup default 2024-03-09T14:50:23.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 started sync module instance, tier type = 2024-03-09T14:50:23.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.731+0000 7f2c660c1980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:23.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.755+0000 7f2c660c1980 20 add_watcher() i=0 2024-03-09T14:50:23.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.755+0000 7f2c660c1980 20 add_watcher() i=1 2024-03-09T14:50:23.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.755+0000 7f2c660c1980 20 add_watcher() i=2 2024-03-09T14:50:23.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.755+0000 7f2c660c1980 20 add_watcher() i=3 2024-03-09T14:50:23.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.755+0000 7f2c660c1980 20 add_watcher() i=4 2024-03-09T14:50:23.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.755+0000 7f2c660c1980 20 add_watcher() i=5 2024-03-09T14:50:23.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.755+0000 7f2c660c1980 20 add_watcher() i=6 2024-03-09T14:50:23.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.755+0000 7f2c660c1980 20 add_watcher() i=7 2024-03-09T14:50:23.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.755+0000 7f2c660c1980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:23.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.759+0000 7f2b1a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:23.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.759+0000 7f2c660c1980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:23.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.759+0000 7f2c660c1980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:23.763 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.759+0000 7f2c660c1980 5 note: GC not initialized 2024-03-09T14:50:23.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.759+0000 7f2b197e2640 20 reqs_thread_entry: start 2024-03-09T14:50:23.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.759+0000 7f2c660c1980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:23.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.759+0000 7f2c660c1980 20 Filter name: none 2024-03-09T14:50:23.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.759+0000 7f2b127fc640 20 reqs_thread_entry: start 2024-03-09T14:50:23.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 10 cache get: name=default.rgw.meta+root+error-at-set-target-layout : miss 2024-03-09T14:50:23.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:23.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 10 cache put: name=default.rgw.meta+root+error-at-set-target-layout info.flags=0x11 2024-03-09T14:50:23.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 10 adding default.rgw.meta+root+error-at-set-target-layout to cache LRU end 2024-03-09T14:50:23.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 : miss 2024-03-09T14:50:23.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:23.765 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 info.flags=0x17 2024-03-09T14:50:23.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 10 adding default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 to cache LRU end 2024-03-09T14:50:23.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:50:23.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:23.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.763+0000 7f2c660c1980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 2024-03-09T14:50:23.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.767+0000 7f2c660c1980 -1 Injecting error=-125 at location=set_target_layout 2024-03-09T14:50:23.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.767+0000 7f2c660c1980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 info.flags=0x17 2024-03-09T14:50:23.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.767+0000 7f2c660c1980 10 moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 to cache LRU end 2024-03-09T14:50:23.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.767+0000 7f2c660c1980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:23.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.767+0000 7f2c660c1980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20, ofs0, ns] 2024-03-09T14:50:23.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.767+0000 7f2b42ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562397 cookie 94073278287952 notifier 6297 bl.length()=917 2024-03-09T14:50:23.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.767+0000 7f2b42ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 info.flags=0x17 2024-03-09T14:50:23.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.767+0000 7f2b42ffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 to cache LRU end 2024-03-09T14:50:23.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.767+0000 7f2b42ffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:23.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.771+0000 7f2c660c1980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.20]): orig_sources= new_sources= 2024-03-09T14:50:23.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.771+0000 7f2c660c1980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.20]): potential sources added=[] removed=[] 2024-03-09T14:50:23.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.771+0000 7f2c660c1980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.20]): orig_dests= new_dests= 2024-03-09T14:50:23.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.771+0000 7f2c660c1980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.20]): potential dests added=[] removed=[] 2024-03-09T14:50:23.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.775+0000 7f2c660c1980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.20.1.0 2024-03-09T14:50:23.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.775+0000 7f2c660c1980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.20.1.0 2024-03-09T14:50:23.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.775+0000 7f2c660c1980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.20.1.1 2024-03-09T14:50:23.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.775+0000 7f2c660c1980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.20.1.1 2024-03-09T14:50:23.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.775+0000 7f2c660c1980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.20.0 2024-03-09T14:50:23.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.779+0000 7f2c660c1980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 : hit (requested=0x17, cached=0x17) 2024-03-09T14:50:23.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.779+0000 7f2c660c1980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 2024-03-09T14:50:23.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.779+0000 7f2c660c1980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 info.flags=0x17 2024-03-09T14:50:23.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.779+0000 7f2c660c1980 10 moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 to cache LRU end 2024-03-09T14:50:23.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.779+0000 7f2c660c1980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:23.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.779+0000 7f2c660c1980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20, ofs0, ns] 2024-03-09T14:50:23.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.779+0000 7f2b427fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562398 cookie 94073278287952 notifier 6297 bl.length()=885 2024-03-09T14:50:23.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.779+0000 7f2b427fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 info.flags=0x17 2024-03-09T14:50:23.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.779+0000 7f2b427fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 to cache LRU end 2024-03-09T14:50:23.784 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.779+0000 7f2b427fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:23.786 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.783+0000 7f2c660c1980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.20]): orig_sources= new_sources= 2024-03-09T14:50:23.786 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.783+0000 7f2c660c1980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.20]): potential sources added=[] removed=[] 2024-03-09T14:50:23.786 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.783+0000 7f2c660c1980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.20]): orig_dests= new_dests= 2024-03-09T14:50:23.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.783+0000 7f2c660c1980 20 handle_bi_update(): bucket=:error-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.20]): potential dests added=[] removed=[] 2024-03-09T14:50:23.788 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.787+0000 7f2c660c1980 1 execute INFO: reshard of bucket "error-at-set-target-layout" completed successfully 2024-03-09T14:50:23.790 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.787+0000 7f2b1a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:23.795 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.791+0000 7f2c660c1980 20 remove_watcher() i=0 2024-03-09T14:50:23.796 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.791+0000 7f2c660c1980 2 removed watcher, disabling cache 2024-03-09T14:50:23.800 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.799+0000 7f2c660c1980 20 remove_watcher() i=1 2024-03-09T14:50:23.802 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.799+0000 7f2c660c1980 20 remove_watcher() i=2 2024-03-09T14:50:23.805 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.803+0000 7f2c660c1980 20 remove_watcher() i=3 2024-03-09T14:50:23.807 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.803+0000 7f2c660c1980 20 remove_watcher() i=4 2024-03-09T14:50:23.810 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.807+0000 7f2c660c1980 20 remove_watcher() i=5 2024-03-09T14:50:23.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.811+0000 7f2c660c1980 20 remove_watcher() i=6 2024-03-09T14:50:23.813 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.811+0000 7f2c660c1980 20 remove_watcher() i=7 2024-03-09T14:50:23.839 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2024-03-09T14:50:23.866 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:23.866 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:23.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.907+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.911+0000 7f83bef09980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.911+0000 7f83bef09980 20 realm 2024-03-09T14:50:23.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.911+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.911+0000 7f83bef09980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.911+0000 7f83bef09980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:23.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.911+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.911+0000 7f83bef09980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:23.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.911+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.911+0000 7f83bef09980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:23.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.911+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:23.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.915+0000 7f83bef09980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:23.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.915+0000 7f83bef09980 20 searching for the correct realm 2024-03-09T14:50:23.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.919+0000 7f83bd222640 0 -- 172.21.15.1:0/829667610 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f83a4055da0 msgr2=0x7f83a4067a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:23.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:23.927+0000 7f83bca21640 0 -- 172.21.15.1:0/829667610 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f83a406f920 msgr2=0x7f83a40672e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:24.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.143+0000 7f83bef09980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:24.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.143+0000 7f83bef09980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:24.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.143+0000 7f83bef09980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:24.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.143+0000 7f83bef09980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:24.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.143+0000 7f83bef09980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:24.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.143+0000 7f83bef09980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:24.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.143+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:24.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.143+0000 7f83bef09980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:24.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.143+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:24.149 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:24.149 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:24.149 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:24.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 20 zone default found 2024-03-09T14:50:24.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 4 Realm: () 2024-03-09T14:50:24.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:24.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:24.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:24.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 20 zonegroup default 2024-03-09T14:50:24.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:24.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:24.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:24.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:24.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.147+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:24.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.151+0000 7f83bef09980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:24.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.151+0000 7f83bef09980 20 started sync module instance, tier type = 2024-03-09T14:50:24.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.151+0000 7f83bef09980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:24.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.183+0000 7f83bef09980 20 add_watcher() i=0 2024-03-09T14:50:24.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.183+0000 7f83bef09980 20 add_watcher() i=1 2024-03-09T14:50:24.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.183+0000 7f83bef09980 20 add_watcher() i=2 2024-03-09T14:50:24.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.183+0000 7f83bef09980 20 add_watcher() i=3 2024-03-09T14:50:24.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.183+0000 7f83bef09980 20 add_watcher() i=4 2024-03-09T14:50:24.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.183+0000 7f83bef09980 20 add_watcher() i=5 2024-03-09T14:50:24.188 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.183+0000 7f83bef09980 20 add_watcher() i=6 2024-03-09T14:50:24.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.191+0000 7f83bef09980 20 add_watcher() i=7 2024-03-09T14:50:24.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.191+0000 7f83bef09980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:24.196 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.195+0000 7f82737e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:24.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.195+0000 7f83bef09980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:24.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.195+0000 7f83bef09980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:24.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.195+0000 7f83bef09980 5 note: GC not initialized 2024-03-09T14:50:24.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.195+0000 7f82727e4640 20 reqs_thread_entry: start 2024-03-09T14:50:24.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.195+0000 7f83bef09980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:24.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.195+0000 7f83bef09980 20 Filter name: none 2024-03-09T14:50:24.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.195+0000 7f826bfff640 20 reqs_thread_entry: start 2024-03-09T14:50:24.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.195+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:24.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.195+0000 7f83bef09980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:24.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.195+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:24.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.199+0000 7f83bef09980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:24.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.199+0000 7f83bef09980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:24.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.199+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:24.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.199+0000 7f83bef09980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:24.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.199+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:24.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.199+0000 7f83bef09980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:24.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.199+0000 7f83bef09980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:24.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.199+0000 7f83bef09980 20 rados->read ofs=0 len=0 2024-03-09T14:50:24.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.199+0000 7f83bef09980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:24.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.203+0000 7f82737e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:24.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.207+0000 7f83bef09980 20 remove_watcher() i=0 2024-03-09T14:50:24.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.207+0000 7f83bef09980 2 removed watcher, disabling cache 2024-03-09T14:50:24.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.211+0000 7f83bef09980 20 remove_watcher() i=1 2024-03-09T14:50:24.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.211+0000 7f83bef09980 20 remove_watcher() i=2 2024-03-09T14:50:24.217 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.215+0000 7f83bef09980 20 remove_watcher() i=3 2024-03-09T14:50:24.220 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.215+0000 7f83bef09980 20 remove_watcher() i=4 2024-03-09T14:50:24.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.219+0000 7f83bef09980 20 remove_watcher() i=5 2024-03-09T14:50:24.222 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.219+0000 7f83bef09980 20 remove_watcher() i=6 2024-03-09T14:50:24.224 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:24.223+0000 7f83bef09980 20 remove_watcher() i=7 2024-03-09T14:50:24.245 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:24.246 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:50:24.246 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-set-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.20 num_objects: 20 size_kb: 1 num_shards: 2 2024-03-09T14:50:24.246 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:24.248 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:24.249 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:24.292 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:24.292 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:24.937 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:24.937 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:25.118 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard bucket with abort at set_target_layout 2024-03-09T14:50:25.118 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:25.121 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:25.121 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:25.245 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:25.245 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:25.336 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:25.337 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:25.436 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:25.436 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:25.537 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:25.537 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:25.635 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:25.635 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:25.734 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:25.734 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:25.836 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:25.836 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:25.923 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:25.923 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:26.023 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:26.023 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:26.131 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:26.131 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:26.229 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:26.229 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:26.334 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:26.334 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:26.434 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:26.435 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:26.537 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:26.537 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:26.638 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:26.639 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:26.749 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:26.749 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:26.853 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:26.853 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:26.951 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:26.951 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:27.050 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:27.050 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:27.149 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:27.149 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:27.248 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:27.248 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:27.346 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket abort-at-set-target-layout 2024-03-09T14:50:27.380 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:27.380 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:27.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.423+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.423+0000 7f874c878980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.423+0000 7f874c878980 20 realm 2024-03-09T14:50:27.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.423+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.423+0000 7f874c878980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.423+0000 7f874c878980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:27.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.423+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.427+0000 7f874c878980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.427+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.427+0000 7f874c878980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:27.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.427+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.427+0000 7f874c878980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:27.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.427+0000 7f874c878980 20 searching for the correct realm 2024-03-09T14:50:27.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.435+0000 7f8749b8f640 0 -- 172.21.15.1:0/307196045 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55e1ffc1d130 msgr2=0x55e1ffc1f5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:27.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.439+0000 7f8749b8f640 0 -- 172.21.15.1:0/307196045 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f873c0bf020 msgr2=0x7f873c009e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:27.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:27.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:27.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:27.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:27.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:27.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:27.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:27.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:27.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 zone default found 2024-03-09T14:50:27.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 4 Realm: () 2024-03-09T14:50:27.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:27.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:27.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:27.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 zonegroup default 2024-03-09T14:50:27.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.447+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.451+0000 7f874c878980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.451+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.451+0000 7f874c878980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.451+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.451+0000 7f874c878980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.451+0000 7f874c878980 20 started sync module instance, tier type = 2024-03-09T14:50:27.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.451+0000 7f874c878980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:27.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.471+0000 7f874c878980 20 add_watcher() i=0 2024-03-09T14:50:27.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.471+0000 7f874c878980 20 add_watcher() i=1 2024-03-09T14:50:27.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.471+0000 7f874c878980 20 add_watcher() i=2 2024-03-09T14:50:27.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.471+0000 7f874c878980 20 add_watcher() i=3 2024-03-09T14:50:27.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.471+0000 7f874c878980 20 add_watcher() i=4 2024-03-09T14:50:27.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.471+0000 7f874c878980 20 add_watcher() i=5 2024-03-09T14:50:27.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.471+0000 7f874c878980 20 add_watcher() i=6 2024-03-09T14:50:27.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.471+0000 7f874c878980 20 add_watcher() i=7 2024-03-09T14:50:27.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.471+0000 7f874c878980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:27.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.475+0000 7f8600fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:27.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.475+0000 7f874c878980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:27.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.475+0000 7f874c878980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:27.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.475+0000 7f874c878980 5 note: GC not initialized 2024-03-09T14:50:27.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.475+0000 7f857b7fe640 20 reqs_thread_entry: start 2024-03-09T14:50:27.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.475+0000 7f874c878980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:27.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.475+0000 7f874c878980 20 Filter name: none 2024-03-09T14:50:27.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.475+0000 7f85797fa640 20 reqs_thread_entry: start 2024-03-09T14:50:27.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.475+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.475+0000 7f874c878980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:27.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.475+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.480 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.479+0000 7f874c878980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:27.480 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.479+0000 7f874c878980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:27.480 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.479+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.479+0000 7f874c878980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:27.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.479+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.479+0000 7f874c878980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:27.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.479+0000 7f874c878980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:27.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.483+0000 7f874c878980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.485 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.483+0000 7f874c878980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.483+0000 7f8600fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:27.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.487+0000 7f874c878980 20 remove_watcher() i=0 2024-03-09T14:50:27.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.487+0000 7f874c878980 2 removed watcher, disabling cache 2024-03-09T14:50:27.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.491+0000 7f874c878980 20 remove_watcher() i=1 2024-03-09T14:50:27.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.491+0000 7f874c878980 20 remove_watcher() i=2 2024-03-09T14:50:27.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.495+0000 7f874c878980 20 remove_watcher() i=3 2024-03-09T14:50:27.499 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.495+0000 7f874c878980 20 remove_watcher() i=4 2024-03-09T14:50:27.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.499+0000 7f874c878980 20 remove_watcher() i=5 2024-03-09T14:50:27.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.499+0000 7f874c878980 20 remove_watcher() i=6 2024-03-09T14:50:27.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.499+0000 7f874c878980 20 remove_watcher() i=7 2024-03-09T14:50:27.518 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.519 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:50:27.519 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: abort-at-set-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:50:27.519 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.519 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket abort-at-set-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-abort-at set_target_layout 2024-03-09T14:50:27.550 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:27.550 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:27.593 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.591+0000 7f92a5129980 20 realm 2024-03-09T14:50:27.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.591+0000 7f92a5129980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:27.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.595+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:27.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.595+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.595+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:27.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.595+0000 7f92a5129980 20 searching for the correct realm 2024-03-09T14:50:27.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.603+0000 7f92a3442640 0 -- 172.21.15.1:0/1929550071 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x5597a56801e0 msgr2=0x5597a5682660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:50:27.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.607+0000 7f92a2c41640 0 -- 172.21.15.1:0/1929550071 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f929804d3e0 msgr2=0x7f929804d860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:27.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:27.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:27.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:27.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:27.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:27.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:27.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.615+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.615+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.615+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:27.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:27.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 zone default found 2024-03-09T14:50:27.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 4 Realm: () 2024-03-09T14:50:27.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:27.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:27.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:27.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 zonegroup default 2024-03-09T14:50:27.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 started sync module instance, tier type = 2024-03-09T14:50:27.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.619+0000 7f92a5129980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:27.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=0 2024-03-09T14:50:27.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=1 2024-03-09T14:50:27.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=2 2024-03-09T14:50:27.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=3 2024-03-09T14:50:27.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=4 2024-03-09T14:50:27.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=5 2024-03-09T14:50:27.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=6 2024-03-09T14:50:27.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=7 2024-03-09T14:50:27.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.635+0000 7f92a5129980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:27.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f91597e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:27.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f92a5129980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:27.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f92a5129980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:27.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f92a5129980 5 note: GC not initialized 2024-03-09T14:50:27.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f90d3fff640 20 reqs_thread_entry: start 2024-03-09T14:50:27.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f92a5129980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:27.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f92a5129980 20 Filter name: none 2024-03-09T14:50:27.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f90d1ffb640 20 reqs_thread_entry: start 2024-03-09T14:50:27.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f92a5129980 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2024-03-09T14:50:27.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:27.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f92a5129980 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2024-03-09T14:50:27.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f92a5129980 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2024-03-09T14:50:27.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f92a5129980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 : miss 2024-03-09T14:50:27.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.639+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.644 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.643+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:27.644 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.643+0000 7f92a5129980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 info.flags=0x17 2024-03-09T14:50:27.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.643+0000 7f92a5129980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 to cache LRU end 2024-03-09T14:50:27.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.643+0000 7f92a5129980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:50:27.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.643+0000 7f92a5129980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:27.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.643+0000 7f92a5129980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 2024-03-09T14:50:27.650 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f92a5129980 time 2024-03-09T14:50:27.651783+0000 2024-03-09T14:50:27.650 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:50:27.651 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:50:27.651 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f92a89fed7b] 2024-03-09T14:50:27.651 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f92a89fef2f] 2024-03-09T14:50:27.651 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5597a42f1c51] 2024-03-09T14:50:27.651 INFO:tasks.workunit.client.0.smithi001.stderr: 4: radosgw-admin(+0x8d9bf2) [0x5597a4305bf2] 2024-03-09T14:50:27.651 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5597a431222b] 2024-03-09T14:50:27.651 INFO:tasks.workunit.client.0.smithi001.stderr: 6: main() 2024-03-09T14:50:27.651 INFO:tasks.workunit.client.0.smithi001.stderr: 7: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f92a7869d90] 2024-03-09T14:50:27.652 INFO:tasks.workunit.client.0.smithi001.stderr: 8: __libc_start_main() 2024-03-09T14:50:27.652 INFO:tasks.workunit.client.0.smithi001.stderr: 9: _start() 2024-03-09T14:50:27.652 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.647+0000 7f92a5129980 -1 ./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f92a5129980 time 2024-03-09T14:50:27.651783+0000 2024-03-09T14:50:27.652 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:50:27.652 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.652 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:50:27.652 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f92a89fed7b] 2024-03-09T14:50:27.652 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f92a89fef2f] 2024-03-09T14:50:27.652 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5597a42f1c51] 2024-03-09T14:50:27.652 INFO:tasks.workunit.client.0.smithi001.stderr: 4: radosgw-admin(+0x8d9bf2) [0x5597a4305bf2] 2024-03-09T14:50:27.653 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5597a431222b] 2024-03-09T14:50:27.653 INFO:tasks.workunit.client.0.smithi001.stderr: 6: main() 2024-03-09T14:50:27.653 INFO:tasks.workunit.client.0.smithi001.stderr: 7: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f92a7869d90] 2024-03-09T14:50:27.653 INFO:tasks.workunit.client.0.smithi001.stderr: 8: __libc_start_main() 2024-03-09T14:50:27.653 INFO:tasks.workunit.client.0.smithi001.stderr: 9: _start() 2024-03-09T14:50:27.653 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.653 INFO:tasks.workunit.client.0.smithi001.stderr:*** Caught signal (Aborted) ** 2024-03-09T14:50:27.653 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f92a5129980 thread_name:radosgw-admin 2024-03-09T14:50:27.680 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:50:27.680 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f92a7882520] 2024-03-09T14:50:27.680 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:50:27.680 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:50:27.680 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:50:27.680 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f92a89fedd5] 2024-03-09T14:50:27.680 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f92a89fef2f] 2024-03-09T14:50:27.680 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5597a42f1c51] 2024-03-09T14:50:27.680 INFO:tasks.workunit.client.0.smithi001.stderr: 8: radosgw-admin(+0x8d9bf2) [0x5597a4305bf2] 2024-03-09T14:50:27.680 INFO:tasks.workunit.client.0.smithi001.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5597a431222b] 2024-03-09T14:50:27.680 INFO:tasks.workunit.client.0.smithi001.stderr: 10: main() 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f92a7869d90] 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: 12: __libc_start_main() 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: 13: _start() 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.651+0000 7f92a5129980 -1 *** Caught signal (Aborted) ** 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f92a5129980 thread_name:radosgw-admin 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f92a7882520] 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f92a89fedd5] 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f92a89fef2f] 2024-03-09T14:50:27.681 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5597a42f1c51] 2024-03-09T14:50:27.682 INFO:tasks.workunit.client.0.smithi001.stderr: 8: radosgw-admin(+0x8d9bf2) [0x5597a4305bf2] 2024-03-09T14:50:27.682 INFO:tasks.workunit.client.0.smithi001.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5597a431222b] 2024-03-09T14:50:27.682 INFO:tasks.workunit.client.0.smithi001.stderr: 10: main() 2024-03-09T14:50:27.682 INFO:tasks.workunit.client.0.smithi001.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f92a7869d90] 2024-03-09T14:50:27.682 INFO:tasks.workunit.client.0.smithi001.stderr: 12: __libc_start_main() 2024-03-09T14:50:27.682 INFO:tasks.workunit.client.0.smithi001.stderr: 13: _start() 2024-03-09T14:50:27.682 INFO:tasks.workunit.client.0.smithi001.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-09T14:50:27.682 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.682 INFO:tasks.workunit.client.0.smithi001.stderr:--- begin dump of recent events --- 2024-03-09T14:50:27.682 INFO:tasks.workunit.client.0.smithi001.stderr: -499> 2024-03-09T14:50:27.567+0000 7f92a2440640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.682 INFO:tasks.workunit.client.0.smithi001.stderr: -498> 2024-03-09T14:50:27.567+0000 7f92a2440640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.682 INFO:tasks.workunit.client.0.smithi001.stderr: -497> 2024-03-09T14:50:27.567+0000 7f92a2440640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.683 INFO:tasks.workunit.client.0.smithi001.stderr: -496> 2024-03-09T14:50:27.567+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.683 INFO:tasks.workunit.client.0.smithi001.stderr: -495> 2024-03-09T14:50:27.567+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.683 INFO:tasks.workunit.client.0.smithi001.stderr: -494> 2024-03-09T14:50:27.567+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.683 INFO:tasks.workunit.client.0.smithi001.stderr: -493> 2024-03-09T14:50:27.567+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.683 INFO:tasks.workunit.client.0.smithi001.stderr: -492> 2024-03-09T14:50:27.567+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.683 INFO:tasks.workunit.client.0.smithi001.stderr: -491> 2024-03-09T14:50:27.567+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.683 INFO:tasks.workunit.client.0.smithi001.stderr: -490> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): get_auth_request con 0x5597a544f890 auth_method 0 2024-03-09T14:50:27.683 INFO:tasks.workunit.client.0.smithi001.stderr: -489> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.684 INFO:tasks.workunit.client.0.smithi001.stderr: -488> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.684 INFO:tasks.workunit.client.0.smithi001.stderr: -487> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.684 INFO:tasks.workunit.client.0.smithi001.stderr: -486> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): handle_auth_done global_id 5975 payload 274 2024-03-09T14:50:27.684 INFO:tasks.workunit.client.0.smithi001.stderr: -485> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient: _finish_hunting 0 2024-03-09T14:50:27.684 INFO:tasks.workunit.client.0.smithi001.stderr: -484> 2024-03-09T14:50:27.571+0000 7f92a2c41640 1 monclient: found mon.noname-c 2024-03-09T14:50:27.684 INFO:tasks.workunit.client.0.smithi001.stderr: -483> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.684 INFO:tasks.workunit.client.0.smithi001.stderr: -482> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.684 INFO:tasks.workunit.client.0.smithi001.stderr: -481> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:50:27.685 INFO:tasks.workunit.client.0.smithi001.stderr: -480> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: dump: 2024-03-09T14:50:27.685 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.685 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.685 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.685 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.685 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.685 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.685 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.686 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.686 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.686 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.686 INFO:tasks.workunit.client.0.smithi001.stderr: -479> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: _finish_auth 0 2024-03-09T14:50:27.686 INFO:tasks.workunit.client.0.smithi001.stderr: -478> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: _check_auth_tickets 2024-03-09T14:50:27.686 INFO:tasks.workunit.client.0.smithi001.stderr: -477> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:50:27.687 INFO:tasks.workunit.client.0.smithi001.stderr: -476> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.687 INFO:tasks.workunit.client.0.smithi001.stderr: -475> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:50:27.687 INFO:tasks.workunit.client.0.smithi001.stderr: -474> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: dump: 2024-03-09T14:50:27.688 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.688 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.688 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.688 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.688 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.688 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.688 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.689 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.689 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.689 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.689 INFO:tasks.workunit.client.0.smithi001.stderr: -473> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 monclient: authenticate success, global_id 5975 2024-03-09T14:50:27.689 INFO:tasks.workunit.client.0.smithi001.stderr: -472> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: get_monmap_and_config success 2024-03-09T14:50:27.689 INFO:tasks.workunit.client.0.smithi001.stderr: -471> 2024-03-09T14:50:27.571+0000 7f92a5129980 4 set_mon_vals no callback set 2024-03-09T14:50:27.689 INFO:tasks.workunit.client.0.smithi001.stderr: -470> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: shutdown 2024-03-09T14:50:27.689 INFO:tasks.workunit.client.0.smithi001.stderr: -469> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 asok(0x5597a5388bb0) unregister_commands rotate-key 2024-03-09T14:50:27.690 INFO:tasks.workunit.client.0.smithi001.stderr: -468> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: build_initial_monmap 2024-03-09T14:50:27.690 INFO:tasks.workunit.client.0.smithi001.stderr: -467> 2024-03-09T14:50:27.571+0000 7f92a5129980 1 build_initial for_mkfs: 0 2024-03-09T14:50:27.690 INFO:tasks.workunit.client.0.smithi001.stderr: -466> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: monmap: 2024-03-09T14:50:27.690 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:50:27.690 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:50:27.690 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:50:27.690 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:50:27.691 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:50:27.691 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.691 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:50:27.691 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:50:27.691 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:50:27.691 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.691 INFO:tasks.workunit.client.0.smithi001.stderr: -465> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding auth protocol: cephx 2024-03-09T14:50:27.692 INFO:tasks.workunit.client.0.smithi001.stderr: -464> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding auth protocol: cephx 2024-03-09T14:50:27.692 INFO:tasks.workunit.client.0.smithi001.stderr: -463> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding auth protocol: cephx 2024-03-09T14:50:27.692 INFO:tasks.workunit.client.0.smithi001.stderr: -462> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: secure 2024-03-09T14:50:27.692 INFO:tasks.workunit.client.0.smithi001.stderr: -461> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: crc 2024-03-09T14:50:27.692 INFO:tasks.workunit.client.0.smithi001.stderr: -460> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: secure 2024-03-09T14:50:27.693 INFO:tasks.workunit.client.0.smithi001.stderr: -459> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: crc 2024-03-09T14:50:27.693 INFO:tasks.workunit.client.0.smithi001.stderr: -458> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: secure 2024-03-09T14:50:27.693 INFO:tasks.workunit.client.0.smithi001.stderr: -457> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: crc 2024-03-09T14:50:27.693 INFO:tasks.workunit.client.0.smithi001.stderr: -456> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: crc 2024-03-09T14:50:27.693 INFO:tasks.workunit.client.0.smithi001.stderr: -455> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: secure 2024-03-09T14:50:27.693 INFO:tasks.workunit.client.0.smithi001.stderr: -454> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: crc 2024-03-09T14:50:27.693 INFO:tasks.workunit.client.0.smithi001.stderr: -453> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: secure 2024-03-09T14:50:27.694 INFO:tasks.workunit.client.0.smithi001.stderr: -452> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: crc 2024-03-09T14:50:27.694 INFO:tasks.workunit.client.0.smithi001.stderr: -451> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: secure 2024-03-09T14:50:27.694 INFO:tasks.workunit.client.0.smithi001.stderr: -450> 2024-03-09T14:50:27.571+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.694 INFO:tasks.workunit.client.0.smithi001.stderr: -449> 2024-03-09T14:50:27.571+0000 7f92a5129980 1 librados: starting msgr at 2024-03-09T14:50:27.694 INFO:tasks.workunit.client.0.smithi001.stderr: -448> 2024-03-09T14:50:27.571+0000 7f92a5129980 1 librados: starting objecter 2024-03-09T14:50:27.694 INFO:tasks.workunit.client.0.smithi001.stderr: -447> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command objecter_requests hook 0x5597a5363670 2024-03-09T14:50:27.694 INFO:tasks.workunit.client.0.smithi001.stderr: -446> 2024-03-09T14:50:27.571+0000 7f92a5129980 1 librados: setting wanted keys 2024-03-09T14:50:27.694 INFO:tasks.workunit.client.0.smithi001.stderr: -445> 2024-03-09T14:50:27.571+0000 7f92a5129980 1 librados: calling monclient init 2024-03-09T14:50:27.694 INFO:tasks.workunit.client.0.smithi001.stderr: -444> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: init 2024-03-09T14:50:27.694 INFO:tasks.workunit.client.0.smithi001.stderr: -443> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding auth protocol: cephx 2024-03-09T14:50:27.694 INFO:tasks.workunit.client.0.smithi001.stderr: -442> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding auth protocol: cephx 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -441> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding auth protocol: cephx 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -440> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: secure 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -439> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: crc 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -438> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: secure 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -437> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: crc 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -436> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: secure 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -435> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: crc 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -434> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: crc 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -433> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: secure 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -432> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: crc 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -431> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: secure 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -430> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: crc 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -429> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: secure 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -428> 2024-03-09T14:50:27.571+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.695 INFO:tasks.workunit.client.0.smithi001.stderr: -427> 2024-03-09T14:50:27.571+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.696 INFO:tasks.workunit.client.0.smithi001.stderr: -426> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command rotate-key hook 0x5597a5597248 2024-03-09T14:50:27.696 INFO:tasks.workunit.client.0.smithi001.stderr: -425> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: _reopen_session rank -1 2024-03-09T14:50:27.696 INFO:tasks.workunit.client.0.smithi001.stderr: -424> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: _add_conns ranks=[2,1,0] 2024-03-09T14:50:27.696 INFO:tasks.workunit.client.0.smithi001.stderr: -423> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-c con 0x5597a5423840 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:50:27.696 INFO:tasks.workunit.client.0.smithi001.stderr: -422> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-b con 0x5597a5423c20 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:50:27.697 INFO:tasks.workunit.client.0.smithi001.stderr: -421> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-a con 0x5597a544f890 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:50:27.697 INFO:tasks.workunit.client.0.smithi001.stderr: -420> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.697 INFO:tasks.workunit.client.0.smithi001.stderr: -419> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.697 INFO:tasks.workunit.client.0.smithi001.stderr: -418> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.697 INFO:tasks.workunit.client.0.smithi001.stderr: -417> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): _renew_subs 2024-03-09T14:50:27.697 INFO:tasks.workunit.client.0.smithi001.stderr: -416> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): authenticate will time out at 2321.318576s 2024-03-09T14:50:27.698 INFO:tasks.workunit.client.0.smithi001.stderr: -415> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): get_auth_request con 0x5597a5423840 auth_method 0 2024-03-09T14:50:27.698 INFO:tasks.workunit.client.0.smithi001.stderr: -414> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.698 INFO:tasks.workunit.client.0.smithi001.stderr: -413> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.698 INFO:tasks.workunit.client.0.smithi001.stderr: -412> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.698 INFO:tasks.workunit.client.0.smithi001.stderr: -411> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): get_auth_request con 0x5597a544f890 auth_method 0 2024-03-09T14:50:27.698 INFO:tasks.workunit.client.0.smithi001.stderr: -410> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.698 INFO:tasks.workunit.client.0.smithi001.stderr: -409> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.698 INFO:tasks.workunit.client.0.smithi001.stderr: -408> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.698 INFO:tasks.workunit.client.0.smithi001.stderr: -407> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.698 INFO:tasks.workunit.client.0.smithi001.stderr: -406> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.698 INFO:tasks.workunit.client.0.smithi001.stderr: -405> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.699 INFO:tasks.workunit.client.0.smithi001.stderr: -404> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.699 INFO:tasks.workunit.client.0.smithi001.stderr: -403> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.699 INFO:tasks.workunit.client.0.smithi001.stderr: -402> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.699 INFO:tasks.workunit.client.0.smithi001.stderr: -401> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): handle_auth_done global_id 5978 payload 824 2024-03-09T14:50:27.699 INFO:tasks.workunit.client.0.smithi001.stderr: -400> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient: _finish_hunting 0 2024-03-09T14:50:27.699 INFO:tasks.workunit.client.0.smithi001.stderr: -399> 2024-03-09T14:50:27.571+0000 7f92a2c41640 1 monclient: found mon.noname-c 2024-03-09T14:50:27.699 INFO:tasks.workunit.client.0.smithi001.stderr: -398> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.699 INFO:tasks.workunit.client.0.smithi001.stderr: -397> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.699 INFO:tasks.workunit.client.0.smithi001.stderr: -396> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr: -395> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: dump: 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr: -394> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: _finish_auth 0 2024-03-09T14:50:27.700 INFO:tasks.workunit.client.0.smithi001.stderr: -393> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: _check_auth_tickets 2024-03-09T14:50:27.701 INFO:tasks.workunit.client.0.smithi001.stderr: -392> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:50:27.701 INFO:tasks.workunit.client.0.smithi001.stderr: -391> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.701 INFO:tasks.workunit.client.0.smithi001.stderr: -390> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:50:27.701 INFO:tasks.workunit.client.0.smithi001.stderr: -389> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: dump: 2024-03-09T14:50:27.701 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.701 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.701 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.701 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.701 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.701 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.701 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.701 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.702 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.702 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.702 INFO:tasks.workunit.client.0.smithi001.stderr: -388> 2024-03-09T14:50:27.571+0000 7f9190e09640 4 set_mon_vals no callback set 2024-03-09T14:50:27.702 INFO:tasks.workunit.client.0.smithi001.stderr: -387> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 monclient: authenticate success, global_id 5978 2024-03-09T14:50:27.702 INFO:tasks.workunit.client.0.smithi001.stderr: -386> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: _renew_subs 2024-03-09T14:50:27.702 INFO:tasks.workunit.client.0.smithi001.stderr: -385> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.702 INFO:tasks.workunit.client.0.smithi001.stderr: -384> 2024-03-09T14:50:27.575+0000 7f92a5129980 10 monclient: _renew_subs 2024-03-09T14:50:27.702 INFO:tasks.workunit.client.0.smithi001.stderr: -383> 2024-03-09T14:50:27.575+0000 7f92a5129980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.702 INFO:tasks.workunit.client.0.smithi001.stderr: -382> 2024-03-09T14:50:27.575+0000 7f92a5129980 1 librados: init done 2024-03-09T14:50:27.702 INFO:tasks.workunit.client.0.smithi001.stderr: -381> 2024-03-09T14:50:27.575+0000 7f918f606640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr: -380> 2024-03-09T14:50:27.575+0000 7f918f606640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr: -379> 2024-03-09T14:50:27.575+0000 7f918f606640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr: -378> 2024-03-09T14:50:27.575+0000 7f92a2440640 10 monclient: get_auth_request con 0x7f9188066040 auth_method 0 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr: -377> 2024-03-09T14:50:27.575+0000 7f92a3442640 10 monclient: get_auth_request con 0x5597a55a0a00 auth_method 0 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr: -376> 2024-03-09T14:50:27.575+0000 7f92a2c41640 10 monclient: get_auth_request con 0x5597a55e1cb0 auth_method 0 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr: -375> 2024-03-09T14:50:27.579+0000 7f92a2440640 10 monclient: get_auth_request con 0x5597a55e6d40 auth_method 0 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr: -374> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient: build_initial_monmap 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr: -373> 2024-03-09T14:50:27.579+0000 7f92a5129980 1 build_initial for_mkfs: 0 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr: -372> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient: monmap: 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:50:27.703 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.704 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:50:27.704 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:50:27.704 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:50:27.704 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.704 INFO:tasks.workunit.client.0.smithi001.stderr: -371> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding auth protocol: cephx 2024-03-09T14:50:27.704 INFO:tasks.workunit.client.0.smithi001.stderr: -370> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding auth protocol: cephx 2024-03-09T14:50:27.704 INFO:tasks.workunit.client.0.smithi001.stderr: -369> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding auth protocol: cephx 2024-03-09T14:50:27.704 INFO:tasks.workunit.client.0.smithi001.stderr: -368> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: secure 2024-03-09T14:50:27.704 INFO:tasks.workunit.client.0.smithi001.stderr: -367> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: crc 2024-03-09T14:50:27.704 INFO:tasks.workunit.client.0.smithi001.stderr: -366> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: secure 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -365> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: crc 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -364> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: secure 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -363> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: crc 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -362> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: crc 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -361> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: secure 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -360> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: crc 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -359> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: secure 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -358> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: crc 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -357> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: secure 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -356> 2024-03-09T14:50:27.579+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -355> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command objecter_requests cmddesc objecter_requests hook 0x5597a55f0300 EEXIST 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -354> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient: init 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -353> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding auth protocol: cephx 2024-03-09T14:50:27.705 INFO:tasks.workunit.client.0.smithi001.stderr: -352> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding auth protocol: cephx 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -351> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding auth protocol: cephx 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -350> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: secure 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -349> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: crc 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -348> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: secure 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -347> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: crc 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -346> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: secure 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -345> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: crc 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -344> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: crc 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -343> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: secure 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -342> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: crc 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -341> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: secure 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -340> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: crc 2024-03-09T14:50:27.706 INFO:tasks.workunit.client.0.smithi001.stderr: -339> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: secure 2024-03-09T14:50:27.707 INFO:tasks.workunit.client.0.smithi001.stderr: -338> 2024-03-09T14:50:27.579+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.707 INFO:tasks.workunit.client.0.smithi001.stderr: -337> 2024-03-09T14:50:27.579+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.707 INFO:tasks.workunit.client.0.smithi001.stderr: -336> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command rotate-key cmddesc rotate-key hook 0x5597a55e9c80 EEXIST 2024-03-09T14:50:27.707 INFO:tasks.workunit.client.0.smithi001.stderr: -335> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient: _reopen_session rank -1 2024-03-09T14:50:27.707 INFO:tasks.workunit.client.0.smithi001.stderr: -334> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient: _add_conns ranks=[0,1,2] 2024-03-09T14:50:27.707 INFO:tasks.workunit.client.0.smithi001.stderr: -333> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-a con 0x5597a5614f60 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:50:27.707 INFO:tasks.workunit.client.0.smithi001.stderr: -332> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-b con 0x5597a5615340 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:50:27.707 INFO:tasks.workunit.client.0.smithi001.stderr: -331> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-c con 0x5597a5610500 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:50:27.707 INFO:tasks.workunit.client.0.smithi001.stderr: -330> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.707 INFO:tasks.workunit.client.0.smithi001.stderr: -329> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -328> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -327> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): _renew_subs 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -326> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): authenticate will time out at 2321.327879s 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -325> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): get_auth_request con 0x5597a5610500 auth_method 0 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -324> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -323> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -322> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -321> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): get_auth_request con 0x5597a5614f60 auth_method 0 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -320> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -319> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -318> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -317> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.708 INFO:tasks.workunit.client.0.smithi001.stderr: -316> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.709 INFO:tasks.workunit.client.0.smithi001.stderr: -315> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.710 INFO:tasks.workunit.client.0.smithi001.stderr: -314> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.710 INFO:tasks.workunit.client.0.smithi001.stderr: -313> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.710 INFO:tasks.workunit.client.0.smithi001.stderr: -312> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.710 INFO:tasks.workunit.client.0.smithi001.stderr: -311> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): get_auth_request con 0x5597a5615340 auth_method 0 2024-03-09T14:50:27.710 INFO:tasks.workunit.client.0.smithi001.stderr: -310> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.710 INFO:tasks.workunit.client.0.smithi001.stderr: -309> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.710 INFO:tasks.workunit.client.0.smithi001.stderr: -308> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.710 INFO:tasks.workunit.client.0.smithi001.stderr: -307> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_done global_id 6351 payload 824 2024-03-09T14:50:27.710 INFO:tasks.workunit.client.0.smithi001.stderr: -306> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient: _finish_hunting 0 2024-03-09T14:50:27.710 INFO:tasks.workunit.client.0.smithi001.stderr: -305> 2024-03-09T14:50:27.583+0000 7f92a2c41640 1 monclient: found mon.noname-a 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr: -304> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr: -303> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr: -302> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr: -301> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: dump: 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.711 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.712 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.712 INFO:tasks.workunit.client.0.smithi001.stderr: -300> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: _finish_auth 0 2024-03-09T14:50:27.712 INFO:tasks.workunit.client.0.smithi001.stderr: -299> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: _check_auth_tickets 2024-03-09T14:50:27.712 INFO:tasks.workunit.client.0.smithi001.stderr: -298> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:50:27.712 INFO:tasks.workunit.client.0.smithi001.stderr: -297> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.712 INFO:tasks.workunit.client.0.smithi001.stderr: -296> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:50:27.712 INFO:tasks.workunit.client.0.smithi001.stderr: -295> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: dump: 2024-03-09T14:50:27.712 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.712 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.712 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.713 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.713 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.713 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.713 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.713 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.713 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.713 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.713 INFO:tasks.workunit.client.0.smithi001.stderr: -294> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 monclient: authenticate success, global_id 6351 2024-03-09T14:50:27.713 INFO:tasks.workunit.client.0.smithi001.stderr: -293> 2024-03-09T14:50:27.583+0000 7f92a3c43640 4 set_mon_vals no callback set 2024-03-09T14:50:27.713 INFO:tasks.workunit.client.0.smithi001.stderr: -292> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: _renew_subs 2024-03-09T14:50:27.713 INFO:tasks.workunit.client.0.smithi001.stderr: -291> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr: -290> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: _renew_subs 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr: -289> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr: -288> 2024-03-09T14:50:27.583+0000 7f918ce01640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr: -287> 2024-03-09T14:50:27.583+0000 7f918ce01640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr: -286> 2024-03-09T14:50:27.583+0000 7f918ce01640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr: -285> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: get_monmap_and_config 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr: -284> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: build_initial_monmap 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr: -283> 2024-03-09T14:50:27.583+0000 7f92a5129980 1 build_initial for_mkfs: 0 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr: -282> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: monmap: 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:50:27.714 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr: -281> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding auth protocol: cephx 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr: -280> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding auth protocol: cephx 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr: -279> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding auth protocol: cephx 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr: -278> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: secure 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr: -277> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: crc 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr: -276> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: secure 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr: -275> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: crc 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr: -274> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: secure 2024-03-09T14:50:27.715 INFO:tasks.workunit.client.0.smithi001.stderr: -273> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: crc 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -272> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: crc 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -271> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: secure 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -270> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: crc 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -269> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: secure 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -268> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: crc 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -267> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: secure 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -266> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient: get_auth_request con 0x7f917c065f80 auth_method 0 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -265> 2024-03-09T14:50:27.583+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -264> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: init 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -263> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding auth protocol: cephx 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -262> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding auth protocol: cephx 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -261> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding auth protocol: cephx 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -260> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: secure 2024-03-09T14:50:27.716 INFO:tasks.workunit.client.0.smithi001.stderr: -259> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: crc 2024-03-09T14:50:27.717 INFO:tasks.workunit.client.0.smithi001.stderr: -258> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: secure 2024-03-09T14:50:27.717 INFO:tasks.workunit.client.0.smithi001.stderr: -257> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: crc 2024-03-09T14:50:27.717 INFO:tasks.workunit.client.0.smithi001.stderr: -256> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: secure 2024-03-09T14:50:27.717 INFO:tasks.workunit.client.0.smithi001.stderr: -255> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: crc 2024-03-09T14:50:27.717 INFO:tasks.workunit.client.0.smithi001.stderr: -254> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: crc 2024-03-09T14:50:27.717 INFO:tasks.workunit.client.0.smithi001.stderr: -253> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: secure 2024-03-09T14:50:27.717 INFO:tasks.workunit.client.0.smithi001.stderr: -252> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: crc 2024-03-09T14:50:27.717 INFO:tasks.workunit.client.0.smithi001.stderr: -251> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: secure 2024-03-09T14:50:27.717 INFO:tasks.workunit.client.0.smithi001.stderr: -250> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: crc 2024-03-09T14:50:27.717 INFO:tasks.workunit.client.0.smithi001.stderr: -249> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: secure 2024-03-09T14:50:27.717 INFO:tasks.workunit.client.0.smithi001.stderr: -248> 2024-03-09T14:50:27.583+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.718 INFO:tasks.workunit.client.0.smithi001.stderr: -247> 2024-03-09T14:50:27.583+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.718 INFO:tasks.workunit.client.0.smithi001.stderr: -246> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command rotate-key cmddesc rotate-key hook 0x7ffd35927d38 EEXIST 2024-03-09T14:50:27.718 INFO:tasks.workunit.client.0.smithi001.stderr: -245> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: _reopen_session rank -1 2024-03-09T14:50:27.718 INFO:tasks.workunit.client.0.smithi001.stderr: -244> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: _add_conns ranks=[2,0,1] 2024-03-09T14:50:27.718 INFO:tasks.workunit.client.0.smithi001.stderr: -243> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-c con 0x5597a5660f80 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:50:27.718 INFO:tasks.workunit.client.0.smithi001.stderr: -242> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-a con 0x5597a5661360 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:50:27.718 INFO:tasks.workunit.client.0.smithi001.stderr: -241> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-b con 0x5597a566c270 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:50:27.718 INFO:tasks.workunit.client.0.smithi001.stderr: -240> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.718 INFO:tasks.workunit.client.0.smithi001.stderr: -239> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.718 INFO:tasks.workunit.client.0.smithi001.stderr: -238> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.718 INFO:tasks.workunit.client.0.smithi001.stderr: -237> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): _renew_subs 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -236> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): authenticate will time out at 2321.331663s 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -235> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): get_auth_request con 0x5597a5661360 auth_method 0 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -234> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -233> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -232> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -231> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): get_auth_request con 0x5597a5660f80 auth_method 0 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -230> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -229> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -228> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -227> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -226> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -225> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -224> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.719 INFO:tasks.workunit.client.0.smithi001.stderr: -223> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.720 INFO:tasks.workunit.client.0.smithi001.stderr: -222> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.720 INFO:tasks.workunit.client.0.smithi001.stderr: -221> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): handle_auth_done global_id 5984 payload 274 2024-03-09T14:50:27.720 INFO:tasks.workunit.client.0.smithi001.stderr: -220> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient: _finish_hunting 0 2024-03-09T14:50:27.720 INFO:tasks.workunit.client.0.smithi001.stderr: -219> 2024-03-09T14:50:27.587+0000 7f92a2c41640 1 monclient: found mon.noname-c 2024-03-09T14:50:27.720 INFO:tasks.workunit.client.0.smithi001.stderr: -218> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.720 INFO:tasks.workunit.client.0.smithi001.stderr: -217> 2024-03-09T14:50:27.587+0000 7f92a3442640 10 monclient: get_auth_request con 0x5597a566c270 auth_method 0 2024-03-09T14:50:27.720 INFO:tasks.workunit.client.0.smithi001.stderr: -216> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.720 INFO:tasks.workunit.client.0.smithi001.stderr: -215> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:50:27.720 INFO:tasks.workunit.client.0.smithi001.stderr: -214> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: dump: 2024-03-09T14:50:27.720 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr: -213> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: _finish_auth 0 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr: -212> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: _check_auth_tickets 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr: -211> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 monclient: authenticate success, global_id 5984 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr: -210> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:50:27.721 INFO:tasks.workunit.client.0.smithi001.stderr: -209> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.722 INFO:tasks.workunit.client.0.smithi001.stderr: -208> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:50:27.722 INFO:tasks.workunit.client.0.smithi001.stderr: -207> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: dump: 2024-03-09T14:50:27.722 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.722 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.722 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.722 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.722 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.723 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.723 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.723 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.723 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.723 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.723 INFO:tasks.workunit.client.0.smithi001.stderr: -206> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: get_monmap_and_config success 2024-03-09T14:50:27.724 INFO:tasks.workunit.client.0.smithi001.stderr: -205> 2024-03-09T14:50:27.587+0000 7f92a5129980 4 set_mon_vals no callback set 2024-03-09T14:50:27.724 INFO:tasks.workunit.client.0.smithi001.stderr: -204> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: shutdown 2024-03-09T14:50:27.724 INFO:tasks.workunit.client.0.smithi001.stderr: -203> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: build_initial_monmap 2024-03-09T14:50:27.724 INFO:tasks.workunit.client.0.smithi001.stderr: -202> 2024-03-09T14:50:27.587+0000 7f92a5129980 1 build_initial for_mkfs: 0 2024-03-09T14:50:27.724 INFO:tasks.workunit.client.0.smithi001.stderr: -201> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: monmap: 2024-03-09T14:50:27.724 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:50:27.724 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:50:27.724 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:50:27.724 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:50:27.725 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:50:27.725 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.725 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:50:27.725 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:50:27.725 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:50:27.725 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.725 INFO:tasks.workunit.client.0.smithi001.stderr: -200> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding auth protocol: cephx 2024-03-09T14:50:27.725 INFO:tasks.workunit.client.0.smithi001.stderr: -199> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding auth protocol: cephx 2024-03-09T14:50:27.726 INFO:tasks.workunit.client.0.smithi001.stderr: -198> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding auth protocol: cephx 2024-03-09T14:50:27.726 INFO:tasks.workunit.client.0.smithi001.stderr: -197> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: secure 2024-03-09T14:50:27.726 INFO:tasks.workunit.client.0.smithi001.stderr: -196> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: crc 2024-03-09T14:50:27.726 INFO:tasks.workunit.client.0.smithi001.stderr: -195> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: secure 2024-03-09T14:50:27.726 INFO:tasks.workunit.client.0.smithi001.stderr: -194> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: crc 2024-03-09T14:50:27.726 INFO:tasks.workunit.client.0.smithi001.stderr: -193> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: secure 2024-03-09T14:50:27.726 INFO:tasks.workunit.client.0.smithi001.stderr: -192> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: crc 2024-03-09T14:50:27.726 INFO:tasks.workunit.client.0.smithi001.stderr: -191> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: crc 2024-03-09T14:50:27.727 INFO:tasks.workunit.client.0.smithi001.stderr: -190> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: secure 2024-03-09T14:50:27.727 INFO:tasks.workunit.client.0.smithi001.stderr: -189> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: crc 2024-03-09T14:50:27.727 INFO:tasks.workunit.client.0.smithi001.stderr: -188> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: secure 2024-03-09T14:50:27.727 INFO:tasks.workunit.client.0.smithi001.stderr: -187> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: crc 2024-03-09T14:50:27.727 INFO:tasks.workunit.client.0.smithi001.stderr: -186> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: secure 2024-03-09T14:50:27.727 INFO:tasks.workunit.client.0.smithi001.stderr: -185> 2024-03-09T14:50:27.587+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.728 INFO:tasks.workunit.client.0.smithi001.stderr: -184> 2024-03-09T14:50:27.587+0000 7f92a5129980 1 librados: starting msgr at 2024-03-09T14:50:27.728 INFO:tasks.workunit.client.0.smithi001.stderr: -183> 2024-03-09T14:50:27.587+0000 7f92a5129980 1 librados: starting objecter 2024-03-09T14:50:27.728 INFO:tasks.workunit.client.0.smithi001.stderr: -182> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command objecter_requests cmddesc objecter_requests hook 0x5597a565d260 EEXIST 2024-03-09T14:50:27.728 INFO:tasks.workunit.client.0.smithi001.stderr: -181> 2024-03-09T14:50:27.587+0000 7f92a5129980 1 librados: setting wanted keys 2024-03-09T14:50:27.728 INFO:tasks.workunit.client.0.smithi001.stderr: -180> 2024-03-09T14:50:27.587+0000 7f92a5129980 1 librados: calling monclient init 2024-03-09T14:50:27.728 INFO:tasks.workunit.client.0.smithi001.stderr: -179> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: init 2024-03-09T14:50:27.728 INFO:tasks.workunit.client.0.smithi001.stderr: -178> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding auth protocol: cephx 2024-03-09T14:50:27.728 INFO:tasks.workunit.client.0.smithi001.stderr: -177> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding auth protocol: cephx 2024-03-09T14:50:27.728 INFO:tasks.workunit.client.0.smithi001.stderr: -176> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding auth protocol: cephx 2024-03-09T14:50:27.728 INFO:tasks.workunit.client.0.smithi001.stderr: -175> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: secure 2024-03-09T14:50:27.729 INFO:tasks.workunit.client.0.smithi001.stderr: -174> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: crc 2024-03-09T14:50:27.729 INFO:tasks.workunit.client.0.smithi001.stderr: -173> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: secure 2024-03-09T14:50:27.729 INFO:tasks.workunit.client.0.smithi001.stderr: -172> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: crc 2024-03-09T14:50:27.729 INFO:tasks.workunit.client.0.smithi001.stderr: -171> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: secure 2024-03-09T14:50:27.729 INFO:tasks.workunit.client.0.smithi001.stderr: -170> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: crc 2024-03-09T14:50:27.729 INFO:tasks.workunit.client.0.smithi001.stderr: -169> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: crc 2024-03-09T14:50:27.729 INFO:tasks.workunit.client.0.smithi001.stderr: -168> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: secure 2024-03-09T14:50:27.729 INFO:tasks.workunit.client.0.smithi001.stderr: -167> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: crc 2024-03-09T14:50:27.729 INFO:tasks.workunit.client.0.smithi001.stderr: -166> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: secure 2024-03-09T14:50:27.729 INFO:tasks.workunit.client.0.smithi001.stderr: -165> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: crc 2024-03-09T14:50:27.729 INFO:tasks.workunit.client.0.smithi001.stderr: -164> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: secure 2024-03-09T14:50:27.730 INFO:tasks.workunit.client.0.smithi001.stderr: -163> 2024-03-09T14:50:27.587+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.730 INFO:tasks.workunit.client.0.smithi001.stderr: -162> 2024-03-09T14:50:27.587+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.730 INFO:tasks.workunit.client.0.smithi001.stderr: -161> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command rotate-key cmddesc rotate-key hook 0x5597a5631dc8 EEXIST 2024-03-09T14:50:27.730 INFO:tasks.workunit.client.0.smithi001.stderr: -160> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: _reopen_session rank -1 2024-03-09T14:50:27.730 INFO:tasks.workunit.client.0.smithi001.stderr: -159> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: _add_conns ranks=[2,0,1] 2024-03-09T14:50:27.730 INFO:tasks.workunit.client.0.smithi001.stderr: -158> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-c con 0x5597a5660f80 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:50:27.730 INFO:tasks.workunit.client.0.smithi001.stderr: -157> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-a con 0x5597a566c270 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:50:27.731 INFO:tasks.workunit.client.0.smithi001.stderr: -156> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-b con 0x5597a565d850 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:50:27.731 INFO:tasks.workunit.client.0.smithi001.stderr: -155> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.731 INFO:tasks.workunit.client.0.smithi001.stderr: -154> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.731 INFO:tasks.workunit.client.0.smithi001.stderr: -153> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.731 INFO:tasks.workunit.client.0.smithi001.stderr: -152> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): _renew_subs 2024-03-09T14:50:27.731 INFO:tasks.workunit.client.0.smithi001.stderr: -151> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): authenticate will time out at 2321.334229s 2024-03-09T14:50:27.731 INFO:tasks.workunit.client.0.smithi001.stderr: -150> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): get_auth_request con 0x5597a5660f80 auth_method 0 2024-03-09T14:50:27.731 INFO:tasks.workunit.client.0.smithi001.stderr: -149> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.732 INFO:tasks.workunit.client.0.smithi001.stderr: -148> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.732 INFO:tasks.workunit.client.0.smithi001.stderr: -147> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.732 INFO:tasks.workunit.client.0.smithi001.stderr: -146> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): get_auth_request con 0x5597a566c270 auth_method 0 2024-03-09T14:50:27.732 INFO:tasks.workunit.client.0.smithi001.stderr: -145> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.732 INFO:tasks.workunit.client.0.smithi001.stderr: -144> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.732 INFO:tasks.workunit.client.0.smithi001.stderr: -143> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.732 INFO:tasks.workunit.client.0.smithi001.stderr: -142> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.732 INFO:tasks.workunit.client.0.smithi001.stderr: -141> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.733 INFO:tasks.workunit.client.0.smithi001.stderr: -140> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.733 INFO:tasks.workunit.client.0.smithi001.stderr: -139> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.733 INFO:tasks.workunit.client.0.smithi001.stderr: -138> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.733 INFO:tasks.workunit.client.0.smithi001.stderr: -137> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.733 INFO:tasks.workunit.client.0.smithi001.stderr: -136> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): handle_auth_done global_id 5987 payload 824 2024-03-09T14:50:27.733 INFO:tasks.workunit.client.0.smithi001.stderr: -135> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient: _finish_hunting 0 2024-03-09T14:50:27.733 INFO:tasks.workunit.client.0.smithi001.stderr: -134> 2024-03-09T14:50:27.587+0000 7f92a2c41640 1 monclient: found mon.noname-c 2024-03-09T14:50:27.734 INFO:tasks.workunit.client.0.smithi001.stderr: -133> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.734 INFO:tasks.workunit.client.0.smithi001.stderr: -132> 2024-03-09T14:50:27.587+0000 7f92a3442640 10 monclient: get_auth_request con 0x5597a565d850 auth_method 0 2024-03-09T14:50:27.734 INFO:tasks.workunit.client.0.smithi001.stderr: -131> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.734 INFO:tasks.workunit.client.0.smithi001.stderr: -130> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:50:27.735 INFO:tasks.workunit.client.0.smithi001.stderr: -129> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: dump: 2024-03-09T14:50:27.735 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.735 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.735 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.735 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.735 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.735 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.735 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.736 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.736 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.736 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.736 INFO:tasks.workunit.client.0.smithi001.stderr: -128> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: _finish_auth 0 2024-03-09T14:50:27.736 INFO:tasks.workunit.client.0.smithi001.stderr: -127> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: _check_auth_tickets 2024-03-09T14:50:27.736 INFO:tasks.workunit.client.0.smithi001.stderr: -126> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:50:27.736 INFO:tasks.workunit.client.0.smithi001.stderr: -125> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.736 INFO:tasks.workunit.client.0.smithi001.stderr: -124> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:50:27.737 INFO:tasks.workunit.client.0.smithi001.stderr: -123> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: dump: 2024-03-09T14:50:27.737 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.737 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.737 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.737 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.737 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.737 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.737 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.738 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.738 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.738 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.738 INFO:tasks.workunit.client.0.smithi001.stderr: -122> 2024-03-09T14:50:27.587+0000 7f91867fc640 4 set_mon_vals no callback set 2024-03-09T14:50:27.738 INFO:tasks.workunit.client.0.smithi001.stderr: -121> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 monclient: authenticate success, global_id 5987 2024-03-09T14:50:27.738 INFO:tasks.workunit.client.0.smithi001.stderr: -120> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: _renew_subs 2024-03-09T14:50:27.739 INFO:tasks.workunit.client.0.smithi001.stderr: -119> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.739 INFO:tasks.workunit.client.0.smithi001.stderr: -118> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: _renew_subs 2024-03-09T14:50:27.739 INFO:tasks.workunit.client.0.smithi001.stderr: -117> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.739 INFO:tasks.workunit.client.0.smithi001.stderr: -116> 2024-03-09T14:50:27.587+0000 7f92a5129980 1 librados: init done 2024-03-09T14:50:27.739 INFO:tasks.workunit.client.0.smithi001.stderr: -115> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command cr dump hook 0x5597a565ae88 2024-03-09T14:50:27.739 INFO:tasks.workunit.client.0.smithi001.stderr: -114> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.739 INFO:tasks.workunit.client.0.smithi001.stderr: -113> 2024-03-09T14:50:27.591+0000 7f9184ff9640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:50:27.740 INFO:tasks.workunit.client.0.smithi001.stderr: -112> 2024-03-09T14:50:27.591+0000 7f9184ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:50:27.740 INFO:tasks.workunit.client.0.smithi001.stderr: -111> 2024-03-09T14:50:27.591+0000 7f9184ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:50:27.740 INFO:tasks.workunit.client.0.smithi001.stderr: -110> 2024-03-09T14:50:27.591+0000 7f92a3442640 10 monclient: get_auth_request con 0x5597a56801e0 auth_method 0 2024-03-09T14:50:27.740 INFO:tasks.workunit.client.0.smithi001.stderr: -109> 2024-03-09T14:50:27.591+0000 7f92a2440640 10 monclient: get_auth_request con 0x7f916c066040 auth_method 0 2024-03-09T14:50:27.740 INFO:tasks.workunit.client.0.smithi001.stderr: -108> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.741 INFO:tasks.workunit.client.0.smithi001.stderr: -107> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 realm 2024-03-09T14:50:27.741 INFO:tasks.workunit.client.0.smithi001.stderr: -106> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.741 INFO:tasks.workunit.client.0.smithi001.stderr: -105> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.741 INFO:tasks.workunit.client.0.smithi001.stderr: -104> 2024-03-09T14:50:27.591+0000 7f92a5129980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:27.741 INFO:tasks.workunit.client.0.smithi001.stderr: -103> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.741 INFO:tasks.workunit.client.0.smithi001.stderr: -102> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.741 INFO:tasks.workunit.client.0.smithi001.stderr: -101> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.742 INFO:tasks.workunit.client.0.smithi001.stderr: -100> 2024-03-09T14:50:27.591+0000 7f92a2c41640 10 monclient: get_auth_request con 0x5597a5684460 auth_method 0 2024-03-09T14:50:27.742 INFO:tasks.workunit.client.0.smithi001.stderr: -99> 2024-03-09T14:50:27.595+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:27.742 INFO:tasks.workunit.client.0.smithi001.stderr: -98> 2024-03-09T14:50:27.595+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.742 INFO:tasks.workunit.client.0.smithi001.stderr: -97> 2024-03-09T14:50:27.595+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:27.742 INFO:tasks.workunit.client.0.smithi001.stderr: -96> 2024-03-09T14:50:27.595+0000 7f92a5129980 20 searching for the correct realm 2024-03-09T14:50:27.742 INFO:tasks.workunit.client.0.smithi001.stderr: -95> 2024-03-09T14:50:27.595+0000 7f92a2440640 10 monclient: get_auth_request con 0x5597a56896e0 auth_method 0 2024-03-09T14:50:27.742 INFO:tasks.workunit.client.0.smithi001.stderr: -94> 2024-03-09T14:50:27.595+0000 7f92a3442640 10 monclient: get_auth_request con 0x7f9298008040 auth_method 0 2024-03-09T14:50:27.742 INFO:tasks.workunit.client.0.smithi001.stderr: -93> 2024-03-09T14:50:27.599+0000 7f92a2c41640 10 monclient: get_auth_request con 0x7f929c0c7040 auth_method 0 2024-03-09T14:50:27.742 INFO:tasks.workunit.client.0.smithi001.stderr: -92> 2024-03-09T14:50:27.599+0000 7f92a2440640 10 monclient: get_auth_request con 0x7f9294061060 auth_method 0 2024-03-09T14:50:27.742 INFO:tasks.workunit.client.0.smithi001.stderr: -91> 2024-03-09T14:50:27.603+0000 7f92a3442640 0 -- 172.21.15.1:0/1929550071 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x5597a56801e0 msgr2=0x5597a5682660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:50:27.743 INFO:tasks.workunit.client.0.smithi001.stderr: -90> 2024-03-09T14:50:27.603+0000 7f9184ff9640 1 client.5987.objecter ms_handle_reset 0x5597a56801e0 session 0x5597a567fb60 osd.1 2024-03-09T14:50:27.743 INFO:tasks.workunit.client.0.smithi001.stderr: -89> 2024-03-09T14:50:27.603+0000 7f9184ff9640 10 monclient: _renew_subs 2024-03-09T14:50:27.743 INFO:tasks.workunit.client.0.smithi001.stderr: -88> 2024-03-09T14:50:27.603+0000 7f9184ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.743 INFO:tasks.workunit.client.0.smithi001.stderr: -87> 2024-03-09T14:50:27.603+0000 7f92a3442640 10 monclient: get_auth_request con 0x7f916c075e60 auth_method 0 2024-03-09T14:50:27.743 INFO:tasks.workunit.client.0.smithi001.stderr: -86> 2024-03-09T14:50:27.603+0000 7f92a3442640 10 monclient: get_auth_request con 0x7f929804e960 auth_method 0 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -85> 2024-03-09T14:50:27.607+0000 7f92a2c41640 10 monclient: get_auth_request con 0x7f929804d3e0 auth_method 0 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -84> 2024-03-09T14:50:27.607+0000 7f92a2c41640 0 -- 172.21.15.1:0/1929550071 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f929804d3e0 msgr2=0x7f929804d860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -83> 2024-03-09T14:50:27.607+0000 7f9184ff9640 1 client.5987.objecter ms_handle_reset 0x7f929804d3e0 session 0x7f929803ee10 osd.0 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -82> 2024-03-09T14:50:27.607+0000 7f92a2c41640 10 monclient: get_auth_request con 0x5597a56801e0 auth_method 0 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -81> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -80> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -79> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -78> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -77> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -76> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -75> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -74> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -73> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.744 INFO:tasks.workunit.client.0.smithi001.stderr: -72> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -71> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -70> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -69> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 zone default found 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -68> 2024-03-09T14:50:27.619+0000 7f92a5129980 4 Realm: () 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -67> 2024-03-09T14:50:27.619+0000 7f92a5129980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -66> 2024-03-09T14:50:27.619+0000 7f92a5129980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -65> 2024-03-09T14:50:27.619+0000 7f92a5129980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -64> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 zonegroup default 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -63> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -62> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -61> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -60> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -59> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.745 INFO:tasks.workunit.client.0.smithi001.stderr: -58> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -57> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 started sync module instance, tier type = 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -56> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -55> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=0 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -54> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=1 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -53> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=2 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -52> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=3 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -51> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=4 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -50> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=5 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -49> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=6 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -48> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=7 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -47> 2024-03-09T14:50:27.635+0000 7f92a5129980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -46> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding auth protocol: cephx 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -45> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding auth protocol: cephx 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -44> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding auth protocol: cephx 2024-03-09T14:50:27.746 INFO:tasks.workunit.client.0.smithi001.stderr: -43> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: secure 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -42> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: crc 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -41> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: secure 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -40> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: crc 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -39> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: secure 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -38> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: crc 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -37> 2024-03-09T14:50:27.639+0000 7f91597e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -36> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: crc 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -35> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: secure 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -34> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: crc 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -33> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: secure 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -32> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: crc 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -31> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: secure 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -30> 2024-03-09T14:50:27.639+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.747 INFO:tasks.workunit.client.0.smithi001.stderr: -29> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -28> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -27> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command cache list hook 0x5597a565cba0 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -26> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command cache inspect hook 0x5597a565cba0 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -25> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command cache erase hook 0x5597a565cba0 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -24> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command cache zap hook 0x5597a565cba0 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -23> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 note: GC not initialized 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -22> 2024-03-09T14:50:27.639+0000 7f90d3fff640 20 reqs_thread_entry: start 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -21> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command sync trace show hook 0x7f929c0047f0 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -20> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command sync trace history hook 0x7f929c0047f0 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -19> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command sync trace active hook 0x7f929c0047f0 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -18> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command sync trace active_short hook 0x7f929c0047f0 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -17> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -16> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 Filter name: none 2024-03-09T14:50:27.748 INFO:tasks.workunit.client.0.smithi001.stderr: -15> 2024-03-09T14:50:27.639+0000 7f90d1ffb640 20 reqs_thread_entry: start 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -14> 2024-03-09T14:50:27.639+0000 7f92a5129980 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -13> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -12> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -11> 2024-03-09T14:50:27.639+0000 7f92a5129980 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -10> 2024-03-09T14:50:27.639+0000 7f92a5129980 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -9> 2024-03-09T14:50:27.639+0000 7f92a5129980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 : miss 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -8> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -7> 2024-03-09T14:50:27.643+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -6> 2024-03-09T14:50:27.643+0000 7f92a5129980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 info.flags=0x17 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -5> 2024-03-09T14:50:27.643+0000 7f92a5129980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 to cache LRU end 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -4> 2024-03-09T14:50:27.643+0000 7f92a5129980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -3> 2024-03-09T14:50:27.643+0000 7f92a5129980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -2> 2024-03-09T14:50:27.643+0000 7f92a5129980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 2024-03-09T14:50:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: -1> 2024-03-09T14:50:27.647+0000 7f92a5129980 -1 ./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f92a5129980 time 2024-03-09T14:50:27.651783+0000 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f92a89fed7b] 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f92a89fef2f] 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5597a42f1c51] 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: 4: radosgw-admin(+0x8d9bf2) [0x5597a4305bf2] 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5597a431222b] 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: 6: main() 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: 7: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f92a7869d90] 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: 8: __libc_start_main() 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: 9: _start() 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.750 INFO:tasks.workunit.client.0.smithi001.stderr: 0> 2024-03-09T14:50:27.651+0000 7f92a5129980 -1 *** Caught signal (Aborted) ** 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f92a5129980 thread_name:radosgw-admin 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f92a7882520] 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f92a89fedd5] 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f92a89fef2f] 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5597a42f1c51] 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 8: radosgw-admin(+0x8d9bf2) [0x5597a4305bf2] 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5597a431222b] 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 10: main() 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f92a7869d90] 2024-03-09T14:50:27.751 INFO:tasks.workunit.client.0.smithi001.stderr: 12: __libc_start_main() 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 13: _start() 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr:--- logging levels --- 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 none 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 lockdep 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 context 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 crush 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_balancer 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_locker 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log_expire 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_migrator 2024-03-09T14:50:27.752 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 buffer 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 timer 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 filer 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 striper 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 objecter 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rados 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_mirror 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_replay 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_pwl 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 journaler 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objectcacher 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 immutable_obj_cache 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 client 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 osd 2024-03-09T14:50:27.753 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 optracker 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objclass 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 filestore 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 journal 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 ms 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mon 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 0/10 monc 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 paxos 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 tp 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 auth 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 crypto 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 finisher 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 reserver 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 heartbeatmap 2024-03-09T14:50:27.754 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 perfcounter 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 20/20 rgw 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_sync 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_datacache 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_access 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_dbstore 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_flight 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_lifecycle 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 javaclient 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 asok 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 throttle 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 refs 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 compressor 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluestore 2024-03-09T14:50:27.755 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluefs 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 bdev 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 kstore 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 4/ 5 rocksdb 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 fuse 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 2/ 5 mgr 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mgrc 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 dpdk 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 eventtrace 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 prioritycache 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 test 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephfs_mirror 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephsqlite 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 crimson_interrupt 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore 2024-03-09T14:50:27.756 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_onode 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_odata 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_omap 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_tm 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_t 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cleaner 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_epm 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_lba 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_fixedkv_tree 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cache 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_journal 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_device 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_backref 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 alienstore 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mclock 2024-03-09T14:50:27.757 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cyanstore 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 ceph_exporter 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 memstore 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 trace 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: -2/-2 (syslog threshold) 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: 99/99 (stderr threshold) 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr:--- pthread ID / name mapping for recent threads --- 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: 7f90d1ffb640 / http_manager 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: 7f90d3fff640 / http_manager 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: 7f91597e2640 / rgw_dt_lg_renew 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: 7f9184ff9640 / ms_dispatch 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: 7f91867fc640 / io_context_pool 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: 7f918ce01640 / ms_dispatch 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: 7f918f606640 / ms_dispatch 2024-03-09T14:50:27.758 INFO:tasks.workunit.client.0.smithi001.stderr: 7f9190e09640 / io_context_pool 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr: 7f92a2440640 / msgr-worker-2 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr: 7f92a2c41640 / msgr-worker-1 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr: 7f92a3442640 / msgr-worker-0 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr: 7f92a3c43640 / io_context_pool 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr: 7f92a5129980 / radosgw-admin 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr: max_recent 500 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr: max_new 1000 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr: log_file /var/log/ceph/ceph-client.admin.80714.log 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr:--- end dump of recent events --- 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr:--- begin dump of recent events --- 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr: -499> 2024-03-09T14:50:27.567+0000 7f92a2440640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr: -498> 2024-03-09T14:50:27.567+0000 7f92a2440640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr: -497> 2024-03-09T14:50:27.567+0000 7f92a2440640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.759 INFO:tasks.workunit.client.0.smithi001.stderr: -496> 2024-03-09T14:50:27.567+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.760 INFO:tasks.workunit.client.0.smithi001.stderr: -495> 2024-03-09T14:50:27.567+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.760 INFO:tasks.workunit.client.0.smithi001.stderr: -494> 2024-03-09T14:50:27.567+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.760 INFO:tasks.workunit.client.0.smithi001.stderr: -493> 2024-03-09T14:50:27.567+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.760 INFO:tasks.workunit.client.0.smithi001.stderr: -492> 2024-03-09T14:50:27.567+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.760 INFO:tasks.workunit.client.0.smithi001.stderr: -491> 2024-03-09T14:50:27.567+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.760 INFO:tasks.workunit.client.0.smithi001.stderr: -490> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): get_auth_request con 0x5597a544f890 auth_method 0 2024-03-09T14:50:27.760 INFO:tasks.workunit.client.0.smithi001.stderr: -489> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.760 INFO:tasks.workunit.client.0.smithi001.stderr: -488> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.760 INFO:tasks.workunit.client.0.smithi001.stderr: -487> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.760 INFO:tasks.workunit.client.0.smithi001.stderr: -486> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): handle_auth_done global_id 5975 payload 274 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr: -485> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient: _finish_hunting 0 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr: -484> 2024-03-09T14:50:27.571+0000 7f92a2c41640 1 monclient: found mon.noname-c 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr: -483> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr: -482> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr: -481> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr: -480> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: dump: 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.761 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr: -479> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: _finish_auth 0 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr: -478> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: _check_auth_tickets 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr: -477> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr: -476> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr: -475> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr: -474> 2024-03-09T14:50:27.571+0000 7f9190e09640 10 monclient: dump: 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.762 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr: -473> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 monclient: authenticate success, global_id 5975 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr: -472> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: get_monmap_and_config success 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr: -471> 2024-03-09T14:50:27.571+0000 7f92a5129980 4 set_mon_vals no callback set 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr: -470> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: shutdown 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr: -469> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 asok(0x5597a5388bb0) unregister_commands rotate-key 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr: -468> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: build_initial_monmap 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr: -467> 2024-03-09T14:50:27.571+0000 7f92a5129980 1 build_initial for_mkfs: 0 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr: -466> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: monmap: 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:50:27.763 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr: -465> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding auth protocol: cephx 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr: -464> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding auth protocol: cephx 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr: -463> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding auth protocol: cephx 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr: -462> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: secure 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr: -461> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: crc 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr: -460> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: secure 2024-03-09T14:50:27.764 INFO:tasks.workunit.client.0.smithi001.stderr: -459> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: crc 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -458> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: secure 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -457> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: crc 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -456> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: crc 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -455> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: secure 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -454> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: crc 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -453> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: secure 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -452> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: crc 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -451> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a55c8360) adding con mode: secure 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -450> 2024-03-09T14:50:27.571+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -449> 2024-03-09T14:50:27.571+0000 7f92a5129980 1 librados: starting msgr at 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -448> 2024-03-09T14:50:27.571+0000 7f92a5129980 1 librados: starting objecter 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -447> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command objecter_requests hook 0x5597a5363670 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -446> 2024-03-09T14:50:27.571+0000 7f92a5129980 1 librados: setting wanted keys 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -445> 2024-03-09T14:50:27.571+0000 7f92a5129980 1 librados: calling monclient init 2024-03-09T14:50:27.765 INFO:tasks.workunit.client.0.smithi001.stderr: -444> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: init 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -443> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding auth protocol: cephx 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -442> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding auth protocol: cephx 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -441> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding auth protocol: cephx 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -440> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: secure 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -439> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: crc 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -438> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: secure 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -437> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: crc 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -436> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: secure 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -435> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: crc 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -434> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: crc 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -433> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: secure 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -432> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: crc 2024-03-09T14:50:27.766 INFO:tasks.workunit.client.0.smithi001.stderr: -431> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: secure 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -430> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: crc 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -429> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 AuthRegistry(0x5597a5597100) adding con mode: secure 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -428> 2024-03-09T14:50:27.571+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -427> 2024-03-09T14:50:27.571+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -426> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command rotate-key hook 0x5597a5597248 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -425> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: _reopen_session rank -1 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -424> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: _add_conns ranks=[2,1,0] 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -423> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-c con 0x5597a5423840 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -422> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-b con 0x5597a5423c20 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -421> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-a con 0x5597a544f890 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -420> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -419> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -418> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.767 INFO:tasks.workunit.client.0.smithi001.stderr: -417> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): _renew_subs 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -416> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient(hunting): authenticate will time out at 2321.318576s 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -415> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): get_auth_request con 0x5597a5423840 auth_method 0 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -414> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -413> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -412> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -411> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): get_auth_request con 0x5597a544f890 auth_method 0 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -410> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -409> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -408> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -407> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -406> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -405> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -404> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.768 INFO:tasks.workunit.client.0.smithi001.stderr: -403> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr: -402> 2024-03-09T14:50:27.571+0000 7f92a3442640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr: -401> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient(hunting): handle_auth_done global_id 5978 payload 824 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr: -400> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient: _finish_hunting 0 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr: -399> 2024-03-09T14:50:27.571+0000 7f92a2c41640 1 monclient: found mon.noname-c 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr: -398> 2024-03-09T14:50:27.571+0000 7f92a2c41640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr: -397> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr: -396> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr: -395> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: dump: 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.769 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr: -394> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: _finish_auth 0 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr: -393> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: _check_auth_tickets 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr: -392> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr: -391> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr: -390> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr: -389> 2024-03-09T14:50:27.571+0000 7f918f606640 10 monclient: dump: 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.770 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr: -388> 2024-03-09T14:50:27.571+0000 7f9190e09640 4 set_mon_vals no callback set 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr: -387> 2024-03-09T14:50:27.571+0000 7f92a5129980 5 monclient: authenticate success, global_id 5978 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr: -386> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: _renew_subs 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr: -385> 2024-03-09T14:50:27.571+0000 7f92a5129980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr: -384> 2024-03-09T14:50:27.575+0000 7f92a5129980 10 monclient: _renew_subs 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr: -383> 2024-03-09T14:50:27.575+0000 7f92a5129980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr: -382> 2024-03-09T14:50:27.575+0000 7f92a5129980 1 librados: init done 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr: -381> 2024-03-09T14:50:27.575+0000 7f918f606640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr: -380> 2024-03-09T14:50:27.575+0000 7f918f606640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:50:27.771 INFO:tasks.workunit.client.0.smithi001.stderr: -379> 2024-03-09T14:50:27.575+0000 7f918f606640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr: -378> 2024-03-09T14:50:27.575+0000 7f92a2440640 10 monclient: get_auth_request con 0x7f9188066040 auth_method 0 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr: -377> 2024-03-09T14:50:27.575+0000 7f92a3442640 10 monclient: get_auth_request con 0x5597a55a0a00 auth_method 0 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr: -376> 2024-03-09T14:50:27.575+0000 7f92a2c41640 10 monclient: get_auth_request con 0x5597a55e1cb0 auth_method 0 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr: -375> 2024-03-09T14:50:27.579+0000 7f92a2440640 10 monclient: get_auth_request con 0x5597a55e6d40 auth_method 0 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr: -374> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient: build_initial_monmap 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr: -373> 2024-03-09T14:50:27.579+0000 7f92a5129980 1 build_initial for_mkfs: 0 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr: -372> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient: monmap: 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:50:27.772 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: -371> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding auth protocol: cephx 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: -370> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding auth protocol: cephx 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: -369> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding auth protocol: cephx 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: -368> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: secure 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: -367> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: crc 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: -366> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: secure 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: -365> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: crc 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: -364> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: secure 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: -363> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: crc 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: -362> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: crc 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: -361> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: secure 2024-03-09T14:50:27.773 INFO:tasks.workunit.client.0.smithi001.stderr: -360> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: crc 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -359> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: secure 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -358> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: crc 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -357> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55eab50) adding con mode: secure 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -356> 2024-03-09T14:50:27.579+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -355> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command objecter_requests cmddesc objecter_requests hook 0x5597a55f0300 EEXIST 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -354> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient: init 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -353> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding auth protocol: cephx 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -352> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding auth protocol: cephx 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -351> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding auth protocol: cephx 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -350> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: secure 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -349> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: crc 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -348> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: secure 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -347> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: crc 2024-03-09T14:50:27.774 INFO:tasks.workunit.client.0.smithi001.stderr: -346> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: secure 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -345> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: crc 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -344> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: crc 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -343> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: secure 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -342> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: crc 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -341> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: secure 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -340> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: crc 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -339> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 AuthRegistry(0x5597a55e9b38) adding con mode: secure 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -338> 2024-03-09T14:50:27.579+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -337> 2024-03-09T14:50:27.579+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -336> 2024-03-09T14:50:27.579+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command rotate-key cmddesc rotate-key hook 0x5597a55e9c80 EEXIST 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -335> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient: _reopen_session rank -1 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -334> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient: _add_conns ranks=[0,1,2] 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -333> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-a con 0x5597a5614f60 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:50:27.775 INFO:tasks.workunit.client.0.smithi001.stderr: -332> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-b con 0x5597a5615340 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -331> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-c con 0x5597a5610500 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -330> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -329> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -328> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -327> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): _renew_subs 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -326> 2024-03-09T14:50:27.579+0000 7f92a5129980 10 monclient(hunting): authenticate will time out at 2321.327879s 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -325> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): get_auth_request con 0x5597a5610500 auth_method 0 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -324> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -323> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -322> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -321> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): get_auth_request con 0x5597a5614f60 auth_method 0 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -320> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -319> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -318> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.776 INFO:tasks.workunit.client.0.smithi001.stderr: -317> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -316> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -315> 2024-03-09T14:50:27.583+0000 7f92a3442640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -314> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -313> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -312> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -311> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): get_auth_request con 0x5597a5615340 auth_method 0 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -310> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -309> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -308> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -307> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_done global_id 6351 payload 824 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -306> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient: _finish_hunting 0 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -305> 2024-03-09T14:50:27.583+0000 7f92a2c41640 1 monclient: found mon.noname-a 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -304> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -303> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.777 INFO:tasks.workunit.client.0.smithi001.stderr: -302> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr: -301> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: dump: 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr: -300> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: _finish_auth 0 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr: -299> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: _check_auth_tickets 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr: -298> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:50:27.778 INFO:tasks.workunit.client.0.smithi001.stderr: -297> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr: -296> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr: -295> 2024-03-09T14:50:27.583+0000 7f918ce01640 10 monclient: dump: 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr: -294> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 monclient: authenticate success, global_id 6351 2024-03-09T14:50:27.779 INFO:tasks.workunit.client.0.smithi001.stderr: -293> 2024-03-09T14:50:27.583+0000 7f92a3c43640 4 set_mon_vals no callback set 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr: -292> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: _renew_subs 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr: -291> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr: -290> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: _renew_subs 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr: -289> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr: -288> 2024-03-09T14:50:27.583+0000 7f918ce01640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr: -287> 2024-03-09T14:50:27.583+0000 7f918ce01640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr: -286> 2024-03-09T14:50:27.583+0000 7f918ce01640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr: -285> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: get_monmap_and_config 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr: -284> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: build_initial_monmap 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr: -283> 2024-03-09T14:50:27.583+0000 7f92a5129980 1 build_initial for_mkfs: 0 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr: -282> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: monmap: 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:50:27.780 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr: -281> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding auth protocol: cephx 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr: -280> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding auth protocol: cephx 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr: -279> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding auth protocol: cephx 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr: -278> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: secure 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr: -277> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: crc 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr: -276> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: secure 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr: -275> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: crc 2024-03-09T14:50:27.781 INFO:tasks.workunit.client.0.smithi001.stderr: -274> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: secure 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -273> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: crc 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -272> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: crc 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -271> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: secure 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -270> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: crc 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -269> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: secure 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -268> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: crc 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -267> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x5597a5632ab0) adding con mode: secure 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -266> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient: get_auth_request con 0x7f917c065f80 auth_method 0 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -265> 2024-03-09T14:50:27.583+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -264> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: init 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -263> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding auth protocol: cephx 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -262> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding auth protocol: cephx 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -261> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding auth protocol: cephx 2024-03-09T14:50:27.782 INFO:tasks.workunit.client.0.smithi001.stderr: -260> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: secure 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -259> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: crc 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -258> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: secure 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -257> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: crc 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -256> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: secure 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -255> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: crc 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -254> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: crc 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -253> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: secure 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -252> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: crc 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -251> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: secure 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -250> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: crc 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -249> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 AuthRegistry(0x7ffd35927bf0) adding con mode: secure 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -248> 2024-03-09T14:50:27.583+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -247> 2024-03-09T14:50:27.583+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -246> 2024-03-09T14:50:27.583+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command rotate-key cmddesc rotate-key hook 0x7ffd35927d38 EEXIST 2024-03-09T14:50:27.783 INFO:tasks.workunit.client.0.smithi001.stderr: -245> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: _reopen_session rank -1 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -244> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient: _add_conns ranks=[2,0,1] 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -243> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-c con 0x5597a5660f80 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -242> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-a con 0x5597a5661360 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -241> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-b con 0x5597a566c270 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -240> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -239> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -238> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -237> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): _renew_subs 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -236> 2024-03-09T14:50:27.583+0000 7f92a5129980 10 monclient(hunting): authenticate will time out at 2321.331663s 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -235> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): get_auth_request con 0x5597a5661360 auth_method 0 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -234> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -233> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -232> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -231> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): get_auth_request con 0x5597a5660f80 auth_method 0 2024-03-09T14:50:27.784 INFO:tasks.workunit.client.0.smithi001.stderr: -230> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -229> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -228> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -227> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -226> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -225> 2024-03-09T14:50:27.583+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -224> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -223> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -222> 2024-03-09T14:50:27.583+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -221> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): handle_auth_done global_id 5984 payload 274 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -220> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient: _finish_hunting 0 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -219> 2024-03-09T14:50:27.587+0000 7f92a2c41640 1 monclient: found mon.noname-c 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -218> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -217> 2024-03-09T14:50:27.587+0000 7f92a3442640 10 monclient: get_auth_request con 0x5597a566c270 auth_method 0 2024-03-09T14:50:27.785 INFO:tasks.workunit.client.0.smithi001.stderr: -216> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr: -215> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr: -214> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: dump: 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr: -213> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: _finish_auth 0 2024-03-09T14:50:27.786 INFO:tasks.workunit.client.0.smithi001.stderr: -212> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: _check_auth_tickets 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr: -211> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 monclient: authenticate success, global_id 5984 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr: -210> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr: -209> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr: -208> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr: -207> 2024-03-09T14:50:27.587+0000 7f91867fc640 10 monclient: dump: 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.787 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr: -206> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: get_monmap_and_config success 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr: -205> 2024-03-09T14:50:27.587+0000 7f92a5129980 4 set_mon_vals no callback set 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr: -204> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: shutdown 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr: -203> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: build_initial_monmap 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr: -202> 2024-03-09T14:50:27.587+0000 7f92a5129980 1 build_initial for_mkfs: 0 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr: -201> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: monmap: 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:50:27.788 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -200> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding auth protocol: cephx 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -199> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding auth protocol: cephx 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -198> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding auth protocol: cephx 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -197> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: secure 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -196> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: crc 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -195> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: secure 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -194> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: crc 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -193> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: secure 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -192> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: crc 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -191> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: crc 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -190> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: secure 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -189> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: crc 2024-03-09T14:50:27.789 INFO:tasks.workunit.client.0.smithi001.stderr: -188> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: secure 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -187> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: crc 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -186> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5636aa0) adding con mode: secure 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -185> 2024-03-09T14:50:27.587+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -184> 2024-03-09T14:50:27.587+0000 7f92a5129980 1 librados: starting msgr at 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -183> 2024-03-09T14:50:27.587+0000 7f92a5129980 1 librados: starting objecter 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -182> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command objecter_requests cmddesc objecter_requests hook 0x5597a565d260 EEXIST 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -181> 2024-03-09T14:50:27.587+0000 7f92a5129980 1 librados: setting wanted keys 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -180> 2024-03-09T14:50:27.587+0000 7f92a5129980 1 librados: calling monclient init 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -179> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: init 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -178> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding auth protocol: cephx 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -177> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding auth protocol: cephx 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -176> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding auth protocol: cephx 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -175> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: secure 2024-03-09T14:50:27.790 INFO:tasks.workunit.client.0.smithi001.stderr: -174> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: crc 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -173> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: secure 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -172> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: crc 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -171> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: secure 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -170> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: crc 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -169> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: crc 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -168> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: secure 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -167> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: crc 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -166> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: secure 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -165> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: crc 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -164> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 AuthRegistry(0x5597a5631c80) adding con mode: secure 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -163> 2024-03-09T14:50:27.587+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -162> 2024-03-09T14:50:27.587+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -161> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command rotate-key cmddesc rotate-key hook 0x5597a5631dc8 EEXIST 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -160> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: _reopen_session rank -1 2024-03-09T14:50:27.791 INFO:tasks.workunit.client.0.smithi001.stderr: -159> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: _add_conns ranks=[2,0,1] 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -158> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-c con 0x5597a5660f80 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -157> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-a con 0x5597a566c270 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -156> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): picked mon.noname-b con 0x5597a565d850 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -155> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -154> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -153> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): start opening mon connection 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -152> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): _renew_subs 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -151> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient(hunting): authenticate will time out at 2321.334229s 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -150> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): get_auth_request con 0x5597a5660f80 auth_method 0 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -149> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -148> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -147> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -146> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): get_auth_request con 0x5597a566c270 auth_method 0 2024-03-09T14:50:27.792 INFO:tasks.workunit.client.0.smithi001.stderr: -145> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -144> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -143> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -142> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -141> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -140> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -139> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -138> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -137> 2024-03-09T14:50:27.587+0000 7f92a2440640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -136> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient(hunting): handle_auth_done global_id 5987 payload 824 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -135> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient: _finish_hunting 0 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -134> 2024-03-09T14:50:27.587+0000 7f92a2c41640 1 monclient: found mon.noname-c 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -133> 2024-03-09T14:50:27.587+0000 7f92a2c41640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -132> 2024-03-09T14:50:27.587+0000 7f92a3442640 10 monclient: get_auth_request con 0x5597a565d850 auth_method 0 2024-03-09T14:50:27.793 INFO:tasks.workunit.client.0.smithi001.stderr: -131> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr: -130> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr: -129> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: dump: 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr: -128> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: _finish_auth 0 2024-03-09T14:50:27.794 INFO:tasks.workunit.client.0.smithi001.stderr: -127> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: _check_auth_tickets 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr: -126> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr: -125> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr: -124> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr: -123> 2024-03-09T14:50:27.587+0000 7f9184ff9640 10 monclient: dump: 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.795 INFO:tasks.workunit.client.0.smithi001.stderr: -122> 2024-03-09T14:50:27.587+0000 7f91867fc640 4 set_mon_vals no callback set 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -121> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 monclient: authenticate success, global_id 5987 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -120> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: _renew_subs 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -119> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -118> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: _renew_subs 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -117> 2024-03-09T14:50:27.587+0000 7f92a5129980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -116> 2024-03-09T14:50:27.587+0000 7f92a5129980 1 librados: init done 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -115> 2024-03-09T14:50:27.587+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command cr dump hook 0x5597a565ae88 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -114> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -113> 2024-03-09T14:50:27.591+0000 7f9184ff9640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -112> 2024-03-09T14:50:27.591+0000 7f9184ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -111> 2024-03-09T14:50:27.591+0000 7f9184ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -110> 2024-03-09T14:50:27.591+0000 7f92a3442640 10 monclient: get_auth_request con 0x5597a56801e0 auth_method 0 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -109> 2024-03-09T14:50:27.591+0000 7f92a2440640 10 monclient: get_auth_request con 0x7f916c066040 auth_method 0 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -108> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.796 INFO:tasks.workunit.client.0.smithi001.stderr: -107> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 realm 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -106> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -105> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -104> 2024-03-09T14:50:27.591+0000 7f92a5129980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -103> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -102> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -101> 2024-03-09T14:50:27.591+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -100> 2024-03-09T14:50:27.591+0000 7f92a2c41640 10 monclient: get_auth_request con 0x5597a5684460 auth_method 0 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -99> 2024-03-09T14:50:27.595+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -98> 2024-03-09T14:50:27.595+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -97> 2024-03-09T14:50:27.595+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -96> 2024-03-09T14:50:27.595+0000 7f92a5129980 20 searching for the correct realm 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -95> 2024-03-09T14:50:27.595+0000 7f92a2440640 10 monclient: get_auth_request con 0x5597a56896e0 auth_method 0 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -94> 2024-03-09T14:50:27.595+0000 7f92a3442640 10 monclient: get_auth_request con 0x7f9298008040 auth_method 0 2024-03-09T14:50:27.797 INFO:tasks.workunit.client.0.smithi001.stderr: -93> 2024-03-09T14:50:27.599+0000 7f92a2c41640 10 monclient: get_auth_request con 0x7f929c0c7040 auth_method 0 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -92> 2024-03-09T14:50:27.599+0000 7f92a2440640 10 monclient: get_auth_request con 0x7f9294061060 auth_method 0 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -91> 2024-03-09T14:50:27.603+0000 7f92a3442640 0 -- 172.21.15.1:0/1929550071 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x5597a56801e0 msgr2=0x5597a5682660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -90> 2024-03-09T14:50:27.603+0000 7f9184ff9640 1 client.5987.objecter ms_handle_reset 0x5597a56801e0 session 0x5597a567fb60 osd.1 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -89> 2024-03-09T14:50:27.603+0000 7f9184ff9640 10 monclient: _renew_subs 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -88> 2024-03-09T14:50:27.603+0000 7f9184ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -87> 2024-03-09T14:50:27.603+0000 7f92a3442640 10 monclient: get_auth_request con 0x7f916c075e60 auth_method 0 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -86> 2024-03-09T14:50:27.603+0000 7f92a3442640 10 monclient: get_auth_request con 0x7f929804e960 auth_method 0 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -85> 2024-03-09T14:50:27.607+0000 7f92a2c41640 10 monclient: get_auth_request con 0x7f929804d3e0 auth_method 0 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -84> 2024-03-09T14:50:27.607+0000 7f92a2c41640 0 -- 172.21.15.1:0/1929550071 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f929804d3e0 msgr2=0x7f929804d860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -83> 2024-03-09T14:50:27.607+0000 7f9184ff9640 1 client.5987.objecter ms_handle_reset 0x7f929804d3e0 session 0x7f929803ee10 osd.0 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -82> 2024-03-09T14:50:27.607+0000 7f92a2c41640 10 monclient: get_auth_request con 0x5597a56801e0 auth_method 0 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -81> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -80> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:27.798 INFO:tasks.workunit.client.0.smithi001.stderr: -79> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -78> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -77> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -76> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -75> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -74> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -73> 2024-03-09T14:50:27.615+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -72> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -71> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -70> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -69> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 zone default found 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -68> 2024-03-09T14:50:27.619+0000 7f92a5129980 4 Realm: () 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -67> 2024-03-09T14:50:27.619+0000 7f92a5129980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -66> 2024-03-09T14:50:27.619+0000 7f92a5129980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:27.799 INFO:tasks.workunit.client.0.smithi001.stderr: -65> 2024-03-09T14:50:27.619+0000 7f92a5129980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -64> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 zonegroup default 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -63> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -62> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -61> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -60> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -59> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -58> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -57> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 started sync module instance, tier type = 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -56> 2024-03-09T14:50:27.619+0000 7f92a5129980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -55> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=0 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -54> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=1 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -53> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=2 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -52> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=3 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -51> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=4 2024-03-09T14:50:27.800 INFO:tasks.workunit.client.0.smithi001.stderr: -50> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=5 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -49> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=6 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -48> 2024-03-09T14:50:27.635+0000 7f92a5129980 20 add_watcher() i=7 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -47> 2024-03-09T14:50:27.635+0000 7f92a5129980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -46> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding auth protocol: cephx 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -45> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding auth protocol: cephx 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -44> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding auth protocol: cephx 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -43> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: secure 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -42> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: crc 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -41> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: secure 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -40> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: crc 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -39> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: secure 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -38> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: crc 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -37> 2024-03-09T14:50:27.639+0000 7f91597e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -36> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: crc 2024-03-09T14:50:27.801 INFO:tasks.workunit.client.0.smithi001.stderr: -35> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: secure 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -34> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: crc 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -33> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: secure 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -32> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: crc 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -31> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 AuthRegistry(0x7ffd35928080) adding con mode: secure 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -30> 2024-03-09T14:50:27.639+0000 7f92a5129980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -29> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -28> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -27> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command cache list hook 0x5597a565cba0 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -26> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command cache inspect hook 0x5597a565cba0 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -25> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command cache erase hook 0x5597a565cba0 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -24> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command cache zap hook 0x5597a565cba0 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -23> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 note: GC not initialized 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -22> 2024-03-09T14:50:27.639+0000 7f90d3fff640 20 reqs_thread_entry: start 2024-03-09T14:50:27.802 INFO:tasks.workunit.client.0.smithi001.stderr: -21> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command sync trace show hook 0x7f929c0047f0 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -20> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command sync trace history hook 0x7f929c0047f0 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -19> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command sync trace active hook 0x7f929c0047f0 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -18> 2024-03-09T14:50:27.639+0000 7f92a5129980 5 asok(0x5597a5388bb0) register_command sync trace active_short hook 0x7f929c0047f0 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -17> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -16> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 Filter name: none 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -15> 2024-03-09T14:50:27.639+0000 7f90d1ffb640 20 reqs_thread_entry: start 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -14> 2024-03-09T14:50:27.639+0000 7f92a5129980 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -13> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -12> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -11> 2024-03-09T14:50:27.639+0000 7f92a5129980 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -10> 2024-03-09T14:50:27.639+0000 7f92a5129980 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -9> 2024-03-09T14:50:27.639+0000 7f92a5129980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 : miss 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -8> 2024-03-09T14:50:27.639+0000 7f92a5129980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -7> 2024-03-09T14:50:27.643+0000 7f92a5129980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:27.803 INFO:tasks.workunit.client.0.smithi001.stderr: -6> 2024-03-09T14:50:27.643+0000 7f92a5129980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 info.flags=0x17 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: -5> 2024-03-09T14:50:27.643+0000 7f92a5129980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 to cache LRU end 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: -4> 2024-03-09T14:50:27.643+0000 7f92a5129980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: -3> 2024-03-09T14:50:27.643+0000 7f92a5129980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: -2> 2024-03-09T14:50:27.643+0000 7f92a5129980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: -1> 2024-03-09T14:50:27.647+0000 7f92a5129980 -1 ./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f92a5129980 time 2024-03-09T14:50:27.651783+0000 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f92a89fed7b] 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f92a89fef2f] 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5597a42f1c51] 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: 4: radosgw-admin(+0x8d9bf2) [0x5597a4305bf2] 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5597a431222b] 2024-03-09T14:50:27.804 INFO:tasks.workunit.client.0.smithi001.stderr: 6: main() 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 7: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f92a7869d90] 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 8: __libc_start_main() 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 9: _start() 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 0> 2024-03-09T14:50:27.651+0000 7f92a5129980 -1 *** Caught signal (Aborted) ** 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f92a5129980 thread_name:radosgw-admin 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f92a7882520] 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f92a89fedd5] 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f92a89fef2f] 2024-03-09T14:50:27.805 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5597a42f1c51] 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: 8: radosgw-admin(+0x8d9bf2) [0x5597a4305bf2] 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5597a431222b] 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: 10: main() 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f92a7869d90] 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: 12: __libc_start_main() 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: 13: _start() 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr:--- logging levels --- 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 none 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 lockdep 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 context 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 crush 2024-03-09T14:50:27.806 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_balancer 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_locker 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log_expire 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_migrator 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 buffer 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 timer 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 filer 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 striper 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 objecter 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rados 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_mirror 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_replay 2024-03-09T14:50:27.807 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_pwl 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 journaler 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objectcacher 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 immutable_obj_cache 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 client 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 osd 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 optracker 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objclass 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 filestore 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 journal 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 ms 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mon 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 0/10 monc 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 paxos 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 tp 2024-03-09T14:50:27.808 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 auth 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 crypto 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 finisher 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 reserver 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 heartbeatmap 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 perfcounter 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 20/20 rgw 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_sync 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_datacache 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_access 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_dbstore 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_flight 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_lifecycle 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 javaclient 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 asok 2024-03-09T14:50:27.809 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 throttle 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 refs 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 compressor 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluestore 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluefs 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 bdev 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 kstore 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 4/ 5 rocksdb 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 fuse 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 2/ 5 mgr 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mgrc 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 dpdk 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 eventtrace 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 prioritycache 2024-03-09T14:50:27.810 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 test 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephfs_mirror 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephsqlite 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 crimson_interrupt 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_onode 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_odata 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_omap 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_tm 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_t 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cleaner 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_epm 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_lba 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_fixedkv_tree 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cache 2024-03-09T14:50:27.811 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_journal 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_device 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_backref 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 alienstore 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mclock 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cyanstore 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 ceph_exporter 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 memstore 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 trace 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: -2/-2 (syslog threshold) 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 99/99 (stderr threshold) 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr:--- pthread ID / name mapping for recent threads --- 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 7f90d1ffb640 / http_manager 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 7f90d3fff640 / http_manager 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 7f91597e2640 / rgw_dt_lg_renew 2024-03-09T14:50:27.812 INFO:tasks.workunit.client.0.smithi001.stderr: 7f9184ff9640 / ms_dispatch 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr: 7f91867fc640 / io_context_pool 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr: 7f918ce01640 / ms_dispatch 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr: 7f918f606640 / ms_dispatch 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr: 7f9190e09640 / io_context_pool 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr: 7f92a2440640 / msgr-worker-2 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr: 7f92a2c41640 / msgr-worker-1 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr: 7f92a3442640 / msgr-worker-0 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr: 7f92a3c43640 / io_context_pool 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr: 7f92a5129980 / radosgw-admin 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr: max_recent 500 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr: max_new 1000 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr: log_file /var/lib/ceph/crash/2024-03-09T14:50:27.654056Z_f68a0c25-a939-410f-ac65-ec5157af9daa/log 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr:--- end dump of recent events --- 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr:Aborted 2024-03-09T14:50:27.813 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket abort-at-set-target-layout 2024-03-09T14:50:27.819 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:27.819 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:27.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.727+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.727+0000 7f15f4889980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.727+0000 7f15f4889980 20 realm 2024-03-09T14:50:27.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.727+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.819 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.727+0000 7f15f4889980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.727+0000 7f15f4889980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.727+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.727+0000 7f15f4889980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.727+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.731+0000 7f15f4889980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.731+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.731+0000 7f15f4889980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.731+0000 7f15f4889980 20 searching for the correct realm 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.739+0000 7f15f1ba0640 0 -- 172.21.15.1:0/2483061524 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x55c55623fbf0 msgr2=0x55c556241fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.743+0000 7f15f23a1640 0 -- 172.21.15.1:0/2483061524 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f15e8022430 msgr2=0x7f15e8022870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:27.820 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 zone default found 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 4 Realm: () 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:27.821 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 zonegroup default 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.751+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.755+0000 7f15f4889980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.755+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.755+0000 7f15f4889980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.755+0000 7f15f4889980 20 started sync module instance, tier type = 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.755+0000 7f15f4889980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.771+0000 7f15f4889980 20 add_watcher() i=0 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.771+0000 7f15f4889980 20 add_watcher() i=1 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.771+0000 7f15f4889980 20 add_watcher() i=2 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.771+0000 7f15f4889980 20 add_watcher() i=3 2024-03-09T14:50:27.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.771+0000 7f15f4889980 20 add_watcher() i=4 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.771+0000 7f15f4889980 20 add_watcher() i=5 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.771+0000 7f15f4889980 20 add_watcher() i=6 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.771+0000 7f15f4889980 20 add_watcher() i=7 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.771+0000 7f15f4889980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f14a8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f15f4889980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f15f4889980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f15f4889980 5 note: GC not initialized 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f14237fe640 20 reqs_thread_entry: start 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f15f4889980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f15f4889980 20 Filter name: none 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f14217fa640 20 reqs_thread_entry: start 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f15f4889980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:27.823 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f15f4889980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f15f4889980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.775+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.779+0000 7f15f4889980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.779+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.779+0000 7f15f4889980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.779+0000 7f15f4889980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.779+0000 7f15f4889980 20 rados->read ofs=0 len=0 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.779+0000 7f15f4889980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.783+0000 7f14a8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.787+0000 7f15f4889980 20 remove_watcher() i=0 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.787+0000 7f15f4889980 2 removed watcher, disabling cache 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.787+0000 7f15f4889980 20 remove_watcher() i=1 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.791+0000 7f15f4889980 20 remove_watcher() i=2 2024-03-09T14:50:27.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.791+0000 7f15f4889980 20 remove_watcher() i=3 2024-03-09T14:50:27.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.795+0000 7f15f4889980 20 remove_watcher() i=4 2024-03-09T14:50:27.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.795+0000 7f15f4889980 20 remove_watcher() i=5 2024-03-09T14:50:27.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.799+0000 7f15f4889980 20 remove_watcher() i=6 2024-03-09T14:50:27.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.799+0000 7f15f4889980 20 remove_watcher() i=7 2024-03-09T14:50:27.825 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.825 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:50:27.825 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: abort-at-set-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:50:27.825 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:27.825 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:27.825 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:27.855 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:27.855 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:27.918 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket abort-at-set-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2024-03-09T14:50:27.955 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:27.955 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:28.000 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.995+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.001 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.999+0000 7f6430e24980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:28.001 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.999+0000 7f6430e24980 20 realm 2024-03-09T14:50:28.001 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.999+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.999+0000 7f6430e24980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:28.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.999+0000 7f6430e24980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:28.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.999+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.999+0000 7f6430e24980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:28.002 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.999+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.999+0000 7f6430e24980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:28.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:27.999+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.003+0000 7f6430e24980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:28.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.003+0000 7f6430e24980 20 searching for the correct realm 2024-03-09T14:50:28.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.007+0000 7f642e93c640 0 -- 172.21.15.1:0/2794252017 >> [v2:172.21.15.1:6824/1666056338,v1:172.21.15.1:6825/1666056338] conn(0x7f64280b8040 msgr2=0x7f64280c6400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:28.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.011+0000 7f642f13d640 0 -- 172.21.15.1:0/2794252017 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f6424026040 msgr2=0x7f642408d050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:28.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:28.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:28.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:28.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:28.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:28.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:28.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:28.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:28.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:28.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 zone default found 2024-03-09T14:50:28.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 4 Realm: () 2024-03-09T14:50:28.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:28.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:28.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:28.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 zonegroup default 2024-03-09T14:50:28.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:28.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.023+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.027+0000 7f6430e24980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:28.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.027+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.027+0000 7f6430e24980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:28.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.027+0000 7f6430e24980 20 started sync module instance, tier type = 2024-03-09T14:50:28.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.027+0000 7f6430e24980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:28.044 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.043+0000 7f6430e24980 20 add_watcher() i=0 2024-03-09T14:50:28.044 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.043+0000 7f6430e24980 20 add_watcher() i=1 2024-03-09T14:50:28.044 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.043+0000 7f6430e24980 20 add_watcher() i=2 2024-03-09T14:50:28.044 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.043+0000 7f6430e24980 20 add_watcher() i=3 2024-03-09T14:50:28.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.043+0000 7f6430e24980 20 add_watcher() i=4 2024-03-09T14:50:28.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.043+0000 7f6430e24980 20 add_watcher() i=5 2024-03-09T14:50:28.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.043+0000 7f6430e24980 20 add_watcher() i=6 2024-03-09T14:50:28.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.043+0000 7f6430e24980 20 add_watcher() i=7 2024-03-09T14:50:28.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.043+0000 7f6430e24980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:28.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.043+0000 7f62e57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:28.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:28.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:28.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 5 note: GC not initialized 2024-03-09T14:50:28.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f625ffff640 20 reqs_thread_entry: start 2024-03-09T14:50:28.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:28.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 20 Filter name: none 2024-03-09T14:50:28.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f625dffb640 20 reqs_thread_entry: start 2024-03-09T14:50:28.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2024-03-09T14:50:28.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:28.050 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2024-03-09T14:50:28.050 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2024-03-09T14:50:28.050 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 : miss 2024-03-09T14:50:28.050 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 20 rados->read ofs=0 len=0 2024-03-09T14:50:28.050 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:28.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 info.flags=0x17 2024-03-09T14:50:28.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 to cache LRU end 2024-03-09T14:50:28.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:50:28.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:28.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.047+0000 7f6430e24980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 2024-03-09T14:50:28.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.051+0000 7f6430e24980 0 INFO: RGWReshardLock::lock found lock on abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 to be held by another RGW process; skipping for now 2024-03-09T14:50:28.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.051+0000 7f62e57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:28.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.055+0000 7f6430e24980 20 remove_watcher() i=0 2024-03-09T14:50:28.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.055+0000 7f6430e24980 2 removed watcher, disabling cache 2024-03-09T14:50:28.061 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.059+0000 7f6430e24980 20 remove_watcher() i=1 2024-03-09T14:50:28.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.059+0000 7f6430e24980 20 remove_watcher() i=2 2024-03-09T14:50:28.065 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.063+0000 7f6430e24980 20 remove_watcher() i=3 2024-03-09T14:50:28.067 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.063+0000 7f6430e24980 20 remove_watcher() i=4 2024-03-09T14:50:28.068 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.067+0000 7f6430e24980 20 remove_watcher() i=5 2024-03-09T14:50:28.069 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.067+0000 7f6430e24980 20 remove_watcher() i=6 2024-03-09T14:50:28.070 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:28.067+0000 7f6430e24980 20 remove_watcher() i=7 2024-03-09T14:50:28.084 INFO:tasks.workunit.client.0.smithi001.stderr:waiting 30 seconds for reshard lock to expire... 2024-03-09T14:50:58.114 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket abort-at-set-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2024-03-09T14:50:58.152 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:58.152 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:58.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.195+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.195+0000 7f1090d7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.195+0000 7f1090d7d980 20 realm 2024-03-09T14:50:58.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.195+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.195+0000 7f1090d7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.195+0000 7f1090d7d980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:58.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.195+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.195+0000 7f1090d7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.195+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.199+0000 7f1090d7d980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:58.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.199+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.199+0000 7f1090d7d980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:58.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.199+0000 7f1090d7d980 20 searching for the correct realm 2024-03-09T14:50:58.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.207+0000 7f108f096640 0 -- 172.21.15.1:0/3251325732 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f1078026040 msgr2=0x7f1078050950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:58.426 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.423+0000 7f108e895640 0 -- 172.21.15.1:0/3251325732 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x55df6fb00dc0 msgr2=0x55df6fb031b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:58.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.431+0000 7f1090d7d980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:58.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.431+0000 7f1090d7d980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:58.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.431+0000 7f1090d7d980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:58.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.431+0000 7f1090d7d980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:58.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.431+0000 7f1090d7d980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:58.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.431+0000 7f1090d7d980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:58.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.431+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.431+0000 7f1090d7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.431+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.431+0000 7f1090d7d980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:58.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.431+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:58.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 20 zone default found 2024-03-09T14:50:58.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 4 Realm: () 2024-03-09T14:50:58.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:58.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:58.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:58.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 20 zonegroup default 2024-03-09T14:50:58.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.437 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 20 started sync module instance, tier type = 2024-03-09T14:50:58.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.435+0000 7f1090d7d980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:58.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.463+0000 7f1090d7d980 20 add_watcher() i=0 2024-03-09T14:50:58.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.463+0000 7f1090d7d980 20 add_watcher() i=1 2024-03-09T14:50:58.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.463+0000 7f1090d7d980 20 add_watcher() i=2 2024-03-09T14:50:58.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.463+0000 7f1090d7d980 20 add_watcher() i=3 2024-03-09T14:50:58.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.463+0000 7f1090d7d980 20 add_watcher() i=4 2024-03-09T14:50:58.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.463+0000 7f1090d7d980 20 add_watcher() i=5 2024-03-09T14:50:58.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.463+0000 7f1090d7d980 20 add_watcher() i=6 2024-03-09T14:50:58.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.463+0000 7f1090d7d980 20 add_watcher() i=7 2024-03-09T14:50:58.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.463+0000 7f1090d7d980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:58.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.467+0000 7f0f45fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:58.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f1090d7d980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:58.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f1090d7d980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:58.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f1090d7d980 5 note: GC not initialized 2024-03-09T14:50:58.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f0f44fe1640 20 reqs_thread_entry: start 2024-03-09T14:50:58.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f1090d7d980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:58.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f1090d7d980 20 Filter name: none 2024-03-09T14:50:58.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f0f3dffb640 20 reqs_thread_entry: start 2024-03-09T14:50:58.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f1090d7d980 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2024-03-09T14:50:58.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f1090d7d980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:58.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f1090d7d980 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2024-03-09T14:50:58.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f1090d7d980 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2024-03-09T14:50:58.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f1090d7d980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 : miss 2024-03-09T14:50:58.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.471+0000 7f1090d7d980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.475+0000 7f1090d7d980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:58.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.475+0000 7f1090d7d980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 info.flags=0x17 2024-03-09T14:50:58.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.475+0000 7f1090d7d980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 to cache LRU end 2024-03-09T14:50:58.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.475+0000 7f1090d7d980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:50:58.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.475+0000 7f1090d7d980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:58.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.475+0000 7f1090d7d980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 2024-03-09T14:50:58.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.483+0000 7f1090d7d980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 info.flags=0x17 2024-03-09T14:50:58.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.483+0000 7f1090d7d980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 to cache LRU end 2024-03-09T14:50:58.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.483+0000 7f1090d7d980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:58.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.483+0000 7f1090d7d980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21, ofs0, ns] 2024-03-09T14:50:58.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.483+0000 7f0f727fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562441 cookie 94418139911728 notifier 6414 bl.length()=917 2024-03-09T14:50:58.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.483+0000 7f0f727fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 info.flags=0x17 2024-03-09T14:50:58.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.483+0000 7f0f727fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 to cache LRU end 2024-03-09T14:50:58.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.483+0000 7f0f727fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:58.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.487+0000 7f1090d7d980 20 handle_bi_update(): bucket=:abort-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.21]): orig_sources= new_sources= 2024-03-09T14:50:58.490 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.487+0000 7f1090d7d980 20 handle_bi_update(): bucket=:abort-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.21]): potential sources added=[] removed=[] 2024-03-09T14:50:58.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.487+0000 7f1090d7d980 20 handle_bi_update(): bucket=:abort-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.21]): orig_dests= new_dests= 2024-03-09T14:50:58.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.487+0000 7f1090d7d980 20 handle_bi_update(): bucket=:abort-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.21]): potential dests added=[] removed=[] 2024-03-09T14:50:58.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.491+0000 7f1090d7d980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.21.1.0 2024-03-09T14:50:58.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.491+0000 7f1090d7d980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.21.1.0 2024-03-09T14:50:58.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.491+0000 7f1090d7d980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.21.1.1 2024-03-09T14:50:58.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.491+0000 7f1090d7d980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.21.1.1 2024-03-09T14:50:58.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.491+0000 7f1090d7d980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.21.0 2024-03-09T14:50:58.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.499+0000 7f1090d7d980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 : hit (requested=0x17, cached=0x17) 2024-03-09T14:50:58.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.499+0000 7f1090d7d980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 2024-03-09T14:50:58.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.503+0000 7f1090d7d980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 info.flags=0x17 2024-03-09T14:50:58.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.503+0000 7f1090d7d980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 to cache LRU end 2024-03-09T14:50:58.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.503+0000 7f1090d7d980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:58.505 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.503+0000 7f1090d7d980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21, ofs0, ns] 2024-03-09T14:50:58.505 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.503+0000 7f0f71ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562442 cookie 94418139911728 notifier 6414 bl.length()=885 2024-03-09T14:50:58.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.503+0000 7f0f71ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 info.flags=0x17 2024-03-09T14:50:58.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.503+0000 7f0f71ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 to cache LRU end 2024-03-09T14:50:58.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.503+0000 7f0f71ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:50:58.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.503+0000 7f1090d7d980 20 handle_bi_update(): bucket=:abort-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.21]): orig_sources= new_sources= 2024-03-09T14:50:58.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.503+0000 7f1090d7d980 20 handle_bi_update(): bucket=:abort-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.21]): potential sources added=[] removed=[] 2024-03-09T14:50:58.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.503+0000 7f1090d7d980 20 handle_bi_update(): bucket=:abort-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.21]): orig_dests= new_dests= 2024-03-09T14:50:58.508 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.503+0000 7f1090d7d980 20 handle_bi_update(): bucket=:abort-at-set-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.21]): potential dests added=[] removed=[] 2024-03-09T14:50:58.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.507+0000 7f1090d7d980 1 execute INFO: reshard of bucket "abort-at-set-target-layout" completed successfully 2024-03-09T14:50:58.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.511+0000 7f0f45fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:58.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.515+0000 7f1090d7d980 20 remove_watcher() i=0 2024-03-09T14:50:58.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.515+0000 7f1090d7d980 2 removed watcher, disabling cache 2024-03-09T14:50:58.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.519+0000 7f1090d7d980 20 remove_watcher() i=1 2024-03-09T14:50:58.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.523+0000 7f1090d7d980 20 remove_watcher() i=2 2024-03-09T14:50:58.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.523+0000 7f1090d7d980 20 remove_watcher() i=3 2024-03-09T14:50:58.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.527+0000 7f1090d7d980 20 remove_watcher() i=4 2024-03-09T14:50:58.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.527+0000 7f1090d7d980 20 remove_watcher() i=5 2024-03-09T14:50:58.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.531+0000 7f1090d7d980 20 remove_watcher() i=6 2024-03-09T14:50:58.535 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.531+0000 7f1090d7d980 20 remove_watcher() i=7 2024-03-09T14:50:58.550 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket abort-at-set-target-layout 2024-03-09T14:50:58.579 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:50:58.579 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:50:58.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.619+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.623+0000 7f472ad75980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.623+0000 7f472ad75980 20 realm 2024-03-09T14:50:58.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.623+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.623+0000 7f472ad75980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.623+0000 7f472ad75980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:50:58.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.623+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.623+0000 7f472ad75980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.623+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.623+0000 7f472ad75980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:58.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.623+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.623+0000 7f472ad75980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:50:58.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.623+0000 7f472ad75980 20 searching for the correct realm 2024-03-09T14:50:58.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.631+0000 7f472908e640 0 -- 172.21.15.1:0/1701340186 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55a2f93392e0 msgr2=0x55a2f933b730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:58.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.639+0000 7f4723fff640 0 -- 172.21.15.1:0/1701340186 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f471c038030 msgr2=0x7f471c075a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:50:58.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.643+0000 7f472ad75980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:50:58.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.643+0000 7f472ad75980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:50:58.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.643+0000 7f472ad75980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:50:58.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.643+0000 7f472ad75980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:50:58.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.643+0000 7f472ad75980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:50:58.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.643+0000 7f472ad75980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:50:58.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.643+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.643+0000 7f472ad75980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.643+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:50:58.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:50:58.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 zone default found 2024-03-09T14:50:58.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 4 Realm: () 2024-03-09T14:50:58.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:50:58.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:50:58.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:50:58.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 zonegroup default 2024-03-09T14:50:58.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 started sync module instance, tier type = 2024-03-09T14:50:58.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.647+0000 7f472ad75980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:50:58.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.663+0000 7f472ad75980 20 add_watcher() i=0 2024-03-09T14:50:58.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.663+0000 7f472ad75980 20 add_watcher() i=1 2024-03-09T14:50:58.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.663+0000 7f472ad75980 20 add_watcher() i=2 2024-03-09T14:50:58.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.663+0000 7f472ad75980 20 add_watcher() i=3 2024-03-09T14:50:58.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.663+0000 7f472ad75980 20 add_watcher() i=4 2024-03-09T14:50:58.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.663+0000 7f472ad75980 20 add_watcher() i=5 2024-03-09T14:50:58.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.663+0000 7f472ad75980 20 add_watcher() i=6 2024-03-09T14:50:58.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.663+0000 7f472ad75980 20 add_watcher() i=7 2024-03-09T14:50:58.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.663+0000 7f472ad75980 2 all 8 watchers are set, enabling cache 2024-03-09T14:50:58.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.667+0000 7f45defe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:58.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.667+0000 7f472ad75980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:50:58.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.667+0000 7f472ad75980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:50:58.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.667+0000 7f472ad75980 5 note: GC not initialized 2024-03-09T14:50:58.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.667+0000 7f45ddfe3640 20 reqs_thread_entry: start 2024-03-09T14:50:58.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.667+0000 7f472ad75980 20 init_complete bucket index max shards: 1 2024-03-09T14:50:58.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.667+0000 7f472ad75980 20 Filter name: none 2024-03-09T14:50:58.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.667+0000 7f454f7fe640 20 reqs_thread_entry: start 2024-03-09T14:50:58.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.667+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.671+0000 7f472ad75980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:58.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.671+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.671+0000 7f472ad75980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:58.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.671+0000 7f472ad75980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:58.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.671+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.671+0000 7f472ad75980 20 rados_obj.operate() r=0 bl.length=194 2024-03-09T14:50:58.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.671+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.671+0000 7f472ad75980 20 rados_obj.operate() r=0 bl.length=408 2024-03-09T14:50:58.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.671+0000 7f472ad75980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:50:58.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.675+0000 7f472ad75980 20 rados->read ofs=0 len=0 2024-03-09T14:50:58.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.675+0000 7f472ad75980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:50:58.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.675+0000 7f45defe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:50:58.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.679+0000 7f472ad75980 20 remove_watcher() i=0 2024-03-09T14:50:58.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.679+0000 7f472ad75980 2 removed watcher, disabling cache 2024-03-09T14:50:58.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.683+0000 7f472ad75980 20 remove_watcher() i=1 2024-03-09T14:50:58.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.683+0000 7f472ad75980 20 remove_watcher() i=2 2024-03-09T14:50:58.689 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.687+0000 7f472ad75980 20 remove_watcher() i=3 2024-03-09T14:50:58.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.687+0000 7f472ad75980 20 remove_watcher() i=4 2024-03-09T14:50:58.692 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.691+0000 7f472ad75980 20 remove_watcher() i=5 2024-03-09T14:50:58.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.691+0000 7f472ad75980 20 remove_watcher() i=6 2024-03-09T14:50:58.695 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:50:58.691+0000 7f472ad75980 20 remove_watcher() i=7 2024-03-09T14:50:58.709 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:58.709 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:50:58.709 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: abort-at-set-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.21 num_objects: 19 size_kb: 1 num_shards: 2 2024-03-09T14:50:58.709 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:58.710 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:58.710 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:58.730 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:58.730 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:59.285 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:59.285 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:59.468 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard bucket with EIO injected at block_writes 2024-03-09T14:50:59.468 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:50:59.471 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:59.471 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:59.591 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:59.591 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:59.680 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:59.680 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:59.781 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:59.781 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:59.880 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:59.881 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:50:59.979 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:50:59.979 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:00.076 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:00.076 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:00.173 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:00.173 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:00.273 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:00.273 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:00.376 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:00.376 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:00.476 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:00.476 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:00.573 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:00.574 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:00.682 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:00.683 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:00.779 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:00.779 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:00.877 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:00.878 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:00.974 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:00.974 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:01.072 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:01.073 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:01.180 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:01.180 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:01.276 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:01.276 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:01.371 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:01.372 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:01.473 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:01.473 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:01.572 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:01.572 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:01.667 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-block-writes 2024-03-09T14:51:01.699 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:01.699 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:01.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.743+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:01.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.743+0000 7f1770b68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:01.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.743+0000 7f1770b68980 20 realm 2024-03-09T14:51:01.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.743+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:01.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.743+0000 7f1770b68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:01.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.743+0000 7f1770b68980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:01.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.743+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:01.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.743+0000 7f1770b68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:01.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.743+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:01.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.747+0000 7f1770b68980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:01.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.747+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:01.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.747+0000 7f1770b68980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:01.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.747+0000 7f1770b68980 20 searching for the correct realm 2024-03-09T14:51:01.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.755+0000 7f176ee81640 0 -- 172.21.15.1:0/3658195510 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f1764008050 msgr2=0x7f17640669e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:01.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.967+0000 7f176ee81640 0 -- 172.21.15.1:0/3658195510 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f1764017030 msgr2=0x7f1764074b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:51:01.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.979+0000 7f1770b68980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:01.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.979+0000 7f1770b68980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:01.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.979+0000 7f1770b68980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:01.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.979+0000 7f1770b68980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:01.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.979+0000 7f1770b68980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:01.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.979+0000 7f1770b68980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:01.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.979+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:01.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.979+0000 7f1770b68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:01.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.979+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:01.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:01.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:01.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:01.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 zone default found 2024-03-09T14:51:01.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 4 Realm: () 2024-03-09T14:51:01.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:01.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:01.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:01.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 zonegroup default 2024-03-09T14:51:01.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:01.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:01.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:01.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:01.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:01.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:01.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 started sync module instance, tier type = 2024-03-09T14:51:01.988 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:01.983+0000 7f1770b68980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:02.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.011+0000 7f1770b68980 20 add_watcher() i=0 2024-03-09T14:51:02.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.011+0000 7f1770b68980 20 add_watcher() i=1 2024-03-09T14:51:02.012 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.011+0000 7f1770b68980 20 add_watcher() i=2 2024-03-09T14:51:02.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.011+0000 7f1770b68980 20 add_watcher() i=3 2024-03-09T14:51:02.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.011+0000 7f1770b68980 20 add_watcher() i=4 2024-03-09T14:51:02.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.011+0000 7f1770b68980 20 add_watcher() i=5 2024-03-09T14:51:02.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.011+0000 7f1770b68980 20 add_watcher() i=6 2024-03-09T14:51:02.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.011+0000 7f1770b68980 20 add_watcher() i=7 2024-03-09T14:51:02.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.011+0000 7f1770b68980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:02.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.015+0000 7f16257e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:02.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f1770b68980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:02.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f1770b68980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:02.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f1770b68980 5 note: GC not initialized 2024-03-09T14:51:02.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f161f7fe640 20 reqs_thread_entry: start 2024-03-09T14:51:02.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f1770b68980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:02.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f1770b68980 20 Filter name: none 2024-03-09T14:51:02.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f161d7fa640 20 reqs_thread_entry: start 2024-03-09T14:51:02.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f1770b68980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:02.022 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f1770b68980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:02.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f1770b68980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:02.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.019+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.023+0000 7f1770b68980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:02.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.023+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.023+0000 7f1770b68980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:02.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.023+0000 7f1770b68980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:02.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.023+0000 7f1770b68980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.023+0000 7f1770b68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.027+0000 7f16257e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:02.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.031+0000 7f1770b68980 20 remove_watcher() i=0 2024-03-09T14:51:02.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.031+0000 7f1770b68980 2 removed watcher, disabling cache 2024-03-09T14:51:02.036 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.031+0000 7f1770b68980 20 remove_watcher() i=1 2024-03-09T14:51:02.038 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.035+0000 7f1770b68980 20 remove_watcher() i=2 2024-03-09T14:51:02.041 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.039+0000 7f1770b68980 20 remove_watcher() i=3 2024-03-09T14:51:02.043 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.039+0000 7f1770b68980 20 remove_watcher() i=4 2024-03-09T14:51:02.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.043+0000 7f1770b68980 20 remove_watcher() i=5 2024-03-09T14:51:02.046 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.043+0000 7f1770b68980 20 remove_watcher() i=6 2024-03-09T14:51:02.048 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.047+0000 7f1770b68980 20 remove_watcher() i=7 2024-03-09T14:51:02.064 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:02.065 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:02.065 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-block-writes id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:51:02.065 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:02.065 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket error-at-block-writes --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-error-at block_writes 2024-03-09T14:51:02.099 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:02.099 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:02.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.143+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.143+0000 7f7408c61980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.147 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.143+0000 7f7408c61980 20 realm 2024-03-09T14:51:02.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.143+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.147+0000 7f7408c61980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.147+0000 7f7408c61980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:02.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.147+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.147+0000 7f7408c61980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.147+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.147+0000 7f7408c61980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:02.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.147+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.147+0000 7f7408c61980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:02.150 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.147+0000 7f7408c61980 20 searching for the correct realm 2024-03-09T14:51:02.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.155+0000 7f7406f7a640 0 -- 172.21.15.1:0/2964220216 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f73f0013030 msgr2=0x7f73f006e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:51:02.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.371+0000 7f7406779640 0 -- 172.21.15.1:0/2964220216 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x55dc243e6fa0 msgr2=0x55dc243e93b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:02.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.379+0000 7f7408c61980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:02.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.379+0000 7f7408c61980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:02.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.379+0000 7f7408c61980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:02.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.379+0000 7f7408c61980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:02.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.379+0000 7f7408c61980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:02.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.379+0000 7f7408c61980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:02.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.379+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.379+0000 7f7408c61980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.379+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.379+0000 7f7408c61980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:02.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.379+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:02.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 20 zone default found 2024-03-09T14:51:02.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 4 Realm: () 2024-03-09T14:51:02.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:02.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:02.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:02.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 20 zonegroup default 2024-03-09T14:51:02.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.387 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 20 started sync module instance, tier type = 2024-03-09T14:51:02.387 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.383+0000 7f7408c61980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:02.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.407+0000 7f7408c61980 20 add_watcher() i=0 2024-03-09T14:51:02.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.407+0000 7f7408c61980 20 add_watcher() i=1 2024-03-09T14:51:02.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.407+0000 7f7408c61980 20 add_watcher() i=2 2024-03-09T14:51:02.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.407+0000 7f7408c61980 20 add_watcher() i=3 2024-03-09T14:51:02.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.407+0000 7f7408c61980 20 add_watcher() i=4 2024-03-09T14:51:02.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.407+0000 7f7408c61980 20 add_watcher() i=5 2024-03-09T14:51:02.410 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.407+0000 7f7408c61980 20 add_watcher() i=6 2024-03-09T14:51:02.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.411+0000 7f7408c61980 20 add_watcher() i=7 2024-03-09T14:51:02.412 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.411+0000 7f7408c61980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:02.416 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f72bd7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:02.417 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f7408c61980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:02.417 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f7408c61980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:02.417 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f7408c61980 5 note: GC not initialized 2024-03-09T14:51:02.418 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f72b77fe640 20 reqs_thread_entry: start 2024-03-09T14:51:02.418 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f7408c61980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:02.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f7408c61980 20 Filter name: none 2024-03-09T14:51:02.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f72b57fa640 20 reqs_thread_entry: start 2024-03-09T14:51:02.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f7408c61980 10 cache get: name=default.rgw.meta+root+error-at-block-writes : miss 2024-03-09T14:51:02.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.419 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f7408c61980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:02.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f7408c61980 10 cache put: name=default.rgw.meta+root+error-at-block-writes info.flags=0x11 2024-03-09T14:51:02.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f7408c61980 10 adding default.rgw.meta+root+error-at-block-writes to cache LRU end 2024-03-09T14:51:02.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f7408c61980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 : miss 2024-03-09T14:51:02.420 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.415+0000 7f7408c61980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.419+0000 7f7408c61980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:02.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.419+0000 7f7408c61980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 info.flags=0x17 2024-03-09T14:51:02.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.419+0000 7f7408c61980 10 adding default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 to cache LRU end 2024-03-09T14:51:02.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.419+0000 7f7408c61980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:02.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.419+0000 7f7408c61980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:02.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.419+0000 7f7408c61980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 2024-03-09T14:51:02.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.423+0000 7f7408c61980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 info.flags=0x17 2024-03-09T14:51:02.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.423+0000 7f7408c61980 10 moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 to cache LRU end 2024-03-09T14:51:02.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.423+0000 7f7408c61980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:02.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.423+0000 7f7408c61980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22, ofs0, ns] 2024-03-09T14:51:02.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.427+0000 7f72e9ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562395 cookie 94403989272896 notifier 6086 bl.length()=907 2024-03-09T14:51:02.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.427+0000 7f72e9ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 info.flags=0x17 2024-03-09T14:51:02.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.427+0000 7f72e9ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 to cache LRU end 2024-03-09T14:51:02.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.427+0000 7f72e9ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:02.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.427+0000 7f7408c61980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): orig_sources= new_sources= 2024-03-09T14:51:02.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.427+0000 7f7408c61980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): potential sources added=[] removed=[] 2024-03-09T14:51:02.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.427+0000 7f7408c61980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): orig_dests= new_dests= 2024-03-09T14:51:02.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.427+0000 7f7408c61980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): potential dests added=[] removed=[] 2024-03-09T14:51:02.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.427+0000 7f7408c61980 -1 Injecting error=-5 at location=block_writes 2024-03-09T14:51:02.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.427+0000 7f7408c61980 0 ERROR: init_reshard failed to pause writes to the current index: (5) Input/output error 2024-03-09T14:51:02.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.431+0000 7f7408c61980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 : hit (requested=0x17, cached=0x17) 2024-03-09T14:51:02.433 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.431+0000 7f7408c61980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 2024-03-09T14:51:02.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.431+0000 7f7408c61980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 info.flags=0x17 2024-03-09T14:51:02.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.431+0000 7f7408c61980 10 moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 to cache LRU end 2024-03-09T14:51:02.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.431+0000 7f7408c61980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:02.435 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.431+0000 7f7408c61980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22, ofs0, ns] 2024-03-09T14:51:02.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.431+0000 7f72e9ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562396 cookie 94403989272896 notifier 6086 bl.length()=875 2024-03-09T14:51:02.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.431+0000 7f72e9ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 info.flags=0x17 2024-03-09T14:51:02.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.431+0000 7f72e9ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 to cache LRU end 2024-03-09T14:51:02.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.431+0000 7f72e9ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:02.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.435+0000 7f7408c61980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): orig_sources= new_sources= 2024-03-09T14:51:02.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.435+0000 7f7408c61980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): potential sources added=[] removed=[] 2024-03-09T14:51:02.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.435+0000 7f7408c61980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): orig_dests= new_dests= 2024-03-09T14:51:02.438 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.435+0000 7f7408c61980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): potential dests added=[] removed=[] 2024-03-09T14:51:02.442 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.439+0000 7f72bd7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:02.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.443+0000 7f7408c61980 20 remove_watcher() i=0 2024-03-09T14:51:02.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.443+0000 7f7408c61980 2 removed watcher, disabling cache 2024-03-09T14:51:02.450 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.447+0000 7f7408c61980 20 remove_watcher() i=1 2024-03-09T14:51:02.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.451+0000 7f7408c61980 20 remove_watcher() i=2 2024-03-09T14:51:02.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.451+0000 7f7408c61980 20 remove_watcher() i=3 2024-03-09T14:51:02.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.455+0000 7f7408c61980 20 remove_watcher() i=4 2024-03-09T14:51:02.459 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.455+0000 7f7408c61980 20 remove_watcher() i=5 2024-03-09T14:51:02.460 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.459+0000 7f7408c61980 20 remove_watcher() i=6 2024-03-09T14:51:02.462 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.459+0000 7f7408c61980 20 remove_watcher() i=7 2024-03-09T14:51:02.479 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-block-writes 2024-03-09T14:51:02.507 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:02.507 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:02.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.547+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.551+0000 7f042aeed980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.551+0000 7f042aeed980 20 realm 2024-03-09T14:51:02.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.551+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.551+0000 7f042aeed980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.551+0000 7f042aeed980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:02.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.551+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.551+0000 7f042aeed980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.551+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.551+0000 7f042aeed980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:02.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.551+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.551+0000 7f042aeed980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:02.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.551+0000 7f042aeed980 20 searching for the correct realm 2024-03-09T14:51:02.559 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.555+0000 7f0428a05640 0 -- 172.21.15.1:0/3633984843 >> [v2:172.21.15.1:6824/1666056338,v1:172.21.15.1:6825/1666056338] conn(0x562ac2085d50 msgr2=0x7f042002a930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:02.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.563+0000 7f0429206640 0 -- 172.21.15.1:0/3633984843 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f041c00cc20 msgr2=0x7f041c084170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:02.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:02.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:02.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:02.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:02.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:02.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:02.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:02.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:02.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 zone default found 2024-03-09T14:51:02.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 4 Realm: () 2024-03-09T14:51:02.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:02.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:02.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:02.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 zonegroup default 2024-03-09T14:51:02.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.581 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.575+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.579+0000 7f042aeed980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.579+0000 7f042aeed980 20 started sync module instance, tier type = 2024-03-09T14:51:02.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.579+0000 7f042aeed980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:02.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.591+0000 7f042aeed980 20 add_watcher() i=0 2024-03-09T14:51:02.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.591+0000 7f042aeed980 20 add_watcher() i=1 2024-03-09T14:51:02.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.591+0000 7f042aeed980 20 add_watcher() i=2 2024-03-09T14:51:02.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.591+0000 7f042aeed980 20 add_watcher() i=3 2024-03-09T14:51:02.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.591+0000 7f042aeed980 20 add_watcher() i=4 2024-03-09T14:51:02.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.591+0000 7f042aeed980 20 add_watcher() i=5 2024-03-09T14:51:02.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.591+0000 7f042aeed980 20 add_watcher() i=6 2024-03-09T14:51:02.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.595+0000 7f042aeed980 20 add_watcher() i=7 2024-03-09T14:51:02.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.595+0000 7f042aeed980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:02.599 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.595+0000 7f02defe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:02.600 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.595+0000 7f042aeed980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:02.600 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.595+0000 7f042aeed980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:02.600 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.595+0000 7f042aeed980 5 note: GC not initialized 2024-03-09T14:51:02.600 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f02ddfe3640 20 reqs_thread_entry: start 2024-03-09T14:51:02.600 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f042aeed980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:02.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f042aeed980 20 Filter name: none 2024-03-09T14:51:02.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f024f7fe640 20 reqs_thread_entry: start 2024-03-09T14:51:02.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f042aeed980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:02.601 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f042aeed980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:02.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f042aeed980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:02.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f042aeed980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:02.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f042aeed980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:02.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.599+0000 7f042aeed980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:02.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.603+0000 7f042aeed980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.603+0000 7f042aeed980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.603+0000 7f02defe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:02.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.607+0000 7f042aeed980 20 remove_watcher() i=0 2024-03-09T14:51:02.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.607+0000 7f042aeed980 2 removed watcher, disabling cache 2024-03-09T14:51:02.612 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.611+0000 7f042aeed980 20 remove_watcher() i=1 2024-03-09T14:51:02.615 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.611+0000 7f042aeed980 20 remove_watcher() i=2 2024-03-09T14:51:02.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.615+0000 7f042aeed980 20 remove_watcher() i=3 2024-03-09T14:51:02.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.615+0000 7f042aeed980 20 remove_watcher() i=4 2024-03-09T14:51:02.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.619+0000 7f042aeed980 20 remove_watcher() i=5 2024-03-09T14:51:02.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.619+0000 7f042aeed980 20 remove_watcher() i=6 2024-03-09T14:51:02.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.619+0000 7f042aeed980 20 remove_watcher() i=7 2024-03-09T14:51:02.637 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:02.637 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:02.637 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-block-writes id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:51:02.638 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:02.638 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:02.639 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:02.680 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:02.680 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:02.746 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket error-at-block-writes --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2024-03-09T14:51:02.781 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:02.782 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:02.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.823+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.826 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.823+0000 7fc93d131980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.823+0000 7fc93d131980 20 realm 2024-03-09T14:51:02.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.823+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.823+0000 7fc93d131980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.823+0000 7fc93d131980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:02.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.823+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.823+0000 7fc93d131980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.823+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.827+0000 7fc93d131980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:02.828 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.827+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.827+0000 7fc93d131980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:02.829 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.827+0000 7fc93d131980 20 searching for the correct realm 2024-03-09T14:51:02.835 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.831+0000 7fc93a448640 0 -- 172.21.15.1:0/63575549 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55e5bea9b3b0 msgr2=0x55e5bea9d870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:51:02.842 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.839+0000 7fc93ac49640 0 -- 172.21.15.1:0/63575549 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fc930058ac0 msgr2=0x7fc930018400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:02.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:02.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:02.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:02.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:02.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:02.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:02.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.852 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:02.852 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.852 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:02.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 zone default found 2024-03-09T14:51:02.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 4 Realm: () 2024-03-09T14:51:02.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:02.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:02.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:02.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 zonegroup default 2024-03-09T14:51:02.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.847+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.851+0000 7fc93d131980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.853 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.851+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.854 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.851+0000 7fc93d131980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.854 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.851+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.854 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.851+0000 7fc93d131980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:02.854 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.851+0000 7fc93d131980 20 started sync module instance, tier type = 2024-03-09T14:51:02.854 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.851+0000 7fc93d131980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:02.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.867+0000 7fc93d131980 20 add_watcher() i=0 2024-03-09T14:51:02.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.867+0000 7fc93d131980 20 add_watcher() i=1 2024-03-09T14:51:02.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.867+0000 7fc93d131980 20 add_watcher() i=2 2024-03-09T14:51:02.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.867+0000 7fc93d131980 20 add_watcher() i=3 2024-03-09T14:51:02.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.867+0000 7fc93d131980 20 add_watcher() i=4 2024-03-09T14:51:02.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.867+0000 7fc93d131980 20 add_watcher() i=5 2024-03-09T14:51:02.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.867+0000 7fc93d131980 20 add_watcher() i=6 2024-03-09T14:51:02.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.867+0000 7fc93d131980 20 add_watcher() i=7 2024-03-09T14:51:02.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.867+0000 7fc93d131980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:02.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc7f17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:02.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:02.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:02.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 5 note: GC not initialized 2024-03-09T14:51:02.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc76bfff640 20 reqs_thread_entry: start 2024-03-09T14:51:02.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:02.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 20 Filter name: none 2024-03-09T14:51:02.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc769ffb640 20 reqs_thread_entry: start 2024-03-09T14:51:02.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 10 cache get: name=default.rgw.meta+root+error-at-block-writes : miss 2024-03-09T14:51:02.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 10 cache put: name=default.rgw.meta+root+error-at-block-writes info.flags=0x11 2024-03-09T14:51:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 10 adding default.rgw.meta+root+error-at-block-writes to cache LRU end 2024-03-09T14:51:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 : miss 2024-03-09T14:51:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 20 rados->read ofs=0 len=0 2024-03-09T14:51:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 info.flags=0x17 2024-03-09T14:51:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 10 adding default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 to cache LRU end 2024-03-09T14:51:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.871+0000 7fc93d131980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 2024-03-09T14:51:02.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.879+0000 7fc93d131980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 info.flags=0x17 2024-03-09T14:51:02.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.879+0000 7fc93d131980 10 moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 to cache LRU end 2024-03-09T14:51:02.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.879+0000 7fc93d131980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:02.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.879+0000 7fc93d131980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22, ofs0, ns] 2024-03-09T14:51:02.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.879+0000 7fc81e7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562397 cookie 94445234719376 notifier 6119 bl.length()=907 2024-03-09T14:51:02.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.879+0000 7fc81e7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 info.flags=0x17 2024-03-09T14:51:02.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.879+0000 7fc81e7fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 to cache LRU end 2024-03-09T14:51:02.883 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.879+0000 7fc81e7fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:02.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.883+0000 7fc93d131980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): orig_sources= new_sources= 2024-03-09T14:51:02.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.883+0000 7fc93d131980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): potential sources added=[] removed=[] 2024-03-09T14:51:02.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.883+0000 7fc93d131980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): orig_dests= new_dests= 2024-03-09T14:51:02.885 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.883+0000 7fc93d131980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): potential dests added=[] removed=[] 2024-03-09T14:51:02.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.887+0000 7fc93d131980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.22.1.0 2024-03-09T14:51:02.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.887+0000 7fc93d131980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.22.1.0 2024-03-09T14:51:02.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.887+0000 7fc93d131980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.22.1.1 2024-03-09T14:51:02.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.887+0000 7fc93d131980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.22.1.1 2024-03-09T14:51:02.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.887+0000 7fc93d131980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.22.0 2024-03-09T14:51:02.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc93d131980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 : hit (requested=0x17, cached=0x17) 2024-03-09T14:51:02.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc93d131980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 2024-03-09T14:51:02.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc93d131980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 info.flags=0x17 2024-03-09T14:51:02.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc93d131980 10 moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 to cache LRU end 2024-03-09T14:51:02.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc93d131980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:02.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc93d131980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22, ofs0, ns] 2024-03-09T14:51:02.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc81dffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562398 cookie 94445234719376 notifier 6119 bl.length()=875 2024-03-09T14:51:02.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc81dffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 info.flags=0x17 2024-03-09T14:51:02.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc81dffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 to cache LRU end 2024-03-09T14:51:02.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc81dffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:02.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc93d131980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): orig_sources= new_sources= 2024-03-09T14:51:02.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc93d131980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): potential sources added=[] removed=[] 2024-03-09T14:51:02.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc93d131980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): orig_dests= new_dests= 2024-03-09T14:51:02.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.891+0000 7fc93d131980 20 handle_bi_update(): bucket=:error-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.22]): potential dests added=[] removed=[] 2024-03-09T14:51:02.898 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.895+0000 7fc93d131980 1 execute INFO: reshard of bucket "error-at-block-writes" completed successfully 2024-03-09T14:51:02.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.899+0000 7fc7f17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:02.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.907+0000 7fc93d131980 20 remove_watcher() i=0 2024-03-09T14:51:02.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.907+0000 7fc93d131980 2 removed watcher, disabling cache 2024-03-09T14:51:02.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.907+0000 7fc93d131980 20 remove_watcher() i=1 2024-03-09T14:51:02.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.911+0000 7fc93d131980 20 remove_watcher() i=2 2024-03-09T14:51:02.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.911+0000 7fc93d131980 20 remove_watcher() i=3 2024-03-09T14:51:02.917 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.915+0000 7fc93d131980 20 remove_watcher() i=4 2024-03-09T14:51:02.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.915+0000 7fc93d131980 20 remove_watcher() i=5 2024-03-09T14:51:02.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.919+0000 7fc93d131980 20 remove_watcher() i=6 2024-03-09T14:51:02.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:02.919+0000 7fc93d131980 20 remove_watcher() i=7 2024-03-09T14:51:02.939 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-block-writes 2024-03-09T14:51:02.966 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:02.966 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:03.011 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.007+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.011+0000 7f11f9c26980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:03.013 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.011+0000 7f11f9c26980 20 realm 2024-03-09T14:51:03.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.011+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.011+0000 7f11f9c26980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:03.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.011+0000 7f11f9c26980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:03.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.011+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.011+0000 7f11f9c26980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:03.014 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.011+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.015 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.011+0000 7f11f9c26980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:03.016 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.011+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.016 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.011+0000 7f11f9c26980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:03.016 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.011+0000 7f11f9c26980 20 searching for the correct realm 2024-03-09T14:51:03.023 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.019+0000 7f11f7f3f640 0 -- 172.21.15.1:0/2670236562 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f11ec008050 msgr2=0x7f11ec0419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:03.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.235+0000 7f11f773e640 0 -- 172.21.15.1:0/2670236562 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x55c7a3d6dea0 msgr2=0x55c7a3d70360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:03.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.243+0000 7f11f9c26980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:03.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.243+0000 7f11f9c26980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:03.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.243+0000 7f11f9c26980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:03.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.243+0000 7f11f9c26980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:03.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.243+0000 7f11f9c26980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:03.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.243+0000 7f11f9c26980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:03.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.243+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.243+0000 7f11f9c26980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:03.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.243+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.243+0000 7f11f9c26980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:03.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.243+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:03.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 20 zone default found 2024-03-09T14:51:03.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 4 Realm: () 2024-03-09T14:51:03.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:03.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:03.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:03.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 20 zonegroup default 2024-03-09T14:51:03.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:03.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:03.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:03.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 20 started sync module instance, tier type = 2024-03-09T14:51:03.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.247+0000 7f11f9c26980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:03.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.271+0000 7f11f9c26980 20 add_watcher() i=0 2024-03-09T14:51:03.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.271+0000 7f11f9c26980 20 add_watcher() i=1 2024-03-09T14:51:03.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.271+0000 7f11f9c26980 20 add_watcher() i=2 2024-03-09T14:51:03.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.271+0000 7f11f9c26980 20 add_watcher() i=3 2024-03-09T14:51:03.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.271+0000 7f11f9c26980 20 add_watcher() i=4 2024-03-09T14:51:03.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.271+0000 7f11f9c26980 20 add_watcher() i=5 2024-03-09T14:51:03.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.271+0000 7f11f9c26980 20 add_watcher() i=6 2024-03-09T14:51:03.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.271+0000 7f11f9c26980 20 add_watcher() i=7 2024-03-09T14:51:03.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.271+0000 7f11f9c26980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:03.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.275+0000 7f10adfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:03.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.275+0000 7f11f9c26980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:03.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.275+0000 7f11f9c26980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:03.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.275+0000 7f11f9c26980 5 note: GC not initialized 2024-03-09T14:51:03.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.275+0000 7f10acfe1640 20 reqs_thread_entry: start 2024-03-09T14:51:03.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.275+0000 7f11f9c26980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:03.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.275+0000 7f11f9c26980 20 Filter name: none 2024-03-09T14:51:03.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.275+0000 7f10267fc640 20 reqs_thread_entry: start 2024-03-09T14:51:03.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.275+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.279+0000 7f11f9c26980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:03.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.279+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.279+0000 7f11f9c26980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:03.282 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.279+0000 7f11f9c26980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:03.282 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.279+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.282 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.279+0000 7f11f9c26980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:03.282 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.279+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.279+0000 7f11f9c26980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:03.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.279+0000 7f11f9c26980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:03.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.283+0000 7f11f9c26980 20 rados->read ofs=0 len=0 2024-03-09T14:51:03.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.283+0000 7f11f9c26980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:03.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.283+0000 7f10adfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:03.291 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.287+0000 7f11f9c26980 20 remove_watcher() i=0 2024-03-09T14:51:03.291 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.287+0000 7f11f9c26980 2 removed watcher, disabling cache 2024-03-09T14:51:03.293 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.291+0000 7f11f9c26980 20 remove_watcher() i=1 2024-03-09T14:51:03.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.295+0000 7f11f9c26980 20 remove_watcher() i=2 2024-03-09T14:51:03.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.295+0000 7f11f9c26980 20 remove_watcher() i=3 2024-03-09T14:51:03.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.299+0000 7f11f9c26980 20 remove_watcher() i=4 2024-03-09T14:51:03.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.299+0000 7f11f9c26980 20 remove_watcher() i=5 2024-03-09T14:51:03.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.299+0000 7f11f9c26980 20 remove_watcher() i=6 2024-03-09T14:51:03.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:03.303+0000 7f11f9c26980 20 remove_watcher() i=7 2024-03-09T14:51:03.328 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:03.329 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:03.329 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-block-writes id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.22 num_objects: 19 size_kb: 1 num_shards: 2 2024-03-09T14:51:03.329 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:03.331 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:03.331 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:03.378 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:03.378 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:04.010 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:04.011 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:04.194 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard bucket with abort at block_writes 2024-03-09T14:51:04.195 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:04.197 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:04.198 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:04.317 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:04.317 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:04.409 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:04.409 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:04.511 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:04.511 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:04.618 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:04.618 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:04.714 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:04.714 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:04.807 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:04.808 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:04.907 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:04.907 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:05.008 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:05.008 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:05.109 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:05.109 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:05.207 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:05.207 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:05.306 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:05.306 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:05.410 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:05.410 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:05.512 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:05.512 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:05.611 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:05.611 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:05.709 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:05.709 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:05.805 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:05.805 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:05.905 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:05.906 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:06.006 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:06.006 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:06.106 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:06.106 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:06.208 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:06.208 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:06.307 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:06.308 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:06.402 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket abort-at-block-writes 2024-03-09T14:51:06.436 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:06.437 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:06.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.483+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.483+0000 7fc232d0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:06.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.483+0000 7fc232d0a980 20 realm 2024-03-09T14:51:06.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.483+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.483+0000 7fc232d0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:06.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.483+0000 7fc232d0a980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:06.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.483+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.483+0000 7fc232d0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:06.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.483+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.487+0000 7fc232d0a980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:06.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.487+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.487+0000 7fc232d0a980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:06.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.487+0000 7fc232d0a980 20 searching for the correct realm 2024-03-09T14:51:06.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.495+0000 7fc230021640 0 -- 172.21.15.1:0/4080336225 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fc22011ec20 msgr2=0x7fc2200f6910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:06.502 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.499+0000 7fc231023640 0 -- 172.21.15.1:0/4080336225 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7fc21805d900 msgr2=0x7fc218076040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:51:06.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.507+0000 7fc232d0a980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:06.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.507+0000 7fc232d0a980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:06.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.507+0000 7fc232d0a980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:06.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.507+0000 7fc232d0a980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:06.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.507+0000 7fc232d0a980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:06.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.507+0000 7fc232d0a980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:06.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.507+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.507+0000 7fc232d0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:06.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.507+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:06.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:06.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 zone default found 2024-03-09T14:51:06.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 4 Realm: () 2024-03-09T14:51:06.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:06.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:06.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:06.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 zonegroup default 2024-03-09T14:51:06.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:06.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:06.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:06.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 started sync module instance, tier type = 2024-03-09T14:51:06.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.511+0000 7fc232d0a980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:06.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.527+0000 7fc232d0a980 20 add_watcher() i=0 2024-03-09T14:51:06.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.527+0000 7fc232d0a980 20 add_watcher() i=1 2024-03-09T14:51:06.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.527+0000 7fc232d0a980 20 add_watcher() i=2 2024-03-09T14:51:06.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.527+0000 7fc232d0a980 20 add_watcher() i=3 2024-03-09T14:51:06.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.527+0000 7fc232d0a980 20 add_watcher() i=4 2024-03-09T14:51:06.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.527+0000 7fc232d0a980 20 add_watcher() i=5 2024-03-09T14:51:06.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.527+0000 7fc232d0a980 20 add_watcher() i=6 2024-03-09T14:51:06.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.531+0000 7fc232d0a980 20 add_watcher() i=7 2024-03-09T14:51:06.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.531+0000 7fc232d0a980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:06.534 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.531+0000 7fc0e6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:06.535 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.531+0000 7fc232d0a980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:06.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.531+0000 7fc232d0a980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:06.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.531+0000 7fc232d0a980 5 note: GC not initialized 2024-03-09T14:51:06.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.531+0000 7fc0e5fe3640 20 reqs_thread_entry: start 2024-03-09T14:51:06.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.531+0000 7fc232d0a980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:06.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.531+0000 7fc232d0a980 20 Filter name: none 2024-03-09T14:51:06.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.531+0000 7fc0577fe640 20 reqs_thread_entry: start 2024-03-09T14:51:06.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.531+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.535+0000 7fc232d0a980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:06.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.535+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.535+0000 7fc232d0a980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:06.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.535+0000 7fc232d0a980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:06.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.535+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.538 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.535+0000 7fc232d0a980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:06.538 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.535+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.538 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.535+0000 7fc232d0a980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:06.538 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.535+0000 7fc232d0a980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:06.539 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.535+0000 7fc232d0a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.539 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.535+0000 7fc232d0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:06.540 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.539+0000 7fc0e6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:06.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.543+0000 7fc232d0a980 20 remove_watcher() i=0 2024-03-09T14:51:06.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.543+0000 7fc232d0a980 2 removed watcher, disabling cache 2024-03-09T14:51:06.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.547+0000 7fc232d0a980 20 remove_watcher() i=1 2024-03-09T14:51:06.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.547+0000 7fc232d0a980 20 remove_watcher() i=2 2024-03-09T14:51:06.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.547+0000 7fc232d0a980 20 remove_watcher() i=3 2024-03-09T14:51:06.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.551+0000 7fc232d0a980 20 remove_watcher() i=4 2024-03-09T14:51:06.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.551+0000 7fc232d0a980 20 remove_watcher() i=5 2024-03-09T14:51:06.556 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.555+0000 7fc232d0a980 20 remove_watcher() i=6 2024-03-09T14:51:06.558 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.555+0000 7fc232d0a980 20 remove_watcher() i=7 2024-03-09T14:51:06.571 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:06.572 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:06.572 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: abort-at-block-writes id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:51:06.572 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:06.572 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket abort-at-block-writes --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-abort-at block_writes 2024-03-09T14:51:06.606 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:06.606 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:06.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:06.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.859+0000 7f7832c7f980 20 realm 2024-03-09T14:51:06.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:06.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.859+0000 7f7832c7f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:06.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:06.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.863+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:06.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.863+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:06.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.863+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:06.865 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.863+0000 7f7832c7f980 20 searching for the correct realm 2024-03-09T14:51:06.870 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:06.867+0000 7f782ff96640 0 -- 172.21.15.1:0/2595872606 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f78200a8bc0 msgr2=0x7f7820108af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:07.085 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.083+0000 7f7830797640 0 -- 172.21.15.1:0/2595872606 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f782403b930 msgr2=0x7f782402d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:07.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:07.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:07.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:07.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:07.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:07.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:07.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.299+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.299+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.299+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:07.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:07.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 zone default found 2024-03-09T14:51:07.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 4 Realm: () 2024-03-09T14:51:07.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:07.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:07.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:07.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 zonegroup default 2024-03-09T14:51:07.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 started sync module instance, tier type = 2024-03-09T14:51:07.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.303+0000 7f7832c7f980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:07.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=0 2024-03-09T14:51:07.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=1 2024-03-09T14:51:07.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=2 2024-03-09T14:51:07.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=3 2024-03-09T14:51:07.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=4 2024-03-09T14:51:07.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=5 2024-03-09T14:51:07.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=6 2024-03-09T14:51:07.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=7 2024-03-09T14:51:07.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.331+0000 7f7832c7f980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:07.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.335+0000 7f76e6fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:07.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.335+0000 7f7832c7f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:07.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.335+0000 7f7832c7f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:07.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.335+0000 7f7832c7f980 5 note: GC not initialized 2024-03-09T14:51:07.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.335+0000 7f76e5fdb640 20 reqs_thread_entry: start 2024-03-09T14:51:07.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.335+0000 7f7832c7f980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:07.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 20 Filter name: none 2024-03-09T14:51:07.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f76577fe640 20 reqs_thread_entry: start 2024-03-09T14:51:07.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2024-03-09T14:51:07.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.341 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:07.341 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2024-03-09T14:51:07.341 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2024-03-09T14:51:07.341 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 : miss 2024-03-09T14:51:07.341 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.342 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:07.342 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:07.342 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:07.342 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:07.342 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:07.342 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.339+0000 7f7832c7f980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 2024-03-09T14:51:07.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.347+0000 7f7832c7f980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:07.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.347+0000 7f7832c7f980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:07.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.347+0000 7f7832c7f980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:07.349 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.347+0000 7f7832c7f980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23, ofs0, ns] 2024-03-09T14:51:07.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.347+0000 7f77077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562446 cookie 94851686727232 notifier 6173 bl.length()=907 2024-03-09T14:51:07.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.347+0000 7f77077fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:07.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.347+0000 7f77077fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:07.350 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.347+0000 7f77077fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:07.354 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.351+0000 7f7832c7f980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): orig_sources= new_sources= 2024-03-09T14:51:07.354 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.351+0000 7f7832c7f980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): potential sources added=[] removed=[] 2024-03-09T14:51:07.354 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.351+0000 7f7832c7f980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): orig_dests= new_dests= 2024-03-09T14:51:07.354 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f7832c7f980 time 2024-03-09T14:51:07.356062+0000 2024-03-09T14:51:07.355 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:51:07.355 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.351+0000 7f7832c7f980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): potential dests added=[] removed=[] 2024-03-09T14:51:07.356 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f7836554d7b] 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f7836554f2f] 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x56445f879c51] 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a5) [0x56445f89a285] 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr: 5: main() 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f78353bfd90] 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr: 7: __libc_start_main() 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr: 8: _start() 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr:*** Caught signal (Aborted) ** 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f7832c7f980 thread_name:radosgw-admin 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.355+0000 7f7832c7f980 -1 ./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f7832c7f980 time 2024-03-09T14:51:07.356062+0000 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:51:07.357 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.358 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:07.358 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f7836554d7b] 2024-03-09T14:51:07.358 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f7836554f2f] 2024-03-09T14:51:07.358 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x56445f879c51] 2024-03-09T14:51:07.358 INFO:tasks.workunit.client.0.smithi001.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a5) [0x56445f89a285] 2024-03-09T14:51:07.358 INFO:tasks.workunit.client.0.smithi001.stderr: 5: main() 2024-03-09T14:51:07.358 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f78353bfd90] 2024-03-09T14:51:07.358 INFO:tasks.workunit.client.0.smithi001.stderr: 7: __libc_start_main() 2024-03-09T14:51:07.358 INFO:tasks.workunit.client.0.smithi001.stderr: 8: _start() 2024-03-09T14:51:07.358 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.359 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:07.359 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f78353d8520] 2024-03-09T14:51:07.359 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f7836554dd5] 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f7836554f2f] 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x56445f879c51] 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a5) [0x56445f89a285] 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr: 9: main() 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f78353bfd90] 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr: 11: __libc_start_main() 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr: 12: _start() 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.355+0000 7f7832c7f980 -1 *** Caught signal (Aborted) ** 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f7832c7f980 thread_name:radosgw-admin 2024-03-09T14:51:07.360 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.361 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:07.361 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f78353d8520] 2024-03-09T14:51:07.361 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:51:07.361 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:51:07.361 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:51:07.361 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f7836554dd5] 2024-03-09T14:51:07.361 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f7836554f2f] 2024-03-09T14:51:07.361 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x56445f879c51] 2024-03-09T14:51:07.361 INFO:tasks.workunit.client.0.smithi001.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a5) [0x56445f89a285] 2024-03-09T14:51:07.361 INFO:tasks.workunit.client.0.smithi001.stderr: 9: main() 2024-03-09T14:51:07.362 INFO:tasks.workunit.client.0.smithi001.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f78353bfd90] 2024-03-09T14:51:07.362 INFO:tasks.workunit.client.0.smithi001.stderr: 11: __libc_start_main() 2024-03-09T14:51:07.362 INFO:tasks.workunit.client.0.smithi001.stderr: 12: _start() 2024-03-09T14:51:07.362 INFO:tasks.workunit.client.0.smithi001.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-09T14:51:07.362 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.362 INFO:tasks.workunit.client.0.smithi001.stderr:--- begin dump of recent events --- 2024-03-09T14:51:07.406 INFO:tasks.workunit.client.0.smithi001.stderr: -499> 2024-03-09T14:51:06.835+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.406 INFO:tasks.workunit.client.0.smithi001.stderr: -498> 2024-03-09T14:51:06.835+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.406 INFO:tasks.workunit.client.0.smithi001.stderr: -497> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): get_auth_request con 0x7f7818002100 auth_method 0 2024-03-09T14:51:07.406 INFO:tasks.workunit.client.0.smithi001.stderr: -496> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.406 INFO:tasks.workunit.client.0.smithi001.stderr: -495> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.406 INFO:tasks.workunit.client.0.smithi001.stderr: -494> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.406 INFO:tasks.workunit.client.0.smithi001.stderr: -493> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): handle_auth_done global_id 6543 payload 274 2024-03-09T14:51:07.406 INFO:tasks.workunit.client.0.smithi001.stderr: -492> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient: _finish_hunting 0 2024-03-09T14:51:07.406 INFO:tasks.workunit.client.0.smithi001.stderr: -491> 2024-03-09T14:51:06.835+0000 7f7830f98640 1 monclient: found mon.noname-a 2024-03-09T14:51:07.406 INFO:tasks.workunit.client.0.smithi001.stderr: -490> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr: -489> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr: -488> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr: -487> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: dump: 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.407 INFO:tasks.workunit.client.0.smithi001.stderr: -486> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: _finish_auth 0 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr: -485> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: _check_auth_tickets 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr: -484> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr: -483> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 monclient: authenticate success, global_id 6543 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr: -482> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: get_monmap_and_config success 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr: -481> 2024-03-09T14:51:06.835+0000 7f7832c7f980 4 set_mon_vals no callback set 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr: -480> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr: -479> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr: -478> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: dump: 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.408 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr: -477> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: shutdown 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr: -476> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 asok(0x564460e7dbb0) unregister_commands rotate-key 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr: -475> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: build_initial_monmap 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr: -474> 2024-03-09T14:51:06.835+0000 7f7832c7f980 1 build_initial for_mkfs: 0 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr: -473> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: monmap: 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:07.409 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.410 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:07.410 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:07.410 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:07.410 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.410 INFO:tasks.workunit.client.0.smithi001.stderr: -472> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding auth protocol: cephx 2024-03-09T14:51:07.410 INFO:tasks.workunit.client.0.smithi001.stderr: -471> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding auth protocol: cephx 2024-03-09T14:51:07.410 INFO:tasks.workunit.client.0.smithi001.stderr: -470> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding auth protocol: cephx 2024-03-09T14:51:07.410 INFO:tasks.workunit.client.0.smithi001.stderr: -469> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: secure 2024-03-09T14:51:07.410 INFO:tasks.workunit.client.0.smithi001.stderr: -468> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: crc 2024-03-09T14:51:07.410 INFO:tasks.workunit.client.0.smithi001.stderr: -467> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: secure 2024-03-09T14:51:07.410 INFO:tasks.workunit.client.0.smithi001.stderr: -466> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: crc 2024-03-09T14:51:07.410 INFO:tasks.workunit.client.0.smithi001.stderr: -465> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: secure 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -464> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: crc 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -463> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: crc 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -462> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: secure 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -461> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: crc 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -460> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: secure 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -459> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: crc 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -458> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: secure 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -457> 2024-03-09T14:51:06.835+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -456> 2024-03-09T14:51:06.835+0000 7f7832c7f980 1 librados: starting msgr at 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -455> 2024-03-09T14:51:06.835+0000 7f7832c7f980 1 librados: starting objecter 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -454> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command objecter_requests hook 0x564460f3bc20 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -453> 2024-03-09T14:51:06.835+0000 7f7832c7f980 1 librados: setting wanted keys 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -452> 2024-03-09T14:51:06.835+0000 7f7832c7f980 1 librados: calling monclient init 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -451> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: init 2024-03-09T14:51:07.411 INFO:tasks.workunit.client.0.smithi001.stderr: -450> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding auth protocol: cephx 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -449> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding auth protocol: cephx 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -448> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding auth protocol: cephx 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -447> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: secure 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -446> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: crc 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -445> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: secure 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -444> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: crc 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -443> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: secure 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -442> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: crc 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -441> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: crc 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -440> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: secure 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -439> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: crc 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -438> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: secure 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -437> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: crc 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -436> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: secure 2024-03-09T14:51:07.412 INFO:tasks.workunit.client.0.smithi001.stderr: -435> 2024-03-09T14:51:06.835+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -434> 2024-03-09T14:51:06.835+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -433> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command rotate-key hook 0x564461086f48 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -432> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -431> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: _add_conns ranks=[0,1,2] 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -430> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-a con 0x56446108aef0 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -429> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-b con 0x5644610b5d80 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -428> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-c con 0x7f7818002100 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -427> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -426> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -425> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -424> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -423> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): authenticate will time out at 2360.583696s 2024-03-09T14:51:07.413 INFO:tasks.workunit.client.0.smithi001.stderr: -422> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): get_auth_request con 0x56446108aef0 auth_method 0 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -421> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -420> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -419> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -418> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): get_auth_request con 0x7f7818002100 auth_method 0 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -417> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -416> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -415> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -414> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -413> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -412> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -411> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -410> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.414 INFO:tasks.workunit.client.0.smithi001.stderr: -409> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr: -408> 2024-03-09T14:51:06.839+0000 7f7830797640 10 monclient(hunting): handle_auth_done global_id 6546 payload 824 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr: -407> 2024-03-09T14:51:06.839+0000 7f7830797640 10 monclient: _finish_hunting 0 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr: -406> 2024-03-09T14:51:06.839+0000 7f7830797640 1 monclient: found mon.noname-a 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr: -405> 2024-03-09T14:51:06.839+0000 7f7830797640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr: -404> 2024-03-09T14:51:06.839+0000 7f782ff96640 10 monclient: get_auth_request con 0x5644610b5d80 auth_method 0 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr: -403> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr: -402> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr: -401> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: dump: 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.415 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr: -400> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: _finish_auth 0 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr: -399> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: _check_auth_tickets 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr: -398> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr: -397> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr: -396> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr: -395> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: dump: 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.416 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.417 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.417 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.417 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.417 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.417 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.417 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.417 INFO:tasks.workunit.client.0.smithi001.stderr: -394> 2024-03-09T14:51:06.839+0000 7f7832c7f980 5 monclient: authenticate success, global_id 6546 2024-03-09T14:51:07.418 INFO:tasks.workunit.client.0.smithi001.stderr: -393> 2024-03-09T14:51:06.839+0000 7f7832c7f980 10 monclient: _renew_subs 2024-03-09T14:51:07.418 INFO:tasks.workunit.client.0.smithi001.stderr: -392> 2024-03-09T14:51:06.839+0000 7f7832c7f980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.418 INFO:tasks.workunit.client.0.smithi001.stderr: -391> 2024-03-09T14:51:06.839+0000 7f771e60c640 4 set_mon_vals no callback set 2024-03-09T14:51:07.418 INFO:tasks.workunit.client.0.smithi001.stderr: -390> 2024-03-09T14:51:06.839+0000 7f7832c7f980 10 monclient: _renew_subs 2024-03-09T14:51:07.418 INFO:tasks.workunit.client.0.smithi001.stderr: -389> 2024-03-09T14:51:06.839+0000 7f7832c7f980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.418 INFO:tasks.workunit.client.0.smithi001.stderr: -388> 2024-03-09T14:51:06.839+0000 7f7832c7f980 1 librados: init done 2024-03-09T14:51:07.418 INFO:tasks.workunit.client.0.smithi001.stderr: -387> 2024-03-09T14:51:06.839+0000 7f771ce09640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:51:07.418 INFO:tasks.workunit.client.0.smithi001.stderr: -386> 2024-03-09T14:51:06.839+0000 7f771ce09640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:07.418 INFO:tasks.workunit.client.0.smithi001.stderr: -385> 2024-03-09T14:51:06.839+0000 7f771ce09640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:07.418 INFO:tasks.workunit.client.0.smithi001.stderr: -384> 2024-03-09T14:51:06.839+0000 7f782ff96640 10 monclient: get_auth_request con 0x7f7718066040 auth_method 0 2024-03-09T14:51:07.418 INFO:tasks.workunit.client.0.smithi001.stderr: -383> 2024-03-09T14:51:06.839+0000 7f7830f98640 10 monclient: get_auth_request con 0x5644610c9010 auth_method 0 2024-03-09T14:51:07.418 INFO:tasks.workunit.client.0.smithi001.stderr: -382> 2024-03-09T14:51:06.843+0000 7f7830797640 10 monclient: get_auth_request con 0x5644610d2300 auth_method 0 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr: -381> 2024-03-09T14:51:06.843+0000 7f782ff96640 10 monclient: get_auth_request con 0x5644610d7390 auth_method 0 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr: -380> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: build_initial_monmap 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr: -379> 2024-03-09T14:51:06.847+0000 7f7832c7f980 1 build_initial for_mkfs: 0 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr: -378> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: monmap: 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.419 INFO:tasks.workunit.client.0.smithi001.stderr: -377> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding auth protocol: cephx 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -376> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding auth protocol: cephx 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -375> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding auth protocol: cephx 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -374> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: secure 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -373> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: crc 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -372> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: secure 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -371> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: crc 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -370> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: secure 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -369> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: crc 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -368> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: crc 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -367> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: secure 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -366> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: crc 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -365> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: secure 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -364> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: crc 2024-03-09T14:51:07.420 INFO:tasks.workunit.client.0.smithi001.stderr: -363> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: secure 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -362> 2024-03-09T14:51:06.847+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -361> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command objecter_requests cmddesc objecter_requests hook 0x5644610dc5a0 EEXIST 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -360> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: init 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -359> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding auth protocol: cephx 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -358> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding auth protocol: cephx 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -357> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding auth protocol: cephx 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -356> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: secure 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -355> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: crc 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -354> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: secure 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -353> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: crc 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -352> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: secure 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -351> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: crc 2024-03-09T14:51:07.421 INFO:tasks.workunit.client.0.smithi001.stderr: -350> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: crc 2024-03-09T14:51:07.422 INFO:tasks.workunit.client.0.smithi001.stderr: -349> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: secure 2024-03-09T14:51:07.422 INFO:tasks.workunit.client.0.smithi001.stderr: -348> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: crc 2024-03-09T14:51:07.422 INFO:tasks.workunit.client.0.smithi001.stderr: -347> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: secure 2024-03-09T14:51:07.422 INFO:tasks.workunit.client.0.smithi001.stderr: -346> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: crc 2024-03-09T14:51:07.422 INFO:tasks.workunit.client.0.smithi001.stderr: -345> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: secure 2024-03-09T14:51:07.422 INFO:tasks.workunit.client.0.smithi001.stderr: -344> 2024-03-09T14:51:06.847+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.422 INFO:tasks.workunit.client.0.smithi001.stderr: -343> 2024-03-09T14:51:06.847+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.422 INFO:tasks.workunit.client.0.smithi001.stderr: -342> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command rotate-key cmddesc rotate-key hook 0x5644610da490 EEXIST 2024-03-09T14:51:07.422 INFO:tasks.workunit.client.0.smithi001.stderr: -341> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:07.422 INFO:tasks.workunit.client.0.smithi001.stderr: -340> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: _add_conns ranks=[0,2,1] 2024-03-09T14:51:07.422 INFO:tasks.workunit.client.0.smithi001.stderr: -339> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-a con 0x564461105740 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:07.422 INFO:tasks.workunit.client.0.smithi001.stderr: -338> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-c con 0x5644611094c0 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:07.423 INFO:tasks.workunit.client.0.smithi001.stderr: -337> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-b con 0x564461100c50 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:07.423 INFO:tasks.workunit.client.0.smithi001.stderr: -336> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.423 INFO:tasks.workunit.client.0.smithi001.stderr: -335> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.423 INFO:tasks.workunit.client.0.smithi001.stderr: -334> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.423 INFO:tasks.workunit.client.0.smithi001.stderr: -333> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:07.423 INFO:tasks.workunit.client.0.smithi001.stderr: -332> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): authenticate will time out at 2360.594160s 2024-03-09T14:51:07.423 INFO:tasks.workunit.client.0.smithi001.stderr: -331> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): get_auth_request con 0x5644611094c0 auth_method 0 2024-03-09T14:51:07.423 INFO:tasks.workunit.client.0.smithi001.stderr: -330> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.423 INFO:tasks.workunit.client.0.smithi001.stderr: -329> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.423 INFO:tasks.workunit.client.0.smithi001.stderr: -328> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.423 INFO:tasks.workunit.client.0.smithi001.stderr: -327> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): get_auth_request con 0x564461105740 auth_method 0 2024-03-09T14:51:07.423 INFO:tasks.workunit.client.0.smithi001.stderr: -326> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -325> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -324> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -323> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -322> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -321> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -320> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -319> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -318> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -317> 2024-03-09T14:51:06.847+0000 7f7830f98640 10 monclient(hunting): get_auth_request con 0x564461100c50 auth_method 0 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -316> 2024-03-09T14:51:06.847+0000 7f7830f98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -315> 2024-03-09T14:51:06.847+0000 7f7830f98640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -314> 2024-03-09T14:51:06.847+0000 7f7830f98640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -313> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): handle_auth_done global_id 6549 payload 824 2024-03-09T14:51:07.424 INFO:tasks.workunit.client.0.smithi001.stderr: -312> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient: _finish_hunting 0 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr: -311> 2024-03-09T14:51:06.847+0000 7f7830797640 1 monclient: found mon.noname-a 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr: -310> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr: -309> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr: -308> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr: -307> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: dump: 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.425 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr: -306> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: _finish_auth 0 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr: -305> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: _check_auth_tickets 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr: -304> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr: -303> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr: -302> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr: -301> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: dump: 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.426 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr: -300> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 monclient: authenticate success, global_id 6549 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr: -299> 2024-03-09T14:51:06.847+0000 7f7831799640 4 set_mon_vals no callback set 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr: -298> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: _renew_subs 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr: -297> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr: -296> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: _renew_subs 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr: -295> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr: -294> 2024-03-09T14:51:06.851+0000 7f7715ffb640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr: -293> 2024-03-09T14:51:06.851+0000 7f7715ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr: -292> 2024-03-09T14:51:06.851+0000 7f7715ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr: -291> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: get_monmap_and_config 2024-03-09T14:51:07.427 INFO:tasks.workunit.client.0.smithi001.stderr: -290> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: build_initial_monmap 2024-03-09T14:51:07.428 INFO:tasks.workunit.client.0.smithi001.stderr: -289> 2024-03-09T14:51:06.851+0000 7f7832c7f980 1 build_initial for_mkfs: 0 2024-03-09T14:51:07.428 INFO:tasks.workunit.client.0.smithi001.stderr: -288> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: monmap: 2024-03-09T14:51:07.428 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:07.428 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:07.428 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:07.428 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:07.428 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:07.428 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.428 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:07.428 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:07.428 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -287> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding auth protocol: cephx 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -286> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding auth protocol: cephx 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -285> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding auth protocol: cephx 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -284> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -283> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -282> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -281> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -280> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -279> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -278> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -277> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -276> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.429 INFO:tasks.workunit.client.0.smithi001.stderr: -275> 2024-03-09T14:51:06.851+0000 7f782ff96640 10 monclient: get_auth_request con 0x7f7708065f50 auth_method 0 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -274> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -273> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -272> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -271> 2024-03-09T14:51:06.851+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -270> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: init 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -269> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding auth protocol: cephx 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -268> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding auth protocol: cephx 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -267> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding auth protocol: cephx 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -266> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: secure 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -265> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: crc 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -264> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: secure 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -263> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: crc 2024-03-09T14:51:07.430 INFO:tasks.workunit.client.0.smithi001.stderr: -262> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: secure 2024-03-09T14:51:07.431 INFO:tasks.workunit.client.0.smithi001.stderr: -261> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: crc 2024-03-09T14:51:07.431 INFO:tasks.workunit.client.0.smithi001.stderr: -260> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: crc 2024-03-09T14:51:07.431 INFO:tasks.workunit.client.0.smithi001.stderr: -259> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: secure 2024-03-09T14:51:07.431 INFO:tasks.workunit.client.0.smithi001.stderr: -258> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: crc 2024-03-09T14:51:07.431 INFO:tasks.workunit.client.0.smithi001.stderr: -257> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: secure 2024-03-09T14:51:07.431 INFO:tasks.workunit.client.0.smithi001.stderr: -256> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: crc 2024-03-09T14:51:07.431 INFO:tasks.workunit.client.0.smithi001.stderr: -255> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: secure 2024-03-09T14:51:07.431 INFO:tasks.workunit.client.0.smithi001.stderr: -254> 2024-03-09T14:51:06.851+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.431 INFO:tasks.workunit.client.0.smithi001.stderr: -253> 2024-03-09T14:51:06.851+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -252> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command rotate-key cmddesc rotate-key hook 0x7ffec1e83ec8 EEXIST 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -251> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -250> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: _add_conns ranks=[2,1,0] 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -249> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-c con 0x5644611572e0 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -248> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-b con 0x5644611576c0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -247> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-a con 0x5644611274f0 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -246> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -245> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -244> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -243> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -242> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): authenticate will time out at 2360.598802s 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -241> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): get_auth_request con 0x5644611572e0 auth_method 0 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -240> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.432 INFO:tasks.workunit.client.0.smithi001.stderr: -239> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -238> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -237> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): get_auth_request con 0x5644611274f0 auth_method 0 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -236> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -235> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -234> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -233> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -232> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -231> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -230> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -229> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -228> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -227> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): handle_auth_done global_id 6170 payload 274 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -226> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient: _finish_hunting 0 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -225> 2024-03-09T14:51:06.851+0000 7f7830797640 1 monclient: found mon.noname-c 2024-03-09T14:51:07.433 INFO:tasks.workunit.client.0.smithi001.stderr: -224> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr: -223> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr: -222> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr: -221> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: dump: 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.434 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr: -220> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: _finish_auth 0 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr: -219> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: _check_auth_tickets 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr: -218> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr: -217> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 monclient: authenticate success, global_id 6170 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr: -216> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: get_monmap_and_config success 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr: -215> 2024-03-09T14:51:06.851+0000 7f7832c7f980 4 set_mon_vals no callback set 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr: -214> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr: -213> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr: -212> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: dump: 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.435 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.436 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.436 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.436 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.436 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.436 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.436 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.436 INFO:tasks.workunit.client.0.smithi001.stderr: -211> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: shutdown 2024-03-09T14:51:07.436 INFO:tasks.workunit.client.0.smithi001.stderr: -210> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: build_initial_monmap 2024-03-09T14:51:07.436 INFO:tasks.workunit.client.0.smithi001.stderr: -209> 2024-03-09T14:51:06.855+0000 7f7832c7f980 1 build_initial for_mkfs: 0 2024-03-09T14:51:07.436 INFO:tasks.workunit.client.0.smithi001.stderr: -208> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: monmap: 2024-03-09T14:51:07.436 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr: -207> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding auth protocol: cephx 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr: -206> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding auth protocol: cephx 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr: -205> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding auth protocol: cephx 2024-03-09T14:51:07.437 INFO:tasks.workunit.client.0.smithi001.stderr: -204> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -203> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -202> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -201> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -200> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -199> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -198> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -197> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -196> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -195> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -194> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -193> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -192> 2024-03-09T14:51:06.855+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -191> 2024-03-09T14:51:06.855+0000 7f7832c7f980 1 librados: starting msgr at 2024-03-09T14:51:07.438 INFO:tasks.workunit.client.0.smithi001.stderr: -190> 2024-03-09T14:51:06.855+0000 7f7832c7f980 1 librados: starting objecter 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -189> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command objecter_requests cmddesc objecter_requests hook 0x5644611519c0 EEXIST 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -188> 2024-03-09T14:51:06.855+0000 7f7832c7f980 1 librados: setting wanted keys 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -187> 2024-03-09T14:51:06.855+0000 7f7832c7f980 1 librados: calling monclient init 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -186> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: init 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -185> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding auth protocol: cephx 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -184> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding auth protocol: cephx 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -183> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding auth protocol: cephx 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -182> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: secure 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -181> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: crc 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -180> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: secure 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -179> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: crc 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -178> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: secure 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -177> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: crc 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -176> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: crc 2024-03-09T14:51:07.439 INFO:tasks.workunit.client.0.smithi001.stderr: -175> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: secure 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -174> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: crc 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -173> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: secure 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -172> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: crc 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -171> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: secure 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -170> 2024-03-09T14:51:06.855+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -169> 2024-03-09T14:51:06.855+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -168> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command rotate-key cmddesc rotate-key hook 0x564461103278 EEXIST 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -167> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -166> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: _add_conns ranks=[2,0,1] 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -165> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-c con 0x5644611274f0 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -164> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-a con 0x5644611572e0 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -163> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-b con 0x564461150180 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -162> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.440 INFO:tasks.workunit.client.0.smithi001.stderr: -161> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -160> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -159> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -158> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): authenticate will time out at 2360.601789s 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -157> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): get_auth_request con 0x5644611274f0 auth_method 0 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -156> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -155> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -154> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -153> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): get_auth_request con 0x5644611572e0 auth_method 0 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -152> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -151> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -150> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -149> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -148> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.441 INFO:tasks.workunit.client.0.smithi001.stderr: -147> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr: -146> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr: -145> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr: -144> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr: -143> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): handle_auth_done global_id 6173 payload 824 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr: -142> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient: _finish_hunting 0 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr: -141> 2024-03-09T14:51:06.855+0000 7f7830797640 1 monclient: found mon.noname-c 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr: -140> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr: -139> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr: -138> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr: -137> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: dump: 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.442 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.443 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.443 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.443 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.443 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.443 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.443 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.443 INFO:tasks.workunit.client.0.smithi001.stderr: -136> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: _finish_auth 0 2024-03-09T14:51:07.443 INFO:tasks.workunit.client.0.smithi001.stderr: -135> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: _check_auth_tickets 2024-03-09T14:51:07.443 INFO:tasks.workunit.client.0.smithi001.stderr: -134> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:07.443 INFO:tasks.workunit.client.0.smithi001.stderr: -133> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.443 INFO:tasks.workunit.client.0.smithi001.stderr: -132> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:51:07.443 INFO:tasks.workunit.client.0.smithi001.stderr: -131> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: dump: 2024-03-09T14:51:07.444 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.444 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.444 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.444 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.444 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.444 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.444 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.444 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.444 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.444 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.444 INFO:tasks.workunit.client.0.smithi001.stderr: -130> 2024-03-09T14:51:06.855+0000 7f77077fe640 4 set_mon_vals no callback set 2024-03-09T14:51:07.444 INFO:tasks.workunit.client.0.smithi001.stderr: -129> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 monclient: authenticate success, global_id 6173 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -128> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: _renew_subs 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -127> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -126> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: _renew_subs 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -125> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -124> 2024-03-09T14:51:06.855+0000 7f7832c7f980 1 librados: init done 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -123> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command cr dump hook 0x7f78200b2f98 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -122> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -121> 2024-03-09T14:51:06.859+0000 7f7705ffb640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -120> 2024-03-09T14:51:06.859+0000 7f7705ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -119> 2024-03-09T14:51:06.859+0000 7f7705ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -118> 2024-03-09T14:51:06.859+0000 7f7830f98640 10 monclient: get_auth_request con 0x5644611706c0 auth_method 0 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -117> 2024-03-09T14:51:06.859+0000 7f782ff96640 10 monclient: get_auth_request con 0x7f76f8066070 auth_method 0 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -116> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.445 INFO:tasks.workunit.client.0.smithi001.stderr: -115> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 realm 2024-03-09T14:51:07.446 INFO:tasks.workunit.client.0.smithi001.stderr: -114> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.446 INFO:tasks.workunit.client.0.smithi001.stderr: -113> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.446 INFO:tasks.workunit.client.0.smithi001.stderr: -112> 2024-03-09T14:51:06.859+0000 7f7832c7f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:07.446 INFO:tasks.workunit.client.0.smithi001.stderr: -111> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.446 INFO:tasks.workunit.client.0.smithi001.stderr: -110> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.446 INFO:tasks.workunit.client.0.smithi001.stderr: -109> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.446 INFO:tasks.workunit.client.0.smithi001.stderr: -108> 2024-03-09T14:51:06.859+0000 7f7830797640 10 monclient: get_auth_request con 0x564461174910 auth_method 0 2024-03-09T14:51:07.446 INFO:tasks.workunit.client.0.smithi001.stderr: -107> 2024-03-09T14:51:06.863+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:07.446 INFO:tasks.workunit.client.0.smithi001.stderr: -106> 2024-03-09T14:51:06.863+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -105> 2024-03-09T14:51:06.863+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -104> 2024-03-09T14:51:06.863+0000 7f7832c7f980 20 searching for the correct realm 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -103> 2024-03-09T14:51:06.863+0000 7f782ff96640 10 monclient: get_auth_request con 0x564461179b50 auth_method 0 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -102> 2024-03-09T14:51:06.863+0000 7f7830f98640 10 monclient: get_auth_request con 0x7f7824008040 auth_method 0 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -101> 2024-03-09T14:51:06.867+0000 7f7830797640 10 monclient: get_auth_request con 0x564460f17260 auth_method 0 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -100> 2024-03-09T14:51:06.867+0000 7f782ff96640 0 -- 172.21.15.1:0/2595872606 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f78200a8bc0 msgr2=0x7f7820108af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -99> 2024-03-09T14:51:06.867+0000 7f782ff96640 10 monclient: get_auth_request con 0x7f78200a8bc0 auth_method 0 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -98> 2024-03-09T14:51:07.071+0000 7f782ff96640 10 monclient: get_auth_request con 0x7f78200a8bc0 auth_method 0 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -97> 2024-03-09T14:51:07.075+0000 7f7830f98640 10 monclient: get_auth_request con 0x7f782404c040 auth_method 0 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -96> 2024-03-09T14:51:07.083+0000 7f7830797640 10 monclient: get_auth_request con 0x7f782403b930 auth_method 0 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -95> 2024-03-09T14:51:07.083+0000 7f7830797640 0 -- 172.21.15.1:0/2595872606 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f782403b930 msgr2=0x7f782402d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -94> 2024-03-09T14:51:07.287+0000 7f7830797640 10 monclient: get_auth_request con 0x7f782403b930 auth_method 0 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -93> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -92> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:07.447 INFO:tasks.workunit.client.0.smithi001.stderr: -91> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -90> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -89> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -88> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -87> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -86> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -85> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -84> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -83> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -82> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -81> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 zone default found 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -80> 2024-03-09T14:51:07.303+0000 7f7832c7f980 4 Realm: () 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -79> 2024-03-09T14:51:07.303+0000 7f7832c7f980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -78> 2024-03-09T14:51:07.303+0000 7f7832c7f980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -77> 2024-03-09T14:51:07.303+0000 7f7832c7f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:07.448 INFO:tasks.workunit.client.0.smithi001.stderr: -76> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 zonegroup default 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -75> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -74> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -73> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -72> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -71> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -70> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -69> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 started sync module instance, tier type = 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -68> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -67> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=0 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -66> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=1 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -65> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=2 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -64> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=3 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -63> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=4 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -62> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=5 2024-03-09T14:51:07.449 INFO:tasks.workunit.client.0.smithi001.stderr: -61> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=6 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -60> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=7 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -59> 2024-03-09T14:51:07.331+0000 7f7832c7f980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -58> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding auth protocol: cephx 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -57> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding auth protocol: cephx 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -56> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding auth protocol: cephx 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -55> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: secure 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -54> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: crc 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -53> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: secure 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -52> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: crc 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -51> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: secure 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -50> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: crc 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -49> 2024-03-09T14:51:07.335+0000 7f76e6fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -48> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: crc 2024-03-09T14:51:07.450 INFO:tasks.workunit.client.0.smithi001.stderr: -47> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: secure 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -46> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: crc 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -45> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: secure 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -44> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: crc 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -43> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: secure 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -42> 2024-03-09T14:51:07.335+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -41> 2024-03-09T14:51:07.335+0000 7f7832c7f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -40> 2024-03-09T14:51:07.335+0000 7f7832c7f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -39> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command cache list hook 0x56446115a0f0 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -38> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command cache inspect hook 0x56446115a0f0 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -37> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command cache erase hook 0x56446115a0f0 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -36> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command cache zap hook 0x56446115a0f0 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -35> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 note: GC not initialized 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -34> 2024-03-09T14:51:07.335+0000 7f76e5fdb640 20 reqs_thread_entry: start 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -33> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command sync trace show hook 0x7f7818001d40 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -32> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command sync trace history hook 0x7f7818001d40 2024-03-09T14:51:07.451 INFO:tasks.workunit.client.0.smithi001.stderr: -31> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command sync trace active hook 0x7f7818001d40 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -30> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command sync trace active_short hook 0x7f7818001d40 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -29> 2024-03-09T14:51:07.335+0000 7f7832c7f980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -28> 2024-03-09T14:51:07.339+0000 7f7832c7f980 20 Filter name: none 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -27> 2024-03-09T14:51:07.339+0000 7f76577fe640 20 reqs_thread_entry: start 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -26> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -25> 2024-03-09T14:51:07.339+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -24> 2024-03-09T14:51:07.339+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -23> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -22> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -21> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 : miss 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -20> 2024-03-09T14:51:07.339+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -19> 2024-03-09T14:51:07.339+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -18> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -17> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:07.452 INFO:tasks.workunit.client.0.smithi001.stderr: -16> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -15> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -14> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -13> 2024-03-09T14:51:07.347+0000 7f7832c7f980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -12> 2024-03-09T14:51:07.347+0000 7f7832c7f980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -11> 2024-03-09T14:51:07.347+0000 7f7832c7f980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -10> 2024-03-09T14:51:07.347+0000 7f7832c7f980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23, ofs0, ns] 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -9> 2024-03-09T14:51:07.347+0000 7f77077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562446 cookie 94851686727232 notifier 6173 bl.length()=907 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -8> 2024-03-09T14:51:07.347+0000 7f77077fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -7> 2024-03-09T14:51:07.347+0000 7f77077fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -6> 2024-03-09T14:51:07.347+0000 7f77077fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -5> 2024-03-09T14:51:07.351+0000 7f7832c7f980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): orig_sources= new_sources= 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -4> 2024-03-09T14:51:07.351+0000 7f7832c7f980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): potential sources added=[] removed=[] 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -3> 2024-03-09T14:51:07.351+0000 7f7832c7f980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): orig_dests= new_dests= 2024-03-09T14:51:07.453 INFO:tasks.workunit.client.0.smithi001.stderr: -2> 2024-03-09T14:51:07.351+0000 7f7832c7f980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): potential dests added=[] removed=[] 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: -1> 2024-03-09T14:51:07.355+0000 7f7832c7f980 -1 ./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f7832c7f980 time 2024-03-09T14:51:07.356062+0000 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f7836554d7b] 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f7836554f2f] 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x56445f879c51] 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a5) [0x56445f89a285] 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: 5: main() 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f78353bfd90] 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: 7: __libc_start_main() 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: 8: _start() 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.454 INFO:tasks.workunit.client.0.smithi001.stderr: 0> 2024-03-09T14:51:07.355+0000 7f7832c7f980 -1 *** Caught signal (Aborted) ** 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f7832c7f980 thread_name:radosgw-admin 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f78353d8520] 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f7836554dd5] 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f7836554f2f] 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x56445f879c51] 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a5) [0x56445f89a285] 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 9: main() 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f78353bfd90] 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 11: __libc_start_main() 2024-03-09T14:51:07.455 INFO:tasks.workunit.client.0.smithi001.stderr: 12: _start() 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr:--- logging levels --- 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 none 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 lockdep 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 context 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 crush 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_balancer 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_locker 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log_expire 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_migrator 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 buffer 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 timer 2024-03-09T14:51:07.456 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 filer 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 striper 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 objecter 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rados 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_mirror 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_replay 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_pwl 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 journaler 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objectcacher 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 immutable_obj_cache 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 client 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 osd 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 optracker 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objclass 2024-03-09T14:51:07.457 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 filestore 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 journal 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 ms 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mon 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 0/10 monc 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 paxos 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 tp 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 auth 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 crypto 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 finisher 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 reserver 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 heartbeatmap 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 perfcounter 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 20/20 rgw 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_sync 2024-03-09T14:51:07.458 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_datacache 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_access 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_dbstore 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_flight 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_lifecycle 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 javaclient 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 asok 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 throttle 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 refs 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 compressor 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluestore 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluefs 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 bdev 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 kstore 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 4/ 5 rocksdb 2024-03-09T14:51:07.459 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 fuse 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 2/ 5 mgr 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mgrc 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 dpdk 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 eventtrace 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 prioritycache 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 test 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephfs_mirror 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephsqlite 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 crimson_interrupt 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_onode 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_odata 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_omap 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_tm 2024-03-09T14:51:07.460 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_t 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cleaner 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_epm 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_lba 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_fixedkv_tree 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cache 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_journal 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_device 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_backref 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 alienstore 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mclock 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cyanstore 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 ceph_exporter 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 memstore 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 trace 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: -2/-2 (syslog threshold) 2024-03-09T14:51:07.461 INFO:tasks.workunit.client.0.smithi001.stderr: 99/99 (stderr threshold) 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr:--- pthread ID / name mapping for recent threads --- 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f76577fe640 / http_manager 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f76e5fdb640 / http_manager 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f76e6fdd640 / rgw_dt_lg_renew 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f7705ffb640 / ms_dispatch 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f77077fe640 / io_context_pool 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f7715ffb640 / ms_dispatch 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f771ce09640 / ms_dispatch 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f771e60c640 / io_context_pool 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f782ff96640 / msgr-worker-2 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f7830797640 / msgr-worker-1 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f7830f98640 / msgr-worker-0 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f7831799640 / io_context_pool 2024-03-09T14:51:07.462 INFO:tasks.workunit.client.0.smithi001.stderr: 7f7832c7f980 / radosgw-admin 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: max_recent 500 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: max_new 1000 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: log_file /var/log/ceph/ceph-client.admin.87205.log 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr:--- end dump of recent events --- 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr:--- begin dump of recent events --- 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: -499> 2024-03-09T14:51:06.835+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: -498> 2024-03-09T14:51:06.835+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: -497> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): get_auth_request con 0x7f7818002100 auth_method 0 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: -496> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: -495> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: -494> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: -493> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): handle_auth_done global_id 6543 payload 274 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: -492> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient: _finish_hunting 0 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: -491> 2024-03-09T14:51:06.835+0000 7f7830f98640 1 monclient: found mon.noname-a 2024-03-09T14:51:07.463 INFO:tasks.workunit.client.0.smithi001.stderr: -490> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr: -489> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr: -488> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr: -487> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: dump: 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.464 INFO:tasks.workunit.client.0.smithi001.stderr: -486> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: _finish_auth 0 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr: -485> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: _check_auth_tickets 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr: -484> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr: -483> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 monclient: authenticate success, global_id 6543 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr: -482> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: get_monmap_and_config success 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr: -481> 2024-03-09T14:51:06.835+0000 7f7832c7f980 4 set_mon_vals no callback set 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr: -480> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr: -479> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr: -478> 2024-03-09T14:51:06.835+0000 7f771e60c640 10 monclient: dump: 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.465 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr: -477> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: shutdown 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr: -476> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 asok(0x564460e7dbb0) unregister_commands rotate-key 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr: -475> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: build_initial_monmap 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr: -474> 2024-03-09T14:51:06.835+0000 7f7832c7f980 1 build_initial for_mkfs: 0 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr: -473> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: monmap: 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:07.466 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: -472> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding auth protocol: cephx 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: -471> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding auth protocol: cephx 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: -470> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding auth protocol: cephx 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: -469> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: secure 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: -468> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: crc 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: -467> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: secure 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: -466> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: crc 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: -465> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: secure 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: -464> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: crc 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: -463> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: crc 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: -462> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: secure 2024-03-09T14:51:07.467 INFO:tasks.workunit.client.0.smithi001.stderr: -461> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: crc 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -460> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: secure 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -459> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: crc 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -458> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461088c40) adding con mode: secure 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -457> 2024-03-09T14:51:06.835+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -456> 2024-03-09T14:51:06.835+0000 7f7832c7f980 1 librados: starting msgr at 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -455> 2024-03-09T14:51:06.835+0000 7f7832c7f980 1 librados: starting objecter 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -454> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command objecter_requests hook 0x564460f3bc20 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -453> 2024-03-09T14:51:06.835+0000 7f7832c7f980 1 librados: setting wanted keys 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -452> 2024-03-09T14:51:06.835+0000 7f7832c7f980 1 librados: calling monclient init 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -451> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: init 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -450> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding auth protocol: cephx 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -449> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding auth protocol: cephx 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -448> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding auth protocol: cephx 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -447> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: secure 2024-03-09T14:51:07.468 INFO:tasks.workunit.client.0.smithi001.stderr: -446> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: crc 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -445> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: secure 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -444> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: crc 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -443> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: secure 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -442> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: crc 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -441> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: crc 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -440> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: secure 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -439> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: crc 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -438> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: secure 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -437> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: crc 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -436> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 AuthRegistry(0x564461086e00) adding con mode: secure 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -435> 2024-03-09T14:51:06.835+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -434> 2024-03-09T14:51:06.835+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -433> 2024-03-09T14:51:06.835+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command rotate-key hook 0x564461086f48 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -432> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:07.469 INFO:tasks.workunit.client.0.smithi001.stderr: -431> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient: _add_conns ranks=[0,1,2] 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -430> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-a con 0x56446108aef0 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -429> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-b con 0x5644610b5d80 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -428> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-c con 0x7f7818002100 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -427> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -426> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -425> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -424> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -423> 2024-03-09T14:51:06.835+0000 7f7832c7f980 10 monclient(hunting): authenticate will time out at 2360.583696s 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -422> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): get_auth_request con 0x56446108aef0 auth_method 0 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -421> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -420> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -419> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -418> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): get_auth_request con 0x7f7818002100 auth_method 0 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -417> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.470 INFO:tasks.workunit.client.0.smithi001.stderr: -416> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -415> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -414> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -413> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -412> 2024-03-09T14:51:06.835+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -411> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -410> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -409> 2024-03-09T14:51:06.835+0000 7f7830f98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -408> 2024-03-09T14:51:06.839+0000 7f7830797640 10 monclient(hunting): handle_auth_done global_id 6546 payload 824 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -407> 2024-03-09T14:51:06.839+0000 7f7830797640 10 monclient: _finish_hunting 0 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -406> 2024-03-09T14:51:06.839+0000 7f7830797640 1 monclient: found mon.noname-a 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -405> 2024-03-09T14:51:06.839+0000 7f7830797640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -404> 2024-03-09T14:51:06.839+0000 7f782ff96640 10 monclient: get_auth_request con 0x5644610b5d80 auth_method 0 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -403> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.471 INFO:tasks.workunit.client.0.smithi001.stderr: -402> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr: -401> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: dump: 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr: -400> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: _finish_auth 0 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr: -399> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: _check_auth_tickets 2024-03-09T14:51:07.472 INFO:tasks.workunit.client.0.smithi001.stderr: -398> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr: -397> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr: -396> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr: -395> 2024-03-09T14:51:06.839+0000 7f771ce09640 10 monclient: dump: 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr: -394> 2024-03-09T14:51:06.839+0000 7f7832c7f980 5 monclient: authenticate success, global_id 6546 2024-03-09T14:51:07.473 INFO:tasks.workunit.client.0.smithi001.stderr: -393> 2024-03-09T14:51:06.839+0000 7f7832c7f980 10 monclient: _renew_subs 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -392> 2024-03-09T14:51:06.839+0000 7f7832c7f980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -391> 2024-03-09T14:51:06.839+0000 7f771e60c640 4 set_mon_vals no callback set 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -390> 2024-03-09T14:51:06.839+0000 7f7832c7f980 10 monclient: _renew_subs 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -389> 2024-03-09T14:51:06.839+0000 7f7832c7f980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -388> 2024-03-09T14:51:06.839+0000 7f7832c7f980 1 librados: init done 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -387> 2024-03-09T14:51:06.839+0000 7f771ce09640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -386> 2024-03-09T14:51:06.839+0000 7f771ce09640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -385> 2024-03-09T14:51:06.839+0000 7f771ce09640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -384> 2024-03-09T14:51:06.839+0000 7f782ff96640 10 monclient: get_auth_request con 0x7f7718066040 auth_method 0 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -383> 2024-03-09T14:51:06.839+0000 7f7830f98640 10 monclient: get_auth_request con 0x5644610c9010 auth_method 0 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -382> 2024-03-09T14:51:06.843+0000 7f7830797640 10 monclient: get_auth_request con 0x5644610d2300 auth_method 0 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -381> 2024-03-09T14:51:06.843+0000 7f782ff96640 10 monclient: get_auth_request con 0x5644610d7390 auth_method 0 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -380> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: build_initial_monmap 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -379> 2024-03-09T14:51:06.847+0000 7f7832c7f980 1 build_initial for_mkfs: 0 2024-03-09T14:51:07.474 INFO:tasks.workunit.client.0.smithi001.stderr: -378> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: monmap: 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr: -377> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding auth protocol: cephx 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr: -376> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding auth protocol: cephx 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr: -375> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding auth protocol: cephx 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr: -374> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: secure 2024-03-09T14:51:07.475 INFO:tasks.workunit.client.0.smithi001.stderr: -373> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: crc 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -372> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: secure 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -371> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: crc 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -370> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: secure 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -369> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: crc 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -368> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: crc 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -367> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: secure 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -366> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: crc 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -365> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: secure 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -364> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: crc 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -363> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610db360) adding con mode: secure 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -362> 2024-03-09T14:51:06.847+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -361> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command objecter_requests cmddesc objecter_requests hook 0x5644610dc5a0 EEXIST 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -360> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: init 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -359> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding auth protocol: cephx 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -358> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding auth protocol: cephx 2024-03-09T14:51:07.476 INFO:tasks.workunit.client.0.smithi001.stderr: -357> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding auth protocol: cephx 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -356> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: secure 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -355> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: crc 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -354> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: secure 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -353> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: crc 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -352> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: secure 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -351> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: crc 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -350> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: crc 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -349> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: secure 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -348> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: crc 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -347> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: secure 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -346> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: crc 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -345> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 AuthRegistry(0x5644610da348) adding con mode: secure 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -344> 2024-03-09T14:51:06.847+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -343> 2024-03-09T14:51:06.847+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.477 INFO:tasks.workunit.client.0.smithi001.stderr: -342> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command rotate-key cmddesc rotate-key hook 0x5644610da490 EEXIST 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -341> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -340> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: _add_conns ranks=[0,2,1] 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -339> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-a con 0x564461105740 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -338> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-c con 0x5644611094c0 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -337> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-b con 0x564461100c50 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -336> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -335> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -334> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -333> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -332> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient(hunting): authenticate will time out at 2360.594160s 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -331> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): get_auth_request con 0x5644611094c0 auth_method 0 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -330> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -329> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.478 INFO:tasks.workunit.client.0.smithi001.stderr: -328> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -327> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): get_auth_request con 0x564461105740 auth_method 0 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -326> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -325> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -324> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -323> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -322> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -321> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -320> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -319> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -318> 2024-03-09T14:51:06.847+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -317> 2024-03-09T14:51:06.847+0000 7f7830f98640 10 monclient(hunting): get_auth_request con 0x564461100c50 auth_method 0 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -316> 2024-03-09T14:51:06.847+0000 7f7830f98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -315> 2024-03-09T14:51:06.847+0000 7f7830f98640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.479 INFO:tasks.workunit.client.0.smithi001.stderr: -314> 2024-03-09T14:51:06.847+0000 7f7830f98640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr: -313> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient(hunting): handle_auth_done global_id 6549 payload 824 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr: -312> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient: _finish_hunting 0 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr: -311> 2024-03-09T14:51:06.847+0000 7f7830797640 1 monclient: found mon.noname-a 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr: -310> 2024-03-09T14:51:06.847+0000 7f7830797640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr: -309> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr: -308> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr: -307> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: dump: 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.480 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr: -306> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: _finish_auth 0 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr: -305> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: _check_auth_tickets 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr: -304> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr: -303> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr: -302> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr: -301> 2024-03-09T14:51:06.847+0000 7f7715ffb640 10 monclient: dump: 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.481 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: -300> 2024-03-09T14:51:06.847+0000 7f7832c7f980 5 monclient: authenticate success, global_id 6549 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: -299> 2024-03-09T14:51:06.847+0000 7f7831799640 4 set_mon_vals no callback set 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: -298> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: _renew_subs 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: -297> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: -296> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: _renew_subs 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: -295> 2024-03-09T14:51:06.847+0000 7f7832c7f980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: -294> 2024-03-09T14:51:06.851+0000 7f7715ffb640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: -293> 2024-03-09T14:51:06.851+0000 7f7715ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: -292> 2024-03-09T14:51:06.851+0000 7f7715ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: -291> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: get_monmap_and_config 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: -290> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: build_initial_monmap 2024-03-09T14:51:07.482 INFO:tasks.workunit.client.0.smithi001.stderr: -289> 2024-03-09T14:51:06.851+0000 7f7832c7f980 1 build_initial for_mkfs: 0 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr: -288> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: monmap: 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr: -287> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding auth protocol: cephx 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr: -286> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding auth protocol: cephx 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr: -285> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding auth protocol: cephx 2024-03-09T14:51:07.483 INFO:tasks.workunit.client.0.smithi001.stderr: -284> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -283> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -282> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -281> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -280> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -279> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -278> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -277> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -276> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -275> 2024-03-09T14:51:06.851+0000 7f782ff96640 10 monclient: get_auth_request con 0x7f7708065f50 auth_method 0 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -274> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -273> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -272> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -271> 2024-03-09T14:51:06.851+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -270> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: init 2024-03-09T14:51:07.484 INFO:tasks.workunit.client.0.smithi001.stderr: -269> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding auth protocol: cephx 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -268> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding auth protocol: cephx 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -267> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding auth protocol: cephx 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -266> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: secure 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -265> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: crc 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -264> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: secure 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -263> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: crc 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -262> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: secure 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -261> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: crc 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -260> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: crc 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -259> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: secure 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -258> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: crc 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -257> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: secure 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -256> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: crc 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -255> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e83d80) adding con mode: secure 2024-03-09T14:51:07.485 INFO:tasks.workunit.client.0.smithi001.stderr: -254> 2024-03-09T14:51:06.851+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -253> 2024-03-09T14:51:06.851+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -252> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command rotate-key cmddesc rotate-key hook 0x7ffec1e83ec8 EEXIST 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -251> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -250> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: _add_conns ranks=[2,1,0] 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -249> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-c con 0x5644611572e0 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -248> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-b con 0x5644611576c0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -247> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-a con 0x5644611274f0 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -246> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -245> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -244> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -243> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -242> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient(hunting): authenticate will time out at 2360.598802s 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -241> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): get_auth_request con 0x5644611572e0 auth_method 0 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -240> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.486 INFO:tasks.workunit.client.0.smithi001.stderr: -239> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -238> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -237> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): get_auth_request con 0x5644611274f0 auth_method 0 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -236> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -235> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -234> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -233> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -232> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -231> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -230> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -229> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -228> 2024-03-09T14:51:06.851+0000 7f7830f98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -227> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient(hunting): handle_auth_done global_id 6170 payload 274 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -226> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient: _finish_hunting 0 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -225> 2024-03-09T14:51:06.851+0000 7f7830797640 1 monclient: found mon.noname-c 2024-03-09T14:51:07.487 INFO:tasks.workunit.client.0.smithi001.stderr: -224> 2024-03-09T14:51:06.851+0000 7f7830797640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr: -223> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr: -222> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr: -221> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: dump: 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr: -220> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: _finish_auth 0 2024-03-09T14:51:07.488 INFO:tasks.workunit.client.0.smithi001.stderr: -219> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: _check_auth_tickets 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr: -218> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr: -217> 2024-03-09T14:51:06.851+0000 7f7832c7f980 5 monclient: authenticate success, global_id 6170 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr: -216> 2024-03-09T14:51:06.851+0000 7f7832c7f980 10 monclient: get_monmap_and_config success 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr: -215> 2024-03-09T14:51:06.851+0000 7f7832c7f980 4 set_mon_vals no callback set 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr: -214> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr: -213> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr: -212> 2024-03-09T14:51:06.851+0000 7f77077fe640 10 monclient: dump: 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.489 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr: -211> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: shutdown 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr: -210> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: build_initial_monmap 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr: -209> 2024-03-09T14:51:06.855+0000 7f7832c7f980 1 build_initial for_mkfs: 0 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr: -208> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: monmap: 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:07.490 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -207> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding auth protocol: cephx 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -206> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding auth protocol: cephx 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -205> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding auth protocol: cephx 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -204> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -203> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -202> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -201> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -200> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -199> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -198> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -197> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -196> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -195> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -194> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: crc 2024-03-09T14:51:07.491 INFO:tasks.workunit.client.0.smithi001.stderr: -193> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461122b30) adding con mode: secure 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -192> 2024-03-09T14:51:06.855+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -191> 2024-03-09T14:51:06.855+0000 7f7832c7f980 1 librados: starting msgr at 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -190> 2024-03-09T14:51:06.855+0000 7f7832c7f980 1 librados: starting objecter 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -189> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command objecter_requests cmddesc objecter_requests hook 0x5644611519c0 EEXIST 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -188> 2024-03-09T14:51:06.855+0000 7f7832c7f980 1 librados: setting wanted keys 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -187> 2024-03-09T14:51:06.855+0000 7f7832c7f980 1 librados: calling monclient init 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -186> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: init 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -185> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding auth protocol: cephx 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -184> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding auth protocol: cephx 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -183> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding auth protocol: cephx 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -182> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: secure 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -181> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: crc 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -180> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: secure 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -179> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: crc 2024-03-09T14:51:07.492 INFO:tasks.workunit.client.0.smithi001.stderr: -178> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: secure 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -177> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: crc 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -176> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: crc 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -175> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: secure 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -174> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: crc 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -173> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: secure 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -172> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: crc 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -171> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 AuthRegistry(0x564461103130) adding con mode: secure 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -170> 2024-03-09T14:51:06.855+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -169> 2024-03-09T14:51:06.855+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -168> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command rotate-key cmddesc rotate-key hook 0x564461103278 EEXIST 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -167> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -166> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: _add_conns ranks=[2,0,1] 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -165> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-c con 0x5644611274f0 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -164> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-a con 0x5644611572e0 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:07.493 INFO:tasks.workunit.client.0.smithi001.stderr: -163> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): picked mon.noname-b con 0x564461150180 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -162> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -161> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -160> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -159> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -158> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient(hunting): authenticate will time out at 2360.601789s 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -157> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): get_auth_request con 0x5644611274f0 auth_method 0 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -156> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -155> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -154> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -153> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): get_auth_request con 0x5644611572e0 auth_method 0 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -152> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -151> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -150> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -149> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.494 INFO:tasks.workunit.client.0.smithi001.stderr: -148> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr: -147> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr: -146> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr: -145> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr: -144> 2024-03-09T14:51:06.855+0000 7f782ff96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr: -143> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient(hunting): handle_auth_done global_id 6173 payload 824 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr: -142> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient: _finish_hunting 0 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr: -141> 2024-03-09T14:51:06.855+0000 7f7830797640 1 monclient: found mon.noname-c 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr: -140> 2024-03-09T14:51:06.855+0000 7f7830797640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr: -139> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr: -138> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr: -137> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: dump: 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.495 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr: -136> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: _finish_auth 0 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr: -135> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: _check_auth_tickets 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr: -134> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr: -133> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr: -132> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr: -131> 2024-03-09T14:51:06.855+0000 7f7705ffb640 10 monclient: dump: 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.496 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr: -130> 2024-03-09T14:51:06.855+0000 7f77077fe640 4 set_mon_vals no callback set 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr: -129> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 monclient: authenticate success, global_id 6173 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr: -128> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: _renew_subs 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr: -127> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr: -126> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: _renew_subs 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr: -125> 2024-03-09T14:51:06.855+0000 7f7832c7f980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr: -124> 2024-03-09T14:51:06.855+0000 7f7832c7f980 1 librados: init done 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr: -123> 2024-03-09T14:51:06.855+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command cr dump hook 0x7f78200b2f98 2024-03-09T14:51:07.497 INFO:tasks.workunit.client.0.smithi001.stderr: -122> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -121> 2024-03-09T14:51:06.859+0000 7f7705ffb640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -120> 2024-03-09T14:51:06.859+0000 7f7705ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -119> 2024-03-09T14:51:06.859+0000 7f7705ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -118> 2024-03-09T14:51:06.859+0000 7f7830f98640 10 monclient: get_auth_request con 0x5644611706c0 auth_method 0 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -117> 2024-03-09T14:51:06.859+0000 7f782ff96640 10 monclient: get_auth_request con 0x7f76f8066070 auth_method 0 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -116> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -115> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 realm 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -114> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -113> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -112> 2024-03-09T14:51:06.859+0000 7f7832c7f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -111> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -110> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -109> 2024-03-09T14:51:06.859+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -108> 2024-03-09T14:51:06.859+0000 7f7830797640 10 monclient: get_auth_request con 0x564461174910 auth_method 0 2024-03-09T14:51:07.498 INFO:tasks.workunit.client.0.smithi001.stderr: -107> 2024-03-09T14:51:06.863+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -106> 2024-03-09T14:51:06.863+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -105> 2024-03-09T14:51:06.863+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -104> 2024-03-09T14:51:06.863+0000 7f7832c7f980 20 searching for the correct realm 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -103> 2024-03-09T14:51:06.863+0000 7f782ff96640 10 monclient: get_auth_request con 0x564461179b50 auth_method 0 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -102> 2024-03-09T14:51:06.863+0000 7f7830f98640 10 monclient: get_auth_request con 0x7f7824008040 auth_method 0 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -101> 2024-03-09T14:51:06.867+0000 7f7830797640 10 monclient: get_auth_request con 0x564460f17260 auth_method 0 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -100> 2024-03-09T14:51:06.867+0000 7f782ff96640 0 -- 172.21.15.1:0/2595872606 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f78200a8bc0 msgr2=0x7f7820108af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -99> 2024-03-09T14:51:06.867+0000 7f782ff96640 10 monclient: get_auth_request con 0x7f78200a8bc0 auth_method 0 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -98> 2024-03-09T14:51:07.071+0000 7f782ff96640 10 monclient: get_auth_request con 0x7f78200a8bc0 auth_method 0 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -97> 2024-03-09T14:51:07.075+0000 7f7830f98640 10 monclient: get_auth_request con 0x7f782404c040 auth_method 0 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -96> 2024-03-09T14:51:07.083+0000 7f7830797640 10 monclient: get_auth_request con 0x7f782403b930 auth_method 0 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -95> 2024-03-09T14:51:07.083+0000 7f7830797640 0 -- 172.21.15.1:0/2595872606 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f782403b930 msgr2=0x7f782402d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -94> 2024-03-09T14:51:07.287+0000 7f7830797640 10 monclient: get_auth_request con 0x7f782403b930 auth_method 0 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -93> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:07.499 INFO:tasks.workunit.client.0.smithi001.stderr: -92> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -91> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -90> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -89> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -88> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -87> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -86> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -85> 2024-03-09T14:51:07.299+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -84> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -83> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -82> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -81> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 zone default found 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -80> 2024-03-09T14:51:07.303+0000 7f7832c7f980 4 Realm: () 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -79> 2024-03-09T14:51:07.303+0000 7f7832c7f980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -78> 2024-03-09T14:51:07.303+0000 7f7832c7f980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:07.500 INFO:tasks.workunit.client.0.smithi001.stderr: -77> 2024-03-09T14:51:07.303+0000 7f7832c7f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -76> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 zonegroup default 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -75> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -74> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -73> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -72> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -71> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -70> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -69> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 started sync module instance, tier type = 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -68> 2024-03-09T14:51:07.303+0000 7f7832c7f980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -67> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=0 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -66> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=1 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -65> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=2 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -64> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=3 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -63> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=4 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -62> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=5 2024-03-09T14:51:07.501 INFO:tasks.workunit.client.0.smithi001.stderr: -61> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=6 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -60> 2024-03-09T14:51:07.331+0000 7f7832c7f980 20 add_watcher() i=7 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -59> 2024-03-09T14:51:07.331+0000 7f7832c7f980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -58> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding auth protocol: cephx 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -57> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding auth protocol: cephx 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -56> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding auth protocol: cephx 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -55> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: secure 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -54> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: crc 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -53> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: secure 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -52> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: crc 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -51> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: secure 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -50> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: crc 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -49> 2024-03-09T14:51:07.335+0000 7f76e6fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -48> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: crc 2024-03-09T14:51:07.502 INFO:tasks.workunit.client.0.smithi001.stderr: -47> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: secure 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -46> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: crc 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -45> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: secure 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -44> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: crc 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -43> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 AuthRegistry(0x7ffec1e84210) adding con mode: secure 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -42> 2024-03-09T14:51:07.335+0000 7f7832c7f980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -41> 2024-03-09T14:51:07.335+0000 7f7832c7f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -40> 2024-03-09T14:51:07.335+0000 7f7832c7f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -39> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command cache list hook 0x56446115a0f0 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -38> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command cache inspect hook 0x56446115a0f0 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -37> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command cache erase hook 0x56446115a0f0 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -36> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command cache zap hook 0x56446115a0f0 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -35> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 note: GC not initialized 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -34> 2024-03-09T14:51:07.335+0000 7f76e5fdb640 20 reqs_thread_entry: start 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -33> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command sync trace show hook 0x7f7818001d40 2024-03-09T14:51:07.503 INFO:tasks.workunit.client.0.smithi001.stderr: -32> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command sync trace history hook 0x7f7818001d40 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -31> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command sync trace active hook 0x7f7818001d40 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -30> 2024-03-09T14:51:07.335+0000 7f7832c7f980 5 asok(0x564460e7dbb0) register_command sync trace active_short hook 0x7f7818001d40 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -29> 2024-03-09T14:51:07.335+0000 7f7832c7f980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -28> 2024-03-09T14:51:07.339+0000 7f7832c7f980 20 Filter name: none 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -27> 2024-03-09T14:51:07.339+0000 7f76577fe640 20 reqs_thread_entry: start 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -26> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -25> 2024-03-09T14:51:07.339+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -24> 2024-03-09T14:51:07.339+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -23> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -22> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -21> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 : miss 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -20> 2024-03-09T14:51:07.339+0000 7f7832c7f980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -19> 2024-03-09T14:51:07.339+0000 7f7832c7f980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:07.504 INFO:tasks.workunit.client.0.smithi001.stderr: -18> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -17> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -16> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -15> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -14> 2024-03-09T14:51:07.339+0000 7f7832c7f980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -13> 2024-03-09T14:51:07.347+0000 7f7832c7f980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -12> 2024-03-09T14:51:07.347+0000 7f7832c7f980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -11> 2024-03-09T14:51:07.347+0000 7f7832c7f980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -10> 2024-03-09T14:51:07.347+0000 7f7832c7f980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23, ofs0, ns] 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -9> 2024-03-09T14:51:07.347+0000 7f77077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562446 cookie 94851686727232 notifier 6173 bl.length()=907 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -8> 2024-03-09T14:51:07.347+0000 7f77077fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -7> 2024-03-09T14:51:07.347+0000 7f77077fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -6> 2024-03-09T14:51:07.347+0000 7f77077fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -5> 2024-03-09T14:51:07.351+0000 7f7832c7f980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): orig_sources= new_sources= 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -4> 2024-03-09T14:51:07.351+0000 7f7832c7f980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): potential sources added=[] removed=[] 2024-03-09T14:51:07.505 INFO:tasks.workunit.client.0.smithi001.stderr: -3> 2024-03-09T14:51:07.351+0000 7f7832c7f980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): orig_dests= new_dests= 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: -2> 2024-03-09T14:51:07.351+0000 7f7832c7f980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): potential dests added=[] removed=[] 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: -1> 2024-03-09T14:51:07.355+0000 7f7832c7f980 -1 ./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f7832c7f980 time 2024-03-09T14:51:07.356062+0000 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f7836554d7b] 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f7836554f2f] 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x56445f879c51] 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a5) [0x56445f89a285] 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: 5: main() 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f78353bfd90] 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: 7: __libc_start_main() 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: 8: _start() 2024-03-09T14:51:07.506 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 0> 2024-03-09T14:51:07.355+0000 7f7832c7f980 -1 *** Caught signal (Aborted) ** 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f7832c7f980 thread_name:radosgw-admin 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f78353d8520] 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f7836554dd5] 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f7836554f2f] 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x56445f879c51] 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a5) [0x56445f89a285] 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 9: main() 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f78353bfd90] 2024-03-09T14:51:07.507 INFO:tasks.workunit.client.0.smithi001.stderr: 11: __libc_start_main() 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 12: _start() 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr:--- logging levels --- 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 none 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 lockdep 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 context 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 crush 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_balancer 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_locker 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log_expire 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_migrator 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 buffer 2024-03-09T14:51:07.508 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 timer 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 filer 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 striper 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 objecter 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rados 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_mirror 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_replay 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_pwl 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 journaler 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objectcacher 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 immutable_obj_cache 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 client 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 osd 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 optracker 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objclass 2024-03-09T14:51:07.509 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 filestore 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 journal 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 ms 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mon 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 0/10 monc 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 paxos 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 tp 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 auth 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 crypto 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 finisher 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 reserver 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 heartbeatmap 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 perfcounter 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 20/20 rgw 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_sync 2024-03-09T14:51:07.510 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_datacache 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_access 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_dbstore 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_flight 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_lifecycle 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 javaclient 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 asok 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 throttle 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 refs 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 compressor 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluestore 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluefs 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 bdev 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 kstore 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 4/ 5 rocksdb 2024-03-09T14:51:07.511 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 fuse 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 2/ 5 mgr 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mgrc 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 dpdk 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 eventtrace 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 prioritycache 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 test 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephfs_mirror 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephsqlite 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 crimson_interrupt 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_onode 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_odata 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_omap 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_tm 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_t 2024-03-09T14:51:07.512 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cleaner 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_epm 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_lba 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_fixedkv_tree 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cache 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_journal 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_device 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_backref 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 alienstore 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mclock 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cyanstore 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 ceph_exporter 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 memstore 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 trace 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: -2/-2 (syslog threshold) 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr: 99/99 (stderr threshold) 2024-03-09T14:51:07.513 INFO:tasks.workunit.client.0.smithi001.stderr:--- pthread ID / name mapping for recent threads --- 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f76577fe640 / http_manager 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f76e5fdb640 / http_manager 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f76e6fdd640 / rgw_dt_lg_renew 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f7705ffb640 / ms_dispatch 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f77077fe640 / io_context_pool 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f7715ffb640 / ms_dispatch 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f771ce09640 / ms_dispatch 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f771e60c640 / io_context_pool 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f782ff96640 / msgr-worker-2 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f7830797640 / msgr-worker-1 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f7830f98640 / msgr-worker-0 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f7831799640 / io_context_pool 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: 7f7832c7f980 / radosgw-admin 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: max_recent 500 2024-03-09T14:51:07.514 INFO:tasks.workunit.client.0.smithi001.stderr: max_new 1000 2024-03-09T14:51:07.515 INFO:tasks.workunit.client.0.smithi001.stderr: log_file /var/lib/ceph/crash/2024-03-09T14:51:07.361438Z_12f377c5-7eb1-476c-83ae-c0904597df39/log 2024-03-09T14:51:07.515 INFO:tasks.workunit.client.0.smithi001.stderr:--- end dump of recent events --- 2024-03-09T14:51:07.515 INFO:tasks.workunit.client.0.smithi001.stderr:Aborted 2024-03-09T14:51:07.515 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket abort-at-block-writes 2024-03-09T14:51:07.515 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:07.515 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:07.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.443+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.447+0000 7fecf59c3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.447+0000 7fecf59c3980 20 realm 2024-03-09T14:51:07.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.447+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.447+0000 7fecf59c3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.447+0000 7fecf59c3980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:07.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.447+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.447+0000 7fecf59c3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.447+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.447+0000 7fecf59c3980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.447+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.447+0000 7fecf59c3980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.447+0000 7fecf59c3980 20 searching for the correct realm 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.455+0000 7fecf3cdc640 0 -- 172.21.15.1:0/2120352614 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x5585d9e90350 msgr2=0x5585d9e927a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.463+0000 7fecf2cda640 0 -- 172.21.15.1:0/2120352614 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fece405e970 msgr2=0x7fece40555b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 zone default found 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 4 Realm: () 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 zonegroup default 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 started sync module instance, tier type = 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.471+0000 7fecf59c3980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.487+0000 7fecf59c3980 20 add_watcher() i=0 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.487+0000 7fecf59c3980 20 add_watcher() i=1 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.487+0000 7fecf59c3980 20 add_watcher() i=2 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.487+0000 7fecf59c3980 20 add_watcher() i=3 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.487+0000 7fecf59c3980 20 add_watcher() i=4 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.487+0000 7fecf59c3980 20 add_watcher() i=5 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.487+0000 7fecf59c3980 20 add_watcher() i=6 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.491+0000 7fecf59c3980 20 add_watcher() i=7 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.491+0000 7fecf59c3980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.491+0000 7feba9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:07.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 5 note: GC not initialized 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7feba8fe1640 20 reqs_thread_entry: start 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 Filter name: none 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7feb227fc640 20 reqs_thread_entry: start 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 rados_obj.operate() r=0 bl.length=435 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 rados_obj.operate() r=0 bl.length=435 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:07.524 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.495+0000 7fecf59c3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.499+0000 7feba9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:07.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.503+0000 7fecf59c3980 20 remove_watcher() i=0 2024-03-09T14:51:07.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.503+0000 7fecf59c3980 2 removed watcher, disabling cache 2024-03-09T14:51:07.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.503+0000 7fecf59c3980 20 remove_watcher() i=1 2024-03-09T14:51:07.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.507+0000 7fecf59c3980 20 remove_watcher() i=2 2024-03-09T14:51:07.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.511+0000 7fecf59c3980 20 remove_watcher() i=3 2024-03-09T14:51:07.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.511+0000 7fecf59c3980 20 remove_watcher() i=4 2024-03-09T14:51:07.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.511+0000 7fecf59c3980 20 remove_watcher() i=5 2024-03-09T14:51:07.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.515+0000 7fecf59c3980 20 remove_watcher() i=6 2024-03-09T14:51:07.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.515+0000 7fecf59c3980 20 remove_watcher() i=7 2024-03-09T14:51:07.532 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.532 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:07.532 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: abort-at-block-writes id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:51:07.532 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:07.533 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:07.533 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:07.574 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:07.574 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:07.642 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket abort-at-block-writes --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2024-03-09T14:51:07.679 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:07.680 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:07.726 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.723+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.727+0000 7fb7d14a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.727+0000 7fb7d14a7980 20 realm 2024-03-09T14:51:07.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.727+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.727+0000 7fb7d14a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.727+0000 7fb7d14a7980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:07.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.727+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.727+0000 7fb7d14a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:07.729 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.727+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.727+0000 7fb7d14a7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:07.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.727+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:07.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.727+0000 7fb7d14a7980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:07.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.727+0000 7fb7d14a7980 20 searching for the correct realm 2024-03-09T14:51:07.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.735+0000 7fb7cf7c0640 0 -- 172.21.15.1:0/1861001548 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7fb7c40b64b0 msgr2=0x7fb7c40dc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:07.954 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:07.951+0000 7fb7cefbf640 0 -- 172.21.15.1:0/1861001548 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x55d661f91a40 msgr2=0x55d661f93e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:08.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.159+0000 7fb7d14a7980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:08.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.159+0000 7fb7d14a7980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:08.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.159+0000 7fb7d14a7980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:08.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.159+0000 7fb7d14a7980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:08.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.159+0000 7fb7d14a7980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:08.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.159+0000 7fb7d14a7980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:08.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.159+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:08.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.159+0000 7fb7d14a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:08.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.159+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:08.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.159+0000 7fb7d14a7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:08.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.159+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:08.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:08.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 20 zone default found 2024-03-09T14:51:08.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 4 Realm: () 2024-03-09T14:51:08.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:08.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:08.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:08.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 20 zonegroup default 2024-03-09T14:51:08.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:08.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:08.165 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:08.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:08.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:08.166 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:08.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 20 started sync module instance, tier type = 2024-03-09T14:51:08.167 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.163+0000 7fb7d14a7980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:08.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.187+0000 7fb7d14a7980 20 add_watcher() i=0 2024-03-09T14:51:08.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.187+0000 7fb7d14a7980 20 add_watcher() i=1 2024-03-09T14:51:08.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.187+0000 7fb7d14a7980 20 add_watcher() i=2 2024-03-09T14:51:08.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.187+0000 7fb7d14a7980 20 add_watcher() i=3 2024-03-09T14:51:08.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.187+0000 7fb7d14a7980 20 add_watcher() i=4 2024-03-09T14:51:08.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.187+0000 7fb7d14a7980 20 add_watcher() i=5 2024-03-09T14:51:08.192 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.187+0000 7fb7d14a7980 20 add_watcher() i=6 2024-03-09T14:51:08.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.191+0000 7fb7d14a7980 20 add_watcher() i=7 2024-03-09T14:51:08.193 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.191+0000 7fb7d14a7980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:08.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.195+0000 7fb6857e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:08.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.195+0000 7fb7d14a7980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:08.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.195+0000 7fb7d14a7980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:08.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.195+0000 7fb7d14a7980 5 note: GC not initialized 2024-03-09T14:51:08.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.195+0000 7fb5fffff640 20 reqs_thread_entry: start 2024-03-09T14:51:08.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.195+0000 7fb7d14a7980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:08.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.195+0000 7fb7d14a7980 20 Filter name: none 2024-03-09T14:51:08.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.195+0000 7fb5fdffb640 20 reqs_thread_entry: start 2024-03-09T14:51:08.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2024-03-09T14:51:08.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:08.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:08.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2024-03-09T14:51:08.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2024-03-09T14:51:08.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 : miss 2024-03-09T14:51:08.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:08.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 20 rados_obj.operate() r=0 bl.length=435 2024-03-09T14:51:08.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:08.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:08.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:08.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:08.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 2024-03-09T14:51:08.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.199+0000 7fb7d14a7980 0 INFO: RGWReshardLock::lock found lock on abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to be held by another RGW process; skipping for now 2024-03-09T14:51:08.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.203+0000 7fb6857e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:08.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.207+0000 7fb7d14a7980 20 remove_watcher() i=0 2024-03-09T14:51:08.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.207+0000 7fb7d14a7980 2 removed watcher, disabling cache 2024-03-09T14:51:08.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.211+0000 7fb7d14a7980 20 remove_watcher() i=1 2024-03-09T14:51:08.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.211+0000 7fb7d14a7980 20 remove_watcher() i=2 2024-03-09T14:51:08.217 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.215+0000 7fb7d14a7980 20 remove_watcher() i=3 2024-03-09T14:51:08.220 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.215+0000 7fb7d14a7980 20 remove_watcher() i=4 2024-03-09T14:51:08.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.219+0000 7fb7d14a7980 20 remove_watcher() i=5 2024-03-09T14:51:08.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.219+0000 7fb7d14a7980 20 remove_watcher() i=6 2024-03-09T14:51:08.225 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:08.223+0000 7fb7d14a7980 20 remove_watcher() i=7 2024-03-09T14:51:08.250 INFO:tasks.workunit.client.0.smithi001.stderr:waiting 30 seconds for reshard lock to expire... 2024-03-09T14:51:38.269 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket abort-at-block-writes --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2024-03-09T14:51:38.308 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:38.308 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:38.356 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.354+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.354+0000 7fb9c68d0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.354+0000 7fb9c68d0980 20 realm 2024-03-09T14:51:38.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.354+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.354+0000 7fb9c68d0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.354+0000 7fb9c68d0980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:38.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.354+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.354+0000 7fb9c68d0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.358 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.354+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.358+0000 7fb9c68d0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:38.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.358+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.358+0000 7fb9c68d0980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:38.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.358+0000 7fb9c68d0980 20 searching for the correct realm 2024-03-09T14:51:38.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.362+0000 7fb9c3be7640 0 -- 172.21.15.1:0/3529840193 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x55de4b3fdf80 msgr2=0x55de4b400440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:38.374 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.370+0000 7fb9c43e8640 0 -- 172.21.15.1:0/3529840193 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fb9b80747e0 msgr2=0x7fb9b8067fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:38.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.378+0000 7fb9c68d0980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:38.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.378+0000 7fb9c68d0980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:38.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.378+0000 7fb9c68d0980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:38.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.378+0000 7fb9c68d0980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:38.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.378+0000 7fb9c68d0980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:38.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.378+0000 7fb9c68d0980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:38.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.378+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.378+0000 7fb9c68d0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.378+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.378+0000 7fb9c68d0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:38.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.378+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:38.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 20 zone default found 2024-03-09T14:51:38.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 4 Realm: () 2024-03-09T14:51:38.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:38.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:38.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:38.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 20 zonegroup default 2024-03-09T14:51:38.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 20 started sync module instance, tier type = 2024-03-09T14:51:38.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.382+0000 7fb9c68d0980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:38.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.398+0000 7fb9c68d0980 20 add_watcher() i=0 2024-03-09T14:51:38.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.398+0000 7fb9c68d0980 20 add_watcher() i=1 2024-03-09T14:51:38.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.398+0000 7fb9c68d0980 20 add_watcher() i=2 2024-03-09T14:51:38.402 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.398+0000 7fb9c68d0980 20 add_watcher() i=3 2024-03-09T14:51:38.402 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.398+0000 7fb9c68d0980 20 add_watcher() i=4 2024-03-09T14:51:38.402 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.398+0000 7fb9c68d0980 20 add_watcher() i=5 2024-03-09T14:51:38.402 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.398+0000 7fb9c68d0980 20 add_watcher() i=6 2024-03-09T14:51:38.403 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.398+0000 7fb9c68d0980 20 add_watcher() i=7 2024-03-09T14:51:38.403 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.398+0000 7fb9c68d0980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:38.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb87afdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:38.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:38.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:38.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 5 note: GC not initialized 2024-03-09T14:51:38.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb879fdb640 20 reqs_thread_entry: start 2024-03-09T14:51:38.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:38.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 20 Filter name: none 2024-03-09T14:51:38.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb7eb7fe640 20 reqs_thread_entry: start 2024-03-09T14:51:38.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2024-03-09T14:51:38.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:38.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2024-03-09T14:51:38.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2024-03-09T14:51:38.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 : miss 2024-03-09T14:51:38.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 20 rados_obj.operate() r=0 bl.length=435 2024-03-09T14:51:38.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:38.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:38.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:38.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:38.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.402+0000 7fb9c68d0980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 2024-03-09T14:51:38.409 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.406+0000 7fb9c68d0980 10 init_target_layout removing existing target index objects from a previous reshard attempt 2024-03-09T14:51:38.417 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.414+0000 7fb9c68d0980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:38.417 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.414+0000 7fb9c68d0980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:38.417 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.414+0000 7fb9c68d0980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:38.417 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.414+0000 7fb9c68d0980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23, ofs0, ns] 2024-03-09T14:51:38.418 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.414+0000 7fb89b7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562447 cookie 94413233593664 notifier 6612 bl.length()=907 2024-03-09T14:51:38.418 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.414+0000 7fb89b7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:38.418 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.414+0000 7fb89b7fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:38.418 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.414+0000 7fb89b7fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:38.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.418+0000 7fb9c68d0980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): orig_sources= new_sources= 2024-03-09T14:51:38.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.418+0000 7fb9c68d0980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): potential sources added=[] removed=[] 2024-03-09T14:51:38.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.418+0000 7fb9c68d0980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): orig_dests= new_dests= 2024-03-09T14:51:38.421 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.418+0000 7fb9c68d0980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): potential dests added=[] removed=[] 2024-03-09T14:51:38.423 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.418+0000 7fb9c68d0980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.23.2.0 2024-03-09T14:51:38.423 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.418+0000 7fb9c68d0980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.23.2.0 2024-03-09T14:51:38.424 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.418+0000 7fb9c68d0980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.23.2.1 2024-03-09T14:51:38.424 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.418+0000 7fb9c68d0980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.23.2.1 2024-03-09T14:51:38.424 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.418+0000 7fb9c68d0980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.23.0 2024-03-09T14:51:38.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.426+0000 7fb9c68d0980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 : hit (requested=0x17, cached=0x17) 2024-03-09T14:51:38.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.426+0000 7fb9c68d0980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 2024-03-09T14:51:38.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.426+0000 7fb9c68d0980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:38.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.426+0000 7fb9c68d0980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:38.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.426+0000 7fb9c68d0980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:38.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.426+0000 7fb9c68d0980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23, ofs0, ns] 2024-03-09T14:51:38.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.426+0000 7fb89affd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562448 cookie 94413233593664 notifier 6612 bl.length()=875 2024-03-09T14:51:38.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.426+0000 7fb89affd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 info.flags=0x17 2024-03-09T14:51:38.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.426+0000 7fb89affd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 to cache LRU end 2024-03-09T14:51:38.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.426+0000 7fb89affd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:38.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.430+0000 7fb9c68d0980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): orig_sources= new_sources= 2024-03-09T14:51:38.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.430+0000 7fb9c68d0980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): potential sources added=[] removed=[] 2024-03-09T14:51:38.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.430+0000 7fb9c68d0980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): orig_dests= new_dests= 2024-03-09T14:51:38.434 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.430+0000 7fb9c68d0980 20 handle_bi_update(): bucket=:abort-at-block-writes[8c209108-e1cd-4d45-9608-6f491550b24c.4644.23]): potential dests added=[] removed=[] 2024-03-09T14:51:38.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.434+0000 7fb9c68d0980 1 execute INFO: reshard of bucket "abort-at-block-writes" completed successfully 2024-03-09T14:51:38.439 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.434+0000 7fb87afdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:38.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.442+0000 7fb9c68d0980 20 remove_watcher() i=0 2024-03-09T14:51:38.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.442+0000 7fb9c68d0980 2 removed watcher, disabling cache 2024-03-09T14:51:38.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.442+0000 7fb9c68d0980 20 remove_watcher() i=1 2024-03-09T14:51:38.449 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.446+0000 7fb9c68d0980 20 remove_watcher() i=2 2024-03-09T14:51:38.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.446+0000 7fb9c68d0980 20 remove_watcher() i=3 2024-03-09T14:51:38.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.450+0000 7fb9c68d0980 20 remove_watcher() i=4 2024-03-09T14:51:38.455 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.450+0000 7fb9c68d0980 20 remove_watcher() i=5 2024-03-09T14:51:38.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.454+0000 7fb9c68d0980 20 remove_watcher() i=6 2024-03-09T14:51:38.458 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.454+0000 7fb9c68d0980 20 remove_watcher() i=7 2024-03-09T14:51:38.475 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket abort-at-block-writes 2024-03-09T14:51:38.502 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:38.502 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:38.549 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.546+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.546+0000 7f2e207b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.546+0000 7f2e207b8980 20 realm 2024-03-09T14:51:38.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.546+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.546+0000 7f2e207b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.546+0000 7f2e207b8980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:38.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.546+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.546+0000 7f2e207b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.546+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.550+0000 7f2e207b8980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:38.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.550+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.550+0000 7f2e207b8980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:38.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.550+0000 7f2e207b8980 20 searching for the correct realm 2024-03-09T14:51:38.560 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.558+0000 7f2e1dacf640 0 -- 172.21.15.1:0/1301112487 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f2e1003b070 msgr2=0x7f2e10019a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:51:38.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.566+0000 7f2e1dacf640 0 -- 172.21.15.1:0/1301112487 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f2ce8075e60 msgr2=0x7f2ce8078290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:38.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.570+0000 7f2e207b8980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:38.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.570+0000 7f2e207b8980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:38.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.570+0000 7f2e207b8980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:38.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.570+0000 7f2e207b8980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:38.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.570+0000 7f2e207b8980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:38.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.570+0000 7f2e207b8980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:38.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.570+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.570+0000 7f2e207b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.570+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:38.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:38.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 zone default found 2024-03-09T14:51:38.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 4 Realm: () 2024-03-09T14:51:38.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:38.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:38.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:38.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 zonegroup default 2024-03-09T14:51:38.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 started sync module instance, tier type = 2024-03-09T14:51:38.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.574+0000 7f2e207b8980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:38.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.590+0000 7f2e207b8980 20 add_watcher() i=0 2024-03-09T14:51:38.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.590+0000 7f2e207b8980 20 add_watcher() i=1 2024-03-09T14:51:38.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.590+0000 7f2e207b8980 20 add_watcher() i=2 2024-03-09T14:51:38.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.590+0000 7f2e207b8980 20 add_watcher() i=3 2024-03-09T14:51:38.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.590+0000 7f2e207b8980 20 add_watcher() i=4 2024-03-09T14:51:38.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.590+0000 7f2e207b8980 20 add_watcher() i=5 2024-03-09T14:51:38.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.590+0000 7f2e207b8980 20 add_watcher() i=6 2024-03-09T14:51:38.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.590+0000 7f2e207b8980 20 add_watcher() i=7 2024-03-09T14:51:38.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.590+0000 7f2e207b8980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:38.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2cd57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:38.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:38.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:38.603 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 5 note: GC not initialized 2024-03-09T14:51:38.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2ccf7fe640 20 reqs_thread_entry: start 2024-03-09T14:51:38.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:38.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 20 Filter name: none 2024-03-09T14:51:38.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2ccd7fa640 20 reqs_thread_entry: start 2024-03-09T14:51:38.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:38.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.605 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:38.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:38.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 20 rados_obj.operate() r=0 bl.length=189 2024-03-09T14:51:38.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.594+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.598+0000 7f2e207b8980 20 rados_obj.operate() r=0 bl.length=403 2024-03-09T14:51:38.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.598+0000 7f2e207b8980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:38.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.598+0000 7f2e207b8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:38.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.598+0000 7f2e207b8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:38.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.598+0000 7f2cd57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:38.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.606+0000 7f2e207b8980 20 remove_watcher() i=0 2024-03-09T14:51:38.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.606+0000 7f2e207b8980 2 removed watcher, disabling cache 2024-03-09T14:51:38.611 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.606+0000 7f2e207b8980 20 remove_watcher() i=1 2024-03-09T14:51:38.612 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.610+0000 7f2e207b8980 20 remove_watcher() i=2 2024-03-09T14:51:38.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.610+0000 7f2e207b8980 20 remove_watcher() i=3 2024-03-09T14:51:38.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.614+0000 7f2e207b8980 20 remove_watcher() i=4 2024-03-09T14:51:38.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.614+0000 7f2e207b8980 20 remove_watcher() i=5 2024-03-09T14:51:38.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.614+0000 7f2e207b8980 20 remove_watcher() i=6 2024-03-09T14:51:38.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:38.618+0000 7f2e207b8980 20 remove_watcher() i=7 2024-03-09T14:51:38.634 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:38.634 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:38.634 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: abort-at-block-writes id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.23 num_objects: 19 size_kb: 1 num_shards: 2 2024-03-09T14:51:38.634 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:38.635 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:38.635 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:38.659 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:38.659 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:39.179 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:39.179 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:39.361 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard bucket with EIO injected at commit_target_layout 2024-03-09T14:51:39.361 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:39.364 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:39.364 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:39.484 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:39.484 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:39.576 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:39.577 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:39.678 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:39.679 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:39.778 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:39.778 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:39.878 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:39.879 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:39.977 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:39.978 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:40.078 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:40.078 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:40.178 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:40.179 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:40.275 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:40.275 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:40.377 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:40.377 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:40.475 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:40.476 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:40.579 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:40.579 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:40.681 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:40.681 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:40.784 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:40.785 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:40.887 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:40.887 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:40.986 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:40.987 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:41.085 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:41.085 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:41.185 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:41.185 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:41.291 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:41.291 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:41.395 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:41.396 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:41.494 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:41.494 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:41.597 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2024-03-09T14:51:41.629 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:41.629 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:41.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.670+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:41.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.670+0000 7fcd3861d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:41.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.670+0000 7fcd3861d980 20 realm 2024-03-09T14:51:41.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.670+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:41.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.670+0000 7fcd3861d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:41.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.670+0000 7fcd3861d980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:41.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.670+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:41.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.670+0000 7fcd3861d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:41.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.670+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:41.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.674+0000 7fcd3861d980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:41.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.674+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:41.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.674+0000 7fcd3861d980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:41.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.674+0000 7fcd3861d980 20 searching for the correct realm 2024-03-09T14:51:41.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.678+0000 7fcd2ffff640 0 -- 172.21.15.1:0/3786677695 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fcd28015070 msgr2=0x7fcd28074870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:41.897 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:41.894+0000 7fcd36135640 0 -- 172.21.15.1:0/3786677695 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fcd2011dbd0 msgr2=0x7fcd20123410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:42.115 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.110+0000 7fcd3861d980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:42.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.110+0000 7fcd3861d980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:42.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.110+0000 7fcd3861d980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:42.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.110+0000 7fcd3861d980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:42.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.110+0000 7fcd3861d980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:42.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.110+0000 7fcd3861d980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:42.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.110+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:42.117 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:42.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 20 zone default found 2024-03-09T14:51:42.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 4 Realm: () 2024-03-09T14:51:42.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:42.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:42.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:42.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 20 zonegroup default 2024-03-09T14:51:42.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.119 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.114+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.118+0000 7fcd3861d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.118+0000 7fcd3861d980 20 started sync module instance, tier type = 2024-03-09T14:51:42.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.118+0000 7fcd3861d980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:42.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.138+0000 7fcd3861d980 20 add_watcher() i=0 2024-03-09T14:51:42.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.138+0000 7fcd3861d980 20 add_watcher() i=1 2024-03-09T14:51:42.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.138+0000 7fcd3861d980 20 add_watcher() i=2 2024-03-09T14:51:42.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.138+0000 7fcd3861d980 20 add_watcher() i=3 2024-03-09T14:51:42.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.138+0000 7fcd3861d980 20 add_watcher() i=4 2024-03-09T14:51:42.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.138+0000 7fcd3861d980 20 add_watcher() i=5 2024-03-09T14:51:42.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.138+0000 7fcd3861d980 20 add_watcher() i=6 2024-03-09T14:51:42.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.142+0000 7fcd3861d980 20 add_watcher() i=7 2024-03-09T14:51:42.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.142+0000 7fcd3861d980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:42.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.146+0000 7fcbecfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:42.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.146+0000 7fcd3861d980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:42.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.146+0000 7fcd3861d980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:42.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.150+0000 7fcd3861d980 5 note: GC not initialized 2024-03-09T14:51:42.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.150+0000 7fcb677fe640 20 reqs_thread_entry: start 2024-03-09T14:51:42.152 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.150+0000 7fcd3861d980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:42.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.150+0000 7fcd3861d980 20 Filter name: none 2024-03-09T14:51:42.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.150+0000 7fcb657fa640 20 reqs_thread_entry: start 2024-03-09T14:51:42.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.150+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.150+0000 7fcd3861d980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:42.154 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.150+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.154+0000 7fcd3861d980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:42.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.154+0000 7fcd3861d980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:42.156 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.154+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.154+0000 7fcd3861d980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:42.158 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.154+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.159 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.154+0000 7fcd3861d980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:42.159 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.154+0000 7fcd3861d980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:42.160 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.158+0000 7fcd3861d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.158+0000 7fcd3861d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.158+0000 7fcbecfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:42.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.162+0000 7fcd3861d980 20 remove_watcher() i=0 2024-03-09T14:51:42.168 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.162+0000 7fcd3861d980 2 removed watcher, disabling cache 2024-03-09T14:51:42.170 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.166+0000 7fcd3861d980 20 remove_watcher() i=1 2024-03-09T14:51:42.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.170+0000 7fcd3861d980 20 remove_watcher() i=2 2024-03-09T14:51:42.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.170+0000 7fcd3861d980 20 remove_watcher() i=3 2024-03-09T14:51:42.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.174+0000 7fcd3861d980 20 remove_watcher() i=4 2024-03-09T14:51:42.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.178+0000 7fcd3861d980 20 remove_watcher() i=5 2024-03-09T14:51:42.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.178+0000 7fcd3861d980 20 remove_watcher() i=6 2024-03-09T14:51:42.183 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.178+0000 7fcd3861d980 20 remove_watcher() i=7 2024-03-09T14:51:42.199 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:42.199 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:42.200 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-commit-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:51:42.200 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:42.200 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket error-at-commit-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-error-at commit_target_layout 2024-03-09T14:51:42.232 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:42.232 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:42.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.274+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.274+0000 7f8bebade980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.274+0000 7f8bebade980 20 realm 2024-03-09T14:51:42.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.274+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.274+0000 7f8bebade980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.274+0000 7f8bebade980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:42.278 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.274+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.274+0000 7f8bebade980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.274+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.278+0000 7f8bebade980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:42.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.278+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.278+0000 7f8bebade980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:42.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.278+0000 7f8bebade980 20 searching for the correct realm 2024-03-09T14:51:42.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.282+0000 7f8be9df7640 0 -- 172.21.15.1:0/3154822964 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f8be00fa920 msgr2=0x7f8be00cc6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:42.293 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.290+0000 7f8be95f6640 0 -- 172.21.15.1:0/3154822964 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f8be0112730 msgr2=0x7f8be00cc6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:42.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.506+0000 7f8bebade980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:42.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.506+0000 7f8bebade980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:42.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.506+0000 7f8bebade980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:42.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.506+0000 7f8bebade980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:42.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.506+0000 7f8bebade980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:42.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.506+0000 7f8bebade980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:42.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.506+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:42.513 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:42.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 20 zone default found 2024-03-09T14:51:42.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 4 Realm: () 2024-03-09T14:51:42.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:42.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:42.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:42.514 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 20 zonegroup default 2024-03-09T14:51:42.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.515 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.510+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.514+0000 7f8bebade980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.514+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.514+0000 7f8bebade980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.514+0000 7f8bebade980 20 started sync module instance, tier type = 2024-03-09T14:51:42.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.514+0000 7f8bebade980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:42.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.538+0000 7f8bebade980 20 add_watcher() i=0 2024-03-09T14:51:42.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.538+0000 7f8bebade980 20 add_watcher() i=1 2024-03-09T14:51:42.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.538+0000 7f8bebade980 20 add_watcher() i=2 2024-03-09T14:51:42.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.538+0000 7f8bebade980 20 add_watcher() i=3 2024-03-09T14:51:42.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.538+0000 7f8bebade980 20 add_watcher() i=4 2024-03-09T14:51:42.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.538+0000 7f8bebade980 20 add_watcher() i=5 2024-03-09T14:51:42.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.538+0000 7f8bebade980 20 add_watcher() i=6 2024-03-09T14:51:42.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.542+0000 7f8bebade980 20 add_watcher() i=7 2024-03-09T14:51:42.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.542+0000 7f8bebade980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:42.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8a9f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:42.549 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8bebade980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:42.549 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8bebade980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:42.549 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8bebade980 5 note: GC not initialized 2024-03-09T14:51:42.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8a9e7e4640 20 reqs_thread_entry: start 2024-03-09T14:51:42.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8bebade980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:42.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8bebade980 20 Filter name: none 2024-03-09T14:51:42.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8a0ffff640 20 reqs_thread_entry: start 2024-03-09T14:51:42.550 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8bebade980 10 cache get: name=default.rgw.meta+root+error-at-commit-target-layout : miss 2024-03-09T14:51:42.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8bebade980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:42.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8bebade980 10 cache put: name=default.rgw.meta+root+error-at-commit-target-layout info.flags=0x11 2024-03-09T14:51:42.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8bebade980 10 adding default.rgw.meta+root+error-at-commit-target-layout to cache LRU end 2024-03-09T14:51:42.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8bebade980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 : miss 2024-03-09T14:51:42.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.546+0000 7f8bebade980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.550+0000 7f8bebade980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:42.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.550+0000 7f8bebade980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 info.flags=0x17 2024-03-09T14:51:42.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.550+0000 7f8bebade980 10 adding default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 to cache LRU end 2024-03-09T14:51:42.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.550+0000 7f8bebade980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:42.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.550+0000 7f8bebade980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:42.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.550+0000 7f8bebade980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 2024-03-09T14:51:42.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.558+0000 7f8bebade980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 info.flags=0x17 2024-03-09T14:51:42.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.558+0000 7f8bebade980 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 to cache LRU end 2024-03-09T14:51:42.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.558+0000 7f8bebade980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:42.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.558+0000 7f8bebade980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24, ofs0, ns] 2024-03-09T14:51:42.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.558+0000 7f8accff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562401 cookie 94895867596112 notifier 6272 bl.length()=923 2024-03-09T14:51:42.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.558+0000 7f8accff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 info.flags=0x17 2024-03-09T14:51:42.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.558+0000 7f8accff9640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 to cache LRU end 2024-03-09T14:51:42.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.558+0000 7f8accff9640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:42.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.562+0000 7f8bebade980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.24]): orig_sources= new_sources= 2024-03-09T14:51:42.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.562+0000 7f8bebade980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.24]): potential sources added=[] removed=[] 2024-03-09T14:51:42.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.562+0000 7f8bebade980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.24]): orig_dests= new_dests= 2024-03-09T14:51:42.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.562+0000 7f8bebade980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.24]): potential dests added=[] removed=[] 2024-03-09T14:51:42.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.566+0000 7f8bebade980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.24.1.0 2024-03-09T14:51:42.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.566+0000 7f8bebade980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.24.1.0 2024-03-09T14:51:42.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.566+0000 7f8bebade980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.24.1.1 2024-03-09T14:51:42.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.566+0000 7f8bebade980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.24.1.1 2024-03-09T14:51:42.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.566+0000 7f8bebade980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.24.0 2024-03-09T14:51:42.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.570+0000 7f8bebade980 -1 Injecting error=-5 at location=commit_target_layout 2024-03-09T14:51:42.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.570+0000 7f8bebade980 0 ERROR: commit_reshard failed to commit target index layout: (5) Input/output error 2024-03-09T14:51:42.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.578+0000 7f8a9f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:42.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.586+0000 7f8bebade980 20 remove_watcher() i=0 2024-03-09T14:51:42.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.586+0000 7f8bebade980 2 removed watcher, disabling cache 2024-03-09T14:51:42.591 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.586+0000 7f8bebade980 20 remove_watcher() i=1 2024-03-09T14:51:42.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.590+0000 7f8bebade980 20 remove_watcher() i=2 2024-03-09T14:51:42.596 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.594+0000 7f8bebade980 20 remove_watcher() i=3 2024-03-09T14:51:42.599 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.594+0000 7f8bebade980 20 remove_watcher() i=4 2024-03-09T14:51:42.600 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.598+0000 7f8bebade980 20 remove_watcher() i=5 2024-03-09T14:51:42.602 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.598+0000 7f8bebade980 20 remove_watcher() i=6 2024-03-09T14:51:42.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.598+0000 7f8bebade980 20 remove_watcher() i=7 2024-03-09T14:51:42.621 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2024-03-09T14:51:42.649 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:42.649 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:42.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.690+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.694+0000 7fb8a9b51980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.694+0000 7fb8a9b51980 20 realm 2024-03-09T14:51:42.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.694+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.694+0000 7fb8a9b51980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.694+0000 7fb8a9b51980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:42.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.694+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.694+0000 7fb8a9b51980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.697 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.694+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.694+0000 7fb8a9b51980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:42.698 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.694+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.699 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.694+0000 7fb8a9b51980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:42.699 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.694+0000 7fb8a9b51980 20 searching for the correct realm 2024-03-09T14:51:42.706 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.702+0000 7fb8a6e68640 0 -- 172.21.15.1:0/1910414957 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fb89805c080 msgr2=0x7fb89808aac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:42.712 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.710+0000 7fb8a6e68640 0 -- 172.21.15.1:0/1910414957 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fb770075e90 msgr2=0x7fb7700782c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:42.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.714+0000 7fb8a9b51980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:42.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.714+0000 7fb8a9b51980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:42.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.714+0000 7fb8a9b51980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:42.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.714+0000 7fb8a9b51980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:42.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.714+0000 7fb8a9b51980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:42.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.714+0000 7fb8a9b51980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:42.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.714+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:42.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:42.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 zone default found 2024-03-09T14:51:42.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 4 Realm: () 2024-03-09T14:51:42.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:42.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:42.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:42.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 zonegroup default 2024-03-09T14:51:42.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 started sync module instance, tier type = 2024-03-09T14:51:42.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.718+0000 7fb8a9b51980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:42.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.734+0000 7fb8a9b51980 20 add_watcher() i=0 2024-03-09T14:51:42.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.734+0000 7fb8a9b51980 20 add_watcher() i=1 2024-03-09T14:51:42.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.734+0000 7fb8a9b51980 20 add_watcher() i=2 2024-03-09T14:51:42.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.734+0000 7fb8a9b51980 20 add_watcher() i=3 2024-03-09T14:51:42.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.734+0000 7fb8a9b51980 20 add_watcher() i=4 2024-03-09T14:51:42.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.734+0000 7fb8a9b51980 20 add_watcher() i=5 2024-03-09T14:51:42.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.734+0000 7fb8a9b51980 20 add_watcher() i=6 2024-03-09T14:51:42.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.738+0000 7fb8a9b51980 20 add_watcher() i=7 2024-03-09T14:51:42.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.738+0000 7fb8a9b51980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:42.743 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.738+0000 7fb75e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:42.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.738+0000 7fb8a9b51980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:42.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.738+0000 7fb8a9b51980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:42.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.738+0000 7fb8a9b51980 5 note: GC not initialized 2024-03-09T14:51:42.744 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.738+0000 7fb75d7e2640 20 reqs_thread_entry: start 2024-03-09T14:51:42.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.742+0000 7fb8a9b51980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:42.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.742+0000 7fb8a9b51980 20 Filter name: none 2024-03-09T14:51:42.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.742+0000 7fb7567fc640 20 reqs_thread_entry: start 2024-03-09T14:51:42.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.742+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.742+0000 7fb8a9b51980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:42.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.742+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.742+0000 7fb8a9b51980 20 rados_obj.operate() r=0 bl.length=443 2024-03-09T14:51:42.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.742+0000 7fb8a9b51980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:42.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.742+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.742+0000 7fb8a9b51980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:42.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.742+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.746+0000 7fb8a9b51980 20 rados_obj.operate() r=0 bl.length=443 2024-03-09T14:51:42.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.746+0000 7fb8a9b51980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:42.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.746+0000 7fb8a9b51980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.751 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.746+0000 7fb8a9b51980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.750+0000 7fb75e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:42.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.754+0000 7fb8a9b51980 20 remove_watcher() i=0 2024-03-09T14:51:42.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.754+0000 7fb8a9b51980 2 removed watcher, disabling cache 2024-03-09T14:51:42.759 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.754+0000 7fb8a9b51980 20 remove_watcher() i=1 2024-03-09T14:51:42.761 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.758+0000 7fb8a9b51980 20 remove_watcher() i=2 2024-03-09T14:51:42.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.762+0000 7fb8a9b51980 20 remove_watcher() i=3 2024-03-09T14:51:42.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.762+0000 7fb8a9b51980 20 remove_watcher() i=4 2024-03-09T14:51:42.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.762+0000 7fb8a9b51980 20 remove_watcher() i=5 2024-03-09T14:51:42.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.766+0000 7fb8a9b51980 20 remove_watcher() i=6 2024-03-09T14:51:42.771 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.766+0000 7fb8a9b51980 20 remove_watcher() i=7 2024-03-09T14:51:42.784 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:42.784 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:42.784 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-commit-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:51:42.785 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:42.786 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:42.786 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:42.833 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:42.834 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:42.894 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket error-at-commit-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2024-03-09T14:51:42.930 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:42.930 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:42.974 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.970+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.970+0000 7fbffefbf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.970+0000 7fbffefbf980 20 realm 2024-03-09T14:51:42.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.970+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.974+0000 7fbffefbf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.974+0000 7fbffefbf980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:42.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.974+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.974+0000 7fbffefbf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:42.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.974+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.978 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.974+0000 7fbffefbf980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:42.978 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.974+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:42.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.974+0000 7fbffefbf980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:42.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.974+0000 7fbffefbf980 20 searching for the correct realm 2024-03-09T14:51:42.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.982+0000 7fbffc2d6640 0 -- 172.21.15.1:0/114889579 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x5626244e3cc0 msgr2=0x5626244e6180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:51:42.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:42.990+0000 7fbffcad7640 0 -- 172.21.15.1:0/114889579 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fbfe4060910 msgr2=0x7fbfe4020430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:43.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.206+0000 7fbffefbf980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:43.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.206+0000 7fbffefbf980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:43.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.206+0000 7fbffefbf980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:43.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.206+0000 7fbffefbf980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:43.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.206+0000 7fbffefbf980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:43.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.206+0000 7fbffefbf980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:43.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.206+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.206+0000 7fbffefbf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:43.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.206+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:43.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:43.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 zone default found 2024-03-09T14:51:43.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 4 Realm: () 2024-03-09T14:51:43.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:43.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:43.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:43.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 zonegroup default 2024-03-09T14:51:43.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:43.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:43.214 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:43.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 started sync module instance, tier type = 2024-03-09T14:51:43.215 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.210+0000 7fbffefbf980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:43.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.234+0000 7fbffefbf980 20 add_watcher() i=0 2024-03-09T14:51:43.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.234+0000 7fbffefbf980 20 add_watcher() i=1 2024-03-09T14:51:43.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.234+0000 7fbffefbf980 20 add_watcher() i=2 2024-03-09T14:51:43.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.234+0000 7fbffefbf980 20 add_watcher() i=3 2024-03-09T14:51:43.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.234+0000 7fbffefbf980 20 add_watcher() i=4 2024-03-09T14:51:43.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.234+0000 7fbffefbf980 20 add_watcher() i=5 2024-03-09T14:51:43.239 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.234+0000 7fbffefbf980 20 add_watcher() i=6 2024-03-09T14:51:43.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.238+0000 7fbffefbf980 20 add_watcher() i=7 2024-03-09T14:51:43.241 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.238+0000 7fbffefbf980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:43.245 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.242+0000 7fbeb37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:43.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.242+0000 7fbffefbf980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:43.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.242+0000 7fbffefbf980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:43.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.242+0000 7fbffefbf980 5 note: GC not initialized 2024-03-09T14:51:43.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.242+0000 7fbeb27e4640 20 reqs_thread_entry: start 2024-03-09T14:51:43.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.242+0000 7fbffefbf980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:43.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.242+0000 7fbffefbf980 20 Filter name: none 2024-03-09T14:51:43.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.242+0000 7fbeb07e0640 20 reqs_thread_entry: start 2024-03-09T14:51:43.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.242+0000 7fbffefbf980 10 cache get: name=default.rgw.meta+root+error-at-commit-target-layout : miss 2024-03-09T14:51:43.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.242+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.246+0000 7fbffefbf980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:43.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.246+0000 7fbffefbf980 10 cache put: name=default.rgw.meta+root+error-at-commit-target-layout info.flags=0x11 2024-03-09T14:51:43.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.246+0000 7fbffefbf980 10 adding default.rgw.meta+root+error-at-commit-target-layout to cache LRU end 2024-03-09T14:51:43.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.246+0000 7fbffefbf980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 : miss 2024-03-09T14:51:43.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.246+0000 7fbffefbf980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.246+0000 7fbffefbf980 20 rados_obj.operate() r=0 bl.length=443 2024-03-09T14:51:43.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.246+0000 7fbffefbf980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 info.flags=0x17 2024-03-09T14:51:43.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.246+0000 7fbffefbf980 10 adding default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 to cache LRU end 2024-03-09T14:51:43.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.246+0000 7fbffefbf980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:43.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.246+0000 7fbffefbf980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:43.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.246+0000 7fbffefbf980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 2024-03-09T14:51:43.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.250+0000 7fbffefbf980 10 init_target_layout removing existing target index objects from a previous reshard attempt 2024-03-09T14:51:43.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.258+0000 7fbffefbf980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 info.flags=0x17 2024-03-09T14:51:43.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.258+0000 7fbffefbf980 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 to cache LRU end 2024-03-09T14:51:43.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.258+0000 7fbffefbf980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:43.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.258+0000 7fbffefbf980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24, ofs0, ns] 2024-03-09T14:51:43.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.258+0000 7fbed3fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562402 cookie 94721817880096 notifier 6711 bl.length()=923 2024-03-09T14:51:43.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.258+0000 7fbed3fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 info.flags=0x17 2024-03-09T14:51:43.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.258+0000 7fbed3fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 to cache LRU end 2024-03-09T14:51:43.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.258+0000 7fbed3fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:43.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.262+0000 7fbffefbf980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.24]): orig_sources= new_sources= 2024-03-09T14:51:43.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.262+0000 7fbffefbf980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.24]): potential sources added=[] removed=[] 2024-03-09T14:51:43.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.262+0000 7fbffefbf980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.24]): orig_dests= new_dests= 2024-03-09T14:51:43.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.262+0000 7fbffefbf980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.24]): potential dests added=[] removed=[] 2024-03-09T14:51:43.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.266+0000 7fbffefbf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.24.2.0 2024-03-09T14:51:43.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.266+0000 7fbffefbf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.24.2.0 2024-03-09T14:51:43.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.266+0000 7fbffefbf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.24.2.1 2024-03-09T14:51:43.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.266+0000 7fbffefbf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.24.2.1 2024-03-09T14:51:43.268 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.266+0000 7fbffefbf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.24.0 2024-03-09T14:51:43.272 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.270+0000 7fbffefbf980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 : hit (requested=0x17, cached=0x17) 2024-03-09T14:51:43.272 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.270+0000 7fbffefbf980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 2024-03-09T14:51:43.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.274+0000 7fbffefbf980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 info.flags=0x17 2024-03-09T14:51:43.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.274+0000 7fbffefbf980 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 to cache LRU end 2024-03-09T14:51:43.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.274+0000 7fbffefbf980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:43.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.274+0000 7fbffefbf980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24, ofs0, ns] 2024-03-09T14:51:43.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.274+0000 7fbed37fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562403 cookie 94721817880096 notifier 6711 bl.length()=891 2024-03-09T14:51:43.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.274+0000 7fbed37fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 info.flags=0x17 2024-03-09T14:51:43.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.274+0000 7fbed37fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 to cache LRU end 2024-03-09T14:51:43.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.274+0000 7fbed37fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:43.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.274+0000 7fbffefbf980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.24]): orig_sources= new_sources= 2024-03-09T14:51:43.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.274+0000 7fbffefbf980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.24]): potential sources added=[] removed=[] 2024-03-09T14:51:43.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.274+0000 7fbffefbf980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.24]): orig_dests= new_dests= 2024-03-09T14:51:43.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.274+0000 7fbffefbf980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.24]): potential dests added=[] removed=[] 2024-03-09T14:51:43.282 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.278+0000 7fbffefbf980 1 execute INFO: reshard of bucket "error-at-commit-target-layout" completed successfully 2024-03-09T14:51:43.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.282+0000 7fbeb37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:43.291 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.286+0000 7fbffefbf980 20 remove_watcher() i=0 2024-03-09T14:51:43.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.286+0000 7fbffefbf980 2 removed watcher, disabling cache 2024-03-09T14:51:43.294 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.290+0000 7fbffefbf980 20 remove_watcher() i=1 2024-03-09T14:51:43.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.294+0000 7fbffefbf980 20 remove_watcher() i=2 2024-03-09T14:51:43.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.298+0000 7fbffefbf980 20 remove_watcher() i=3 2024-03-09T14:51:43.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.298+0000 7fbffefbf980 20 remove_watcher() i=4 2024-03-09T14:51:43.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.302+0000 7fbffefbf980 20 remove_watcher() i=5 2024-03-09T14:51:43.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.302+0000 7fbffefbf980 20 remove_watcher() i=6 2024-03-09T14:51:43.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.306+0000 7fbffefbf980 20 remove_watcher() i=7 2024-03-09T14:51:43.327 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2024-03-09T14:51:43.359 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:43.359 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:43.403 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.398+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.402+0000 7f4d9fab8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:43.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.402+0000 7f4d9fab8980 20 realm 2024-03-09T14:51:43.405 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.402+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.402+0000 7f4d9fab8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:43.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.402+0000 7f4d9fab8980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:43.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.402+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.402+0000 7f4d9fab8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:43.406 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.402+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.402+0000 7f4d9fab8980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:43.407 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.402+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.406+0000 7f4d9fab8980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:43.408 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.406+0000 7f4d9fab8980 20 searching for the correct realm 2024-03-09T14:51:43.415 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.410+0000 7f4d9cdcf640 0 -- 172.21.15.1:0/3641239362 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f4d9003ba80 msgr2=0x7f4d90085a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:43.422 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.418+0000 7f4d9cdcf640 0 -- 172.21.15.1:0/3641239362 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f4c68075e90 msgr2=0x7f4c680782c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:43.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:43.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:43.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:43.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:43.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:43.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:43.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:43.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:43.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:43.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 zone default found 2024-03-09T14:51:43.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 4 Realm: () 2024-03-09T14:51:43.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:43.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:43.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:43.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 zonegroup default 2024-03-09T14:51:43.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:43.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:43.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:43.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 started sync module instance, tier type = 2024-03-09T14:51:43.432 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.426+0000 7f4d9fab8980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:43.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.442+0000 7f4d9fab8980 20 add_watcher() i=0 2024-03-09T14:51:43.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.442+0000 7f4d9fab8980 20 add_watcher() i=1 2024-03-09T14:51:43.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.442+0000 7f4d9fab8980 20 add_watcher() i=2 2024-03-09T14:51:43.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.442+0000 7f4d9fab8980 20 add_watcher() i=3 2024-03-09T14:51:43.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.442+0000 7f4d9fab8980 20 add_watcher() i=4 2024-03-09T14:51:43.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.442+0000 7f4d9fab8980 20 add_watcher() i=5 2024-03-09T14:51:43.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.442+0000 7f4d9fab8980 20 add_watcher() i=6 2024-03-09T14:51:43.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.446+0000 7f4d9fab8980 20 add_watcher() i=7 2024-03-09T14:51:43.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.446+0000 7f4d9fab8980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:43.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.446+0000 7f4c53fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:43.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.446+0000 7f4d9fab8980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:43.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.446+0000 7f4d9fab8980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:43.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.446+0000 7f4d9fab8980 5 note: GC not initialized 2024-03-09T14:51:43.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.446+0000 7f4c52fe5640 20 reqs_thread_entry: start 2024-03-09T14:51:43.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.446+0000 7f4d9fab8980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:43.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.446+0000 7f4d9fab8980 20 Filter name: none 2024-03-09T14:51:43.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.450+0000 7f4c50fe1640 20 reqs_thread_entry: start 2024-03-09T14:51:43.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.450+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.450+0000 7f4d9fab8980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:43.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.450+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.450+0000 7f4d9fab8980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:43.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.450+0000 7f4d9fab8980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:43.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.450+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.455 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.450+0000 7f4d9fab8980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:43.455 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.450+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.454+0000 7f4d9fab8980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:43.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.454+0000 7f4d9fab8980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:43.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.454+0000 7f4d9fab8980 20 rados->read ofs=0 len=0 2024-03-09T14:51:43.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.454+0000 7f4d9fab8980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:43.458 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.454+0000 7f4c53fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:43.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.458+0000 7f4d9fab8980 20 remove_watcher() i=0 2024-03-09T14:51:43.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.458+0000 7f4d9fab8980 2 removed watcher, disabling cache 2024-03-09T14:51:43.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.462+0000 7f4d9fab8980 20 remove_watcher() i=1 2024-03-09T14:51:43.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.462+0000 7f4d9fab8980 20 remove_watcher() i=2 2024-03-09T14:51:43.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.466+0000 7f4d9fab8980 20 remove_watcher() i=3 2024-03-09T14:51:43.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.466+0000 7f4d9fab8980 20 remove_watcher() i=4 2024-03-09T14:51:43.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.470+0000 7f4d9fab8980 20 remove_watcher() i=5 2024-03-09T14:51:43.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.470+0000 7f4d9fab8980 20 remove_watcher() i=6 2024-03-09T14:51:43.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:43.470+0000 7f4d9fab8980 20 remove_watcher() i=7 2024-03-09T14:51:43.488 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:43.489 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:43.489 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-commit-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.24 num_objects: 19 size_kb: 1 num_shards: 2 2024-03-09T14:51:43.489 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:43.490 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:43.490 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:43.514 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:43.514 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:44.065 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:44.065 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:44.247 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard bucket with ECANCELED injected at commit_target_layout 2024-03-09T14:51:44.247 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:44.250 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:44.250 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:44.373 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:44.373 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:44.463 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:44.464 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:44.562 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:44.563 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:44.666 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:44.666 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:44.765 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:44.765 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:44.869 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:44.869 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:44.968 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:44.968 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:45.070 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:45.070 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:45.173 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:45.173 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:45.278 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:45.278 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:45.379 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:45.379 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:45.473 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:45.474 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:45.578 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:45.578 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:45.680 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:45.680 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:45.783 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:45.783 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:45.887 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:45.887 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:45.986 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:45.986 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:46.087 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:46.087 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:46.188 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:46.188 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:46.291 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:46.291 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:46.397 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:46.398 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:46.546 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2024-03-09T14:51:46.581 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:46.581 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:46.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.622+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.622+0000 7f7e3d49d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:46.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.622+0000 7f7e3d49d980 20 realm 2024-03-09T14:51:46.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.622+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.626+0000 7f7e3d49d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:46.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.626+0000 7f7e3d49d980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:46.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.626+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.626+0000 7f7e3d49d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:46.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.626+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.626+0000 7f7e3d49d980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:46.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.626+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.626+0000 7f7e3d49d980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:46.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.626+0000 7f7e3d49d980 20 searching for the correct realm 2024-03-09T14:51:46.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.634+0000 7f7e3b7b6640 0 -- 172.21.15.1:0/3443027420 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f7e30018070 msgr2=0x7f7e30053750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:46.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.638+0000 7f7e3afb5640 0 -- 172.21.15.1:0/3443027420 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f7e3005f2d0 msgr2=0x7f7e3006f180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:46.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.854+0000 7f7e3d49d980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:46.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.854+0000 7f7e3d49d980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:46.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.854+0000 7f7e3d49d980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:46.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.854+0000 7f7e3d49d980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:46.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.854+0000 7f7e3d49d980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:46.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.854+0000 7f7e3d49d980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:46.859 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.854+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.860 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.854+0000 7f7e3d49d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:46.860 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.854+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.860 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:46.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:46.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 zone default found 2024-03-09T14:51:46.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 4 Realm: () 2024-03-09T14:51:46.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:46.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:46.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:46.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 zonegroup default 2024-03-09T14:51:46.861 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:46.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.862 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:46.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.863 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:46.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 started sync module instance, tier type = 2024-03-09T14:51:46.864 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.858+0000 7f7e3d49d980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:46.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.882+0000 7f7e3d49d980 20 add_watcher() i=0 2024-03-09T14:51:46.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.882+0000 7f7e3d49d980 20 add_watcher() i=1 2024-03-09T14:51:46.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.882+0000 7f7e3d49d980 20 add_watcher() i=2 2024-03-09T14:51:46.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.882+0000 7f7e3d49d980 20 add_watcher() i=3 2024-03-09T14:51:46.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.882+0000 7f7e3d49d980 20 add_watcher() i=4 2024-03-09T14:51:46.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.882+0000 7f7e3d49d980 20 add_watcher() i=5 2024-03-09T14:51:46.887 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.882+0000 7f7e3d49d980 20 add_watcher() i=6 2024-03-09T14:51:46.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.886+0000 7f7e3d49d980 20 add_watcher() i=7 2024-03-09T14:51:46.889 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.886+0000 7f7e3d49d980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:46.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.890+0000 7f7cf17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:46.894 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.890+0000 7f7e3d49d980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:46.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.890+0000 7f7e3d49d980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:46.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.890+0000 7f7e3d49d980 5 note: GC not initialized 2024-03-09T14:51:46.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.890+0000 7f7c6bfff640 20 reqs_thread_entry: start 2024-03-09T14:51:46.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.890+0000 7f7e3d49d980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:46.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.890+0000 7f7e3d49d980 20 Filter name: none 2024-03-09T14:51:46.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.890+0000 7f7c69ffb640 20 reqs_thread_entry: start 2024-03-09T14:51:46.896 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.890+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.897 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.894+0000 7f7e3d49d980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:46.897 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.894+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.898 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.894+0000 7f7e3d49d980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:46.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.894+0000 7f7e3d49d980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:46.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.894+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.894+0000 7f7e3d49d980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:46.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.894+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.900 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.898+0000 7f7e3d49d980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:46.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.898+0000 7f7e3d49d980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:46.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.898+0000 7f7e3d49d980 20 rados->read ofs=0 len=0 2024-03-09T14:51:46.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.898+0000 7f7e3d49d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:46.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.898+0000 7f7cf17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:46.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.906+0000 7f7e3d49d980 20 remove_watcher() i=0 2024-03-09T14:51:46.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.906+0000 7f7e3d49d980 2 removed watcher, disabling cache 2024-03-09T14:51:46.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.906+0000 7f7e3d49d980 20 remove_watcher() i=1 2024-03-09T14:51:46.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.910+0000 7f7e3d49d980 20 remove_watcher() i=2 2024-03-09T14:51:46.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.910+0000 7f7e3d49d980 20 remove_watcher() i=3 2024-03-09T14:51:46.917 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.914+0000 7f7e3d49d980 20 remove_watcher() i=4 2024-03-09T14:51:46.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.914+0000 7f7e3d49d980 20 remove_watcher() i=5 2024-03-09T14:51:46.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.918+0000 7f7e3d49d980 20 remove_watcher() i=6 2024-03-09T14:51:46.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:46.918+0000 7f7e3d49d980 20 remove_watcher() i=7 2024-03-09T14:51:46.937 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:46.937 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:46.937 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-commit-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:51:46.937 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:46.937 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket error-at-commit-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-error-at commit_target_layout --inject-error-code 125 2024-03-09T14:51:46.973 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:46.973 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:47.017 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.014+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.014+0000 7fd0ffe9a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.014+0000 7fd0ffe9a980 20 realm 2024-03-09T14:51:47.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.014+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.014+0000 7fd0ffe9a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.014+0000 7fd0ffe9a980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:47.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.014+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.014+0000 7fd0ffe9a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.019 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.014+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.018+0000 7fd0ffe9a980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:47.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.018+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.018+0000 7fd0ffe9a980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:47.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.018+0000 7fd0ffe9a980 20 searching for the correct realm 2024-03-09T14:51:47.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.026+0000 7fd0fe1b3640 0 -- 172.21.15.1:0/2100950514 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7fd0f407c040 msgr2=0x7fd0f407f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:47.245 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.242+0000 7fd0fd9b2640 0 -- 172.21.15.1:0/2100950514 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x56390b5062a0 msgr2=0x56390b508690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:47.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:47.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:47.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:47.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:47.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:47.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:47.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:47.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 zone default found 2024-03-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 4 Realm: () 2024-03-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 zonegroup default 2024-03-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.250+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.254+0000 7fd0ffe9a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.254+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.254+0000 7fd0ffe9a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.254+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.254+0000 7fd0ffe9a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.254+0000 7fd0ffe9a980 20 started sync module instance, tier type = 2024-03-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.254+0000 7fd0ffe9a980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.274+0000 7fd0ffe9a980 20 add_watcher() i=0 2024-03-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.274+0000 7fd0ffe9a980 20 add_watcher() i=1 2024-03-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.274+0000 7fd0ffe9a980 20 add_watcher() i=2 2024-03-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.274+0000 7fd0ffe9a980 20 add_watcher() i=3 2024-03-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.274+0000 7fd0ffe9a980 20 add_watcher() i=4 2024-03-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.274+0000 7fd0ffe9a980 20 add_watcher() i=5 2024-03-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.274+0000 7fd0ffe9a980 20 add_watcher() i=6 2024-03-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.278+0000 7fd0ffe9a980 20 add_watcher() i=7 2024-03-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.278+0000 7fd0ffe9a980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:47.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fcfaf7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:47.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fd0ffe9a980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:47.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fd0ffe9a980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:47.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fd0ffe9a980 5 note: GC not initialized 2024-03-09T14:51:47.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fcfae7fc640 20 reqs_thread_entry: start 2024-03-09T14:51:47.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fd0ffe9a980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:47.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fd0ffe9a980 20 Filter name: none 2024-03-09T14:51:47.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fcf1ffff640 20 reqs_thread_entry: start 2024-03-09T14:51:47.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fd0ffe9a980 10 cache get: name=default.rgw.meta+root+error-at-commit-target-layout : miss 2024-03-09T14:51:47.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.288 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fd0ffe9a980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:47.288 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fd0ffe9a980 10 cache put: name=default.rgw.meta+root+error-at-commit-target-layout info.flags=0x11 2024-03-09T14:51:47.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fd0ffe9a980 10 adding default.rgw.meta+root+error-at-commit-target-layout to cache LRU end 2024-03-09T14:51:47.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fd0ffe9a980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 : miss 2024-03-09T14:51:47.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.282+0000 7fd0ffe9a980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.286+0000 7fd0ffe9a980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:47.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.286+0000 7fd0ffe9a980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 info.flags=0x17 2024-03-09T14:51:47.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.286+0000 7fd0ffe9a980 10 adding default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 to cache LRU end 2024-03-09T14:51:47.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.286+0000 7fd0ffe9a980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:47.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.286+0000 7fd0ffe9a980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:47.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.286+0000 7fd0ffe9a980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 2024-03-09T14:51:47.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.294+0000 7fd0ffe9a980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 info.flags=0x17 2024-03-09T14:51:47.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.294+0000 7fd0ffe9a980 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 to cache LRU end 2024-03-09T14:51:47.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.294+0000 7fd0ffe9a980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:47.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.294+0000 7fd0ffe9a980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25, ofs0, ns] 2024-03-09T14:51:47.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.294+0000 7fcfe17fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562406 cookie 94803002990368 notifier 6768 bl.length()=923 2024-03-09T14:51:47.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.294+0000 7fcfe17fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 info.flags=0x17 2024-03-09T14:51:47.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.294+0000 7fcfe17fa640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 to cache LRU end 2024-03-09T14:51:47.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.294+0000 7fcfe17fa640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:47.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.298+0000 7fd0ffe9a980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.25]): orig_sources= new_sources= 2024-03-09T14:51:47.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.298+0000 7fd0ffe9a980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.25]): potential sources added=[] removed=[] 2024-03-09T14:51:47.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.298+0000 7fd0ffe9a980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.25]): orig_dests= new_dests= 2024-03-09T14:51:47.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.298+0000 7fd0ffe9a980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.25]): potential dests added=[] removed=[] 2024-03-09T14:51:47.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.298+0000 7fd0ffe9a980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.25.1.0 2024-03-09T14:51:47.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.298+0000 7fd0ffe9a980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.25.1.0 2024-03-09T14:51:47.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.298+0000 7fd0ffe9a980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.25.1.1 2024-03-09T14:51:47.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.298+0000 7fd0ffe9a980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.25.1.1 2024-03-09T14:51:47.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.298+0000 7fd0ffe9a980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.25.0 2024-03-09T14:51:47.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.302+0000 7fd0ffe9a980 -1 Injecting error=-125 at location=commit_target_layout 2024-03-09T14:51:47.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.302+0000 7fd0ffe9a980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 : hit (requested=0x17, cached=0x17) 2024-03-09T14:51:47.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.302+0000 7fd0ffe9a980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 2024-03-09T14:51:47.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.306+0000 7fd0ffe9a980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 info.flags=0x17 2024-03-09T14:51:47.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.306+0000 7fd0ffe9a980 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 to cache LRU end 2024-03-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.306+0000 7fd0ffe9a980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.306+0000 7fd0ffe9a980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25, ofs0, ns] 2024-03-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.306+0000 7fcfe0ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562407 cookie 94803002990368 notifier 6768 bl.length()=891 2024-03-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.306+0000 7fcfe0ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 info.flags=0x17 2024-03-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.306+0000 7fcfe0ff9640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 to cache LRU end 2024-03-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.306+0000 7fcfe0ff9640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.310+0000 7fd0ffe9a980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.25]): orig_sources= new_sources= 2024-03-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.310+0000 7fd0ffe9a980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.25]): potential sources added=[] removed=[] 2024-03-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.310+0000 7fd0ffe9a980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.25]): orig_dests= new_dests= 2024-03-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.310+0000 7fd0ffe9a980 20 handle_bi_update(): bucket=:error-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.25]): potential dests added=[] removed=[] 2024-03-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.310+0000 7fd0ffe9a980 1 execute INFO: reshard of bucket "error-at-commit-target-layout" completed successfully 2024-03-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.314+0000 7fcfaf7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.318+0000 7fd0ffe9a980 20 remove_watcher() i=0 2024-03-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.318+0000 7fd0ffe9a980 2 removed watcher, disabling cache 2024-03-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.322+0000 7fd0ffe9a980 20 remove_watcher() i=1 2024-03-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.322+0000 7fd0ffe9a980 20 remove_watcher() i=2 2024-03-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.326+0000 7fd0ffe9a980 20 remove_watcher() i=3 2024-03-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.330+0000 7fd0ffe9a980 20 remove_watcher() i=4 2024-03-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.330+0000 7fd0ffe9a980 20 remove_watcher() i=5 2024-03-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.330+0000 7fd0ffe9a980 20 remove_watcher() i=6 2024-03-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.334+0000 7fd0ffe9a980 20 remove_watcher() i=7 2024-03-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2024-03-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.422+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.422+0000 7efcdb759980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.422+0000 7efcdb759980 20 realm 2024-03-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.422+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.422+0000 7efcdb759980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.422+0000 7efcdb759980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.422+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.422+0000 7efcdb759980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.422+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.426+0000 7efcdb759980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.426+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.426+0000 7efcdb759980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.426+0000 7efcdb759980 20 searching for the correct realm 2024-03-09T14:51:47.436 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.434+0000 7efcd8a70640 0 -- 172.21.15.1:0/3507109761 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7efccc068be0 msgr2=0x7efccc053f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:51:47.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.438+0000 7efcd8a70640 0 -- 172.21.15.1:0/3507109761 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7efba4075e10 msgr2=0x7efba4078240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:47.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.446+0000 7efcdb759980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:47.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.446+0000 7efcdb759980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:47.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.446+0000 7efcdb759980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:47.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.446+0000 7efcdb759980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:47.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.446+0000 7efcdb759980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:47.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.446+0000 7efcdb759980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:47.451 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.446+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.446+0000 7efcdb759980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.446+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:47.452 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 zone default found 2024-03-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 4 Realm: () 2024-03-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 zonegroup default 2024-03-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 started sync module instance, tier type = 2024-03-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.450+0000 7efcdb759980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.466+0000 7efcdb759980 20 add_watcher() i=0 2024-03-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.466+0000 7efcdb759980 20 add_watcher() i=1 2024-03-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.466+0000 7efcdb759980 20 add_watcher() i=2 2024-03-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.466+0000 7efcdb759980 20 add_watcher() i=3 2024-03-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.466+0000 7efcdb759980 20 add_watcher() i=4 2024-03-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.466+0000 7efcdb759980 20 add_watcher() i=5 2024-03-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.466+0000 7efcdb759980 20 add_watcher() i=6 2024-03-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.466+0000 7efcdb759980 20 add_watcher() i=7 2024-03-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.466+0000 7efcdb759980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:47.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.470+0000 7efb8f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.470+0000 7efcdb759980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.470+0000 7efcdb759980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.470+0000 7efcdb759980 5 note: GC not initialized 2024-03-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.470+0000 7efb8e7e4640 20 reqs_thread_entry: start 2024-03-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.470+0000 7efcdb759980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.470+0000 7efcdb759980 20 Filter name: none 2024-03-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.470+0000 7efafffff640 20 reqs_thread_entry: start 2024-03-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.470+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.474+0000 7efcdb759980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.474+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.474+0000 7efcdb759980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:47.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.474+0000 7efcdb759980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.474+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.474+0000 7efcdb759980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.474+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.474+0000 7efcdb759980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:47.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.474+0000 7efcdb759980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:47.480 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.478+0000 7efcdb759980 20 rados->read ofs=0 len=0 2024-03-09T14:51:47.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.478+0000 7efcdb759980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:47.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.478+0000 7efb8f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.482+0000 7efcdb759980 20 remove_watcher() i=0 2024-03-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.482+0000 7efcdb759980 2 removed watcher, disabling cache 2024-03-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.486+0000 7efcdb759980 20 remove_watcher() i=1 2024-03-09T14:51:47.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.486+0000 7efcdb759980 20 remove_watcher() i=2 2024-03-09T14:51:47.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.490+0000 7efcdb759980 20 remove_watcher() i=3 2024-03-09T14:51:47.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.490+0000 7efcdb759980 20 remove_watcher() i=4 2024-03-09T14:51:47.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.494+0000 7efcdb759980 20 remove_watcher() i=5 2024-03-09T14:51:47.497 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.494+0000 7efcdb759980 20 remove_watcher() i=6 2024-03-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:47.494+0000 7efcdb759980 20 remove_watcher() i=7 2024-03-09T14:51:47.513 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:47.513 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:47.513 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-commit-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.25 num_objects: 20 size_kb: 1 num_shards: 2 2024-03-09T14:51:47.513 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:47.514 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:47.515 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:47.984 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:47.984 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:48.166 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard bucket with abort at commit_target_layout 2024-03-09T14:51:48.166 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:48.169 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:48.169 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:48.294 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:48.294 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:48.396 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:48.396 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:48.501 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:48.501 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:48.600 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:48.601 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:48.702 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:48.702 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:48.803 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:48.803 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:48.904 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:48.904 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:49.004 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:49.004 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:49.104 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:49.104 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:49.201 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:49.201 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:49.303 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:49.303 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:49.396 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:49.396 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:49.500 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:49.500 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:49.602 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:49.602 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:49.700 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:49.701 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:49.796 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:49.797 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:49.897 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:49.897 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:49.994 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:49.995 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:50.098 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:50.098 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:50.188 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:50.188 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:50.289 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:50.290 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:51:50.385 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket abort-at-commit-target-layout 2024-03-09T14:51:50.419 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:50.419 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:50.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.462+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.462+0000 7f7307d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:50.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.462+0000 7f7307d81980 20 realm 2024-03-09T14:51:50.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.462+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.462+0000 7f7307d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:50.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.462+0000 7f7307d81980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:50.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.462+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.462+0000 7f7307d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:50.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.462+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.466+0000 7f7307d81980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:50.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.466+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.466+0000 7f7307d81980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:50.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.466+0000 7f7307d81980 20 searching for the correct realm 2024-03-09T14:51:50.476 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.474+0000 7f72fffff640 0 -- 172.21.15.1:0/155339844 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x55d09a01e320 msgr2=0x55d09a020710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:50.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.478+0000 7f7305899640 0 -- 172.21.15.1:0/155339844 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f72f0055b50 msgr2=0x7f72f0052810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:50.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.486+0000 7f7307d81980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:50.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.486+0000 7f7307d81980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:50.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.486+0000 7f7307d81980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:50.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.486+0000 7f7307d81980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:50.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.486+0000 7f7307d81980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:50.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.486+0000 7f7307d81980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:50.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.486+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:50.492 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:50.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:50.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 zone default found 2024-03-09T14:51:50.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 4 Realm: () 2024-03-09T14:51:50.493 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:50.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:50.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:50.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 zonegroup default 2024-03-09T14:51:50.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:50.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:50.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:50.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 started sync module instance, tier type = 2024-03-09T14:51:50.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.490+0000 7f7307d81980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:50.510 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.506+0000 7f7307d81980 20 add_watcher() i=0 2024-03-09T14:51:50.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.506+0000 7f7307d81980 20 add_watcher() i=1 2024-03-09T14:51:50.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.506+0000 7f7307d81980 20 add_watcher() i=2 2024-03-09T14:51:50.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.506+0000 7f7307d81980 20 add_watcher() i=3 2024-03-09T14:51:50.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.506+0000 7f7307d81980 20 add_watcher() i=4 2024-03-09T14:51:50.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.506+0000 7f7307d81980 20 add_watcher() i=5 2024-03-09T14:51:50.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.506+0000 7f7307d81980 20 add_watcher() i=6 2024-03-09T14:51:50.511 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.506+0000 7f7307d81980 20 add_watcher() i=7 2024-03-09T14:51:50.512 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.506+0000 7f7307d81980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:50.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f71bbfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:50.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:50.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:50.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 5 note: GC not initialized 2024-03-09T14:51:50.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f71bafe5640 20 reqs_thread_entry: start 2024-03-09T14:51:50.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:50.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 Filter name: none 2024-03-09T14:51:50.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f71b8fe1640 20 reqs_thread_entry: start 2024-03-09T14:51:50.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:50.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:50.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:50.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:50.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:50.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.514+0000 7f7307d81980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:50.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.518+0000 7f7307d81980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.518+0000 7f7307d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:50.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.518+0000 7f71bbfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:50.526 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.522+0000 7f7307d81980 20 remove_watcher() i=0 2024-03-09T14:51:50.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.522+0000 7f7307d81980 2 removed watcher, disabling cache 2024-03-09T14:51:50.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.526+0000 7f7307d81980 20 remove_watcher() i=1 2024-03-09T14:51:50.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.526+0000 7f7307d81980 20 remove_watcher() i=2 2024-03-09T14:51:50.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.530+0000 7f7307d81980 20 remove_watcher() i=3 2024-03-09T14:51:50.534 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.530+0000 7f7307d81980 20 remove_watcher() i=4 2024-03-09T14:51:50.536 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.534+0000 7f7307d81980 20 remove_watcher() i=5 2024-03-09T14:51:50.537 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.534+0000 7f7307d81980 20 remove_watcher() i=6 2024-03-09T14:51:50.539 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.534+0000 7f7307d81980 20 remove_watcher() i=7 2024-03-09T14:51:50.552 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:50.552 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:50.552 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: abort-at-commit-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:51:50.552 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:50.552 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket abort-at-commit-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-abort-at commit_target_layout 2024-03-09T14:51:50.583 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:50.583 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:50.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:50.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.622+0000 7f39a48b7980 20 realm 2024-03-09T14:51:50.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:50.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.622+0000 7f39a48b7980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:50.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:50.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.626+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:50.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.626+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:50.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.626+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:50.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.626+0000 7f39a48b7980 20 searching for the correct realm 2024-03-09T14:51:50.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:50.630+0000 7f39a1bce640 0 -- 172.21.15.1:0/536104429 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x7f386c075d40 msgr2=0x7f386c078170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:51.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.246+0000 7f39a23cf640 0 -- 172.21.15.1:0/536104429 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f399810e240 msgr2=0x7f399810a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:51.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:51.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:51.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:51.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:51.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:51.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:51.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.462+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:51.469 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:51.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 20 zone default found 2024-03-09T14:51:51.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 4 Realm: () 2024-03-09T14:51:51.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:51.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:51.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:51.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 20 zonegroup default 2024-03-09T14:51:51.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.470+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.470+0000 7f39a48b7980 20 started sync module instance, tier type = 2024-03-09T14:51:51.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.470+0000 7f39a48b7980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:51.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=0 2024-03-09T14:51:51.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=1 2024-03-09T14:51:51.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=2 2024-03-09T14:51:51.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=3 2024-03-09T14:51:51.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=4 2024-03-09T14:51:51.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=5 2024-03-09T14:51:51.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=6 2024-03-09T14:51:51.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.494+0000 7f39a48b7980 20 add_watcher() i=7 2024-03-09T14:51:51.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.494+0000 7f39a48b7980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:51.503 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.498+0000 7f38597e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:51.503 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.498+0000 7f39a48b7980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:51.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.498+0000 7f39a48b7980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:51.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.498+0000 7f39a48b7980 5 note: GC not initialized 2024-03-09T14:51:51.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f38537fe640 20 reqs_thread_entry: start 2024-03-09T14:51:51.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:51.504 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 20 Filter name: none 2024-03-09T14:51:51.505 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f38517fa640 20 reqs_thread_entry: start 2024-03-09T14:51:51.505 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 10 cache get: name=default.rgw.meta+root+abort-at-commit-target-layout : miss 2024-03-09T14:51:51.505 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:51.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 10 cache put: name=default.rgw.meta+root+abort-at-commit-target-layout info.flags=0x11 2024-03-09T14:51:51.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 10 adding default.rgw.meta+root+abort-at-commit-target-layout to cache LRU end 2024-03-09T14:51:51.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 : miss 2024-03-09T14:51:51.506 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:51.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:51:51.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:51:51.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:51.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:51.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.502+0000 7f39a48b7980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 2024-03-09T14:51:51.516 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.514+0000 7f39a48b7980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:51:51.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.514+0000 7f39a48b7980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:51:51.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.514+0000 7f39a48b7980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:51.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.514+0000 7f39a48b7980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26, ofs0, ns] 2024-03-09T14:51:51.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.514+0000 7f3885ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562410 cookie 94723478501152 notifier 6825 bl.length()=923 2024-03-09T14:51:51.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.514+0000 7f3885ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:51:51.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.514+0000 7f3885ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:51:51.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.514+0000 7f3885ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:51.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.518+0000 7f39a48b7980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): orig_sources= new_sources= 2024-03-09T14:51:51.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.518+0000 7f39a48b7980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): potential sources added=[] removed=[] 2024-03-09T14:51:51.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.518+0000 7f39a48b7980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): orig_dests= new_dests= 2024-03-09T14:51:51.520 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.518+0000 7f39a48b7980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): potential dests added=[] removed=[] 2024-03-09T14:51:51.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.518+0000 7f39a48b7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.0 2024-03-09T14:51:51.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.518+0000 7f39a48b7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.0 2024-03-09T14:51:51.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.518+0000 7f39a48b7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.1 2024-03-09T14:51:51.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.518+0000 7f39a48b7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.1 2024-03-09T14:51:51.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.518+0000 7f39a48b7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.0 2024-03-09T14:51:51.527 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f39a48b7980 time 2024-03-09T14:51:51.529450+0000 2024-03-09T14:51:51.527 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:51:51.530 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:51.530 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f39a818cd7b] 2024-03-09T14:51:51.530 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f39a818cf2f] 2024-03-09T14:51:51.530 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5626864afc51] 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x577) [0x5626864d0557] 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr: 5: main() 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f39a6ff7d90] 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr: 7: __libc_start_main() 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr: 8: _start() 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr:*** Caught signal (Aborted) ** 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f39a48b7980 thread_name:radosgw-admin 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.526+0000 7f39a48b7980 -1 ./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f39a48b7980 time 2024-03-09T14:51:51.529450+0000 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f39a818cd7b] 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f39a818cf2f] 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5626864afc51] 2024-03-09T14:51:51.531 INFO:tasks.workunit.client.0.smithi001.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x577) [0x5626864d0557] 2024-03-09T14:51:51.532 INFO:tasks.workunit.client.0.smithi001.stderr: 5: main() 2024-03-09T14:51:51.532 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f39a6ff7d90] 2024-03-09T14:51:51.532 INFO:tasks.workunit.client.0.smithi001.stderr: 7: __libc_start_main() 2024-03-09T14:51:51.532 INFO:tasks.workunit.client.0.smithi001.stderr: 8: _start() 2024-03-09T14:51:51.532 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.555 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:51.555 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f39a7010520] 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f39a818cdd5] 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f39a818cf2f] 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5626864afc51] 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x577) [0x5626864d0557] 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: 9: main() 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f39a6ff7d90] 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: 11: __libc_start_main() 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: 12: _start() 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.530+0000 7f39a48b7980 -1 *** Caught signal (Aborted) ** 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f39a48b7980 thread_name:radosgw-admin 2024-03-09T14:51:51.556 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f39a7010520] 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f39a818cdd5] 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f39a818cf2f] 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5626864afc51] 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x577) [0x5626864d0557] 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: 9: main() 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f39a6ff7d90] 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: 11: __libc_start_main() 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: 12: _start() 2024-03-09T14:51:51.557 INFO:tasks.workunit.client.0.smithi001.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr:--- begin dump of recent events --- 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr: -499> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 monclient: authenticate success, global_id 6389 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr: -498> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: get_monmap_and_config success 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr: -497> 2024-03-09T14:51:50.602+0000 7f39a48b7980 4 set_mon_vals no callback set 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr: -496> 2024-03-09T14:51:50.602+0000 7f3890608640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr: -495> 2024-03-09T14:51:50.602+0000 7f3890608640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr: -494> 2024-03-09T14:51:50.602+0000 7f3890608640 10 monclient: dump: 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.558 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr: -493> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: shutdown 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr: -492> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 asok(0x562687196bb0) unregister_commands rotate-key 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr: -491> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: build_initial_monmap 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr: -490> 2024-03-09T14:51:50.602+0000 7f39a48b7980 1 build_initial for_mkfs: 0 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr: -489> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: monmap: 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:51.559 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr: -488> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding auth protocol: cephx 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr: -487> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding auth protocol: cephx 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr: -486> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding auth protocol: cephx 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr: -485> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: secure 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr: -484> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: crc 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr: -483> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: secure 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr: -482> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: crc 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr: -481> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: secure 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr: -480> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: crc 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr: -479> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: crc 2024-03-09T14:51:51.560 INFO:tasks.workunit.client.0.smithi001.stderr: -478> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: secure 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -477> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: crc 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -476> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: secure 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -475> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: crc 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -474> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: secure 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -473> 2024-03-09T14:51:50.602+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -472> 2024-03-09T14:51:50.602+0000 7f39a48b7980 1 librados: starting msgr at 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -471> 2024-03-09T14:51:50.602+0000 7f39a48b7980 1 librados: starting objecter 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -470> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command objecter_requests hook 0x5626871c2f40 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -469> 2024-03-09T14:51:50.602+0000 7f39a48b7980 1 librados: setting wanted keys 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -468> 2024-03-09T14:51:50.602+0000 7f39a48b7980 1 librados: calling monclient init 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -467> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: init 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -466> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding auth protocol: cephx 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -465> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding auth protocol: cephx 2024-03-09T14:51:51.561 INFO:tasks.workunit.client.0.smithi001.stderr: -464> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding auth protocol: cephx 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -463> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: secure 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -462> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: crc 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -461> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: secure 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -460> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: crc 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -459> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: secure 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -458> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: crc 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -457> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: crc 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -456> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: secure 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -455> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: crc 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -454> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: secure 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -453> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: crc 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -452> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: secure 2024-03-09T14:51:51.562 INFO:tasks.workunit.client.0.smithi001.stderr: -451> 2024-03-09T14:51:50.602+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -450> 2024-03-09T14:51:50.602+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -449> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command rotate-key hook 0x5626873a2d38 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -448> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -447> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: _add_conns ranks=[2,0,1] 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -446> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-c con 0x56268725ba60 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -445> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-a con 0x56268725be40 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -444> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-b con 0x5626872ec590 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -443> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -442> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -441> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -440> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -439> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): authenticate will time out at 2404.351096s 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -438> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): get_auth_request con 0x56268725be40 auth_method 0 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -437> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.563 INFO:tasks.workunit.client.0.smithi001.stderr: -436> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -435> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -434> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): get_auth_request con 0x56268725ba60 auth_method 0 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -433> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -432> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -431> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -430> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -429> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -428> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -427> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -426> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -425> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -424> 2024-03-09T14:51:50.602+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request con 0x5626872ec590 auth_method 0 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -423> 2024-03-09T14:51:50.602+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -422> 2024-03-09T14:51:50.602+0000 7f39a2bd0640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.564 INFO:tasks.workunit.client.0.smithi001.stderr: -421> 2024-03-09T14:51:50.602+0000 7f39a2bd0640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr: -420> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): handle_auth_done global_id 6816 payload 824 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr: -419> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient: _finish_hunting 0 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr: -418> 2024-03-09T14:51:50.602+0000 7f39a1bce640 1 monclient: found mon.noname-a 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr: -417> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr: -416> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr: -415> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr: -414> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: dump: 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.565 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr: -413> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: _finish_auth 0 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr: -412> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: _check_auth_tickets 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr: -411> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr: -410> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr: -409> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr: -408> 2024-03-09T14:51:50.606+0000 7f388ee05640 10 monclient: dump: 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.566 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr: -407> 2024-03-09T14:51:50.606+0000 7f39a48b7980 5 monclient: authenticate success, global_id 6816 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr: -406> 2024-03-09T14:51:50.602+0000 7f3890608640 4 set_mon_vals no callback set 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr: -405> 2024-03-09T14:51:50.606+0000 7f39a48b7980 10 monclient: _renew_subs 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr: -404> 2024-03-09T14:51:50.606+0000 7f39a48b7980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr: -403> 2024-03-09T14:51:50.606+0000 7f39a48b7980 10 monclient: _renew_subs 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr: -402> 2024-03-09T14:51:50.606+0000 7f39a48b7980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr: -401> 2024-03-09T14:51:50.606+0000 7f39a48b7980 1 librados: init done 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr: -400> 2024-03-09T14:51:50.606+0000 7f388ee05640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr: -399> 2024-03-09T14:51:50.606+0000 7f388ee05640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr: -398> 2024-03-09T14:51:50.606+0000 7f388ee05640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:51.567 INFO:tasks.workunit.client.0.smithi001.stderr: -397> 2024-03-09T14:51:50.606+0000 7f39a23cf640 10 monclient: get_auth_request con 0x7f3888066070 auth_method 0 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr: -396> 2024-03-09T14:51:50.606+0000 7f39a2bd0640 10 monclient: get_auth_request con 0x5626873cfc50 auth_method 0 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr: -395> 2024-03-09T14:51:50.606+0000 7f39a23cf640 10 monclient: get_auth_request con 0x5626873ed480 auth_method 0 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr: -394> 2024-03-09T14:51:50.610+0000 7f39a1bce640 10 monclient: get_auth_request con 0x5626873f2510 auth_method 0 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr: -393> 2024-03-09T14:51:50.610+0000 7f39a48b7980 10 monclient: build_initial_monmap 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr: -392> 2024-03-09T14:51:50.610+0000 7f39a48b7980 1 build_initial for_mkfs: 0 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr: -391> 2024-03-09T14:51:50.610+0000 7f39a48b7980 10 monclient: monmap: 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:51.568 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -390> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding auth protocol: cephx 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -389> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding auth protocol: cephx 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -388> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding auth protocol: cephx 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -387> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: secure 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -386> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: crc 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -385> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: secure 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -384> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: crc 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -383> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: secure 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -382> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: crc 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -381> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: crc 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -380> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: secure 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -379> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: crc 2024-03-09T14:51:51.569 INFO:tasks.workunit.client.0.smithi001.stderr: -378> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: secure 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -377> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: crc 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -376> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: secure 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -375> 2024-03-09T14:51:50.610+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -374> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command objecter_requests cmddesc objecter_requests hook 0x5626873fbc60 EEXIST 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -373> 2024-03-09T14:51:50.610+0000 7f39a48b7980 10 monclient: init 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -372> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding auth protocol: cephx 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -371> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding auth protocol: cephx 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -370> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding auth protocol: cephx 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -369> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: secure 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -368> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: crc 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -367> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: secure 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -366> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: crc 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -365> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: secure 2024-03-09T14:51:51.570 INFO:tasks.workunit.client.0.smithi001.stderr: -364> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: crc 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -363> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: crc 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -362> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: secure 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -361> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: crc 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -360> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: secure 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -359> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: crc 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -358> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: secure 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -357> 2024-03-09T14:51:50.614+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -356> 2024-03-09T14:51:50.614+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -355> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command rotate-key cmddesc rotate-key hook 0x5626873f5600 EEXIST 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -354> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -353> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: _add_conns ranks=[0,1,2] 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -352> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-a con 0x5626874208c0 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -351> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-b con 0x562687420ca0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:51.571 INFO:tasks.workunit.client.0.smithi001.stderr: -350> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-c con 0x56268741be60 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -349> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -348> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -347> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -346> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -345> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): authenticate will time out at 2404.360492s 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -344> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): get_auth_request con 0x5626874208c0 auth_method 0 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -343> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -342> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -341> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -340> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request con 0x56268741be60 auth_method 0 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -339> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -338> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -337> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -336> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.572 INFO:tasks.workunit.client.0.smithi001.stderr: -335> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr: -334> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr: -333> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr: -332> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr: -331> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr: -330> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): handle_auth_done global_id 6819 payload 824 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr: -329> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient: _finish_hunting 0 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr: -328> 2024-03-09T14:51:50.614+0000 7f39a23cf640 1 monclient: found mon.noname-a 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr: -327> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr: -326> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr: -325> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr: -324> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: dump: 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.573 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr: -323> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: _finish_auth 0 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr: -322> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: _check_auth_tickets 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr: -321> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr: -320> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr: -319> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr: -318> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: dump: 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.574 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr: -317> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 monclient: authenticate success, global_id 6819 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr: -316> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: _renew_subs 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr: -315> 2024-03-09T14:51:50.614+0000 7f39a33d1640 4 set_mon_vals no callback set 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr: -314> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr: -313> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: _renew_subs 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr: -312> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.575 INFO:tasks.workunit.client.0.smithi001.stderr: -311> 2024-03-09T14:51:50.614+0000 7f3887fff640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr: -310> 2024-03-09T14:51:50.614+0000 7f3887fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr: -309> 2024-03-09T14:51:50.614+0000 7f3887fff640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr: -308> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: get_monmap_and_config 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr: -307> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: build_initial_monmap 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr: -306> 2024-03-09T14:51:50.614+0000 7f39a48b7980 1 build_initial for_mkfs: 0 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr: -305> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: monmap: 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:51.576 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -304> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding auth protocol: cephx 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -303> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding auth protocol: cephx 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -302> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding auth protocol: cephx 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -301> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -300> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -299> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -298> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -297> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -296> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -295> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -294> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -293> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -292> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.577 INFO:tasks.workunit.client.0.smithi001.stderr: -291> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -290> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -289> 2024-03-09T14:51:50.614+0000 7f39a1bce640 10 monclient: get_auth_request con 0x7f387c065f80 auth_method 0 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -288> 2024-03-09T14:51:50.614+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -287> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: init 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -286> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding auth protocol: cephx 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -285> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding auth protocol: cephx 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -284> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding auth protocol: cephx 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -283> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: secure 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -282> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: crc 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -281> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: secure 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -280> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: crc 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -279> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: secure 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -278> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: crc 2024-03-09T14:51:51.578 INFO:tasks.workunit.client.0.smithi001.stderr: -277> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: crc 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -276> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: secure 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -275> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: crc 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -274> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: secure 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -273> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: crc 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -272> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: secure 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -271> 2024-03-09T14:51:50.614+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -270> 2024-03-09T14:51:50.618+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -269> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command rotate-key cmddesc rotate-key hook 0x7ffe50bada68 EEXIST 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -268> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -267> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _add_conns ranks=[2,1,0] 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -266> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-c con 0x562687464af0 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -265> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-b con 0x56268746fab0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -264> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-a con 0x562687465410 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -263> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.579 INFO:tasks.workunit.client.0.smithi001.stderr: -262> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -261> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -260> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -259> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): authenticate will time out at 2404.364280s 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -258> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): get_auth_request con 0x562687464af0 auth_method 0 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -257> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -256> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -255> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -254> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request con 0x562687465410 auth_method 0 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -253> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -252> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -251> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -250> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -249> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.580 INFO:tasks.workunit.client.0.smithi001.stderr: -248> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr: -247> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr: -246> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr: -245> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr: -244> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_done global_id 6822 payload 274 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr: -243> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient: _finish_hunting 0 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr: -242> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 1 monclient: found mon.noname-a 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr: -241> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr: -240> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient: get_auth_request con 0x56268746fab0 auth_method 0 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr: -239> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr: -238> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr: -237> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: dump: 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.581 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr: -236> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: _finish_auth 0 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr: -235> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: _check_auth_tickets 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr: -234> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr: -233> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr: -232> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr: -231> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: dump: 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.582 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr: -230> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 monclient: authenticate success, global_id 6822 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr: -229> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: get_monmap_and_config success 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr: -228> 2024-03-09T14:51:50.618+0000 7f39a48b7980 4 set_mon_vals no callback set 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr: -227> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: shutdown 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr: -226> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: build_initial_monmap 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr: -225> 2024-03-09T14:51:50.618+0000 7f39a48b7980 1 build_initial for_mkfs: 0 2024-03-09T14:51:51.583 INFO:tasks.workunit.client.0.smithi001.stderr: -224> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: monmap: 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr: -223> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding auth protocol: cephx 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr: -222> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding auth protocol: cephx 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr: -221> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding auth protocol: cephx 2024-03-09T14:51:51.584 INFO:tasks.workunit.client.0.smithi001.stderr: -220> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -219> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -218> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -217> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -216> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -215> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -214> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -213> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -212> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -211> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -210> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -209> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -208> 2024-03-09T14:51:50.618+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -207> 2024-03-09T14:51:50.618+0000 7f39a48b7980 1 librados: starting msgr at 2024-03-09T14:51:51.585 INFO:tasks.workunit.client.0.smithi001.stderr: -206> 2024-03-09T14:51:50.618+0000 7f39a48b7980 1 librados: starting objecter 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -205> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command objecter_requests cmddesc objecter_requests hook 0x56268746d800 EEXIST 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -204> 2024-03-09T14:51:50.618+0000 7f39a48b7980 1 librados: setting wanted keys 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -203> 2024-03-09T14:51:50.618+0000 7f39a48b7980 1 librados: calling monclient init 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -202> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: init 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -201> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding auth protocol: cephx 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -200> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding auth protocol: cephx 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -199> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding auth protocol: cephx 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -198> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: secure 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -197> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: crc 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -196> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: secure 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -195> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: crc 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -194> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: secure 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -193> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: crc 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -192> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: crc 2024-03-09T14:51:51.586 INFO:tasks.workunit.client.0.smithi001.stderr: -191> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: secure 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -190> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: crc 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -189> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: secure 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -188> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: crc 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -187> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: secure 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -186> 2024-03-09T14:51:50.618+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -185> 2024-03-09T14:51:50.618+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -184> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command rotate-key cmddesc rotate-key hook 0x56268743d728 EEXIST 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -183> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -182> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _add_conns ranks=[1,0,2] 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -181> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-b con 0x562687464af0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -180> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-a con 0x562687465410 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -179> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-c con 0x56268746fab0 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -178> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.587 INFO:tasks.workunit.client.0.smithi001.stderr: -177> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -176> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -175> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -174> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): authenticate will time out at 2404.366969s 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -173> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): get_auth_request con 0x562687465410 auth_method 0 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -172> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -171> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -170> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -169> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request con 0x56268746fab0 auth_method 0 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -168> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -167> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -166> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -165> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -164> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -163> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.588 INFO:tasks.workunit.client.0.smithi001.stderr: -162> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -161> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -160> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -159> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): get_auth_request con 0x562687464af0 auth_method 0 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -158> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -157> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -156> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -155> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): handle_auth_done global_id 6825 payload 824 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -154> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient: _finish_hunting 0 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -153> 2024-03-09T14:51:50.618+0000 7f39a1bce640 1 monclient: found mon.noname-a 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -152> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -151> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -150> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr: -149> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: dump: 2024-03-09T14:51:51.589 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr: -148> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: _finish_auth 0 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr: -147> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: _check_auth_tickets 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr: -146> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 monclient: authenticate success, global_id 6825 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr: -145> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:51.590 INFO:tasks.workunit.client.0.smithi001.stderr: -144> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr: -143> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr: -142> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: dump: 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr: -141> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _renew_subs 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr: -140> 2024-03-09T14:51:50.618+0000 7f3885ffb640 4 set_mon_vals no callback set 2024-03-09T14:51:51.591 INFO:tasks.workunit.client.0.smithi001.stderr: -139> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -138> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _renew_subs 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -137> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -136> 2024-03-09T14:51:50.618+0000 7f39a48b7980 1 librados: init done 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -135> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command cr dump hook 0x5626874699f8 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -134> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -133> 2024-03-09T14:51:50.622+0000 7f3873fff640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -132> 2024-03-09T14:51:50.622+0000 7f3873fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -131> 2024-03-09T14:51:50.622+0000 7f3873fff640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -130> 2024-03-09T14:51:50.622+0000 7f39a23cf640 10 monclient: get_auth_request con 0x7f386c066070 auth_method 0 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -129> 2024-03-09T14:51:50.622+0000 7f39a2bd0640 10 monclient: get_auth_request con 0x56268748c560 auth_method 0 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -128> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -127> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 realm 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -126> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -125> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.592 INFO:tasks.workunit.client.0.smithi001.stderr: -124> 2024-03-09T14:51:50.622+0000 7f39a48b7980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -123> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -122> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -121> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -120> 2024-03-09T14:51:50.622+0000 7f39a23cf640 10 monclient: get_auth_request con 0x562687490750 auth_method 0 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -119> 2024-03-09T14:51:50.626+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -118> 2024-03-09T14:51:50.626+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -117> 2024-03-09T14:51:50.626+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -116> 2024-03-09T14:51:50.626+0000 7f39a48b7980 20 searching for the correct realm 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -115> 2024-03-09T14:51:50.626+0000 7f39a1bce640 10 monclient: get_auth_request con 0x562687495980 auth_method 0 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -114> 2024-03-09T14:51:50.626+0000 7f39a2bd0640 10 monclient: get_auth_request con 0x7f39980c0020 auth_method 0 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -113> 2024-03-09T14:51:50.630+0000 7f39a23cf640 10 monclient: get_auth_request con 0x7f399c028040 auth_method 0 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -112> 2024-03-09T14:51:50.630+0000 7f39a1bce640 10 monclient: get_auth_request con 0x7f3994079040 auth_method 0 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -111> 2024-03-09T14:51:50.630+0000 7f3873fff640 1 client.6825.objecter ms_handle_reset 0x562687495980 session 0x562687495300 osd.3 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -110> 2024-03-09T14:51:50.630+0000 7f3873fff640 10 monclient: _renew_subs 2024-03-09T14:51:51.593 INFO:tasks.workunit.client.0.smithi001.stderr: -109> 2024-03-09T14:51:50.630+0000 7f3873fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -108> 2024-03-09T14:51:50.630+0000 7f39a1bce640 10 monclient: get_auth_request con 0x7f386c075d40 auth_method 0 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -107> 2024-03-09T14:51:50.630+0000 7f39a1bce640 0 -- 172.21.15.1:0/536104429 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x7f386c075d40 msgr2=0x7f386c078170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -106> 2024-03-09T14:51:50.834+0000 7f39a1bce640 10 monclient: get_auth_request con 0x7f386c075d40 auth_method 0 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -105> 2024-03-09T14:51:51.238+0000 7f39a1bce640 10 monclient: get_auth_request con 0x7f386c075d40 auth_method 0 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -104> 2024-03-09T14:51:51.238+0000 7f39a2bd0640 10 monclient: get_auth_request con 0x7f39980037c0 auth_method 0 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -103> 2024-03-09T14:51:51.246+0000 7f39a23cf640 10 monclient: get_auth_request con 0x7f399810e240 auth_method 0 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -102> 2024-03-09T14:51:51.246+0000 7f39a23cf640 0 -- 172.21.15.1:0/536104429 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f399810e240 msgr2=0x7f399810a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -101> 2024-03-09T14:51:51.450+0000 7f39a23cf640 10 monclient: get_auth_request con 0x7f399810e240 auth_method 0 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -100> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -99> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -98> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -97> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -96> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:51.594 INFO:tasks.workunit.client.0.smithi001.stderr: -95> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -94> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -93> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -92> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -91> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -90> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -89> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -88> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 zone default found 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -87> 2024-03-09T14:51:51.466+0000 7f39a48b7980 4 Realm: () 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -86> 2024-03-09T14:51:51.466+0000 7f39a48b7980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -85> 2024-03-09T14:51:51.466+0000 7f39a48b7980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -84> 2024-03-09T14:51:51.466+0000 7f39a48b7980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -83> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 zonegroup default 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -82> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -81> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.595 INFO:tasks.workunit.client.0.smithi001.stderr: -80> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -79> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -78> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -77> 2024-03-09T14:51:51.470+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -76> 2024-03-09T14:51:51.470+0000 7f39a48b7980 20 started sync module instance, tier type = 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -75> 2024-03-09T14:51:51.470+0000 7f39a48b7980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -74> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=0 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -73> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=1 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -72> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=2 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -71> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=3 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -70> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=4 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -69> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=5 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -68> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=6 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -67> 2024-03-09T14:51:51.494+0000 7f39a48b7980 20 add_watcher() i=7 2024-03-09T14:51:51.596 INFO:tasks.workunit.client.0.smithi001.stderr: -66> 2024-03-09T14:51:51.494+0000 7f39a48b7980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -65> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding auth protocol: cephx 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -64> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding auth protocol: cephx 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -63> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding auth protocol: cephx 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -62> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: secure 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -61> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: crc 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -60> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: secure 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -59> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: crc 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -58> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: secure 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -57> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: crc 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -56> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: crc 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -55> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: secure 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -54> 2024-03-09T14:51:51.498+0000 7f38597e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -53> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: crc 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -52> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: secure 2024-03-09T14:51:51.597 INFO:tasks.workunit.client.0.smithi001.stderr: -51> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: crc 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -50> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: secure 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -49> 2024-03-09T14:51:51.498+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -48> 2024-03-09T14:51:51.498+0000 7f39a48b7980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -47> 2024-03-09T14:51:51.498+0000 7f39a48b7980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -46> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command cache list hook 0x562687472190 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -45> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command cache inspect hook 0x562687472190 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -44> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command cache erase hook 0x562687472190 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -43> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command cache zap hook 0x562687472190 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -42> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 note: GC not initialized 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -41> 2024-03-09T14:51:51.502+0000 7f38537fe640 20 reqs_thread_entry: start 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -40> 2024-03-09T14:51:51.502+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command sync trace show hook 0x7f3998004470 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -39> 2024-03-09T14:51:51.502+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command sync trace history hook 0x7f3998004470 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -38> 2024-03-09T14:51:51.502+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command sync trace active hook 0x7f3998004470 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -37> 2024-03-09T14:51:51.502+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command sync trace active_short hook 0x7f3998004470 2024-03-09T14:51:51.598 INFO:tasks.workunit.client.0.smithi001.stderr: -36> 2024-03-09T14:51:51.502+0000 7f39a48b7980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -35> 2024-03-09T14:51:51.502+0000 7f39a48b7980 20 Filter name: none 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -34> 2024-03-09T14:51:51.502+0000 7f38517fa640 20 reqs_thread_entry: start 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -33> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 cache get: name=default.rgw.meta+root+abort-at-commit-target-layout : miss 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -32> 2024-03-09T14:51:51.502+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -31> 2024-03-09T14:51:51.502+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -30> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 cache put: name=default.rgw.meta+root+abort-at-commit-target-layout info.flags=0x11 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -29> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 adding default.rgw.meta+root+abort-at-commit-target-layout to cache LRU end 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -28> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 : miss 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -27> 2024-03-09T14:51:51.502+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -26> 2024-03-09T14:51:51.502+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -25> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -24> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -23> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:51.599 INFO:tasks.workunit.client.0.smithi001.stderr: -22> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -21> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -20> 2024-03-09T14:51:51.506+0000 7f3873fff640 1 client.6825.objecter ms_handle_reset 0x7f3994079040 session 0x7f3994084200 osd.4 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -19> 2024-03-09T14:51:51.506+0000 7f39a1bce640 10 monclient: get_auth_request con 0x562687495980 auth_method 0 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -18> 2024-03-09T14:51:51.514+0000 7f39a48b7980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -17> 2024-03-09T14:51:51.514+0000 7f39a48b7980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -16> 2024-03-09T14:51:51.514+0000 7f39a48b7980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -15> 2024-03-09T14:51:51.514+0000 7f39a48b7980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26, ofs0, ns] 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -14> 2024-03-09T14:51:51.514+0000 7f3885ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562410 cookie 94723478501152 notifier 6825 bl.length()=923 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -13> 2024-03-09T14:51:51.514+0000 7f3885ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -12> 2024-03-09T14:51:51.514+0000 7f3885ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -11> 2024-03-09T14:51:51.514+0000 7f3885ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -10> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): orig_sources= new_sources= 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -9> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): potential sources added=[] removed=[] 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -8> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): orig_dests= new_dests= 2024-03-09T14:51:51.600 INFO:tasks.workunit.client.0.smithi001.stderr: -7> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): potential dests added=[] removed=[] 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: -6> 2024-03-09T14:51:51.518+0000 7f39a48b7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.0 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: -5> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.0 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: -4> 2024-03-09T14:51:51.518+0000 7f39a48b7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.1 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: -3> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.1 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: -2> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.0 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: -1> 2024-03-09T14:51:51.526+0000 7f39a48b7980 -1 ./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f39a48b7980 time 2024-03-09T14:51:51.529450+0000 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f39a818cd7b] 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f39a818cf2f] 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5626864afc51] 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x577) [0x5626864d0557] 2024-03-09T14:51:51.601 INFO:tasks.workunit.client.0.smithi001.stderr: 5: main() 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f39a6ff7d90] 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 7: __libc_start_main() 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 8: _start() 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 0> 2024-03-09T14:51:51.530+0000 7f39a48b7980 -1 *** Caught signal (Aborted) ** 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f39a48b7980 thread_name:radosgw-admin 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f39a7010520] 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f39a818cdd5] 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f39a818cf2f] 2024-03-09T14:51:51.602 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5626864afc51] 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x577) [0x5626864d0557] 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 9: main() 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f39a6ff7d90] 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 11: __libc_start_main() 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 12: _start() 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr:--- logging levels --- 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 none 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 lockdep 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 context 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 crush 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_balancer 2024-03-09T14:51:51.603 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_locker 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log_expire 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_migrator 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 buffer 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 timer 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 filer 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 striper 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 objecter 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rados 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_mirror 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_replay 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_pwl 2024-03-09T14:51:51.604 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 journaler 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objectcacher 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 immutable_obj_cache 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 client 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 osd 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 optracker 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objclass 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 filestore 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 journal 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 ms 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mon 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 0/10 monc 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 paxos 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 tp 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 auth 2024-03-09T14:51:51.605 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 crypto 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 finisher 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 reserver 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 heartbeatmap 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 perfcounter 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 20/20 rgw 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_sync 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_datacache 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_access 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_dbstore 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_flight 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_lifecycle 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 javaclient 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 asok 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 throttle 2024-03-09T14:51:51.606 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 refs 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 compressor 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluestore 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluefs 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 bdev 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 kstore 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 4/ 5 rocksdb 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 fuse 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 2/ 5 mgr 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mgrc 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 dpdk 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 eventtrace 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 prioritycache 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 test 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephfs_mirror 2024-03-09T14:51:51.607 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephsqlite 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 crimson_interrupt 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_onode 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_odata 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_omap 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_tm 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_t 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cleaner 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_epm 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_lba 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_fixedkv_tree 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cache 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_journal 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_device 2024-03-09T14:51:51.608 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_backref 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 alienstore 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mclock 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cyanstore 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 ceph_exporter 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 memstore 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 trace 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: -2/-2 (syslog threshold) 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 99/99 (stderr threshold) 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr:--- pthread ID / name mapping for recent threads --- 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 7f38517fa640 / http_manager 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 7f38537fe640 / http_manager 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 7f38597e2640 / rgw_dt_lg_renew 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 7f3873fff640 / ms_dispatch 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 7f3885ffb640 / io_context_pool 2024-03-09T14:51:51.609 INFO:tasks.workunit.client.0.smithi001.stderr: 7f3887fff640 / ms_dispatch 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr: 7f388ee05640 / ms_dispatch 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr: 7f3890608640 / io_context_pool 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr: 7f39a1bce640 / msgr-worker-2 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr: 7f39a23cf640 / msgr-worker-1 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr: 7f39a2bd0640 / msgr-worker-0 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr: 7f39a33d1640 / io_context_pool 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr: 7f39a48b7980 / radosgw-admin 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr: max_recent 500 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr: max_new 1000 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr: log_file /var/log/ceph/ceph-client.admin.95466.log 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr:--- end dump of recent events --- 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr:--- begin dump of recent events --- 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr: -499> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 monclient: authenticate success, global_id 6389 2024-03-09T14:51:51.610 INFO:tasks.workunit.client.0.smithi001.stderr: -498> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: get_monmap_and_config success 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr: -497> 2024-03-09T14:51:50.602+0000 7f39a48b7980 4 set_mon_vals no callback set 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr: -496> 2024-03-09T14:51:50.602+0000 7f3890608640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr: -495> 2024-03-09T14:51:50.602+0000 7f3890608640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr: -494> 2024-03-09T14:51:50.602+0000 7f3890608640 10 monclient: dump: 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.611 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr: -493> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: shutdown 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr: -492> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 asok(0x562687196bb0) unregister_commands rotate-key 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr: -491> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: build_initial_monmap 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr: -490> 2024-03-09T14:51:50.602+0000 7f39a48b7980 1 build_initial for_mkfs: 0 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr: -489> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: monmap: 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:51.612 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -488> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding auth protocol: cephx 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -487> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding auth protocol: cephx 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -486> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding auth protocol: cephx 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -485> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: secure 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -484> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: crc 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -483> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: secure 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -482> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: crc 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -481> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: secure 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -480> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: crc 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -479> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: crc 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -478> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: secure 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -477> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: crc 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -476> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: secure 2024-03-09T14:51:51.613 INFO:tasks.workunit.client.0.smithi001.stderr: -475> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: crc 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -474> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873d2660) adding con mode: secure 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -473> 2024-03-09T14:51:50.602+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -472> 2024-03-09T14:51:50.602+0000 7f39a48b7980 1 librados: starting msgr at 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -471> 2024-03-09T14:51:50.602+0000 7f39a48b7980 1 librados: starting objecter 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -470> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command objecter_requests hook 0x5626871c2f40 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -469> 2024-03-09T14:51:50.602+0000 7f39a48b7980 1 librados: setting wanted keys 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -468> 2024-03-09T14:51:50.602+0000 7f39a48b7980 1 librados: calling monclient init 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -467> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: init 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -466> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding auth protocol: cephx 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -465> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding auth protocol: cephx 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -464> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding auth protocol: cephx 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -463> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: secure 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -462> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: crc 2024-03-09T14:51:51.614 INFO:tasks.workunit.client.0.smithi001.stderr: -461> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: secure 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -460> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: crc 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -459> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: secure 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -458> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: crc 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -457> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: crc 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -456> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: secure 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -455> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: crc 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -454> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: secure 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -453> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: crc 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -452> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 AuthRegistry(0x5626873a2bf0) adding con mode: secure 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -451> 2024-03-09T14:51:50.602+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -450> 2024-03-09T14:51:50.602+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -449> 2024-03-09T14:51:50.602+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command rotate-key hook 0x5626873a2d38 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -448> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -447> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient: _add_conns ranks=[2,0,1] 2024-03-09T14:51:51.615 INFO:tasks.workunit.client.0.smithi001.stderr: -446> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-c con 0x56268725ba60 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -445> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-a con 0x56268725be40 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -444> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-b con 0x5626872ec590 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -443> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -442> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -441> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -440> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -439> 2024-03-09T14:51:50.602+0000 7f39a48b7980 10 monclient(hunting): authenticate will time out at 2404.351096s 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -438> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): get_auth_request con 0x56268725be40 auth_method 0 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -437> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -436> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -435> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -434> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): get_auth_request con 0x56268725ba60 auth_method 0 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -433> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.616 INFO:tasks.workunit.client.0.smithi001.stderr: -432> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -431> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -430> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -429> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -428> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -427> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -426> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -425> 2024-03-09T14:51:50.602+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -424> 2024-03-09T14:51:50.602+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request con 0x5626872ec590 auth_method 0 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -423> 2024-03-09T14:51:50.602+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -422> 2024-03-09T14:51:50.602+0000 7f39a2bd0640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -421> 2024-03-09T14:51:50.602+0000 7f39a2bd0640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -420> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient(hunting): handle_auth_done global_id 6816 payload 824 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -419> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient: _finish_hunting 0 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -418> 2024-03-09T14:51:50.602+0000 7f39a1bce640 1 monclient: found mon.noname-a 2024-03-09T14:51:51.617 INFO:tasks.workunit.client.0.smithi001.stderr: -417> 2024-03-09T14:51:50.602+0000 7f39a1bce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr: -416> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr: -415> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr: -414> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: dump: 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.618 INFO:tasks.workunit.client.0.smithi001.stderr: -413> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: _finish_auth 0 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr: -412> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: _check_auth_tickets 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr: -411> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr: -410> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr: -409> 2024-03-09T14:51:50.602+0000 7f388ee05640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr: -408> 2024-03-09T14:51:50.606+0000 7f388ee05640 10 monclient: dump: 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.619 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -407> 2024-03-09T14:51:50.606+0000 7f39a48b7980 5 monclient: authenticate success, global_id 6816 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -406> 2024-03-09T14:51:50.602+0000 7f3890608640 4 set_mon_vals no callback set 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -405> 2024-03-09T14:51:50.606+0000 7f39a48b7980 10 monclient: _renew_subs 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -404> 2024-03-09T14:51:50.606+0000 7f39a48b7980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -403> 2024-03-09T14:51:50.606+0000 7f39a48b7980 10 monclient: _renew_subs 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -402> 2024-03-09T14:51:50.606+0000 7f39a48b7980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -401> 2024-03-09T14:51:50.606+0000 7f39a48b7980 1 librados: init done 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -400> 2024-03-09T14:51:50.606+0000 7f388ee05640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -399> 2024-03-09T14:51:50.606+0000 7f388ee05640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -398> 2024-03-09T14:51:50.606+0000 7f388ee05640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -397> 2024-03-09T14:51:50.606+0000 7f39a23cf640 10 monclient: get_auth_request con 0x7f3888066070 auth_method 0 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -396> 2024-03-09T14:51:50.606+0000 7f39a2bd0640 10 monclient: get_auth_request con 0x5626873cfc50 auth_method 0 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -395> 2024-03-09T14:51:50.606+0000 7f39a23cf640 10 monclient: get_auth_request con 0x5626873ed480 auth_method 0 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -394> 2024-03-09T14:51:50.610+0000 7f39a1bce640 10 monclient: get_auth_request con 0x5626873f2510 auth_method 0 2024-03-09T14:51:51.620 INFO:tasks.workunit.client.0.smithi001.stderr: -393> 2024-03-09T14:51:50.610+0000 7f39a48b7980 10 monclient: build_initial_monmap 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr: -392> 2024-03-09T14:51:50.610+0000 7f39a48b7980 1 build_initial for_mkfs: 0 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr: -391> 2024-03-09T14:51:50.610+0000 7f39a48b7980 10 monclient: monmap: 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr: -390> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding auth protocol: cephx 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr: -389> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding auth protocol: cephx 2024-03-09T14:51:51.621 INFO:tasks.workunit.client.0.smithi001.stderr: -388> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding auth protocol: cephx 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -387> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: secure 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -386> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: crc 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -385> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: secure 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -384> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: crc 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -383> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: secure 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -382> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: crc 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -381> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: crc 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -380> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: secure 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -379> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: crc 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -378> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: secure 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -377> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: crc 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -376> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f64d0) adding con mode: secure 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -375> 2024-03-09T14:51:50.610+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.622 INFO:tasks.workunit.client.0.smithi001.stderr: -374> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command objecter_requests cmddesc objecter_requests hook 0x5626873fbc60 EEXIST 2024-03-09T14:51:51.623 INFO:tasks.workunit.client.0.smithi001.stderr: -373> 2024-03-09T14:51:50.610+0000 7f39a48b7980 10 monclient: init 2024-03-09T14:51:51.623 INFO:tasks.workunit.client.0.smithi001.stderr: -372> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding auth protocol: cephx 2024-03-09T14:51:51.623 INFO:tasks.workunit.client.0.smithi001.stderr: -371> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding auth protocol: cephx 2024-03-09T14:51:51.623 INFO:tasks.workunit.client.0.smithi001.stderr: -370> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding auth protocol: cephx 2024-03-09T14:51:51.623 INFO:tasks.workunit.client.0.smithi001.stderr: -369> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: secure 2024-03-09T14:51:51.623 INFO:tasks.workunit.client.0.smithi001.stderr: -368> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: crc 2024-03-09T14:51:51.623 INFO:tasks.workunit.client.0.smithi001.stderr: -367> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: secure 2024-03-09T14:51:51.623 INFO:tasks.workunit.client.0.smithi001.stderr: -366> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: crc 2024-03-09T14:51:51.623 INFO:tasks.workunit.client.0.smithi001.stderr: -365> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: secure 2024-03-09T14:51:51.623 INFO:tasks.workunit.client.0.smithi001.stderr: -364> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: crc 2024-03-09T14:51:51.623 INFO:tasks.workunit.client.0.smithi001.stderr: -363> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: crc 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -362> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: secure 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -361> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: crc 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -360> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: secure 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -359> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: crc 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -358> 2024-03-09T14:51:50.610+0000 7f39a48b7980 5 AuthRegistry(0x5626873f54b8) adding con mode: secure 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -357> 2024-03-09T14:51:50.614+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -356> 2024-03-09T14:51:50.614+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -355> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command rotate-key cmddesc rotate-key hook 0x5626873f5600 EEXIST 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -354> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -353> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: _add_conns ranks=[0,1,2] 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -352> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-a con 0x5626874208c0 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -351> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-b con 0x562687420ca0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -350> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-c con 0x56268741be60 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -349> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.624 INFO:tasks.workunit.client.0.smithi001.stderr: -348> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -347> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -346> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -345> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient(hunting): authenticate will time out at 2404.360492s 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -344> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): get_auth_request con 0x5626874208c0 auth_method 0 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -343> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -342> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -341> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -340> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request con 0x56268741be60 auth_method 0 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -339> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -338> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -337> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -336> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -335> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.625 INFO:tasks.workunit.client.0.smithi001.stderr: -334> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr: -333> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr: -332> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr: -331> 2024-03-09T14:51:50.614+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr: -330> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient(hunting): handle_auth_done global_id 6819 payload 824 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr: -329> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient: _finish_hunting 0 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr: -328> 2024-03-09T14:51:50.614+0000 7f39a23cf640 1 monclient: found mon.noname-a 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr: -327> 2024-03-09T14:51:50.614+0000 7f39a23cf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr: -326> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr: -325> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr: -324> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: dump: 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.626 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr: -323> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: _finish_auth 0 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr: -322> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: _check_auth_tickets 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr: -321> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr: -320> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr: -319> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr: -318> 2024-03-09T14:51:50.614+0000 7f3887fff640 10 monclient: dump: 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.627 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr: -317> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 monclient: authenticate success, global_id 6819 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr: -316> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: _renew_subs 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr: -315> 2024-03-09T14:51:50.614+0000 7f39a33d1640 4 set_mon_vals no callback set 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr: -314> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr: -313> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: _renew_subs 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr: -312> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr: -311> 2024-03-09T14:51:50.614+0000 7f3887fff640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:51:51.628 INFO:tasks.workunit.client.0.smithi001.stderr: -310> 2024-03-09T14:51:50.614+0000 7f3887fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:51.629 INFO:tasks.workunit.client.0.smithi001.stderr: -309> 2024-03-09T14:51:50.614+0000 7f3887fff640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:51.629 INFO:tasks.workunit.client.0.smithi001.stderr: -308> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: get_monmap_and_config 2024-03-09T14:51:51.629 INFO:tasks.workunit.client.0.smithi001.stderr: -307> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: build_initial_monmap 2024-03-09T14:51:51.629 INFO:tasks.workunit.client.0.smithi001.stderr: -306> 2024-03-09T14:51:50.614+0000 7f39a48b7980 1 build_initial for_mkfs: 0 2024-03-09T14:51:51.629 INFO:tasks.workunit.client.0.smithi001.stderr: -305> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: monmap: 2024-03-09T14:51:51.629 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:51.629 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:51.629 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:51.629 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:51.629 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:51.629 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.629 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:51.630 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:51.630 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:51.630 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.630 INFO:tasks.workunit.client.0.smithi001.stderr: -304> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding auth protocol: cephx 2024-03-09T14:51:51.630 INFO:tasks.workunit.client.0.smithi001.stderr: -303> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding auth protocol: cephx 2024-03-09T14:51:51.630 INFO:tasks.workunit.client.0.smithi001.stderr: -302> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding auth protocol: cephx 2024-03-09T14:51:51.630 INFO:tasks.workunit.client.0.smithi001.stderr: -301> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.630 INFO:tasks.workunit.client.0.smithi001.stderr: -300> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.630 INFO:tasks.workunit.client.0.smithi001.stderr: -299> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.630 INFO:tasks.workunit.client.0.smithi001.stderr: -298> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -297> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -296> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -295> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -294> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -293> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -292> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -291> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -290> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -289> 2024-03-09T14:51:50.614+0000 7f39a1bce640 10 monclient: get_auth_request con 0x7f387c065f80 auth_method 0 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -288> 2024-03-09T14:51:50.614+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -287> 2024-03-09T14:51:50.614+0000 7f39a48b7980 10 monclient: init 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -286> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding auth protocol: cephx 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -285> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding auth protocol: cephx 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -284> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding auth protocol: cephx 2024-03-09T14:51:51.631 INFO:tasks.workunit.client.0.smithi001.stderr: -283> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: secure 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -282> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: crc 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -281> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: secure 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -280> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: crc 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -279> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: secure 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -278> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: crc 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -277> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: crc 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -276> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: secure 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -275> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: crc 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -274> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: secure 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -273> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: crc 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -272> 2024-03-09T14:51:50.614+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50bad920) adding con mode: secure 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -271> 2024-03-09T14:51:50.614+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -270> 2024-03-09T14:51:50.618+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.632 INFO:tasks.workunit.client.0.smithi001.stderr: -269> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command rotate-key cmddesc rotate-key hook 0x7ffe50bada68 EEXIST 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -268> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -267> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _add_conns ranks=[2,1,0] 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -266> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-c con 0x562687464af0 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -265> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-b con 0x56268746fab0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -264> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-a con 0x562687465410 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -263> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -262> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -261> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -260> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -259> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): authenticate will time out at 2404.364280s 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -258> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): get_auth_request con 0x562687464af0 auth_method 0 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -257> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -256> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -255> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.633 INFO:tasks.workunit.client.0.smithi001.stderr: -254> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request con 0x562687465410 auth_method 0 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -253> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -252> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -251> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -250> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -249> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -248> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -247> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -246> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -245> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -244> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_done global_id 6822 payload 274 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -243> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient: _finish_hunting 0 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -242> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 1 monclient: found mon.noname-a 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -241> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -240> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient: get_auth_request con 0x56268746fab0 auth_method 0 2024-03-09T14:51:51.634 INFO:tasks.workunit.client.0.smithi001.stderr: -239> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr: -238> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr: -237> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: dump: 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr: -236> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: _finish_auth 0 2024-03-09T14:51:51.635 INFO:tasks.workunit.client.0.smithi001.stderr: -235> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: _check_auth_tickets 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr: -234> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr: -233> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr: -232> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr: -231> 2024-03-09T14:51:50.618+0000 7f3885ffb640 10 monclient: dump: 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.636 INFO:tasks.workunit.client.0.smithi001.stderr: -230> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 monclient: authenticate success, global_id 6822 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr: -229> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: get_monmap_and_config success 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr: -228> 2024-03-09T14:51:50.618+0000 7f39a48b7980 4 set_mon_vals no callback set 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr: -227> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: shutdown 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr: -226> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: build_initial_monmap 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr: -225> 2024-03-09T14:51:50.618+0000 7f39a48b7980 1 build_initial for_mkfs: 0 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr: -224> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: monmap: 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:51:51.637 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -223> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding auth protocol: cephx 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -222> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding auth protocol: cephx 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -221> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding auth protocol: cephx 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -220> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -219> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -218> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -217> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -216> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -215> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -214> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -213> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -212> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.638 INFO:tasks.workunit.client.0.smithi001.stderr: -211> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -210> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: crc 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -209> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743e410) adding con mode: secure 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -208> 2024-03-09T14:51:50.618+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -207> 2024-03-09T14:51:50.618+0000 7f39a48b7980 1 librados: starting msgr at 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -206> 2024-03-09T14:51:50.618+0000 7f39a48b7980 1 librados: starting objecter 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -205> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command objecter_requests cmddesc objecter_requests hook 0x56268746d800 EEXIST 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -204> 2024-03-09T14:51:50.618+0000 7f39a48b7980 1 librados: setting wanted keys 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -203> 2024-03-09T14:51:50.618+0000 7f39a48b7980 1 librados: calling monclient init 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -202> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: init 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -201> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding auth protocol: cephx 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -200> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding auth protocol: cephx 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -199> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding auth protocol: cephx 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -198> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: secure 2024-03-09T14:51:51.639 INFO:tasks.workunit.client.0.smithi001.stderr: -197> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: crc 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -196> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: secure 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -195> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: crc 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -194> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: secure 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -193> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: crc 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -192> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: crc 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -191> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: secure 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -190> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: crc 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -189> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: secure 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -188> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: crc 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -187> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 AuthRegistry(0x56268743d5e0) adding con mode: secure 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -186> 2024-03-09T14:51:50.618+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -185> 2024-03-09T14:51:50.618+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -184> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command rotate-key cmddesc rotate-key hook 0x56268743d728 EEXIST 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -183> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _reopen_session rank -1 2024-03-09T14:51:51.640 INFO:tasks.workunit.client.0.smithi001.stderr: -182> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _add_conns ranks=[1,0,2] 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -181> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-b con 0x562687464af0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -180> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-a con 0x562687465410 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -179> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): picked mon.noname-c con 0x56268746fab0 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -178> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -177> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -176> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): start opening mon connection 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -175> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): _renew_subs 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -174> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient(hunting): authenticate will time out at 2404.366969s 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -173> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): get_auth_request con 0x562687465410 auth_method 0 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -172> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -171> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -170> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -169> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request con 0x56268746fab0 auth_method 0 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -168> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.641 INFO:tasks.workunit.client.0.smithi001.stderr: -167> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -166> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -165> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -164> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -163> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -162> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -161> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -160> 2024-03-09T14:51:50.618+0000 7f39a2bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -159> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): get_auth_request con 0x562687464af0 auth_method 0 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -158> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -157> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -156> 2024-03-09T14:51:50.618+0000 7f39a23cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -155> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient(hunting): handle_auth_done global_id 6825 payload 824 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -154> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient: _finish_hunting 0 2024-03-09T14:51:51.642 INFO:tasks.workunit.client.0.smithi001.stderr: -153> 2024-03-09T14:51:50.618+0000 7f39a1bce640 1 monclient: found mon.noname-a 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr: -152> 2024-03-09T14:51:50.618+0000 7f39a1bce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr: -151> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr: -150> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr: -149> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: dump: 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.643 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr: -148> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: _finish_auth 0 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr: -147> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: _check_auth_tickets 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr: -146> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 monclient: authenticate success, global_id 6825 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr: -145> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr: -144> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr: -143> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr: -142> 2024-03-09T14:51:50.618+0000 7f3873fff640 10 monclient: dump: 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:51:51.644 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -141> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _renew_subs 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -140> 2024-03-09T14:51:50.618+0000 7f3885ffb640 4 set_mon_vals no callback set 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -139> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -138> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _renew_subs 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -137> 2024-03-09T14:51:50.618+0000 7f39a48b7980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -136> 2024-03-09T14:51:50.618+0000 7f39a48b7980 1 librados: init done 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -135> 2024-03-09T14:51:50.618+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command cr dump hook 0x5626874699f8 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -134> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -133> 2024-03-09T14:51:50.622+0000 7f3873fff640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -132> 2024-03-09T14:51:50.622+0000 7f3873fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -131> 2024-03-09T14:51:50.622+0000 7f3873fff640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -130> 2024-03-09T14:51:50.622+0000 7f39a23cf640 10 monclient: get_auth_request con 0x7f386c066070 auth_method 0 2024-03-09T14:51:51.645 INFO:tasks.workunit.client.0.smithi001.stderr: -129> 2024-03-09T14:51:50.622+0000 7f39a2bd0640 10 monclient: get_auth_request con 0x56268748c560 auth_method 0 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -128> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -127> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 realm 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -126> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -125> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -124> 2024-03-09T14:51:50.622+0000 7f39a48b7980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -123> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -122> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -121> 2024-03-09T14:51:50.622+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -120> 2024-03-09T14:51:50.622+0000 7f39a23cf640 10 monclient: get_auth_request con 0x562687490750 auth_method 0 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -119> 2024-03-09T14:51:50.626+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -118> 2024-03-09T14:51:50.626+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -117> 2024-03-09T14:51:50.626+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -116> 2024-03-09T14:51:50.626+0000 7f39a48b7980 20 searching for the correct realm 2024-03-09T14:51:51.646 INFO:tasks.workunit.client.0.smithi001.stderr: -115> 2024-03-09T14:51:50.626+0000 7f39a1bce640 10 monclient: get_auth_request con 0x562687495980 auth_method 0 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -114> 2024-03-09T14:51:50.626+0000 7f39a2bd0640 10 monclient: get_auth_request con 0x7f39980c0020 auth_method 0 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -113> 2024-03-09T14:51:50.630+0000 7f39a23cf640 10 monclient: get_auth_request con 0x7f399c028040 auth_method 0 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -112> 2024-03-09T14:51:50.630+0000 7f39a1bce640 10 monclient: get_auth_request con 0x7f3994079040 auth_method 0 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -111> 2024-03-09T14:51:50.630+0000 7f3873fff640 1 client.6825.objecter ms_handle_reset 0x562687495980 session 0x562687495300 osd.3 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -110> 2024-03-09T14:51:50.630+0000 7f3873fff640 10 monclient: _renew_subs 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -109> 2024-03-09T14:51:50.630+0000 7f3873fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -108> 2024-03-09T14:51:50.630+0000 7f39a1bce640 10 monclient: get_auth_request con 0x7f386c075d40 auth_method 0 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -107> 2024-03-09T14:51:50.630+0000 7f39a1bce640 0 -- 172.21.15.1:0/536104429 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x7f386c075d40 msgr2=0x7f386c078170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -106> 2024-03-09T14:51:50.834+0000 7f39a1bce640 10 monclient: get_auth_request con 0x7f386c075d40 auth_method 0 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -105> 2024-03-09T14:51:51.238+0000 7f39a1bce640 10 monclient: get_auth_request con 0x7f386c075d40 auth_method 0 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -104> 2024-03-09T14:51:51.238+0000 7f39a2bd0640 10 monclient: get_auth_request con 0x7f39980037c0 auth_method 0 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -103> 2024-03-09T14:51:51.246+0000 7f39a23cf640 10 monclient: get_auth_request con 0x7f399810e240 auth_method 0 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -102> 2024-03-09T14:51:51.246+0000 7f39a23cf640 0 -- 172.21.15.1:0/536104429 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f399810e240 msgr2=0x7f399810a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -101> 2024-03-09T14:51:51.450+0000 7f39a23cf640 10 monclient: get_auth_request con 0x7f399810e240 auth_method 0 2024-03-09T14:51:51.647 INFO:tasks.workunit.client.0.smithi001.stderr: -100> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -99> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -98> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -97> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -96> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -95> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -94> 2024-03-09T14:51:51.462+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -93> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -92> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -91> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -90> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -89> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -88> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 zone default found 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -87> 2024-03-09T14:51:51.466+0000 7f39a48b7980 4 Realm: () 2024-03-09T14:51:51.648 INFO:tasks.workunit.client.0.smithi001.stderr: -86> 2024-03-09T14:51:51.466+0000 7f39a48b7980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -85> 2024-03-09T14:51:51.466+0000 7f39a48b7980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -84> 2024-03-09T14:51:51.466+0000 7f39a48b7980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -83> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 zonegroup default 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -82> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -81> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -80> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -79> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -78> 2024-03-09T14:51:51.466+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -77> 2024-03-09T14:51:51.470+0000 7f39a48b7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -76> 2024-03-09T14:51:51.470+0000 7f39a48b7980 20 started sync module instance, tier type = 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -75> 2024-03-09T14:51:51.470+0000 7f39a48b7980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -74> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=0 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -73> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=1 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -72> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=2 2024-03-09T14:51:51.649 INFO:tasks.workunit.client.0.smithi001.stderr: -71> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=3 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -70> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=4 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -69> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=5 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -68> 2024-03-09T14:51:51.490+0000 7f39a48b7980 20 add_watcher() i=6 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -67> 2024-03-09T14:51:51.494+0000 7f39a48b7980 20 add_watcher() i=7 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -66> 2024-03-09T14:51:51.494+0000 7f39a48b7980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -65> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding auth protocol: cephx 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -64> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding auth protocol: cephx 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -63> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding auth protocol: cephx 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -62> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: secure 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -61> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: crc 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -60> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: secure 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -59> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: crc 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -58> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: secure 2024-03-09T14:51:51.650 INFO:tasks.workunit.client.0.smithi001.stderr: -57> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: crc 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -56> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: crc 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -55> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: secure 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -54> 2024-03-09T14:51:51.498+0000 7f38597e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -53> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: crc 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -52> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: secure 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -51> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: crc 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -50> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 AuthRegistry(0x7ffe50baddb0) adding con mode: secure 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -49> 2024-03-09T14:51:51.498+0000 7f39a48b7980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -48> 2024-03-09T14:51:51.498+0000 7f39a48b7980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -47> 2024-03-09T14:51:51.498+0000 7f39a48b7980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -46> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command cache list hook 0x562687472190 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -45> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command cache inspect hook 0x562687472190 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -44> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command cache erase hook 0x562687472190 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -43> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command cache zap hook 0x562687472190 2024-03-09T14:51:51.651 INFO:tasks.workunit.client.0.smithi001.stderr: -42> 2024-03-09T14:51:51.498+0000 7f39a48b7980 5 note: GC not initialized 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -41> 2024-03-09T14:51:51.502+0000 7f38537fe640 20 reqs_thread_entry: start 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -40> 2024-03-09T14:51:51.502+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command sync trace show hook 0x7f3998004470 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -39> 2024-03-09T14:51:51.502+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command sync trace history hook 0x7f3998004470 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -38> 2024-03-09T14:51:51.502+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command sync trace active hook 0x7f3998004470 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -37> 2024-03-09T14:51:51.502+0000 7f39a48b7980 5 asok(0x562687196bb0) register_command sync trace active_short hook 0x7f3998004470 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -36> 2024-03-09T14:51:51.502+0000 7f39a48b7980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -35> 2024-03-09T14:51:51.502+0000 7f39a48b7980 20 Filter name: none 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -34> 2024-03-09T14:51:51.502+0000 7f38517fa640 20 reqs_thread_entry: start 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -33> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 cache get: name=default.rgw.meta+root+abort-at-commit-target-layout : miss 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -32> 2024-03-09T14:51:51.502+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -31> 2024-03-09T14:51:51.502+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -30> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 cache put: name=default.rgw.meta+root+abort-at-commit-target-layout info.flags=0x11 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -29> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 adding default.rgw.meta+root+abort-at-commit-target-layout to cache LRU end 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -28> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 : miss 2024-03-09T14:51:51.652 INFO:tasks.workunit.client.0.smithi001.stderr: -27> 2024-03-09T14:51:51.502+0000 7f39a48b7980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -26> 2024-03-09T14:51:51.502+0000 7f39a48b7980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -25> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -24> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -23> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -22> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -21> 2024-03-09T14:51:51.502+0000 7f39a48b7980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -20> 2024-03-09T14:51:51.506+0000 7f3873fff640 1 client.6825.objecter ms_handle_reset 0x7f3994079040 session 0x7f3994084200 osd.4 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -19> 2024-03-09T14:51:51.506+0000 7f39a1bce640 10 monclient: get_auth_request con 0x562687495980 auth_method 0 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -18> 2024-03-09T14:51:51.514+0000 7f39a48b7980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -17> 2024-03-09T14:51:51.514+0000 7f39a48b7980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -16> 2024-03-09T14:51:51.514+0000 7f39a48b7980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -15> 2024-03-09T14:51:51.514+0000 7f39a48b7980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26, ofs0, ns] 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -14> 2024-03-09T14:51:51.514+0000 7f3885ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562410 cookie 94723478501152 notifier 6825 bl.length()=923 2024-03-09T14:51:51.653 INFO:tasks.workunit.client.0.smithi001.stderr: -13> 2024-03-09T14:51:51.514+0000 7f3885ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: -12> 2024-03-09T14:51:51.514+0000 7f3885ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: -11> 2024-03-09T14:51:51.514+0000 7f3885ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: -10> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): orig_sources= new_sources= 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: -9> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): potential sources added=[] removed=[] 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: -8> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): orig_dests= new_dests= 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: -7> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): potential dests added=[] removed=[] 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: -6> 2024-03-09T14:51:51.518+0000 7f39a48b7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.0 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: -5> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.0 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: -4> 2024-03-09T14:51:51.518+0000 7f39a48b7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.1 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: -3> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.1 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: -2> 2024-03-09T14:51:51.518+0000 7f39a48b7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.0 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: -1> 2024-03-09T14:51:51.526+0000 7f39a48b7980 -1 ./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f39a48b7980 time 2024-03-09T14:51:51.529450+0000 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.654 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f39a818cd7b] 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f39a818cf2f] 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5626864afc51] 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x577) [0x5626864d0557] 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: 5: main() 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f39a6ff7d90] 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: 7: __libc_start_main() 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: 8: _start() 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: 0> 2024-03-09T14:51:51.530+0000 7f39a48b7980 -1 *** Caught signal (Aborted) ** 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f39a48b7980 thread_name:radosgw-admin 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:51:51.655 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f39a7010520] 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f39a818cdd5] 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f39a818cf2f] 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5626864afc51] 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x577) [0x5626864d0557] 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 9: main() 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f39a6ff7d90] 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 11: __libc_start_main() 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 12: _start() 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr:--- logging levels --- 2024-03-09T14:51:51.656 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 none 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 lockdep 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 context 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 crush 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_balancer 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_locker 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log_expire 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_migrator 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 buffer 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 timer 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 filer 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 striper 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 objecter 2024-03-09T14:51:51.657 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rados 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_mirror 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_replay 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_pwl 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 journaler 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objectcacher 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 immutable_obj_cache 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 client 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 osd 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 optracker 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objclass 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 filestore 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 journal 2024-03-09T14:51:51.658 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 ms 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mon 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 0/10 monc 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 paxos 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 tp 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 auth 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 crypto 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 finisher 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 reserver 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 heartbeatmap 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 perfcounter 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 20/20 rgw 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_sync 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_datacache 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_access 2024-03-09T14:51:51.659 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_dbstore 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_flight 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_lifecycle 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 javaclient 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 asok 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 throttle 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 refs 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 compressor 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluestore 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluefs 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 bdev 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 kstore 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 4/ 5 rocksdb 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 fuse 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 2/ 5 mgr 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mgrc 2024-03-09T14:51:51.660 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 dpdk 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 eventtrace 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 prioritycache 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 test 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephfs_mirror 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephsqlite 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 crimson_interrupt 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_onode 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_odata 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_omap 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_tm 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_t 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cleaner 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_epm 2024-03-09T14:51:51.661 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_lba 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_fixedkv_tree 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cache 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_journal 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_device 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_backref 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 alienstore 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mclock 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cyanstore 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 ceph_exporter 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 memstore 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 trace 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: -2/-2 (syslog threshold) 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr: 99/99 (stderr threshold) 2024-03-09T14:51:51.662 INFO:tasks.workunit.client.0.smithi001.stderr:--- pthread ID / name mapping for recent threads --- 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f38517fa640 / http_manager 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f38537fe640 / http_manager 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f38597e2640 / rgw_dt_lg_renew 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f3873fff640 / ms_dispatch 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f3885ffb640 / io_context_pool 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f3887fff640 / ms_dispatch 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f388ee05640 / ms_dispatch 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f3890608640 / io_context_pool 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f39a1bce640 / msgr-worker-2 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f39a23cf640 / msgr-worker-1 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f39a2bd0640 / msgr-worker-0 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f39a33d1640 / io_context_pool 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: 7f39a48b7980 / radosgw-admin 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: max_recent 500 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: max_new 1000 2024-03-09T14:51:51.663 INFO:tasks.workunit.client.0.smithi001.stderr: log_file /var/lib/ceph/crash/2024-03-09T14:51:51.534883Z_ec3b351e-262f-4df4-8aa9-024a27abcbe7/log 2024-03-09T14:51:51.664 INFO:tasks.workunit.client.0.smithi001.stderr:--- end dump of recent events --- 2024-03-09T14:51:51.664 INFO:tasks.workunit.client.0.smithi001.stderr:Aborted 2024-03-09T14:51:51.664 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket abort-at-commit-target-layout 2024-03-09T14:51:51.666 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:51:51.666 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:51:51.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.622+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.622+0000 7f4b02b13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.622+0000 7f4b02b13980 20 realm 2024-03-09T14:51:51.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.622+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.622+0000 7f4b02b13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.622+0000 7f4b02b13980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.622+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.622+0000 7f4b02b13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.622+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.626+0000 7f4b02b13980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.626+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.626+0000 7f4b02b13980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.626+0000 7f4b02b13980 20 searching for the correct realm 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.630+0000 7f4b00e2c640 0 -- 172.21.15.1:0/278637101 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55c280aa3aa0 msgr2=0x55c280aa5f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.638+0000 7f4affe2a640 0 -- 172.21.15.1:0/278637101 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f4af0084bd0 msgr2=0x7f4af0064230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:51:51.667 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 zone default found 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 4 Realm: () 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:51:51.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:51:51.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 zonegroup default 2024-03-09T14:51:51.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.646+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.650+0000 7f4b02b13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.650+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.650+0000 7f4b02b13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.650+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.650+0000 7f4b02b13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.650+0000 7f4b02b13980 20 started sync module instance, tier type = 2024-03-09T14:51:51.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.650+0000 7f4b02b13980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:51:51.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.666+0000 7f4b02b13980 20 add_watcher() i=0 2024-03-09T14:51:51.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.666+0000 7f4b02b13980 20 add_watcher() i=1 2024-03-09T14:51:51.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.666+0000 7f4b02b13980 20 add_watcher() i=2 2024-03-09T14:51:51.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.666+0000 7f4b02b13980 20 add_watcher() i=3 2024-03-09T14:51:51.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.666+0000 7f4b02b13980 20 add_watcher() i=4 2024-03-09T14:51:51.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.666+0000 7f4b02b13980 20 add_watcher() i=5 2024-03-09T14:51:51.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.666+0000 7f4b02b13980 20 add_watcher() i=6 2024-03-09T14:51:51.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.666+0000 7f4b02b13980 20 add_watcher() i=7 2024-03-09T14:51:51.671 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.666+0000 7f4b02b13980 2 all 8 watchers are set, enabling cache 2024-03-09T14:51:51.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f49b77de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:51.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f4b02b13980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:51:51.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f4b02b13980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:51:51.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f4b02b13980 5 note: GC not initialized 2024-03-09T14:51:51.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f49b67dc640 20 reqs_thread_entry: start 2024-03-09T14:51:51.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f4b02b13980 20 init_complete bucket index max shards: 1 2024-03-09T14:51:51.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f4b02b13980 20 Filter name: none 2024-03-09T14:51:51.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f49af7fe640 20 reqs_thread_entry: start 2024-03-09T14:51:51.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f4b02b13980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:51.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f4b02b13980 20 rados_obj.operate() r=0 bl.length=443 2024-03-09T14:51:51.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f4b02b13980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:51.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.670+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.674+0000 7f4b02b13980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:51:51.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.674+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.674+0000 7f4b02b13980 20 rados_obj.operate() r=0 bl.length=443 2024-03-09T14:51:51.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.674+0000 7f4b02b13980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:51:51.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.674+0000 7f4b02b13980 20 rados->read ofs=0 len=0 2024-03-09T14:51:51.677 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.674+0000 7f4b02b13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:51:51.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.674+0000 7f49b77de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:51:51.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.678+0000 7f4b02b13980 20 remove_watcher() i=0 2024-03-09T14:51:51.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.678+0000 7f4b02b13980 2 removed watcher, disabling cache 2024-03-09T14:51:51.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.682+0000 7f4b02b13980 20 remove_watcher() i=1 2024-03-09T14:51:51.688 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.686+0000 7f4b02b13980 20 remove_watcher() i=2 2024-03-09T14:51:51.690 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.686+0000 7f4b02b13980 20 remove_watcher() i=3 2024-03-09T14:51:51.692 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.690+0000 7f4b02b13980 20 remove_watcher() i=4 2024-03-09T14:51:51.693 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.690+0000 7f4b02b13980 20 remove_watcher() i=5 2024-03-09T14:51:51.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.690+0000 7f4b02b13980 20 remove_watcher() i=6 2024-03-09T14:51:51.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:51:51.694+0000 7f4b02b13980 20 remove_watcher() i=7 2024-03-09T14:51:51.711 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.711 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:51:51.711 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: abort-at-commit-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:51:51.712 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:51:51.712 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:51:51.713 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:21.962 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:21.963 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:22.050 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket abort-at-commit-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2024-03-09T14:52:22.086 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:52:22.087 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:52:22.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.130+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.130+0000 7f2353142980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.130+0000 7f2353142980 20 realm 2024-03-09T14:52:22.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.130+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.130+0000 7f2353142980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.136 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.130+0000 7f2353142980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:52:22.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.130+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.130+0000 7f2353142980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.130+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.134+0000 7f2353142980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:22.138 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.134+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.134+0000 7f2353142980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:52:22.139 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.134+0000 7f2353142980 20 searching for the correct realm 2024-03-09T14:52:22.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.142+0000 7f2350459640 0 -- 172.21.15.1:0/1718213886 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x55c829d5f300 msgr2=0x55c829d617c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:22.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.146+0000 7f2350c5a640 0 -- 172.21.15.1:0/1718213886 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f2344001690 msgr2=0x7f234404f590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:52:22.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:52:22.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:52:22.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:52:22.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:52:22.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:52:22.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:52:22.161 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:22.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:52:22.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 zone default found 2024-03-09T14:52:22.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 4 Realm: () 2024-03-09T14:52:22.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:52:22.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:52:22.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:52:22.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 zonegroup default 2024-03-09T14:52:22.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.158+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.162+0000 7f2353142980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.162+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.162+0000 7f2353142980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.162+0000 7f2353142980 20 started sync module instance, tier type = 2024-03-09T14:52:22.164 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.162+0000 7f2353142980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:52:22.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.178+0000 7f2353142980 20 add_watcher() i=0 2024-03-09T14:52:22.180 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.178+0000 7f2353142980 20 add_watcher() i=1 2024-03-09T14:52:22.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.178+0000 7f2353142980 20 add_watcher() i=2 2024-03-09T14:52:22.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.178+0000 7f2353142980 20 add_watcher() i=3 2024-03-09T14:52:22.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.178+0000 7f2353142980 20 add_watcher() i=4 2024-03-09T14:52:22.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.178+0000 7f2353142980 20 add_watcher() i=5 2024-03-09T14:52:22.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.178+0000 7f2353142980 20 add_watcher() i=6 2024-03-09T14:52:22.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.178+0000 7f2353142980 20 add_watcher() i=7 2024-03-09T14:52:22.181 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.178+0000 7f2353142980 2 all 8 watchers are set, enabling cache 2024-03-09T14:52:22.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2206fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:22.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:52:22.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:52:22.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 5 note: GC not initialized 2024-03-09T14:52:22.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2205fe3640 20 reqs_thread_entry: start 2024-03-09T14:52:22.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 20 init_complete bucket index max shards: 1 2024-03-09T14:52:22.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 20 Filter name: none 2024-03-09T14:52:22.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f21777fe640 20 reqs_thread_entry: start 2024-03-09T14:52:22.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 10 cache get: name=default.rgw.meta+root+abort-at-commit-target-layout : miss 2024-03-09T14:52:22.185 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:52:22.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 10 cache put: name=default.rgw.meta+root+abort-at-commit-target-layout info.flags=0x11 2024-03-09T14:52:22.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 10 adding default.rgw.meta+root+abort-at-commit-target-layout to cache LRU end 2024-03-09T14:52:22.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 : miss 2024-03-09T14:52:22.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:52:22.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:52:22.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:52:22.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:52:22.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:22.187 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.182+0000 7f2353142980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 2024-03-09T14:52:22.194 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.190+0000 7f2353142980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:52:22.194 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.190+0000 7f2353142980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:52:22.194 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.190+0000 7f2353142980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:22.194 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.190+0000 7f2353142980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26, ofs0, ns] 2024-03-09T14:52:22.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.190+0000 7f2227fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562412 cookie 94318183728368 notifier 6858 bl.length()=923 2024-03-09T14:52:22.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.190+0000 7f2227fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:52:22.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.190+0000 7f2227fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:52:22.195 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.190+0000 7f2227fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:22.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.194+0000 7f2353142980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): orig_sources= new_sources= 2024-03-09T14:52:22.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.194+0000 7f2353142980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): potential sources added=[] removed=[] 2024-03-09T14:52:22.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.194+0000 7f2353142980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): orig_dests= new_dests= 2024-03-09T14:52:22.197 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.194+0000 7f2353142980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): potential dests added=[] removed=[] 2024-03-09T14:52:22.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.194+0000 7f2353142980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.0 2024-03-09T14:52:22.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.194+0000 7f2353142980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.0 2024-03-09T14:52:22.198 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.194+0000 7f2353142980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.1 2024-03-09T14:52:22.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.194+0000 7f2353142980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.1.1 2024-03-09T14:52:22.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.194+0000 7f2353142980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.26.0 2024-03-09T14:52:22.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.202+0000 7f2353142980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 : hit (requested=0x17, cached=0x17) 2024-03-09T14:52:22.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.202+0000 7f2353142980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 2024-03-09T14:52:22.205 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.202+0000 7f2353142980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:52:22.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.202+0000 7f2353142980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:52:22.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.202+0000 7f2353142980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:22.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.202+0000 7f2353142980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26, ofs0, ns] 2024-03-09T14:52:22.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.202+0000 7f2227fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562413 cookie 94318183728368 notifier 6858 bl.length()=891 2024-03-09T14:52:22.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.202+0000 7f2227fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 info.flags=0x17 2024-03-09T14:52:22.206 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.202+0000 7f2227fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 to cache LRU end 2024-03-09T14:52:22.207 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.202+0000 7f2227fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:22.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.206+0000 7f2353142980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): orig_sources= new_sources= 2024-03-09T14:52:22.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.206+0000 7f2353142980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): potential sources added=[] removed=[] 2024-03-09T14:52:22.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.206+0000 7f2353142980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): orig_dests= new_dests= 2024-03-09T14:52:22.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.206+0000 7f2353142980 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[8c209108-e1cd-4d45-9608-6f491550b24c.4644.26]): potential dests added=[] removed=[] 2024-03-09T14:52:22.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.206+0000 7f2353142980 1 execute INFO: reshard of bucket "abort-at-commit-target-layout" completed successfully 2024-03-09T14:52:22.213 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.210+0000 7f2206fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:22.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.214+0000 7f2353142980 20 remove_watcher() i=0 2024-03-09T14:52:22.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.214+0000 7f2353142980 2 removed watcher, disabling cache 2024-03-09T14:52:22.221 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.218+0000 7f2353142980 20 remove_watcher() i=1 2024-03-09T14:52:22.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.218+0000 7f2353142980 20 remove_watcher() i=2 2024-03-09T14:52:22.225 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.222+0000 7f2353142980 20 remove_watcher() i=3 2024-03-09T14:52:22.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.222+0000 7f2353142980 20 remove_watcher() i=4 2024-03-09T14:52:22.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.226+0000 7f2353142980 20 remove_watcher() i=5 2024-03-09T14:52:22.230 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.226+0000 7f2353142980 20 remove_watcher() i=6 2024-03-09T14:52:22.232 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.230+0000 7f2353142980 20 remove_watcher() i=7 2024-03-09T14:52:22.247 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket abort-at-commit-target-layout 2024-03-09T14:52:22.274 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:52:22.274 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:52:22.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.314+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.314+0000 7f5e1311b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.314+0000 7f5e1311b980 20 realm 2024-03-09T14:52:22.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.314+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.318+0000 7f5e1311b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.318+0000 7f5e1311b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:52:22.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.318+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.318+0000 7f5e1311b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.318+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.318+0000 7f5e1311b980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:22.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.318+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.323 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.318+0000 7f5e1311b980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:52:22.323 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.318+0000 7f5e1311b980 20 searching for the correct realm 2024-03-09T14:52:22.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.326+0000 7f5e10432640 0 -- 172.21.15.1:0/2933956302 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f5e00045020 msgr2=0x7f5e00003610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:22.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.334+0000 7f5e10432640 0 -- 172.21.15.1:0/2933956302 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f5cdc075ec0 msgr2=0x7f5cdc0782f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:22.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:52:22.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:52:22.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:52:22.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:52:22.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:52:22.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:52:22.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:52:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 zone default found 2024-03-09T14:52:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 4 Realm: () 2024-03-09T14:52:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:52:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:52:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:52:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 zonegroup default 2024-03-09T14:52:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.346 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 started sync module instance, tier type = 2024-03-09T14:52:22.347 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.342+0000 7f5e1311b980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:52:22.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.358+0000 7f5e1311b980 20 add_watcher() i=0 2024-03-09T14:52:22.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.358+0000 7f5e1311b980 20 add_watcher() i=1 2024-03-09T14:52:22.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.358+0000 7f5e1311b980 20 add_watcher() i=2 2024-03-09T14:52:22.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.358+0000 7f5e1311b980 20 add_watcher() i=3 2024-03-09T14:52:22.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.358+0000 7f5e1311b980 20 add_watcher() i=4 2024-03-09T14:52:22.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.358+0000 7f5e1311b980 20 add_watcher() i=5 2024-03-09T14:52:22.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.358+0000 7f5e1311b980 20 add_watcher() i=6 2024-03-09T14:52:22.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.362+0000 7f5e1311b980 20 add_watcher() i=7 2024-03-09T14:52:22.364 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.362+0000 7f5e1311b980 2 all 8 watchers are set, enabling cache 2024-03-09T14:52:22.367 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.362+0000 7f5cc6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:22.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.362+0000 7f5e1311b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:52:22.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.362+0000 7f5e1311b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:52:22.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.362+0000 7f5e1311b980 5 note: GC not initialized 2024-03-09T14:52:22.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.362+0000 7f5cc5fe3640 20 reqs_thread_entry: start 2024-03-09T14:52:22.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.362+0000 7f5e1311b980 20 init_complete bucket index max shards: 1 2024-03-09T14:52:22.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 Filter name: none 2024-03-09T14:52:22.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5c377fe640 20 reqs_thread_entry: start 2024-03-09T14:52:22.368 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:52:22.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:52:22.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:52:22.369 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.370 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 rados_obj.operate() r=0 bl.length=197 2024-03-09T14:52:22.370 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.370 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 rados_obj.operate() r=0 bl.length=411 2024-03-09T14:52:22.370 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:52:22.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:22.371 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.366+0000 7f5e1311b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:22.372 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.370+0000 7f5cc6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:22.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.374+0000 7f5e1311b980 20 remove_watcher() i=0 2024-03-09T14:52:22.378 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.374+0000 7f5e1311b980 2 removed watcher, disabling cache 2024-03-09T14:52:22.379 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.374+0000 7f5e1311b980 20 remove_watcher() i=1 2024-03-09T14:52:22.381 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.378+0000 7f5e1311b980 20 remove_watcher() i=2 2024-03-09T14:52:22.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.378+0000 7f5e1311b980 20 remove_watcher() i=3 2024-03-09T14:52:22.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.382+0000 7f5e1311b980 20 remove_watcher() i=4 2024-03-09T14:52:22.387 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.382+0000 7f5e1311b980 20 remove_watcher() i=5 2024-03-09T14:52:22.388 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.386+0000 7f5e1311b980 20 remove_watcher() i=6 2024-03-09T14:52:22.389 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:22.386+0000 7f5e1311b980 20 remove_watcher() i=7 2024-03-09T14:52:22.402 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:22.402 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:52:22.402 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: abort-at-commit-target-layout id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.26 num_objects: 19 size_kb: 1 num_shards: 2 2024-03-09T14:52:22.403 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:22.404 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:22.404 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:22.429 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:22.429 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:22.998 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:22.998 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:23.180 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard bucket with EIO injected at do_reshard 2024-03-09T14:52:23.181 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:23.184 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:23.184 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:23.304 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:23.304 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:23.392 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:23.392 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:23.497 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:23.497 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:23.594 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:23.594 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:23.694 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:23.694 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:23.790 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:23.790 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:23.892 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:23.892 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:23.988 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:23.988 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:24.087 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:24.087 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:24.185 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:24.185 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:24.288 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:24.288 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:24.388 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:24.388 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:24.489 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:24.490 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:24.592 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:24.592 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:24.693 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:24.693 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:24.796 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:24.796 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:24.893 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:24.894 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:24.994 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:24.994 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:25.095 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:25.096 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:25.193 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:25.193 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:25.286 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:25.286 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:25.383 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-do-reshard 2024-03-09T14:52:25.416 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:52:25.416 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:52:25.461 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.458+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.458+0000 7f0e8e306980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:25.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.458+0000 7f0e8e306980 20 realm 2024-03-09T14:52:25.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.458+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.458+0000 7f0e8e306980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:25.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.458+0000 7f0e8e306980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:52:25.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.458+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.458+0000 7f0e8e306980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:25.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.458+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.462+0000 7f0e8e306980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:25.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.462+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.462+0000 7f0e8e306980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:52:25.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.462+0000 7f0e8e306980 20 searching for the correct realm 2024-03-09T14:52:25.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.466+0000 7f0e8b61d640 0 -- 172.21.15.1:0/1050267795 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f0e7c06ab30 msgr2=0x7f0e7c055e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:25.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.682+0000 7f0e8be1e640 0 -- 172.21.15.1:0/1050267795 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f0e7404ae50 msgr2=0x7f0e74088220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:25.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.898+0000 7f0e8e306980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:52:25.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.898+0000 7f0e8e306980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:52:25.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.898+0000 7f0e8e306980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:52:25.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.898+0000 7f0e8e306980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:52:25.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.898+0000 7f0e8e306980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:52:25.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.898+0000 7f0e8e306980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:52:25.904 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.898+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.904 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.898+0000 7f0e8e306980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:25.904 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.898+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.904 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:25.904 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.905 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:52:25.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 20 zone default found 2024-03-09T14:52:25.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 4 Realm: () 2024-03-09T14:52:25.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:52:25.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:52:25.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:52:25.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 20 zonegroup default 2024-03-09T14:52:25.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.906 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:25.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:25.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.902+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.906+0000 7f0e8e306980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:25.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.906+0000 7f0e8e306980 20 started sync module instance, tier type = 2024-03-09T14:52:25.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.906+0000 7f0e8e306980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:52:25.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.930+0000 7f0e8e306980 20 add_watcher() i=0 2024-03-09T14:52:25.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.930+0000 7f0e8e306980 20 add_watcher() i=1 2024-03-09T14:52:25.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.930+0000 7f0e8e306980 20 add_watcher() i=2 2024-03-09T14:52:25.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.930+0000 7f0e8e306980 20 add_watcher() i=3 2024-03-09T14:52:25.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.930+0000 7f0e8e306980 20 add_watcher() i=4 2024-03-09T14:52:25.934 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.930+0000 7f0e8e306980 20 add_watcher() i=5 2024-03-09T14:52:25.934 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.930+0000 7f0e8e306980 20 add_watcher() i=6 2024-03-09T14:52:25.935 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.930+0000 7f0e8e306980 20 add_watcher() i=7 2024-03-09T14:52:25.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.930+0000 7f0e8e306980 2 all 8 watchers are set, enabling cache 2024-03-09T14:52:25.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.938+0000 7f0d427e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:25.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.938+0000 7f0e8e306980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:52:25.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.938+0000 7f0e8e306980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:52:25.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.938+0000 7f0e8e306980 5 note: GC not initialized 2024-03-09T14:52:25.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.938+0000 7f0d417e2640 20 reqs_thread_entry: start 2024-03-09T14:52:25.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.938+0000 7f0e8e306980 20 init_complete bucket index max shards: 1 2024-03-09T14:52:25.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.938+0000 7f0e8e306980 20 Filter name: none 2024-03-09T14:52:25.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.938+0000 7f0d3a7fc640 20 reqs_thread_entry: start 2024-03-09T14:52:25.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.938+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.938+0000 7f0e8e306980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:25.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.938+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.942+0000 7f0e8e306980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:25.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.942+0000 7f0e8e306980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:52:25.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.942+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.944 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.942+0000 7f0e8e306980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:25.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.942+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.942+0000 7f0e8e306980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:25.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.942+0000 7f0e8e306980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:52:25.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.942+0000 7f0e8e306980 20 rados->read ofs=0 len=0 2024-03-09T14:52:25.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.942+0000 7f0e8e306980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:25.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.946+0000 7f0d427e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:25.954 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.950+0000 7f0e8e306980 20 remove_watcher() i=0 2024-03-09T14:52:25.954 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.950+0000 7f0e8e306980 2 removed watcher, disabling cache 2024-03-09T14:52:25.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.954+0000 7f0e8e306980 20 remove_watcher() i=1 2024-03-09T14:52:25.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.954+0000 7f0e8e306980 20 remove_watcher() i=2 2024-03-09T14:52:25.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.958+0000 7f0e8e306980 20 remove_watcher() i=3 2024-03-09T14:52:25.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.962+0000 7f0e8e306980 20 remove_watcher() i=4 2024-03-09T14:52:25.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.962+0000 7f0e8e306980 20 remove_watcher() i=5 2024-03-09T14:52:25.967 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.962+0000 7f0e8e306980 20 remove_watcher() i=6 2024-03-09T14:52:25.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:25.966+0000 7f0e8e306980 20 remove_watcher() i=7 2024-03-09T14:52:25.984 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:25.984 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:52:25.984 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-do-reshard id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:52:25.984 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:25.985 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket error-at-do-reshard --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-error-at do_reshard 2024-03-09T14:52:26.016 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:52:26.016 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:52:26.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.058+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.064 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.058+0000 7fcc2ee9b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.064 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.058+0000 7fcc2ee9b980 20 realm 2024-03-09T14:52:26.064 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.058+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.064 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.058+0000 7fcc2ee9b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.064 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.058+0000 7fcc2ee9b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:52:26.064 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.058+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.065 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.058+0000 7fcc2ee9b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.065 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.058+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.062+0000 7fcc2ee9b980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:26.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.062+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.062+0000 7fcc2ee9b980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:52:26.066 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.062+0000 7fcc2ee9b980 20 searching for the correct realm 2024-03-09T14:52:26.072 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.070+0000 7fcc2c1b2640 0 -- 172.21.15.1:0/704560623 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fcc1c034060 msgr2=0x7fcc1c05e930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:52:26.080 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.078+0000 7fcc2c1b2640 0 -- 172.21.15.1:0/704560623 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fcaf8075ec0 msgr2=0x7fcaf80782f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:26.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.082+0000 7fcc2ee9b980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:52:26.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.082+0000 7fcc2ee9b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:52:26.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.082+0000 7fcc2ee9b980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:52:26.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.082+0000 7fcc2ee9b980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:52:26.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.082+0000 7fcc2ee9b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:52:26.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.082+0000 7fcc2ee9b980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:52:26.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.082+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.088 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:26.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:52:26.089 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 zone default found 2024-03-09T14:52:26.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 4 Realm: () 2024-03-09T14:52:26.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:52:26.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:52:26.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:52:26.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 zonegroup default 2024-03-09T14:52:26.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.090 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 started sync module instance, tier type = 2024-03-09T14:52:26.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.086+0000 7fcc2ee9b980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:52:26.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.102+0000 7fcc2ee9b980 20 add_watcher() i=0 2024-03-09T14:52:26.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.102+0000 7fcc2ee9b980 20 add_watcher() i=1 2024-03-09T14:52:26.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.102+0000 7fcc2ee9b980 20 add_watcher() i=2 2024-03-09T14:52:26.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.102+0000 7fcc2ee9b980 20 add_watcher() i=3 2024-03-09T14:52:26.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.102+0000 7fcc2ee9b980 20 add_watcher() i=4 2024-03-09T14:52:26.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.102+0000 7fcc2ee9b980 20 add_watcher() i=5 2024-03-09T14:52:26.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.102+0000 7fcc2ee9b980 20 add_watcher() i=6 2024-03-09T14:52:26.108 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.106+0000 7fcc2ee9b980 20 add_watcher() i=7 2024-03-09T14:52:26.108 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.106+0000 7fcc2ee9b980 2 all 8 watchers are set, enabling cache 2024-03-09T14:52:26.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.106+0000 7fcae2fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:26.111 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.106+0000 7fcc2ee9b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:52:26.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.106+0000 7fcc2ee9b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:52:26.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.106+0000 7fcc2ee9b980 5 note: GC not initialized 2024-03-09T14:52:26.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcae1fe3640 20 reqs_thread_entry: start 2024-03-09T14:52:26.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 20 init_complete bucket index max shards: 1 2024-03-09T14:52:26.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 20 Filter name: none 2024-03-09T14:52:26.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fca537fe640 20 reqs_thread_entry: start 2024-03-09T14:52:26.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 10 cache get: name=default.rgw.meta+root+error-at-do-reshard : miss 2024-03-09T14:52:26.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:26.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 10 cache put: name=default.rgw.meta+root+error-at-do-reshard info.flags=0x11 2024-03-09T14:52:26.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 10 adding default.rgw.meta+root+error-at-do-reshard to cache LRU end 2024-03-09T14:52:26.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 : miss 2024-03-09T14:52:26.113 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:26.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 info.flags=0x17 2024-03-09T14:52:26.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 10 adding default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 to cache LRU end 2024-03-09T14:52:26.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:52:26.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:26.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.110+0000 7fcc2ee9b980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 2024-03-09T14:52:26.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.118+0000 7fcc2ee9b980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 info.flags=0x17 2024-03-09T14:52:26.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.118+0000 7fcc2ee9b980 10 moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 to cache LRU end 2024-03-09T14:52:26.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.118+0000 7fcc2ee9b980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:26.120 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.118+0000 7fcc2ee9b980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27, ofs0, ns] 2024-03-09T14:52:26.121 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.118+0000 7fcb03fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562451 cookie 94193500254592 notifier 6909 bl.length()=903 2024-03-09T14:52:26.121 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.118+0000 7fcb03fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 info.flags=0x17 2024-03-09T14:52:26.121 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.118+0000 7fcb03fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 to cache LRU end 2024-03-09T14:52:26.121 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.118+0000 7fcb03fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:26.123 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.118+0000 7fcc2ee9b980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): orig_sources= new_sources= 2024-03-09T14:52:26.124 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.118+0000 7fcc2ee9b980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): potential sources added=[] removed=[] 2024-03-09T14:52:26.124 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.118+0000 7fcc2ee9b980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): orig_dests= new_dests= 2024-03-09T14:52:26.124 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.118+0000 7fcc2ee9b980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): potential dests added=[] removed=[] 2024-03-09T14:52:26.127 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.122+0000 7fcc2ee9b980 -1 Injecting error=-5 at location=do_reshard 2024-03-09T14:52:26.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.130+0000 7fcc2ee9b980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 : hit (requested=0x17, cached=0x17) 2024-03-09T14:52:26.132 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.130+0000 7fcc2ee9b980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 2024-03-09T14:52:26.133 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.130+0000 7fcc2ee9b980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 info.flags=0x17 2024-03-09T14:52:26.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.130+0000 7fcc2ee9b980 10 moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 to cache LRU end 2024-03-09T14:52:26.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.130+0000 7fcc2ee9b980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:26.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.130+0000 7fcc2ee9b980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27, ofs0, ns] 2024-03-09T14:52:26.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.130+0000 7fcb037fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562452 cookie 94193500254592 notifier 6909 bl.length()=871 2024-03-09T14:52:26.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.130+0000 7fcb037fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 info.flags=0x17 2024-03-09T14:52:26.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.130+0000 7fcb037fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 to cache LRU end 2024-03-09T14:52:26.134 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.130+0000 7fcb037fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:26.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.134+0000 7fcc2ee9b980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): orig_sources= new_sources= 2024-03-09T14:52:26.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.134+0000 7fcc2ee9b980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): potential sources added=[] removed=[] 2024-03-09T14:52:26.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.134+0000 7fcc2ee9b980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): orig_dests= new_dests= 2024-03-09T14:52:26.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.134+0000 7fcc2ee9b980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): potential dests added=[] removed=[] 2024-03-09T14:52:26.137 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.134+0000 7fcc2ee9b980 1 execute INFO: reshard of bucket "error-at-do-reshard" canceled due to errors 2024-03-09T14:52:26.140 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.138+0000 7fcae2fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:26.145 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.142+0000 7fcc2ee9b980 20 remove_watcher() i=0 2024-03-09T14:52:26.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.142+0000 7fcc2ee9b980 2 removed watcher, disabling cache 2024-03-09T14:52:26.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.146+0000 7fcc2ee9b980 20 remove_watcher() i=1 2024-03-09T14:52:26.153 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.150+0000 7fcc2ee9b980 20 remove_watcher() i=2 2024-03-09T14:52:26.155 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.150+0000 7fcc2ee9b980 20 remove_watcher() i=3 2024-03-09T14:52:26.157 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.154+0000 7fcc2ee9b980 20 remove_watcher() i=4 2024-03-09T14:52:26.160 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.158+0000 7fcc2ee9b980 20 remove_watcher() i=5 2024-03-09T14:52:26.162 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.158+0000 7fcc2ee9b980 20 remove_watcher() i=6 2024-03-09T14:52:26.163 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.158+0000 7fcc2ee9b980 20 remove_watcher() i=7 2024-03-09T14:52:26.178 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-do-reshard 2024-03-09T14:52:26.204 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:52:26.205 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:52:26.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.246+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.246+0000 7fd974f13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.246+0000 7fd974f13980 20 realm 2024-03-09T14:52:26.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.246+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.246+0000 7fd974f13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.246+0000 7fd974f13980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:52:26.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.246+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.250 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.246+0000 7fd974f13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.246+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.250+0000 7fd974f13980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:26.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.250+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.250+0000 7fd974f13980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:52:26.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.250+0000 7fd974f13980 20 searching for the correct realm 2024-03-09T14:52:26.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.254+0000 7fd97222a640 0 -- 172.21.15.1:0/3159868857 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x556374520340 msgr2=0x556374522730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:26.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.262+0000 7fd972a2b640 0 -- 172.21.15.1:0/3159868857 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fd95c019400 msgr2=0x7fd95c019860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:26.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:52:26.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:52:26.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:52:26.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:52:26.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:52:26.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:52:26.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.274 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:26.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:52:26.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 zone default found 2024-03-09T14:52:26.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 4 Realm: () 2024-03-09T14:52:26.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:52:26.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:52:26.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:52:26.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 zonegroup default 2024-03-09T14:52:26.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.270+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.274+0000 7fd974f13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.274+0000 7fd974f13980 20 started sync module instance, tier type = 2024-03-09T14:52:26.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.274+0000 7fd974f13980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:52:26.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.290+0000 7fd974f13980 20 add_watcher() i=0 2024-03-09T14:52:26.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.290+0000 7fd974f13980 20 add_watcher() i=1 2024-03-09T14:52:26.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.290+0000 7fd974f13980 20 add_watcher() i=2 2024-03-09T14:52:26.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.290+0000 7fd974f13980 20 add_watcher() i=3 2024-03-09T14:52:26.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.290+0000 7fd974f13980 20 add_watcher() i=4 2024-03-09T14:52:26.295 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.290+0000 7fd974f13980 20 add_watcher() i=5 2024-03-09T14:52:26.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.290+0000 7fd974f13980 20 add_watcher() i=6 2024-03-09T14:52:26.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.294+0000 7fd974f13980 20 add_watcher() i=7 2024-03-09T14:52:26.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.294+0000 7fd974f13980 2 all 8 watchers are set, enabling cache 2024-03-09T14:52:26.299 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.294+0000 7fd8297e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:26.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.294+0000 7fd974f13980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:52:26.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.294+0000 7fd974f13980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:52:26.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.294+0000 7fd974f13980 5 note: GC not initialized 2024-03-09T14:52:26.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.294+0000 7fd7a3fff640 20 reqs_thread_entry: start 2024-03-09T14:52:26.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.294+0000 7fd974f13980 20 init_complete bucket index max shards: 1 2024-03-09T14:52:26.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.294+0000 7fd974f13980 20 Filter name: none 2024-03-09T14:52:26.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.294+0000 7fd7a1ffb640 20 reqs_thread_entry: start 2024-03-09T14:52:26.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.298+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.298+0000 7fd974f13980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:26.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.298+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.298+0000 7fd974f13980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:26.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.298+0000 7fd974f13980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:52:26.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.298+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.298+0000 7fd974f13980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:26.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.298+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.298+0000 7fd974f13980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:26.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.298+0000 7fd974f13980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:52:26.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.298+0000 7fd974f13980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.298+0000 7fd974f13980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.302+0000 7fd8297e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:26.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.306+0000 7fd974f13980 20 remove_watcher() i=0 2024-03-09T14:52:26.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.306+0000 7fd974f13980 2 removed watcher, disabling cache 2024-03-09T14:52:26.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.306+0000 7fd974f13980 20 remove_watcher() i=1 2024-03-09T14:52:26.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.310+0000 7fd974f13980 20 remove_watcher() i=2 2024-03-09T14:52:26.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.310+0000 7fd974f13980 20 remove_watcher() i=3 2024-03-09T14:52:26.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.314+0000 7fd974f13980 20 remove_watcher() i=4 2024-03-09T14:52:26.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.314+0000 7fd974f13980 20 remove_watcher() i=5 2024-03-09T14:52:26.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.318+0000 7fd974f13980 20 remove_watcher() i=6 2024-03-09T14:52:26.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.318+0000 7fd974f13980 20 remove_watcher() i=7 2024-03-09T14:52:26.335 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:26.335 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:52:26.335 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-do-reshard id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:52:26.335 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:26.336 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:26.336 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:26.378 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:26.378 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:26.438 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket error-at-do-reshard --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2024-03-09T14:52:26.474 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:52:26.474 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:52:26.517 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.514+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.518 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.514+0000 7f06ca2a2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.514+0000 7f06ca2a2980 20 realm 2024-03-09T14:52:26.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.514+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.514+0000 7f06ca2a2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.514+0000 7f06ca2a2980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:52:26.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.514+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.514+0000 7f06ca2a2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.519 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.514+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.518+0000 7f06ca2a2980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:26.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.518+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.518+0000 7f06ca2a2980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:52:26.521 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.518+0000 7f06ca2a2980 20 searching for the correct realm 2024-03-09T14:52:26.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.522+0000 7f06c75b9640 0 -- 172.21.15.1:0/2296558790 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x55e2018b3b10 msgr2=0x55e2018b5f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:26.534 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.530+0000 7f06c7dba640 0 -- 172.21.15.1:0/2296558790 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f06b0074560 msgr2=0x7f06b0062f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:26.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.538+0000 7f06ca2a2980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:52:26.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.538+0000 7f06ca2a2980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:52:26.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.538+0000 7f06ca2a2980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:52:26.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.538+0000 7f06ca2a2980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:52:26.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.538+0000 7f06ca2a2980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:52:26.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.538+0000 7f06ca2a2980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:52:26.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.538+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.538+0000 7f06ca2a2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.538+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:26.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:52:26.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 zone default found 2024-03-09T14:52:26.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 4 Realm: () 2024-03-09T14:52:26.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:52:26.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:52:26.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:52:26.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 zonegroup default 2024-03-09T14:52:26.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 started sync module instance, tier type = 2024-03-09T14:52:26.546 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.542+0000 7f06ca2a2980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:52:26.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.558+0000 7f06ca2a2980 20 add_watcher() i=0 2024-03-09T14:52:26.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.558+0000 7f06ca2a2980 20 add_watcher() i=1 2024-03-09T14:52:26.562 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.558+0000 7f06ca2a2980 20 add_watcher() i=2 2024-03-09T14:52:26.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.558+0000 7f06ca2a2980 20 add_watcher() i=3 2024-03-09T14:52:26.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.558+0000 7f06ca2a2980 20 add_watcher() i=4 2024-03-09T14:52:26.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.558+0000 7f06ca2a2980 20 add_watcher() i=5 2024-03-09T14:52:26.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.558+0000 7f06ca2a2980 20 add_watcher() i=6 2024-03-09T14:52:26.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.558+0000 7f06ca2a2980 20 add_watcher() i=7 2024-03-09T14:52:26.563 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.558+0000 7f06ca2a2980 2 all 8 watchers are set, enabling cache 2024-03-09T14:52:26.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f057d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:26.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f06ca2a2980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:52:26.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f06ca2a2980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:52:26.566 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f06ca2a2980 5 note: GC not initialized 2024-03-09T14:52:26.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f04f7fff640 20 reqs_thread_entry: start 2024-03-09T14:52:26.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f06ca2a2980 20 init_complete bucket index max shards: 1 2024-03-09T14:52:26.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f06ca2a2980 20 Filter name: none 2024-03-09T14:52:26.567 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f04f5ffb640 20 reqs_thread_entry: start 2024-03-09T14:52:26.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f06ca2a2980 10 cache get: name=default.rgw.meta+root+error-at-do-reshard : miss 2024-03-09T14:52:26.568 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f06ca2a2980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:26.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f06ca2a2980 10 cache put: name=default.rgw.meta+root+error-at-do-reshard info.flags=0x11 2024-03-09T14:52:26.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f06ca2a2980 10 adding default.rgw.meta+root+error-at-do-reshard to cache LRU end 2024-03-09T14:52:26.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f06ca2a2980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 : miss 2024-03-09T14:52:26.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.562+0000 7f06ca2a2980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.566+0000 7f06ca2a2980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:26.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.566+0000 7f06ca2a2980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 info.flags=0x17 2024-03-09T14:52:26.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.566+0000 7f06ca2a2980 10 adding default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 to cache LRU end 2024-03-09T14:52:26.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.566+0000 7f06ca2a2980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:52:26.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.566+0000 7f06ca2a2980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:26.569 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.566+0000 7f06ca2a2980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 2024-03-09T14:52:26.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.570+0000 7f06ca2a2980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 info.flags=0x17 2024-03-09T14:52:26.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.570+0000 7f06ca2a2980 10 moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 to cache LRU end 2024-03-09T14:52:26.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.570+0000 7f06ca2a2980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:26.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.570+0000 7f06ca2a2980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27, ofs0, ns] 2024-03-09T14:52:26.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.570+0000 7f05aa7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562453 cookie 94429176889872 notifier 6518 bl.length()=903 2024-03-09T14:52:26.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.570+0000 7f05aa7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 info.flags=0x17 2024-03-09T14:52:26.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.570+0000 7f05aa7fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 to cache LRU end 2024-03-09T14:52:26.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.570+0000 7f05aa7fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:26.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.574+0000 7f06ca2a2980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): orig_sources= new_sources= 2024-03-09T14:52:26.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.574+0000 7f06ca2a2980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): potential sources added=[] removed=[] 2024-03-09T14:52:26.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.574+0000 7f06ca2a2980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): orig_dests= new_dests= 2024-03-09T14:52:26.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.574+0000 7f06ca2a2980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): potential dests added=[] removed=[] 2024-03-09T14:52:26.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.578+0000 7f06ca2a2980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.27.1.0 2024-03-09T14:52:26.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.578+0000 7f06ca2a2980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.27.1.0 2024-03-09T14:52:26.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.578+0000 7f06ca2a2980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.27.1.1 2024-03-09T14:52:26.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.578+0000 7f06ca2a2980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.27.1.1 2024-03-09T14:52:26.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.578+0000 7f06ca2a2980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.27.0 2024-03-09T14:52:26.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.582+0000 7f06ca2a2980 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 : hit (requested=0x17, cached=0x17) 2024-03-09T14:52:26.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.582+0000 7f06ca2a2980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 2024-03-09T14:52:26.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.586+0000 7f06ca2a2980 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 info.flags=0x17 2024-03-09T14:52:26.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.586+0000 7f06ca2a2980 10 moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 to cache LRU end 2024-03-09T14:52:26.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.586+0000 7f06ca2a2980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:26.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.586+0000 7f06ca2a2980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27, ofs0, ns] 2024-03-09T14:52:26.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.586+0000 7f05a9ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562454 cookie 94429176889872 notifier 6518 bl.length()=871 2024-03-09T14:52:26.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.586+0000 7f05a9ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 info.flags=0x17 2024-03-09T14:52:26.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.586+0000 7f05a9ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 to cache LRU end 2024-03-09T14:52:26.590 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.586+0000 7f05a9ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:26.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.590+0000 7f06ca2a2980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): orig_sources= new_sources= 2024-03-09T14:52:26.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.590+0000 7f06ca2a2980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): potential sources added=[] removed=[] 2024-03-09T14:52:26.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.590+0000 7f06ca2a2980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): orig_dests= new_dests= 2024-03-09T14:52:26.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.590+0000 7f06ca2a2980 20 handle_bi_update(): bucket=:error-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.27]): potential dests added=[] removed=[] 2024-03-09T14:52:26.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.590+0000 7f06ca2a2980 1 execute INFO: reshard of bucket "error-at-do-reshard" completed successfully 2024-03-09T14:52:26.599 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.594+0000 7f057d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:26.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.602+0000 7f06ca2a2980 20 remove_watcher() i=0 2024-03-09T14:52:26.604 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.602+0000 7f06ca2a2980 2 removed watcher, disabling cache 2024-03-09T14:52:26.606 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.602+0000 7f06ca2a2980 20 remove_watcher() i=1 2024-03-09T14:52:26.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.606+0000 7f06ca2a2980 20 remove_watcher() i=2 2024-03-09T14:52:26.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.606+0000 7f06ca2a2980 20 remove_watcher() i=3 2024-03-09T14:52:26.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.610+0000 7f06ca2a2980 20 remove_watcher() i=4 2024-03-09T14:52:26.614 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.610+0000 7f06ca2a2980 20 remove_watcher() i=5 2024-03-09T14:52:26.616 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.610+0000 7f06ca2a2980 20 remove_watcher() i=6 2024-03-09T14:52:26.617 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.614+0000 7f06ca2a2980 20 remove_watcher() i=7 2024-03-09T14:52:26.633 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket error-at-do-reshard 2024-03-09T14:52:26.662 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:52:26.662 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:52:26.705 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.702+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.707 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.702+0000 7f4dc449d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.707 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.702+0000 7f4dc449d980 20 realm 2024-03-09T14:52:26.707 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.702+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.702+0000 7f4dc449d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.702+0000 7f4dc449d980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:52:26.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.702+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.702+0000 7f4dc449d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.702+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.706+0000 7f4dc449d980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:26.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.706+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.706+0000 7f4dc449d980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:52:26.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.706+0000 7f4dc449d980 20 searching for the correct realm 2024-03-09T14:52:26.714 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.710+0000 7f4dc1fb5640 0 -- 172.21.15.1:0/1989733713 >> [v2:172.21.15.1:6824/1666056338,v1:172.21.15.1:6825/1666056338] conn(0x7f4dbc01f040 msgr2=0x7f4dbc02d970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:26.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.718+0000 7f4dc27b6640 0 -- 172.21.15.1:0/1989733713 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f4db8109040 msgr2=0x7f4db810e8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:26.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.726+0000 7f4dc449d980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:52:26.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.726+0000 7f4dc449d980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:52:26.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.726+0000 7f4dc449d980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:52:26.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.726+0000 7f4dc449d980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:52:26.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.726+0000 7f4dc449d980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:52:26.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.726+0000 7f4dc449d980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:52:26.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.726+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.726+0000 7f4dc449d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.731 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.726+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.726+0000 7f4dc449d980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:26.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.726+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:52:26.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 20 zone default found 2024-03-09T14:52:26.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 4 Realm: () 2024-03-09T14:52:26.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:52:26.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:52:26.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:52:26.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 20 zonegroup default 2024-03-09T14:52:26.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.733 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 20 started sync module instance, tier type = 2024-03-09T14:52:26.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.730+0000 7f4dc449d980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:52:26.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.746+0000 7f4dc449d980 20 add_watcher() i=0 2024-03-09T14:52:26.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.746+0000 7f4dc449d980 20 add_watcher() i=1 2024-03-09T14:52:26.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.746+0000 7f4dc449d980 20 add_watcher() i=2 2024-03-09T14:52:26.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.746+0000 7f4dc449d980 20 add_watcher() i=3 2024-03-09T14:52:26.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.746+0000 7f4dc449d980 20 add_watcher() i=4 2024-03-09T14:52:26.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.746+0000 7f4dc449d980 20 add_watcher() i=5 2024-03-09T14:52:26.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.746+0000 7f4dc449d980 20 add_watcher() i=6 2024-03-09T14:52:26.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.746+0000 7f4dc449d980 20 add_watcher() i=7 2024-03-09T14:52:26.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.746+0000 7f4dc449d980 2 all 8 watchers are set, enabling cache 2024-03-09T14:52:26.752 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4c77fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:26.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:52:26.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:52:26.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 5 note: GC not initialized 2024-03-09T14:52:26.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4c76fe5640 20 reqs_thread_entry: start 2024-03-09T14:52:26.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 init_complete bucket index max shards: 1 2024-03-09T14:52:26.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 Filter name: none 2024-03-09T14:52:26.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4c74fe1640 20 reqs_thread_entry: start 2024-03-09T14:52:26.753 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:26.754 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:26.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:52:26.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:26.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:26.755 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.750+0000 7f4dc449d980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:52:26.756 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.754+0000 7f4dc449d980 20 rados->read ofs=0 len=0 2024-03-09T14:52:26.757 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.754+0000 7f4dc449d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:26.758 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.754+0000 7f4c77fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:26.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.758+0000 7f4dc449d980 20 remove_watcher() i=0 2024-03-09T14:52:26.762 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.758+0000 7f4dc449d980 2 removed watcher, disabling cache 2024-03-09T14:52:26.764 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.762+0000 7f4dc449d980 20 remove_watcher() i=1 2024-03-09T14:52:26.766 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.762+0000 7f4dc449d980 20 remove_watcher() i=2 2024-03-09T14:52:26.768 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.766+0000 7f4dc449d980 20 remove_watcher() i=3 2024-03-09T14:52:26.770 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.766+0000 7f4dc449d980 20 remove_watcher() i=4 2024-03-09T14:52:26.772 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.770+0000 7f4dc449d980 20 remove_watcher() i=5 2024-03-09T14:52:26.773 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.770+0000 7f4dc449d980 20 remove_watcher() i=6 2024-03-09T14:52:26.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:26.770+0000 7f4dc449d980 20 remove_watcher() i=7 2024-03-09T14:52:26.788 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:26.789 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:52:26.789 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: error-at-do-reshard id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.27 num_objects: 19 size_kb: 1 num_shards: 2 2024-03-09T14:52:26.789 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:26.790 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:26.790 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:26.813 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:26.813 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:27.347 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:27.347 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:27.527 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: reshard bucket with abort at do_reshard 2024-03-09T14:52:27.528 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:27.533 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:27.533 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:27.658 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:27.658 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:27.748 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:27.749 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:27.852 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:27.852 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:27.952 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:27.952 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:28.050 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:28.050 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:28.152 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:28.152 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:28.254 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:28.254 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:28.350 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:28.351 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:28.448 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:28.448 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:28.547 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:28.547 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:28.646 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:28.646 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:28.752 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:28.752 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:28.852 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:28.852 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:28.951 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:28.951 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:29.052 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:29.052 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:29.148 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:29.149 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:29.249 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:29.249 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:29.349 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:29.349 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:29.449 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:29.449 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:29.543 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:29.543 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:29.644 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:29.644 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:52:29.743 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket abort-at-do-reshard 2024-03-09T14:52:29.777 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:52:29.777 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:52:29.822 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.818+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.822+0000 7f002bdf0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:29.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.822+0000 7f002bdf0980 20 realm 2024-03-09T14:52:29.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.822+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.822+0000 7f002bdf0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:29.824 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.822+0000 7f002bdf0980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:52:29.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.822+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.822+0000 7f002bdf0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:29.825 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.822+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.826 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.822+0000 7f002bdf0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:29.826 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.822+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.822+0000 7f002bdf0980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:52:29.827 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.822+0000 7f002bdf0980 20 searching for the correct realm 2024-03-09T14:52:29.833 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.830+0000 7f0029107640 0 -- 172.21.15.1:0/1261340139 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55fab30a4ef0 msgr2=0x55fab30a73b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:29.840 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.834+0000 7f0029908640 0 -- 172.21.15.1:0/1261340139 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f001405ec00 msgr2=0x7f0014054460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:29.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:52:29.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:52:29.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:52:29.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:52:29.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:52:29.848 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:52:29.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:29.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:29.849 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:52:29.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 zone default found 2024-03-09T14:52:29.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 4 Realm: () 2024-03-09T14:52:29.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:52:29.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:52:29.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:52:29.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 zonegroup default 2024-03-09T14:52:29.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:29.850 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:29.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:29.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 started sync module instance, tier type = 2024-03-09T14:52:29.851 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.846+0000 7f002bdf0980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:52:29.867 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.862+0000 7f002bdf0980 20 add_watcher() i=0 2024-03-09T14:52:29.867 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.862+0000 7f002bdf0980 20 add_watcher() i=1 2024-03-09T14:52:29.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.862+0000 7f002bdf0980 20 add_watcher() i=2 2024-03-09T14:52:29.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.862+0000 7f002bdf0980 20 add_watcher() i=3 2024-03-09T14:52:29.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.862+0000 7f002bdf0980 20 add_watcher() i=4 2024-03-09T14:52:29.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.862+0000 7f002bdf0980 20 add_watcher() i=5 2024-03-09T14:52:29.868 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.862+0000 7f002bdf0980 20 add_watcher() i=6 2024-03-09T14:52:29.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.866+0000 7f002bdf0980 20 add_watcher() i=7 2024-03-09T14:52:29.869 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.866+0000 7f002bdf0980 2 all 8 watchers are set, enabling cache 2024-03-09T14:52:29.871 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.866+0000 7efedbfff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:29.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.866+0000 7f002bdf0980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:52:29.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.866+0000 7f002bdf0980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:52:29.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.866+0000 7f002bdf0980 5 note: GC not initialized 2024-03-09T14:52:29.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.866+0000 7efedaffd640 20 reqs_thread_entry: start 2024-03-09T14:52:29.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.866+0000 7f002bdf0980 20 init_complete bucket index max shards: 1 2024-03-09T14:52:29.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.866+0000 7f002bdf0980 20 Filter name: none 2024-03-09T14:52:29.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.866+0000 7efed8ff9640 20 reqs_thread_entry: start 2024-03-09T14:52:29.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.866+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.870+0000 7f002bdf0980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:29.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.870+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.870+0000 7f002bdf0980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:29.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.870+0000 7f002bdf0980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:52:29.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.870+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.870+0000 7f002bdf0980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:29.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.870+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.870+0000 7f002bdf0980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:29.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.870+0000 7f002bdf0980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:52:29.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.870+0000 7f002bdf0980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.874+0000 7f002bdf0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:29.877 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.874+0000 7efedbfff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:29.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.878+0000 7f002bdf0980 20 remove_watcher() i=0 2024-03-09T14:52:29.882 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.878+0000 7f002bdf0980 2 removed watcher, disabling cache 2024-03-09T14:52:29.884 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.882+0000 7f002bdf0980 20 remove_watcher() i=1 2024-03-09T14:52:29.886 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.882+0000 7f002bdf0980 20 remove_watcher() i=2 2024-03-09T14:52:29.888 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.886+0000 7f002bdf0980 20 remove_watcher() i=3 2024-03-09T14:52:29.891 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.886+0000 7f002bdf0980 20 remove_watcher() i=4 2024-03-09T14:52:29.892 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.890+0000 7f002bdf0980 20 remove_watcher() i=5 2024-03-09T14:52:29.893 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.890+0000 7f002bdf0980 20 remove_watcher() i=6 2024-03-09T14:52:29.895 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.890+0000 7f002bdf0980 20 remove_watcher() i=7 2024-03-09T14:52:29.909 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:29.909 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:52:29.909 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: abort-at-do-reshard id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:52:29.909 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:29.909 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket abort-at-do-reshard --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-abort-at do_reshard 2024-03-09T14:52:29.941 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:52:29.941 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:52:29.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:29.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.982+0000 7f6c01dee980 20 realm 2024-03-09T14:52:29.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:29.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.982+0000 7f6c01dee980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:52:29.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:29.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.986+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:29.989 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.986+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:29.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.986+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:52:29.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.986+0000 7f6c01dee980 20 searching for the correct realm 2024-03-09T14:52:29.995 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:29.990+0000 7f6bff105640 0 -- 172.21.15.1:0/173291078 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f6bf0123040 msgr2=0x7f6bf010e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:30.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.206+0000 7f6bff906640 0 -- 172.21.15.1:0/173291078 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f6bf40736d0 msgr2=0x7f6bf4052600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:52:30.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:52:30.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:52:30.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:52:30.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:52:30.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:52:30.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:52:30.427 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.422+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.422+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.422+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:30.428 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:52:30.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 zone default found 2024-03-09T14:52:30.429 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 4 Realm: () 2024-03-09T14:52:30.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:52:30.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:52:30.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:52:30.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 zonegroup default 2024-03-09T14:52:30.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.430 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 started sync module instance, tier type = 2024-03-09T14:52:30.431 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.426+0000 7f6c01dee980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:52:30.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=0 2024-03-09T14:52:30.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=1 2024-03-09T14:52:30.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=2 2024-03-09T14:52:30.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=3 2024-03-09T14:52:30.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=4 2024-03-09T14:52:30.456 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=5 2024-03-09T14:52:30.457 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=6 2024-03-09T14:52:30.458 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=7 2024-03-09T14:52:30.459 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.454+0000 7f6c01dee980 2 all 8 watchers are set, enabling cache 2024-03-09T14:52:30.462 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.458+0000 7f6ab5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:30.463 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.458+0000 7f6c01dee980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:52:30.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.458+0000 7f6c01dee980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:52:30.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.458+0000 7f6c01dee980 5 note: GC not initialized 2024-03-09T14:52:30.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6ab4fe1640 20 reqs_thread_entry: start 2024-03-09T14:52:30.464 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 20 init_complete bucket index max shards: 1 2024-03-09T14:52:30.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 20 Filter name: none 2024-03-09T14:52:30.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6a2e7fc640 20 reqs_thread_entry: start 2024-03-09T14:52:30.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 10 cache get: name=default.rgw.meta+root+abort-at-do-reshard : miss 2024-03-09T14:52:30.465 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:30.466 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 10 cache put: name=default.rgw.meta+root+abort-at-do-reshard info.flags=0x11 2024-03-09T14:52:30.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 10 adding default.rgw.meta+root+abort-at-do-reshard to cache LRU end 2024-03-09T14:52:30.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 : miss 2024-03-09T14:52:30.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:30.467 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:52:30.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:52:30.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:52:30.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:30.468 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.462+0000 7f6c01dee980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 2024-03-09T14:52:30.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.470+0000 7f6c01dee980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:52:30.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.470+0000 7f6c01dee980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:52:30.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.470+0000 7f6c01dee980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:30.474 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.470+0000 7f6c01dee980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28, ofs0, ns] 2024-03-09T14:52:30.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.470+0000 7f6adeffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562429 cookie 94662218958480 notifier 7002 bl.length()=903 2024-03-09T14:52:30.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.470+0000 7f6adeffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:52:30.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.470+0000 7f6adeffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:52:30.475 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.470+0000 7f6adeffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:30.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.474+0000 7f6c01dee980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): orig_sources= new_sources= 2024-03-09T14:52:30.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.474+0000 7f6c01dee980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): potential sources added=[] removed=[] 2024-03-09T14:52:30.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.474+0000 7f6c01dee980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): orig_dests= new_dests= 2024-03-09T14:52:30.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.474+0000 7f6c01dee980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): potential dests added=[] removed=[] 2024-03-09T14:52:30.480 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f6c01dee980 time 2024-03-09T14:52:30.481885+0000 2024-03-09T14:52:30.480 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:52:30.482 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:52:30.482 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f6c056c3d7b] 2024-03-09T14:52:30.482 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f6c056c3f2f] 2024-03-09T14:52:30.482 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5618425f2c51] 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x315) [0x5618426132f5] 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr: 5: main() 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6c0452ed90] 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr: 7: __libc_start_main() 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr: 8: _start() 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr:*** Caught signal (Aborted) ** 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f6c01dee980 thread_name:radosgw-admin 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.478+0000 7f6c01dee980 -1 ./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f6c01dee980 time 2024-03-09T14:52:30.481885+0000 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f6c056c3d7b] 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f6c056c3f2f] 2024-03-09T14:52:30.483 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5618425f2c51] 2024-03-09T14:52:30.484 INFO:tasks.workunit.client.0.smithi001.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x315) [0x5618426132f5] 2024-03-09T14:52:30.484 INFO:tasks.workunit.client.0.smithi001.stderr: 5: main() 2024-03-09T14:52:30.484 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6c0452ed90] 2024-03-09T14:52:30.484 INFO:tasks.workunit.client.0.smithi001.stderr: 7: __libc_start_main() 2024-03-09T14:52:30.484 INFO:tasks.workunit.client.0.smithi001.stderr: 8: _start() 2024-03-09T14:52:30.484 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.485 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:52:30.485 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f6c04547520] 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6c056c3dd5] 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f6c056c3f2f] 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5618425f2c51] 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x315) [0x5618426132f5] 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: 9: main() 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6c0452ed90] 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: 11: __libc_start_main() 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: 12: _start() 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.482+0000 7f6c01dee980 -1 *** Caught signal (Aborted) ** 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f6c01dee980 thread_name:radosgw-admin 2024-03-09T14:52:30.486 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f6c04547520] 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6c056c3dd5] 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f6c056c3f2f] 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5618425f2c51] 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x315) [0x5618426132f5] 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: 9: main() 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6c0452ed90] 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: 11: __libc_start_main() 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: 12: _start() 2024-03-09T14:52:30.487 INFO:tasks.workunit.client.0.smithi001.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-09T14:52:30.488 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.488 INFO:tasks.workunit.client.0.smithi001.stderr:--- begin dump of recent events --- 2024-03-09T14:52:30.522 INFO:tasks.workunit.client.0.smithi001.stderr: -499> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): get_auth_request con 0x7f6be80022b0 auth_method 0 2024-03-09T14:52:30.522 INFO:tasks.workunit.client.0.smithi001.stderr: -498> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.522 INFO:tasks.workunit.client.0.smithi001.stderr: -497> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.522 INFO:tasks.workunit.client.0.smithi001.stderr: -496> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.522 INFO:tasks.workunit.client.0.smithi001.stderr: -495> 2024-03-09T14:52:29.958+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.522 INFO:tasks.workunit.client.0.smithi001.stderr: -494> 2024-03-09T14:52:29.958+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.522 INFO:tasks.workunit.client.0.smithi001.stderr: -493> 2024-03-09T14:52:29.958+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.522 INFO:tasks.workunit.client.0.smithi001.stderr: -492> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.522 INFO:tasks.workunit.client.0.smithi001.stderr: -491> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.522 INFO:tasks.workunit.client.0.smithi001.stderr: -490> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.522 INFO:tasks.workunit.client.0.smithi001.stderr: -489> 2024-03-09T14:52:29.958+0000 7f6c00107640 10 monclient(hunting): handle_auth_done global_id 6551 payload 274 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr: -488> 2024-03-09T14:52:29.958+0000 7f6c00107640 10 monclient: _finish_hunting 0 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr: -487> 2024-03-09T14:52:29.958+0000 7f6c00107640 1 monclient: found mon.noname-c 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr: -486> 2024-03-09T14:52:29.958+0000 7f6c00107640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr: -485> 2024-03-09T14:52:29.958+0000 7f6bff105640 10 monclient: get_auth_request con 0x561843e01100 auth_method 0 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr: -484> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr: -483> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr: -482> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: dump: 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.523 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr: -481> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: _finish_auth 0 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr: -480> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: _check_auth_tickets 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr: -479> 2024-03-09T14:52:29.958+0000 7f6c01dee980 5 monclient: authenticate success, global_id 6551 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr: -478> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr: -477> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr: -476> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr: -475> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: dump: 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.524 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr: -474> 2024-03-09T14:52:29.958+0000 7f6c01dee980 10 monclient: get_monmap_and_config success 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr: -473> 2024-03-09T14:52:29.958+0000 7f6c01dee980 4 set_mon_vals no callback set 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr: -472> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: shutdown 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr: -471> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 asok(0x561843beebb0) unregister_commands rotate-key 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr: -470> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: build_initial_monmap 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr: -469> 2024-03-09T14:52:29.962+0000 7f6c01dee980 1 build_initial for_mkfs: 0 2024-03-09T14:52:30.525 INFO:tasks.workunit.client.0.smithi001.stderr: -468> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: monmap: 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr: -467> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding auth protocol: cephx 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr: -466> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding auth protocol: cephx 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr: -465> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding auth protocol: cephx 2024-03-09T14:52:30.526 INFO:tasks.workunit.client.0.smithi001.stderr: -464> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: secure 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -463> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: crc 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -462> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: secure 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -461> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: crc 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -460> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: secure 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -459> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: crc 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -458> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: crc 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -457> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: secure 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -456> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: crc 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -455> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: secure 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -454> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: crc 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -453> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: secure 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -452> 2024-03-09T14:52:29.962+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -451> 2024-03-09T14:52:29.962+0000 7f6c01dee980 1 librados: starting msgr at 2024-03-09T14:52:30.527 INFO:tasks.workunit.client.0.smithi001.stderr: -450> 2024-03-09T14:52:29.962+0000 7f6c01dee980 1 librados: starting objecter 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -449> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command objecter_requests hook 0x561843c285d0 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -448> 2024-03-09T14:52:29.962+0000 7f6c01dee980 1 librados: setting wanted keys 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -447> 2024-03-09T14:52:29.962+0000 7f6c01dee980 1 librados: calling monclient init 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -446> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: init 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -445> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding auth protocol: cephx 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -444> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding auth protocol: cephx 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -443> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding auth protocol: cephx 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -442> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: secure 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -441> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: crc 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -440> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: secure 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -439> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: crc 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -438> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: secure 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -437> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: crc 2024-03-09T14:52:30.528 INFO:tasks.workunit.client.0.smithi001.stderr: -436> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: crc 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -435> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: secure 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -434> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: crc 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -433> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: secure 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -432> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: crc 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -431> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: secure 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -430> 2024-03-09T14:52:29.962+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -429> 2024-03-09T14:52:29.962+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -428> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command rotate-key hook 0x561843dfd168 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -427> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: _reopen_session rank -1 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -426> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: _add_conns ranks=[0,2,1] 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -425> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-a con 0x561843e01100 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -424> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-c con 0x561843e2bd30 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:52:30.529 INFO:tasks.workunit.client.0.smithi001.stderr: -423> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-b con 0x7f6be80022b0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -422> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -421> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -420> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -419> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): _renew_subs 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -418> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): authenticate will time out at 2443.709934s 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -417> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): get_auth_request con 0x561843e2bd30 auth_method 0 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -416> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -415> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -414> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -413> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): get_auth_request con 0x561843e01100 auth_method 0 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -412> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -411> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.530 INFO:tasks.workunit.client.0.smithi001.stderr: -410> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -409> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -408> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -407> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -406> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -405> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -404> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -403> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): handle_auth_done global_id 6554 payload 824 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -402> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient: _finish_hunting 0 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -401> 2024-03-09T14:52:29.962+0000 7f6bff105640 1 monclient: found mon.noname-c 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -400> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -399> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -398> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr: -397> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: dump: 2024-03-09T14:52:30.531 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr: -396> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: _finish_auth 0 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr: -395> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: _check_auth_tickets 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr: -394> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr: -393> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.532 INFO:tasks.workunit.client.0.smithi001.stderr: -392> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr: -391> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: dump: 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr: -390> 2024-03-09T14:52:29.962+0000 7f6aede0b640 4 set_mon_vals no callback set 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr: -389> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 monclient: authenticate success, global_id 6554 2024-03-09T14:52:30.533 INFO:tasks.workunit.client.0.smithi001.stderr: -388> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: _renew_subs 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -387> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -386> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: _renew_subs 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -385> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -384> 2024-03-09T14:52:29.962+0000 7f6c01dee980 1 librados: init done 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -383> 2024-03-09T14:52:29.966+0000 7f6ae7fff640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -382> 2024-03-09T14:52:29.966+0000 7f6ae7fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -381> 2024-03-09T14:52:29.966+0000 7f6ae7fff640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -380> 2024-03-09T14:52:29.966+0000 7f6bff906640 10 monclient: get_auth_request con 0x7f6ae8066040 auth_method 0 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -379> 2024-03-09T14:52:29.966+0000 7f6c00107640 10 monclient: get_auth_request con 0x561843e3ece0 auth_method 0 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -378> 2024-03-09T14:52:29.966+0000 7f6bff906640 10 monclient: get_auth_request con 0x561843e48380 auth_method 0 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -377> 2024-03-09T14:52:29.970+0000 7f6bff105640 10 monclient: get_auth_request con 0x561843e4cf30 auth_method 0 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -376> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient: build_initial_monmap 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -375> 2024-03-09T14:52:29.970+0000 7f6c01dee980 1 build_initial for_mkfs: 0 2024-03-09T14:52:30.534 INFO:tasks.workunit.client.0.smithi001.stderr: -374> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient: monmap: 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr: -373> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding auth protocol: cephx 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr: -372> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding auth protocol: cephx 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr: -371> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding auth protocol: cephx 2024-03-09T14:52:30.535 INFO:tasks.workunit.client.0.smithi001.stderr: -370> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: secure 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -369> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: crc 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -368> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: secure 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -367> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: crc 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -366> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: secure 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -365> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: crc 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -364> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: crc 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -363> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: secure 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -362> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: crc 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -361> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: secure 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -360> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: crc 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -359> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: secure 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -358> 2024-03-09T14:52:29.970+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -357> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command objecter_requests cmddesc objecter_requests hook 0x561843e53bf0 EEXIST 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -356> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient: init 2024-03-09T14:52:30.536 INFO:tasks.workunit.client.0.smithi001.stderr: -355> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding auth protocol: cephx 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -354> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding auth protocol: cephx 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -353> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding auth protocol: cephx 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -352> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: secure 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -351> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: crc 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -350> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: secure 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -349> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: crc 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -348> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: secure 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -347> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: crc 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -346> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: crc 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -345> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: secure 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -344> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: crc 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -343> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: secure 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -342> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: crc 2024-03-09T14:52:30.537 INFO:tasks.workunit.client.0.smithi001.stderr: -341> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: secure 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -340> 2024-03-09T14:52:29.970+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -339> 2024-03-09T14:52:29.970+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -338> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command rotate-key cmddesc rotate-key hook 0x561843e50000 EEXIST 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -337> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient: _reopen_session rank -1 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -336> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient: _add_conns ranks=[2,1,0] 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -335> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-c con 0x561843e7b3d0 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -334> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-b con 0x561843e7b7b0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -333> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-a con 0x561843e8e0e0 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -332> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -331> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -330> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -329> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): _renew_subs 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -328> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): authenticate will time out at 2443.719470s 2024-03-09T14:52:30.538 INFO:tasks.workunit.client.0.smithi001.stderr: -327> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): get_auth_request con 0x561843e7b3d0 auth_method 0 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -326> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -325> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -324> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -323> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): get_auth_request con 0x561843e8e0e0 auth_method 0 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -322> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -321> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -320> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -319> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -318> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -317> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -316> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -315> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -314> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -313> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): handle_auth_done global_id 6557 payload 824 2024-03-09T14:52:30.539 INFO:tasks.workunit.client.0.smithi001.stderr: -312> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient: _finish_hunting 0 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr: -311> 2024-03-09T14:52:29.970+0000 7f6bff906640 1 monclient: found mon.noname-c 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr: -310> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr: -309> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr: -308> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr: -307> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: dump: 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.540 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr: -306> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: _finish_auth 0 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr: -305> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: _check_auth_tickets 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr: -304> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr: -303> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr: -302> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr: -301> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: dump: 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.541 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr: -300> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 monclient: authenticate success, global_id 6557 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr: -299> 2024-03-09T14:52:29.974+0000 7f6c00908640 4 set_mon_vals no callback set 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr: -298> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: _renew_subs 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr: -297> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr: -296> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: _renew_subs 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr: -295> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr: -294> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr: -293> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr: -292> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr: -291> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: get_monmap_and_config 2024-03-09T14:52:30.542 INFO:tasks.workunit.client.0.smithi001.stderr: -290> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: build_initial_monmap 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr: -289> 2024-03-09T14:52:29.974+0000 7f6c01dee980 1 build_initial for_mkfs: 0 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr: -288> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: monmap: 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr: -287> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding auth protocol: cephx 2024-03-09T14:52:30.543 INFO:tasks.workunit.client.0.smithi001.stderr: -286> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding auth protocol: cephx 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -285> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding auth protocol: cephx 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -284> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -283> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -282> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -281> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -280> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -279> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -278> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -277> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -276> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -275> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -274> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -273> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.544 INFO:tasks.workunit.client.0.smithi001.stderr: -272> 2024-03-09T14:52:29.974+0000 7f6bff105640 10 monclient: get_auth_request con 0x7f6ad8065f50 auth_method 0 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -271> 2024-03-09T14:52:29.974+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -270> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: init 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -269> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding auth protocol: cephx 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -268> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding auth protocol: cephx 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -267> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding auth protocol: cephx 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -266> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: secure 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -265> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: crc 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -264> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: secure 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -263> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: crc 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -262> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: secure 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -261> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: crc 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -260> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: crc 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -259> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: secure 2024-03-09T14:52:30.545 INFO:tasks.workunit.client.0.smithi001.stderr: -258> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: crc 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -257> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: secure 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -256> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: crc 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -255> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: secure 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -254> 2024-03-09T14:52:29.974+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -253> 2024-03-09T14:52:29.974+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -252> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command rotate-key cmddesc rotate-key hook 0x7ffc990aab08 EEXIST 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -251> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: _reopen_session rank -1 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -250> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: _add_conns ranks=[0,1,2] 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -249> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-a con 0x561843ecd750 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -248> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-b con 0x561843e9e640 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -247> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-c con 0x561843ec9110 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -246> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -245> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -244> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.546 INFO:tasks.workunit.client.0.smithi001.stderr: -243> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): _renew_subs 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -242> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): authenticate will time out at 2443.723556s 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -241> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): get_auth_request con 0x561843ecd750 auth_method 0 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -240> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -239> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -238> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -237> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): get_auth_request con 0x561843ec9110 auth_method 0 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -236> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -235> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -234> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -233> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -232> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -231> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.547 INFO:tasks.workunit.client.0.smithi001.stderr: -230> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr: -229> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr: -228> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr: -227> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): handle_auth_done global_id 6999 payload 274 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr: -226> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient: _finish_hunting 0 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr: -225> 2024-03-09T14:52:29.974+0000 7f6bff906640 1 monclient: found mon.noname-a 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr: -224> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr: -223> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr: -222> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr: -221> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: dump: 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.548 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr: -220> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: _finish_auth 0 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr: -219> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: _check_auth_tickets 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr: -218> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 monclient: authenticate success, global_id 6999 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr: -217> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr: -216> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: get_monmap_and_config success 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr: -215> 2024-03-09T14:52:29.978+0000 7f6c01dee980 4 set_mon_vals no callback set 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr: -214> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr: -213> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:52:30.549 INFO:tasks.workunit.client.0.smithi001.stderr: -212> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: dump: 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr: -211> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: shutdown 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr: -210> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: build_initial_monmap 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr: -209> 2024-03-09T14:52:29.978+0000 7f6c01dee980 1 build_initial for_mkfs: 0 2024-03-09T14:52:30.550 INFO:tasks.workunit.client.0.smithi001.stderr: -208> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: monmap: 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr: -207> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding auth protocol: cephx 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr: -206> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding auth protocol: cephx 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr: -205> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding auth protocol: cephx 2024-03-09T14:52:30.551 INFO:tasks.workunit.client.0.smithi001.stderr: -204> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -203> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -202> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -201> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -200> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -199> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -198> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -197> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -196> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -195> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -194> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -193> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -192> 2024-03-09T14:52:29.978+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -191> 2024-03-09T14:52:29.978+0000 7f6c01dee980 1 librados: starting msgr at 2024-03-09T14:52:30.552 INFO:tasks.workunit.client.0.smithi001.stderr: -190> 2024-03-09T14:52:29.978+0000 7f6c01dee980 1 librados: starting objecter 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -189> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command objecter_requests cmddesc objecter_requests hook 0x561843e9b2d0 EEXIST 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -188> 2024-03-09T14:52:29.978+0000 7f6c01dee980 1 librados: setting wanted keys 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -187> 2024-03-09T14:52:29.978+0000 7f6c01dee980 1 librados: calling monclient init 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -186> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: init 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -185> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding auth protocol: cephx 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -184> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding auth protocol: cephx 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -183> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding auth protocol: cephx 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -182> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: secure 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -181> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: crc 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -180> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: secure 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -179> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: crc 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -178> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: secure 2024-03-09T14:52:30.553 INFO:tasks.workunit.client.0.smithi001.stderr: -177> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: crc 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -176> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: crc 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -175> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: secure 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -174> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: crc 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -173> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: secure 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -172> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: crc 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -171> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: secure 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -170> 2024-03-09T14:52:29.978+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -169> 2024-03-09T14:52:29.978+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -168> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command rotate-key cmddesc rotate-key hook 0x561843e981a8 EEXIST 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -167> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: _reopen_session rank -1 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -166> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: _add_conns ranks=[0,1,2] 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -165> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-a con 0x561843e9e640 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -164> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-b con 0x561843ec9110 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -163> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-c con 0x561843ecd750 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:52:30.554 INFO:tasks.workunit.client.0.smithi001.stderr: -162> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -161> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -160> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -159> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): _renew_subs 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -158> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): authenticate will time out at 2443.726078s 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -157> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): get_auth_request con 0x561843e9e640 auth_method 0 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -156> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -155> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -154> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -153> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): get_auth_request con 0x561843ecd750 auth_method 0 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -152> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -151> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -150> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -149> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.555 INFO:tasks.workunit.client.0.smithi001.stderr: -148> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr: -147> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr: -146> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr: -145> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr: -144> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr: -143> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): handle_auth_done global_id 7002 payload 824 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr: -142> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient: _finish_hunting 0 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr: -141> 2024-03-09T14:52:29.978+0000 7f6bff906640 1 monclient: found mon.noname-a 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr: -140> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr: -139> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr: -138> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr: -137> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: dump: 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.556 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr: -136> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: _finish_auth 0 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr: -135> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: _check_auth_tickets 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr: -134> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr: -133> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr: -132> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr: -131> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: dump: 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.557 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr: -130> 2024-03-09T14:52:29.978+0000 7f6adeffd640 4 set_mon_vals no callback set 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr: -129> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 monclient: authenticate success, global_id 7002 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr: -128> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: _renew_subs 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr: -127> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr: -126> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: _renew_subs 2024-03-09T14:52:30.558 INFO:tasks.workunit.client.0.smithi001.stderr: -125> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -124> 2024-03-09T14:52:29.978+0000 7f6c01dee980 1 librados: init done 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -123> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command cr dump hook 0x561843ec7e38 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -122> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -121> 2024-03-09T14:52:29.982+0000 7f6add7fa640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -120> 2024-03-09T14:52:29.982+0000 7f6add7fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -119> 2024-03-09T14:52:29.982+0000 7f6add7fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -118> 2024-03-09T14:52:29.982+0000 7f6bff105640 10 monclient: get_auth_request con 0x7f6ac8066040 auth_method 0 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -117> 2024-03-09T14:52:29.982+0000 7f6c00107640 10 monclient: get_auth_request con 0x561843ee6be0 auth_method 0 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -116> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -115> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 realm 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -114> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -113> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -112> 2024-03-09T14:52:29.982+0000 7f6c01dee980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -111> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.559 INFO:tasks.workunit.client.0.smithi001.stderr: -110> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -109> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -108> 2024-03-09T14:52:29.982+0000 7f6bff906640 10 monclient: get_auth_request con 0x561843eeae10 auth_method 0 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -107> 2024-03-09T14:52:29.986+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -106> 2024-03-09T14:52:29.986+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -105> 2024-03-09T14:52:29.986+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -104> 2024-03-09T14:52:29.986+0000 7f6c01dee980 20 searching for the correct realm 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -103> 2024-03-09T14:52:29.986+0000 7f6bff105640 10 monclient: get_auth_request con 0x561843ef0050 auth_method 0 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -102> 2024-03-09T14:52:29.990+0000 7f6c00107640 10 monclient: get_auth_request con 0x7f6be8003100 auth_method 0 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -101> 2024-03-09T14:52:29.990+0000 7f6bff906640 10 monclient: get_auth_request con 0x7f6be8003be0 auth_method 0 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -100> 2024-03-09T14:52:29.990+0000 7f6bff105640 10 monclient: get_auth_request con 0x7f6bf0123040 auth_method 0 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -99> 2024-03-09T14:52:29.990+0000 7f6bff105640 0 -- 172.21.15.1:0/173291078 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f6bf0123040 msgr2=0x7f6bf010e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -98> 2024-03-09T14:52:30.194+0000 7f6bff105640 10 monclient: get_auth_request con 0x7f6bf0123040 auth_method 0 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -97> 2024-03-09T14:52:30.198+0000 7f6c00107640 10 monclient: get_auth_request con 0x7f6bf407c810 auth_method 0 2024-03-09T14:52:30.560 INFO:tasks.workunit.client.0.smithi001.stderr: -96> 2024-03-09T14:52:30.206+0000 7f6bff906640 10 monclient: get_auth_request con 0x7f6bf40736d0 auth_method 0 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -95> 2024-03-09T14:52:30.206+0000 7f6bff906640 0 -- 172.21.15.1:0/173291078 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f6bf40736d0 msgr2=0x7f6bf4052600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -94> 2024-03-09T14:52:30.410+0000 7f6bff906640 10 monclient: get_auth_request con 0x7f6bf40736d0 auth_method 0 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -93> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -92> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -91> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -90> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -89> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -88> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -87> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -86> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -85> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -84> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -83> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.561 INFO:tasks.workunit.client.0.smithi001.stderr: -82> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:52:30.562 INFO:tasks.workunit.client.0.smithi001.stderr: -81> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 zone default found 2024-03-09T14:52:30.562 INFO:tasks.workunit.client.0.smithi001.stderr: -80> 2024-03-09T14:52:30.426+0000 7f6c01dee980 4 Realm: () 2024-03-09T14:52:30.562 INFO:tasks.workunit.client.0.smithi001.stderr: -79> 2024-03-09T14:52:30.426+0000 7f6c01dee980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:52:30.562 INFO:tasks.workunit.client.0.smithi001.stderr: -78> 2024-03-09T14:52:30.426+0000 7f6c01dee980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:52:30.562 INFO:tasks.workunit.client.0.smithi001.stderr: -77> 2024-03-09T14:52:30.426+0000 7f6c01dee980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:52:30.562 INFO:tasks.workunit.client.0.smithi001.stderr: -76> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 zonegroup default 2024-03-09T14:52:30.562 INFO:tasks.workunit.client.0.smithi001.stderr: -75> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.562 INFO:tasks.workunit.client.0.smithi001.stderr: -74> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.562 INFO:tasks.workunit.client.0.smithi001.stderr: -73> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.562 INFO:tasks.workunit.client.0.smithi001.stderr: -72> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.562 INFO:tasks.workunit.client.0.smithi001.stderr: -71> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -70> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -69> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 started sync module instance, tier type = 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -68> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -67> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=0 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -66> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=1 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -65> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=2 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -64> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=3 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -63> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=4 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -62> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=5 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -61> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=6 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -60> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=7 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -59> 2024-03-09T14:52:30.454+0000 7f6c01dee980 2 all 8 watchers are set, enabling cache 2024-03-09T14:52:30.563 INFO:tasks.workunit.client.0.smithi001.stderr: -58> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding auth protocol: cephx 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -57> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding auth protocol: cephx 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -56> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding auth protocol: cephx 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -55> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: secure 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -54> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: crc 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -53> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: secure 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -52> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: crc 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -51> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: secure 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -50> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: crc 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -49> 2024-03-09T14:52:30.458+0000 7f6ab5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -48> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: crc 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -47> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: secure 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -46> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: crc 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -45> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: secure 2024-03-09T14:52:30.564 INFO:tasks.workunit.client.0.smithi001.stderr: -44> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: crc 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -43> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: secure 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -42> 2024-03-09T14:52:30.458+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -41> 2024-03-09T14:52:30.458+0000 7f6c01dee980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -40> 2024-03-09T14:52:30.458+0000 7f6c01dee980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -39> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command cache list hook 0x561843ebf9f0 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -38> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command cache inspect hook 0x561843ebf9f0 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -37> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command cache erase hook 0x561843ebf9f0 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -36> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command cache zap hook 0x561843ebf9f0 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -35> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 note: GC not initialized 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -34> 2024-03-09T14:52:30.462+0000 7f6ab4fe1640 20 reqs_thread_entry: start 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -33> 2024-03-09T14:52:30.462+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command sync trace show hook 0x7f6bf0001df0 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -32> 2024-03-09T14:52:30.462+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command sync trace history hook 0x7f6bf0001df0 2024-03-09T14:52:30.565 INFO:tasks.workunit.client.0.smithi001.stderr: -31> 2024-03-09T14:52:30.462+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command sync trace active hook 0x7f6bf0001df0 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -30> 2024-03-09T14:52:30.462+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command sync trace active_short hook 0x7f6bf0001df0 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -29> 2024-03-09T14:52:30.462+0000 7f6c01dee980 20 init_complete bucket index max shards: 1 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -28> 2024-03-09T14:52:30.462+0000 7f6c01dee980 20 Filter name: none 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -27> 2024-03-09T14:52:30.462+0000 7f6a2e7fc640 20 reqs_thread_entry: start 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -26> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 cache get: name=default.rgw.meta+root+abort-at-do-reshard : miss 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -25> 2024-03-09T14:52:30.462+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -24> 2024-03-09T14:52:30.462+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -23> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 cache put: name=default.rgw.meta+root+abort-at-do-reshard info.flags=0x11 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -22> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 adding default.rgw.meta+root+abort-at-do-reshard to cache LRU end 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -21> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 : miss 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -20> 2024-03-09T14:52:30.462+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -19> 2024-03-09T14:52:30.462+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -18> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:52:30.566 INFO:tasks.workunit.client.0.smithi001.stderr: -17> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -16> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -15> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -14> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -13> 2024-03-09T14:52:30.470+0000 7f6c01dee980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -12> 2024-03-09T14:52:30.470+0000 7f6c01dee980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -11> 2024-03-09T14:52:30.470+0000 7f6c01dee980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -10> 2024-03-09T14:52:30.470+0000 7f6c01dee980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28, ofs0, ns] 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -9> 2024-03-09T14:52:30.470+0000 7f6adeffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562429 cookie 94662218958480 notifier 7002 bl.length()=903 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -8> 2024-03-09T14:52:30.470+0000 7f6adeffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -7> 2024-03-09T14:52:30.470+0000 7f6adeffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -6> 2024-03-09T14:52:30.470+0000 7f6adeffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -5> 2024-03-09T14:52:30.474+0000 7f6c01dee980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): orig_sources= new_sources= 2024-03-09T14:52:30.567 INFO:tasks.workunit.client.0.smithi001.stderr: -4> 2024-03-09T14:52:30.474+0000 7f6c01dee980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): potential sources added=[] removed=[] 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: -3> 2024-03-09T14:52:30.474+0000 7f6c01dee980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): orig_dests= new_dests= 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: -2> 2024-03-09T14:52:30.474+0000 7f6c01dee980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): potential dests added=[] removed=[] 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: -1> 2024-03-09T14:52:30.478+0000 7f6c01dee980 -1 ./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f6c01dee980 time 2024-03-09T14:52:30.481885+0000 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f6c056c3d7b] 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f6c056c3f2f] 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5618425f2c51] 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x315) [0x5618426132f5] 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: 5: main() 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6c0452ed90] 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: 7: __libc_start_main() 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: 8: _start() 2024-03-09T14:52:30.568 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: 0> 2024-03-09T14:52:30.482+0000 7f6c01dee980 -1 *** Caught signal (Aborted) ** 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f6c01dee980 thread_name:radosgw-admin 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f6c04547520] 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6c056c3dd5] 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f6c056c3f2f] 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5618425f2c51] 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x315) [0x5618426132f5] 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: 9: main() 2024-03-09T14:52:30.569 INFO:tasks.workunit.client.0.smithi001.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6c0452ed90] 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: 11: __libc_start_main() 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: 12: _start() 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr:--- logging levels --- 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 none 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 lockdep 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 context 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 crush 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_balancer 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_locker 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log 2024-03-09T14:52:30.570 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log_expire 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_migrator 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 buffer 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 timer 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 filer 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 striper 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 objecter 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rados 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_mirror 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_replay 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_pwl 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 journaler 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objectcacher 2024-03-09T14:52:30.571 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 immutable_obj_cache 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 client 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 osd 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 optracker 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objclass 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 filestore 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 journal 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 ms 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mon 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 0/10 monc 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 paxos 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 tp 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 auth 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 crypto 2024-03-09T14:52:30.572 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 finisher 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 reserver 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 heartbeatmap 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 perfcounter 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 20/20 rgw 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_sync 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_datacache 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_access 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_dbstore 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_flight 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_lifecycle 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 javaclient 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 asok 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 throttle 2024-03-09T14:52:30.573 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 refs 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 compressor 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluestore 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluefs 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 bdev 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 kstore 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 4/ 5 rocksdb 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 fuse 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 2/ 5 mgr 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mgrc 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 dpdk 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 eventtrace 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 prioritycache 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 test 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephfs_mirror 2024-03-09T14:52:30.574 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephsqlite 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 crimson_interrupt 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_onode 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_odata 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_omap 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_tm 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_t 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cleaner 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_epm 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_lba 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_fixedkv_tree 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cache 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_journal 2024-03-09T14:52:30.575 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_device 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_backref 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 alienstore 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mclock 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cyanstore 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 ceph_exporter 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 memstore 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 trace 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: -2/-2 (syslog threshold) 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 99/99 (stderr threshold) 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr:--- pthread ID / name mapping for recent threads --- 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6a2e7fc640 / http_manager 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6ab4fe1640 / http_manager 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6ab5fe3640 / rgw_dt_lg_renew 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6add7fa640 / ms_dispatch 2024-03-09T14:52:30.576 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6adeffd640 / io_context_pool 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6ae57fa640 / ms_dispatch 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6ae7fff640 / ms_dispatch 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6aede0b640 / io_context_pool 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6bff105640 / msgr-worker-2 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6bff906640 / msgr-worker-1 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6c00107640 / msgr-worker-0 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6c00908640 / io_context_pool 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6c01dee980 / radosgw-admin 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: max_recent 500 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: max_new 1000 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: log_file /var/log/ceph/ceph-client.admin.101366.log 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr:--- end dump of recent events --- 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr:--- begin dump of recent events --- 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: -499> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): get_auth_request con 0x7f6be80022b0 auth_method 0 2024-03-09T14:52:30.577 INFO:tasks.workunit.client.0.smithi001.stderr: -498> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -497> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -496> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -495> 2024-03-09T14:52:29.958+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -494> 2024-03-09T14:52:29.958+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -493> 2024-03-09T14:52:29.958+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -492> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -491> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -490> 2024-03-09T14:52:29.958+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -489> 2024-03-09T14:52:29.958+0000 7f6c00107640 10 monclient(hunting): handle_auth_done global_id 6551 payload 274 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -488> 2024-03-09T14:52:29.958+0000 7f6c00107640 10 monclient: _finish_hunting 0 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -487> 2024-03-09T14:52:29.958+0000 7f6c00107640 1 monclient: found mon.noname-c 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -486> 2024-03-09T14:52:29.958+0000 7f6c00107640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -485> 2024-03-09T14:52:29.958+0000 7f6bff105640 10 monclient: get_auth_request con 0x561843e01100 auth_method 0 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -484> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.578 INFO:tasks.workunit.client.0.smithi001.stderr: -483> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr: -482> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: dump: 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr: -481> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: _finish_auth 0 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr: -480> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: _check_auth_tickets 2024-03-09T14:52:30.579 INFO:tasks.workunit.client.0.smithi001.stderr: -479> 2024-03-09T14:52:29.958+0000 7f6c01dee980 5 monclient: authenticate success, global_id 6551 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr: -478> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr: -477> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr: -476> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr: -475> 2024-03-09T14:52:29.958+0000 7f6aede0b640 10 monclient: dump: 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.580 INFO:tasks.workunit.client.0.smithi001.stderr: -474> 2024-03-09T14:52:29.958+0000 7f6c01dee980 10 monclient: get_monmap_and_config success 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr: -473> 2024-03-09T14:52:29.958+0000 7f6c01dee980 4 set_mon_vals no callback set 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr: -472> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: shutdown 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr: -471> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 asok(0x561843beebb0) unregister_commands rotate-key 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr: -470> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: build_initial_monmap 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr: -469> 2024-03-09T14:52:29.962+0000 7f6c01dee980 1 build_initial for_mkfs: 0 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr: -468> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: monmap: 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:52:30.581 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -467> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding auth protocol: cephx 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -466> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding auth protocol: cephx 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -465> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding auth protocol: cephx 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -464> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: secure 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -463> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: crc 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -462> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: secure 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -461> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: crc 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -460> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: secure 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -459> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: crc 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -458> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: crc 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -457> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: secure 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -456> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: crc 2024-03-09T14:52:30.582 INFO:tasks.workunit.client.0.smithi001.stderr: -455> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: secure 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -454> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: crc 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -453> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfee60) adding con mode: secure 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -452> 2024-03-09T14:52:29.962+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -451> 2024-03-09T14:52:29.962+0000 7f6c01dee980 1 librados: starting msgr at 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -450> 2024-03-09T14:52:29.962+0000 7f6c01dee980 1 librados: starting objecter 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -449> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command objecter_requests hook 0x561843c285d0 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -448> 2024-03-09T14:52:29.962+0000 7f6c01dee980 1 librados: setting wanted keys 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -447> 2024-03-09T14:52:29.962+0000 7f6c01dee980 1 librados: calling monclient init 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -446> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: init 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -445> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding auth protocol: cephx 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -444> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding auth protocol: cephx 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -443> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding auth protocol: cephx 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -442> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: secure 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -441> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: crc 2024-03-09T14:52:30.583 INFO:tasks.workunit.client.0.smithi001.stderr: -440> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: secure 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -439> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: crc 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -438> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: secure 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -437> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: crc 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -436> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: crc 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -435> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: secure 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -434> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: crc 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -433> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: secure 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -432> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: crc 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -431> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 AuthRegistry(0x561843dfd020) adding con mode: secure 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -430> 2024-03-09T14:52:29.962+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -429> 2024-03-09T14:52:29.962+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -428> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command rotate-key hook 0x561843dfd168 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -427> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: _reopen_session rank -1 2024-03-09T14:52:30.584 INFO:tasks.workunit.client.0.smithi001.stderr: -426> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: _add_conns ranks=[0,2,1] 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -425> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-a con 0x561843e01100 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -424> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-c con 0x561843e2bd30 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -423> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-b con 0x7f6be80022b0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -422> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -421> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -420> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -419> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): _renew_subs 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -418> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient(hunting): authenticate will time out at 2443.709934s 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -417> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): get_auth_request con 0x561843e2bd30 auth_method 0 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -416> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -415> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -414> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -413> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): get_auth_request con 0x561843e01100 auth_method 0 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -412> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.585 INFO:tasks.workunit.client.0.smithi001.stderr: -411> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -410> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -409> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -408> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -407> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -406> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -405> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -404> 2024-03-09T14:52:29.962+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -403> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient(hunting): handle_auth_done global_id 6554 payload 824 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -402> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient: _finish_hunting 0 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -401> 2024-03-09T14:52:29.962+0000 7f6bff105640 1 monclient: found mon.noname-c 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -400> 2024-03-09T14:52:29.962+0000 7f6bff105640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -399> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.586 INFO:tasks.workunit.client.0.smithi001.stderr: -398> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr: -397> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: dump: 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr: -396> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: _finish_auth 0 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr: -395> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: _check_auth_tickets 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr: -394> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:52:30.587 INFO:tasks.workunit.client.0.smithi001.stderr: -393> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr: -392> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr: -391> 2024-03-09T14:52:29.962+0000 7f6ae7fff640 10 monclient: dump: 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr: -390> 2024-03-09T14:52:29.962+0000 7f6aede0b640 4 set_mon_vals no callback set 2024-03-09T14:52:30.588 INFO:tasks.workunit.client.0.smithi001.stderr: -389> 2024-03-09T14:52:29.962+0000 7f6c01dee980 5 monclient: authenticate success, global_id 6554 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -388> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: _renew_subs 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -387> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -386> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: _renew_subs 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -385> 2024-03-09T14:52:29.962+0000 7f6c01dee980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -384> 2024-03-09T14:52:29.962+0000 7f6c01dee980 1 librados: init done 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -383> 2024-03-09T14:52:29.966+0000 7f6ae7fff640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -382> 2024-03-09T14:52:29.966+0000 7f6ae7fff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -381> 2024-03-09T14:52:29.966+0000 7f6ae7fff640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -380> 2024-03-09T14:52:29.966+0000 7f6bff906640 10 monclient: get_auth_request con 0x7f6ae8066040 auth_method 0 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -379> 2024-03-09T14:52:29.966+0000 7f6c00107640 10 monclient: get_auth_request con 0x561843e3ece0 auth_method 0 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -378> 2024-03-09T14:52:29.966+0000 7f6bff906640 10 monclient: get_auth_request con 0x561843e48380 auth_method 0 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -377> 2024-03-09T14:52:29.970+0000 7f6bff105640 10 monclient: get_auth_request con 0x561843e4cf30 auth_method 0 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -376> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient: build_initial_monmap 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -375> 2024-03-09T14:52:29.970+0000 7f6c01dee980 1 build_initial for_mkfs: 0 2024-03-09T14:52:30.589 INFO:tasks.workunit.client.0.smithi001.stderr: -374> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient: monmap: 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr: -373> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding auth protocol: cephx 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr: -372> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding auth protocol: cephx 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr: -371> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding auth protocol: cephx 2024-03-09T14:52:30.590 INFO:tasks.workunit.client.0.smithi001.stderr: -370> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: secure 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -369> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: crc 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -368> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: secure 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -367> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: crc 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -366> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: secure 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -365> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: crc 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -364> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: crc 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -363> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: secure 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -362> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: crc 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -361> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: secure 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -360> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: crc 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -359> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e50ed0) adding con mode: secure 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -358> 2024-03-09T14:52:29.970+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -357> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command objecter_requests cmddesc objecter_requests hook 0x561843e53bf0 EEXIST 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -356> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient: init 2024-03-09T14:52:30.591 INFO:tasks.workunit.client.0.smithi001.stderr: -355> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding auth protocol: cephx 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -354> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding auth protocol: cephx 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -353> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding auth protocol: cephx 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -352> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: secure 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -351> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: crc 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -350> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: secure 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -349> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: crc 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -348> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: secure 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -347> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: crc 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -346> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: crc 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -345> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: secure 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -344> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: crc 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -343> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: secure 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -342> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: crc 2024-03-09T14:52:30.592 INFO:tasks.workunit.client.0.smithi001.stderr: -341> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 AuthRegistry(0x561843e4feb8) adding con mode: secure 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -340> 2024-03-09T14:52:29.970+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -339> 2024-03-09T14:52:29.970+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -338> 2024-03-09T14:52:29.970+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command rotate-key cmddesc rotate-key hook 0x561843e50000 EEXIST 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -337> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient: _reopen_session rank -1 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -336> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient: _add_conns ranks=[2,1,0] 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -335> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-c con 0x561843e7b3d0 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -334> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-b con 0x561843e7b7b0 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -333> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-a con 0x561843e8e0e0 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -332> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -331> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -330> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -329> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): _renew_subs 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -328> 2024-03-09T14:52:29.970+0000 7f6c01dee980 10 monclient(hunting): authenticate will time out at 2443.719470s 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -327> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): get_auth_request con 0x561843e7b3d0 auth_method 0 2024-03-09T14:52:30.593 INFO:tasks.workunit.client.0.smithi001.stderr: -326> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -325> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -324> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -323> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): get_auth_request con 0x561843e8e0e0 auth_method 0 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -322> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -321> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -320> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -319> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -318> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -317> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -316> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -315> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -314> 2024-03-09T14:52:29.970+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -313> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient(hunting): handle_auth_done global_id 6557 payload 824 2024-03-09T14:52:30.594 INFO:tasks.workunit.client.0.smithi001.stderr: -312> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient: _finish_hunting 0 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr: -311> 2024-03-09T14:52:29.970+0000 7f6bff906640 1 monclient: found mon.noname-c 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr: -310> 2024-03-09T14:52:29.970+0000 7f6bff906640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr: -309> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr: -308> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr: -307> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: dump: 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.595 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr: -306> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: _finish_auth 0 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr: -305> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: _check_auth_tickets 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr: -304> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr: -303> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr: -302> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr: -301> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 10 monclient: dump: 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.596 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: -300> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 monclient: authenticate success, global_id 6557 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: -299> 2024-03-09T14:52:29.974+0000 7f6c00908640 4 set_mon_vals no callback set 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: -298> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: _renew_subs 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: -297> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: -296> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: _renew_subs 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: -295> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.1:3301/0 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: -294> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: -293> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: -292> 2024-03-09T14:52:29.974+0000 7f6ae57fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: -291> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: get_monmap_and_config 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: -290> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: build_initial_monmap 2024-03-09T14:52:30.597 INFO:tasks.workunit.client.0.smithi001.stderr: -289> 2024-03-09T14:52:29.974+0000 7f6c01dee980 1 build_initial for_mkfs: 0 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr: -288> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: monmap: 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr: -287> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding auth protocol: cephx 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr: -286> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding auth protocol: cephx 2024-03-09T14:52:30.598 INFO:tasks.workunit.client.0.smithi001.stderr: -285> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding auth protocol: cephx 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -284> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -283> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -282> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -281> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -280> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -279> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -278> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -277> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -276> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -275> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -274> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -273> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -272> 2024-03-09T14:52:29.974+0000 7f6bff105640 10 monclient: get_auth_request con 0x7f6ad8065f50 auth_method 0 2024-03-09T14:52:30.599 INFO:tasks.workunit.client.0.smithi001.stderr: -271> 2024-03-09T14:52:29.974+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -270> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: init 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -269> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding auth protocol: cephx 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -268> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding auth protocol: cephx 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -267> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding auth protocol: cephx 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -266> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: secure 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -265> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: crc 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -264> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: secure 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -263> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: crc 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -262> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: secure 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -261> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: crc 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -260> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: crc 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -259> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: secure 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -258> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: crc 2024-03-09T14:52:30.600 INFO:tasks.workunit.client.0.smithi001.stderr: -257> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: secure 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -256> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: crc 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -255> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aa9c0) adding con mode: secure 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -254> 2024-03-09T14:52:29.974+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -253> 2024-03-09T14:52:29.974+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -252> 2024-03-09T14:52:29.974+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command rotate-key cmddesc rotate-key hook 0x7ffc990aab08 EEXIST 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -251> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: _reopen_session rank -1 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -250> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient: _add_conns ranks=[0,1,2] 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -249> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-a con 0x561843ecd750 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -248> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-b con 0x561843e9e640 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -247> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-c con 0x561843ec9110 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -246> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -245> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -244> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.601 INFO:tasks.workunit.client.0.smithi001.stderr: -243> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): _renew_subs 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -242> 2024-03-09T14:52:29.974+0000 7f6c01dee980 10 monclient(hunting): authenticate will time out at 2443.723556s 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -241> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): get_auth_request con 0x561843ecd750 auth_method 0 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -240> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -239> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -238> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -237> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): get_auth_request con 0x561843ec9110 auth_method 0 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -236> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -235> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -234> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -233> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -232> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -231> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -230> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.602 INFO:tasks.workunit.client.0.smithi001.stderr: -229> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr: -228> 2024-03-09T14:52:29.974+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr: -227> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient(hunting): handle_auth_done global_id 6999 payload 274 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr: -226> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient: _finish_hunting 0 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr: -225> 2024-03-09T14:52:29.974+0000 7f6bff906640 1 monclient: found mon.noname-a 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr: -224> 2024-03-09T14:52:29.974+0000 7f6bff906640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr: -223> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr: -222> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr: -221> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: dump: 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.603 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr: -220> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: _finish_auth 0 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr: -219> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: _check_auth_tickets 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr: -218> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 monclient: authenticate success, global_id 6999 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr: -217> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr: -216> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: get_monmap_and_config success 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr: -215> 2024-03-09T14:52:29.978+0000 7f6c01dee980 4 set_mon_vals no callback set 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr: -214> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr: -213> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr: -212> 2024-03-09T14:52:29.978+0000 7f6adeffd640 10 monclient: dump: 2024-03-09T14:52:30.604 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr: -211> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: shutdown 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr: -210> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: build_initial_monmap 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr: -209> 2024-03-09T14:52:29.978+0000 7f6c01dee980 1 build_initial for_mkfs: 0 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr: -208> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: monmap: 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 0 2024-03-09T14:52:30.605 INFO:tasks.workunit.client.0.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 0.000000 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr:created 0.000000 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.noname-a 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.noname-b 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.noname-c 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr: -207> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding auth protocol: cephx 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr: -206> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding auth protocol: cephx 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr: -205> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding auth protocol: cephx 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr: -204> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr: -203> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.606 INFO:tasks.workunit.client.0.smithi001.stderr: -202> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -201> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -200> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -199> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -198> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -197> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -196> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -195> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -194> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: crc 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -193> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98e90) adding con mode: secure 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -192> 2024-03-09T14:52:29.978+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -191> 2024-03-09T14:52:29.978+0000 7f6c01dee980 1 librados: starting msgr at 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -190> 2024-03-09T14:52:29.978+0000 7f6c01dee980 1 librados: starting objecter 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -189> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command objecter_requests cmddesc objecter_requests hook 0x561843e9b2d0 EEXIST 2024-03-09T14:52:30.607 INFO:tasks.workunit.client.0.smithi001.stderr: -188> 2024-03-09T14:52:29.978+0000 7f6c01dee980 1 librados: setting wanted keys 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -187> 2024-03-09T14:52:29.978+0000 7f6c01dee980 1 librados: calling monclient init 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -186> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: init 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -185> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding auth protocol: cephx 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -184> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding auth protocol: cephx 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -183> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding auth protocol: cephx 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -182> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: secure 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -181> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: crc 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -180> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: secure 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -179> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: crc 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -178> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: secure 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -177> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: crc 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -176> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: crc 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -175> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: secure 2024-03-09T14:52:30.608 INFO:tasks.workunit.client.0.smithi001.stderr: -174> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: crc 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -173> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: secure 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -172> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: crc 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -171> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 AuthRegistry(0x561843e98060) adding con mode: secure 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -170> 2024-03-09T14:52:29.978+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -169> 2024-03-09T14:52:29.978+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -168> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command rotate-key cmddesc rotate-key hook 0x561843e981a8 EEXIST 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -167> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: _reopen_session rank -1 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -166> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: _add_conns ranks=[0,1,2] 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -165> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-a con 0x561843e9e640 addr [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -164> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-b con 0x561843ec9110 addr [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -163> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): picked mon.noname-c con 0x561843ecd750 addr [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -162> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.609 INFO:tasks.workunit.client.0.smithi001.stderr: -161> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -160> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): start opening mon connection 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -159> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): _renew_subs 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -158> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient(hunting): authenticate will time out at 2443.726078s 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -157> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): get_auth_request con 0x561843e9e640 auth_method 0 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -156> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -155> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -154> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -153> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): get_auth_request con 0x561843ecd750 auth_method 0 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -152> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -151> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): _init_auth method 2 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -150> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): _init_auth creating new auth 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -149> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -148> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -147> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.610 INFO:tasks.workunit.client.0.smithi001.stderr: -146> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr: -145> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr: -144> 2024-03-09T14:52:29.978+0000 7f6c00107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr: -143> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient(hunting): handle_auth_done global_id 7002 payload 824 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr: -142> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient: _finish_hunting 0 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr: -141> 2024-03-09T14:52:29.978+0000 7f6bff906640 1 monclient: found mon.noname-a 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr: -140> 2024-03-09T14:52:29.978+0000 7f6bff906640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.1:3300/0 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr: -139> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr: -138> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr: -137> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: dump: 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.611 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr: -136> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: _finish_auth 0 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr: -135> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: _check_auth_tickets 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr: -134> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr: -133> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr: -132> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr: -131> 2024-03-09T14:52:29.978+0000 7f6add7fa640 10 monclient: dump: 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr:epoch 1 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr:fsid fe441b42-34a9-46d6-87e9-ac75455181ab 2024-03-09T14:52:30.612 INFO:tasks.workunit.client.0.smithi001.stderr:last_changed 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr:created 2024-03-09T14:27:24.720043+0000 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr:min_mon_release 19 (squid) 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr:election_strategy: 1 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr:2: [v2:172.21.15.1:3301/0,v1:172.21.15.1:6790/0] mon.c 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr: -130> 2024-03-09T14:52:29.978+0000 7f6adeffd640 4 set_mon_vals no callback set 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr: -129> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 monclient: authenticate success, global_id 7002 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr: -128> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: _renew_subs 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr: -127> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr: -126> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: _renew_subs 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr: -125> 2024-03-09T14:52:29.978+0000 7f6c01dee980 10 monclient: _send_mon_message to mon.a at v2:172.21.15.1:3300/0 2024-03-09T14:52:30.613 INFO:tasks.workunit.client.0.smithi001.stderr: -124> 2024-03-09T14:52:29.978+0000 7f6c01dee980 1 librados: init done 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -123> 2024-03-09T14:52:29.978+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command cr dump hook 0x561843ec7e38 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -122> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -121> 2024-03-09T14:52:29.982+0000 7f6add7fa640 4 mgrc handle_mgr_map Got map version 5 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -120> 2024-03-09T14:52:29.982+0000 7f6add7fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -119> 2024-03-09T14:52:29.982+0000 7f6add7fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.53:6832/1277640141,v1:172.21.15.53:6833/1277640141] 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -118> 2024-03-09T14:52:29.982+0000 7f6bff105640 10 monclient: get_auth_request con 0x7f6ac8066040 auth_method 0 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -117> 2024-03-09T14:52:29.982+0000 7f6c00107640 10 monclient: get_auth_request con 0x561843ee6be0 auth_method 0 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -116> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -115> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 realm 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -114> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -113> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -112> 2024-03-09T14:52:29.982+0000 7f6c01dee980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -111> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.614 INFO:tasks.workunit.client.0.smithi001.stderr: -110> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -109> 2024-03-09T14:52:29.982+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -108> 2024-03-09T14:52:29.982+0000 7f6bff906640 10 monclient: get_auth_request con 0x561843eeae10 auth_method 0 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -107> 2024-03-09T14:52:29.986+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -106> 2024-03-09T14:52:29.986+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -105> 2024-03-09T14:52:29.986+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -104> 2024-03-09T14:52:29.986+0000 7f6c01dee980 20 searching for the correct realm 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -103> 2024-03-09T14:52:29.986+0000 7f6bff105640 10 monclient: get_auth_request con 0x561843ef0050 auth_method 0 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -102> 2024-03-09T14:52:29.990+0000 7f6c00107640 10 monclient: get_auth_request con 0x7f6be8003100 auth_method 0 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -101> 2024-03-09T14:52:29.990+0000 7f6bff906640 10 monclient: get_auth_request con 0x7f6be8003be0 auth_method 0 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -100> 2024-03-09T14:52:29.990+0000 7f6bff105640 10 monclient: get_auth_request con 0x7f6bf0123040 auth_method 0 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -99> 2024-03-09T14:52:29.990+0000 7f6bff105640 0 -- 172.21.15.1:0/173291078 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f6bf0123040 msgr2=0x7f6bf010e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -98> 2024-03-09T14:52:30.194+0000 7f6bff105640 10 monclient: get_auth_request con 0x7f6bf0123040 auth_method 0 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -97> 2024-03-09T14:52:30.198+0000 7f6c00107640 10 monclient: get_auth_request con 0x7f6bf407c810 auth_method 0 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -96> 2024-03-09T14:52:30.206+0000 7f6bff906640 10 monclient: get_auth_request con 0x7f6bf40736d0 auth_method 0 2024-03-09T14:52:30.615 INFO:tasks.workunit.client.0.smithi001.stderr: -95> 2024-03-09T14:52:30.206+0000 7f6bff906640 0 -- 172.21.15.1:0/173291078 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f6bf40736d0 msgr2=0x7f6bf4052600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -94> 2024-03-09T14:52:30.410+0000 7f6bff906640 10 monclient: get_auth_request con 0x7f6bf40736d0 auth_method 0 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -93> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -92> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -91> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -90> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -89> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -88> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -87> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -86> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -85> 2024-03-09T14:52:30.422+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -84> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -83> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -82> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:52:30.616 INFO:tasks.workunit.client.0.smithi001.stderr: -81> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 zone default found 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -80> 2024-03-09T14:52:30.426+0000 7f6c01dee980 4 Realm: () 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -79> 2024-03-09T14:52:30.426+0000 7f6c01dee980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -78> 2024-03-09T14:52:30.426+0000 7f6c01dee980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -77> 2024-03-09T14:52:30.426+0000 7f6c01dee980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -76> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 zonegroup default 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -75> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -74> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -73> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -72> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -71> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -70> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -69> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 started sync module instance, tier type = 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -68> 2024-03-09T14:52:30.426+0000 7f6c01dee980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -67> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=0 2024-03-09T14:52:30.617 INFO:tasks.workunit.client.0.smithi001.stderr: -66> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=1 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -65> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=2 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -64> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=3 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -63> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=4 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -62> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=5 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -61> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=6 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -60> 2024-03-09T14:52:30.454+0000 7f6c01dee980 20 add_watcher() i=7 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -59> 2024-03-09T14:52:30.454+0000 7f6c01dee980 2 all 8 watchers are set, enabling cache 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -58> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding auth protocol: cephx 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -57> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding auth protocol: cephx 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -56> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding auth protocol: cephx 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -55> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: secure 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -54> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: crc 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -53> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: secure 2024-03-09T14:52:30.618 INFO:tasks.workunit.client.0.smithi001.stderr: -52> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: crc 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -51> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: secure 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -50> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: crc 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -49> 2024-03-09T14:52:30.458+0000 7f6ab5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -48> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: crc 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -47> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: secure 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -46> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: crc 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -45> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: secure 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -44> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: crc 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -43> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 AuthRegistry(0x7ffc990aae50) adding con mode: secure 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -42> 2024-03-09T14:52:30.458+0000 7f6c01dee980 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -41> 2024-03-09T14:52:30.458+0000 7f6c01dee980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -40> 2024-03-09T14:52:30.458+0000 7f6c01dee980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -39> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command cache list hook 0x561843ebf9f0 2024-03-09T14:52:30.619 INFO:tasks.workunit.client.0.smithi001.stderr: -38> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command cache inspect hook 0x561843ebf9f0 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -37> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command cache erase hook 0x561843ebf9f0 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -36> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command cache zap hook 0x561843ebf9f0 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -35> 2024-03-09T14:52:30.458+0000 7f6c01dee980 5 note: GC not initialized 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -34> 2024-03-09T14:52:30.462+0000 7f6ab4fe1640 20 reqs_thread_entry: start 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -33> 2024-03-09T14:52:30.462+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command sync trace show hook 0x7f6bf0001df0 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -32> 2024-03-09T14:52:30.462+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command sync trace history hook 0x7f6bf0001df0 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -31> 2024-03-09T14:52:30.462+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command sync trace active hook 0x7f6bf0001df0 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -30> 2024-03-09T14:52:30.462+0000 7f6c01dee980 5 asok(0x561843beebb0) register_command sync trace active_short hook 0x7f6bf0001df0 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -29> 2024-03-09T14:52:30.462+0000 7f6c01dee980 20 init_complete bucket index max shards: 1 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -28> 2024-03-09T14:52:30.462+0000 7f6c01dee980 20 Filter name: none 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -27> 2024-03-09T14:52:30.462+0000 7f6a2e7fc640 20 reqs_thread_entry: start 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -26> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 cache get: name=default.rgw.meta+root+abort-at-do-reshard : miss 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -25> 2024-03-09T14:52:30.462+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -24> 2024-03-09T14:52:30.462+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:30.620 INFO:tasks.workunit.client.0.smithi001.stderr: -23> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 cache put: name=default.rgw.meta+root+abort-at-do-reshard info.flags=0x11 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -22> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 adding default.rgw.meta+root+abort-at-do-reshard to cache LRU end 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -21> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 : miss 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -20> 2024-03-09T14:52:30.462+0000 7f6c01dee980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -19> 2024-03-09T14:52:30.462+0000 7f6c01dee980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -18> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -17> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -16> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -15> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -14> 2024-03-09T14:52:30.462+0000 7f6c01dee980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -13> 2024-03-09T14:52:30.470+0000 7f6c01dee980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -12> 2024-03-09T14:52:30.470+0000 7f6c01dee980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -11> 2024-03-09T14:52:30.470+0000 7f6c01dee980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -10> 2024-03-09T14:52:30.470+0000 7f6c01dee980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28, ofs0, ns] 2024-03-09T14:52:30.621 INFO:tasks.workunit.client.0.smithi001.stderr: -9> 2024-03-09T14:52:30.470+0000 7f6adeffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562429 cookie 94662218958480 notifier 7002 bl.length()=903 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: -8> 2024-03-09T14:52:30.470+0000 7f6adeffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: -7> 2024-03-09T14:52:30.470+0000 7f6adeffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: -6> 2024-03-09T14:52:30.470+0000 7f6adeffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: -5> 2024-03-09T14:52:30.474+0000 7f6c01dee980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): orig_sources= new_sources= 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: -4> 2024-03-09T14:52:30.474+0000 7f6c01dee980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): potential sources added=[] removed=[] 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: -3> 2024-03-09T14:52:30.474+0000 7f6c01dee980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): orig_dests= new_dests= 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: -2> 2024-03-09T14:52:30.474+0000 7f6c01dee980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): potential dests added=[] removed=[] 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: -1> 2024-03-09T14:52:30.478+0000 7f6c01dee980 -1 ./src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f6c01dee980 time 2024-03-09T14:52:30.481885+0000 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr:./src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f6c056c3d7b] 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f6c056c3f2f] 2024-03-09T14:52:30.622 INFO:tasks.workunit.client.0.smithi001.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5618425f2c51] 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x315) [0x5618426132f5] 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: 5: main() 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6c0452ed90] 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: 7: __libc_start_main() 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: 8: _start() 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: 0> 2024-03-09T14:52:30.482+0000 7f6c01dee980 -1 *** Caught signal (Aborted) ** 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: in thread 7f6c01dee980 thread_name:radosgw-admin 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: ceph version 19.0.0-2079-gc00d9c60 (c00d9c609fb42a2a25f1c196bed064ca8bb02413) squid (dev) 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f6c04547520] 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: 2: pthread_kill() 2024-03-09T14:52:30.623 INFO:tasks.workunit.client.0.smithi001.stderr: 3: raise() 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: 4: abort() 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6c056c3dd5] 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f6c056c3f2f] 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5618425f2c51] 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x315) [0x5618426132f5] 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: 9: main() 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6c0452ed90] 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: 11: __libc_start_main() 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: 12: _start() 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr:--- logging levels --- 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 none 2024-03-09T14:52:30.624 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 lockdep 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 context 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 crush 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_balancer 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_locker 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_log_expire 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mds_migrator 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 buffer 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 timer 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 filer 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 striper 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 1 objecter 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rados 2024-03-09T14:52:30.625 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_mirror 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_replay 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 rbd_pwl 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 journaler 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objectcacher 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 immutable_obj_cache 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 client 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 osd 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 optracker 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 objclass 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 filestore 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 journal 2024-03-09T14:52:30.626 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 ms 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mon 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 0/10 monc 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 paxos 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 tp 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 auth 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 crypto 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 finisher 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 reserver 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 heartbeatmap 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 perfcounter 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 20/20 rgw 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_sync 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_datacache 2024-03-09T14:52:30.627 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_access 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_dbstore 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_flight 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 rgw_lifecycle 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 javaclient 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 asok 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 1 throttle 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 0 refs 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 compressor 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluestore 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 bluefs 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 3 bdev 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 kstore 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 4/ 5 rocksdb 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 fuse 2024-03-09T14:52:30.628 INFO:tasks.workunit.client.0.smithi001.stderr: 2/ 5 mgr 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mgrc 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 dpdk 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 eventtrace 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 prioritycache 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 test 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephfs_mirror 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cephsqlite 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 crimson_interrupt 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_onode 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_odata 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_omap 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_tm 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_t 2024-03-09T14:52:30.629 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cleaner 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_epm 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_lba 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_fixedkv_tree 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_cache 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_journal 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_device 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 seastore_backref 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 alienstore 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 mclock 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 0/ 5 cyanstore 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 ceph_exporter 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 memstore 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: 1/ 5 trace 2024-03-09T14:52:30.630 INFO:tasks.workunit.client.0.smithi001.stderr: -2/-2 (syslog threshold) 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 99/99 (stderr threshold) 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr:--- pthread ID / name mapping for recent threads --- 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6a2e7fc640 / http_manager 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6ab4fe1640 / http_manager 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6ab5fe3640 / rgw_dt_lg_renew 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6add7fa640 / ms_dispatch 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6adeffd640 / io_context_pool 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6ae57fa640 / ms_dispatch 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6ae7fff640 / ms_dispatch 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6aede0b640 / io_context_pool 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6bff105640 / msgr-worker-2 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6bff906640 / msgr-worker-1 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6c00107640 / msgr-worker-0 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6c00908640 / io_context_pool 2024-03-09T14:52:30.631 INFO:tasks.workunit.client.0.smithi001.stderr: 7f6c01dee980 / radosgw-admin 2024-03-09T14:52:30.632 INFO:tasks.workunit.client.0.smithi001.stderr: max_recent 500 2024-03-09T14:52:30.632 INFO:tasks.workunit.client.0.smithi001.stderr: max_new 1000 2024-03-09T14:52:30.632 INFO:tasks.workunit.client.0.smithi001.stderr: log_file /var/lib/ceph/crash/2024-03-09T14:52:30.487295Z_8996beb6-88e1-45d9-a0a0-fff104cd3674/log 2024-03-09T14:52:30.632 INFO:tasks.workunit.client.0.smithi001.stderr:--- end dump of recent events --- 2024-03-09T14:52:30.632 INFO:tasks.workunit.client.0.smithi001.stderr:Aborted 2024-03-09T14:52:30.632 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket abort-at-do-reshard 2024-03-09T14:52:30.636 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:52:30.636 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:52:30.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.570+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.574+0000 7f6b32e2c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.574+0000 7f6b32e2c980 20 realm 2024-03-09T14:52:30.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.574+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.574+0000 7f6b32e2c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.574+0000 7f6b32e2c980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.574+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.574+0000 7f6b32e2c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.574+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.574+0000 7f6b32e2c980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.574+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.574+0000 7f6b32e2c980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.574+0000 7f6b32e2c980 20 searching for the correct realm 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.582+0000 7f6b30143640 0 -- 172.21.15.1:0/3269126420 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f6b200d3040 msgr2=0x7f6b200d3930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.590+0000 7f6b30143640 0 -- 172.21.15.1:0/3269126420 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f69fc075de0 msgr2=0x7f69fc078210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.594+0000 7f6b32e2c980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.594+0000 7f6b32e2c980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.594+0000 7f6b32e2c980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.594+0000 7f6b32e2c980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:52:30.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.594+0000 7f6b32e2c980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.594+0000 7f6b32e2c980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.594+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 zone default found 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 4 Realm: () 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 zonegroup default 2024-03-09T14:52:30.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 started sync module instance, tier type = 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.598+0000 7f6b32e2c980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.614+0000 7f6b32e2c980 20 add_watcher() i=0 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.614+0000 7f6b32e2c980 20 add_watcher() i=1 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.614+0000 7f6b32e2c980 20 add_watcher() i=2 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.614+0000 7f6b32e2c980 20 add_watcher() i=3 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.614+0000 7f6b32e2c980 20 add_watcher() i=4 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.614+0000 7f6b32e2c980 20 add_watcher() i=5 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.614+0000 7f6b32e2c980 20 add_watcher() i=6 2024-03-09T14:52:30.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.618+0000 7f6b32e2c980 20 add_watcher() i=7 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.618+0000 7f6b32e2c980 2 all 8 watchers are set, enabling cache 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.618+0000 7f69e6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.618+0000 7f6b32e2c980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.618+0000 7f6b32e2c980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.618+0000 7f6b32e2c980 5 note: GC not initialized 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.618+0000 7f69e5fe3640 20 reqs_thread_entry: start 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.618+0000 7f6b32e2c980 20 init_complete bucket index max shards: 1 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.618+0000 7f6b32e2c980 20 Filter name: none 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.618+0000 7f69577fe640 20 reqs_thread_entry: start 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.618+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.622+0000 7f6b32e2c980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.622+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.622+0000 7f6b32e2c980 20 rados_obj.operate() r=0 bl.length=433 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.622+0000 7f6b32e2c980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:52:30.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.622+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.622+0000 7f6b32e2c980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:52:30.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.622+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.622+0000 7f6b32e2c980 20 rados_obj.operate() r=0 bl.length=433 2024-03-09T14:52:30.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.622+0000 7f6b32e2c980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:52:30.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.622+0000 7f6b32e2c980 20 rados->read ofs=0 len=0 2024-03-09T14:52:30.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.622+0000 7f6b32e2c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:52:30.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.626+0000 7f69e6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:52:30.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.630+0000 7f6b32e2c980 20 remove_watcher() i=0 2024-03-09T14:52:30.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.630+0000 7f6b32e2c980 2 removed watcher, disabling cache 2024-03-09T14:52:30.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.630+0000 7f6b32e2c980 20 remove_watcher() i=1 2024-03-09T14:52:30.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.634+0000 7f6b32e2c980 20 remove_watcher() i=2 2024-03-09T14:52:30.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.638+0000 7f6b32e2c980 20 remove_watcher() i=3 2024-03-09T14:52:30.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.638+0000 7f6b32e2c980 20 remove_watcher() i=4 2024-03-09T14:52:30.644 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.642+0000 7f6b32e2c980 20 remove_watcher() i=5 2024-03-09T14:52:30.645 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.642+0000 7f6b32e2c980 20 remove_watcher() i=6 2024-03-09T14:52:30.646 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:52:30.642+0000 7f6b32e2c980 20 remove_watcher() i=7 2024-03-09T14:52:30.660 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.660 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:52:30.660 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: abort-at-do-reshard id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 num_objects: 20 size_kb: 1 num_shards: 1 2024-03-09T14:52:30.660 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:52:30.662 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:52:30.662 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:00.848 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:00.848 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:00.938 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket abort-at-do-reshard --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2024-03-09T14:53:00.976 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:00.976 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:01.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.021+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.025+0000 7f15db1f6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.025+0000 7f15db1f6980 20 realm 2024-03-09T14:53:01.025 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.025+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.025+0000 7f15db1f6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.025+0000 7f15db1f6980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:01.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.025+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.025+0000 7f15db1f6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.025+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.025+0000 7f15db1f6980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:01.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.025+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.029+0000 7f15db1f6980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:01.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.029+0000 7f15db1f6980 20 searching for the correct realm 2024-03-09T14:53:01.031 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.029+0000 7f15d8d0e640 0 -- 172.21.15.1:0/3576431168 >> [v2:172.21.15.1:6824/1666056338,v1:172.21.15.1:6825/1666056338] conn(0x55d7d18be9b0 msgr2=0x7f15d00bb3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:01.244 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.241+0000 7f15d950f640 0 -- 172.21.15.1:0/3576431168 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f15cc03e070 msgr2=0x7f15cc058430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:01.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:01.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:01.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:01.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:01.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:01.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:01.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:01.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:01.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 zone default found 2024-03-09T14:53:01.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 4 Realm: () 2024-03-09T14:53:01.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:01.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:01.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:01.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 zonegroup default 2024-03-09T14:53:01.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.264 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.264 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.261+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.264 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.265+0000 7f15db1f6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.264 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.265+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.265+0000 7f15db1f6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.265+0000 7f15db1f6980 20 started sync module instance, tier type = 2024-03-09T14:53:01.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.265+0000 7f15db1f6980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:01.289 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.289+0000 7f15db1f6980 20 add_watcher() i=0 2024-03-09T14:53:01.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.289+0000 7f15db1f6980 20 add_watcher() i=1 2024-03-09T14:53:01.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.289+0000 7f15db1f6980 20 add_watcher() i=2 2024-03-09T14:53:01.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.289+0000 7f15db1f6980 20 add_watcher() i=3 2024-03-09T14:53:01.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.289+0000 7f15db1f6980 20 add_watcher() i=4 2024-03-09T14:53:01.290 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.289+0000 7f15db1f6980 20 add_watcher() i=5 2024-03-09T14:53:01.291 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.289+0000 7f15db1f6980 20 add_watcher() i=6 2024-03-09T14:53:01.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.293+0000 7f15db1f6980 20 add_watcher() i=7 2024-03-09T14:53:01.292 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.293+0000 7f15db1f6980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:01.296 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f148efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:01.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f15db1f6980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:01.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f15db1f6980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:01.297 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f15db1f6980 5 note: GC not initialized 2024-03-09T14:53:01.298 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f148dfe3640 20 reqs_thread_entry: start 2024-03-09T14:53:01.299 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f15db1f6980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:01.299 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f15db1f6980 20 Filter name: none 2024-03-09T14:53:01.299 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f13ff7fe640 20 reqs_thread_entry: start 2024-03-09T14:53:01.299 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f15db1f6980 10 cache get: name=default.rgw.meta+root+abort-at-do-reshard : miss 2024-03-09T14:53:01.299 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f15db1f6980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:53:01.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f15db1f6980 10 cache put: name=default.rgw.meta+root+abort-at-do-reshard info.flags=0x11 2024-03-09T14:53:01.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f15db1f6980 10 adding default.rgw.meta+root+abort-at-do-reshard to cache LRU end 2024-03-09T14:53:01.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f15db1f6980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 : miss 2024-03-09T14:53:01.300 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.297+0000 7f15db1f6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.301+0000 7f15db1f6980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:53:01.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.301+0000 7f15db1f6980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:53:01.301 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.301+0000 7f15db1f6980 10 adding default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:53:01.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.301+0000 7f15db1f6980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:53:01.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.301+0000 7f15db1f6980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:53:01.302 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.301+0000 7f15db1f6980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 2024-03-09T14:53:01.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.305+0000 7f15db1f6980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:53:01.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.305+0000 7f15db1f6980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:53:01.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.305+0000 7f15db1f6980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:53:01.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.305+0000 7f15db1f6980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28, ofs0, ns] 2024-03-09T14:53:01.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.305+0000 7f14affff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562431 cookie 94385424322608 notifier 6590 bl.length()=903 2024-03-09T14:53:01.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.305+0000 7f14affff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:53:01.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.305+0000 7f14affff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:53:01.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.305+0000 7f14affff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:53:01.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.309+0000 7f15db1f6980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): orig_sources= new_sources= 2024-03-09T14:53:01.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.309+0000 7f15db1f6980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): potential sources added=[] removed=[] 2024-03-09T14:53:01.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.309+0000 7f15db1f6980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): orig_dests= new_dests= 2024-03-09T14:53:01.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.309+0000 7f15db1f6980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): potential dests added=[] removed=[] 2024-03-09T14:53:01.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.313+0000 7f15db1f6980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.28.1.0 2024-03-09T14:53:01.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.313+0000 7f15db1f6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.28.1.0 2024-03-09T14:53:01.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.313+0000 7f15db1f6980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.28.1.1 2024-03-09T14:53:01.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.313+0000 7f15db1f6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.28.1.1 2024-03-09T14:53:01.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.313+0000 7f15db1f6980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.28.0 2024-03-09T14:53:01.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.313+0000 7f15db1f6980 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 : hit (requested=0x17, cached=0x17) 2024-03-09T14:53:01.315 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.313+0000 7f15db1f6980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 2024-03-09T14:53:01.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.317+0000 7f15db1f6980 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:53:01.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.317+0000 7f15db1f6980 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:53:01.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.317+0000 7f15db1f6980 10 updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:53:01.317 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.317+0000 7f15db1f6980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28, ofs0, ns] 2024-03-09T14:53:01.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.317+0000 7f14af7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562432 cookie 94385424322608 notifier 6590 bl.length()=871 2024-03-09T14:53:01.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.317+0000 7f14af7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 info.flags=0x17 2024-03-09T14:53:01.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.317+0000 7f14af7fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 to cache LRU end 2024-03-09T14:53:01.319 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.317+0000 7f14af7fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2024-03-09T14:53:01.321 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.321+0000 7f15db1f6980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): orig_sources= new_sources= 2024-03-09T14:53:01.321 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.321+0000 7f15db1f6980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): potential sources added=[] removed=[] 2024-03-09T14:53:01.321 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.321+0000 7f15db1f6980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): orig_dests= new_dests= 2024-03-09T14:53:01.321 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.321+0000 7f15db1f6980 20 handle_bi_update(): bucket=:abort-at-do-reshard[8c209108-e1cd-4d45-9608-6f491550b24c.4644.28]): potential dests added=[] removed=[] 2024-03-09T14:53:01.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.321+0000 7f15db1f6980 1 execute INFO: reshard of bucket "abort-at-do-reshard" completed successfully 2024-03-09T14:53:01.325 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.325+0000 7f148efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:01.331 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.329+0000 7f15db1f6980 20 remove_watcher() i=0 2024-03-09T14:53:01.331 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.329+0000 7f15db1f6980 2 removed watcher, disabling cache 2024-03-09T14:53:01.333 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.333+0000 7f15db1f6980 20 remove_watcher() i=1 2024-03-09T14:53:01.335 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.333+0000 7f15db1f6980 20 remove_watcher() i=2 2024-03-09T14:53:01.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.337+0000 7f15db1f6980 20 remove_watcher() i=3 2024-03-09T14:53:01.340 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.341+0000 7f15db1f6980 20 remove_watcher() i=4 2024-03-09T14:53:01.342 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.341+0000 7f15db1f6980 20 remove_watcher() i=5 2024-03-09T14:53:01.344 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.345+0000 7f15db1f6980 20 remove_watcher() i=6 2024-03-09T14:53:01.345 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.345+0000 7f15db1f6980 20 remove_watcher() i=7 2024-03-09T14:53:01.370 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket abort-at-do-reshard 2024-03-09T14:53:01.397 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:01.397 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:01.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.441+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.445+0000 7fdb04c7e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.445+0000 7fdb04c7e980 20 realm 2024-03-09T14:53:01.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.445+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.445+0000 7fdb04c7e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.445+0000 7fdb04c7e980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:01.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.445+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.445+0000 7fdb04c7e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.445+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.445+0000 7fdb04c7e980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:01.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.445+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.445+0000 7fdb04c7e980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:01.448 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.449+0000 7fdb04c7e980 20 searching for the correct realm 2024-03-09T14:53:01.455 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.453+0000 7fdb02f97640 0 -- 172.21.15.1:0/4016815950 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7fdaf8003020 msgr2=0x7fdaf8018910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:53:01.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.669+0000 7fdb02796640 0 -- 172.21.15.1:0/4016815950 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x56237a7338a0 msgr2=0x56237a735d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:01.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.677+0000 7fdb04c7e980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:01.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.677+0000 7fdb04c7e980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:01.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.677+0000 7fdb04c7e980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:01.678 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.677+0000 7fdb04c7e980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:01.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.677+0000 7fdb04c7e980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:01.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.677+0000 7fdb04c7e980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:01.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.677+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.677+0000 7fdb04c7e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.679 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.677+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.677+0000 7fdb04c7e980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:01.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.677+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:01.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 20 zone default found 2024-03-09T14:53:01.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 4 Realm: () 2024-03-09T14:53:01.681 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:01.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:01.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:01.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 20 zonegroup default 2024-03-09T14:53:01.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 20 started sync module instance, tier type = 2024-03-09T14:53:01.683 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.681+0000 7fdb04c7e980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:01.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.705+0000 7fdb04c7e980 20 add_watcher() i=0 2024-03-09T14:53:01.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.705+0000 7fdb04c7e980 20 add_watcher() i=1 2024-03-09T14:53:01.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.705+0000 7fdb04c7e980 20 add_watcher() i=2 2024-03-09T14:53:01.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.705+0000 7fdb04c7e980 20 add_watcher() i=3 2024-03-09T14:53:01.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.705+0000 7fdb04c7e980 20 add_watcher() i=4 2024-03-09T14:53:01.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.705+0000 7fdb04c7e980 20 add_watcher() i=5 2024-03-09T14:53:01.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.705+0000 7fdb04c7e980 20 add_watcher() i=6 2024-03-09T14:53:01.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.709+0000 7fdb04c7e980 20 add_watcher() i=7 2024-03-09T14:53:01.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.709+0000 7fdb04c7e980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:01.714 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.713+0000 7fd9b8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:01.715 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.713+0000 7fdb04c7e980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:01.715 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.713+0000 7fdb04c7e980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:01.715 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.713+0000 7fdb04c7e980 5 note: GC not initialized 2024-03-09T14:53:01.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.713+0000 7fd9337fe640 20 reqs_thread_entry: start 2024-03-09T14:53:01.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.713+0000 7fdb04c7e980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:01.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.717+0000 7fdb04c7e980 20 Filter name: none 2024-03-09T14:53:01.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.717+0000 7fd9317fa640 20 reqs_thread_entry: start 2024-03-09T14:53:01.716 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.717+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.717+0000 7fdb04c7e980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:53:01.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.717+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.718 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.717+0000 7fdb04c7e980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:53:01.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.717+0000 7fdb04c7e980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:01.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.717+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.717+0000 7fdb04c7e980 20 rados_obj.operate() r=0 bl.length=187 2024-03-09T14:53:01.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.717+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.719 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.717+0000 7fdb04c7e980 20 rados_obj.operate() r=0 bl.length=401 2024-03-09T14:53:01.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.717+0000 7fdb04c7e980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:01.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.721+0000 7fdb04c7e980 20 rados->read ofs=0 len=0 2024-03-09T14:53:01.721 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.721+0000 7fdb04c7e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:01.723 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.721+0000 7fd9b8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:01.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.729+0000 7fdb04c7e980 20 remove_watcher() i=0 2024-03-09T14:53:01.728 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.729+0000 7fdb04c7e980 2 removed watcher, disabling cache 2024-03-09T14:53:01.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.729+0000 7fdb04c7e980 20 remove_watcher() i=1 2024-03-09T14:53:01.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.733+0000 7fdb04c7e980 20 remove_watcher() i=2 2024-03-09T14:53:01.734 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.733+0000 7fdb04c7e980 20 remove_watcher() i=3 2024-03-09T14:53:01.736 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.737+0000 7fdb04c7e980 20 remove_watcher() i=4 2024-03-09T14:53:01.738 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.737+0000 7fdb04c7e980 20 remove_watcher() i=5 2024-03-09T14:53:01.740 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.737+0000 7fdb04c7e980 20 remove_watcher() i=6 2024-03-09T14:53:01.741 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:01.741+0000 7fdb04c7e980 20 remove_watcher() i=7 2024-03-09T14:53:01.758 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:53:01.758 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:53:01.758 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: abort-at-do-reshard id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.28 num_objects: 19 size_kb: 1 num_shards: 2 2024-03-09T14:53:01.759 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:53:01.761 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:01.761 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:01.790 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:01.790 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:02.377 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:02.377 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:02.558 INFO:tasks.workunit.client.0.smithi001.stderr: test: reshard versioned bucket 2024-03-09T14:53:02.559 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket myver 2024-03-09T14:53:02.591 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:02.591 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:02.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.637+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.637+0000 7fdb9ac68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:02.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.637+0000 7fdb9ac68980 20 realm 2024-03-09T14:53:02.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.637+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.637+0000 7fdb9ac68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:02.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.637+0000 7fdb9ac68980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:02.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.637+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.641+0000 7fdb9ac68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:02.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.641+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.641+0000 7fdb9ac68980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:02.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.641+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.641+0000 7fdb9ac68980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:02.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.641+0000 7fdb9ac68980 20 searching for the correct realm 2024-03-09T14:53:02.647 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.645+0000 7fdb97f7f640 0 -- 172.21.15.1:0/1995899401 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fdb880c4040 msgr2=0x7fdb880b8ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:02.656 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.653+0000 7fdb98780640 0 -- 172.21.15.1:0/1995899401 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fdb8c052f90 msgr2=0x7fdb8c06c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:53:02.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.869+0000 7fdb9ac68980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:02.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.869+0000 7fdb9ac68980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:02.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.869+0000 7fdb9ac68980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:02.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.869+0000 7fdb9ac68980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:02.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.869+0000 7fdb9ac68980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:02.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.869+0000 7fdb9ac68980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:02.872 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.869+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:02.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:02.873 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.874 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:02.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 20 zone default found 2024-03-09T14:53:02.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 4 Realm: () 2024-03-09T14:53:02.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:02.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:02.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:02.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 20 zonegroup default 2024-03-09T14:53:02.875 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.873+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.877+0000 7fdb9ac68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:02.876 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.877+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.877 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.877+0000 7fdb9ac68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:02.877 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.877+0000 7fdb9ac68980 20 started sync module instance, tier type = 2024-03-09T14:53:02.878 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.877+0000 7fdb9ac68980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:02.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.901+0000 7fdb9ac68980 20 add_watcher() i=0 2024-03-09T14:53:02.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.901+0000 7fdb9ac68980 20 add_watcher() i=1 2024-03-09T14:53:02.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.901+0000 7fdb9ac68980 20 add_watcher() i=2 2024-03-09T14:53:02.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.901+0000 7fdb9ac68980 20 add_watcher() i=3 2024-03-09T14:53:02.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.901+0000 7fdb9ac68980 20 add_watcher() i=4 2024-03-09T14:53:02.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.901+0000 7fdb9ac68980 20 add_watcher() i=5 2024-03-09T14:53:02.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.901+0000 7fdb9ac68980 20 add_watcher() i=6 2024-03-09T14:53:02.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.901+0000 7fdb9ac68980 20 add_watcher() i=7 2024-03-09T14:53:02.903 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.901+0000 7fdb9ac68980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:02.907 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.905+0000 7fda4efdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:02.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fdb9ac68980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:02.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fdb9ac68980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:02.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fdb9ac68980 5 note: GC not initialized 2024-03-09T14:53:02.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fda4dfdb640 20 reqs_thread_entry: start 2024-03-09T14:53:02.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fdb9ac68980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:02.909 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fdb9ac68980 20 Filter name: none 2024-03-09T14:53:02.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fd9bf7fe640 20 reqs_thread_entry: start 2024-03-09T14:53:02.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.910 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fdb9ac68980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:02.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fdb9ac68980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:02.911 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fdb9ac68980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:02.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.909+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.913+0000 7fdb9ac68980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:02.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.913+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.913+0000 7fdb9ac68980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:02.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.913+0000 7fdb9ac68980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:02.913 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.913+0000 7fdb9ac68980 20 rados->read ofs=0 len=0 2024-03-09T14:53:02.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.913+0000 7fdb9ac68980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:02.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.913+0000 7fda4efdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:02.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.921+0000 7fdb9ac68980 20 remove_watcher() i=0 2024-03-09T14:53:02.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.921+0000 7fdb9ac68980 2 removed watcher, disabling cache 2024-03-09T14:53:02.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.921+0000 7fdb9ac68980 20 remove_watcher() i=1 2024-03-09T14:53:02.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.925+0000 7fdb9ac68980 20 remove_watcher() i=2 2024-03-09T14:53:02.929 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.929+0000 7fdb9ac68980 20 remove_watcher() i=3 2024-03-09T14:53:02.932 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.929+0000 7fdb9ac68980 20 remove_watcher() i=4 2024-03-09T14:53:02.933 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.933+0000 7fdb9ac68980 20 remove_watcher() i=5 2024-03-09T14:53:02.935 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.933+0000 7fdb9ac68980 20 remove_watcher() i=6 2024-03-09T14:53:02.936 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:02.937+0000 7fdb9ac68980 20 remove_watcher() i=7 2024-03-09T14:53:02.953 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:53:02.953 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:53:02.953 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: myver id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 num_objects: 0 size_kb: 0 num_shards: 1 2024-03-09T14:53:02.953 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:53:02.954 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket myver --num-shards 2 2024-03-09T14:53:02.987 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:02.988 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:03.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.033+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.033+0000 7f44c61ee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.033+0000 7f44c61ee980 20 realm 2024-03-09T14:53:03.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.033+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.033+0000 7f44c61ee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.033+0000 7f44c61ee980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:03.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.033+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.033+0000 7f44c61ee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.033+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.036 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.033+0000 7f44c61ee980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:03.036 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.033+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.036 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.037+0000 7f44c61ee980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:03.036 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.037+0000 7f44c61ee980 20 searching for the correct realm 2024-03-09T14:53:03.043 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.041+0000 7f44c3505640 0 -- 172.21.15.1:0/1767083470 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x564c8225de00 msgr2=0x564c822602c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:03.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.049+0000 7f44c3d06640 0 -- 172.21.15.1:0/1767083470 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f44b80752d0 msgr2=0x7f44b8021860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:03.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:03.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:03.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:03.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:03.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:03.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:03.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.061 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:03.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:03.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 zone default found 2024-03-09T14:53:03.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 4 Realm: () 2024-03-09T14:53:03.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:03.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:03.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:03.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 zonegroup default 2024-03-09T14:53:03.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.062 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.057+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.061+0000 7f44c61ee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.061+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.061+0000 7f44c61ee980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.061+0000 7f44c61ee980 20 started sync module instance, tier type = 2024-03-09T14:53:03.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.061+0000 7f44c61ee980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:03.076 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.077+0000 7f44c61ee980 20 add_watcher() i=0 2024-03-09T14:53:03.076 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.077+0000 7f44c61ee980 20 add_watcher() i=1 2024-03-09T14:53:03.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.077+0000 7f44c61ee980 20 add_watcher() i=2 2024-03-09T14:53:03.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.077+0000 7f44c61ee980 20 add_watcher() i=3 2024-03-09T14:53:03.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.077+0000 7f44c61ee980 20 add_watcher() i=4 2024-03-09T14:53:03.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.077+0000 7f44c61ee980 20 add_watcher() i=5 2024-03-09T14:53:03.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.077+0000 7f44c61ee980 20 add_watcher() i=6 2024-03-09T14:53:03.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.077+0000 7f44c61ee980 20 add_watcher() i=7 2024-03-09T14:53:03.077 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.077+0000 7f44c61ee980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:03.080 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f4379fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:03.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:03.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:03.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 5 note: GC not initialized 2024-03-09T14:53:03.081 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f4378fe1640 20 reqs_thread_entry: start 2024-03-09T14:53:03.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:03.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 20 Filter name: none 2024-03-09T14:53:03.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f42f27fc640 20 reqs_thread_entry: start 2024-03-09T14:53:03.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 10 cache get: name=default.rgw.meta+root+myver : miss 2024-03-09T14:53:03.082 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:03.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 10 cache put: name=default.rgw.meta+root+myver info.flags=0x11 2024-03-09T14:53:03.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 10 adding default.rgw.meta+root+myver to cache LRU end 2024-03-09T14:53:03.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 : miss 2024-03-09T14:53:03.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:03.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:03.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 10 adding default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:03.083 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:53:03.084 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:03.084 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.081+0000 7f44c61ee980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 2024-03-09T14:53:03.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.089+0000 7f44c61ee980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:03.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.089+0000 7f44c61ee980 10 moving default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:03.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.089+0000 7f44c61ee980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:03.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.089+0000 7f44c61ee980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18, ofs0, ns] 2024-03-09T14:53:03.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.089+0000 7f43a2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562417 cookie 94886601051472 notifier 6644 bl.length()=813 2024-03-09T14:53:03.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.089+0000 7f43a2ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:03.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.089+0000 7f43a2ffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:03.091 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.089+0000 7f43a2ffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:03.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.093+0000 7f44c61ee980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): orig_sources= new_sources= 2024-03-09T14:53:03.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.093+0000 7f44c61ee980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): potential sources added=[] removed=[] 2024-03-09T14:53:03.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.093+0000 7f44c61ee980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): orig_dests= new_dests= 2024-03-09T14:53:03.093 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.093+0000 7f44c61ee980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): potential dests added=[] removed=[] 2024-03-09T14:53:03.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.093+0000 7f44c61ee980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.1.0 2024-03-09T14:53:03.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.093+0000 7f44c61ee980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.1.0 2024-03-09T14:53:03.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.093+0000 7f44c61ee980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.1.1 2024-03-09T14:53:03.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.093+0000 7f44c61ee980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.1.1 2024-03-09T14:53:03.096 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.093+0000 7f44c61ee980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.0 2024-03-09T14:53:03.097 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.097+0000 7f44c61ee980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 : hit (requested=0x17, cached=0x17) 2024-03-09T14:53:03.097 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.097+0000 7f44c61ee980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 2024-03-09T14:53:03.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.097+0000 7f44c61ee980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:03.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.097+0000 7f44c61ee980 10 moving default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:03.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.097+0000 7f44c61ee980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:03.098 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.097+0000 7f44c61ee980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18, ofs0, ns] 2024-03-09T14:53:03.099 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.097+0000 7f43a27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562418 cookie 94886601051472 notifier 6644 bl.length()=781 2024-03-09T14:53:03.099 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.097+0000 7f43a27fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:03.099 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.097+0000 7f43a27fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:03.099 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.097+0000 7f43a27fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:03.100 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.101+0000 7f44c61ee980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): orig_sources= new_sources= 2024-03-09T14:53:03.100 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.101+0000 7f44c61ee980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): potential sources added=[] removed=[] 2024-03-09T14:53:03.100 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.101+0000 7f44c61ee980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): orig_dests= new_dests= 2024-03-09T14:53:03.100 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.101+0000 7f44c61ee980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): potential dests added=[] removed=[] 2024-03-09T14:53:03.103 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.101+0000 7f44c61ee980 1 execute INFO: reshard of bucket "myver" completed successfully 2024-03-09T14:53:03.107 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.105+0000 7f4379fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:03.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.109+0000 7f44c61ee980 20 remove_watcher() i=0 2024-03-09T14:53:03.112 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.109+0000 7f44c61ee980 2 removed watcher, disabling cache 2024-03-09T14:53:03.114 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.113+0000 7f44c61ee980 20 remove_watcher() i=1 2024-03-09T14:53:03.116 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.117+0000 7f44c61ee980 20 remove_watcher() i=2 2024-03-09T14:53:03.118 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.117+0000 7f44c61ee980 20 remove_watcher() i=3 2024-03-09T14:53:03.121 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.121+0000 7f44c61ee980 20 remove_watcher() i=4 2024-03-09T14:53:03.122 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.121+0000 7f44c61ee980 20 remove_watcher() i=5 2024-03-09T14:53:03.123 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.121+0000 7f44c61ee980 20 remove_watcher() i=6 2024-03-09T14:53:03.124 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.125+0000 7f44c61ee980 20 remove_watcher() i=7 2024-03-09T14:53:03.139 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket myver 2024-03-09T14:53:03.165 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:03.165 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:03.208 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.205+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.209+0000 7fbaa76a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.209+0000 7fbaa76a6980 20 realm 2024-03-09T14:53:03.209 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.209+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.209+0000 7fbaa76a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.209+0000 7fbaa76a6980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:03.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.209+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.209+0000 7fbaa76a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.210 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.209+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.209+0000 7fbaa76a6980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:03.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.209+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.213+0000 7fbaa76a6980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:03.212 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.213+0000 7fbaa76a6980 20 searching for the correct realm 2024-03-09T14:53:03.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.217+0000 7fbaa59bf640 0 -- 172.21.15.1:0/591290267 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7fba9c018070 msgr2=0x7fba9c07a8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:03.224 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.225+0000 7fbaa51be640 0 -- 172.21.15.1:0/591290267 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fba9c053490 msgr2=0x7fba9c077120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:03.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.441+0000 7fbaa76a6980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:03.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.441+0000 7fbaa76a6980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:03.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.441+0000 7fbaa76a6980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:03.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.441+0000 7fbaa76a6980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:03.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.441+0000 7fbaa76a6980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:03.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.441+0000 7fbaa76a6980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:03.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.441+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.443 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.441+0000 7fbaa76a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.441+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:03.444 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:03.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 zone default found 2024-03-09T14:53:03.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 4 Realm: () 2024-03-09T14:53:03.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:03.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:03.445 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:03.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 zonegroup default 2024-03-09T14:53:03.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.446 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 started sync module instance, tier type = 2024-03-09T14:53:03.447 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.445+0000 7fbaa76a6980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:03.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.469+0000 7fbaa76a6980 20 add_watcher() i=0 2024-03-09T14:53:03.470 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.469+0000 7fbaa76a6980 20 add_watcher() i=1 2024-03-09T14:53:03.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.469+0000 7fbaa76a6980 20 add_watcher() i=2 2024-03-09T14:53:03.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.469+0000 7fbaa76a6980 20 add_watcher() i=3 2024-03-09T14:53:03.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.469+0000 7fbaa76a6980 20 add_watcher() i=4 2024-03-09T14:53:03.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.469+0000 7fbaa76a6980 20 add_watcher() i=5 2024-03-09T14:53:03.471 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.469+0000 7fbaa76a6980 20 add_watcher() i=6 2024-03-09T14:53:03.472 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.473+0000 7fbaa76a6980 20 add_watcher() i=7 2024-03-09T14:53:03.473 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.473+0000 7fbaa76a6980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:03.477 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.477+0000 7fb95bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:03.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.477+0000 7fbaa76a6980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:03.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.477+0000 7fbaa76a6980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:03.478 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.477+0000 7fbaa76a6980 5 note: GC not initialized 2024-03-09T14:53:03.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.477+0000 7fb95afe5640 20 reqs_thread_entry: start 2024-03-09T14:53:03.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.477+0000 7fbaa76a6980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:03.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.477+0000 7fbaa76a6980 20 Filter name: none 2024-03-09T14:53:03.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.477+0000 7fb958fe1640 20 reqs_thread_entry: start 2024-03-09T14:53:03.479 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.477+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.480 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.481+0000 7fbaa76a6980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:03.480 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.481+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.481+0000 7fbaa76a6980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:03.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.481+0000 7fbaa76a6980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:03.481 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.481+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.481+0000 7fbaa76a6980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:03.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.481+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.481+0000 7fbaa76a6980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:03.482 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.481+0000 7fbaa76a6980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:03.483 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.481+0000 7fbaa76a6980 20 rados->read ofs=0 len=0 2024-03-09T14:53:03.484 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.485+0000 7fbaa76a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:03.486 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.485+0000 7fb95bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:03.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.489+0000 7fbaa76a6980 20 remove_watcher() i=0 2024-03-09T14:53:03.491 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.489+0000 7fbaa76a6980 2 removed watcher, disabling cache 2024-03-09T14:53:03.494 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.493+0000 7fbaa76a6980 20 remove_watcher() i=1 2024-03-09T14:53:03.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.497+0000 7fbaa76a6980 20 remove_watcher() i=2 2024-03-09T14:53:03.499 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.497+0000 7fbaa76a6980 20 remove_watcher() i=3 2024-03-09T14:53:03.501 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.501+0000 7fbaa76a6980 20 remove_watcher() i=4 2024-03-09T14:53:03.503 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.501+0000 7fbaa76a6980 20 remove_watcher() i=5 2024-03-09T14:53:03.505 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.505+0000 7fbaa76a6980 20 remove_watcher() i=6 2024-03-09T14:53:03.507 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:03.505+0000 7fbaa76a6980 20 remove_watcher() i=7 2024-03-09T14:53:03.531 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:53:03.531 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:53:03.531 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: myver id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 num_objects: 0 size_kb: 0 num_shards: 2 2024-03-09T14:53:03.531 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:53:03.535 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:03.535 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:03.565 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:03.565 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:03.653 INFO:tasks.workunit.client.0.smithi001.stderr: test: reshard removes olh entries with empty name 2024-03-09T14:53:03.658 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:03.658 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:03.786 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:03.786 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:04.092 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket a-bucket 2024-03-09T14:53:04.126 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:04.126 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:04.173 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.173+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.173+0000 7f561bf2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.173+0000 7f561bf2f980 20 realm 2024-03-09T14:53:04.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.173+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.173+0000 7f561bf2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.173+0000 7f561bf2f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:04.175 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.173+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.173+0000 7f561bf2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.176 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.173+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.177+0000 7f561bf2f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:04.177 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.177+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.177+0000 7f561bf2f980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:04.178 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.177+0000 7f561bf2f980 20 searching for the correct realm 2024-03-09T14:53:04.184 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.185+0000 7f561a248640 0 -- 172.21.15.1:0/4136876941 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x563e58b77520 msgr2=0x563e58b79990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:53:04.191 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.189+0000 7f5619a47640 0 -- 172.21.15.1:0/4136876941 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f5604113870 msgr2=0x7f56040aa3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:04.199 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.197+0000 7f561bf2f980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:04.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.197+0000 7f561bf2f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:04.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.197+0000 7f561bf2f980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:04.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.197+0000 7f561bf2f980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:04.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.197+0000 7f561bf2f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:04.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.197+0000 7f561bf2f980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:04.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.197+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.197+0000 7f561bf2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.197+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:04.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:04.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 zone default found 2024-03-09T14:53:04.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 4 Realm: () 2024-03-09T14:53:04.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:04.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:04.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:04.201 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 zonegroup default 2024-03-09T14:53:04.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 started sync module instance, tier type = 2024-03-09T14:53:04.203 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.201+0000 7f561bf2f980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:04.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.217+0000 7f561bf2f980 20 add_watcher() i=0 2024-03-09T14:53:04.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.217+0000 7f561bf2f980 20 add_watcher() i=1 2024-03-09T14:53:04.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.217+0000 7f561bf2f980 20 add_watcher() i=2 2024-03-09T14:53:04.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.217+0000 7f561bf2f980 20 add_watcher() i=3 2024-03-09T14:53:04.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.217+0000 7f561bf2f980 20 add_watcher() i=4 2024-03-09T14:53:04.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.217+0000 7f561bf2f980 20 add_watcher() i=5 2024-03-09T14:53:04.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.217+0000 7f561bf2f980 20 add_watcher() i=6 2024-03-09T14:53:04.219 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.217+0000 7f561bf2f980 20 add_watcher() i=7 2024-03-09T14:53:04.220 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.217+0000 7f561bf2f980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:04.222 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.221+0000 7f54cb7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:04.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.221+0000 7f561bf2f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:04.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.221+0000 7f561bf2f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:04.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.221+0000 7f561bf2f980 5 note: GC not initialized 2024-03-09T14:53:04.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.221+0000 7f54ca7fc640 20 reqs_thread_entry: start 2024-03-09T14:53:04.223 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.221+0000 7f561bf2f980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:04.224 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.221+0000 7f561bf2f980 20 Filter name: none 2024-03-09T14:53:04.224 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.221+0000 7f543bfff640 20 reqs_thread_entry: start 2024-03-09T14:53:04.224 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.221+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.224 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.225+0000 7f561bf2f980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:53:04.224 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.225+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.225 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.225+0000 7f561bf2f980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:53:04.225 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.225+0000 7f561bf2f980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:04.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.225+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.225+0000 7f561bf2f980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:53:04.226 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.225+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.229+0000 7f561bf2f980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:53:04.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.229+0000 7f561bf2f980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:04.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.229+0000 7f561bf2f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.229+0000 7f561bf2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.231 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.229+0000 7f54cb7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:04.236 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.233+0000 7f561bf2f980 20 remove_watcher() i=0 2024-03-09T14:53:04.236 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.233+0000 7f561bf2f980 2 removed watcher, disabling cache 2024-03-09T14:53:04.238 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.237+0000 7f561bf2f980 20 remove_watcher() i=1 2024-03-09T14:53:04.240 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.241+0000 7f561bf2f980 20 remove_watcher() i=2 2024-03-09T14:53:04.243 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.241+0000 7f561bf2f980 20 remove_watcher() i=3 2024-03-09T14:53:04.245 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.245+0000 7f561bf2f980 20 remove_watcher() i=4 2024-03-09T14:53:04.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.245+0000 7f561bf2f980 20 remove_watcher() i=5 2024-03-09T14:53:04.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.245+0000 7f561bf2f980 20 remove_watcher() i=6 2024-03-09T14:53:04.249 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.249+0000 7f561bf2f980 20 remove_watcher() i=7 2024-03-09T14:53:04.263 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:53:04.263 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:53:04.263 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: a-bucket id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 num_objects: 0 size_kb: 0 num_shards: 3 2024-03-09T14:53:04.263 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:53:04.263 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket layout --bucket a-bucket 2024-03-09T14:53:04.289 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:04.289 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:04.334 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.333+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.333+0000 7f384e1d4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.333+0000 7f384e1d4980 20 realm 2024-03-09T14:53:04.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.333+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.337+0000 7f384e1d4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.337+0000 7f384e1d4980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:04.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.337+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.337+0000 7f384e1d4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.336 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.337+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.337+0000 7f384e1d4980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:04.338 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.337+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.337+0000 7f384e1d4980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:04.339 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.337+0000 7f384e1d4980 20 searching for the correct realm 2024-03-09T14:53:04.348 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.345+0000 7f384c4ed640 0 -- 172.21.15.1:0/2630306203 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55677ecd8a70 msgr2=0x55677ecdaf10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:04.351 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.349+0000 7f384bcec640 0 -- 172.21.15.1:0/2630306203 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f384010e1b0 msgr2=0x7f384010e610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:04.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.357+0000 7f384e1d4980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:04.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.357+0000 7f384e1d4980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:04.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.357+0000 7f384e1d4980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:04.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.357+0000 7f384e1d4980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:04.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.357+0000 7f384e1d4980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:04.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.357+0000 7f384e1d4980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:04.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.357+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.360 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.357+0000 7f384e1d4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.357+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.361 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:04.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:04.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 zone default found 2024-03-09T14:53:04.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 4 Realm: () 2024-03-09T14:53:04.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:04.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:04.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:04.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 zonegroup default 2024-03-09T14:53:04.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.362 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 started sync module instance, tier type = 2024-03-09T14:53:04.363 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.361+0000 7f384e1d4980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:04.379 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.377+0000 7f384e1d4980 20 add_watcher() i=0 2024-03-09T14:53:04.379 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.377+0000 7f384e1d4980 20 add_watcher() i=1 2024-03-09T14:53:04.379 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.377+0000 7f384e1d4980 20 add_watcher() i=2 2024-03-09T14:53:04.379 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.377+0000 7f384e1d4980 20 add_watcher() i=3 2024-03-09T14:53:04.379 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.377+0000 7f384e1d4980 20 add_watcher() i=4 2024-03-09T14:53:04.379 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.377+0000 7f384e1d4980 20 add_watcher() i=5 2024-03-09T14:53:04.379 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.377+0000 7f384e1d4980 20 add_watcher() i=6 2024-03-09T14:53:04.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.377+0000 7f384e1d4980 20 add_watcher() i=7 2024-03-09T14:53:04.380 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.377+0000 7f384e1d4980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:04.382 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.381+0000 7f37027e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:04.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.381+0000 7f384e1d4980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:04.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.381+0000 7f384e1d4980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:04.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.381+0000 7f384e1d4980 5 note: GC not initialized 2024-03-09T14:53:04.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.381+0000 7f37017e2640 20 reqs_thread_entry: start 2024-03-09T14:53:04.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.381+0000 7f384e1d4980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:04.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.381+0000 7f384e1d4980 20 Filter name: none 2024-03-09T14:53:04.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.381+0000 7f36fa7fc640 20 reqs_thread_entry: start 2024-03-09T14:53:04.383 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.381+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.385+0000 7f384e1d4980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:53:04.384 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.385+0000 7f384e1d4980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.385+0000 7f384e1d4980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:53:04.385 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.385+0000 7f384e1d4980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:04.386 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.385+0000 7f37027e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:04.391 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.389+0000 7f384e1d4980 20 remove_watcher() i=0 2024-03-09T14:53:04.391 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.389+0000 7f384e1d4980 2 removed watcher, disabling cache 2024-03-09T14:53:04.393 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.393+0000 7f384e1d4980 20 remove_watcher() i=1 2024-03-09T14:53:04.395 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.393+0000 7f384e1d4980 20 remove_watcher() i=2 2024-03-09T14:53:04.397 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.397+0000 7f384e1d4980 20 remove_watcher() i=3 2024-03-09T14:53:04.399 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.397+0000 7f384e1d4980 20 remove_watcher() i=4 2024-03-09T14:53:04.401 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.401+0000 7f384e1d4980 20 remove_watcher() i=5 2024-03-09T14:53:04.402 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.401+0000 7f384e1d4980 20 remove_watcher() i=6 2024-03-09T14:53:04.403 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.401+0000 7f384e1d4980 20 remove_watcher() i=7 2024-03-09T14:53:04.417 INFO:tasks.workunit.client.0.smithi001.stderr:rados -p default.rgw.buckets.index setomapval .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.0 --omap-key-file /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/olh_noname_key < /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/olh_noname_val 2024-03-09T14:53:04.443 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:04.475 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:04.476 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket a-bucket 2024-03-09T14:53:04.486 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:04.486 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:04.527 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.525+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.529+0000 7f50fc23f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.529+0000 7f50fc23f980 20 realm 2024-03-09T14:53:04.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.529+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.529+0000 7f50fc23f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.529+0000 7f50fc23f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:04.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.529+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.529+0000 7f50fc23f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.529+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.529+0000 7f50fc23f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:04.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.529+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.533+0000 7f50fc23f980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:04.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.533+0000 7f50fc23f980 20 searching for the correct realm 2024-03-09T14:53:04.538 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.537+0000 7f50fa558640 0 -- 172.21.15.1:0/2660198874 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x564fef165d80 msgr2=0x564fef1681d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:04.545 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.545+0000 7f50f9556640 0 -- 172.21.15.1:0/2660198874 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f50ec0ce040 msgr2=0x7f50ec0e9ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:04.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.549+0000 7f50fc23f980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:04.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.549+0000 7f50fc23f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:04.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.549+0000 7f50fc23f980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:04.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.549+0000 7f50fc23f980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:04.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.549+0000 7f50fc23f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:04.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.549+0000 7f50fc23f980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:04.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.549+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.552 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:04.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:04.553 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 zone default found 2024-03-09T14:53:04.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 4 Realm: () 2024-03-09T14:53:04.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:04.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:04.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:04.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 zonegroup default 2024-03-09T14:53:04.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 started sync module instance, tier type = 2024-03-09T14:53:04.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.553+0000 7f50fc23f980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:04.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.569+0000 7f50fc23f980 20 add_watcher() i=0 2024-03-09T14:53:04.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.569+0000 7f50fc23f980 20 add_watcher() i=1 2024-03-09T14:53:04.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.569+0000 7f50fc23f980 20 add_watcher() i=2 2024-03-09T14:53:04.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.569+0000 7f50fc23f980 20 add_watcher() i=3 2024-03-09T14:53:04.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.569+0000 7f50fc23f980 20 add_watcher() i=4 2024-03-09T14:53:04.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.569+0000 7f50fc23f980 20 add_watcher() i=5 2024-03-09T14:53:04.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.569+0000 7f50fc23f980 20 add_watcher() i=6 2024-03-09T14:53:04.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.569+0000 7f50fc23f980 20 add_watcher() i=7 2024-03-09T14:53:04.571 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.569+0000 7f50fc23f980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:04.574 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.573+0000 7f4fab7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:04.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.573+0000 7f50fc23f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:04.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.573+0000 7f50fc23f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:04.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.573+0000 7f50fc23f980 5 note: GC not initialized 2024-03-09T14:53:04.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.573+0000 7f4faa7fc640 20 reqs_thread_entry: start 2024-03-09T14:53:04.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.573+0000 7f50fc23f980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:04.575 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.573+0000 7f50fc23f980 20 Filter name: none 2024-03-09T14:53:04.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.573+0000 7f4f1bfff640 20 reqs_thread_entry: start 2024-03-09T14:53:04.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.573+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.573+0000 7f50fc23f980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:53:04.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.573+0000 7f50fc23f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:53:04.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:04.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 20 INFO: main: max_entries=1000, index=gen=2, max_shards=3 2024-03-09T14:53:04.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 20 INFO: main: starting shard=0 2024-03-09T14:53:04.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.0 2024-03-09T14:53:04.577 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.0 2024-03-09T14:53:04.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2024-03-09T14:53:04.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 20 INFO: main: starting shard=1 2024-03-09T14:53:04.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.1 2024-03-09T14:53:04.578 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.1 2024-03-09T14:53:04.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2024-03-09T14:53:04.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 20 INFO: main: starting shard=2 2024-03-09T14:53:04.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.2 2024-03-09T14:53:04.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.577+0000 7f50fc23f980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.2 2024-03-09T14:53:04.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.581+0000 7f50fc23f980 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2024-03-09T14:53:04.581 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.581+0000 7f50fc23f980 20 INFO: main: done 2024-03-09T14:53:04.581 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.581+0000 7f4fab7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:04.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.585+0000 7f50fc23f980 20 remove_watcher() i=0 2024-03-09T14:53:04.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.585+0000 7f50fc23f980 2 removed watcher, disabling cache 2024-03-09T14:53:04.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.589+0000 7f50fc23f980 20 remove_watcher() i=1 2024-03-09T14:53:04.590 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.589+0000 7f50fc23f980 20 remove_watcher() i=2 2024-03-09T14:53:04.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.593+0000 7f50fc23f980 20 remove_watcher() i=3 2024-03-09T14:53:04.594 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.593+0000 7f50fc23f980 20 remove_watcher() i=4 2024-03-09T14:53:04.595 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.593+0000 7f50fc23f980 20 remove_watcher() i=5 2024-03-09T14:53:04.597 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.597+0000 7f50fc23f980 20 remove_watcher() i=6 2024-03-09T14:53:04.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.597+0000 7f50fc23f980 20 remove_watcher() i=7 2024-03-09T14:53:04.611 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket a-bucket --num-shards 1 --yes-i-really-mean-it 2024-03-09T14:53:04.641 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:04.641 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:04.684 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.681+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.685+0000 7f5ee3451980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.685+0000 7f5ee3451980 20 realm 2024-03-09T14:53:04.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.685+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.685+0000 7f5ee3451980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.685+0000 7f5ee3451980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:04.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.685+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.685+0000 7f5ee3451980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.686 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.685+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.685+0000 7f5ee3451980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:04.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.685+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.687 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.685+0000 7f5ee3451980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:04.688 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.685+0000 7f5ee3451980 20 searching for the correct realm 2024-03-09T14:53:04.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.693+0000 7f5ee0768640 0 -- 172.21.15.1:0/3150752335 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f5ed00f5120 msgr2=0x7f5ed0109160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:04.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.697+0000 7f5ee0f69640 0 -- 172.21.15.1:0/3150752335 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f5ed4050920 msgr2=0x7f5ed403e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:04.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:04.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:04.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:04.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:04.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:04.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:04.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.918 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:04.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.919 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:04.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 zone default found 2024-03-09T14:53:04.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 4 Realm: () 2024-03-09T14:53:04.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:04.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:04.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:04.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 zonegroup default 2024-03-09T14:53:04.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.917+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.920 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.921+0000 7f5ee3451980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.921+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.921+0000 7f5ee3451980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.921 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.921+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.921+0000 7f5ee3451980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:04.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.921+0000 7f5ee3451980 20 started sync module instance, tier type = 2024-03-09T14:53:04.922 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.921+0000 7f5ee3451980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:04.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.945+0000 7f5ee3451980 20 add_watcher() i=0 2024-03-09T14:53:04.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.945+0000 7f5ee3451980 20 add_watcher() i=1 2024-03-09T14:53:04.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.945+0000 7f5ee3451980 20 add_watcher() i=2 2024-03-09T14:53:04.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.945+0000 7f5ee3451980 20 add_watcher() i=3 2024-03-09T14:53:04.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.945+0000 7f5ee3451980 20 add_watcher() i=4 2024-03-09T14:53:04.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.945+0000 7f5ee3451980 20 add_watcher() i=5 2024-03-09T14:53:04.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.945+0000 7f5ee3451980 20 add_watcher() i=6 2024-03-09T14:53:04.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.945+0000 7f5ee3451980 20 add_watcher() i=7 2024-03-09T14:53:04.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.945+0000 7f5ee3451980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:04.952 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.949+0000 7f5d96fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:04.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5ee3451980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:04.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5ee3451980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:04.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5ee3451980 5 note: GC not initialized 2024-03-09T14:53:04.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5d95fe3640 20 reqs_thread_entry: start 2024-03-09T14:53:04.953 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5ee3451980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:04.954 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5ee3451980 20 Filter name: none 2024-03-09T14:53:04.954 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5d077fe640 20 reqs_thread_entry: start 2024-03-09T14:53:04.954 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5ee3451980 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2024-03-09T14:53:04.954 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5ee3451980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:53:04.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5ee3451980 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x11 2024-03-09T14:53:04.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5ee3451980 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2024-03-09T14:53:04.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5ee3451980 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 : miss 2024-03-09T14:53:04.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.953+0000 7f5ee3451980 20 rados->read ofs=0 len=0 2024-03-09T14:53:04.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.957+0000 7f5ee3451980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:53:04.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.957+0000 7f5ee3451980 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:53:04.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.957+0000 7f5ee3451980 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:53:04.956 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.957+0000 7f5ee3451980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:53:04.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.957+0000 7f5ee3451980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:04.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.957+0000 7f5ee3451980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 2024-03-09T14:53:04.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.961+0000 7f5ee3451980 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:53:04.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.961+0000 7f5ee3451980 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:53:04.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.961+0000 7f5ee3451980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:04.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.961+0000 7f5ee3451980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17, ofs0, ns] 2024-03-09T14:53:04.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.961+0000 7f5db77fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562419 cookie 93847706414496 notifier 7206 bl.length()=819 2024-03-09T14:53:04.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.961+0000 7f5db77fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:53:04.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.961+0000 7f5db77fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:53:04.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.961+0000 7f5db77fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:04.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.965+0000 7f5ee3451980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): orig_sources= new_sources= 2024-03-09T14:53:04.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.965+0000 7f5ee3451980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): potential sources added=[] removed=[] 2024-03-09T14:53:04.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.965+0000 7f5ee3451980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): orig_dests= new_dests= 2024-03-09T14:53:04.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.965+0000 7f5ee3451980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): potential dests added=[] removed=[] 2024-03-09T14:53:04.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.969+0000 7f5ee3451980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.3.0 2024-03-09T14:53:04.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.969+0000 7f5ee3451980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.3.0 2024-03-09T14:53:04.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.969+0000 7f5ee3451980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.0 2024-03-09T14:53:04.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.969+0000 7f5ee3451980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.0 2024-03-09T14:53:04.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.969+0000 7f5ee3451980 10 Dropping entry with empty name, idx=�1001_04/57/0457f727ec113e418d5b16d206b200ed068c0533554883ce811df7c932a3df68/2018_12_11/2889999/3386469/metadata.gz 2024-03-09T14:53:04.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.969+0000 7f5ee3451980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.1 2024-03-09T14:53:04.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.969+0000 7f5ee3451980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.1 2024-03-09T14:53:04.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.969+0000 7f5ee3451980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.2 2024-03-09T14:53:04.971 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.969+0000 7f5ee3451980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.2.2 2024-03-09T14:53:04.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.973+0000 7f5ee3451980 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 : hit (requested=0x17, cached=0x17) 2024-03-09T14:53:04.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.973+0000 7f5ee3451980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 2024-03-09T14:53:04.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.973+0000 7f5ee3451980 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:53:04.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.973+0000 7f5ee3451980 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:53:04.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.973+0000 7f5ee3451980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:04.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.973+0000 7f5ee3451980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17, ofs0, ns] 2024-03-09T14:53:04.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.977+0000 7f5db7fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562420 cookie 93847706414496 notifier 7206 bl.length()=787 2024-03-09T14:53:04.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.977+0000 7f5db7fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 info.flags=0x17 2024-03-09T14:53:04.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.977+0000 7f5db7fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17 to cache LRU end 2024-03-09T14:53:04.976 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.977+0000 7f5db7fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:04.978 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.977+0000 7f5ee3451980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): orig_sources= new_sources= 2024-03-09T14:53:04.978 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.977+0000 7f5ee3451980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): potential sources added=[] removed=[] 2024-03-09T14:53:04.978 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.977+0000 7f5ee3451980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): orig_dests= new_dests= 2024-03-09T14:53:04.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.977+0000 7f5ee3451980 20 handle_bi_update(): bucket=:a-bucket[8c209108-e1cd-4d45-9608-6f491550b24c.4644.17]): potential dests added=[] removed=[] 2024-03-09T14:53:04.981 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.981+0000 7f5ee3451980 1 execute INFO: reshard of bucket "a-bucket" completed successfully 2024-03-09T14:53:04.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.985+0000 7f5d96fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:04.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.989+0000 7f5ee3451980 20 remove_watcher() i=0 2024-03-09T14:53:04.990 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.989+0000 7f5ee3451980 2 removed watcher, disabling cache 2024-03-09T14:53:04.992 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.989+0000 7f5ee3451980 20 remove_watcher() i=1 2024-03-09T14:53:04.994 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.993+0000 7f5ee3451980 20 remove_watcher() i=2 2024-03-09T14:53:04.996 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.997+0000 7f5ee3451980 20 remove_watcher() i=3 2024-03-09T14:53:04.998 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:04.997+0000 7f5ee3451980 20 remove_watcher() i=4 2024-03-09T14:53:05.000 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.001+0000 7f5ee3451980 20 remove_watcher() i=5 2024-03-09T14:53:05.001 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.001+0000 7f5ee3451980 20 remove_watcher() i=6 2024-03-09T14:53:05.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.001+0000 7f5ee3451980 20 remove_watcher() i=7 2024-03-09T14:53:05.019 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket a-bucket 2024-03-09T14:53:05.050 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:05.050 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:05.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.305+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.305+0000 7f223e4de980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.305+0000 7f223e4de980 20 realm 2024-03-09T14:53:05.305 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.305+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.305+0000 7f223e4de980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.305+0000 7f223e4de980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:05.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.305+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.305+0000 7f223e4de980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.306 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.305+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.305+0000 7f223e4de980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:05.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.305+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.309+0000 7f223e4de980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:05.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.309+0000 7f223e4de980 20 searching for the correct realm 2024-03-09T14:53:05.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.309+0000 7f223c7f7640 0 -- 172.21.15.1:0/747960726 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f2230039070 msgr2=0x7f223000a450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:05.522 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.521+0000 7f223c7f7640 0 -- 172.21.15.1:0/747960726 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f2230008070 msgr2=0x7f2230055960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:05.745 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.745+0000 7f223e4de980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:05.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.745+0000 7f223e4de980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:05.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.745+0000 7f223e4de980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:05.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.745+0000 7f223e4de980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:05.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.745+0000 7f223e4de980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:05.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.745+0000 7f223e4de980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:05.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.745+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.745+0000 7f223e4de980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.746 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.745+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.745+0000 7f223e4de980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:05.747 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.745+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:05.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 20 zone default found 2024-03-09T14:53:05.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 4 Realm: () 2024-03-09T14:53:05.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:05.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:05.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:05.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 20 zonegroup default 2024-03-09T14:53:05.748 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.749 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 20 started sync module instance, tier type = 2024-03-09T14:53:05.750 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.749+0000 7f223e4de980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:05.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.773+0000 7f223e4de980 20 add_watcher() i=0 2024-03-09T14:53:05.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.773+0000 7f223e4de980 20 add_watcher() i=1 2024-03-09T14:53:05.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.773+0000 7f223e4de980 20 add_watcher() i=2 2024-03-09T14:53:05.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.773+0000 7f223e4de980 20 add_watcher() i=3 2024-03-09T14:53:05.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.773+0000 7f223e4de980 20 add_watcher() i=4 2024-03-09T14:53:05.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.773+0000 7f223e4de980 20 add_watcher() i=5 2024-03-09T14:53:05.774 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.773+0000 7f223e4de980 20 add_watcher() i=6 2024-03-09T14:53:05.776 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.777+0000 7f223e4de980 20 add_watcher() i=7 2024-03-09T14:53:05.777 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.777+0000 7f223e4de980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:05.780 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.781+0000 7f20f1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:05.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.781+0000 7f223e4de980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:05.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.781+0000 7f223e4de980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:05.781 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.781+0000 7f223e4de980 5 note: GC not initialized 2024-03-09T14:53:05.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.781+0000 7f20f0fe1640 20 reqs_thread_entry: start 2024-03-09T14:53:05.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.781+0000 7f223e4de980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:05.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.781+0000 7f223e4de980 20 Filter name: none 2024-03-09T14:53:05.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.781+0000 7f206a7fc640 20 reqs_thread_entry: start 2024-03-09T14:53:05.782 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.781+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.781+0000 7f223e4de980 20 rados_obj.operate() r=0 bl.length=176 2024-03-09T14:53:05.783 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.781+0000 7f223e4de980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.785+0000 7f223e4de980 20 rados_obj.operate() r=0 bl.length=390 2024-03-09T14:53:05.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.785+0000 7f223e4de980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:05.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.785+0000 7f223e4de980 20 INFO: main: max_entries=1000, index=gen=3, max_shards=1 2024-03-09T14:53:05.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.785+0000 7f223e4de980 20 INFO: main: starting shard=0 2024-03-09T14:53:05.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.785+0000 7f223e4de980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.3.0 2024-03-09T14:53:05.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.785+0000 7f223e4de980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.17.3.0 2024-03-09T14:53:05.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.785+0000 7f223e4de980 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2024-03-09T14:53:05.785 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.785+0000 7f223e4de980 20 INFO: main: done 2024-03-09T14:53:05.787 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.785+0000 7f20f1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:05.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.793+0000 7f223e4de980 20 remove_watcher() i=0 2024-03-09T14:53:05.792 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.793+0000 7f223e4de980 2 removed watcher, disabling cache 2024-03-09T14:53:05.795 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.793+0000 7f223e4de980 20 remove_watcher() i=1 2024-03-09T14:53:05.797 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.797+0000 7f223e4de980 20 remove_watcher() i=2 2024-03-09T14:53:05.799 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.797+0000 7f223e4de980 20 remove_watcher() i=3 2024-03-09T14:53:05.801 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.801+0000 7f223e4de980 20 remove_watcher() i=4 2024-03-09T14:53:05.803 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.801+0000 7f223e4de980 20 remove_watcher() i=5 2024-03-09T14:53:05.804 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.805+0000 7f223e4de980 20 remove_watcher() i=6 2024-03-09T14:53:05.806 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.805+0000 7f223e4de980 20 remove_watcher() i=7 2024-03-09T14:53:05.821 INFO:tasks.workunit.client.0.smithi001.stderr: test: PUT succeeds during reshard 2024-03-09T14:53:05.822 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket myver 2024-03-09T14:53:05.853 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:05.853 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:05.897 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.897+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.897+0000 7f62ec9fc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.897+0000 7f62ec9fc980 20 realm 2024-03-09T14:53:05.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.897+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.897+0000 7f62ec9fc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.897+0000 7f62ec9fc980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:05.899 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.897+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.900 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.897+0000 7f62ec9fc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.900 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.897+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.901+0000 7f62ec9fc980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:05.901 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.901+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.901+0000 7f62ec9fc980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:05.902 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.901+0000 7f62ec9fc980 20 searching for the correct realm 2024-03-09T14:53:05.908 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.909+0000 7f62ead15640 0 -- 172.21.15.1:0/1387447037 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x561bb002e740 msgr2=0x561bb0030b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:05.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.913+0000 7f62e9d13640 0 -- 172.21.15.1:0/1387447037 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f62dc01b040 msgr2=0x7f62dc038470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:05.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.921+0000 7f62ec9fc980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:05.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.921+0000 7f62ec9fc980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:05.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.921+0000 7f62ec9fc980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:05.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.921+0000 7f62ec9fc980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:05.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.921+0000 7f62ec9fc980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:05.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.921+0000 7f62ec9fc980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:05.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.921+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.921+0000 7f62ec9fc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.924 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.921+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.921+0000 7f62ec9fc980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:05.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.921+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:05.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 20 zone default found 2024-03-09T14:53:05.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 4 Realm: () 2024-03-09T14:53:05.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:05.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:05.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:05.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 20 zonegroup default 2024-03-09T14:53:05.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.925 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 20 started sync module instance, tier type = 2024-03-09T14:53:05.926 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.925+0000 7f62ec9fc980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:05.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.941+0000 7f62ec9fc980 20 add_watcher() i=0 2024-03-09T14:53:05.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.941+0000 7f62ec9fc980 20 add_watcher() i=1 2024-03-09T14:53:05.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.941+0000 7f62ec9fc980 20 add_watcher() i=2 2024-03-09T14:53:05.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.941+0000 7f62ec9fc980 20 add_watcher() i=3 2024-03-09T14:53:05.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.941+0000 7f62ec9fc980 20 add_watcher() i=4 2024-03-09T14:53:05.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.941+0000 7f62ec9fc980 20 add_watcher() i=5 2024-03-09T14:53:05.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.941+0000 7f62ec9fc980 20 add_watcher() i=6 2024-03-09T14:53:05.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.941+0000 7f62ec9fc980 20 add_watcher() i=7 2024-03-09T14:53:05.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.941+0000 7f62ec9fc980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:05.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f61a17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:05.945 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:05.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:05.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 5 note: GC not initialized 2024-03-09T14:53:05.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f619b7fe640 20 reqs_thread_entry: start 2024-03-09T14:53:05.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:05.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 Filter name: none 2024-03-09T14:53:05.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f61997fa640 20 reqs_thread_entry: start 2024-03-09T14:53:05.946 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:05.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:05.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:05.947 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:05.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:05.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.945+0000 7f62ec9fc980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:05.948 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.949+0000 7f62ec9fc980 20 rados->read ofs=0 len=0 2024-03-09T14:53:05.949 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.949+0000 7f62ec9fc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:05.950 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.949+0000 7f61a17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:05.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.953+0000 7f62ec9fc980 20 remove_watcher() i=0 2024-03-09T14:53:05.955 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.953+0000 7f62ec9fc980 2 removed watcher, disabling cache 2024-03-09T14:53:05.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.957+0000 7f62ec9fc980 20 remove_watcher() i=1 2024-03-09T14:53:05.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.957+0000 7f62ec9fc980 20 remove_watcher() i=2 2024-03-09T14:53:05.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.961+0000 7f62ec9fc980 20 remove_watcher() i=3 2024-03-09T14:53:05.963 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.961+0000 7f62ec9fc980 20 remove_watcher() i=4 2024-03-09T14:53:05.964 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.965+0000 7f62ec9fc980 20 remove_watcher() i=5 2024-03-09T14:53:05.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.965+0000 7f62ec9fc980 20 remove_watcher() i=6 2024-03-09T14:53:05.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:05.965+0000 7f62ec9fc980 20 remove_watcher() i=7 2024-03-09T14:53:05.980 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:53:05.980 INFO:tasks.workunit.client.0.smithi001.stderr:BUCKET_STATS: 2024-03-09T14:53:05.980 INFO:tasks.workunit.client.0.smithi001.stderr:bucket: myver id: 8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 num_objects: 0 size_kb: 0 num_shards: 2 2024-03-09T14:53:05.980 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:53:05.981 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin --inject-delay-at=do_reshard --inject-delay-ms=5000 bucket reshard --bucket myver --num-shards 3 2024-03-09T14:53:06.010 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:06.011 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:06.054 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.053+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.056 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.057+0000 7f150840f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:06.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.057+0000 7f150840f980 20 realm 2024-03-09T14:53:06.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.057+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.057+0000 7f150840f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:06.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.057+0000 7f150840f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:06.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.057+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.057+0000 7f150840f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:06.057 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.057+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.057+0000 7f150840f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:06.058 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.057+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.057+0000 7f150840f980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:06.059 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.057+0000 7f150840f980 20 searching for the correct realm 2024-03-09T14:53:06.063 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.061+0000 7f1505726640 0 -- 172.21.15.1:0/2823309647 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f14f804e030 msgr2=0x7f14f801f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:53:06.277 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.277+0000 7f1505f27640 0 -- 172.21.15.1:0/2823309647 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f14fc07bba0 msgr2=0x7f14fc075030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:53:06.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.493+0000 7f150840f980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:06.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.493+0000 7f150840f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:06.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.493+0000 7f150840f980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:06.495 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.493+0000 7f150840f980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:06.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.493+0000 7f150840f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:06.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.493+0000 7f150840f980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:06.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.493+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.493+0000 7f150840f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:06.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.493+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:06.496 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:06.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 20 zone default found 2024-03-09T14:53:06.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 4 Realm: () 2024-03-09T14:53:06.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:06.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:06.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:06.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 20 zonegroup default 2024-03-09T14:53:06.498 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.499 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:06.499 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.499 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:06.499 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.497+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.500 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.501+0000 7f150840f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:06.500 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.501+0000 7f150840f980 20 started sync module instance, tier type = 2024-03-09T14:53:06.500 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.501+0000 7f150840f980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:06.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.521+0000 7f150840f980 20 add_watcher() i=0 2024-03-09T14:53:06.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.521+0000 7f150840f980 20 add_watcher() i=1 2024-03-09T14:53:06.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.521+0000 7f150840f980 20 add_watcher() i=2 2024-03-09T14:53:06.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.521+0000 7f150840f980 20 add_watcher() i=3 2024-03-09T14:53:06.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.521+0000 7f150840f980 20 add_watcher() i=4 2024-03-09T14:53:06.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.521+0000 7f150840f980 20 add_watcher() i=5 2024-03-09T14:53:06.523 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.521+0000 7f150840f980 20 add_watcher() i=6 2024-03-09T14:53:06.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.525+0000 7f150840f980 20 add_watcher() i=7 2024-03-09T14:53:06.525 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.525+0000 7f150840f980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:06.529 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.529+0000 7f13b77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:06.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.529+0000 7f150840f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:06.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.529+0000 7f150840f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:06.530 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.529+0000 7f150840f980 5 note: GC not initialized 2024-03-09T14:53:06.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.529+0000 7f13b67fc640 20 reqs_thread_entry: start 2024-03-09T14:53:06.531 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.529+0000 7f150840f980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:06.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.529+0000 7f150840f980 20 Filter name: none 2024-03-09T14:53:06.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.529+0000 7f1327fff640 20 reqs_thread_entry: start 2024-03-09T14:53:06.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.529+0000 7f150840f980 10 cache get: name=default.rgw.meta+root+myver : miss 2024-03-09T14:53:06.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.529+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.532 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.533+0000 7f150840f980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:06.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.533+0000 7f150840f980 10 cache put: name=default.rgw.meta+root+myver info.flags=0x11 2024-03-09T14:53:06.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.533+0000 7f150840f980 10 adding default.rgw.meta+root+myver to cache LRU end 2024-03-09T14:53:06.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.533+0000 7f150840f980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 : miss 2024-03-09T14:53:06.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.533+0000 7f150840f980 20 rados->read ofs=0 len=0 2024-03-09T14:53:06.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.533+0000 7f150840f980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:06.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.533+0000 7f150840f980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:06.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.533+0000 7f150840f980 10 adding default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:06.533 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.533+0000 7f150840f980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:53:06.534 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.533+0000 7f150840f980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:06.534 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.533+0000 7f150840f980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 2024-03-09T14:53:06.542 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.541+0000 7f150840f980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:06.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.541+0000 7f150840f980 10 moving default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:06.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.541+0000 7f150840f980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:06.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.541+0000 7f150840f980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18, ofs0, ns] 2024-03-09T14:53:06.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.541+0000 7f13e97fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562421 cookie 94562463799584 notifier 7263 bl.length()=813 2024-03-09T14:53:06.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.541+0000 7f13e97fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:06.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.541+0000 7f13e97fa640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:06.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.541+0000 7f13e97fa640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:06.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.541+0000 7f150840f980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): orig_sources= new_sources= 2024-03-09T14:53:06.543 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.541+0000 7f150840f980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): potential sources added=[] removed=[] 2024-03-09T14:53:06.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.541+0000 7f150840f980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): orig_dests= new_dests= 2024-03-09T14:53:06.544 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.541+0000 7f150840f980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): potential dests added=[] removed=[] 2024-03-09T14:53:06.547 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:06.545+0000 7f150840f980 -1 Injecting delay=5.000000000s at location=do_reshard 2024-03-09T14:53:06.985 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:06.985 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:11.547 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.545+0000 7f150840f980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.0 2024-03-09T14:53:11.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.545+0000 7f150840f980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.0 2024-03-09T14:53:11.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.545+0000 7f150840f980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.1 2024-03-09T14:53:11.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.545+0000 7f150840f980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.1 2024-03-09T14:53:11.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.545+0000 7f150840f980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.2 2024-03-09T14:53:11.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.545+0000 7f150840f980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.2 2024-03-09T14:53:11.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.545+0000 7f150840f980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.1.0 2024-03-09T14:53:11.548 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.545+0000 7f150840f980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.1.0 2024-03-09T14:53:11.549 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.549+0000 7f150840f980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.1.1 2024-03-09T14:53:11.549 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.549+0000 7f150840f980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.1.1 2024-03-09T14:53:11.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.549+0000 7f150840f980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 : hit (requested=0x17, cached=0x17) 2024-03-09T14:53:11.551 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.549+0000 7f150840f980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 2024-03-09T14:53:11.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.553+0000 7f150840f980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:11.554 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.553+0000 7f150840f980 10 moving default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:11.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.553+0000 7f150840f980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:11.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.553+0000 7f150840f980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18, ofs0, ns] 2024-03-09T14:53:11.555 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.553+0000 7f13e97fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562422 cookie 94562463799584 notifier 7263 bl.length()=781 2024-03-09T14:53:11.556 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.553+0000 7f13e97fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:11.556 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.553+0000 7f13e97fa640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:11.556 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.553+0000 7f13e97fa640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:11.560 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.561+0000 7f150840f980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): orig_sources= new_sources= 2024-03-09T14:53:11.560 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.561+0000 7f150840f980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): potential sources added=[] removed=[] 2024-03-09T14:53:11.560 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.561+0000 7f150840f980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): orig_dests= new_dests= 2024-03-09T14:53:11.560 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.561+0000 7f150840f980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): potential dests added=[] removed=[] 2024-03-09T14:53:11.564 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.565+0000 7f150840f980 1 execute INFO: reshard of bucket "myver" completed successfully 2024-03-09T14:53:11.570 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.569+0000 7f13b77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:11.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.577+0000 7f150840f980 20 remove_watcher() i=0 2024-03-09T14:53:11.576 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.577+0000 7f150840f980 2 removed watcher, disabling cache 2024-03-09T14:53:11.579 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.577+0000 7f150840f980 20 remove_watcher() i=1 2024-03-09T14:53:11.581 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.581+0000 7f150840f980 20 remove_watcher() i=2 2024-03-09T14:53:11.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.585+0000 7f150840f980 20 remove_watcher() i=3 2024-03-09T14:53:11.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.585+0000 7f150840f980 20 remove_watcher() i=4 2024-03-09T14:53:11.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.589+0000 7f150840f980 20 remove_watcher() i=5 2024-03-09T14:53:11.591 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.589+0000 7f150840f980 20 remove_watcher() i=6 2024-03-09T14:53:11.593 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:11.593+0000 7f150840f980 20 remove_watcher() i=7 2024-03-09T14:53:12.192 INFO:tasks.workunit.client.0.smithi001.stderr:put object successful 2024-03-09T14:53:12.192 INFO:tasks.workunit.client.0.smithi001.stderr:TEST: check that bucket stats are correct after reshard with unlinked entries 2024-03-09T14:53:12.192 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-03-09T14:53:12.197 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:12.197 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:12.310 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:12.311 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:12.478 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:12.478 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:12.632 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:12.633 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:12.705 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:12.706 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:12.850 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:12.851 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:12.926 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:12.926 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:13.075 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:13.075 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:13.146 INFO:tasks.workunit.client.0.smithi001.stderr:ceph config set client rgw_debug_inject_set_olh_err 2 2024-03-09T14:53:13.582 INFO:tasks.workunit.client.0.smithi001.stderr:ceph config set client rgw_debug_inject_olh_cancel_modification_err true 2024-03-09T14:53:15.001 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:15.001 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:15.169 INFO:tasks.workunit.client.0.smithi001.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2024-03-09T14:53:15.170 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket myver --object x 2024-03-09T14:53:15.207 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:15.207 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:15.257 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.257+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.258 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.257+0000 7fe41adb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.257+0000 7fe41adb3980 20 realm 2024-03-09T14:53:15.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.257+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.257+0000 7fe41adb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.259 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.257+0000 7fe41adb3980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:15.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.257+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.257+0000 7fe41adb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.260 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.257+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.261+0000 7fe41adb3980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:15.261 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.261+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.261+0000 7fe41adb3980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:15.262 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.261+0000 7fe41adb3980 20 searching for the correct realm 2024-03-09T14:53:15.269 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.269+0000 7fe413fff640 0 -- 172.21.15.1:0/947244124 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fe40404e060 msgr2=0x7fe40401b050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:15.276 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.273+0000 7fe413fff640 0 -- 172.21.15.1:0/947244124 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fe2e0075e60 msgr2=0x7fe2e0078290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:15.283 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.281+0000 7fe41adb3980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:15.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.281+0000 7fe41adb3980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:15.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.281+0000 7fe41adb3980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:15.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.281+0000 7fe41adb3980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:15.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.281+0000 7fe41adb3980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:15.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.281+0000 7fe41adb3980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:15.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.281+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.281+0000 7fe41adb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.284 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.281+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:15.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:15.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 zone default found 2024-03-09T14:53:15.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 4 Realm: () 2024-03-09T14:53:15.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:15.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:15.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:15.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 zonegroup default 2024-03-09T14:53:15.285 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.286 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 started sync module instance, tier type = 2024-03-09T14:53:15.287 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.285+0000 7fe41adb3980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:15.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.301+0000 7fe41adb3980 20 add_watcher() i=0 2024-03-09T14:53:15.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.301+0000 7fe41adb3980 20 add_watcher() i=1 2024-03-09T14:53:15.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.301+0000 7fe41adb3980 20 add_watcher() i=2 2024-03-09T14:53:15.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.301+0000 7fe41adb3980 20 add_watcher() i=3 2024-03-09T14:53:15.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.301+0000 7fe41adb3980 20 add_watcher() i=4 2024-03-09T14:53:15.303 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.301+0000 7fe41adb3980 20 add_watcher() i=5 2024-03-09T14:53:15.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.301+0000 7fe41adb3980 20 add_watcher() i=6 2024-03-09T14:53:15.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.305+0000 7fe41adb3980 20 add_watcher() i=7 2024-03-09T14:53:15.304 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.305+0000 7fe41adb3980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:15.307 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.305+0000 7fe2cdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:15.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.305+0000 7fe41adb3980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:15.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.305+0000 7fe41adb3980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:15.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.305+0000 7fe41adb3980 5 note: GC not initialized 2024-03-09T14:53:15.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.305+0000 7fe2ccfe1640 20 reqs_thread_entry: start 2024-03-09T14:53:15.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:15.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 Filter name: none 2024-03-09T14:53:15.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe2467fc640 20 reqs_thread_entry: start 2024-03-09T14:53:15.308 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:15.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:15.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:15.309 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 INFO: main: max_entries=1000, index=gen=2, max_shards=3 2024-03-09T14:53:15.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 INFO: main: starting shard=0 2024-03-09T14:53:15.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.0 2024-03-09T14:53:15.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.0 2024-03-09T14:53:15.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2024-03-09T14:53:15.310 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 INFO: main: starting shard=1 2024-03-09T14:53:15.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.1 2024-03-09T14:53:15.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.1 2024-03-09T14:53:15.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2024-03-09T14:53:15.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 INFO: main: starting shard=2 2024-03-09T14:53:15.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.2 2024-03-09T14:53:15.311 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.2 2024-03-09T14:53:15.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2024-03-09T14:53:15.312 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.309+0000 7fe41adb3980 20 INFO: main: done 2024-03-09T14:53:15.313 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.313+0000 7fe2cdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:15.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.317+0000 7fe41adb3980 20 remove_watcher() i=0 2024-03-09T14:53:15.318 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.317+0000 7fe41adb3980 2 removed watcher, disabling cache 2024-03-09T14:53:15.320 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.321+0000 7fe41adb3980 20 remove_watcher() i=1 2024-03-09T14:53:15.322 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.321+0000 7fe41adb3980 20 remove_watcher() i=2 2024-03-09T14:53:15.324 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.325+0000 7fe41adb3980 20 remove_watcher() i=3 2024-03-09T14:53:15.326 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.325+0000 7fe41adb3980 20 remove_watcher() i=4 2024-03-09T14:53:15.328 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.329+0000 7fe41adb3980 20 remove_watcher() i=5 2024-03-09T14:53:15.329 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.329+0000 7fe41adb3980 20 remove_watcher() i=6 2024-03-09T14:53:15.330 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.329+0000 7fe41adb3980 20 remove_watcher() i=7 2024-03-09T14:53:15.349 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:15.349 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:15.377 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:15.377 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:15.497 INFO:tasks.workunit.client.0.smithi001.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2024-03-09T14:53:15.497 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket myver --object y 2024-03-09T14:53:15.534 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:15.534 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:15.580 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.581+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.581+0000 7fa16a724980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.581+0000 7fa16a724980 20 realm 2024-03-09T14:53:15.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.581+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.582 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.581+0000 7fa16a724980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.581+0000 7fa16a724980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:15.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.581+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.581+0000 7fa16a724980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.581+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.585+0000 7fa16a724980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:15.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.585+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.585 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.585+0000 7fa16a724980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:15.585 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.585+0000 7fa16a724980 20 searching for the correct realm 2024-03-09T14:53:15.592 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.593+0000 7fa168a3d640 0 -- 172.21.15.1:0/3745197966 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x5608e2fdfb90 msgr2=0x5608e2fe2020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:15.598 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.597+0000 7fa16823c640 0 -- 172.21.15.1:0/3745197966 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fa150008050 msgr2=0x7fa15007d900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:15.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.605+0000 7fa16a724980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:15.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.605+0000 7fa16a724980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:15.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.605+0000 7fa16a724980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:15.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.605+0000 7fa16a724980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:15.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.605+0000 7fa16a724980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:15.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.605+0000 7fa16a724980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:15.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.605+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.605+0000 7fa16a724980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.607 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.605+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.605+0000 7fa16a724980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:15.608 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.605+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:15.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 20 zone default found 2024-03-09T14:53:15.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 4 Realm: () 2024-03-09T14:53:15.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:15.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:15.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:15.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 20 zonegroup default 2024-03-09T14:53:15.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.609 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 20 started sync module instance, tier type = 2024-03-09T14:53:15.610 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.609+0000 7fa16a724980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:15.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.625+0000 7fa16a724980 20 add_watcher() i=0 2024-03-09T14:53:15.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.625+0000 7fa16a724980 20 add_watcher() i=1 2024-03-09T14:53:15.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.625+0000 7fa16a724980 20 add_watcher() i=2 2024-03-09T14:53:15.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.625+0000 7fa16a724980 20 add_watcher() i=3 2024-03-09T14:53:15.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.625+0000 7fa16a724980 20 add_watcher() i=4 2024-03-09T14:53:15.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.625+0000 7fa16a724980 20 add_watcher() i=5 2024-03-09T14:53:15.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.625+0000 7fa16a724980 20 add_watcher() i=6 2024-03-09T14:53:15.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.629+0000 7fa16a724980 20 add_watcher() i=7 2024-03-09T14:53:15.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.629+0000 7fa16a724980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:15.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.629+0000 7fa01efdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:15.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.629+0000 7fa16a724980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:15.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.629+0000 7fa16a724980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:15.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.629+0000 7fa16a724980 5 note: GC not initialized 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.629+0000 7fa01dfdb640 20 reqs_thread_entry: start 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.629+0000 7fa16a724980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 Filter name: none 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7f9f8f7fe640 20 reqs_thread_entry: start 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 INFO: main: max_entries=1000, index=gen=2, max_shards=3 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 INFO: main: starting shard=0 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.0 2024-03-09T14:53:15.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.0 2024-03-09T14:53:15.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2024-03-09T14:53:15.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 INFO: main: starting shard=1 2024-03-09T14:53:15.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.1 2024-03-09T14:53:15.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.1 2024-03-09T14:53:15.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2024-03-09T14:53:15.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 INFO: main: starting shard=2 2024-03-09T14:53:15.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.2 2024-03-09T14:53:15.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.2 2024-03-09T14:53:15.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2024-03-09T14:53:15.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.633+0000 7fa16a724980 20 INFO: main: done 2024-03-09T14:53:15.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.637+0000 7fa01efdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:15.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.641+0000 7fa16a724980 20 remove_watcher() i=0 2024-03-09T14:53:15.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.641+0000 7fa16a724980 2 removed watcher, disabling cache 2024-03-09T14:53:15.644 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.645+0000 7fa16a724980 20 remove_watcher() i=1 2024-03-09T14:53:15.646 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.645+0000 7fa16a724980 20 remove_watcher() i=2 2024-03-09T14:53:15.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.649+0000 7fa16a724980 20 remove_watcher() i=3 2024-03-09T14:53:15.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.649+0000 7fa16a724980 20 remove_watcher() i=4 2024-03-09T14:53:15.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.649+0000 7fa16a724980 20 remove_watcher() i=5 2024-03-09T14:53:15.653 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.653+0000 7fa16a724980 20 remove_watcher() i=6 2024-03-09T14:53:15.654 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.653+0000 7fa16a724980 20 remove_watcher() i=7 2024-03-09T14:53:15.670 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:15.670 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:15.698 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:15.698 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:15.829 INFO:tasks.workunit.client.0.smithi001.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2024-03-09T14:53:15.829 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bi list --bucket myver --object z 2024-03-09T14:53:15.865 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:15.865 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:15.912 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.913+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.913+0000 7efd24dc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.913+0000 7efd24dc0980 20 realm 2024-03-09T14:53:15.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.913+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.914 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.913+0000 7efd24dc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.913+0000 7efd24dc0980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:15.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.913+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.913+0000 7efd24dc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.915 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.913+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.917+0000 7efd24dc0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:15.916 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.917+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.917 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.917+0000 7efd24dc0980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:15.917 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.917+0000 7efd24dc0980 20 searching for the correct realm 2024-03-09T14:53:15.923 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.921+0000 7efd220d7640 0 -- 172.21.15.1:0/203097621 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x5615c55d4360 msgr2=0x5615c55d6750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:53:15.931 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.929+0000 7efd228d8640 0 -- 172.21.15.1:0/203097621 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7efd0c00f8e0 msgr2=0x7efd0c0413c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:15.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.937+0000 7efd24dc0980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:15.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.937+0000 7efd24dc0980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:15.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.937+0000 7efd24dc0980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:15.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.937+0000 7efd24dc0980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:15.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.937+0000 7efd24dc0980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:15.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.937+0000 7efd24dc0980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:15.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.937+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.940 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:15.941 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:15.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 zone default found 2024-03-09T14:53:15.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 4 Realm: () 2024-03-09T14:53:15.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:15.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:15.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:15.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 zonegroup default 2024-03-09T14:53:15.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.942 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:15.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 started sync module instance, tier type = 2024-03-09T14:53:15.943 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.941+0000 7efd24dc0980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:15.960 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.961+0000 7efd24dc0980 20 add_watcher() i=0 2024-03-09T14:53:15.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.961+0000 7efd24dc0980 20 add_watcher() i=1 2024-03-09T14:53:15.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.961+0000 7efd24dc0980 20 add_watcher() i=2 2024-03-09T14:53:15.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.961+0000 7efd24dc0980 20 add_watcher() i=3 2024-03-09T14:53:15.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.961+0000 7efd24dc0980 20 add_watcher() i=4 2024-03-09T14:53:15.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.961+0000 7efd24dc0980 20 add_watcher() i=5 2024-03-09T14:53:15.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.961+0000 7efd24dc0980 20 add_watcher() i=6 2024-03-09T14:53:15.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.961+0000 7efd24dc0980 20 add_watcher() i=7 2024-03-09T14:53:15.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.961+0000 7efd24dc0980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:15.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efbd97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:15.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:15.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:15.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 5 note: GC not initialized 2024-03-09T14:53:15.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efb53fff640 20 reqs_thread_entry: start 2024-03-09T14:53:15.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:15.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 Filter name: none 2024-03-09T14:53:15.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efb51ffb640 20 reqs_thread_entry: start 2024-03-09T14:53:15.966 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.967 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:15.967 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 rados->read ofs=0 len=0 2024-03-09T14:53:15.967 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:15.967 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:15.967 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 INFO: main: max_entries=1000, index=gen=2, max_shards=3 2024-03-09T14:53:15.967 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 INFO: main: starting shard=0 2024-03-09T14:53:15.967 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.0 2024-03-09T14:53:15.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.0 2024-03-09T14:53:15.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2024-03-09T14:53:15.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 INFO: main: starting shard=1 2024-03-09T14:53:15.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.1 2024-03-09T14:53:15.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.1 2024-03-09T14:53:15.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2024-03-09T14:53:15.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 INFO: main: starting shard=2 2024-03-09T14:53:15.968 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.2 2024-03-09T14:53:15.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.965+0000 7efd24dc0980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.2 2024-03-09T14:53:15.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.969+0000 7efd24dc0980 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2024-03-09T14:53:15.969 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.969+0000 7efd24dc0980 20 INFO: main: done 2024-03-09T14:53:15.970 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.969+0000 7efbd97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:15.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.973+0000 7efd24dc0980 20 remove_watcher() i=0 2024-03-09T14:53:15.975 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.973+0000 7efd24dc0980 2 removed watcher, disabling cache 2024-03-09T14:53:15.977 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.977+0000 7efd24dc0980 20 remove_watcher() i=1 2024-03-09T14:53:15.979 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.977+0000 7efd24dc0980 20 remove_watcher() i=2 2024-03-09T14:53:15.981 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.981+0000 7efd24dc0980 20 remove_watcher() i=3 2024-03-09T14:53:15.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.981+0000 7efd24dc0980 20 remove_watcher() i=4 2024-03-09T14:53:15.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.985+0000 7efd24dc0980 20 remove_watcher() i=5 2024-03-09T14:53:15.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.985+0000 7efd24dc0980 20 remove_watcher() i=6 2024-03-09T14:53:15.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:15.985+0000 7efd24dc0980 20 remove_watcher() i=7 2024-03-09T14:53:16.002 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:16.002 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:16.029 INFO:tasks.workunit.client.0.smithi001.stderr:ceph config rm client rgw_debug_inject_set_olh_err 2024-03-09T14:53:16.466 INFO:tasks.workunit.client.0.smithi001.stderr:ceph config rm client rgw_debug_inject_olh_cancel_modification_err 2024-03-09T14:53:16.887 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket reshard --bucket myver --num-shards 17 --yes-i-really-mean-it 2024-03-09T14:53:16.915 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:16.915 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:16.957 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.957+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:16.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.957+0000 7f81205cf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:16.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.957+0000 7f81205cf980 20 realm 2024-03-09T14:53:16.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.957+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:16.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.957+0000 7f81205cf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:16.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.957+0000 7f81205cf980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:16.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.957+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:16.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.957+0000 7f81205cf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:16.959 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.957+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:16.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.961+0000 7f81205cf980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:16.961 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.961+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:16.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.961+0000 7f81205cf980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:16.962 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.961+0000 7f81205cf980 20 searching for the correct realm 2024-03-09T14:53:16.965 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.965+0000 7f811e0e7640 0 -- 172.21.15.1:0/1024917907 >> [v2:172.21.15.1:6824/1666056338,v1:172.21.15.1:6825/1666056338] conn(0x555a8aa94f50 msgr2=0x7f8118036920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:16.972 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.973+0000 7f811e0e7640 0 -- 172.21.15.1:0/1024917907 >> [v2:172.21.15.1:6824/1666056338,v1:172.21.15.1:6825/1666056338] conn(0x7f7fe8075e90 msgr2=0x7f7fe80782c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:16.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.981+0000 7f81205cf980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:16.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.981+0000 7f81205cf980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:16.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.981+0000 7f81205cf980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:16.983 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.981+0000 7f81205cf980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:16.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.981+0000 7f81205cf980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:16.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.981+0000 7f81205cf980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:16.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.981+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:16.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.981+0000 7f81205cf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:16.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.981+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:16.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:16.984 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:16.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:16.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 zone default found 2024-03-09T14:53:16.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 4 Realm: () 2024-03-09T14:53:16.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:16.985 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:16.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:16.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 zonegroup default 2024-03-09T14:53:16.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:16.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:16.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:16.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:16.986 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:16.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:16.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 started sync module instance, tier type = 2024-03-09T14:53:16.987 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:16.985+0000 7f81205cf980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:17.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.001+0000 7f81205cf980 20 add_watcher() i=0 2024-03-09T14:53:17.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.001+0000 7f81205cf980 20 add_watcher() i=1 2024-03-09T14:53:17.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.001+0000 7f81205cf980 20 add_watcher() i=2 2024-03-09T14:53:17.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.001+0000 7f81205cf980 20 add_watcher() i=3 2024-03-09T14:53:17.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.001+0000 7f81205cf980 20 add_watcher() i=4 2024-03-09T14:53:17.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.001+0000 7f81205cf980 20 add_watcher() i=5 2024-03-09T14:53:17.003 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.001+0000 7f81205cf980 20 add_watcher() i=6 2024-03-09T14:53:17.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.001+0000 7f81205cf980 20 add_watcher() i=7 2024-03-09T14:53:17.004 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.001+0000 7f81205cf980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:17.006 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.005+0000 7f7fd4fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:17.007 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.005+0000 7f81205cf980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:17.007 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.005+0000 7f81205cf980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:17.007 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.005+0000 7f81205cf980 5 note: GC not initialized 2024-03-09T14:53:17.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.005+0000 7f7f4f7fe640 20 reqs_thread_entry: start 2024-03-09T14:53:17.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.005+0000 7f81205cf980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:17.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.005+0000 7f81205cf980 20 Filter name: none 2024-03-09T14:53:17.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.005+0000 7f7f4d7fa640 20 reqs_thread_entry: start 2024-03-09T14:53:17.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.005+0000 7f81205cf980 10 cache get: name=default.rgw.meta+root+myver : miss 2024-03-09T14:53:17.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.005+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.009+0000 7f81205cf980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:17.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.009+0000 7f81205cf980 10 cache put: name=default.rgw.meta+root+myver info.flags=0x11 2024-03-09T14:53:17.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.009+0000 7f81205cf980 10 adding default.rgw.meta+root+myver to cache LRU end 2024-03-09T14:53:17.008 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.009+0000 7f81205cf980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 : miss 2024-03-09T14:53:17.009 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.009+0000 7f81205cf980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.009 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.009+0000 7f81205cf980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:17.009 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.009+0000 7f81205cf980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:17.009 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.009+0000 7f81205cf980 10 adding default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:17.009 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.009+0000 7f81205cf980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:53:17.009 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.009+0000 7f81205cf980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:17.009 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.009+0000 7f81205cf980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 2024-03-09T14:53:17.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.021+0000 7f81205cf980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:17.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.021+0000 7f81205cf980 10 moving default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:17.020 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.021+0000 7f81205cf980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:17.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.021+0000 7f81205cf980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18, ofs0, ns] 2024-03-09T14:53:17.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.021+0000 7f80017fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562423 cookie 93847364128048 notifier 7368 bl.length()=813 2024-03-09T14:53:17.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.021+0000 7f80017fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:17.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.021+0000 7f80017fa640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:17.021 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.021+0000 7f80017fa640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:17.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.021+0000 7f81205cf980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): orig_sources= new_sources= 2024-03-09T14:53:17.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.021+0000 7f81205cf980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): potential sources added=[] removed=[] 2024-03-09T14:53:17.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.021+0000 7f81205cf980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): orig_dests= new_dests= 2024-03-09T14:53:17.024 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.021+0000 7f81205cf980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): potential dests added=[] removed=[] 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.0 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.0 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.1 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.1 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.2 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.2 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.3 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.3 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.4 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.4 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.5 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.5 2024-03-09T14:53:17.026 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.6 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.6 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.7 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.7 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.8 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.8 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.10 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.10 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.11 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.11 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.12 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.12 2024-03-09T14:53:17.027 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.13 2024-03-09T14:53:17.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.13 2024-03-09T14:53:17.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.14 2024-03-09T14:53:17.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.14 2024-03-09T14:53:17.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.15 2024-03-09T14:53:17.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.15 2024-03-09T14:53:17.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.16 2024-03-09T14:53:17.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.16 2024-03-09T14:53:17.028 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.0 2024-03-09T14:53:17.029 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.0 2024-03-09T14:53:17.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.1 2024-03-09T14:53:17.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.1 2024-03-09T14:53:17.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.2 2024-03-09T14:53:17.030 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.025+0000 7f81205cf980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.2.2 2024-03-09T14:53:17.031 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.029+0000 7f81205cf980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 : hit (requested=0x17, cached=0x17) 2024-03-09T14:53:17.031 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.029+0000 7f81205cf980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 2024-03-09T14:53:17.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.033+0000 7f81205cf980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:17.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.033+0000 7f81205cf980 10 moving default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:17.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.033+0000 7f81205cf980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:17.032 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.033+0000 7f81205cf980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18, ofs0, ns] 2024-03-09T14:53:17.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.033+0000 7f8001ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562424 cookie 93847364128048 notifier 7368 bl.length()=781 2024-03-09T14:53:17.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.033+0000 7f8001ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:17.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.033+0000 7f8001ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:17.033 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.033+0000 7f8001ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:17.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.033+0000 7f81205cf980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): orig_sources= new_sources= 2024-03-09T14:53:17.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.033+0000 7f81205cf980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): potential sources added=[] removed=[] 2024-03-09T14:53:17.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.033+0000 7f81205cf980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): orig_dests= new_dests= 2024-03-09T14:53:17.034 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.033+0000 7f81205cf980 20 handle_bi_update(): bucket=:myver[8c209108-e1cd-4d45-9608-6f491550b24c.4644.18]): potential dests added=[] removed=[] 2024-03-09T14:53:17.037 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.037+0000 7f81205cf980 1 execute INFO: reshard of bucket "myver" completed successfully 2024-03-09T14:53:17.040 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.041+0000 7f7fd4fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:17.044 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.045+0000 7f81205cf980 20 remove_watcher() i=0 2024-03-09T14:53:17.045 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.045+0000 7f81205cf980 2 removed watcher, disabling cache 2024-03-09T14:53:17.047 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.045+0000 7f81205cf980 20 remove_watcher() i=1 2024-03-09T14:53:17.049 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.049+0000 7f81205cf980 20 remove_watcher() i=2 2024-03-09T14:53:17.051 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.049+0000 7f81205cf980 20 remove_watcher() i=3 2024-03-09T14:53:17.053 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.053+0000 7f81205cf980 20 remove_watcher() i=4 2024-03-09T14:53:17.054 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.053+0000 7f81205cf980 20 remove_watcher() i=5 2024-03-09T14:53:17.055 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.053+0000 7f81205cf980 20 remove_watcher() i=6 2024-03-09T14:53:17.056 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.057+0000 7f81205cf980 20 remove_watcher() i=7 2024-03-09T14:53:17.070 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket check unlinked --bucket myver --fix --min-age-hours 0 --rgw-olh-pending-timeout-sec 0 --dump-keys 2024-03-09T14:53:17.102 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:17.103 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:17.144 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.145+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.145+0000 7f7ec47c7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:17.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.145+0000 7f7ec47c7980 20 realm 2024-03-09T14:53:17.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.145+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.145+0000 7f7ec47c7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:17.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.145+0000 7f7ec47c7980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:17.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.145+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.145+0000 7f7ec47c7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:17.146 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.145+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.145+0000 7f7ec47c7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:17.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.145+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.149+0000 7f7ec47c7980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:17.148 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.149+0000 7f7ec47c7980 20 searching for the correct realm 2024-03-09T14:53:17.151 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.149+0000 7f7ec1ade640 0 -- 172.21.15.1:0/2648328305 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f7eb4010040 msgr2=0x7f7eb408d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:53:17.365 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.365+0000 7f7ec22df640 0 -- 172.21.15.1:0/2648328305 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f7eb80d1430 msgr2=0x7f7eb80d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:17.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.581+0000 7f7ec47c7980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:17.583 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.581+0000 7f7ec47c7980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:17.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.581+0000 7f7ec47c7980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:17.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.581+0000 7f7ec47c7980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:17.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.581+0000 7f7ec47c7980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:17.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.581+0000 7f7ec47c7980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:17.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.581+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.581+0000 7f7ec47c7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:17.584 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.581+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.585 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:17.585 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:17.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 20 zone default found 2024-03-09T14:53:17.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 4 Realm: () 2024-03-09T14:53:17.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:17.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:17.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:17.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 20 zonegroup default 2024-03-09T14:53:17.586 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:17.587 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:17.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.585+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.588 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.589+0000 7f7ec47c7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:17.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.589+0000 7f7ec47c7980 20 started sync module instance, tier type = 2024-03-09T14:53:17.589 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.589+0000 7f7ec47c7980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:17.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.613+0000 7f7ec47c7980 20 add_watcher() i=0 2024-03-09T14:53:17.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.613+0000 7f7ec47c7980 20 add_watcher() i=1 2024-03-09T14:53:17.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.613+0000 7f7ec47c7980 20 add_watcher() i=2 2024-03-09T14:53:17.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.613+0000 7f7ec47c7980 20 add_watcher() i=3 2024-03-09T14:53:17.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.613+0000 7f7ec47c7980 20 add_watcher() i=4 2024-03-09T14:53:17.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.613+0000 7f7ec47c7980 20 add_watcher() i=5 2024-03-09T14:53:17.613 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.613+0000 7f7ec47c7980 20 add_watcher() i=6 2024-03-09T14:53:17.615 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.613+0000 7f7ec47c7980 20 add_watcher() i=7 2024-03-09T14:53:17.615 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.613+0000 7f7ec47c7980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:17.619 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.617+0000 7f7d797e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:17.620 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:17.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:17.621 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 5 note: GC not initialized 2024-03-09T14:53:17.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7d737fe640 20 reqs_thread_entry: start 2024-03-09T14:53:17.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:17.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 20 Filter name: none 2024-03-09T14:53:17.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7d717fa640 20 reqs_thread_entry: start 2024-03-09T14:53:17.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 10 cache get: name=default.rgw.meta+root+myver : miss 2024-03-09T14:53:17.622 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:17.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 10 cache put: name=default.rgw.meta+root+myver info.flags=0x11 2024-03-09T14:53:17.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 10 adding default.rgw.meta+root+myver to cache LRU end 2024-03-09T14:53:17.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 : miss 2024-03-09T14:53:17.623 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 20 rados->read ofs=0 len=0 2024-03-09T14:53:17.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:17.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 info.flags=0x17 2024-03-09T14:53:17.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 10 adding default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 to cache LRU end 2024-03-09T14:53:17.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T14:53:17.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.621+0000 7f7ec47c7980 10 updating xattr: name=user.rgw.acl bl.length()=143 2024-03-09T14:53:17.624 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.625+0000 7f7ec47c7980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18 2024-03-09T14:53:17.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.625+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.0 2024-03-09T14:53:17.625 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.625+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.0 2024-03-09T14:53:17.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.625+0000 7f7ec47c7980 1 NOTICE: finished shard 0 (0 entries removed) 2024-03-09T14:53:17.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.625+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.1 2024-03-09T14:53:17.626 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.625+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.1 2024-03-09T14:53:17.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.625+0000 7f7ec47c7980 1 NOTICE: finished shard 1 (0 entries removed) 2024-03-09T14:53:17.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.625+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.2 2024-03-09T14:53:17.627 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.625+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.2 2024-03-09T14:53:17.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.625+0000 7f7ec47c7980 1 NOTICE: finished shard 2 (0 entries removed) 2024-03-09T14:53:17.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.625+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.3 2024-03-09T14:53:17.628 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.625+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.3 2024-03-09T14:53:17.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 1 NOTICE: finished shard 3 (0 entries removed) 2024-03-09T14:53:17.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.4 2024-03-09T14:53:17.629 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.4 2024-03-09T14:53:17.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 1 NOTICE: finished shard 4 (0 entries removed) 2024-03-09T14:53:17.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.5 2024-03-09T14:53:17.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.5 2024-03-09T14:53:17.630 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 1 NOTICE: finished shard 5 (0 entries removed) 2024-03-09T14:53:17.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.6 2024-03-09T14:53:17.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.6 2024-03-09T14:53:17.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 1 NOTICE: finished shard 6 (0 entries removed) 2024-03-09T14:53:17.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.7 2024-03-09T14:53:17.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.7 2024-03-09T14:53:17.631 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 1 NOTICE: finished shard 7 (0 entries removed) 2024-03-09T14:53:17.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.8 2024-03-09T14:53:17.632 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.629+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.8 2024-03-09T14:53:17.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.633+0000 7f7ec47c7980 1 NOTICE: finished shard 8 (0 entries removed) 2024-03-09T14:53:17.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.633+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.633 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.633+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.633+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.10 2024-03-09T14:53:17.634 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.633+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.10 2024-03-09T14:53:17.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.633+0000 7f7ec47c7980 1 NOTICE: finished shard 10 (0 entries removed) 2024-03-09T14:53:17.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.633+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.11 2024-03-09T14:53:17.635 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.633+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.11 2024-03-09T14:53:17.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.637+0000 7f7ec47c7980 1 NOTICE: finished shard 11 (0 entries removed) 2024-03-09T14:53:17.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.637+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.12 2024-03-09T14:53:17.636 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.637+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.12 2024-03-09T14:53:17.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.637+0000 7f7ec47c7980 1 NOTICE: finished shard 12 (0 entries removed) 2024-03-09T14:53:17.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.637+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.13 2024-03-09T14:53:17.637 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.637+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.13 2024-03-09T14:53:17.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.637+0000 7f7ec47c7980 1 NOTICE: finished shard 13 (0 entries removed) 2024-03-09T14:53:17.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.637+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.14 2024-03-09T14:53:17.638 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.637+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.14 2024-03-09T14:53:17.639 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.637+0000 7f7ec47c7980 1 NOTICE: finished shard 14 (0 entries removed) 2024-03-09T14:53:17.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.637+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.15 2024-03-09T14:53:17.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.637+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.15 2024-03-09T14:53:17.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.641+0000 7f7ec47c7980 1 NOTICE: finished shard 15 (0 entries removed) 2024-03-09T14:53:17.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.641+0000 7f7ec47c7980 10 bucket_obj is .dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.16 2024-03-09T14:53:17.640 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.641+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.16 2024-03-09T14:53:17.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.641+0000 7f7ec47c7980 1 NOTICE: finished shard 16 (0 entries removed) 2024-03-09T14:53:17.641 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.641+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.642 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.641+0000 7f7ec47c7980 20 get_obj_state: octx=0x56283a2bfe30 obj=myver:x state=0x56283a568488 s->prefetch_data=0 2024-03-09T14:53:17.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.641+0000 7f7ec47c7980 20 get_obj_state: setting s->obj_tag to ufbyst75bro5pcn5a9nocybjm2z0s2t4 2024-03-09T14:53:17.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.641+0000 7f7ec47c7980 20 get_obj_state_impl: setting s->olh_tag to 2ftxjucvrtsqkz8zr1shwl39b6ze4654 2024-03-09T14:53:17.643 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.641+0000 7f7ec47c7980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=2ftxjucvrtsqkz8zr1shwl39b6ze4654 2024-03-09T14:53:17.646 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.645+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.648 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.649+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.649+0000 7f7ec47c7980 20 olh_log_entry: epoch=2 op=2 key=x[] 2024-03-09T14:53:17.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.649+0000 7f7ec47c7980 20 olh_log_entry: epoch=2 op=3 key=x[K1I0BTCsM4McfEHOqf2xXBsuVh25EGZ] 2024-03-09T14:53:17.649 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.649+0000 7f7ec47c7980 20 get_obj_state: octx=0x56283a2bfe30 obj=myver:_:K1I0BTCsM4McfEHOqf2xXBsuVh25EGZ_x state=0x56283a59b488 s->prefetch_data=0 2024-03-09T14:53:17.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.649+0000 7f7ec47c7980 10 manifest: total_size = 11 2024-03-09T14:53:17.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.649+0000 7f7ec47c7980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.479543729628227847 2024-03-09T14:53:17.650 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.649+0000 7f7ec47c7980 20 get_obj_state: octx=0x56283a2bfe30 obj=myver:_:K1I0BTCsM4McfEHOqf2xXBsuVh25EGZ_x state=0x56283a59b488 s->prefetch_data=0 2024-03-09T14:53:17.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.649+0000 7f7ec47c7980 20 prepare_atomic_modification: state is not atomic. state=0x56283a59b488 2024-03-09T14:53:17.651 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.649+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.653+0000 7f7ec47c7980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:53:17.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.653+0000 7f7ec47c7980 20 RGWObjManifest::operator++(): result: ofs=11 stripe_ofs=11 part_ofs=0 rule->part_size=0 2024-03-09T14:53:17.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.653+0000 7f7ec47c7980 0 deleting objects inline since gc isn't initialized 2024-03-09T14:53:17.655 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.653+0000 7f7ec47c7980 5 delete_objs_inline: removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18__shadow:K1I0BTCsM4McfEHOqf2xXBsuVh25EGZ_.oigb5YPuKZxeVqc599I5QGNxugcOzPl_0 2024-03-09T14:53:17.656 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.657+0000 7f7da5ffb640 20 handle_completion(): completion ok for obj=x[K1I0BTCsM4McfEHOqf2xXBsuVh25EGZ] 2024-03-09T14:53:17.657 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.657+0000 7f7ec47c7980 5 delete_objs_inline: refcount put returned error -2 2024-03-09T14:53:17.660 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.661+0000 7f7ec47c7980 20 get_obj_state: octx=0x56283a2bfe30 obj=myver:x state=0x56283a568488 s->prefetch_data=0 2024-03-09T14:53:17.661 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.661+0000 7f7ec47c7980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=2ftxjucvrtsqkz8zr1shwl39b6ze4654 2024-03-09T14:53:17.666 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.665+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.668 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.669+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.669+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.669 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.669+0000 7f7ec47c7980 20 get_obj_state: octx=0x56283a2bfe30 obj=myver:y state=0x56283a568488 s->prefetch_data=0 2024-03-09T14:53:17.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.669+0000 7f7ec47c7980 20 get_obj_state: setting s->obj_tag to biogezhhlx3fhodmryphuzbcrzx41e79 2024-03-09T14:53:17.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.669+0000 7f7ec47c7980 20 get_obj_state_impl: setting s->olh_tag to qu7i0umofqt3jyekvigaynejve4i7n0z 2024-03-09T14:53:17.670 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.669+0000 7f7ec47c7980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=qu7i0umofqt3jyekvigaynejve4i7n0z 2024-03-09T14:53:17.672 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.669+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.673+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.673+0000 7f7ec47c7980 20 olh_log_entry: epoch=2 op=2 key=y[] 2024-03-09T14:53:17.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.673+0000 7f7ec47c7980 20 olh_log_entry: epoch=2 op=3 key=y[Rd3gOa7F1QSLtg-XewXvJ2LTFWzMDtn] 2024-03-09T14:53:17.674 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.673+0000 7f7ec47c7980 20 get_obj_state: octx=0x56283a2bfe30 obj=myver:_:Rd3gOa7F1QSLtg-XewXvJ2LTFWzMDtn_y state=0x56283a59faa8 s->prefetch_data=0 2024-03-09T14:53:17.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.673+0000 7f7ec47c7980 10 manifest: total_size = 22 2024-03-09T14:53:17.675 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.673+0000 7f7ec47c7980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.4133846618173844318 2024-03-09T14:53:17.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.673+0000 7f7ec47c7980 20 get_obj_state: octx=0x56283a2bfe30 obj=myver:_:Rd3gOa7F1QSLtg-XewXvJ2LTFWzMDtn_y state=0x56283a59faa8 s->prefetch_data=0 2024-03-09T14:53:17.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.673+0000 7f7ec47c7980 20 prepare_atomic_modification: state is not atomic. state=0x56283a59faa8 2024-03-09T14:53:17.676 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.673+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.681+0000 7f7ec47c7980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:53:17.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.681+0000 7f7ec47c7980 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2024-03-09T14:53:17.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.681+0000 7f7ec47c7980 0 deleting objects inline since gc isn't initialized 2024-03-09T14:53:17.680 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.681+0000 7f7ec47c7980 5 delete_objs_inline: removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18__shadow:Rd3gOa7F1QSLtg-XewXvJ2LTFWzMDtn_.bVuBLK1K4ljjuJroUSHTEOIwSa9lY4G_0 2024-03-09T14:53:17.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.681+0000 7f7ec47c7980 5 delete_objs_inline: refcount put returned error -2 2024-03-09T14:53:17.682 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.681+0000 7f7da57fa640 20 handle_completion(): completion ok for obj=y[Rd3gOa7F1QSLtg-XewXvJ2LTFWzMDtn] 2024-03-09T14:53:17.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.685+0000 7f7ec47c7980 20 get_obj_state: octx=0x56283a2bfe30 obj=myver:y state=0x56283a568488 s->prefetch_data=0 2024-03-09T14:53:17.685 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.685+0000 7f7ec47c7980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=qu7i0umofqt3jyekvigaynejve4i7n0z 2024-03-09T14:53:17.688 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.689+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.691 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.689+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.692 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.693+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.693 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.693+0000 7f7ec47c7980 20 get_obj_state: octx=0x56283a2bfe30 obj=myver:z state=0x56283a568488 s->prefetch_data=0 2024-03-09T14:53:17.693 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.693+0000 7f7ec47c7980 20 get_obj_state: setting s->obj_tag to 5j593lw35f4xbxbcxpkcclaj8c22rvrt 2024-03-09T14:53:17.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.693+0000 7f7ec47c7980 20 get_obj_state_impl: setting s->olh_tag to 30xhpm5r7c7pb5j9ji25gabxbs3dvfkr 2024-03-09T14:53:17.694 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.693+0000 7f7ec47c7980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=30xhpm5r7c7pb5j9ji25gabxbs3dvfkr 2024-03-09T14:53:17.696 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.693+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.699 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.697+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.699 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.697+0000 7f7ec47c7980 20 olh_log_entry: epoch=2 op=2 key=z[] 2024-03-09T14:53:17.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.697+0000 7f7ec47c7980 20 olh_log_entry: epoch=2 op=3 key=z[yZVsglM7MVIgLLgylVGLEXVdn5ia8Ss] 2024-03-09T14:53:17.700 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.697+0000 7f7ec47c7980 20 get_obj_state: octx=0x56283a2bfe30 obj=myver:_:yZVsglM7MVIgLLgylVGLEXVdn5ia8Ss_z state=0x56283a59faa8 s->prefetch_data=0 2024-03-09T14:53:17.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.701+0000 7f7ec47c7980 10 manifest: total_size = 22 2024-03-09T14:53:17.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.701+0000 7f7ec47c7980 20 get_obj_state: setting s->obj_tag to 8c209108-e1cd-4d45-9608-6f491550b24c.4644.5519183135840634333 2024-03-09T14:53:17.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.701+0000 7f7ec47c7980 20 get_obj_state: octx=0x56283a2bfe30 obj=myver:_:yZVsglM7MVIgLLgylVGLEXVdn5ia8Ss_z state=0x56283a59faa8 s->prefetch_data=0 2024-03-09T14:53:17.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.701+0000 7f7ec47c7980 20 prepare_atomic_modification: state is not atomic. state=0x56283a59faa8 2024-03-09T14:53:17.701 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.701+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.707 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.705+0000 7f7ec47c7980 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2024-03-09T14:53:17.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.705+0000 7f7ec47c7980 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2024-03-09T14:53:17.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.705+0000 7f7ec47c7980 0 deleting objects inline since gc isn't initialized 2024-03-09T14:53:17.708 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.705+0000 7f7ec47c7980 5 delete_objs_inline: removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18__shadow:yZVsglM7MVIgLLgylVGLEXVdn5ia8Ss_.B4vB09AKHyKSZnjy1fqilXYQ5QNeSWy_0 2024-03-09T14:53:17.709 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.709+0000 7f7da5ffb640 20 handle_completion(): completion ok for obj=z[yZVsglM7MVIgLLgylVGLEXVdn5ia8Ss] 2024-03-09T14:53:17.710 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.709+0000 7f7ec47c7980 5 delete_objs_inline: refcount put returned error -2 2024-03-09T14:53:17.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.713+0000 7f7ec47c7980 20 get_obj_state: octx=0x56283a2bfe30 obj=myver:z state=0x56283a568488 s->prefetch_data=0 2024-03-09T14:53:17.713 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.713+0000 7f7ec47c7980 20 bucket_index_guard_olh_op(): olh_state.olh_tag=30xhpm5r7c7pb5j9ji25gabxbs3dvfkr 2024-03-09T14:53:17.717 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.717+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.721+0000 7f7ec47c7980 20 bucket index object: :.dir.8c209108-e1cd-4d45-9608-6f491550b24c.4644.18.3.9 2024-03-09T14:53:17.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.721+0000 7f7ec47c7980 1 NOTICE: finished shard 9 (3 entries removed) 2024-03-09T14:53:17.720 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.721+0000 7f7ec47c7980 1 NOTICE: finished all shards (3 entries removed) 2024-03-09T14:53:17.722 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.721+0000 7f7d797e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:17.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.725+0000 7f7ec47c7980 20 remove_watcher() i=0 2024-03-09T14:53:17.727 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.725+0000 7f7ec47c7980 2 removed watcher, disabling cache 2024-03-09T14:53:17.730 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.729+0000 7f7ec47c7980 20 remove_watcher() i=1 2024-03-09T14:53:17.732 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.733+0000 7f7ec47c7980 20 remove_watcher() i=2 2024-03-09T14:53:17.735 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.733+0000 7f7ec47c7980 20 remove_watcher() i=3 2024-03-09T14:53:17.737 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.737+0000 7f7ec47c7980 20 remove_watcher() i=4 2024-03-09T14:53:17.739 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.737+0000 7f7ec47c7980 20 remove_watcher() i=5 2024-03-09T14:53:17.741 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.741+0000 7f7ec47c7980 20 remove_watcher() i=6 2024-03-09T14:53:17.742 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:17.741+0000 7f7ec47c7980 20 remove_watcher() i=7 2024-03-09T14:53:17.764 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:17.764 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:17.848 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:17.848 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:18.120 INFO:tasks.workunit.client.0.smithi001.stderr:radosgw-admin bucket stats --bucket myver 2024-03-09T14:53:18.154 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:18.154 INFO:tasks.workunit.client.0.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:18.200 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.197+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.201+0000 7fe603060980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:18.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.201+0000 7fe603060980 20 realm 2024-03-09T14:53:18.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.201+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.201+0000 7fe603060980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:18.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.201+0000 7fe603060980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:18.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.201+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.201+0000 7fe603060980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:18.202 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.201+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.201+0000 7fe603060980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:18.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.201+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.204 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.205+0000 7fe603060980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:18.205 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.205+0000 7fe603060980 20 searching for the correct realm 2024-03-09T14:53:18.211 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.209+0000 7fe601379640 0 -- 172.21.15.1:0/1715657842 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x5569e00ad6e0 msgr2=0x5569e00afba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:18.218 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.217+0000 7fe600377640 0 -- 172.21.15.1:0/1715657842 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fe5f00c4040 msgr2=0x7fe5f010da50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:53:18.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.225+0000 7fe603060980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:18.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.225+0000 7fe603060980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:18.227 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.225+0000 7fe603060980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:18.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.225+0000 7fe603060980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:18.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.225+0000 7fe603060980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:18.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.225+0000 7fe603060980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:18.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.225+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.225+0000 7fe603060980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:18.228 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.225+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:18.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:18.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 zone default found 2024-03-09T14:53:18.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 4 Realm: () 2024-03-09T14:53:18.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:18.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:18.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:18.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 zonegroup default 2024-03-09T14:53:18.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.229 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:18.230 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.231 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:18.231 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.231 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:18.231 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 started sync module instance, tier type = 2024-03-09T14:53:18.231 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.229+0000 7fe603060980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:18.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.245+0000 7fe603060980 20 add_watcher() i=0 2024-03-09T14:53:18.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.245+0000 7fe603060980 20 add_watcher() i=1 2024-03-09T14:53:18.246 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.245+0000 7fe603060980 20 add_watcher() i=2 2024-03-09T14:53:18.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.245+0000 7fe603060980 20 add_watcher() i=3 2024-03-09T14:53:18.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.245+0000 7fe603060980 20 add_watcher() i=4 2024-03-09T14:53:18.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.245+0000 7fe603060980 20 add_watcher() i=5 2024-03-09T14:53:18.247 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.245+0000 7fe603060980 20 add_watcher() i=6 2024-03-09T14:53:18.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.245+0000 7fe603060980 20 add_watcher() i=7 2024-03-09T14:53:18.248 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.245+0000 7fe603060980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:18.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.249+0000 7fe4b6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:18.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.249+0000 7fe603060980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:18.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.249+0000 7fe603060980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:18.251 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.249+0000 7fe603060980 5 note: GC not initialized 2024-03-09T14:53:18.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.249+0000 7fe4b5fe3640 20 reqs_thread_entry: start 2024-03-09T14:53:18.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.249+0000 7fe603060980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:18.252 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.249+0000 7fe603060980 20 Filter name: none 2024-03-09T14:53:18.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.253+0000 7fe4277fe640 20 reqs_thread_entry: start 2024-03-09T14:53:18.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.253+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.253+0000 7fe603060980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:18.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.253+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.253+0000 7fe603060980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:18.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.253+0000 7fe603060980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:18.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.253+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.253+0000 7fe603060980 20 rados_obj.operate() r=0 bl.length=173 2024-03-09T14:53:18.253 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.253+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.253+0000 7fe603060980 20 rados_obj.operate() r=0 bl.length=387 2024-03-09T14:53:18.254 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.253+0000 7fe603060980 20 couldn't put binfo cache entry, might have raced with data changes 2024-03-09T14:53:18.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.257+0000 7fe603060980 20 rados->read ofs=0 len=0 2024-03-09T14:53:18.256 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.257+0000 7fe603060980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:18.258 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.257+0000 7fe4b6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:18.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.261+0000 7fe603060980 20 remove_watcher() i=0 2024-03-09T14:53:18.263 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.261+0000 7fe603060980 2 removed watcher, disabling cache 2024-03-09T14:53:18.265 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.265+0000 7fe603060980 20 remove_watcher() i=1 2024-03-09T14:53:18.267 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.265+0000 7fe603060980 20 remove_watcher() i=2 2024-03-09T14:53:18.269 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.269+0000 7fe603060980 20 remove_watcher() i=3 2024-03-09T14:53:18.271 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.269+0000 7fe603060980 20 remove_watcher() i=4 2024-03-09T14:53:18.272 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.273+0000 7fe603060980 20 remove_watcher() i=5 2024-03-09T14:53:18.273 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.273+0000 7fe603060980 20 remove_watcher() i=6 2024-03-09T14:53:18.275 INFO:tasks.workunit.client.0.smithi001.stderr:2024-03-09T14:53:18.273+0000 7fe603060980 20 remove_watcher() i=7 2024-03-09T14:53:18.289 INFO:tasks.workunit.client.0.smithi001.stderr:{'rgw.main': {'size': 0, 'size_actual': 0, 'size_utilized': 0, 'size_kb': 0, 'size_kb_actual': 0, 'size_kb_utilized': 0, 'num_objects': 0}} 2024-03-09T14:53:18.289 INFO:tasks.workunit.client.0.smithi001.stderr:Deleting bucket a-bucket 2024-03-09T14:53:18.290 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:18.290 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:18.317 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:18.317 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:18.414 INFO:tasks.workunit.client.0.smithi001.stderr:Deleting bucket myver 2024-03-09T14:53:18.417 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:18.417 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:18.556 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings 2024-03-09T14:53:18.556 INFO:tasks.workunit.client.0.smithi001.stderr: warnings.warn( 2024-03-09T14:53:19.020 INFO:tasks.workunit.client.0.smithi001.stderr:Completed resharding tests 2024-03-09T14:53:19.020 INFO:tasks.workunit.client.0.smithi001.stdout:Could not connect to the endpoint URL: "http://localhost:80/" 2024-03-09T14:53:19.020 INFO:tasks.workunit.client.0.smithi001.stdout:Could not connect to the endpoint URL: "http://localhost:8000/" 2024-03-09T14:53:19.021 INFO:tasks.workunit.client.0.smithi001.stdout:connected to https://localhost:443 2024-03-09T14:53:19.131 INFO:tasks.workunit.client.0.smithi001.stderr:+ deactivate 2024-03-09T14:53:19.131 INFO:tasks.workunit.client.0.smithi001.stderr:+ '[' -n /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin ']' 2024-03-09T14:53:19.131 INFO:tasks.workunit.client.0.smithi001.stderr:+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-09T14:53:19.131 INFO:tasks.workunit.client.0.smithi001.stderr:+ export PATH 2024-03-09T14:53:19.132 INFO:tasks.workunit.client.0.smithi001.stderr:+ unset _OLD_VIRTUAL_PATH 2024-03-09T14:53:19.132 INFO:tasks.workunit.client.0.smithi001.stderr:+ '[' -n '' ']' 2024-03-09T14:53:19.132 INFO:tasks.workunit.client.0.smithi001.stderr:+ '[' -n /usr/bin/bash -o -n '' ']' 2024-03-09T14:53:19.132 INFO:tasks.workunit.client.0.smithi001.stderr:+ hash -r 2024-03-09T14:53:19.132 INFO:tasks.workunit.client.0.smithi001.stderr:+ '[' -n '' ']' 2024-03-09T14:53:19.132 INFO:tasks.workunit.client.0.smithi001.stderr:+ unset VIRTUAL_ENV 2024-03-09T14:53:19.132 INFO:tasks.workunit.client.0.smithi001.stderr:+ unset VIRTUAL_ENV_PROMPT 2024-03-09T14:53:19.132 INFO:tasks.workunit.client.0.smithi001.stderr:+ '[' '!' '' = nondestructive ']' 2024-03-09T14:53:19.132 INFO:tasks.workunit.client.0.smithi001.stderr:+ unset -f deactivate 2024-03-09T14:53:19.132 INFO:tasks.workunit.client.0.smithi001.stderr:+ echo OK. 2024-03-09T14:53:19.133 INFO:tasks.workunit.client.0.smithi001.stdout:OK. 2024-03-09T14:53:19.133 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T14:53:19.134 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:53:19.190 INFO:tasks.workunit:Stopping ['rgw/run-reshard.sh'] on client.0... 2024-03-09T14:53:19.190 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-09T14:53:19.615 DEBUG:teuthology.parallel:result is None 2024-03-09T14:53:19.616 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:53:19.629 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:53:19.629 DEBUG:teuthology.orchestra.run.smithi001:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-09T14:53:19.680 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:53:19.680 INFO:teuthology.run_tasks:Running task s3tests-java... 2024-03-09T14:53:19.696 DEBUG:tasks.s3tests_java:S3 Tests Java: __INIT__ 2024-03-09T14:53:19.697 DEBUG:tasks.s3tests_java:S3 Tests Java: SETUP 2024-03-09T14:53:19.697 INFO:tasks.s3tests_java:S3 Tests Java: Downloading test suite... 2024-03-09T14:53:19.697 DEBUG:teuthology.orchestra.run.smithi001:> git clone -b ceph-master https://github.com/ceph/java_s3tests.git /home/ubuntu/cephtest/s3-tests-java 2024-03-09T14:53:19.729 INFO:teuthology.orchestra.run.smithi001.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests-java'... 2024-03-09T14:53:24.354 INFO:tasks.s3tests_java:S3 Tests Java: Installing required packages... 2024-03-09T14:53:24.354 DEBUG:teuthology.orchestra.run.smithi001:> /home/ubuntu/cephtest/s3-tests-java/bootstrap.sh 2024-03-09T14:53:24.446 INFO:teuthology.orchestra.run.smithi001.stderr:/home/ubuntu/cephtest/s3-tests-java/bootstrap.sh: missing required DEB packages. Installing via sudo. 2024-03-09T14:53:24.506 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-03-09T14:53:24.717 INFO:teuthology.orchestra.run.smithi001.stdout:Building dependency tree... 2024-03-09T14:53:24.717 INFO:teuthology.orchestra.run.smithi001.stdout:Reading state information... 2024-03-09T14:53:24.907 INFO:teuthology.orchestra.run.smithi001.stdout:The following packages were automatically installed and are no longer required: 2024-03-09T14:53:24.907 INFO:teuthology.orchestra.run.smithi001.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-09T14:53:24.907 INFO:teuthology.orchestra.run.smithi001.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-09T14:53:24.907 INFO:teuthology.orchestra.run.smithi001.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-09T14:53:24.908 INFO:teuthology.orchestra.run.smithi001.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-09T14:53:24.908 INFO:teuthology.orchestra.run.smithi001.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-09T14:53:24.908 INFO:teuthology.orchestra.run.smithi001.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-09T14:53:24.908 INFO:teuthology.orchestra.run.smithi001.stdout:The following additional packages will be installed: 2024-03-09T14:53:24.909 INFO:teuthology.orchestra.run.smithi001.stdout: libgail-common libgail18 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common 2024-03-09T14:53:24.909 INFO:teuthology.orchestra.run.smithi001.stdout: openjdk-8-jdk-headless openjdk-8-jre openjdk-8-jre-headless 2024-03-09T14:53:24.911 INFO:teuthology.orchestra.run.smithi001.stdout:Suggested packages: 2024-03-09T14:53:24.911 INFO:teuthology.orchestra.run.smithi001.stdout: gvfs openjdk-8-demo openjdk-8-source visualvm libnss-mdns fonts-nanum 2024-03-09T14:53:24.911 INFO:teuthology.orchestra.run.smithi001.stdout: fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei 2024-03-09T14:53:24.911 INFO:teuthology.orchestra.run.smithi001.stdout: fonts-wqy-zenhei fonts-indic 2024-03-09T14:53:24.944 INFO:teuthology.orchestra.run.smithi001.stdout:The following NEW packages will be installed: 2024-03-09T14:53:24.945 INFO:teuthology.orchestra.run.smithi001.stdout: libgail-common libgail18 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common 2024-03-09T14:53:24.945 INFO:teuthology.orchestra.run.smithi001.stdout: openjdk-8-jdk openjdk-8-jdk-headless openjdk-8-jre openjdk-8-jre-headless 2024-03-09T14:53:25.127 INFO:teuthology.orchestra.run.smithi001.stdout:0 upgraded, 9 newly installed, 0 to remove and 275 not upgraded. 2024-03-09T14:53:25.128 INFO:teuthology.orchestra.run.smithi001.stdout:Need to get 46.1 MB of archives. 2024-03-09T14:53:25.128 INFO:teuthology.orchestra.run.smithi001.stdout:After this operation, 155 MB of additional disk space will be used. 2024-03-09T14:53:25.128 INFO:teuthology.orchestra.run.smithi001.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 libgtk2.0-common all 2.24.33-2ubuntu2 [125 kB] 2024-03-09T14:53:25.492 INFO:teuthology.orchestra.run.smithi001.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 libgtk2.0-0 amd64 2.24.33-2ubuntu2 [2,037 kB] 2024-03-09T14:53:25.851 INFO:teuthology.orchestra.run.smithi001.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 libgail18 amd64 2.24.33-2ubuntu2 [15.9 kB] 2024-03-09T14:53:25.851 INFO:teuthology.orchestra.run.smithi001.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 libgail-common amd64 2.24.33-2ubuntu2 [132 kB] 2024-03-09T14:53:25.858 INFO:teuthology.orchestra.run.smithi001.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 libgtk2.0-bin amd64 2.24.33-2ubuntu2 [7,932 B] 2024-03-09T14:53:25.859 INFO:teuthology.orchestra.run.smithi001.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 openjdk-8-jre-headless amd64 8u392-ga-1~22.04 [30.8 MB] 2024-03-09T14:53:26.915 INFO:teuthology.orchestra.run.smithi001.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 openjdk-8-jre amd64 8u392-ga-1~22.04 [75.5 kB] 2024-03-09T14:53:26.916 INFO:teuthology.orchestra.run.smithi001.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 openjdk-8-jdk-headless amd64 8u392-ga-1~22.04 [8,863 kB] 2024-03-09T14:53:27.168 INFO:teuthology.orchestra.run.smithi001.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 openjdk-8-jdk amd64 8u392-ga-1~22.04 [4,006 kB] 2024-03-09T14:53:27.588 INFO:teuthology.orchestra.run.smithi001.stderr:debconf: unable to initialize frontend: Dialog 2024-03-09T14:53:27.589 INFO:teuthology.orchestra.run.smithi001.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2024-03-09T14:53:27.589 INFO:teuthology.orchestra.run.smithi001.stderr:debconf: falling back to frontend: Readline 2024-03-09T14:53:27.607 INFO:teuthology.orchestra.run.smithi001.stderr:debconf: unable to initialize frontend: Readline 2024-03-09T14:53:27.607 INFO:teuthology.orchestra.run.smithi001.stderr:debconf: (This frontend requires a controlling tty.) 2024-03-09T14:53:27.607 INFO:teuthology.orchestra.run.smithi001.stderr:debconf: falling back to frontend: Teletype 2024-03-09T14:53:27.610 INFO:teuthology.orchestra.run.smithi001.stderr:dpkg-preconfigure: unable to re-open stdin: 2024-03-09T14:53:27.645 INFO:teuthology.orchestra.run.smithi001.stdout:Fetched 46.1 MB in 2s (19.8 MB/s) 2024-03-09T14:53:27.757 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libgtk2.0-common. 2024-03-09T14:53:27.825 INFO:teuthology.orchestra.run.smithi001.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174563 files and directories currently installed.) 2024-03-09T14:53:27.831 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../0-libgtk2.0-common_2.24.33-2ubuntu2_all.deb ... 2024-03-09T14:53:27.873 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libgtk2.0-common (2.24.33-2ubuntu2) ... 2024-03-09T14:53:28.277 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libgtk2.0-0:amd64. 2024-03-09T14:53:28.306 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../1-libgtk2.0-0_2.24.33-2ubuntu2_amd64.deb ... 2024-03-09T14:53:28.385 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libgtk2.0-0:amd64 (2.24.33-2ubuntu2) ... 2024-03-09T14:53:28.840 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libgail18:amd64. 2024-03-09T14:53:28.857 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../2-libgail18_2.24.33-2ubuntu2_amd64.deb ... 2024-03-09T14:53:28.898 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libgail18:amd64 (2.24.33-2ubuntu2) ... 2024-03-09T14:53:29.134 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libgail-common:amd64. 2024-03-09T14:53:29.151 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../3-libgail-common_2.24.33-2ubuntu2_amd64.deb ... 2024-03-09T14:53:29.192 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libgail-common:amd64 (2.24.33-2ubuntu2) ... 2024-03-09T14:53:29.462 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libgtk2.0-bin. 2024-03-09T14:53:29.490 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../4-libgtk2.0-bin_2.24.33-2ubuntu2_amd64.deb ... 2024-03-09T14:53:29.528 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libgtk2.0-bin (2.24.33-2ubuntu2) ... 2024-03-09T14:53:29.957 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package openjdk-8-jre-headless:amd64. 2024-03-09T14:53:29.986 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../5-openjdk-8-jre-headless_8u392-ga-1~22.04_amd64.deb ... 2024-03-09T14:53:30.027 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking openjdk-8-jre-headless:amd64 (8u392-ga-1~22.04) ... 2024-03-09T14:53:31.646 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package openjdk-8-jre:amd64. 2024-03-09T14:53:31.675 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../6-openjdk-8-jre_8u392-ga-1~22.04_amd64.deb ... 2024-03-09T14:53:31.716 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking openjdk-8-jre:amd64 (8u392-ga-1~22.04) ... 2024-03-09T14:53:32.159 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package openjdk-8-jdk-headless:amd64. 2024-03-09T14:53:32.188 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../7-openjdk-8-jdk-headless_8u392-ga-1~22.04_amd64.deb ... 2024-03-09T14:53:32.229 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking openjdk-8-jdk-headless:amd64 (8u392-ga-1~22.04) ... 2024-03-09T14:53:33.038 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package openjdk-8-jdk:amd64. 2024-03-09T14:53:33.067 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../8-openjdk-8-jdk_8u392-ga-1~22.04_amd64.deb ... 2024-03-09T14:53:33.108 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking openjdk-8-jdk:amd64 (8u392-ga-1~22.04) ... 2024-03-09T14:53:33.458 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up openjdk-8-jre-headless:amd64 (8u392-ga-1~22.04) ... 2024-03-09T14:53:34.553 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/orbd to provide /usr/bin/orbd (orbd) in auto mode 2024-03-09T14:53:34.581 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode 2024-03-09T14:53:34.605 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode 2024-03-09T14:53:35.004 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libgtk2.0-common (2.24.33-2ubuntu2) ... 2024-03-09T14:53:35.173 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libgtk2.0-0:amd64 (2.24.33-2ubuntu2) ... 2024-03-09T14:53:35.533 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up openjdk-8-jre:amd64 (8u392-ga-1~22.04) ... 2024-03-09T14:53:35.617 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/policytool to provide /usr/bin/policytool (policytool) in auto mode 2024-03-09T14:53:35.684 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up openjdk-8-jdk-headless:amd64 (8u392-ga-1~22.04) ... 2024-03-09T14:53:35.785 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/clhsdb to provide /usr/bin/clhsdb (clhsdb) in auto mode 2024-03-09T14:53:35.814 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/extcheck to provide /usr/bin/extcheck (extcheck) in auto mode 2024-03-09T14:53:35.839 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/hsdb to provide /usr/bin/hsdb (hsdb) in auto mode 2024-03-09T14:53:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/idlj to provide /usr/bin/idlj (idlj) in auto mode 2024-03-09T14:53:35.991 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/javah to provide /usr/bin/javah (javah) in auto mode 2024-03-09T14:53:36.137 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jhat to provide /usr/bin/jhat (jhat) in auto mode 2024-03-09T14:53:36.287 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jsadebugd to provide /usr/bin/jsadebugd (jsadebugd) in auto mode 2024-03-09T14:53:36.397 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/native2ascii to provide /usr/bin/native2ascii (native2ascii) in auto mode 2024-03-09T14:53:36.448 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/schemagen to provide /usr/bin/schemagen (schemagen) in auto mode 2024-03-09T14:53:36.499 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsgen to provide /usr/bin/wsgen (wsgen) in auto mode 2024-03-09T14:53:36.525 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsimport to provide /usr/bin/wsimport (wsimport) in auto mode 2024-03-09T14:53:36.550 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode 2024-03-09T14:53:36.615 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libgail18:amd64 (2.24.33-2ubuntu2) ... 2024-03-09T14:53:36.741 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libgtk2.0-bin (2.24.33-2ubuntu2) ... 2024-03-09T14:53:36.867 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up openjdk-8-jdk:amd64 (8u392-ga-1~22.04) ... 2024-03-09T14:53:36.959 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/appletviewer to provide /usr/bin/appletviewer (appletviewer) in auto mode 2024-03-09T14:53:37.052 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libgail-common:amd64 (2.24.33-2ubuntu2) ... 2024-03-09T14:53:37.180 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-09T14:53:38.221 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-09T14:53:38.314 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-09T14:53:38.607 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-09T14:53:39.103 INFO:teuthology.orchestra.run.smithi001.stdout: Gradle will be installed under /opt/gradle 2024-03-09T14:53:39.124 INFO:teuthology.orchestra.run.smithi001.stderr:--2024-03-09 14:53:39-- https://services.gradle.org/distributions/gradle-6.0.1-bin.zip 2024-03-09T14:53:39.127 INFO:teuthology.orchestra.run.smithi001.stderr:Resolving services.gradle.org (services.gradle.org)... 104.17.128.37, 104.17.129.37, 2606:4700::6811:8125, ... 2024-03-09T14:53:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:Connecting to services.gradle.org (services.gradle.org)|104.17.128.37|:443... connected. 2024-03-09T14:53:39.190 INFO:teuthology.orchestra.run.smithi001.stderr:HTTP request sent, awaiting response... 301 Moved Permanently 2024-03-09T14:53:39.191 INFO:teuthology.orchestra.run.smithi001.stderr:Location: https://github.com/gradle/gradle-distributions/releases/download/v6.0.1/gradle-6.0.1-bin.zip [following] 2024-03-09T14:53:39.191 INFO:teuthology.orchestra.run.smithi001.stderr:--2024-03-09 14:53:39-- https://github.com/gradle/gradle-distributions/releases/download/v6.0.1/gradle-6.0.1-bin.zip 2024-03-09T14:53:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:Resolving github.com (github.com)... 140.82.112.4 2024-03-09T14:53:39.203 INFO:teuthology.orchestra.run.smithi001.stderr:Connecting to github.com (github.com)|140.82.112.4|:443... connected. 2024-03-09T14:53:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:HTTP request sent, awaiting response... 302 Found 2024-03-09T14:53:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/696192900/8f418ba1-c336-448c-95c6-5180382113cb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240309%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240309T145339Z&X-Amz-Expires=300&X-Amz-Signature=4dae2007cfaee787af6f8aec81f133f500f61716f93e9effdf45639e84344372&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=696192900&response-content-disposition=attachment%3B%20filename%3Dgradle-6.0.1-bin.zip&response-content-type=application%2Foctet-stream [following] 2024-03-09T14:53:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:--2024-03-09 14:53:39-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/696192900/8f418ba1-c336-448c-95c6-5180382113cb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240309%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240309T145339Z&X-Amz-Expires=300&X-Amz-Signature=4dae2007cfaee787af6f8aec81f133f500f61716f93e9effdf45639e84344372&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=696192900&response-content-disposition=attachment%3B%20filename%3Dgradle-6.0.1-bin.zip&response-content-type=application%2Foctet-stream 2024-03-09T14:53:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.111.133, ... 2024-03-09T14:53:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected. 2024-03-09T14:53:39.436 INFO:teuthology.orchestra.run.smithi001.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-09T14:53:39.436 INFO:teuthology.orchestra.run.smithi001.stderr:Length: 95129672 (91M) [application/octet-stream] 2024-03-09T14:53:39.436 INFO:teuthology.orchestra.run.smithi001.stderr:Saving to: 'gradle-6.0.1-bin.zip' 2024-03-09T14:53:39.440 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-09T14:53:39.458 INFO:teuthology.orchestra.run.smithi001.stderr: 0K .......... .......... .......... .......... .......... 0% 2.28M 40s 2024-03-09T14:53:39.470 INFO:teuthology.orchestra.run.smithi001.stderr: 50K .......... .......... .......... .......... .......... 0% 4.01M 31s 2024-03-09T14:53:39.473 INFO:teuthology.orchestra.run.smithi001.stderr: 100K .......... .......... .......... .......... .......... 0% 13.9M 23s 2024-03-09T14:53:39.476 INFO:teuthology.orchestra.run.smithi001.stderr: 150K .......... .......... .......... .......... .......... 0% 14.7M 19s 2024-03-09T14:53:39.486 INFO:teuthology.orchestra.run.smithi001.stderr: 200K .......... .......... .......... .......... .......... 0% 5.43M 18s 2024-03-09T14:53:39.487 INFO:teuthology.orchestra.run.smithi001.stderr: 250K .......... .......... .......... .......... .......... 0% 39.4M 16s 2024-03-09T14:53:39.491 INFO:teuthology.orchestra.run.smithi001.stderr: 300K .......... .......... .......... .......... .......... 0% 21.5M 14s 2024-03-09T14:53:39.493 INFO:teuthology.orchestra.run.smithi001.stderr: 350K .......... .......... .......... .......... .......... 0% 34.7M 13s 2024-03-09T14:53:39.493 INFO:teuthology.orchestra.run.smithi001.stderr: 400K .......... .......... .......... .......... .......... 0% 25.4M 12s 2024-03-09T14:53:39.494 INFO:teuthology.orchestra.run.smithi001.stderr: 450K .......... .......... .......... .......... .......... 0% 42.1M 11s 2024-03-09T14:53:39.501 INFO:teuthology.orchestra.run.smithi001.stderr: 500K .......... .......... .......... .......... .......... 0% 7.06M 11s 2024-03-09T14:53:39.502 INFO:teuthology.orchestra.run.smithi001.stderr: 550K .......... .......... .......... .......... .......... 0% 43.0M 10s 2024-03-09T14:53:39.504 INFO:teuthology.orchestra.run.smithi001.stderr: 600K .......... .......... .......... .......... .......... 0% 31.7M 10s 2024-03-09T14:53:39.504 INFO:teuthology.orchestra.run.smithi001.stderr: 650K .......... .......... .......... .......... .......... 0% 94.7M 9s 2024-03-09T14:53:39.504 INFO:teuthology.orchestra.run.smithi001.stderr: 700K .......... .......... .......... .......... .......... 0% 66.1M 8s 2024-03-09T14:53:39.505 INFO:teuthology.orchestra.run.smithi001.stderr: 750K .......... .......... .......... .......... .......... 0% 48.4M 8s 2024-03-09T14:53:39.509 INFO:teuthology.orchestra.run.smithi001.stderr: 800K .......... .......... .......... .......... .......... 0% 38.7M 8s 2024-03-09T14:53:39.509 INFO:teuthology.orchestra.run.smithi001.stderr: 850K .......... .......... .......... .......... .......... 0% 84.3M 7s 2024-03-09T14:53:39.509 INFO:teuthology.orchestra.run.smithi001.stderr: 900K .......... .......... .......... .......... .......... 1% 119M 7s 2024-03-09T14:53:39.509 INFO:teuthology.orchestra.run.smithi001.stderr: 950K .......... .......... .......... .......... .......... 1% 56.6M 7s 2024-03-09T14:53:39.515 INFO:teuthology.orchestra.run.smithi001.stderr: 1000K .......... .......... .......... .......... .......... 1% 8.12M 7s 2024-03-09T14:53:39.516 INFO:teuthology.orchestra.run.smithi001.stderr: 1050K .......... .......... .......... .......... .......... 1% 52.2M 7s 2024-03-09T14:53:40.690 INFO:teuthology.orchestra.run.smithi001.stderr: 1100K .......... .......... .......... .......... .......... 1% 41.6M 6s 2024-03-09T14:53:40.690 INFO:teuthology.orchestra.run.smithi001.stderr: 1150K .......... .......... .......... .......... .......... 1% 54.8M 6s 2024-03-09T14:53:40.690 INFO:teuthology.orchestra.run.smithi001.stderr: 1200K .......... .......... .......... .......... .......... 1% 60.2M 6s 2024-03-09T14:53:40.690 INFO:teuthology.orchestra.run.smithi001.stderr: 1250K .......... .......... .......... .......... .......... 1% 55.2M 6s 2024-03-09T14:53:40.690 INFO:teuthology.orchestra.run.smithi001.stderr: 1300K .......... .......... .......... .......... .......... 1% 39.7M 6s 2024-03-09T14:53:40.690 INFO:teuthology.orchestra.run.smithi001.stderr: 1350K .......... .......... .......... .......... .......... 1% 36.2M 6s 2024-03-09T14:53:40.690 INFO:teuthology.orchestra.run.smithi001.stderr: 1400K .......... .......... .......... .......... .......... 1% 63.5M 5s 2024-03-09T14:53:40.690 INFO:teuthology.orchestra.run.smithi001.stderr: 1450K .......... .......... .......... .......... .......... 1% 75.5M 5s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 1500K .......... .......... .......... .......... .......... 1% 77.3M 5s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 1550K .......... .......... .......... .......... .......... 1% 75.4M 5s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 1600K .......... .......... .......... .......... .......... 1% 115M 5s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 1650K .......... .......... .......... .......... .......... 1% 156M 5s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 1700K .......... .......... .......... .......... .......... 1% 161M 5s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 1750K .......... .......... .......... .......... .......... 1% 159M 5s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 1800K .......... .......... .......... .......... .......... 1% 145M 4s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 1850K .......... .......... .......... .......... .......... 2% 161M 4s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 1900K .......... .......... .......... .......... .......... 2% 162M 4s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 1950K .......... .......... .......... .......... .......... 2% 161M 4s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 2000K .......... .......... .......... .......... .......... 2% 18.4M 4s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 2050K .......... .......... .......... .......... .......... 2% 104M 4s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 2100K .......... .......... .......... .......... .......... 2% 79.7M 4s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 2150K .......... .......... .......... .......... .......... 2% 77.8M 4s 2024-03-09T14:53:40.691 INFO:teuthology.orchestra.run.smithi001.stderr: 2200K .......... .......... .......... .......... .......... 2% 65.0M 4s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2250K .......... .......... .......... .......... .......... 2% 83.1M 4s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2300K .......... .......... .......... .......... .......... 2% 84.5M 4s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2350K .......... .......... .......... .......... .......... 2% 92.3M 4s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2400K .......... .......... .......... .......... .......... 2% 82.6M 4s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2450K .......... .......... .......... .......... .......... 2% 99.9M 4s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2500K .......... .......... .......... .......... .......... 2% 105M 4s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2550K .......... .......... .......... .......... .......... 2% 109M 3s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2600K .......... .......... .......... .......... .......... 2% 90.8M 3s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2650K .......... .......... .......... .......... .......... 2% 118M 3s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2700K .......... .......... .......... .......... .......... 2% 108M 3s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2750K .......... .......... .......... .......... .......... 3% 114M 3s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2800K .......... .......... .......... .......... .......... 3% 86.9M 3s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2850K .......... .......... .......... .......... .......... 3% 116M 3s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2900K .......... .......... .......... .......... .......... 3% 114M 3s 2024-03-09T14:53:40.692 INFO:teuthology.orchestra.run.smithi001.stderr: 2950K .......... .......... .......... .......... .......... 3% 107M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3000K .......... .......... .......... .......... .......... 3% 96.0M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3050K .......... .......... .......... .......... .......... 3% 111M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3100K .......... .......... .......... .......... .......... 3% 117M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3150K .......... .......... .......... .......... .......... 3% 106M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3200K .......... .......... .......... .......... .......... 3% 93.0M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3250K .......... .......... .......... .......... .......... 3% 115M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3300K .......... .......... .......... .......... .......... 3% 104M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3350K .......... .......... .......... .......... .......... 3% 109M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3400K .......... .......... .......... .......... .......... 3% 99.1M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3450K .......... .......... .......... .......... .......... 3% 105M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3500K .......... .......... .......... .......... .......... 3% 112M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3550K .......... .......... .......... .......... .......... 3% 104M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3600K .......... .......... .......... .......... .......... 3% 74.9M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3650K .......... .......... .......... .......... .......... 3% 110M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3700K .......... .......... .......... .......... .......... 4% 106M 3s 2024-03-09T14:53:40.693 INFO:teuthology.orchestra.run.smithi001.stderr: 3750K .......... .......... .......... .......... .......... 4% 104M 3s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 3800K .......... .......... .......... .......... .......... 4% 106M 3s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 3850K .......... .......... .......... .......... .......... 4% 101M 3s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 3900K .......... .......... .......... .......... .......... 4% 114M 3s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 3950K .......... .......... .......... .......... .......... 4% 106M 3s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 4000K .......... .......... .......... .......... .......... 4% 92.7M 2s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 4050K .......... .......... .......... .......... .......... 4% 70.0M 2s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 4100K .......... .......... .......... .......... .......... 4% 72.5M 2s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 4150K .......... .......... .......... .......... .......... 4% 72.6M 2s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 4200K .......... .......... .......... .......... .......... 4% 73.8M 2s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 4250K .......... .......... .......... .......... .......... 4% 62.2M 2s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 4300K .......... .......... .......... .......... .......... 4% 68.6M 2s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 4350K .......... .......... .......... .......... .......... 4% 75.8M 2s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 4400K .......... .......... .......... .......... .......... 4% 75.6M 2s 2024-03-09T14:53:40.694 INFO:teuthology.orchestra.run.smithi001.stderr: 4450K .......... .......... .......... .......... .......... 4% 66.8M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 4500K .......... .......... .......... .......... .......... 4% 71.3M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 4550K .......... .......... .......... .......... .......... 4% 71.5M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 4600K .......... .......... .......... .......... .......... 5% 74.0M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 4650K .......... .......... .......... .......... .......... 5% 59.6M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 4700K .......... .......... .......... .......... .......... 5% 72.2M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 4750K .......... .......... .......... .......... .......... 5% 77.9M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 4800K .......... .......... .......... .......... .......... 5% 89.5M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 4850K .......... .......... .......... .......... .......... 5% 61.1M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 4900K .......... .......... .......... .......... .......... 5% 67.9M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 4950K .......... .......... .......... .......... .......... 5% 69.7M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 5000K .......... .......... .......... .......... .......... 5% 79.4M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 5050K .......... .......... .......... .......... .......... 5% 52.2M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 5100K .......... .......... .......... .......... .......... 5% 70.5M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 5150K .......... .......... .......... .......... .......... 5% 72.4M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 5200K .......... .......... .......... .......... .......... 5% 86.2M 2s 2024-03-09T14:53:40.695 INFO:teuthology.orchestra.run.smithi001.stderr: 5250K .......... .......... .......... .......... .......... 5% 72.9M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5300K .......... .......... .......... .......... .......... 5% 86.4M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5350K .......... .......... .......... .......... .......... 5% 86.8M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5400K .......... .......... .......... .......... .......... 5% 79.4M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5450K .......... .......... .......... .......... .......... 5% 76.2M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5500K .......... .......... .......... .......... .......... 5% 90.7M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5550K .......... .......... .......... .......... .......... 6% 86.2M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5600K .......... .......... .......... .......... .......... 6% 90.3M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5650K .......... .......... .......... .......... .......... 6% 80.3M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5700K .......... .......... .......... .......... .......... 6% 84.2M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5750K .......... .......... .......... .......... .......... 6% 90.0M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5800K .......... .......... .......... .......... .......... 6% 93.7M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5850K .......... .......... .......... .......... .......... 6% 74.2M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5900K .......... .......... .......... .......... .......... 6% 98.0M 2s 2024-03-09T14:53:40.696 INFO:teuthology.orchestra.run.smithi001.stderr: 5950K .......... .......... .......... .......... .......... 6% 121M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6000K .......... .......... .......... .......... .......... 6% 96.7M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6050K .......... .......... .......... .......... .......... 6% 84.0M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6100K .......... .......... .......... .......... .......... 6% 84.2M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6150K .......... .......... .......... .......... .......... 6% 99.9M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6200K .......... .......... .......... .......... .......... 6% 100M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6250K .......... .......... .......... .......... .......... 6% 82.6M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6300K .......... .......... .......... .......... .......... 6% 101M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6350K .......... .......... .......... .......... .......... 6% 99.4M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6400K .......... .......... .......... .......... .......... 6% 101M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6450K .......... .......... .......... .......... .......... 6% 99.2M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6500K .......... .......... .......... .......... .......... 7% 101M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6550K .......... .......... .......... .......... .......... 7% 115M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6600K .......... .......... .......... .......... .......... 7% 111M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6650K .......... .......... .......... .......... .......... 7% 93.4M 2s 2024-03-09T14:53:40.697 INFO:teuthology.orchestra.run.smithi001.stderr: 6700K .......... .......... .......... .......... .......... 7% 113M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 6750K .......... .......... .......... .......... .......... 7% 112M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 6800K .......... .......... .......... .......... .......... 7% 113M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 6850K .......... .......... .......... .......... .......... 7% 96.5M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 6900K .......... .......... .......... .......... .......... 7% 113M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 6950K .......... .......... .......... .......... .......... 7% 93.5M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 7000K .......... .......... .......... .......... .......... 7% 119M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 7050K .......... .......... .......... .......... .......... 7% 95.4M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 7100K .......... .......... .......... .......... .......... 7% 111M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 7150K .......... .......... .......... .......... .......... 7% 111M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 7200K .......... .......... .......... .......... .......... 7% 112M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 7250K .......... .......... .......... .......... .......... 7% 89.8M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 7300K .......... .......... .......... .......... .......... 7% 107M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 7350K .......... .......... .......... .......... .......... 7% 94.1M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 7400K .......... .......... .......... .......... .......... 8% 113M 2s 2024-03-09T14:53:40.698 INFO:teuthology.orchestra.run.smithi001.stderr: 7450K .......... .......... .......... .......... .......... 8% 100M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 7500K .......... .......... .......... .......... .......... 8% 118M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 7550K .......... .......... .......... .......... .......... 8% 120M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 7600K .......... .......... .......... .......... .......... 8% 114M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 7650K .......... .......... .......... .......... .......... 8% 106M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 7700K .......... .......... .......... .......... .......... 8% 104M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 7750K .......... .......... .......... .......... .......... 8% 126M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 7800K .......... .......... .......... .......... .......... 8% 118M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 7850K .......... .......... .......... .......... .......... 8% 100M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 7900K .......... .......... .......... .......... .......... 8% 127M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 7950K .......... .......... .......... .......... .......... 8% 128M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 8000K .......... .......... .......... .......... .......... 8% 126M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 8050K .......... .......... .......... .......... .......... 8% 115M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 8100K .......... .......... .......... .......... .......... 8% 130M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 8150K .......... .......... .......... .......... .......... 8% 130M 2s 2024-03-09T14:53:40.699 INFO:teuthology.orchestra.run.smithi001.stderr: 8200K .......... .......... .......... .......... .......... 8% 129M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8250K .......... .......... .......... .......... .......... 8% 111M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8300K .......... .......... .......... .......... .......... 8% 114M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8350K .......... .......... .......... .......... .......... 9% 130M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8400K .......... .......... .......... .......... .......... 9% 132M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8450K .......... .......... .......... .......... .......... 9% 121M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8500K .......... .......... .......... .......... .......... 9% 133M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8550K .......... .......... .......... .......... .......... 9% 143M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8600K .......... .......... .......... .......... .......... 9% 131M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8650K .......... .......... .......... .......... .......... 9% 112M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8700K .......... .......... .......... .......... .......... 9% 132M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8750K .......... .......... .......... .......... .......... 9% 134M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8800K .......... .......... .......... .......... .......... 9% 139M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8850K .......... .......... .......... .......... .......... 9% 109M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8900K .......... .......... .......... .......... .......... 9% 137M 2s 2024-03-09T14:53:40.700 INFO:teuthology.orchestra.run.smithi001.stderr: 8950K .......... .......... .......... .......... .......... 9% 145M 2s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9000K .......... .......... .......... .......... .......... 9% 147M 2s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9050K .......... .......... .......... .......... .......... 9% 149M 2s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9100K .......... .......... .......... .......... .......... 9% 138M 2s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9150K .......... .......... .......... .......... .......... 9% 148M 2s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9200K .......... .......... .......... .......... .......... 9% 149M 2s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9250K .......... .......... .......... .......... .......... 10% 133M 2s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9300K .......... .......... .......... .......... .......... 10% 144M 1s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9350K .......... .......... .......... .......... .......... 10% 143M 1s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9400K .......... .......... .......... .......... .......... 10% 152M 1s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9450K .......... .......... .......... .......... .......... 10% 122M 1s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9500K .......... .......... .......... .......... .......... 10% 149M 1s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9550K .......... .......... .......... .......... .......... 10% 154M 1s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9600K .......... .......... .......... .......... .......... 10% 145M 1s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9650K .......... .......... .......... .......... .......... 10% 138M 1s 2024-03-09T14:53:40.701 INFO:teuthology.orchestra.run.smithi001.stderr: 9700K .......... .......... .......... .......... .......... 10% 157M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 9750K .......... .......... .......... .......... .......... 10% 158M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 9800K .......... .......... .......... .......... .......... 10% 152M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 9850K .......... .......... .......... .......... .......... 10% 126M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 9900K .......... .......... .......... .......... .......... 10% 155M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 9950K .......... .......... .......... .......... .......... 10% 151M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 10000K .......... .......... .......... .......... .......... 10% 141M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 10050K .......... .......... .......... .......... .......... 10% 138M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 10100K .......... .......... .......... .......... .......... 10% 146M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 10150K .......... .......... .......... .......... .......... 10% 153M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 10200K .......... .......... .......... .......... .......... 11% 157M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 10250K .......... .......... .......... .......... .......... 11% 139M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 10300K .......... .......... .......... .......... .......... 11% 158M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 10350K .......... .......... .......... .......... .......... 11% 144M 1s 2024-03-09T14:53:40.702 INFO:teuthology.orchestra.run.smithi001.stderr: 10400K .......... .......... .......... .......... .......... 11% 148M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 10450K .......... .......... .......... .......... .......... 11% 145M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 10500K .......... .......... .......... .......... .......... 11% 157M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 10550K .......... .......... .......... .......... .......... 11% 164M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 10600K .......... .......... .......... .......... .......... 11% 153M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 10650K .......... .......... .......... .......... .......... 11% 138M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 10700K .......... .......... .......... .......... .......... 11% 158M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 10750K .......... .......... .......... .......... .......... 11% 173M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 10800K .......... .......... .......... .......... .......... 11% 177M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 10850K .......... .......... .......... .......... .......... 11% 148M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 10900K .......... .......... .......... .......... .......... 11% 176M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 10950K .......... .......... .......... .......... .......... 11% 171M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 11000K .......... .......... .......... .......... .......... 11% 171M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 11050K .......... .......... .......... .......... .......... 11% 142M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 11100K .......... .......... .......... .......... .......... 12% 164M 1s 2024-03-09T14:53:40.703 INFO:teuthology.orchestra.run.smithi001.stderr: 11150K .......... .......... .......... .......... .......... 12% 174M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11200K .......... .......... .......... .......... .......... 12% 162M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11250K .......... .......... .......... .......... .......... 12% 129M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11300K .......... .......... .......... .......... .......... 12% 172M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11350K .......... .......... .......... .......... .......... 12% 177M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11400K .......... .......... .......... .......... .......... 12% 180M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11450K .......... .......... .......... .......... .......... 12% 150M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11500K .......... .......... .......... .......... .......... 12% 180M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11550K .......... .......... .......... .......... .......... 12% 177M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11600K .......... .......... .......... .......... .......... 12% 174M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11650K .......... .......... .......... .......... .......... 12% 162M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11700K .......... .......... .......... .......... .......... 12% 179M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11750K .......... .......... .......... .......... .......... 12% 177M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11800K .......... .......... .......... .......... .......... 12% 153M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11850K .......... .......... .......... .......... .......... 12% 145M 1s 2024-03-09T14:53:40.704 INFO:teuthology.orchestra.run.smithi001.stderr: 11900K .......... .......... .......... .......... .......... 12% 182M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 11950K .......... .......... .......... .......... .......... 12% 158M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12000K .......... .......... .......... .......... .......... 12% 179M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12050K .......... .......... .......... .......... .......... 13% 155M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12100K .......... .......... .......... .......... .......... 13% 177M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12150K .......... .......... .......... .......... .......... 13% 191M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12200K .......... .......... .......... .......... .......... 13% 189M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12250K .......... .......... .......... .......... .......... 13% 158M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12300K .......... .......... .......... .......... .......... 13% 189M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12350K .......... .......... .......... .......... .......... 13% 200M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12400K .......... .......... .......... .......... .......... 13% 180M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12450K .......... .......... .......... .......... .......... 13% 167M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12500K .......... .......... .......... .......... .......... 13% 191M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12550K .......... .......... .......... .......... .......... 13% 192M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12600K .......... .......... .......... .......... .......... 13% 179M 1s 2024-03-09T14:53:40.705 INFO:teuthology.orchestra.run.smithi001.stderr: 12650K .......... .......... .......... .......... .......... 13% 99.4M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 12700K .......... .......... .......... .......... .......... 13% 107M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 12750K .......... .......... .......... .......... .......... 13% 118M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 12800K .......... .......... .......... .......... .......... 13% 114M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 12850K .......... .......... .......... .......... .......... 13% 110M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 12900K .......... .......... .......... .......... .......... 13% 103M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 12950K .......... .......... .......... .......... .......... 13% 123M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 13000K .......... .......... .......... .......... .......... 14% 110M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 13050K .......... .......... .......... .......... .......... 14% 94.8M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 13100K .......... .......... .......... .......... .......... 14% 112M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 13150K .......... .......... .......... .......... .......... 14% 115M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 13200K .......... .......... .......... .......... .......... 14% 118M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 13250K .......... .......... .......... .......... .......... 14% 105M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 13300K .......... .......... .......... .......... .......... 14% 116M 1s 2024-03-09T14:53:40.706 INFO:teuthology.orchestra.run.smithi001.stderr: 13350K .......... .......... .......... .......... .......... 14% 109M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 13400K .......... .......... .......... .......... .......... 14% 121M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 13450K .......... .......... .......... .......... .......... 14% 87.1M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 13500K .......... .......... .......... .......... .......... 14% 121M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 13550K .......... .......... .......... .......... .......... 14% 114M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 13600K .......... .......... .......... .......... .......... 14% 120M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 13650K .......... .......... .......... .......... .......... 14% 109M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 13700K .......... .......... .......... .......... .......... 14% 106M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 13750K .......... .......... .......... .......... .......... 14% 120M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 13800K .......... .......... .......... .......... .......... 14% 115M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 13850K .......... .......... .......... .......... .......... 14% 93.4M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 13900K .......... .......... .......... .......... .......... 15% 115M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 13950K .......... .......... .......... .......... .......... 15% 114M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 14000K .......... .......... .......... .......... .......... 15% 120M 1s 2024-03-09T14:53:40.707 INFO:teuthology.orchestra.run.smithi001.stderr: 14050K .......... .......... .......... .......... .......... 15% 107M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14100K .......... .......... .......... .......... .......... 15% 117M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14150K .......... .......... .......... .......... .......... 15% 113M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14200K .......... .......... .......... .......... .......... 15% 118M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14250K .......... .......... .......... .......... .......... 15% 90.9M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14300K .......... .......... .......... .......... .......... 15% 118M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14350K .......... .......... .......... .......... .......... 15% 117M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14400K .......... .......... .......... .......... .......... 15% 115M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14450K .......... .......... .......... .......... .......... 15% 106M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14500K .......... .......... .......... .......... .......... 15% 116M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14550K .......... .......... .......... .......... .......... 15% 117M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14600K .......... .......... .......... .......... .......... 15% 112M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14650K .......... .......... .......... .......... .......... 15% 89.4M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14700K .......... .......... .......... .......... .......... 15% 114M 1s 2024-03-09T14:53:40.708 INFO:teuthology.orchestra.run.smithi001.stderr: 14750K .......... .......... .......... .......... .......... 15% 118M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 14800K .......... .......... .......... .......... .......... 15% 114M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 14850K .......... .......... .......... .......... .......... 16% 98.7M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 14900K .......... .......... .......... .......... .......... 16% 116M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 14950K .......... .......... .......... .......... .......... 16% 119M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 15000K .......... .......... .......... .......... .......... 16% 117M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 15050K .......... .......... .......... .......... .......... 16% 90.5M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 15100K .......... .......... .......... .......... .......... 16% 112M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 15150K .......... .......... .......... .......... .......... 16% 119M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 15200K .......... .......... .......... .......... .......... 16% 117M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 15250K .......... .......... .......... .......... .......... 16% 109M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 15300K .......... .......... .......... .......... .......... 16% 112M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 15350K .......... .......... .......... .......... .......... 16% 115M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 15400K .......... .......... .......... .......... .......... 16% 114M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 15450K .......... .......... .......... .......... .......... 16% 93.1M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 15500K .......... .......... .......... .......... .......... 16% 117M 1s 2024-03-09T14:53:40.709 INFO:teuthology.orchestra.run.smithi001.stderr: 15550K .......... .......... .......... .......... .......... 16% 112M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 15600K .......... .......... .......... .......... .......... 16% 116M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 15650K .......... .......... .......... .......... .......... 16% 108M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 15700K .......... .......... .......... .......... .......... 16% 116M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 15750K .......... .......... .......... .......... .......... 17% 117M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 15800K .......... .......... .......... .......... .......... 17% 104M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 15850K .......... .......... .......... .......... .......... 17% 102M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 15900K .......... .......... .......... .......... .......... 17% 115M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 15950K .......... .......... .......... .......... .......... 17% 112M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 16000K .......... .......... .......... .......... .......... 17% 117M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 16050K .......... .......... .......... .......... .......... 17% 107M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 16100K .......... .......... .......... .......... .......... 17% 117M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 16150K .......... .......... .......... .......... .......... 17% 117M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 16200K .......... .......... .......... .......... .......... 17% 107M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 16250K .......... .......... .......... .......... .......... 17% 92.8M 1s 2024-03-09T14:53:40.710 INFO:teuthology.orchestra.run.smithi001.stderr: 16300K .......... .......... .......... .......... .......... 17% 106M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16350K .......... .......... .......... .......... .......... 17% 103M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16400K .......... .......... .......... .......... .......... 17% 117M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16450K .......... .......... .......... .......... .......... 17% 101M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16500K .......... .......... .......... .......... .......... 17% 117M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16550K .......... .......... .......... .......... .......... 17% 115M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16600K .......... .......... .......... .......... .......... 17% 119M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16650K .......... .......... .......... .......... .......... 17% 86.4M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16700K .......... .......... .......... .......... .......... 18% 123M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16750K .......... .......... .......... .......... .......... 18% 115M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16800K .......... .......... .......... .......... .......... 18% 116M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16850K .......... .......... .......... .......... .......... 18% 110M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16900K .......... .......... .......... .......... .......... 18% 113M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 16950K .......... .......... .......... .......... .......... 18% 116M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 17000K .......... .......... .......... .......... .......... 18% 118M 1s 2024-03-09T14:53:40.711 INFO:teuthology.orchestra.run.smithi001.stderr: 17050K .......... .......... .......... .......... .......... 18% 87.2M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17100K .......... .......... .......... .......... .......... 18% 119M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17150K .......... .......... .......... .......... .......... 18% 105M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17200K .......... .......... .......... .......... .......... 18% 126M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17250K .......... .......... .......... .......... .......... 18% 104M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17300K .......... .......... .......... .......... .......... 18% 115M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17350K .......... .......... .......... .......... .......... 18% 113M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17400K .......... .......... .......... .......... .......... 18% 119M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17450K .......... .......... .......... .......... .......... 18% 92.3M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17500K .......... .......... .......... .......... .......... 18% 113M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17550K .......... .......... .......... .......... .......... 18% 122M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17600K .......... .......... .......... .......... .......... 18% 113M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17650K .......... .......... .......... .......... .......... 19% 103M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17700K .......... .......... .......... .......... .......... 19% 121M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17750K .......... .......... .......... .......... .......... 19% 116M 1s 2024-03-09T14:53:40.712 INFO:teuthology.orchestra.run.smithi001.stderr: 17800K .......... .......... .......... .......... .......... 19% 113M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 17850K .......... .......... .......... .......... .......... 19% 90.1M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 17900K .......... .......... .......... .......... .......... 19% 121M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 17950K .......... .......... .......... .......... .......... 19% 109M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18000K .......... .......... .......... .......... .......... 19% 120M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18050K .......... .......... .......... .......... .......... 19% 98.3M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18100K .......... .......... .......... .......... .......... 19% 119M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18150K .......... .......... .......... .......... .......... 19% 116M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18200K .......... .......... .......... .......... .......... 19% 122M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18250K .......... .......... .......... .......... .......... 19% 88.7M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18300K .......... .......... .......... .......... .......... 19% 115M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18350K .......... .......... .......... .......... .......... 19% 119M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18400K .......... .......... .......... .......... .......... 19% 111M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18450K .......... .......... .......... .......... .......... 19% 112M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18500K .......... .......... .......... .......... .......... 19% 108M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18550K .......... .......... .......... .......... .......... 20% 122M 1s 2024-03-09T14:53:40.713 INFO:teuthology.orchestra.run.smithi001.stderr: 18600K .......... .......... .......... .......... .......... 20% 111M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 18650K .......... .......... .......... .......... .......... 20% 94.4M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 18700K .......... .......... .......... .......... .......... 20% 120M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 18750K .......... .......... .......... .......... .......... 20% 111M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 18800K .......... .......... .......... .......... .......... 20% 118M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 18850K .......... .......... .......... .......... .......... 20% 100M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 18900K .......... .......... .......... .......... .......... 20% 111M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 18950K .......... .......... .......... .......... .......... 20% 121M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 19000K .......... .......... .......... .......... .......... 20% 115M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 19050K .......... .......... .......... .......... .......... 20% 92.0M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 19100K .......... .......... .......... .......... .......... 20% 112M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 19150K .......... .......... .......... .......... .......... 20% 111M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 19200K .......... .......... .......... .......... .......... 20% 98.9M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 19250K .......... .......... .......... .......... .......... 20% 121M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 19300K .......... .......... .......... .......... .......... 20% 135M 1s 2024-03-09T14:53:40.714 INFO:teuthology.orchestra.run.smithi001.stderr: 19350K .......... .......... .......... .......... .......... 20% 121M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 19400K .......... .......... .......... .......... .......... 20% 106M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 19450K .......... .......... .......... .......... .......... 20% 92.5M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 19500K .......... .......... .......... .......... .......... 21% 121M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 19550K .......... .......... .......... .......... .......... 21% 113M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 19600K .......... .......... .......... .......... .......... 21% 117M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 19650K .......... .......... .......... .......... .......... 21% 109M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 19700K .......... .......... .......... .......... .......... 21% 117M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 19750K .......... .......... .......... .......... .......... 21% 117M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 19800K .......... .......... .......... .......... .......... 21% 114M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 19850K .......... .......... .......... .......... .......... 21% 94.8M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 19900K .......... .......... .......... .......... .......... 21% 112M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 19950K .......... .......... .......... .......... .......... 21% 119M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 20000K .......... .......... .......... .......... .......... 21% 108M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 20050K .......... .......... .......... .......... .......... 21% 111M 1s 2024-03-09T14:53:40.715 INFO:teuthology.orchestra.run.smithi001.stderr: 20100K .......... .......... .......... .......... .......... 21% 114M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20150K .......... .......... .......... .......... .......... 21% 119M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20200K .......... .......... .......... .......... .......... 21% 111M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20250K .......... .......... .......... .......... .......... 21% 94.1M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20300K .......... .......... .......... .......... .......... 21% 120M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20350K .......... .......... .......... .......... .......... 21% 109M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20400K .......... .......... .......... .......... .......... 22% 124M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20450K .......... .......... .......... .......... .......... 22% 97.3M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20500K .......... .......... .......... .......... .......... 22% 123M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20550K .......... .......... .......... .......... .......... 22% 114M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20600K .......... .......... .......... .......... .......... 22% 118M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20650K .......... .......... .......... .......... .......... 22% 90.8M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20700K .......... .......... .......... .......... .......... 22% 115M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20750K .......... .......... .......... .......... .......... 22% 119M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20800K .......... .......... .......... .......... .......... 22% 109M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20850K .......... .......... .......... .......... .......... 22% 113M 1s 2024-03-09T14:53:40.716 INFO:teuthology.orchestra.run.smithi001.stderr: 20900K .......... .......... .......... .......... .......... 22% 115M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 20950K .......... .......... .......... .......... .......... 22% 115M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21000K .......... .......... .......... .......... .......... 22% 117M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21050K .......... .......... .......... .......... .......... 22% 88.9M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21100K .......... .......... .......... .......... .......... 22% 119M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21150K .......... .......... .......... .......... .......... 22% 119M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21200K .......... .......... .......... .......... .......... 22% 113M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21250K .......... .......... .......... .......... .......... 22% 109M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21300K .......... .......... .......... .......... .......... 22% 110M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21350K .......... .......... .......... .......... .......... 23% 112M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21400K .......... .......... .......... .......... .......... 23% 118M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21450K .......... .......... .......... .......... .......... 23% 91.8M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21500K .......... .......... .......... .......... .......... 23% 119M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21550K .......... .......... .......... .......... .......... 23% 111M 1s 2024-03-09T14:53:40.717 INFO:teuthology.orchestra.run.smithi001.stderr: 21600K .......... .......... .......... .......... .......... 23% 110M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 21650K .......... .......... .......... .......... .......... 23% 106M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 21700K .......... .......... .......... .......... .......... 23% 116M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 21750K .......... .......... .......... .......... .......... 23% 116M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 21800K .......... .......... .......... .......... .......... 23% 113M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 21850K .......... .......... .......... .......... .......... 23% 93.5M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 21900K .......... .......... .......... .......... .......... 23% 117M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 21950K .......... .......... .......... .......... .......... 23% 116M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 22000K .......... .......... .......... .......... .......... 23% 117M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 22050K .......... .......... .......... .......... .......... 23% 103M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 22100K .......... .......... .......... .......... .......... 23% 117M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 22150K .......... .......... .......... .......... .......... 23% 116M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 22200K .......... .......... .......... .......... .......... 23% 116M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 22250K .......... .......... .......... .......... .......... 24% 92.0M 1s 2024-03-09T14:53:40.718 INFO:teuthology.orchestra.run.smithi001.stderr: 22300K .......... .......... .......... .......... .......... 24% 115M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22350K .......... .......... .......... .......... .......... 24% 117M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22400K .......... .......... .......... .......... .......... 24% 113M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22450K .......... .......... .......... .......... .......... 24% 104M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22500K .......... .......... .......... .......... .......... 24% 119M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22550K .......... .......... .......... .......... .......... 24% 116M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22600K .......... .......... .......... .......... .......... 24% 113M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22650K .......... .......... .......... .......... .......... 24% 91.7M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22700K .......... .......... .......... .......... .......... 24% 117M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22750K .......... .......... .......... .......... .......... 24% 116M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22800K .......... .......... .......... .......... .......... 24% 108M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22850K .......... .......... .......... .......... .......... 24% 113M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22900K .......... .......... .......... .......... .......... 24% 114M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 22950K .......... .......... .......... .......... .......... 24% 116M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 23000K .......... .......... .......... .......... .......... 24% 114M 1s 2024-03-09T14:53:40.719 INFO:teuthology.orchestra.run.smithi001.stderr: 23050K .......... .......... .......... .......... .......... 24% 92.5M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23100K .......... .......... .......... .......... .......... 24% 118M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23150K .......... .......... .......... .......... .......... 24% 114M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23200K .......... .......... .......... .......... .......... 25% 115M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23250K .......... .......... .......... .......... .......... 25% 85.5M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23300K .......... .......... .......... .......... .......... 25% 111M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23350K .......... .......... .......... .......... .......... 25% 111M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23400K .......... .......... .......... .......... .......... 25% 120M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23450K .......... .......... .......... .......... .......... 25% 90.0M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23500K .......... .......... .......... .......... .......... 25% 116M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23550K .......... .......... .......... .......... .......... 25% 114M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23600K .......... .......... .......... .......... .......... 25% 123M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23650K .......... .......... .......... .......... .......... 25% 98.8M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23700K .......... .......... .......... .......... .......... 25% 122M 1s 2024-03-09T14:53:40.720 INFO:teuthology.orchestra.run.smithi001.stderr: 23750K .......... .......... .......... .......... .......... 25% 114M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 23800K .......... .......... .......... .......... .......... 25% 122M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 23850K .......... .......... .......... .......... .......... 25% 92.0M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 23900K .......... .......... .......... .......... .......... 25% 112M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 23950K .......... .......... .......... .......... .......... 25% 114M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 24000K .......... .......... .......... .......... .......... 25% 103M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 24050K .......... .......... .......... .......... .......... 25% 119M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 24100K .......... .......... .......... .......... .......... 25% 113M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 24150K .......... .......... .......... .......... .......... 26% 117M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 24200K .......... .......... .......... .......... .......... 26% 114M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 24250K .......... .......... .......... .......... .......... 26% 90.7M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 24300K .......... .......... .......... .......... .......... 26% 119M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 24350K .......... .......... .......... .......... .......... 26% 93.8M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 24400K .......... .......... .......... .......... .......... 26% 85.1M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 24450K .......... .......... .......... .......... .......... 26% 104M 1s 2024-03-09T14:53:40.721 INFO:teuthology.orchestra.run.smithi001.stderr: 24500K .......... .......... .......... .......... .......... 26% 121M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 24550K .......... .......... .......... .......... .......... 26% 110M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 24600K .......... .......... .......... .......... .......... 26% 118M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 24650K .......... .......... .......... .......... .......... 26% 93.0M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 24700K .......... .......... .......... .......... .......... 26% 111M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 24750K .......... .......... .......... .......... .......... 26% 123M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 24800K .......... .......... .......... .......... .......... 26% 110M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 24850K .......... .......... .......... .......... .......... 26% 102M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 24900K .......... .......... .......... .......... .......... 26% 119M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 24950K .......... .......... .......... .......... .......... 26% 115M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 25000K .......... .......... .......... .......... .......... 26% 104M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 25050K .......... .......... .......... .......... .......... 27% 91.0M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 25100K .......... .......... .......... .......... .......... 27% 120M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 25150K .......... .......... .......... .......... .......... 27% 117M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 25200K .......... .......... .......... .......... .......... 27% 115M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 25250K .......... .......... .......... .......... .......... 27% 101M 1s 2024-03-09T14:53:40.722 INFO:teuthology.orchestra.run.smithi001.stderr: 25300K .......... .......... .......... .......... .......... 27% 123M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25350K .......... .......... .......... .......... .......... 27% 113M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25400K .......... .......... .......... .......... .......... 27% 122M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25450K .......... .......... .......... .......... .......... 27% 92.8M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25500K .......... .......... .......... .......... .......... 27% 111M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25550K .......... .......... .......... .......... .......... 27% 117M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25600K .......... .......... .......... .......... .......... 27% 115M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25650K .......... .......... .......... .......... .......... 27% 108M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25700K .......... .......... .......... .......... .......... 27% 116M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25750K .......... .......... .......... .......... .......... 27% 110M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25800K .......... .......... .......... .......... .......... 27% 116M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25850K .......... .......... .......... .......... .......... 27% 90.9M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25900K .......... .......... .......... .......... .......... 27% 121M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 25950K .......... .......... .......... .......... .......... 27% 113M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 26000K .......... .......... .......... .......... .......... 28% 118M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 26050K .......... .......... .......... .......... .......... 28% 102M 1s 2024-03-09T14:53:40.723 INFO:teuthology.orchestra.run.smithi001.stderr: 26100K .......... .......... .......... .......... .......... 28% 119M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26150K .......... .......... .......... .......... .......... 28% 113M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26200K .......... .......... .......... .......... .......... 28% 120M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26250K .......... .......... .......... .......... .......... 28% 89.0M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26300K .......... .......... .......... .......... .......... 28% 111M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26350K .......... .......... .......... .......... .......... 28% 113M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26400K .......... .......... .......... .......... .......... 28% 121M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26450K .......... .......... .......... .......... .......... 28% 87.3M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26500K .......... .......... .......... .......... .......... 28% 150M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26550K .......... .......... .......... .......... .......... 28% 117M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26600K .......... .......... .......... .......... .......... 28% 114M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26650K .......... .......... .......... .......... .......... 28% 91.1M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26700K .......... .......... .......... .......... .......... 28% 114M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26750K .......... .......... .......... .......... .......... 28% 113M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26800K .......... .......... .......... .......... .......... 28% 122M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26850K .......... .......... .......... .......... .......... 28% 105M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26900K .......... .......... .......... .......... .......... 29% 112M 1s 2024-03-09T14:53:40.724 INFO:teuthology.orchestra.run.smithi001.stderr: 26950K .......... .......... .......... .......... .......... 29% 118M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27000K .......... .......... .......... .......... .......... 29% 117M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27050K .......... .......... .......... .......... .......... 29% 90.2M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27100K .......... .......... .......... .......... .......... 29% 112M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27150K .......... .......... .......... .......... .......... 29% 121M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27200K .......... .......... .......... .......... .......... 29% 116M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27250K .......... .......... .......... .......... .......... 29% 106M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27300K .......... .......... .......... .......... .......... 29% 114M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27350K .......... .......... .......... .......... .......... 29% 120M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27400K .......... .......... .......... .......... .......... 29% 116M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27450K .......... .......... .......... .......... .......... 29% 89.0M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27500K .......... .......... .......... .......... .......... 29% 120M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27550K .......... .......... .......... .......... .......... 29% 113M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27600K .......... .......... .......... .......... .......... 29% 118M 1s 2024-03-09T14:53:40.725 INFO:teuthology.orchestra.run.smithi001.stderr: 27650K .......... .......... .......... .......... .......... 29% 105M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 27700K .......... .......... .......... .......... .......... 29% 116M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 27750K .......... .......... .......... .......... .......... 29% 114M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 27800K .......... .......... .......... .......... .......... 29% 111M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 27850K .......... .......... .......... .......... .......... 30% 92.9M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 27900K .......... .......... .......... .......... .......... 30% 117M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 27950K .......... .......... .......... .......... .......... 30% 122M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 28000K .......... .......... .......... .......... .......... 30% 110M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 28050K .......... .......... .......... .......... .......... 30% 108M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 28100K .......... .......... .......... .......... .......... 30% 110M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 28150K .......... .......... .......... .......... .......... 30% 118M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 28200K .......... .......... .......... .......... .......... 30% 117M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 28250K .......... .......... .......... .......... .......... 30% 91.2M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 28300K .......... .......... .......... .......... .......... 30% 120M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 28350K .......... .......... .......... .......... .......... 30% 114M 1s 2024-03-09T14:53:40.726 INFO:teuthology.orchestra.run.smithi001.stderr: 28400K .......... .......... .......... .......... .......... 30% 112M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 28450K .......... .......... .......... .......... .......... 30% 90.2M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 28500K .......... .......... .......... .......... .......... 30% 153M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 28550K .......... .......... .......... .......... .......... 30% 115M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 28600K .......... .......... .......... .......... .......... 30% 110M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 28650K .......... .......... .......... .......... .......... 30% 93.7M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 28700K .......... .......... .......... .......... .......... 30% 118M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 28750K .......... .......... .......... .......... .......... 31% 119M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 28800K .......... .......... .......... .......... .......... 31% 103M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 28850K .......... .......... .......... .......... .......... 31% 116M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 28900K .......... .......... .......... .......... .......... 31% 111M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 28950K .......... .......... .......... .......... .......... 31% 111M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 29000K .......... .......... .......... .......... .......... 31% 116M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 29050K .......... .......... .......... .......... .......... 31% 91.3M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 29100K .......... .......... .......... .......... .......... 31% 117M 1s 2024-03-09T14:53:40.727 INFO:teuthology.orchestra.run.smithi001.stderr: 29150K .......... .......... .......... .......... .......... 31% 114M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29200K .......... .......... .......... .......... .......... 31% 108M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29250K .......... .......... .......... .......... .......... 31% 94.7M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29300K .......... .......... .......... .......... .......... 31% 113M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29350K .......... .......... .......... .......... .......... 31% 120M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29400K .......... .......... .......... .......... .......... 31% 117M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29450K .......... .......... .......... .......... .......... 31% 91.4M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29500K .......... .......... .......... .......... .......... 31% 114M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29550K .......... .......... .......... .......... .......... 31% 118M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29600K .......... .......... .......... .......... .......... 31% 113M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29650K .......... .......... .......... .......... .......... 31% 109M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29700K .......... .......... .......... .......... .......... 32% 112M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29750K .......... .......... .......... .......... .......... 32% 119M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29800K .......... .......... .......... .......... .......... 32% 116M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29850K .......... .......... .......... .......... .......... 32% 93.0M 1s 2024-03-09T14:53:40.728 INFO:teuthology.orchestra.run.smithi001.stderr: 29900K .......... .......... .......... .......... .......... 32% 114M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 29950K .......... .......... .......... .......... .......... 32% 116M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30000K .......... .......... .......... .......... .......... 32% 115M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30050K .......... .......... .......... .......... .......... 32% 105M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30100K .......... .......... .......... .......... .......... 32% 117M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30150K .......... .......... .......... .......... .......... 32% 112M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30200K .......... .......... .......... .......... .......... 32% 117M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30250K .......... .......... .......... .......... .......... 32% 93.3M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30300K .......... .......... .......... .......... .......... 32% 108M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30350K .......... .......... .......... .......... .......... 32% 121M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30400K .......... .......... .......... .......... .......... 32% 113M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30450K .......... .......... .......... .......... .......... 32% 105M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30500K .......... .......... .......... .......... .......... 32% 117M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30550K .......... .......... .......... .......... .......... 32% 120M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30600K .......... .......... .......... .......... .......... 32% 104M 1s 2024-03-09T14:53:40.729 INFO:teuthology.orchestra.run.smithi001.stderr: 30650K .......... .......... .......... .......... .......... 33% 95.3M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 30700K .......... .......... .......... .......... .......... 33% 122M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 30750K .......... .......... .......... .......... .......... 33% 112M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 30800K .......... .......... .......... .......... .......... 33% 121M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 30850K .......... .......... .......... .......... .......... 33% 104M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 30900K .......... .......... .......... .......... .......... 33% 117M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 30950K .......... .......... .......... .......... .......... 33% 116M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 31000K .......... .......... .......... .......... .......... 33% 119M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 31050K .......... .......... .......... .......... .......... 33% 80.0M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 31100K .......... .......... .......... .......... .......... 33% 110M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 31150K .......... .......... .......... .......... .......... 33% 101M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 31200K .......... .......... .......... .......... .......... 33% 115M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 31250K .......... .......... .......... .......... .......... 33% 102M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 31300K .......... .......... .......... .......... .......... 33% 124M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 31350K .......... .......... .......... .......... .......... 33% 120M 1s 2024-03-09T14:53:40.730 INFO:teuthology.orchestra.run.smithi001.stderr: 31400K .......... .......... .......... .......... .......... 33% 112M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 31450K .......... .......... .......... .......... .......... 33% 93.6M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 31500K .......... .......... .......... .......... .......... 33% 118M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 31550K .......... .......... .......... .......... .......... 34% 107M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 31600K .......... .......... .......... .......... .......... 34% 124M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 31650K .......... .......... .......... .......... .......... 34% 101M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 31700K .......... .......... .......... .......... .......... 34% 114M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 31750K .......... .......... .......... .......... .......... 34% 117M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 31800K .......... .......... .......... .......... .......... 34% 115M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 31850K .......... .......... .......... .......... .......... 34% 88.1M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 31900K .......... .......... .......... .......... .......... 34% 116M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 31950K .......... .......... .......... .......... .......... 34% 119M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 32000K .......... .......... .......... .......... .......... 34% 109M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 32050K .......... .......... .......... .......... .......... 34% 108M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 32100K .......... .......... .......... .......... .......... 34% 116M 1s 2024-03-09T14:53:40.731 INFO:teuthology.orchestra.run.smithi001.stderr: 32150K .......... .......... .......... .......... .......... 34% 115M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32200K .......... .......... .......... .......... .......... 34% 117M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32250K .......... .......... .......... .......... .......... 34% 86.1M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32300K .......... .......... .......... .......... .......... 34% 79.3M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32350K .......... .......... .......... .......... .......... 34% 169M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32400K .......... .......... .......... .......... .......... 34% 149M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32450K .......... .......... .......... .......... .......... 34% 109M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32500K .......... .......... .......... .......... .......... 35% 110M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32550K .......... .......... .......... .......... .......... 35% 109M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32600K .......... .......... .......... .......... .......... 35% 72.0M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32650K .......... .......... .......... .......... .......... 35% 173M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32700K .......... .......... .......... .......... .......... 35% 117M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32750K .......... .......... .......... .......... .......... 35% 114M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32800K .......... .......... .......... .......... .......... 35% 114M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32850K .......... .......... .......... .......... .......... 35% 92.5M 1s 2024-03-09T14:53:40.732 INFO:teuthology.orchestra.run.smithi001.stderr: 32900K .......... .......... .......... .......... .......... 35% 130M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 32950K .......... .......... .......... .......... .......... 35% 118M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33000K .......... .......... .......... .......... .......... 35% 114M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33050K .......... .......... .......... .......... .......... 35% 87.5M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33100K .......... .......... .......... .......... .......... 35% 114M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33150K .......... .......... .......... .......... .......... 35% 117M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33200K .......... .......... .......... .......... .......... 35% 118M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33250K .......... .......... .......... .......... .......... 35% 111M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33300K .......... .......... .......... .......... .......... 35% 111M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33350K .......... .......... .......... .......... .......... 35% 118M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33400K .......... .......... .......... .......... .......... 36% 108M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33450K .......... .......... .......... .......... .......... 36% 73.0M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33500K .......... .......... .......... .......... .......... 36% 94.7M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33550K .......... .......... .......... .......... .......... 36% 101M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33600K .......... .......... .......... .......... .......... 36% 113M 1s 2024-03-09T14:53:40.733 INFO:teuthology.orchestra.run.smithi001.stderr: 33650K .......... .......... .......... .......... .......... 36% 104M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 33700K .......... .......... .......... .......... .......... 36% 120M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 33750K .......... .......... .......... .......... .......... 36% 122M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 33800K .......... .......... .......... .......... .......... 36% 110M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 33850K .......... .......... .......... .......... .......... 36% 92.6M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 33900K .......... .......... .......... .......... .......... 36% 112M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 33950K .......... .......... .......... .......... .......... 36% 110M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 34000K .......... .......... .......... .......... .......... 36% 110M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 34050K .......... .......... .......... .......... .......... 36% 108M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 34100K .......... .......... .......... .......... .......... 36% 110M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 34150K .......... .......... .......... .......... .......... 36% 116M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 34200K .......... .......... .......... .......... .......... 36% 121M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 34250K .......... .......... .......... .......... .......... 36% 86.0M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 34300K .......... .......... .......... .......... .......... 36% 111M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 34350K .......... .......... .......... .......... .......... 37% 104M 1s 2024-03-09T14:53:40.734 INFO:teuthology.orchestra.run.smithi001.stderr: 34400K .......... .......... .......... .......... .......... 37% 105M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 34450K .......... .......... .......... .......... .......... 37% 145M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 34500K .......... .......... .......... .......... .......... 37% 120M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 34550K .......... .......... .......... .......... .......... 37% 115M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 34600K .......... .......... .......... .......... .......... 37% 116M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 34650K .......... .......... .......... .......... .......... 37% 94.1M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 34700K .......... .......... .......... .......... .......... 37% 112M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 34750K .......... .......... .......... .......... .......... 37% 118M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 34800K .......... .......... .......... .......... .......... 37% 117M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 34850K .......... .......... .......... .......... .......... 37% 97.6M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 34900K .......... .......... .......... .......... .......... 37% 117M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 34950K .......... .......... .......... .......... .......... 37% 114M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 35000K .......... .......... .......... .......... .......... 37% 111M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 35050K .......... .......... .......... .......... .......... 37% 93.8M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 35100K .......... .......... .......... .......... .......... 37% 116M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 35150K .......... .......... .......... .......... .......... 37% 110M 1s 2024-03-09T14:53:40.735 INFO:teuthology.orchestra.run.smithi001.stderr: 35200K .......... .......... .......... .......... .......... 37% 118M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35250K .......... .......... .......... .......... .......... 37% 110M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35300K .......... .......... .......... .......... .......... 38% 101M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35350K .......... .......... .......... .......... .......... 38% 103M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35400K .......... .......... .......... .......... .......... 38% 120M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35450K .......... .......... .......... .......... .......... 38% 93.7M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35500K .......... .......... .......... .......... .......... 38% 121M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35550K .......... .......... .......... .......... .......... 38% 112M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35600K .......... .......... .......... .......... .......... 38% 108M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35650K .......... .......... .......... .......... .......... 38% 111M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35700K .......... .......... .......... .......... .......... 38% 116M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35750K .......... .......... .......... .......... .......... 38% 120M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35800K .......... .......... .......... .......... .......... 38% 62.1M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35850K .......... .......... .......... .......... .......... 38% 68.7M 1s 2024-03-09T14:53:40.736 INFO:teuthology.orchestra.run.smithi001.stderr: 35900K .......... .......... .......... .......... .......... 38% 106M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 35950K .......... .......... .......... .......... .......... 38% 118M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36000K .......... .......... .......... .......... .......... 38% 68.8M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36050K .......... .......... .......... .......... .......... 38% 94.1M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36100K .......... .......... .......... .......... .......... 38% 111M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36150K .......... .......... .......... .......... .......... 38% 116M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36200K .......... .......... .......... .......... .......... 39% 122M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36250K .......... .......... .......... .......... .......... 39% 89.9M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36300K .......... .......... .......... .......... .......... 39% 107M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36350K .......... .......... .......... .......... .......... 39% 116M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36400K .......... .......... .......... .......... .......... 39% 116M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36450K .......... .......... .......... .......... .......... 39% 111M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36500K .......... .......... .......... .......... .......... 39% 115M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36550K .......... .......... .......... .......... .......... 39% 111M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36600K .......... .......... .......... .......... .......... 39% 123M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36650K .......... .......... .......... .......... .......... 39% 90.3M 1s 2024-03-09T14:53:40.737 INFO:teuthology.orchestra.run.smithi001.stderr: 36700K .......... .......... .......... .......... .......... 39% 111M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 36750K .......... .......... .......... .......... .......... 39% 120M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 36800K .......... .......... .......... .......... .......... 39% 115M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 36850K .......... .......... .......... .......... .......... 39% 107M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 36900K .......... .......... .......... .......... .......... 39% 116M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 36950K .......... .......... .......... .......... .......... 39% 115M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 37000K .......... .......... .......... .......... .......... 39% 114M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 37050K .......... .......... .......... .......... .......... 39% 92.1M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 37100K .......... .......... .......... .......... .......... 39% 112M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 37150K .......... .......... .......... .......... .......... 40% 111M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 37200K .......... .......... .......... .......... .......... 40% 103M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 37250K .......... .......... .......... .......... .......... 40% 95.3M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 37300K .......... .......... .......... .......... .......... 40% 102M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 37350K .......... .......... .......... .......... .......... 40% 106M 1s 2024-03-09T14:53:40.738 INFO:teuthology.orchestra.run.smithi001.stderr: 37400K .......... .......... .......... .......... .......... 40% 107M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 37450K .......... .......... .......... .......... .......... 40% 84.6M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 37500K .......... .......... .......... .......... .......... 40% 106M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 37550K .......... .......... .......... .......... .......... 40% 108M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 37600K .......... .......... .......... .......... .......... 40% 101M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 37650K .......... .......... .......... .......... .......... 40% 96.5M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 37700K .......... .......... .......... .......... .......... 40% 103M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 37750K .......... .......... .......... .......... .......... 40% 104M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 37800K .......... .......... .......... .......... .......... 40% 106M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 37850K .......... .......... .......... .......... .......... 40% 85.1M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 37900K .......... .......... .......... .......... .......... 40% 106M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 37950K .......... .......... .......... .......... .......... 40% 108M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 38000K .......... .......... .......... .......... .......... 40% 103M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 38050K .......... .......... .......... .......... .......... 41% 97.9M 1s 2024-03-09T14:53:40.739 INFO:teuthology.orchestra.run.smithi001.stderr: 38100K .......... .......... .......... .......... .......... 41% 112M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38150K .......... .......... .......... .......... .......... 41% 187M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38200K .......... .......... .......... .......... .......... 41% 134M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38250K .......... .......... .......... .......... .......... 41% 78.4M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38300K .......... .......... .......... .......... .......... 41% 114M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38350K .......... .......... .......... .......... .......... 41% 106M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38400K .......... .......... .......... .......... .......... 41% 104M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38450K .......... .......... .......... .......... .......... 41% 85.5M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38500K .......... .......... .......... .......... .......... 41% 111M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38550K .......... .......... .......... .......... .......... 41% 112M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38600K .......... .......... .......... .......... .......... 41% 107M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38650K .......... .......... .......... .......... .......... 41% 89.4M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38700K .......... .......... .......... .......... .......... 41% 116M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38750K .......... .......... .......... .......... .......... 41% 111M 1s 2024-03-09T14:53:40.740 INFO:teuthology.orchestra.run.smithi001.stderr: 38800K .......... .......... .......... .......... .......... 41% 114M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 38850K .......... .......... .......... .......... .......... 41% 98.1M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 38900K .......... .......... .......... .......... .......... 41% 116M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 38950K .......... .......... .......... .......... .......... 41% 113M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 39000K .......... .......... .......... .......... .......... 42% 114M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 39050K .......... .......... .......... .......... .......... 42% 89.8M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 39100K .......... .......... .......... .......... .......... 42% 124M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 39150K .......... .......... .......... .......... .......... 42% 113M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 39200K .......... .......... .......... .......... .......... 42% 116M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 39250K .......... .......... .......... .......... .......... 42% 107M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 39300K .......... .......... .......... .......... .......... 42% 108M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 39350K .......... .......... .......... .......... .......... 42% 121M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 39400K .......... .......... .......... .......... .......... 42% 106M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 39450K .......... .......... .......... .......... .......... 42% 87.9M 1s 2024-03-09T14:53:40.741 INFO:teuthology.orchestra.run.smithi001.stderr: 39500K .......... .......... .......... .......... .......... 42% 123M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 39550K .......... .......... .......... .......... .......... 42% 111M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 39600K .......... .......... .......... .......... .......... 42% 114M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 39650K .......... .......... .......... .......... .......... 42% 105M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 39700K .......... .......... .......... .......... .......... 42% 99.0M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 39750K .......... .......... .......... .......... .......... 42% 132M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 39800K .......... .......... .......... .......... .......... 42% 122M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 39850K .......... .......... .......... .......... .......... 42% 90.1M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 39900K .......... .......... .......... .......... .......... 43% 110M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 39950K .......... .......... .......... .......... .......... 43% 120M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 40000K .......... .......... .......... .......... .......... 43% 111M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 40050K .......... .......... .......... .......... .......... 43% 103M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 40100K .......... .......... .......... .......... .......... 43% 120M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 40150K .......... .......... .......... .......... .......... 43% 116M 1s 2024-03-09T14:53:40.742 INFO:teuthology.orchestra.run.smithi001.stderr: 40200K .......... .......... .......... .......... .......... 43% 108M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40250K .......... .......... .......... .......... .......... 43% 92.7M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40300K .......... .......... .......... .......... .......... 43% 117M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40350K .......... .......... .......... .......... .......... 43% 114M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40400K .......... .......... .......... .......... .......... 43% 110M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40450K .......... .......... .......... .......... .......... 43% 95.4M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40500K .......... .......... .......... .......... .......... 43% 112M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40550K .......... .......... .......... .......... .......... 43% 71.2M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40600K .......... .......... .......... .......... .......... 43% 112M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40650K .......... .......... .......... .......... .......... 43% 70.1M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40700K .......... .......... .......... .......... .......... 43% 79.9M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40750K .......... .......... .......... .......... .......... 43% 111M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40800K .......... .......... .......... .......... .......... 43% 109M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40850K .......... .......... .......... .......... .......... 44% 104M 1s 2024-03-09T14:53:40.743 INFO:teuthology.orchestra.run.smithi001.stderr: 40900K .......... .......... .......... .......... .......... 44% 87.5M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 40950K .......... .......... .......... .......... .......... 44% 114M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41000K .......... .......... .......... .......... .......... 44% 115M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41050K .......... .......... .......... .......... .......... 44% 88.0M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41100K .......... .......... .......... .......... .......... 44% 111M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41150K .......... .......... .......... .......... .......... 44% 104M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41200K .......... .......... .......... .......... .......... 44% 115M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41250K .......... .......... .......... .......... .......... 44% 104M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41300K .......... .......... .......... .......... .......... 44% 110M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41350K .......... .......... .......... .......... .......... 44% 113M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41400K .......... .......... .......... .......... .......... 44% 117M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41450K .......... .......... .......... .......... .......... 44% 96.0M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41500K .......... .......... .......... .......... .......... 44% 141M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41550K .......... .......... .......... .......... .......... 44% 120M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41600K .......... .......... .......... .......... .......... 44% 106M 1s 2024-03-09T14:53:40.744 INFO:teuthology.orchestra.run.smithi001.stderr: 41650K .......... .......... .......... .......... .......... 44% 98.5M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 41700K .......... .......... .......... .......... .......... 44% 112M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 41750K .......... .......... .......... .......... .......... 44% 123M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 41800K .......... .......... .......... .......... .......... 45% 106M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 41850K .......... .......... .......... .......... .......... 45% 91.9M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 41900K .......... .......... .......... .......... .......... 45% 117M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 41950K .......... .......... .......... .......... .......... 45% 102M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 42000K .......... .......... .......... .......... .......... 45% 106M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 42050K .......... .......... .......... .......... .......... 45% 98.2M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 42100K .......... .......... .......... .......... .......... 45% 110M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 42150K .......... .......... .......... .......... .......... 45% 116M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 42200K .......... .......... .......... .......... .......... 45% 118M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 42250K .......... .......... .......... .......... .......... 45% 91.4M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 42300K .......... .......... .......... .......... .......... 45% 113M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 42350K .......... .......... .......... .......... .......... 45% 121M 1s 2024-03-09T14:53:40.745 INFO:teuthology.orchestra.run.smithi001.stderr: 42400K .......... .......... .......... .......... .......... 45% 106M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 42450K .......... .......... .......... .......... .......... 45% 107M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 42500K .......... .......... .......... .......... .......... 45% 114M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 42550K .......... .......... .......... .......... .......... 45% 114M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 42600K .......... .......... .......... .......... .......... 45% 116M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 42650K .......... .......... .......... .......... .......... 45% 88.6M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 42700K .......... .......... .......... .......... .......... 46% 118M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 42750K .......... .......... .......... .......... .......... 46% 116M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 42800K .......... .......... .......... .......... .......... 46% 116M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 42850K .......... .......... .......... .......... .......... 46% 105M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 42900K .......... .......... .......... .......... .......... 46% 119M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 42950K .......... .......... .......... .......... .......... 46% 108M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 43000K .......... .......... .......... .......... .......... 46% 66.9M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 43050K .......... .......... .......... .......... .......... 46% 69.8M 1s 2024-03-09T14:53:40.746 INFO:teuthology.orchestra.run.smithi001.stderr: 43100K .......... .......... .......... .......... .......... 46% 90.8M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43150K .......... .......... .......... .......... .......... 46% 101M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43200K .......... .......... .......... .......... .......... 46% 120M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43250K .......... .......... .......... .......... .......... 46% 107M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43300K .......... .......... .......... .......... .......... 46% 112M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43350K .......... .......... .......... .......... .......... 46% 114M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43400K .......... .......... .......... .......... .......... 46% 122M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43450K .......... .......... .......... .......... .......... 46% 62.5M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43500K .......... .......... .......... .......... .......... 46% 124M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43550K .......... .......... .......... .......... .......... 46% 114M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43600K .......... .......... .......... .......... .......... 46% 116M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43650K .......... .......... .......... .......... .......... 47% 108M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43700K .......... .......... .......... .......... .......... 47% 115M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43750K .......... .......... .......... .......... .......... 47% 104M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43800K .......... .......... .......... .......... .......... 47% 121M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43850K .......... .......... .......... .......... .......... 47% 89.3M 1s 2024-03-09T14:53:40.747 INFO:teuthology.orchestra.run.smithi001.stderr: 43900K .......... .......... .......... .......... .......... 47% 111M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 43950K .......... .......... .......... .......... .......... 47% 119M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44000K .......... .......... .......... .......... .......... 47% 109M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44050K .......... .......... .......... .......... .......... 47% 113M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44100K .......... .......... .......... .......... .......... 47% 105M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44150K .......... .......... .......... .......... .......... 47% 131M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44200K .......... .......... .......... .......... .......... 47% 118M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44250K .......... .......... .......... .......... .......... 47% 85.7M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44300K .......... .......... .......... .......... .......... 47% 118M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44350K .......... .......... .......... .......... .......... 47% 108M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44400K .......... .......... .......... .......... .......... 47% 120M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44450K .......... .......... .......... .......... .......... 47% 106M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44500K .......... .......... .......... .......... .......... 47% 114M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44550K .......... .......... .......... .......... .......... 48% 106M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44600K .......... .......... .......... .......... .......... 48% 142M 1s 2024-03-09T14:53:40.748 INFO:teuthology.orchestra.run.smithi001.stderr: 44650K .......... .......... .......... .......... .......... 48% 81.4M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 44700K .......... .......... .......... .......... .......... 48% 116M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 44750K .......... .......... .......... .......... .......... 48% 138M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 44800K .......... .......... .......... .......... .......... 48% 113M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 44850K .......... .......... .......... .......... .......... 48% 111M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 44900K .......... .......... .......... .......... .......... 48% 115M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 44950K .......... .......... .......... .......... .......... 48% 87.3M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 45000K .......... .......... .......... .......... .......... 48% 127M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 45050K .......... .......... .......... .......... .......... 48% 86.4M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 45100K .......... .......... .......... .......... .......... 48% 120M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 45150K .......... .......... .......... .......... .......... 48% 114M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 45200K .......... .......... .......... .......... .......... 48% 115M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 45250K .......... .......... .......... .......... .......... 48% 113M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 45300K .......... .......... .......... .......... .......... 48% 117M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 45350K .......... .......... .......... .......... .......... 48% 107M 1s 2024-03-09T14:53:40.749 INFO:teuthology.orchestra.run.smithi001.stderr: 45400K .......... .......... .......... .......... .......... 48% 82.9M 1s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 45450K .......... .......... .......... .......... .......... 48% 91.0M 1s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 45500K .......... .......... .......... .......... .......... 49% 109M 1s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 45550K .......... .......... .......... .......... .......... 49% 115M 1s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 45600K .......... .......... .......... .......... .......... 49% 120M 1s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 45650K .......... .......... .......... .......... .......... 49% 104M 1s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 45700K .......... .......... .......... .......... .......... 49% 119M 1s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 45750K .......... .......... .......... .......... .......... 49% 116M 0s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 45800K .......... .......... .......... .......... .......... 49% 120M 0s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 45850K .......... .......... .......... .......... .......... 49% 90.3M 0s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 45900K .......... .......... .......... .......... .......... 49% 115M 0s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 45950K .......... .......... .......... .......... .......... 49% 108M 0s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 46000K .......... .......... .......... .......... .......... 49% 127M 0s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 46050K .......... .......... .......... .......... .......... 49% 102M 0s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 46100K .......... .......... .......... .......... .......... 49% 117M 0s 2024-03-09T14:53:40.750 INFO:teuthology.orchestra.run.smithi001.stderr: 46150K .......... .......... .......... .......... .......... 49% 115M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46200K .......... .......... .......... .......... .......... 49% 121M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46250K .......... .......... .......... .......... .......... 49% 84.1M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46300K .......... .......... .......... .......... .......... 49% 111M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46350K .......... .......... .......... .......... .......... 49% 116M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46400K .......... .......... .......... .......... .......... 49% 114M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46450K .......... .......... .......... .......... .......... 50% 116M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46500K .......... .......... .......... .......... .......... 50% 107M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46550K .......... .......... .......... .......... .......... 50% 124M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46600K .......... .......... .......... .......... .......... 50% 113M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46650K .......... .......... .......... .......... .......... 50% 84.6M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46700K .......... .......... .......... .......... .......... 50% 127M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46750K .......... .......... .......... .......... .......... 50% 109M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46800K .......... .......... .......... .......... .......... 50% 116M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46850K .......... .......... .......... .......... .......... 50% 107M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46900K .......... .......... .......... .......... .......... 50% 120M 0s 2024-03-09T14:53:40.751 INFO:teuthology.orchestra.run.smithi001.stderr: 46950K .......... .......... .......... .......... .......... 50% 115M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47000K .......... .......... .......... .......... .......... 50% 112M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47050K .......... .......... .......... .......... .......... 50% 73.8M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47100K .......... .......... .......... .......... .......... 50% 121M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47150K .......... .......... .......... .......... .......... 50% 114M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47200K .......... .......... .......... .......... .......... 50% 114M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47250K .......... .......... .......... .......... .......... 50% 114M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47300K .......... .......... .......... .......... .......... 50% 114M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47350K .......... .......... .......... .......... .......... 51% 114M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47400K .......... .......... .......... .......... .......... 51% 107M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47450K .......... .......... .......... .......... .......... 51% 93.6M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47500K .......... .......... .......... .......... .......... 51% 121M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47550K .......... .......... .......... .......... .......... 51% 111M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47600K .......... .......... .......... .......... .......... 51% 114M 0s 2024-03-09T14:53:40.752 INFO:teuthology.orchestra.run.smithi001.stderr: 47650K .......... .......... .......... .......... .......... 51% 103M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 47700K .......... .......... .......... .......... .......... 51% 125M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 47750K .......... .......... .......... .......... .......... 51% 109M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 47800K .......... .......... .......... .......... .......... 51% 124M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 47850K .......... .......... .......... .......... .......... 51% 88.4M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 47900K .......... .......... .......... .......... .......... 51% 113M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 47950K .......... .......... .......... .......... .......... 51% 125M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 48000K .......... .......... .......... .......... .......... 51% 113M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 48050K .......... .......... .......... .......... .......... 51% 110M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 48100K .......... .......... .......... .......... .......... 51% 117M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 48150K .......... .......... .......... .......... .......... 51% 112M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 48200K .......... .......... .......... .......... .......... 51% 114M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 48250K .......... .......... .......... .......... .......... 51% 86.9M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 48300K .......... .......... .......... .......... .......... 52% 114M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 48350K .......... .......... .......... .......... .......... 52% 108M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 48400K .......... .......... .......... .......... .......... 52% 128M 0s 2024-03-09T14:53:40.753 INFO:teuthology.orchestra.run.smithi001.stderr: 48450K .......... .......... .......... .......... .......... 52% 111M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 48500K .......... .......... .......... .......... .......... 52% 110M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 48550K .......... .......... .......... .......... .......... 52% 109M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 48600K .......... .......... .......... .......... .......... 52% 99.0M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 48650K .......... .......... .......... .......... .......... 52% 90.0M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 48700K .......... .......... .......... .......... .......... 52% 111M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 48750K .......... .......... .......... .......... .......... 52% 112M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 48800K .......... .......... .......... .......... .......... 52% 123M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 48850K .......... .......... .......... .......... .......... 52% 110M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 48900K .......... .......... .......... .......... .......... 52% 113M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 48950K .......... .......... .......... .......... .......... 52% 119M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 49000K .......... .......... .......... .......... .......... 52% 118M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 49050K .......... .......... .......... .......... .......... 52% 83.5M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 49100K .......... .......... .......... .......... .......... 52% 83.9M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 49150K .......... .......... .......... .......... .......... 52% 118M 0s 2024-03-09T14:53:40.754 INFO:teuthology.orchestra.run.smithi001.stderr: 49200K .......... .......... .......... .......... .......... 53% 124M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49250K .......... .......... .......... .......... .......... 53% 106M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49300K .......... .......... .......... .......... .......... 53% 120M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49350K .......... .......... .......... .......... .......... 53% 116M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49400K .......... .......... .......... .......... .......... 53% 105M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49450K .......... .......... .......... .......... .......... 53% 68.3M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49500K .......... .......... .......... .......... .......... 53% 121M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49550K .......... .......... .......... .......... .......... 53% 96.9M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49600K .......... .......... .......... .......... .......... 53% 105M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49650K .......... .......... .......... .......... .......... 53% 114M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49700K .......... .......... .......... .......... .......... 53% 122M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49750K .......... .......... .......... .......... .......... 53% 116M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49800K .......... .......... .......... .......... .......... 53% 116M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49850K .......... .......... .......... .......... .......... 53% 86.9M 0s 2024-03-09T14:53:40.755 INFO:teuthology.orchestra.run.smithi001.stderr: 49900K .......... .......... .......... .......... .......... 53% 116M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 49950K .......... .......... .......... .......... .......... 53% 109M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50000K .......... .......... .......... .......... .......... 53% 120M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50050K .......... .......... .......... .......... .......... 53% 112M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50100K .......... .......... .......... .......... .......... 53% 83.9M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50150K .......... .......... .......... .......... .......... 54% 177M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50200K .......... .......... .......... .......... .......... 54% 117M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50250K .......... .......... .......... .......... .......... 54% 90.5M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50300K .......... .......... .......... .......... .......... 54% 114M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50350K .......... .......... .......... .......... .......... 54% 115M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50400K .......... .......... .......... .......... .......... 54% 116M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50450K .......... .......... .......... .......... .......... 54% 108M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50500K .......... .......... .......... .......... .......... 54% 120M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50550K .......... .......... .......... .......... .......... 54% 110M 0s 2024-03-09T14:53:40.756 INFO:teuthology.orchestra.run.smithi001.stderr: 50600K .......... .......... .......... .......... .......... 54% 118M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 50650K .......... .......... .......... .......... .......... 54% 90.3M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 50700K .......... .......... .......... .......... .......... 54% 120M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 50750K .......... .......... .......... .......... .......... 54% 114M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 50800K .......... .......... .......... .......... .......... 54% 113M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 50850K .......... .......... .......... .......... .......... 54% 110M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 50900K .......... .......... .......... .......... .......... 54% 116M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 50950K .......... .......... .......... .......... .......... 54% 114M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 51000K .......... .......... .......... .......... .......... 54% 110M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 51050K .......... .......... .......... .......... .......... 55% 91.0M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 51100K .......... .......... .......... .......... .......... 55% 110M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 51150K .......... .......... .......... .......... .......... 55% 122M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 51200K .......... .......... .......... .......... .......... 55% 112M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 51250K .......... .......... .......... .......... .......... 55% 114M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 51300K .......... .......... .......... .......... .......... 55% 108M 0s 2024-03-09T14:53:40.757 INFO:teuthology.orchestra.run.smithi001.stderr: 51350K .......... .......... .......... .......... .......... 55% 116M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 51400K .......... .......... .......... .......... .......... 55% 115M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 51450K .......... .......... .......... .......... .......... 55% 90.9M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 51500K .......... .......... .......... .......... .......... 55% 117M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 51550K .......... .......... .......... .......... .......... 55% 112M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 51600K .......... .......... .......... .......... .......... 55% 116M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 51650K .......... .......... .......... .......... .......... 55% 112M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 51700K .......... .......... .......... .......... .......... 55% 107M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 51750K .......... .......... .......... .......... .......... 55% 120M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 51800K .......... .......... .......... .......... .......... 55% 98.4M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 51850K .......... .......... .......... .......... .......... 55% 83.5M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 51900K .......... .......... .......... .......... .......... 55% 109M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 51950K .......... .......... .......... .......... .......... 55% 115M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 52000K .......... .......... .......... .......... .......... 56% 70.2M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 52050K .......... .......... .......... .......... .......... 56% 104M 0s 2024-03-09T14:53:40.758 INFO:teuthology.orchestra.run.smithi001.stderr: 52100K .......... .......... .......... .......... .......... 56% 119M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52150K .......... .......... .......... .......... .......... 56% 101M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52200K .......... .......... .......... .......... .......... 56% 115M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52250K .......... .......... .......... .......... .......... 56% 94.0M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52300K .......... .......... .......... .......... .......... 56% 114M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52350K .......... .......... .......... .......... .......... 56% 115M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52400K .......... .......... .......... .......... .......... 56% 114M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52450K .......... .......... .......... .......... .......... 56% 113M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52500K .......... .......... .......... .......... .......... 56% 117M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52550K .......... .......... .......... .......... .......... 56% 117M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52600K .......... .......... .......... .......... .......... 56% 110M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52650K .......... .......... .......... .......... .......... 56% 92.5M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52700K .......... .......... .......... .......... .......... 56% 115M 0s 2024-03-09T14:53:40.759 INFO:teuthology.orchestra.run.smithi001.stderr: 52750K .......... .......... .......... .......... .......... 56% 117M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 52800K .......... .......... .......... .......... .......... 56% 114M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 52850K .......... .......... .......... .......... .......... 56% 105M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 52900K .......... .......... .......... .......... .......... 56% 109M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 52950K .......... .......... .......... .......... .......... 57% 130M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 53000K .......... .......... .......... .......... .......... 57% 123M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 53050K .......... .......... .......... .......... .......... 57% 84.9M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 53100K .......... .......... .......... .......... .......... 57% 119M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 53150K .......... .......... .......... .......... .......... 57% 115M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 53200K .......... .......... .......... .......... .......... 57% 111M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 53250K .......... .......... .......... .......... .......... 57% 106M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 53300K .......... .......... .......... .......... .......... 57% 127M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 53350K .......... .......... .......... .......... .......... 57% 110M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 53400K .......... .......... .......... .......... .......... 57% 116M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 53450K .......... .......... .......... .......... .......... 57% 45.7M 0s 2024-03-09T14:53:40.760 INFO:teuthology.orchestra.run.smithi001.stderr: 53500K .......... .......... .......... .......... .......... 57% 123M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 53550K .......... .......... .......... .......... .......... 57% 72.0M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 53600K .......... .......... .......... .......... .......... 57% 93.1M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 53650K .......... .......... .......... .......... .......... 57% 109M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 53700K .......... .......... .......... .......... .......... 57% 114M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 53750K .......... .......... .......... .......... .......... 57% 120M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 53800K .......... .......... .......... .......... .......... 57% 114M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 53850K .......... .......... .......... .......... .......... 58% 88.3M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 53900K .......... .......... .......... .......... .......... 58% 92.6M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 53950K .......... .......... .......... .......... .......... 58% 150M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 54000K .......... .......... .......... .......... .......... 58% 117M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 54050K .......... .......... .......... .......... .......... 58% 103M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 54100K .......... .......... .......... .......... .......... 58% 116M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 54150K .......... .......... .......... .......... .......... 58% 117M 0s 2024-03-09T14:53:40.761 INFO:teuthology.orchestra.run.smithi001.stderr: 54200K .......... .......... .......... .......... .......... 58% 115M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54250K .......... .......... .......... .......... .......... 58% 93.7M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54300K .......... .......... .......... .......... .......... 58% 112M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54350K .......... .......... .......... .......... .......... 58% 119M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54400K .......... .......... .......... .......... .......... 58% 111M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54450K .......... .......... .......... .......... .......... 58% 110M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54500K .......... .......... .......... .......... .......... 58% 85.2M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54550K .......... .......... .......... .......... .......... 58% 121M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54600K .......... .......... .......... .......... .......... 58% 115M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54650K .......... .......... .......... .......... .......... 58% 87.8M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54700K .......... .......... .......... .......... .......... 58% 116M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54750K .......... .......... .......... .......... .......... 58% 115M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54800K .......... .......... .......... .......... .......... 59% 115M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54850K .......... .......... .......... .......... .......... 59% 109M 0s 2024-03-09T14:53:40.762 INFO:teuthology.orchestra.run.smithi001.stderr: 54900K .......... .......... .......... .......... .......... 59% 115M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 54950K .......... .......... .......... .......... .......... 59% 110M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55000K .......... .......... .......... .......... .......... 59% 94.4M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55050K .......... .......... .......... .......... .......... 59% 73.9M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55100K .......... .......... .......... .......... .......... 59% 107M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55150K .......... .......... .......... .......... .......... 59% 126M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55200K .......... .......... .......... .......... .......... 59% 88.1M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55250K .......... .......... .......... .......... .......... 59% 106M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55300K .......... .......... .......... .......... .......... 59% 114M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55350K .......... .......... .......... .......... .......... 59% 105M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55400K .......... .......... .......... .......... .......... 59% 129M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55450K .......... .......... .......... .......... .......... 59% 92.7M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55500K .......... .......... .......... .......... .......... 59% 113M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55550K .......... .......... .......... .......... .......... 59% 112M 0s 2024-03-09T14:53:40.763 INFO:teuthology.orchestra.run.smithi001.stderr: 55600K .......... .......... .......... .......... .......... 59% 113M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 55650K .......... .......... .......... .......... .......... 59% 104M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 55700K .......... .......... .......... .......... .......... 60% 127M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 55750K .......... .......... .......... .......... .......... 60% 116M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 55800K .......... .......... .......... .......... .......... 60% 114M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 55850K .......... .......... .......... .......... .......... 60% 72.7M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 55900K .......... .......... .......... .......... .......... 60% 57.5M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 55950K .......... .......... .......... .......... .......... 60% 69.4M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 56000K .......... .......... .......... .......... .......... 60% 111M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 56050K .......... .......... .......... .......... .......... 60% 116M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 56100K .......... .......... .......... .......... .......... 60% 81.1M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 56150K .......... .......... .......... .......... .......... 60% 110M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 56200K .......... .......... .......... .......... .......... 60% 114M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 56250K .......... .......... .......... .......... .......... 60% 92.7M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 56300K .......... .......... .......... .......... .......... 60% 101M 0s 2024-03-09T14:53:40.764 INFO:teuthology.orchestra.run.smithi001.stderr: 56350K .......... .......... .......... .......... .......... 60% 137M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 56400K .......... .......... .......... .......... .......... 60% 97.6M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 56450K .......... .......... .......... .......... .......... 60% 126M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 56500K .......... .......... .......... .......... .......... 60% 122M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 56550K .......... .......... .......... .......... .......... 60% 98.3M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 56600K .......... .......... .......... .......... .......... 60% 115M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 56650K .......... .......... .......... .......... .......... 61% 90.5M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 56700K .......... .......... .......... .......... .......... 61% 117M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 56750K .......... .......... .......... .......... .......... 61% 116M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 56800K .......... .......... .......... .......... .......... 61% 115M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 56850K .......... .......... .......... .......... .......... 61% 108M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 56900K .......... .......... .......... .......... .......... 61% 120M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 56950K .......... .......... .......... .......... .......... 61% 113M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 57000K .......... .......... .......... .......... .......... 61% 116M 0s 2024-03-09T14:53:40.765 INFO:teuthology.orchestra.run.smithi001.stderr: 57050K .......... .......... .......... .......... .......... 61% 91.6M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57100K .......... .......... .......... .......... .......... 61% 69.8M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57150K .......... .......... .......... .......... .......... 61% 116M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57200K .......... .......... .......... .......... .......... 61% 114M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57250K .......... .......... .......... .......... .......... 61% 105M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57300K .......... .......... .......... .......... .......... 61% 121M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57350K .......... .......... .......... .......... .......... 61% 119M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57400K .......... .......... .......... .......... .......... 61% 117M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57450K .......... .......... .......... .......... .......... 61% 90.6M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57500K .......... .......... .......... .......... .......... 61% 111M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57550K .......... .......... .......... .......... .......... 62% 121M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57600K .......... .......... .......... .......... .......... 62% 113M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57650K .......... .......... .......... .......... .......... 62% 109M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57700K .......... .......... .......... .......... .......... 62% 114M 0s 2024-03-09T14:53:40.766 INFO:teuthology.orchestra.run.smithi001.stderr: 57750K .......... .......... .......... .......... .......... 62% 113M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 57800K .......... .......... .......... .......... .......... 62% 108M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 57850K .......... .......... .......... .......... .......... 62% 93.1M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 57900K .......... .......... .......... .......... .......... 62% 116M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 57950K .......... .......... .......... .......... .......... 62% 117M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 58000K .......... .......... .......... .......... .......... 62% 109M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 58050K .......... .......... .......... .......... .......... 62% 115M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 58100K .......... .......... .......... .......... .......... 62% 117M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 58150K .......... .......... .......... .......... .......... 62% 115M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 58200K .......... .......... .......... .......... .......... 62% 116M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 58250K .......... .......... .......... .......... .......... 62% 65.5M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 58300K .......... .......... .......... .......... .......... 62% 110M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 58350K .......... .......... .......... .......... .......... 62% 119M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 58400K .......... .......... .......... .......... .......... 62% 116M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 58450K .......... .......... .......... .......... .......... 62% 78.6M 0s 2024-03-09T14:53:40.767 INFO:teuthology.orchestra.run.smithi001.stderr: 58500K .......... .......... .......... .......... .......... 63% 212M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 58550K .......... .......... .......... .......... .......... 63% 114M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 58600K .......... .......... .......... .......... .......... 63% 107M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 58650K .......... .......... .......... .......... .......... 63% 88.5M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 58700K .......... .......... .......... .......... .......... 63% 99.3M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 58750K .......... .......... .......... .......... .......... 63% 127M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 58800K .......... .......... .......... .......... .......... 63% 123M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 58850K .......... .......... .......... .......... .......... 63% 90.4M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 58900K .......... .......... .......... .......... .......... 63% 130M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 58950K .......... .......... .......... .......... .......... 63% 138M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 59000K .......... .......... .......... .......... .......... 63% 115M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 59050K .......... .......... .......... .......... .......... 63% 90.0M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 59100K .......... .......... .......... .......... .......... 63% 112M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 59150K .......... .......... .......... .......... .......... 63% 112M 0s 2024-03-09T14:53:40.768 INFO:teuthology.orchestra.run.smithi001.stderr: 59200K .......... .......... .......... .......... .......... 63% 118M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59250K .......... .......... .......... .......... .......... 63% 109M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59300K .......... .......... .......... .......... .......... 63% 122M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59350K .......... .......... .......... .......... .......... 63% 112M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59400K .......... .......... .......... .......... .......... 63% 110M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59450K .......... .......... .......... .......... .......... 64% 93.9M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59500K .......... .......... .......... .......... .......... 64% 116M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59550K .......... .......... .......... .......... .......... 64% 108M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59600K .......... .......... .......... .......... .......... 64% 95.7M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59650K .......... .......... .......... .......... .......... 64% 84.3M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59700K .......... .......... .......... .......... .......... 64% 114M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59750K .......... .......... .......... .......... .......... 64% 117M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59800K .......... .......... .......... .......... .......... 64% 118M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59850K .......... .......... .......... .......... .......... 64% 88.2M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59900K .......... .......... .......... .......... .......... 64% 103M 0s 2024-03-09T14:53:40.769 INFO:teuthology.orchestra.run.smithi001.stderr: 59950K .......... .......... .......... .......... .......... 64% 123M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60000K .......... .......... .......... .......... .......... 64% 110M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60050K .......... .......... .......... .......... .......... 64% 113M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60100K .......... .......... .......... .......... .......... 64% 115M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60150K .......... .......... .......... .......... .......... 64% 112M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60200K .......... .......... .......... .......... .......... 64% 121M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60250K .......... .......... .......... .......... .......... 64% 87.7M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60300K .......... .......... .......... .......... .......... 64% 111M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60350K .......... .......... .......... .......... .......... 65% 119M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60400K .......... .......... .......... .......... .......... 65% 115M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60450K .......... .......... .......... .......... .......... 65% 108M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60500K .......... .......... .......... .......... .......... 65% 115M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60550K .......... .......... .......... .......... .......... 65% 116M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60600K .......... .......... .......... .......... .......... 65% 114M 0s 2024-03-09T14:53:40.770 INFO:teuthology.orchestra.run.smithi001.stderr: 60650K .......... .......... .......... .......... .......... 65% 89.4M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 60700K .......... .......... .......... .......... .......... 65% 113M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 60750K .......... .......... .......... .......... .......... 65% 117M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 60800K .......... .......... .......... .......... .......... 65% 118M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 60850K .......... .......... .......... .......... .......... 65% 74.0M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 60900K .......... .......... .......... .......... .......... 65% 15.4M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 60950K .......... .......... .......... .......... .......... 65% 130M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 61000K .......... .......... .......... .......... .......... 65% 243M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 61050K .......... .......... .......... .......... .......... 65% 193M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 61100K .......... .......... .......... .......... .......... 65% 245M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 61150K .......... .......... .......... .......... .......... 65% 257M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 61200K .......... .......... .......... .......... .......... 65% 251M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 61250K .......... .......... .......... .......... .......... 65% 235M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 61300K .......... .......... .......... .......... .......... 66% 253M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 61350K .......... .......... .......... .......... .......... 66% 260M 0s 2024-03-09T14:53:40.771 INFO:teuthology.orchestra.run.smithi001.stderr: 61400K .......... .......... .......... .......... .......... 66% 260M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 61450K .......... .......... .......... .......... .......... 66% 216M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 61500K .......... .......... .......... .......... .......... 66% 267M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 61550K .......... .......... .......... .......... .......... 66% 204M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 61600K .......... .......... .......... .......... .......... 66% 108M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 61650K .......... .......... .......... .......... .......... 66% 108M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 61700K .......... .......... .......... .......... .......... 66% 127M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 61750K .......... .......... .......... .......... .......... 66% 91.1M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 61800K .......... .......... .......... .......... .......... 66% 95.1M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 61850K .......... .......... .......... .......... .......... 66% 86.9M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 61900K .......... .......... .......... .......... .......... 66% 117M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 61950K .......... .......... .......... .......... .......... 66% 113M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 62000K .......... .......... .......... .......... .......... 66% 118M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 62050K .......... .......... .......... .......... .......... 66% 110M 0s 2024-03-09T14:53:40.772 INFO:teuthology.orchestra.run.smithi001.stderr: 62100K .......... .......... .......... .......... .......... 66% 117M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62150K .......... .......... .......... .......... .......... 66% 112M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62200K .......... .......... .......... .......... .......... 67% 117M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62250K .......... .......... .......... .......... .......... 67% 91.7M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62300K .......... .......... .......... .......... .......... 67% 110M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62350K .......... .......... .......... .......... .......... 67% 121M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62400K .......... .......... .......... .......... .......... 67% 106M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62450K .......... .......... .......... .......... .......... 67% 84.3M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62500K .......... .......... .......... .......... .......... 67% 63.1M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62550K .......... .......... .......... .......... .......... 67% 129M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62600K .......... .......... .......... .......... .......... 67% 113M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62650K .......... .......... .......... .......... .......... 67% 91.7M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62700K .......... .......... .......... .......... .......... 67% 105M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62750K .......... .......... .......... .......... .......... 67% 109M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62800K .......... .......... .......... .......... .......... 67% 123M 0s 2024-03-09T14:53:40.773 INFO:teuthology.orchestra.run.smithi001.stderr: 62850K .......... .......... .......... .......... .......... 67% 115M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 62900K .......... .......... .......... .......... .......... 67% 102M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 62950K .......... .......... .......... .......... .......... 67% 112M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 63000K .......... .......... .......... .......... .......... 67% 123M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 63050K .......... .......... .......... .......... .......... 67% 84.8M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 63100K .......... .......... .......... .......... .......... 67% 122M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 63150K .......... .......... .......... .......... .......... 68% 109M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 63200K .......... .......... .......... .......... .......... 68% 103M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 63250K .......... .......... .......... .......... .......... 68% 121M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 63300K .......... .......... .......... .......... .......... 68% 125M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 63350K .......... .......... .......... .......... .......... 68% 92.7M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 63400K .......... .......... .......... .......... .......... 68% 138M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 63450K .......... .......... .......... .......... .......... 68% 95.0M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 63500K .......... .......... .......... .......... .......... 68% 116M 0s 2024-03-09T14:53:40.774 INFO:teuthology.orchestra.run.smithi001.stderr: 63550K .......... .......... .......... .......... .......... 68% 98.9M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 63600K .......... .......... .......... .......... .......... 68% 122M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 63650K .......... .......... .......... .......... .......... 68% 94.7M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 63700K .......... .......... .......... .......... .......... 68% 122M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 63750K .......... .......... .......... .......... .......... 68% 131M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 63800K .......... .......... .......... .......... .......... 68% 95.0M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 63850K .......... .......... .......... .......... .......... 68% 90.3M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 63900K .......... .......... .......... .......... .......... 68% 134M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 63950K .......... .......... .......... .......... .......... 68% 103M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 64000K .......... .......... .......... .......... .......... 68% 133M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 64050K .......... .......... .......... .......... .......... 68% 101M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 64100K .......... .......... .......... .......... .......... 69% 124M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 64150K .......... .......... .......... .......... .......... 69% 116M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 64200K .......... .......... .......... .......... .......... 69% 118M 0s 2024-03-09T14:53:40.775 INFO:teuthology.orchestra.run.smithi001.stderr: 64250K .......... .......... .......... .......... .......... 69% 92.7M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64300K .......... .......... .......... .......... .......... 69% 108M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64350K .......... .......... .......... .......... .......... 69% 113M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64400K .......... .......... .......... .......... .......... 69% 126M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64450K .......... .......... .......... .......... .......... 69% 99.8M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64500K .......... .......... .......... .......... .......... 69% 118M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64550K .......... .......... .......... .......... .......... 69% 118M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64600K .......... .......... .......... .......... .......... 69% 114M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64650K .......... .......... .......... .......... .......... 69% 94.2M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64700K .......... .......... .......... .......... .......... 69% 122M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64750K .......... .......... .......... .......... .......... 69% 116M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64800K .......... .......... .......... .......... .......... 69% 115M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64850K .......... .......... .......... .......... .......... 69% 99.4M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64900K .......... .......... .......... .......... .......... 69% 118M 0s 2024-03-09T14:53:40.776 INFO:teuthology.orchestra.run.smithi001.stderr: 64950K .......... .......... .......... .......... .......... 69% 102M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65000K .......... .......... .......... .......... .......... 70% 134M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65050K .......... .......... .......... .......... .......... 70% 88.5M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65100K .......... .......... .......... .......... .......... 70% 120M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65150K .......... .......... .......... .......... .......... 70% 110M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65200K .......... .......... .......... .......... .......... 70% 123M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65250K .......... .......... .......... .......... .......... 70% 108M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65300K .......... .......... .......... .......... .......... 70% 104M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65350K .......... .......... .......... .......... .......... 70% 108M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65400K .......... .......... .......... .......... .......... 70% 137M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65450K .......... .......... .......... .......... .......... 70% 95.2M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65500K .......... .......... .......... .......... .......... 70% 110M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65550K .......... .......... .......... .......... .......... 70% 121M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65600K .......... .......... .......... .......... .......... 70% 111M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65650K .......... .......... .......... .......... .......... 70% 108M 0s 2024-03-09T14:53:40.777 INFO:teuthology.orchestra.run.smithi001.stderr: 65700K .......... .......... .......... .......... .......... 70% 119M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 65750K .......... .......... .......... .......... .......... 70% 106M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 65800K .......... .......... .......... .......... .......... 70% 114M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 65850K .......... .......... .......... .......... .......... 70% 101M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 65900K .......... .......... .......... .......... .......... 70% 108M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 65950K .......... .......... .......... .......... .......... 71% 117M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 66000K .......... .......... .......... .......... .......... 71% 112M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 66050K .......... .......... .......... .......... .......... 71% 105M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 66100K .......... .......... .......... .......... .......... 71% 127M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 66150K .......... .......... .......... .......... .......... 71% 116M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 66200K .......... .......... .......... .......... .......... 71% 111M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 66250K .......... .......... .......... .......... .......... 71% 95.7M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 66300K .......... .......... .......... .......... .......... 71% 107M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 66350K .......... .......... .......... .......... .......... 71% 118M 0s 2024-03-09T14:53:40.778 INFO:teuthology.orchestra.run.smithi001.stderr: 66400K .......... .......... .......... .......... .......... 71% 121M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 66450K .......... .......... .......... .......... .......... 71% 107M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 66500K .......... .......... .......... .......... .......... 71% 121M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 66550K .......... .......... .......... .......... .......... 71% 110M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 66600K .......... .......... .......... .......... .......... 71% 117M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 66650K .......... .......... .......... .......... .......... 71% 89.2M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 66700K .......... .......... .......... .......... .......... 71% 120M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 66750K .......... .......... .......... .......... .......... 71% 111M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 66800K .......... .......... .......... .......... .......... 71% 120M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 66850K .......... .......... .......... .......... .......... 72% 108M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 66900K .......... .......... .......... .......... .......... 72% 117M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 66950K .......... .......... .......... .......... .......... 72% 116M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 67000K .......... .......... .......... .......... .......... 72% 118M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 67050K .......... .......... .......... .......... .......... 72% 91.2M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 67100K .......... .......... .......... .......... .......... 72% 109M 0s 2024-03-09T14:53:40.779 INFO:teuthology.orchestra.run.smithi001.stderr: 67150K .......... .......... .......... .......... .......... 72% 112M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67200K .......... .......... .......... .......... .......... 72% 116M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67250K .......... .......... .......... .......... .......... 72% 102M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67300K .......... .......... .......... .......... .......... 72% 123M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67350K .......... .......... .......... .......... .......... 72% 102M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67400K .......... .......... .......... .......... .......... 72% 125M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67450K .......... .......... .......... .......... .......... 72% 91.7M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67500K .......... .......... .......... .......... .......... 72% 118M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67550K .......... .......... .......... .......... .......... 72% 108M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67600K .......... .......... .......... .......... .......... 72% 113M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67650K .......... .......... .......... .......... .......... 72% 110M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67700K .......... .......... .......... .......... .......... 72% 116M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67750K .......... .......... .......... .......... .......... 72% 120M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67800K .......... .......... .......... .......... .......... 73% 52.3M 0s 2024-03-09T14:53:40.780 INFO:teuthology.orchestra.run.smithi001.stderr: 67850K .......... .......... .......... .......... .......... 73% 70.5M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 67900K .......... .......... .......... .......... .......... 73% 59.5M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 67950K .......... .......... .......... .......... .......... 73% 114M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 68000K .......... .......... .......... .......... .......... 73% 120M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 68050K .......... .......... .......... .......... .......... 73% 103M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 68100K .......... .......... .......... .......... .......... 73% 116M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 68150K .......... .......... .......... .......... .......... 73% 113M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 68200K .......... .......... .......... .......... .......... 73% 115M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 68250K .......... .......... .......... .......... .......... 73% 92.3M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 68300K .......... .......... .......... .......... .......... 73% 112M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 68350K .......... .......... .......... .......... .......... 73% 112M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 68400K .......... .......... .......... .......... .......... 73% 119M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 68450K .......... .......... .......... .......... .......... 73% 101M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 68500K .......... .......... .......... .......... .......... 73% 112M 0s 2024-03-09T14:53:40.781 INFO:teuthology.orchestra.run.smithi001.stderr: 68550K .......... .......... .......... .......... .......... 73% 113M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 68600K .......... .......... .......... .......... .......... 73% 120M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 68650K .......... .......... .......... .......... .......... 73% 88.5M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 68700K .......... .......... .......... .......... .......... 74% 118M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 68750K .......... .......... .......... .......... .......... 74% 115M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 68800K .......... .......... .......... .......... .......... 74% 111M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 68850K .......... .......... .......... .......... .......... 74% 107M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 68900K .......... .......... .......... .......... .......... 74% 115M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 68950K .......... .......... .......... .......... .......... 74% 109M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 69000K .......... .......... .......... .......... .......... 74% 108M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 69050K .......... .......... .......... .......... .......... 74% 92.7M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 69100K .......... .......... .......... .......... .......... 74% 117M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 69150K .......... .......... .......... .......... .......... 74% 116M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 69200K .......... .......... .......... .......... .......... 74% 111M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 69250K .......... .......... .......... .......... .......... 74% 100M 0s 2024-03-09T14:53:40.782 INFO:teuthology.orchestra.run.smithi001.stderr: 69300K .......... .......... .......... .......... .......... 74% 108M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69350K .......... .......... .......... .......... .......... 74% 122M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69400K .......... .......... .......... .......... .......... 74% 110M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69450K .......... .......... .......... .......... .......... 74% 93.6M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69500K .......... .......... .......... .......... .......... 74% 109M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69550K .......... .......... .......... .......... .......... 74% 123M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69600K .......... .......... .......... .......... .......... 74% 109M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69650K .......... .......... .......... .......... .......... 75% 108M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69700K .......... .......... .......... .......... .......... 75% 120M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69750K .......... .......... .......... .......... .......... 75% 112M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69800K .......... .......... .......... .......... .......... 75% 115M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69850K .......... .......... .......... .......... .......... 75% 89.2M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69900K .......... .......... .......... .......... .......... 75% 116M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 69950K .......... .......... .......... .......... .......... 75% 113M 0s 2024-03-09T14:53:40.783 INFO:teuthology.orchestra.run.smithi001.stderr: 70000K .......... .......... .......... .......... .......... 75% 116M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70050K .......... .......... .......... .......... .......... 75% 111M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70100K .......... .......... .......... .......... .......... 75% 110M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70150K .......... .......... .......... .......... .......... 75% 116M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70200K .......... .......... .......... .......... .......... 75% 109M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70250K .......... .......... .......... .......... .......... 75% 94.2M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70300K .......... .......... .......... .......... .......... 75% 115M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70350K .......... .......... .......... .......... .......... 75% 114M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70400K .......... .......... .......... .......... .......... 75% 118M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70450K .......... .......... .......... .......... .......... 75% 106M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70500K .......... .......... .......... .......... .......... 75% 117M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70550K .......... .......... .......... .......... .......... 75% 108M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70600K .......... .......... .......... .......... .......... 76% 116M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70650K .......... .......... .......... .......... .......... 76% 92.9M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70700K .......... .......... .......... .......... .......... 76% 118M 0s 2024-03-09T14:53:40.784 INFO:teuthology.orchestra.run.smithi001.stderr: 70750K .......... .......... .......... .......... .......... 76% 112M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 70800K .......... .......... .......... .......... .......... 76% 114M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 70850K .......... .......... .......... .......... .......... 76% 73.3M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 70900K .......... .......... .......... .......... .......... 76% 70.0M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 70950K .......... .......... .......... .......... .......... 76% 110M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 71000K .......... .......... .......... .......... .......... 76% 116M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 71050K .......... .......... .......... .......... .......... 76% 91.9M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 71100K .......... .......... .......... .......... .......... 76% 114M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 71150K .......... .......... .......... .......... .......... 76% 112M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 71200K .......... .......... .......... .......... .......... 76% 116M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 71250K .......... .......... .......... .......... .......... 76% 103M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 71300K .......... .......... .......... .......... .......... 76% 118M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 71350K .......... .......... .......... .......... .......... 76% 114M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 71400K .......... .......... .......... .......... .......... 76% 110M 0s 2024-03-09T14:53:40.785 INFO:teuthology.orchestra.run.smithi001.stderr: 71450K .......... .......... .......... .......... .......... 76% 87.7M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 71500K .......... .......... .......... .......... .......... 77% 118M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 71550K .......... .......... .......... .......... .......... 77% 119M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 71600K .......... .......... .......... .......... .......... 77% 112M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 71650K .......... .......... .......... .......... .......... 77% 108M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 71700K .......... .......... .......... .......... .......... 77% 109M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 71750K .......... .......... .......... .......... .......... 77% 118M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 71800K .......... .......... .......... .......... .......... 77% 117M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 71850K .......... .......... .......... .......... .......... 77% 91.6M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 71900K .......... .......... .......... .......... .......... 77% 111M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 71950K .......... .......... .......... .......... .......... 77% 120M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 72000K .......... .......... .......... .......... .......... 77% 121M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 72050K .......... .......... .......... .......... .......... 77% 99.0M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 72100K .......... .......... .......... .......... .......... 77% 115M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 72150K .......... .......... .......... .......... .......... 77% 112M 0s 2024-03-09T14:53:40.786 INFO:teuthology.orchestra.run.smithi001.stderr: 72200K .......... .......... .......... .......... .......... 77% 125M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72250K .......... .......... .......... .......... .......... 77% 90.3M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72300K .......... .......... .......... .......... .......... 77% 116M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72350K .......... .......... .......... .......... .......... 77% 117M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72400K .......... .......... .......... .......... .......... 77% 117M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72450K .......... .......... .......... .......... .......... 78% 103M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72500K .......... .......... .......... .......... .......... 78% 115M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72550K .......... .......... .......... .......... .......... 78% 115M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72600K .......... .......... .......... .......... .......... 78% 115M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72650K .......... .......... .......... .......... .......... 78% 91.9M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72700K .......... .......... .......... .......... .......... 78% 116M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72750K .......... .......... .......... .......... .......... 78% 117M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72800K .......... .......... .......... .......... .......... 78% 113M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72850K .......... .......... .......... .......... .......... 78% 110M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72900K .......... .......... .......... .......... .......... 78% 116M 0s 2024-03-09T14:53:40.787 INFO:teuthology.orchestra.run.smithi001.stderr: 72950K .......... .......... .......... .......... .......... 78% 115M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73000K .......... .......... .......... .......... .......... 78% 86.9M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73050K .......... .......... .......... .......... .......... 78% 42.9M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73100K .......... .......... .......... .......... .......... 78% 117M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73150K .......... .......... .......... .......... .......... 78% 68.7M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73200K .......... .......... .......... .......... .......... 78% 46.2M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73250K .......... .......... .......... .......... .......... 78% 69.2M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73300K .......... .......... .......... .......... .......... 78% 107M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73350K .......... .......... .......... .......... .......... 79% 114M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73400K .......... .......... .......... .......... .......... 79% 117M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73450K .......... .......... .......... .......... .......... 79% 95.7M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73500K .......... .......... .......... .......... .......... 79% 108M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73550K .......... .......... .......... .......... .......... 79% 118M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73600K .......... .......... .......... .......... .......... 79% 117M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73650K .......... .......... .......... .......... .......... 79% 102M 0s 2024-03-09T14:53:40.788 INFO:teuthology.orchestra.run.smithi001.stderr: 73700K .......... .......... .......... .......... .......... 79% 120M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 73750K .......... .......... .......... .......... .......... 79% 116M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 73800K .......... .......... .......... .......... .......... 79% 119M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 73850K .......... .......... .......... .......... .......... 79% 88.7M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 73900K .......... .......... .......... .......... .......... 79% 117M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 73950K .......... .......... .......... .......... .......... 79% 112M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 74000K .......... .......... .......... .......... .......... 79% 123M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 74050K .......... .......... .......... .......... .......... 79% 102M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 74100K .......... .......... .......... .......... .......... 79% 118M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 74150K .......... .......... .......... .......... .......... 79% 118M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 74200K .......... .......... .......... .......... .......... 79% 103M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 74250K .......... .......... .......... .......... .......... 79% 98.6M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 74300K .......... .......... .......... .......... .......... 80% 119M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 74350K .......... .......... .......... .......... .......... 80% 116M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 74400K .......... .......... .......... .......... .......... 80% 115M 0s 2024-03-09T14:53:40.789 INFO:teuthology.orchestra.run.smithi001.stderr: 74450K .......... .......... .......... .......... .......... 80% 80.6M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 74500K .......... .......... .......... .......... .......... 80% 116M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 74550K .......... .......... .......... .......... .......... 80% 112M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 74600K .......... .......... .......... .......... .......... 80% 116M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 74650K .......... .......... .......... .......... .......... 80% 89.7M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 74700K .......... .......... .......... .......... .......... 80% 112M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 74750K .......... .......... .......... .......... .......... 80% 120M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 74800K .......... .......... .......... .......... .......... 80% 113M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 74850K .......... .......... .......... .......... .......... 80% 104M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 74900K .......... .......... .......... .......... .......... 80% 116M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 74950K .......... .......... .......... .......... .......... 80% 121M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 75000K .......... .......... .......... .......... .......... 80% 111M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 75050K .......... .......... .......... .......... .......... 80% 92.9M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 75100K .......... .......... .......... .......... .......... 80% 109M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 75150K .......... .......... .......... .......... .......... 80% 123M 0s 2024-03-09T14:53:40.790 INFO:teuthology.orchestra.run.smithi001.stderr: 75200K .......... .......... .......... .......... .......... 81% 116M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75250K .......... .......... .......... .......... .......... 81% 105M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75300K .......... .......... .......... .......... .......... 81% 112M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75350K .......... .......... .......... .......... .......... 81% 121M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75400K .......... .......... .......... .......... .......... 81% 116M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75450K .......... .......... .......... .......... .......... 81% 88.6M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75500K .......... .......... .......... .......... .......... 81% 116M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75550K .......... .......... .......... .......... .......... 81% 118M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75600K .......... .......... .......... .......... .......... 81% 117M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75650K .......... .......... .......... .......... .......... 81% 101M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75700K .......... .......... .......... .......... .......... 81% 108M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75750K .......... .......... .......... .......... .......... 81% 111M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75800K .......... .......... .......... .......... .......... 81% 117M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75850K .......... .......... .......... .......... .......... 81% 94.2M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75900K .......... .......... .......... .......... .......... 81% 113M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 75950K .......... .......... .......... .......... .......... 81% 120M 0s 2024-03-09T14:53:40.791 INFO:teuthology.orchestra.run.smithi001.stderr: 76000K .......... .......... .......... .......... .......... 81% 110M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76050K .......... .......... .......... .......... .......... 81% 109M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76100K .......... .......... .......... .......... .......... 81% 114M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76150K .......... .......... .......... .......... .......... 82% 119M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76200K .......... .......... .......... .......... .......... 82% 108M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76250K .......... .......... .......... .......... .......... 82% 89.7M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76300K .......... .......... .......... .......... .......... 82% 118M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76350K .......... .......... .......... .......... .......... 82% 120M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76400K .......... .......... .......... .......... .......... 82% 114M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76450K .......... .......... .......... .......... .......... 82% 104M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76500K .......... .......... .......... .......... .......... 82% 110M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76550K .......... .......... .......... .......... .......... 82% 127M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76600K .......... .......... .......... .......... .......... 82% 112M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76650K .......... .......... .......... .......... .......... 82% 90.7M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76700K .......... .......... .......... .......... .......... 82% 118M 0s 2024-03-09T14:53:40.792 INFO:teuthology.orchestra.run.smithi001.stderr: 76750K .......... .......... .......... .......... .......... 82% 112M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 76800K .......... .......... .......... .......... .......... 82% 113M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 76850K .......... .......... .......... .......... .......... 82% 107M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 76900K .......... .......... .......... .......... .......... 82% 120M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 76950K .......... .......... .......... .......... .......... 82% 110M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 77000K .......... .......... .......... .......... .......... 82% 118M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 77050K .......... .......... .......... .......... .......... 82% 89.2M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 77100K .......... .......... .......... .......... .......... 83% 116M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 77150K .......... .......... .......... .......... .......... 83% 115M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 77200K .......... .......... .......... .......... .......... 83% 120M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 77250K .......... .......... .......... .......... .......... 83% 104M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 77300K .......... .......... .......... .......... .......... 83% 108M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 77350K .......... .......... .......... .......... .......... 83% 121M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 77400K .......... .......... .......... .......... .......... 83% 71.8M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 77450K .......... .......... .......... .......... .......... 83% 147M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 77500K .......... .......... .......... .......... .......... 83% 136M 0s 2024-03-09T14:53:40.793 INFO:teuthology.orchestra.run.smithi001.stderr: 77550K .......... .......... .......... .......... .......... 83% 118M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 77600K .......... .......... .......... .......... .......... 83% 118M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 77650K .......... .......... .......... .......... .......... 83% 106M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 77700K .......... .......... .......... .......... .......... 83% 113M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 77750K .......... .......... .......... .......... .......... 83% 115M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 77800K .......... .......... .......... .......... .......... 83% 113M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 77850K .......... .......... .......... .......... .......... 83% 93.3M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 77900K .......... .......... .......... .......... .......... 83% 114M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 77950K .......... .......... .......... .......... .......... 83% 120M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 78000K .......... .......... .......... .......... .......... 84% 116M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 78050K .......... .......... .......... .......... .......... 84% 106M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 78100K .......... .......... .......... .......... .......... 84% 119M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 78150K .......... .......... .......... .......... .......... 84% 66.0M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 78200K .......... .......... .......... .......... .......... 84% 120M 0s 2024-03-09T14:53:40.794 INFO:teuthology.orchestra.run.smithi001.stderr: 78250K .......... .......... .......... .......... .......... 84% 72.0M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78300K .......... .......... .......... .......... .......... 84% 51.9M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78350K .......... .......... .......... .......... .......... 84% 111M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78400K .......... .......... .......... .......... .......... 84% 112M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78450K .......... .......... .......... .......... .......... 84% 107M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78500K .......... .......... .......... .......... .......... 84% 113M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78550K .......... .......... .......... .......... .......... 84% 112M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78600K .......... .......... .......... .......... .......... 84% 111M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78650K .......... .......... .......... .......... .......... 84% 91.4M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78700K .......... .......... .......... .......... .......... 84% 116M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78750K .......... .......... .......... .......... .......... 84% 114M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78800K .......... .......... .......... .......... .......... 84% 98.6M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78850K .......... .......... .......... .......... .......... 84% 111M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78900K .......... .......... .......... .......... .......... 84% 112M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 78950K .......... .......... .......... .......... .......... 85% 117M 0s 2024-03-09T14:53:40.795 INFO:teuthology.orchestra.run.smithi001.stderr: 79000K .......... .......... .......... .......... .......... 85% 112M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79050K .......... .......... .......... .......... .......... 85% 89.2M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79100K .......... .......... .......... .......... .......... 85% 123M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79150K .......... .......... .......... .......... .......... 85% 118M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79200K .......... .......... .......... .......... .......... 85% 107M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79250K .......... .......... .......... .......... .......... 85% 109M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79300K .......... .......... .......... .......... .......... 85% 118M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79350K .......... .......... .......... .......... .......... 85% 111M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79400K .......... .......... .......... .......... .......... 85% 123M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79450K .......... .......... .......... .......... .......... 85% 90.1M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79500K .......... .......... .......... .......... .......... 85% 116M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79550K .......... .......... .......... .......... .......... 85% 111M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79600K .......... .......... .......... .......... .......... 85% 119M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79650K .......... .......... .......... .......... .......... 85% 106M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79700K .......... .......... .......... .......... .......... 85% 106M 0s 2024-03-09T14:53:40.796 INFO:teuthology.orchestra.run.smithi001.stderr: 79750K .......... .......... .......... .......... .......... 85% 125M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 79800K .......... .......... .......... .......... .......... 85% 118M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 79850K .......... .......... .......... .......... .......... 86% 91.1M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 79900K .......... .......... .......... .......... .......... 86% 112M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 79950K .......... .......... .......... .......... .......... 86% 122M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 80000K .......... .......... .......... .......... .......... 86% 113M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 80050K .......... .......... .......... .......... .......... 86% 105M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 80100K .......... .......... .......... .......... .......... 86% 121M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 80150K .......... .......... .......... .......... .......... 86% 111M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 80200K .......... .......... .......... .......... .......... 86% 120M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 80250K .......... .......... .......... .......... .......... 86% 89.3M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 80300K .......... .......... .......... .......... .......... 86% 121M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 80350K .......... .......... .......... .......... .......... 86% 111M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 80400K .......... .......... .......... .......... .......... 86% 123M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 80450K .......... .......... .......... .......... .......... 86% 104M 0s 2024-03-09T14:53:40.797 INFO:teuthology.orchestra.run.smithi001.stderr: 80500K .......... .......... .......... .......... .......... 86% 118M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 80550K .......... .......... .......... .......... .......... 86% 115M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 80600K .......... .......... .......... .......... .......... 86% 87.9M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 80650K .......... .......... .......... .......... .......... 86% 93.6M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 80700K .......... .......... .......... .......... .......... 86% 118M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 80750K .......... .......... .......... .......... .......... 86% 116M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 80800K .......... .......... .......... .......... .......... 87% 115M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 80850K .......... .......... .......... .......... .......... 87% 106M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 80900K .......... .......... .......... .......... .......... 87% 101M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 80950K .......... .......... .......... .......... .......... 87% 111M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 81000K .......... .......... .......... .......... .......... 87% 123M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 81050K .......... .......... .......... .......... .......... 87% 90.7M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 81100K .......... .......... .......... .......... .......... 87% 114M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 81150K .......... .......... .......... .......... .......... 87% 120M 0s 2024-03-09T14:53:40.798 INFO:teuthology.orchestra.run.smithi001.stderr: 81200K .......... .......... .......... .......... .......... 87% 109M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81250K .......... .......... .......... .......... .......... 87% 109M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81300K .......... .......... .......... .......... .......... 87% 76.4M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81350K .......... .......... .......... .......... .......... 87% 182M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81400K .......... .......... .......... .......... .......... 87% 136M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81450K .......... .......... .......... .......... .......... 87% 92.1M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81500K .......... .......... .......... .......... .......... 87% 113M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81550K .......... .......... .......... .......... .......... 87% 119M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81600K .......... .......... .......... .......... .......... 87% 114M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81650K .......... .......... .......... .......... .......... 87% 104M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81700K .......... .......... .......... .......... .......... 87% 118M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81750K .......... .......... .......... .......... .......... 88% 108M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81800K .......... .......... .......... .......... .......... 88% 122M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81850K .......... .......... .......... .......... .......... 88% 81.9M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81900K .......... .......... .......... .......... .......... 88% 118M 0s 2024-03-09T14:53:40.799 INFO:teuthology.orchestra.run.smithi001.stderr: 81950K .......... .......... .......... .......... .......... 88% 111M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82000K .......... .......... .......... .......... .......... 88% 118M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82050K .......... .......... .......... .......... .......... 88% 111M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82100K .......... .......... .......... .......... .......... 88% 110M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82150K .......... .......... .......... .......... .......... 88% 121M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82200K .......... .......... .......... .......... .......... 88% 112M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82250K .......... .......... .......... .......... .......... 88% 89.2M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82300K .......... .......... .......... .......... .......... 88% 114M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82350K .......... .......... .......... .......... .......... 88% 113M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82400K .......... .......... .......... .......... .......... 88% 112M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82450K .......... .......... .......... .......... .......... 88% 109M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82500K .......... .......... .......... .......... .......... 88% 117M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82550K .......... .......... .......... .......... .......... 88% 118M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82600K .......... .......... .......... .......... .......... 88% 117M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82650K .......... .......... .......... .......... .......... 89% 90.6M 0s 2024-03-09T14:53:40.800 INFO:teuthology.orchestra.run.smithi001.stderr: 82700K .......... .......... .......... .......... .......... 89% 120M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 82750K .......... .......... .......... .......... .......... 89% 111M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 82800K .......... .......... .......... .......... .......... 89% 115M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 82850K .......... .......... .......... .......... .......... 89% 110M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 82900K .......... .......... .......... .......... .......... 89% 114M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 82950K .......... .......... .......... .......... .......... 89% 96.9M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 83000K .......... .......... .......... .......... .......... 89% 114M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 83050K .......... .......... .......... .......... .......... 89% 94.1M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 83100K .......... .......... .......... .......... .......... 89% 117M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 83150K .......... .......... .......... .......... .......... 89% 113M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 83200K .......... .......... .......... .......... .......... 89% 117M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 83250K .......... .......... .......... .......... .......... 89% 105M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 83300K .......... .......... .......... .......... .......... 89% 119M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 83350K .......... .......... .......... .......... .......... 89% 108M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 83400K .......... .......... .......... .......... .......... 89% 129M 0s 2024-03-09T14:53:40.801 INFO:teuthology.orchestra.run.smithi001.stderr: 83450K .......... .......... .......... .......... .......... 89% 86.6M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 83500K .......... .......... .......... .......... .......... 89% 119M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 83550K .......... .......... .......... .......... .......... 89% 110M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 83600K .......... .......... .......... .......... .......... 90% 121M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 83650K .......... .......... .......... .......... .......... 90% 109M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 83700K .......... .......... .......... .......... .......... 90% 110M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 83750K .......... .......... .......... .......... .......... 90% 115M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 83800K .......... .......... .......... .......... .......... 90% 117M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 83850K .......... .......... .......... .......... .......... 90% 90.1M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 83900K .......... .......... .......... .......... .......... 90% 117M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 83950K .......... .......... .......... .......... .......... 90% 104M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 84000K .......... .......... .......... .......... .......... 90% 120M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 84050K .......... .......... .......... .......... .......... 90% 112M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 84100K .......... .......... .......... .......... .......... 90% 114M 0s 2024-03-09T14:53:40.802 INFO:teuthology.orchestra.run.smithi001.stderr: 84150K .......... .......... .......... .......... .......... 90% 116M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84200K .......... .......... .......... .......... .......... 90% 116M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84250K .......... .......... .......... .......... .......... 90% 89.9M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84300K .......... .......... .......... .......... .......... 90% 119M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84350K .......... .......... .......... .......... .......... 90% 110M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84400K .......... .......... .......... .......... .......... 90% 115M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84450K .......... .......... .......... .......... .......... 90% 108M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84500K .......... .......... .......... .......... .......... 91% 115M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84550K .......... .......... .......... .......... .......... 91% 113M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84600K .......... .......... .......... .......... .......... 91% 113M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84650K .......... .......... .......... .......... .......... 91% 87.4M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84700K .......... .......... .......... .......... .......... 91% 117M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84750K .......... .......... .......... .......... .......... 91% 118M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84800K .......... .......... .......... .......... .......... 91% 114M 0s 2024-03-09T14:53:40.803 INFO:teuthology.orchestra.run.smithi001.stderr: 84850K .......... .......... .......... .......... .......... 91% 106M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 84900K .......... .......... .......... .......... .......... 91% 125M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 84950K .......... .......... .......... .......... .......... 91% 118M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85000K .......... .......... .......... .......... .......... 91% 111M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85050K .......... .......... .......... .......... .......... 91% 88.4M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85100K .......... .......... .......... .......... .......... 91% 114M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85150K .......... .......... .......... .......... .......... 91% 122M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85200K .......... .......... .......... .......... .......... 91% 111M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85250K .......... .......... .......... .......... .......... 91% 110M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85300K .......... .......... .......... .......... .......... 91% 114M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85350K .......... .......... .......... .......... .......... 91% 114M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85400K .......... .......... .......... .......... .......... 91% 112M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85450K .......... .......... .......... .......... .......... 92% 93.5M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85500K .......... .......... .......... .......... .......... 92% 120M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85550K .......... .......... .......... .......... .......... 92% 93.3M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85600K .......... .......... .......... .......... .......... 92% 104M 0s 2024-03-09T14:53:40.804 INFO:teuthology.orchestra.run.smithi001.stderr: 85650K .......... .......... .......... .......... .......... 92% 147M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 85700K .......... .......... .......... .......... .......... 92% 111M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 85750K .......... .......... .......... .......... .......... 92% 129M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 85800K .......... .......... .......... .......... .......... 92% 115M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 85850K .......... .......... .......... .......... .......... 92% 86.4M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 85900K .......... .......... .......... .......... .......... 92% 115M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 85950K .......... .......... .......... .......... .......... 92% 120M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 86000K .......... .......... .......... .......... .......... 92% 90.4M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 86050K .......... .......... .......... .......... .......... 92% 146M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 86100K .......... .......... .......... .......... .......... 92% 115M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 86150K .......... .......... .......... .......... .......... 92% 116M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 86200K .......... .......... .......... .......... .......... 92% 117M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 86250K .......... .......... .......... .......... .......... 92% 87.9M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 86300K .......... .......... .......... .......... .......... 92% 121M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 86350K .......... .......... .......... .......... .......... 93% 114M 0s 2024-03-09T14:53:40.805 INFO:teuthology.orchestra.run.smithi001.stderr: 86400K .......... .......... .......... .......... .......... 93% 116M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 86450K .......... .......... .......... .......... .......... 93% 99.9M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 86500K .......... .......... .......... .......... .......... 93% 121M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 86550K .......... .......... .......... .......... .......... 93% 117M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 86600K .......... .......... .......... .......... .......... 93% 113M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 86650K .......... .......... .......... .......... .......... 93% 90.0M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 86700K .......... .......... .......... .......... .......... 93% 113M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 86750K .......... .......... .......... .......... .......... 93% 114M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 86800K .......... .......... .......... .......... .......... 93% 119M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 86850K .......... .......... .......... .......... .......... 93% 108M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 86900K .......... .......... .......... .......... .......... 93% 115M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 86950K .......... .......... .......... .......... .......... 93% 117M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 87000K .......... .......... .......... .......... .......... 93% 113M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 87050K .......... .......... .......... .......... .......... 93% 89.1M 0s 2024-03-09T14:53:40.806 INFO:teuthology.orchestra.run.smithi001.stderr: 87100K .......... .......... .......... .......... .......... 93% 122M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87150K .......... .......... .......... .......... .......... 93% 110M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87200K .......... .......... .......... .......... .......... 93% 107M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87250K .......... .......... .......... .......... .......... 93% 111M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87300K .......... .......... .......... .......... .......... 94% 125M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87350K .......... .......... .......... .......... .......... 94% 106M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87400K .......... .......... .......... .......... .......... 94% 118M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87450K .......... .......... .......... .......... .......... 94% 91.9M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87500K .......... .......... .......... .......... .......... 94% 108M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87550K .......... .......... .......... .......... .......... 94% 122M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87600K .......... .......... .......... .......... .......... 94% 112M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87650K .......... .......... .......... .......... .......... 94% 104M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87700K .......... .......... .......... .......... .......... 94% 85.0M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87750K .......... .......... .......... .......... .......... 94% 122M 0s 2024-03-09T14:53:40.807 INFO:teuthology.orchestra.run.smithi001.stderr: 87800K .......... .......... .......... .......... .......... 94% 109M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 87850K .......... .......... .......... .......... .......... 94% 91.0M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 87900K .......... .......... .......... .......... .......... 94% 111M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 87950K .......... .......... .......... .......... .......... 94% 86.4M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 88000K .......... .......... .......... .......... .......... 94% 4.14M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 88050K .......... .......... .......... .......... .......... 94% 178M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 88100K .......... .......... .......... .......... .......... 94% 71.6M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 88150K .......... .......... .......... .......... .......... 94% 60.1M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 88200K .......... .......... .......... .......... .......... 94% 112M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 88250K .......... .......... .......... .......... .......... 95% 85.6M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 88300K .......... .......... .......... .......... .......... 95% 120M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 88350K .......... .......... .......... .......... .......... 95% 114M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 88400K .......... .......... .......... .......... .......... 95% 114M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 88450K .......... .......... .......... .......... .......... 95% 106M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 88500K .......... .......... .......... .......... .......... 95% 116M 0s 2024-03-09T14:53:40.808 INFO:teuthology.orchestra.run.smithi001.stderr: 88550K .......... .......... .......... .......... .......... 95% 62.8M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 88600K .......... .......... .......... .......... .......... 95% 120M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 88650K .......... .......... .......... .......... .......... 95% 88.8M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 88700K .......... .......... .......... .......... .......... 95% 115M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 88750K .......... .......... .......... .......... .......... 95% 111M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 88800K .......... .......... .......... .......... .......... 95% 123M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 88850K .......... .......... .......... .......... .......... 95% 108M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 88900K .......... .......... .......... .......... .......... 95% 122M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 88950K .......... .......... .......... .......... .......... 95% 111M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 89000K .......... .......... .......... .......... .......... 95% 113M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 89050K .......... .......... .......... .......... .......... 95% 93.2M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 89100K .......... .......... .......... .......... .......... 95% 105M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 89150K .......... .......... .......... .......... .......... 96% 126M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 89200K .......... .......... .......... .......... .......... 96% 116M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 89250K .......... .......... .......... .......... .......... 96% 109M 0s 2024-03-09T14:53:40.809 INFO:teuthology.orchestra.run.smithi001.stderr: 89300K .......... .......... .......... .......... .......... 96% 113M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89350K .......... .......... .......... .......... .......... 96% 98.7M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89400K .......... .......... .......... .......... .......... 96% 112M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89450K .......... .......... .......... .......... .......... 96% 88.2M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89500K .......... .......... .......... .......... .......... 96% 149M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89550K .......... .......... .......... .......... .......... 96% 108M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89600K .......... .......... .......... .......... .......... 96% 112M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89650K .......... .......... .......... .......... .......... 96% 116M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89700K .......... .......... .......... .......... .......... 96% 115M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89750K .......... .......... .......... .......... .......... 96% 118M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89800K .......... .......... .......... .......... .......... 96% 113M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89850K .......... .......... .......... .......... .......... 96% 94.0M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89900K .......... .......... .......... .......... .......... 96% 117M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 89950K .......... .......... .......... .......... .......... 96% 118M 0s 2024-03-09T14:53:40.810 INFO:teuthology.orchestra.run.smithi001.stderr: 90000K .......... .......... .......... .......... .......... 96% 111M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90050K .......... .......... .......... .......... .......... 96% 112M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90100K .......... .......... .......... .......... .......... 97% 111M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90150K .......... .......... .......... .......... .......... 97% 118M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90200K .......... .......... .......... .......... .......... 97% 116M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90250K .......... .......... .......... .......... .......... 97% 89.9M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90300K .......... .......... .......... .......... .......... 97% 112M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90350K .......... .......... .......... .......... .......... 97% 119M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90400K .......... .......... .......... .......... .......... 97% 114M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90450K .......... .......... .......... .......... .......... 97% 73.7M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90500K .......... .......... .......... .......... .......... 97% 166M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90550K .......... .......... .......... .......... .......... 97% 148M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90600K .......... .......... .......... .......... .......... 97% 72.6M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90650K .......... .......... .......... .......... .......... 97% 90.3M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90700K .......... .......... .......... .......... .......... 97% 93.8M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90750K .......... .......... .......... .......... .......... 97% 89.2M 0s 2024-03-09T14:53:40.811 INFO:teuthology.orchestra.run.smithi001.stderr: 90800K .......... .......... .......... .......... .......... 97% 124M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 90850K .......... .......... .......... .......... .......... 97% 127M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 90900K .......... .......... .......... .......... .......... 97% 99.6M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 90950K .......... .......... .......... .......... .......... 97% 107M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 91000K .......... .......... .......... .......... .......... 98% 109M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 91050K .......... .......... .......... .......... .......... 98% 86.4M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 91100K .......... .......... .......... .......... .......... 98% 117M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 91150K .......... .......... .......... .......... .......... 98% 110M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 91200K .......... .......... .......... .......... .......... 98% 101M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 91250K .......... .......... .......... .......... .......... 98% 97.6M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 91300K .......... .......... .......... .......... .......... 98% 111M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 91350K .......... .......... .......... .......... .......... 98% 105M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 91400K .......... .......... .......... .......... .......... 98% 113M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 91450K .......... .......... .......... .......... .......... 98% 92.0M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 91500K .......... .......... .......... .......... .......... 98% 110M 0s 2024-03-09T14:53:40.812 INFO:teuthology.orchestra.run.smithi001.stderr: 91550K .......... .......... .......... .......... .......... 98% 111M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 91600K .......... .......... .......... .......... .......... 98% 107M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 91650K .......... .......... .......... .......... .......... 98% 99.7M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 91700K .......... .......... .......... .......... .......... 98% 113M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 91750K .......... .......... .......... .......... .......... 98% 106M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 91800K .......... .......... .......... .......... .......... 98% 112M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 91850K .......... .......... .......... .......... .......... 98% 90.2M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 91900K .......... .......... .......... .......... .......... 98% 112M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 91950K .......... .......... .......... .......... .......... 99% 109M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 92000K .......... .......... .......... .......... .......... 99% 113M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 92050K .......... .......... .......... .......... .......... 99% 101M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 92100K .......... .......... .......... .......... .......... 99% 104M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 92150K .......... .......... .......... .......... .......... 99% 115M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 92200K .......... .......... .......... .......... .......... 99% 111M 0s 2024-03-09T14:53:40.813 INFO:teuthology.orchestra.run.smithi001.stderr: 92250K .......... .......... .......... .......... .......... 99% 92.3M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92300K .......... .......... .......... .......... .......... 99% 115M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92350K .......... .......... .......... .......... .......... 99% 141M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92400K .......... .......... .......... .......... .......... 99% 118M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92450K .......... .......... .......... .......... .......... 99% 99.9M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92500K .......... .......... .......... .......... .......... 99% 116M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92550K .......... .......... .......... .......... .......... 99% 117M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92600K .......... .......... .......... .......... .......... 99% 106M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92650K .......... .......... .......... .......... .......... 99% 90.6M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92700K .......... .......... .......... .......... .......... 99% 124M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92750K .......... .......... .......... .......... .......... 99% 126M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92800K .......... .......... .......... .......... .......... 99% 213M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92850K .......... .......... .......... .......... .......... 99% 177M 0s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 92900K 100% 134G=0.9s 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-09T14:53:40.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09 14:53:40 (98.0 MB/s) - 'gradle-6.0.1-bin.zip' saved [95129672/95129672] 2024-03-09T14:53:40.815 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-09T14:53:40.824 INFO:teuthology.orchestra.run.smithi001.stdout:Archive: gradle-6.0.1-bin.zip 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: creating: /opt/gradle/gradle-6.0.1/ 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/getting-started.html 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/LICENSE 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/NOTICE 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: creating: /opt/gradle/gradle-6.0.1/init.d/ 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/init.d/readme.txt 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: creating: /opt/gradle/gradle-6.0.1/bin/ 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/bin/gradle 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/bin/gradle.bat 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: creating: /opt/gradle/gradle-6.0.1/lib/ 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-api-metadata-6.0.1.jar 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-launcher-6.0.1.jar 2024-03-09T14:53:40.825 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-runtime-api-info-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-kotlin-dsl-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-tooling-api-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-wrapper-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-installation-beacon-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-bootstrap-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-core-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-file-collections-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-model-groovy-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-execution-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-model-core-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-build-cache-packaging-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-build-cache-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-core-api-6.0.1.jar 2024-03-09T14:53:40.826 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-persistent-cache-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-worker-processes-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-logging-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-jvm-services-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-process-services-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-resources-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-messaging-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-native-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-base-services-groovy-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-base-services-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-build-option-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-cli-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-snapshots-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-files-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-kotlin-dsl-tooling-models-6.0.1.jar 2024-03-09T14:53:40.827 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-compiler-embeddable-1.3.50-patched-for-gradle-6.0.1.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-hashing-6.0.1.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-pineapple-6.0.1.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-docs-6.0.1.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/groovy-all-1.3-2.5.8.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/jul-to-slf4j-1.7.28.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/log4j-over-slf4j-1.7.28.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/jcl-over-slf4j-1.7.28.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/slf4j-api-1.7.28.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/guava-27.1-android.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/commons-io-2.6.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/commons-lang-2.6.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/asm-commons-7.1.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/asm-analysis-7.1.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/asm-tree-7.1.jar 2024-03-09T14:53:40.828 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/asm-7.1.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/ant-1.10.7.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-stdlib-jdk8-1.3.50.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/javax.inject-1.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-common-1.3.50.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-jvm-1.3.50.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-jvm-host-embeddable-1.3.50.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-compiler-embeddable-1.3.50.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-compiler-impl-embeddable-1.3.50.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-sam-with-receiver-compiler-plugin-1.3.50.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlinx-metadata-jvm-0.1.0.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/commons-compress-1.19.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/fastutil-8.3.0-min.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/jansi-1.18.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/jsr305-3.0.2.jar 2024-03-09T14:53:40.829 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kryo-2.24.0.jar 2024-03-09T14:53:40.830 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-0.18.jar 2024-03-09T14:53:40.830 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/objenesis-2.6.jar 2024-03-09T14:53:40.830 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/trove4j-1.0.20181211.jar 2024-03-09T14:53:40.830 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/xml-apis-1.4.01.jar 2024-03-09T14:53:40.843 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-reflect-1.3.50.jar 2024-03-09T14:53:40.843 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-stdlib-jdk7-1.3.50.jar 2024-03-09T14:53:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-stdlib-1.3.50.jar 2024-03-09T14:53:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/annotations-13.0.jar 2024-03-09T14:53:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/ant-launcher-1.10.7.jar 2024-03-09T14:53:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/minlog-1.2.jar 2024-03-09T14:53:40.853 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-script-runtime-1.3.50.jar 2024-03-09T14:53:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-daemon-embeddable-1.3.50.jar 2024-03-09T14:53:40.861 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/failureaccess-1.0.1.jar 2024-03-09T14:53:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-stdlib-common-1.3.50.jar 2024-03-09T14:53:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-freebsd-amd64-libcpp-0.18.jar 2024-03-09T14:53:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-freebsd-amd64-libstdcpp-0.18.jar 2024-03-09T14:53:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-freebsd-i386-libcpp-0.18.jar 2024-03-09T14:53:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-freebsd-i386-libstdcpp-0.18.jar 2024-03-09T14:53:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-aarch64-0.18.jar 2024-03-09T14:53:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-amd64-0.18.jar 2024-03-09T14:53:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-i386-0.18.jar 2024-03-09T14:53:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-osx-amd64-0.18.jar 2024-03-09T14:53:40.864 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-windows-amd64-0.18.jar 2024-03-09T14:53:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-windows-amd64-min-0.18.jar 2024-03-09T14:53:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-windows-i386-0.18.jar 2024-03-09T14:53:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-windows-i386-min-0.18.jar 2024-03-09T14:53:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-aarch64-ncurses5-0.18.jar 2024-03-09T14:53:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-amd64-ncurses5-0.18.jar 2024-03-09T14:53:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-amd64-ncurses6-0.18.jar 2024-03-09T14:53:40.866 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-i386-ncurses5-0.18.jar 2024-03-09T14:53:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-i386-ncurses6-0.18.jar 2024-03-09T14:53:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: creating: /opt/gradle/gradle-6.0.1/lib/plugins/ 2024-03-09T14:53:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-antlr-6.0.1.jar 2024-03-09T14:53:40.867 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-code-quality-6.0.1.jar 2024-03-09T14:53:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-signing-6.0.1.jar 2024-03-09T14:53:40.868 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-tooling-native-6.0.1.jar 2024-03-09T14:53:40.869 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ide-native-6.0.1.jar 2024-03-09T14:53:40.870 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-testing-native-6.0.1.jar 2024-03-09T14:53:40.873 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-native-6.0.1.jar 2024-03-09T14:53:40.875 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-kotlin-dsl-provider-plugins-6.0.1.jar 2024-03-09T14:53:40.875 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-plugin-development-6.0.1.jar 2024-03-09T14:53:40.876 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-maven-6.0.1.jar 2024-03-09T14:53:40.877 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ide-play-6.0.1.jar 2024-03-09T14:53:40.879 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ide-6.0.1.jar 2024-03-09T14:53:40.880 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-scala-6.0.1.jar 2024-03-09T14:53:40.880 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ear-6.0.1.jar 2024-03-09T14:53:40.882 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-platform-play-6.0.1.jar 2024-03-09T14:53:40.882 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-javascript-6.0.1.jar 2024-03-09T14:53:40.883 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ivy-6.0.1.jar 2024-03-09T14:53:40.883 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-jacoco-6.0.1.jar 2024-03-09T14:53:40.885 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-build-init-6.0.1.jar 2024-03-09T14:53:40.887 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-kotlin-dsl-tooling-builders-6.0.1.jar 2024-03-09T14:53:40.889 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-plugins-6.0.1.jar 2024-03-09T14:53:40.889 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-test-kit-6.0.1.jar 2024-03-09T14:53:40.889 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-groovy-6.0.1.jar 2024-03-09T14:53:40.890 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-scala-6.0.1.jar 2024-03-09T14:53:40.890 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-testing-junit-platform-6.0.1.jar 2024-03-09T14:53:40.890 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-tooling-api-builders-6.0.1.jar 2024-03-09T14:53:40.891 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-testing-jvm-6.0.1.jar 2024-03-09T14:53:40.894 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-java-6.0.1.jar 2024-03-09T14:53:40.894 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-jvm-6.0.1.jar 2024-03-09T14:53:40.894 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-platform-jvm-6.0.1.jar 2024-03-09T14:53:40.898 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-platform-native-6.0.1.jar 2024-03-09T14:53:40.901 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-diagnostics-6.0.1.jar 2024-03-09T14:53:40.901 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-publish-6.0.1.jar 2024-03-09T14:53:40.902 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-testing-base-6.0.1.jar 2024-03-09T14:53:40.904 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-platform-base-6.0.1.jar 2024-03-09T14:53:40.904 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-plugin-use-6.0.1.jar 2024-03-09T14:53:40.910 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-instant-execution-6.0.1.jar 2024-03-09T14:53:40.911 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-version-control-6.0.1.jar 2024-03-09T14:53:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-dependency-management-6.0.1.jar 2024-03-09T14:53:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-composite-builds-6.0.1.jar 2024-03-09T14:53:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-reporting-6.0.1.jar 2024-03-09T14:53:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-resources-s3-6.0.1.jar 2024-03-09T14:53:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-resources-gcs-6.0.1.jar 2024-03-09T14:53:40.928 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-build-cache-http-6.0.1.jar 2024-03-09T14:53:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-resources-http-6.0.1.jar 2024-03-09T14:53:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-resources-sftp-6.0.1.jar 2024-03-09T14:53:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-build-profile-6.0.1.jar 2024-03-09T14:53:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-workers-6.0.1.jar 2024-03-09T14:53:40.947 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/org.eclipse.jgit-5.5.0.201909110433-r.jar 2024-03-09T14:53:40.947 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/wagon-http-3.0.0.jar 2024-03-09T14:53:40.947 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/wagon-http-shared-3.0.0.jar 2024-03-09T14:53:40.947 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-api-services-storage-v1-rev136-1.25.0.jar 2024-03-09T14:53:40.949 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-api-client-1.25.0.jar 2024-03-09T14:53:40.949 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/asm-util-7.1.jar 2024-03-09T14:53:40.957 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/ivy-2.3.0.jar 2024-03-09T14:53:40.959 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-compat-3.0.4.jar 2024-03-09T14:53:40.963 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-core-3.0.4.jar 2024-03-09T14:53:40.963 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/pmaven-common-0.8-20100325.jar 2024-03-09T14:53:40.964 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/pmaven-groovy-0.8-20100325.jar 2024-03-09T14:53:40.964 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/wagon-file-3.0.0.jar 2024-03-09T14:53:40.965 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-container-default-1.7.1.jar 2024-03-09T14:53:40.965 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-connector-wagon-1.13.1.jar 2024-03-09T14:53:40.968 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/bcpg-jdk15on-1.63.jar 2024-03-09T14:53:40.976 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/rhino-1.7.10.jar 2024-03-09T14:53:40.977 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gson-2.8.5.jar 2024-03-09T14:53:40.979 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/simple-4.1.21.jar 2024-03-09T14:53:40.979 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jatl-0.2.3.jar 2024-03-09T14:53:40.980 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-plugin-api-3.0.4.jar 2024-03-09T14:53:40.980 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-oauth-client-1.25.0.jar 2024-03-09T14:53:40.980 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-http-client-jackson2-1.25.0.jar 2024-03-09T14:53:40.983 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-http-client-1.25.0.jar 2024-03-09T14:53:40.987 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/testng-6.3.1.jar 2024-03-09T14:53:40.988 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/snakeyaml-1.17.jar 2024-03-09T14:53:40.989 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/dd-plist-1.21.jar 2024-03-09T14:53:40.991 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/junit-4.12.jar 2024-03-09T14:53:40.994 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/bsh-2.0b6.jar 2024-03-09T14:53:40.994 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/junit-platform-launcher-1.3.1.jar 2024-03-09T14:53:41.001 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aws-java-sdk-s3-1.11.633.jar 2024-03-09T14:53:41.005 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aws-java-sdk-kms-1.11.633.jar 2024-03-09T14:53:41.011 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aws-java-sdk-core-1.11.633.jar 2024-03-09T14:53:41.016 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/httpclient-4.5.10.jar 2024-03-09T14:53:41.019 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jcifs-1.3.17.jar 2024-03-09T14:53:41.020 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/nekohtml-1.9.22.jar 2024-03-09T14:53:41.029 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/xercesImpl-2.12.0.jar 2024-03-09T14:53:41.031 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jsch-0.1.55.jar 2024-03-09T14:53:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jaxb-impl-2.3.2.jar 2024-03-09T14:53:41.038 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jmespath-java-1.11.633.jar 2024-03-09T14:53:41.046 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jackson-databind-2.9.10.jar 2024-03-09T14:53:41.049 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jackson-core-2.9.10.jar 2024-03-09T14:53:41.049 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jackson-annotations-2.9.10.jar 2024-03-09T14:53:41.053 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/joda-time-2.10.4.jar 2024-03-09T14:53:41.059 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/bcpkix-jdk15on-1.61.jar 2024-03-09T14:53:41.089 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/bcprov-jdk15on-1.63.jar 2024-03-09T14:53:41.091 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/commons-codec-1.13.jar 2024-03-09T14:53:41.091 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-aether-provider-3.0.4.jar 2024-03-09T14:53:41.092 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-impl-1.13.1.jar 2024-03-09T14:53:41.093 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-util-1.13.1.jar 2024-03-09T14:53:41.093 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-spi-1.13.1.jar 2024-03-09T14:53:41.094 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-api-1.13.1.jar 2024-03-09T14:53:41.094 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/hamcrest-core-1.3.jar 2024-03-09T14:53:41.096 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/httpcore-4.4.12.jar 2024-03-09T14:53:41.098 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jcommander-1.72.jar 2024-03-09T14:53:41.098 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jzlib-1.1.3.jar 2024-03-09T14:53:41.098 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-artifact-3.0.4.jar 2024-03-09T14:53:41.099 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-model-builder-3.0.4.jar 2024-03-09T14:53:41.100 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-model-3.0.4.jar 2024-03-09T14:53:41.100 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-repository-metadata-3.0.4.jar 2024-03-09T14:53:41.101 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-settings-builder-3.0.4.jar 2024-03-09T14:53:41.102 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-settings-3.0.4.jar 2024-03-09T14:53:41.102 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-sec-dispatcher-1.3.jar 2024-03-09T14:53:41.102 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-cipher-1.7.jar 2024-03-09T14:53:41.102 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-classworlds-2.5.1.jar 2024-03-09T14:53:41.102 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-component-annotations-1.5.5.jar 2024-03-09T14:53:41.103 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-interpolation-1.14.jar 2024-03-09T14:53:41.103 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/wagon-provider-api-3.0.0.jar 2024-03-09T14:53:41.104 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-utils-3.1.0.jar 2024-03-09T14:53:41.105 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/xbean-reflect-3.7.jar 2024-03-09T14:53:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/ion-java-1.0.2.jar 2024-03-09T14:53:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/junit-platform-engine-1.3.1.jar 2024-03-09T14:53:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/junit-platform-commons-1.3.1.jar 2024-03-09T14:53:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/apiguardian-api-1.0.0.jar 2024-03-09T14:53:41.111 INFO:teuthology.orchestra.run.smithi001.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/opentest4j-1.1.1.jar 2024-03-09T14:53:41.146 INFO:teuthology.orchestra.run.smithi001.stdout:export PATH=/opt/gradle/gradle-6.0.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin 2024-03-09T14:53:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:53:41.612 INFO:teuthology.orchestra.run.smithi001.stdout:Welcome to Gradle 6.0.1! 2024-03-09T14:53:41.612 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:53:41.612 INFO:teuthology.orchestra.run.smithi001.stdout:Here are the highlights of this release: 2024-03-09T14:53:41.612 INFO:teuthology.orchestra.run.smithi001.stdout: - Substantial improvements in dependency management, including 2024-03-09T14:53:41.612 INFO:teuthology.orchestra.run.smithi001.stdout: - Publishing Gradle Module Metadata in addition to pom.xml 2024-03-09T14:53:41.612 INFO:teuthology.orchestra.run.smithi001.stdout: - Advanced control of transitive versions 2024-03-09T14:53:41.612 INFO:teuthology.orchestra.run.smithi001.stdout: - Support for optional features and dependencies 2024-03-09T14:53:41.612 INFO:teuthology.orchestra.run.smithi001.stdout: - Rules to tweak published metadata 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout: - Support for Java 13 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout: - Faster incremental Java and Groovy compilation 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout: - New Zinc compiler for Scala 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout: - VS2019 support 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout: - Support for Gradle Enterprise plugin 3.0 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout:For more details see https://docs.gradle.org/6.0.1/release-notes.html 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout:------------------------------------------------------------ 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout:Gradle 6.0.1 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout:------------------------------------------------------------ 2024-03-09T14:53:41.613 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:53:41.614 INFO:teuthology.orchestra.run.smithi001.stdout:Build time: 2019-11-18 20:25:01 UTC 2024-03-09T14:53:41.614 INFO:teuthology.orchestra.run.smithi001.stdout:Revision: fad121066a68c4701acd362daf4287a7c309a0f5 2024-03-09T14:53:41.615 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:53:41.615 INFO:teuthology.orchestra.run.smithi001.stdout:Kotlin: 1.3.50 2024-03-09T14:53:41.615 INFO:teuthology.orchestra.run.smithi001.stdout:Groovy: 2.5.8 2024-03-09T14:53:41.615 INFO:teuthology.orchestra.run.smithi001.stdout:Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019 2024-03-09T14:53:41.615 INFO:teuthology.orchestra.run.smithi001.stdout:JVM: 11.0.17 (Ubuntu 11.0.17+8-post-Ubuntu-1ubuntu222.04) 2024-03-09T14:53:41.615 INFO:teuthology.orchestra.run.smithi001.stdout:OS: Linux 5.15.0-100-generic amd64 2024-03-09T14:53:41.615 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:53:41.629 DEBUG:tasks.s3tests_java:S3 Tests Java: BEGIN 2024-03-09T14:53:41.630 DEBUG:tasks.s3tests_java:S3 Tests Java: Cluster config is: ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-09T14:53:41.630 DEBUG:tasks.s3tests_java:S3 Tests Java: Host is: ubuntu@smithi001.front.sepia.ceph.com 2024-03-09T14:53:41.630 DEBUG:tasks.s3tests_java:S3 Tests Java: Cluster config is: ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-09T14:53:41.630 DEBUG:tasks.s3tests_java:S3 Tests Java: Host is: ubuntu@smithi053.front.sepia.ceph.com 2024-03-09T14:53:41.630 INFO:tasks.s3tests_java:S3 Tests Java: Creating S3 users... 2024-03-09T14:53:41.855 DEBUG:tasks.s3tests_java:S3 Tests Java: s3tests_conf is {'bucket_prefix': 'test-', 's3main': {'access_key': None, 'access_secret': None, 'region': 'mexico', 'endpoint': None, 'port': None, 'display_name': None, 'email': None, 'is_secure': None, 'SSE': 'AES256', 's3dir': 'folder', 'dir': '../data'}} 2024-03-09T14:53:41.855 DEBUG:tasks.s3tests_java:S3 Tests Java: Creating user tester.client.0 2024-03-09T14:53:41.855 DEBUG:tasks.s3tests_java:S3 Tests Java: s3tests_conf[s3main] is {'access_key': 'KNSHDPEDHGBLMBSLSSNA', 'access_secret': 'mfIqQpV+iY9OH7GQnthZ23MwlexmGQbp2T0stgBLjfAcvoIxIsI2EQ==', 'region': 'mexico', 'endpoint': 'smithi001.front.sepia.ceph.com:443', 'port': 443, 'display_name': 'Ms. tester.client.0', 'email': 'tester.client.0_test@test.test', 'is_secure': True, 'SSE': 'AES256', 's3dir': 'folder', 'dir': '../data', 'user_id': 'tester.client.0', 'host': 'smithi001.front.sepia.ceph.com'} 2024-03-09T14:53:41.855 DEBUG:tasks.s3tests_java:S3 Tests Java: Setion, User = s3main, tester.client.0 2024-03-09T14:53:41.856 INFO:tasks.s3tests_java:['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '-n', 'client.0', 'user', 'create', '--uid', 'tester.client.0', '--display-name', 'Ms. tester.client.0', '--access-key', 'KNSHDPEDHGBLMBSLSSNA', '--secret', 'mfIqQpV+iY9OH7GQnthZ23MwlexmGQbp2T0stgBLjfAcvoIxIsI2EQ==', '--email', 'tester.client.0_test@test.test', '--cluster', 'ceph'] 2024-03-09T14:53:41.856 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid tester.client.0 --display-name 'Ms. tester.client.0' --access-key KNSHDPEDHGBLMBSLSSNA --secret mfIqQpV+iY9OH7GQnthZ23MwlexmGQbp2T0stgBLjfAcvoIxIsI2EQ== --email tester.client.0_test@test.test --cluster ceph 2024-03-09T14:53:41.904 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:41.904 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:41.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.945+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.945+0000 7f06b09c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:41.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.945+0000 7f06b09c5980 20 realm 2024-03-09T14:53:41.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.945+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.945+0000 7f06b09c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:41.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.945+0000 7f06b09c5980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:41.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.945+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.945+0000 7f06b09c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:41.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.945+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.949+0000 7f06b09c5980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:41.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.949+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.949+0000 7f06b09c5980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:41.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.949+0000 7f06b09c5980 20 searching for the correct realm 2024-03-09T14:53:41.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.953+0000 7f06adcdc640 0 -- 172.21.15.1:0/2042518130 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x556048f339c0 msgr2=0x556048f35e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:41.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.961+0000 7f06ae4dd640 0 -- 172.21.15.1:0/2042518130 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f06a405c9a0 msgr2=0x7f06a40731d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 zone default found 2024-03-09T14:53:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 4 Realm: () 2024-03-09T14:53:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 zonegroup default 2024-03-09T14:53:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:41.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.969+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.973+0000 7f06b09c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.973+0000 7f06b09c5980 20 started sync module instance, tier type = 2024-03-09T14:53:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.973+0000 7f06b09c5980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:41.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.985+0000 7f06b09c5980 20 add_watcher() i=0 2024-03-09T14:53:41.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.985+0000 7f06b09c5980 20 add_watcher() i=1 2024-03-09T14:53:41.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.985+0000 7f06b09c5980 20 add_watcher() i=2 2024-03-09T14:53:41.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.985+0000 7f06b09c5980 20 add_watcher() i=3 2024-03-09T14:53:41.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.985+0000 7f06b09c5980 20 add_watcher() i=4 2024-03-09T14:53:41.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.985+0000 7f06b09c5980 20 add_watcher() i=5 2024-03-09T14:53:41.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.985+0000 7f06b09c5980 20 add_watcher() i=6 2024-03-09T14:53:41.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 20 add_watcher() i=7 2024-03-09T14:53:41.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:41.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f0564fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:41.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:41.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:41.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 5 note: GC not initialized 2024-03-09T14:53:41.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f04df7fe640 20 reqs_thread_entry: start 2024-03-09T14:53:41.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:41.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 20 Filter name: none 2024-03-09T14:53:41.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f04dd7fa640 20 reqs_thread_entry: start 2024-03-09T14:53:41.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 10 cache get: name=default.rgw.meta+users.uid+tester.client.0 : miss 2024-03-09T14:53:41.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:41.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x0 2024-03-09T14:53:41.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 10 adding default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2024-03-09T14:53:41.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 10 cache get: name=default.rgw.meta+users.email+tester.client.0_test@test.test : miss 2024-03-09T14:53:41.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.989+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:41.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 10 cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x0 2024-03-09T14:53:41.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 10 adding default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2024-03-09T14:53:41.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 10 cache get: name=default.rgw.meta+users.keys+KNSHDPEDHGBLMBSLSSNA : miss 2024-03-09T14:53:41.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 20 rados->read ofs=0 len=0 2024-03-09T14:53:41.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:41.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 10 cache put: name=default.rgw.meta+users.keys+KNSHDPEDHGBLMBSLSSNA info.flags=0x0 2024-03-09T14:53:41.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 10 adding default.rgw.meta+users.keys+KNSHDPEDHGBLMBSLSSNA to cache LRU end 2024-03-09T14:53:41.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 10 cache get: name=default.rgw.meta+users.keys+KNSHDPEDHGBLMBSLSSNA : hit (negative entry) 2024-03-09T14:53:41.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 10 cache get: name=default.rgw.meta+users.keys+KNSHDPEDHGBLMBSLSSNA : hit (negative entry) 2024-03-09T14:53:41.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x17 2024-03-09T14:53:41.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 10 moving default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2024-03-09T14:53:41.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f06b09c5980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.uid:tester.client.0, ofs0, ns] 2024-03-09T14:53:41.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f0591ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562427 cookie 93872029135104 notifier 7428 bl.length()=670 2024-03-09T14:53:41.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f0591ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x17 2024-03-09T14:53:41.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.993+0000 7f0591ffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2024-03-09T14:53:41.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.997+0000 7f06b09c5980 10 cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x7 2024-03-09T14:53:41.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.997+0000 7f06b09c5980 10 moving default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2024-03-09T14:53:41.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.997+0000 7f06b09c5980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.email:tester.client.0_test@test.test, ofs0, ns] 2024-03-09T14:53:41.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.997+0000 7f0591ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562402 cookie 93872029139152 notifier 7428 bl.length()=200 2024-03-09T14:53:41.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.997+0000 7f0591ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x7 2024-03-09T14:53:41.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:41.997+0000 7f0591ffb640 10 rgw watcher librados: moving default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2024-03-09T14:53:42.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.001+0000 7f06b09c5980 10 cache put: name=default.rgw.meta+users.keys+KNSHDPEDHGBLMBSLSSNA info.flags=0x7 2024-03-09T14:53:42.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.001+0000 7f06b09c5980 10 moving default.rgw.meta+users.keys+KNSHDPEDHGBLMBSLSSNA to cache LRU end 2024-03-09T14:53:42.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.001+0000 7f06b09c5980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.keys:KNSHDPEDHGBLMBSLSSNA, ofs0, ns] 2024-03-09T14:53:42.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.001+0000 7f05917fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562428 cookie 93872029135104 notifier 7428 bl.length()=189 2024-03-09T14:53:42.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.001+0000 7f05917fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+KNSHDPEDHGBLMBSLSSNA info.flags=0x7 2024-03-09T14:53:42.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.001+0000 7f05917fa640 10 rgw watcher librados: moving default.rgw.meta+users.keys+KNSHDPEDHGBLMBSLSSNA to cache LRU end 2024-03-09T14:53:42.004 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T14:53:42.004 INFO:teuthology.orchestra.run.smithi001.stdout: "user_id": "tester.client.0", 2024-03-09T14:53:42.004 INFO:teuthology.orchestra.run.smithi001.stdout: "display_name": "Ms. tester.client.0", 2024-03-09T14:53:42.004 INFO:teuthology.orchestra.run.smithi001.stdout: "email": "tester.client.0_test@test.test", 2024-03-09T14:53:42.004 INFO:teuthology.orchestra.run.smithi001.stdout: "suspended": 0, 2024-03-09T14:53:42.004 INFO:teuthology.orchestra.run.smithi001.stdout: "max_buckets": 1000, 2024-03-09T14:53:42.004 INFO:teuthology.orchestra.run.smithi001.stdout: "subusers": [], 2024-03-09T14:53:42.004 INFO:teuthology.orchestra.run.smithi001.stdout: "keys": [ 2024-03-09T14:53:42.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T14:53:42.005 INFO:teuthology.orchestra.run.smithi001.stdout: "user": "tester.client.0", 2024-03-09T14:53:42.005 INFO:teuthology.orchestra.run.smithi001.stdout: "access_key": "KNSHDPEDHGBLMBSLSSNA", 2024-03-09T14:53:42.005 INFO:teuthology.orchestra.run.smithi001.stdout: "secret_key": "mfIqQpV+iY9OH7GQnthZ23MwlexmGQbp2T0stgBLjfAcvoIxIsI2EQ==", 2024-03-09T14:53:42.005 INFO:teuthology.orchestra.run.smithi001.stdout: "active": true, 2024-03-09T14:53:42.005 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T14:53:41.994468Z" 2024-03-09T14:53:42.005 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T14:53:42.005 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-03-09T14:53:42.005 INFO:teuthology.orchestra.run.smithi001.stdout: "swift_keys": [], 2024-03-09T14:53:42.005 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": [], 2024-03-09T14:53:42.005 INFO:teuthology.orchestra.run.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T14:53:42.005 INFO:teuthology.orchestra.run.smithi001.stdout: "default_placement": "", 2024-03-09T14:53:42.005 INFO:teuthology.orchestra.run.smithi001.stdout: "default_storage_class": "", 2024-03-09T14:53:42.006 INFO:teuthology.orchestra.run.smithi001.stdout: "placement_tags": [], 2024-03-09T14:53:42.006 INFO:teuthology.orchestra.run.smithi001.stdout: "bucket_quota": { 2024-03-09T14:53:42.006 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T14:53:42.006 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T14:53:42.006 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T14:53:42.006 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T14:53:42.006 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T14:53:42.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T14:53:42.006 INFO:teuthology.orchestra.run.smithi001.stdout: "user_quota": { 2024-03-09T14:53:42.006 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T14:53:42.006 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T14:53:42.006 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T14:53:42.007 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T14:53:42.007 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T14:53:42.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T14:53:42.007 INFO:teuthology.orchestra.run.smithi001.stdout: "temp_url_keys": [], 2024-03-09T14:53:42.007 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "rgw", 2024-03-09T14:53:42.007 INFO:teuthology.orchestra.run.smithi001.stdout: "mfa_ids": [], 2024-03-09T14:53:42.007 INFO:teuthology.orchestra.run.smithi001.stdout: "account_id": "", 2024-03-09T14:53:42.007 INFO:teuthology.orchestra.run.smithi001.stdout: "path": "/", 2024-03-09T14:53:42.007 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T14:53:41.994453Z", 2024-03-09T14:53:42.007 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-03-09T14:53:42.007 INFO:teuthology.orchestra.run.smithi001.stdout: "group_ids": [] 2024-03-09T14:53:42.007 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T14:53:42.008 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:53:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.005+0000 7f0564fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:42.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.009+0000 7f06b09c5980 20 remove_watcher() i=0 2024-03-09T14:53:42.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.009+0000 7f06b09c5980 2 removed watcher, disabling cache 2024-03-09T14:53:42.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.013+0000 7f06b09c5980 20 remove_watcher() i=1 2024-03-09T14:53:42.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.013+0000 7f06b09c5980 20 remove_watcher() i=2 2024-03-09T14:53:42.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.017+0000 7f06b09c5980 20 remove_watcher() i=3 2024-03-09T14:53:42.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.017+0000 7f06b09c5980 20 remove_watcher() i=4 2024-03-09T14:53:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.021+0000 7f06b09c5980 20 remove_watcher() i=5 2024-03-09T14:53:42.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.021+0000 7f06b09c5980 20 remove_watcher() i=6 2024-03-09T14:53:42.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.021+0000 7f06b09c5980 20 remove_watcher() i=7 2024-03-09T14:53:42.035 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T14:53:42.035 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/archive/s3-tests-java.client.0.conf 2024-03-09T14:53:42.042 INFO:tasks.s3tests_java:S3 Tests Java: Running tests... 2024-03-09T14:53:42.042 DEBUG:teuthology.orchestra.run.smithi001:> cp /home/ubuntu/cephtest/archive/s3-tests-java.client.0.conf /home/ubuntu/cephtest/s3-tests-java/config.properties 2024-03-09T14:53:42.087 DEBUG:teuthology.orchestra.run.smithi001:> radosgw-admin gc process --include-all 2024-03-09T14:53:42.156 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:42.156 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:42.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.197+0000 7f2422a8d980 20 rados->read ofs=0 len=0 2024-03-09T14:53:42.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.201+0000 7f2422a8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:42.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.201+0000 7f2422a8d980 20 realm 2024-03-09T14:53:42.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.201+0000 7f2422a8d980 20 rados->read ofs=0 len=0 2024-03-09T14:53:42.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.201+0000 7f2422a8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:42.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.201+0000 7f2422a8d980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:42.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.201+0000 7f2422a8d980 20 rados->read ofs=0 len=0 2024-03-09T14:53:42.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.201+0000 7f2422a8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:42.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.201+0000 7f2422a8d980 20 rados->read ofs=0 len=0 2024-03-09T14:53:42.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.201+0000 7f2422a8d980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:42.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.201+0000 7f2422a8d980 20 rados->read ofs=0 len=0 2024-03-09T14:53:42.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.201+0000 7f2422a8d980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:42.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.201+0000 7f2422a8d980 20 searching for the correct realm 2024-03-09T14:53:42.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.205+0000 7f241fda4640 0 -- 172.21.15.1:0/3477198682 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f24080022e0 msgr2=0x7f2410054040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.421+0000 7f24205a5640 0 -- 172.21.15.1:0/3477198682 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f241410b850 msgr2=0x7f24140da160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.637+0000 7f2422a8d980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.637+0000 7f2422a8d980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.637+0000 7f2422a8d980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.637+0000 7f2422a8d980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.637+0000 7f2422a8d980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.637+0000 7f2422a8d980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.637+0000 7f2422a8d980 20 rados->read ofs=0 len=0 2024-03-09T14:53:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.637+0000 7f2422a8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.637+0000 7f2422a8d980 20 rados->read ofs=0 len=0 2024-03-09T14:53:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 rados->read ofs=0 len=0 2024-03-09T14:53:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 zone default found 2024-03-09T14:53:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 4 Realm: () 2024-03-09T14:53:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 zonegroup default 2024-03-09T14:53:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 rados->read ofs=0 len=0 2024-03-09T14:53:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 rados->read ofs=0 len=0 2024-03-09T14:53:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 rados->read ofs=0 len=0 2024-03-09T14:53:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 started sync module instance, tier type = 2024-03-09T14:53:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.641+0000 7f2422a8d980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:42.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.669+0000 7f2422a8d980 20 add_watcher() i=0 2024-03-09T14:53:42.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.669+0000 7f2422a8d980 20 add_watcher() i=1 2024-03-09T14:53:42.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.669+0000 7f2422a8d980 20 add_watcher() i=2 2024-03-09T14:53:42.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.669+0000 7f2422a8d980 20 add_watcher() i=3 2024-03-09T14:53:42.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.669+0000 7f2422a8d980 20 add_watcher() i=4 2024-03-09T14:53:42.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.669+0000 7f2422a8d980 20 add_watcher() i=5 2024-03-09T14:53:42.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.669+0000 7f2422a8d980 20 add_watcher() i=6 2024-03-09T14:53:42.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.669+0000 7f2422a8d980 20 add_watcher() i=7 2024-03-09T14:53:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.669+0000 7f2422a8d980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:42.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.673+0000 7f22d67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:42.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.677+0000 7f2422a8d980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:42.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.677+0000 7f2422a8d980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.769+0000 7f22d57e2640 20 reqs_thread_entry: start 2024-03-09T14:53:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.769+0000 7f2422a8d980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.769+0000 7f2422a8d980 20 Filter name: none 2024-03-09T14:53:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.773+0000 7f2246ffd640 20 reqs_thread_entry: start 2024-03-09T14:53:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.773+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T14:53:42.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.773+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.773+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.773+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker='' 2024-03-09T14:53:42.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.773+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.JeUygVC1yQfDqC_IPNqgVkOubTC1qT-_1_91928a36ca750d9150dccd2ff36d26b8', time=2024-03-09T16:48:23.243278+0000, chain.objs.size()=1 2024-03-09T14:53:42.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.773+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.JeUygVC1yQfDqC_IPNqgVkOubTC1qT-_0 2024-03-09T14:53:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.773+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14750674554078757919', time=2024-03-09T16:48:23.727943+0000, chain.objs.size()=1 2024-03-09T14:53:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.773+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.7__shadow_.1ge7qY3D8o3sXVXQcIcHbF3ZSkNtPkB_0 2024-03-09T14:53:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.777+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6053706572917302993', time=2024-03-09T16:50:20.396387+0000, chain.objs.size()=1 2024-03-09T14:53:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.777+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.NlcTKP_Or8i_MiCNW6y-7Kee4XvzWxm_0 2024-03-09T14:53:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.777+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2542030416596507119', time=2024-03-09T16:50:24.689632+0000, chain.objs.size()=1 2024-03-09T14:53:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.777+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.uOQ7K1BxVwB0W6t5SYpWRhL29cxmrQn_0 2024-03-09T14:53:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.777+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13045328662266162725', time=2024-03-09T16:51:03.745059+0000, chain.objs.size()=1 2024-03-09T14:53:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.777+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.e9-JN8RukLMIbadzex-pUX2RUGPxNxx_0 2024-03-09T14:53:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.777+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13330290326501767540', time=2024-03-09T16:51:47.753189+0000, chain.objs.size()=1 2024-03-09T14:53:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.777+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.mT-67qBIexQWumTn4Z0KXdoJT6Cl8p-_0 2024-03-09T14:53:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.777+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5762468514334850737', time=2024-03-09T16:52:22.779069+0000, chain.objs.size()=1 2024-03-09T14:53:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.777+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.lmWm1NfpszT5EVbfRjKZU-D-ezt9nDf_0 2024-03-09T14:53:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.777+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8264047061900835502', time=2024-03-09T16:52:22.844982+0000, chain.objs.size()=1 2024-03-09T14:53:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.777+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.giev1W_DoTh4mBSPsuQTx5mZ_4AJUhy_0 2024-03-09T14:53:42.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.781+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:42.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.785+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T14:53:42.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.789+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.789+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='_2WVWqnDFZT-Z4WQWlJGi7YMQOma_Vhb', time=2024-03-09T16:48:23.213725+0000, chain.objs.size()=1 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.K2rC165_M7WeAiZsE61r3gabQ8KbHlD_0 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='_iPOmxjM6KaE4nLwN33Mt9luBFK7J3Bb', time=2024-03-09T16:48:23.235056+0000, chain.objs.size()=1 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.am4nZkfyda-CXJgV6DAEI9-rA5g8prB_0 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='_cAoOva1YdXWJE2zH4WiIwMcC3vGTdVU', time=2024-03-09T16:48:23.258638+0000, chain.objs.size()=1 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.R9hHXYlSeqza3oyBXe5Q55Jvr-ebV4a_0 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7712514785575230782', time=2024-03-09T16:50:20.398306+0000, chain.objs.size()=1 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.R_bSo8MaVsHfATZTvfGUQXcAHpEE784_0 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2511493743990630192', time=2024-03-09T16:50:59.035225+0000, chain.objs.size()=1 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.1UVwm_0wVTMRHTuiyIrNT9p-L12FHOW_0 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4355591089642848190', time=2024-03-09T16:53:04.024424+0000, chain.objs.size()=1 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17__shadow_.QSAtRBH6NkvaJwwQO4eT1I-5NLaB9cJ_0 2024-03-09T14:53:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18347929303541978136', time=2024-03-09T16:53:18.065533+0000, chain.objs.size()=1 2024-03-09T14:53:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.793+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18__shadow:BGjuyI9GaKMpoNqias0G83cBEQKSLa1_.uTCc6VWHfc94RzDGJbIs0pC4AEaSFP5_0 2024-03-09T14:53:42.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.801+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.805+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T14:53:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2024-03-09T14:53:42.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='_6zE_D3DnOOxg9xH_ZkKf4Ye-a6z6J31', time=2024-03-09T16:48:23.199277+0000, chain.objs.size()=1 2024-03-09T14:53:42.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.FMUX1JBrs0pSWsFvs13oOPxkmxO24KS_0 2024-03-09T14:53:42.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15213637999648483998', time=2024-03-09T16:48:23.605787+0000, chain.objs.size()=2 2024-03-09T14:53:42.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.13__shadow_.ZxKOD4EmqQaLYg9Sguk6rie-mmGxCts_0 2024-03-09T14:53:42.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.13__shadow_.ZxKOD4EmqQaLYg9Sguk6rie-mmGxCts_1 2024-03-09T14:53:42.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1348032474127964003', time=2024-03-09T16:48:23.748209+0000, chain.objs.size()=1 2024-03-09T14:53:42.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.7__shadow_.S27DIO7XxNaZS-YZ9bxyhqZZJNdEVGL_0 2024-03-09T14:53:42.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.674028238370271995', time=2024-03-09T16:51:47.956019+0000, chain.objs.size()=1 2024-03-09T14:53:42.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.GQ_W2Wtie3TXjXAv14KKrufkrPbiVeY_0 2024-03-09T14:53:42.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12692153984764838339', time=2024-03-09T16:53:04.047664+0000, chain.objs.size()=1 2024-03-09T14:53:42.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.809+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17__shadow_.tCwWtF2Qyn-Jp-Fem-zU9SnGKFOhE_j_0 2024-03-09T14:53:42.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.821+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:42.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.825+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T14:53:42.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=13, truncated=0, next_marker='' 2024-03-09T14:53:42.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13686339807024696114', time=2024-03-09T16:48:23.587638+0000, chain.objs.size()=2 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.13__shadow_.RrZhGDK-UQZ8M1do_JumiygN_CGx8V-_0 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.13__shadow_.RrZhGDK-UQZ8M1do_JumiygN_CGx8V-_1 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9456165913555396894', time=2024-03-09T16:50:20.460538+0000, chain.objs.size()=1 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.wDZOtAhadfsfXNxWrRxJ6Gd_9Q9SuK7_0 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.867736996933626723', time=2024-03-09T16:50:24.719941+0000, chain.objs.size()=1 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.Afg7dNvlzjyczNRZNOgDQ27V36uZ8DJ_0 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5111039363838091054', time=2024-03-09T16:50:24.884243+0000, chain.objs.size()=1 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.7vEdIXU-C119yYq_BR5fOXBWlff6IET_0 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.300948974026489252', time=2024-03-09T16:50:59.073092+0000, chain.objs.size()=1 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.w9jONtJY1roBAsbKfcVQ4gRey3ZvIIh_0 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9009438013796435686', time=2024-03-09T16:50:59.266579+0000, chain.objs.size()=1 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.829+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.NFApZRXhrw7y0NsPVrXPNANHKQVY40N_0 2024-03-09T14:53:42.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.833+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9373528876338640984', time=2024-03-09T16:51:03.759511+0000, chain.objs.size()=1 2024-03-09T14:53:42.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.833+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.mkZMi_ZKSba38Xee5LiZwyx4d6atYr9_0 2024-03-09T14:53:42.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.833+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12936154931885066097', time=2024-03-09T16:51:39.018038+0000, chain.objs.size()=1 2024-03-09T14:53:42.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.833+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.sXCZOaSUv3HRhT7_wsuflG4iu498ZKH_0 2024-03-09T14:53:42.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.833+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3014342855063891586', time=2024-03-09T16:51:44.036790+0000, chain.objs.size()=1 2024-03-09T14:53:42.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.833+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.CrxCa_nsP8ibXSSFZggunYMKxQ4Wp-J_0 2024-03-09T14:53:42.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.833+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3740162439172340417', time=2024-03-09T16:51:47.728559+0000, chain.objs.size()=1 2024-03-09T14:53:42.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.833+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.ZD39p0y4m3bEWclOU3AD8sSRNWC_Bbw_0 2024-03-09T14:53:42.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.833+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6823345631039348253', time=2024-03-09T16:52:22.879226+0000, chain.objs.size()=1 2024-03-09T14:53:42.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.833+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.HQIfr_U6zT6yKosSaX4Rz5aTg8J7A9o_0 2024-03-09T14:53:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.837+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2573581931344008221', time=2024-03-09T16:53:02.150047+0000, chain.objs.size()=1 2024-03-09T14:53:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.837+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.kBhYKY6ukRTy5cM68QQ10BvA2lfRT5O_0 2024-03-09T14:53:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.837+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18131022351723717256', time=2024-03-09T16:53:18.072512+0000, chain.objs.size()=1 2024-03-09T14:53:42.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.837+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18__shadow:tQtS2AteCYWGQ4hQUzC8IMablehZqcB_.WWWIhAUUld9pyD5dnYQDA6Idq5LhlK5_0 2024-03-09T14:53:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.841+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.849+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T14:53:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.849+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.849+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker='' 2024-03-09T14:53:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='_uTDLojna3U_-eEF3ss3mW3Np2WJxPHv', time=2024-03-09T16:48:23.420878+0000, chain.objs.size()=2 2024-03-09T14:53:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.8UCPaMPgVOnLdRqT4AuTvGiwrv9b3TG_0 2024-03-09T14:53:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.8UCPaMPgVOnLdRqT4AuTvGiwrv9b3TG_1 2024-03-09T14:53:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13340010106827674520', time=2024-03-09T16:48:23.738120+0000, chain.objs.size()=1 2024-03-09T14:53:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.7__shadow_.jla_Quf_BuLresvfBWPO24jxvMXfYr2_0 2024-03-09T14:53:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6180210155210084639', time=2024-03-09T16:50:59.033891+0000, chain.objs.size()=1 2024-03-09T14:53:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.2_2iCHKZoRSfgrPm8zGLMpG8R7GTS9J_0 2024-03-09T14:53:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7614736998278716624', time=2024-03-09T16:51:03.919964+0000, chain.objs.size()=1 2024-03-09T14:53:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.3uFZYGweiJYM4KGjW0b7QHNIuYwlB1c_0 2024-03-09T14:53:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12308035178659170265', time=2024-03-09T16:51:47.804458+0000, chain.objs.size()=1 2024-03-09T14:53:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.twDMfBfK1DiX2sIWpC8vGKI808KX_7C_0 2024-03-09T14:53:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13869751829200405459', time=2024-03-09T16:52:22.956890+0000, chain.objs.size()=1 2024-03-09T14:53:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.VT79QoUmLLSyu4vHuAQbsbb8G6_LkP5_0 2024-03-09T14:53:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5115107747611539866', time=2024-03-09T16:52:27.251229+0000, chain.objs.size()=1 2024-03-09T14:53:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.OlOC0MTdDu_GOsK6kMFLITz6n4Xi4P7_0 2024-03-09T14:53:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6105359280321235055', time=2024-03-09T16:53:02.206297+0000, chain.objs.size()=1 2024-03-09T14:53:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.853+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.MoLV_SB8oS60roi3WGYXSzocoHBh7hz_0 2024-03-09T14:53:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.861+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:42.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.865+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T14:53:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.865+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2024-03-09T14:53:42.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7664845607759348143', time=2024-03-09T16:48:23.086932+0000, chain.objs.size()=4 2024-03-09T14:53:42.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.10__multipart_foo.2~MSsVaeq9Oxxzh-JaYXXBj6-l5LjwqhK.1 2024-03-09T14:53:42.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.10__multipart_foo.2~MSsVaeq9Oxxzh-JaYXXBj6-l5LjwqhK.2 2024-03-09T14:53:42.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.10__multipart_foo.2~MSsVaeq9Oxxzh-JaYXXBj6-l5LjwqhK.3 2024-03-09T14:53:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.10__multipart_foo.2~MSsVaeq9Oxxzh-JaYXXBj6-l5LjwqhK.4 2024-03-09T14:53:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.K7A44frPkoX30Kh2NETJKndjf8Z7j22_1_492a66082cf69c835c3e6112cb23f544', time=2024-03-09T16:48:23.296466+0000, chain.objs.size()=1 2024-03-09T14:53:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.K7A44frPkoX30Kh2NETJKndjf8Z7j22_0 2024-03-09T14:53:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2692324105546605272', time=2024-03-09T16:51:03.763788+0000, chain.objs.size()=1 2024-03-09T14:53:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.qGQ5PbypsKkfXJWqF2-DClVf70HQZff_0 2024-03-09T14:53:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14852110118767089577', time=2024-03-09T16:51:39.155172+0000, chain.objs.size()=1 2024-03-09T14:53:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.mWe-4Ml2BGFpv2tr2yHd37Oi5gKYOJ7_0 2024-03-09T14:53:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13110611091186028239', time=2024-03-09T16:51:43.853008+0000, chain.objs.size()=1 2024-03-09T14:53:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.XmW-LpNncVPf3y3ZvZpG4ZN-edriNTB_0 2024-03-09T14:53:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4740385142906022091', time=2024-03-09T16:51:47.715752+0000, chain.objs.size()=1 2024-03-09T14:53:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.869+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.NU2YJIojW_3FrRCu6QsxBYGus59K-cl_0 2024-03-09T14:53:42.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.873+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11334039154924467842', time=2024-03-09T16:52:22.772264+0000, chain.objs.size()=1 2024-03-09T14:53:42.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.873+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.iVGDPCysa-0yUydzAOWzfb6ogs2LW6Y_0 2024-03-09T14:53:42.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.873+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6849596431763365181', time=2024-03-09T16:52:22.774051+0000, chain.objs.size()=1 2024-03-09T14:53:42.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.873+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.UwqEPhADmufBAYFXwcuSSEoXDP8xgFf_0 2024-03-09T14:53:42.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.873+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3882405479638723958', time=2024-03-09T16:52:27.198132+0000, chain.objs.size()=1 2024-03-09T14:53:42.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.873+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.v96P_NwwgWHN7MrryJrp-6QxEe2BvF3_0 2024-03-09T14:53:42.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.873+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11400314235863776682', time=2024-03-09T16:53:00.837581+0000, chain.objs.size()=1 2024-03-09T14:53:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.873+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.ugWuoqTltDinHF4S-zc2ErLJGsw4Tby_0 2024-03-09T14:53:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.873+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15132624771276843085', time=2024-03-09T16:53:02.178128+0000, chain.objs.size()=1 2024-03-09T14:53:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.873+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.84LSRGSFkqynoeOnJxDXR9al-DRx3j2_0 2024-03-09T14:53:42.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.877+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:42.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.881+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T14:53:42.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.885+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2024-03-09T14:53:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12179326023206004145', time=2024-03-09T16:50:20.459919+0000, chain.objs.size()=1 2024-03-09T14:53:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.M62EwjkBBc_J7F4BVAuf4Dk0qx-Qwdw_0 2024-03-09T14:53:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11780526315587613231', time=2024-03-09T16:50:24.903576+0000, chain.objs.size()=1 2024-03-09T14:53:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.6WLNBftI6yBCeqHfU4ozjdA05hZKUO8_0 2024-03-09T14:53:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14982588204703516687', time=2024-03-09T16:50:59.023277+0000, chain.objs.size()=1 2024-03-09T14:53:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.nE6Yw9IFWLgk8AwFvTsh-79uCOUkOJ3_0 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17848564670868961242', time=2024-03-09T16:51:38.963285+0000, chain.objs.size()=1 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.QRFzrL9j3F2Q31qhRfnx4W0bwtv_zIS_0 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11707117336928005467', time=2024-03-09T16:51:43.832888+0000, chain.objs.size()=1 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.GJVFMT6mEJz6MeAmgSQYXm_zxBcBC3p_0 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3588656443266447480', time=2024-03-09T16:51:43.834560+0000, chain.objs.size()=1 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.eSZqbSYA9YXtCRaf8RPyzImJkb2DEye_0 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2730821184372144039', time=2024-03-09T16:52:22.774456+0000, chain.objs.size()=1 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.BBn8E0NZoZwMdAx6Jkk8EuCli-C_Is5_0 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2854268768292599622', time=2024-03-09T16:52:27.334838+0000, chain.objs.size()=1 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.cu-2ij0shzDTJG3X86pbPP0fLMUkm2q_0 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16119165113963470564', time=2024-03-09T16:53:02.169115+0000, chain.objs.size()=1 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.to9uvlb4DN7z_ftjunpMzJMBQiTXjuv_0 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1097801273855638768', time=2024-03-09T16:53:04.077736+0000, chain.objs.size()=1 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17__shadow_.ijunMyRAD9sMTfMqeXj8GSdF4FtfnIs_0 2024-03-09T14:53:42.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8384608141069735306', time=2024-03-09T16:53:12.440748+0000, chain.objs.size()=1 2024-03-09T14:53:42.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.889+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18__shadow:4avFlxHBT2zrCiOIJk3KTGfShXvSUmE_.3WF3ApqBSDRgeiw92L3sm8qyEszbDZn_0 2024-03-09T14:53:42.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.897+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:42.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.905+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T14:53:42.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.905+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=9, truncated=0, next_marker='' 2024-03-09T14:53:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9024160014172828040', time=2024-03-09T16:48:23.136132+0000, chain.objs.size()=4 2024-03-09T14:53:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.9__multipart_foo.2~JkrYhrO3HjDn-TJHm0koE5L_sheBWIE.1 2024-03-09T14:53:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.9__multipart_foo.2~JkrYhrO3HjDn-TJHm0koE5L_sheBWIE.2 2024-03-09T14:53:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.9__multipart_foo.2~JkrYhrO3HjDn-TJHm0koE5L_sheBWIE.3 2024-03-09T14:53:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.9__multipart_foo.2~JkrYhrO3HjDn-TJHm0koE5L_sheBWIE.4 2024-03-09T14:53:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13707277977245231977', time=2024-03-09T16:48:23.594233+0000, chain.objs.size()=2 2024-03-09T14:53:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.13__shadow_._FUyh-tATS0YCp0K2D4uUZEqn73H_PD_0 2024-03-09T14:53:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.13__shadow_._FUyh-tATS0YCp0K2D4uUZEqn73H_PD_1 2024-03-09T14:53:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6102010532539638773', time=2024-03-09T16:50:20.420762+0000, chain.objs.size()=1 2024-03-09T14:53:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.TTecTgUXDRS2wsP_a3EQijNH2_NUq6h_0 2024-03-09T14:53:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14576020339401000593', time=2024-03-09T16:50:20.466664+0000, chain.objs.size()=1 2024-03-09T14:53:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.rVfyCe6n6JeCWwOaLEtuVHevE09-nqB_0 2024-03-09T14:53:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5185573957007706445', time=2024-03-09T16:50:59.033590+0000, chain.objs.size()=1 2024-03-09T14:53:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.CBII5NKirG1DDogn-vJDejHjTHmtkvo_0 2024-03-09T14:53:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10893079117110065964', time=2024-03-09T16:51:03.919271+0000, chain.objs.size()=1 2024-03-09T14:53:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.c3pftH5mmusp8wfIONnX6WcphZ9OhpB_0 2024-03-09T14:53:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12456113744514369453', time=2024-03-09T16:51:39.021311+0000, chain.objs.size()=1 2024-03-09T14:53:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.2Wm-cW9zUcq-h4Uj7FITfufiyjSpm1R_0 2024-03-09T14:53:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10139665679051691810', time=2024-03-09T16:51:43.851726+0000, chain.objs.size()=1 2024-03-09T14:53:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.909+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.jiDX4fTeSCjhwrlb7SV2itr7QdqjG2Q_0 2024-03-09T14:53:42.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.913+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7829278925027957780', time=2024-03-09T16:53:02.124484+0000, chain.objs.size()=1 2024-03-09T14:53:42.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.913+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.lGjBD46E9lYhPLCS6QcwvjlcD3XEkXx_0 2024-03-09T14:53:42.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.917+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:42.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.921+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T14:53:42.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.925+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.925+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.929+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2024-03-09T14:53:42.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.929+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7881267819332822892', time=2024-03-09T16:50:20.468187+0000, chain.objs.size()=1 2024-03-09T14:53:42.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.929+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.CjxiRDT_VJqoA-nmRG8MYnWusa2AT0k_0 2024-03-09T14:53:42.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.929+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11229555221932919630', time=2024-03-09T16:50:20.469804+0000, chain.objs.size()=1 2024-03-09T14:53:42.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.929+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.OgaDmGkAhEgTalfEnPDsf8S7-Rl2M3x_0 2024-03-09T14:53:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.929+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:42.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.937+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T14:53:42.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.937+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.937+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=10, truncated=0, next_marker='' 2024-03-09T14:53:42.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14667829419773088597', time=2024-03-09T16:48:22.887600+0000, chain.objs.size()=1 2024-03-09T14:53:42.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.cd606BUFCfHHbFBKTfYGBhZGXx0OSuq_0 2024-03-09T14:53:42.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.2_49163a9c25627249aa24353e2c2b13b6', time=2024-03-09T16:48:23.351886+0000, chain.objs.size()=3 2024-03-09T14:53:42.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.1 2024-03-09T14:53:42.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.2 2024-03-09T14:53:42.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.3 2024-03-09T14:53:42.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2317020279843432629', time=2024-03-09T16:48:23.385998+0000, chain.objs.size()=3 2024-03-09T14:53:42.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.1 2024-03-09T14:53:42.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.2 2024-03-09T14:53:42.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.3 2024-03-09T14:53:42.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6477235068231068944', time=2024-03-09T16:48:23.512975+0000, chain.objs.size()=3 2024-03-09T14:53:42.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-FROZEN.2~Ab5Ciq80sRaRAuyeLQrsIvc_YNUaFe2.1 2024-03-09T14:53:42.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-FROZEN.2~Ab5Ciq80sRaRAuyeLQrsIvc_YNUaFe2.2 2024-03-09T14:53:42.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-FROZEN.2~Ab5Ciq80sRaRAuyeLQrsIvc_YNUaFe2.3 2024-03-09T14:53:42.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7065859932315877771', time=2024-03-09T16:50:20.482875+0000, chain.objs.size()=1 2024-03-09T14:53:42.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.3sRq-AD5_cFdtFoAS7AP94gNd_uQzL0_0 2024-03-09T14:53:42.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6166162740561024975', time=2024-03-09T16:51:03.970184+0000, chain.objs.size()=1 2024-03-09T14:53:42.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.941+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.uz8OB_3zsTS5v1stOqDoqfp8l8tNPeK_0 2024-03-09T14:53:42.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.945+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.797330750045108810', time=2024-03-09T16:51:39.154467+0000, chain.objs.size()=1 2024-03-09T14:53:42.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.945+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.Vw3-1cy0b6prjDNfWR4hDQth6g-jhX9_0 2024-03-09T14:53:42.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.945+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16787140044628953664', time=2024-03-09T16:51:43.929134+0000, chain.objs.size()=1 2024-03-09T14:53:42.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.945+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.BczAcSRPkhJDGOt04U9dYGzAhEpH1I3_0 2024-03-09T14:53:42.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.945+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6772429709565869573', time=2024-03-09T16:51:47.955920+0000, chain.objs.size()=1 2024-03-09T14:53:42.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.945+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.axn4CAkwnXBlHMur73EYggArrs-Y8TH_0 2024-03-09T14:53:42.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.945+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3867923763088017774', time=2024-03-09T16:52:22.792390+0000, chain.objs.size()=1 2024-03-09T14:53:42.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.945+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.2IWSCqMX-iz8jwiU0Xvo2eg2qQSOlAd_0 2024-03-09T14:53:42.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.949+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:42.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.953+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T14:53:42.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.957+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=12, truncated=0, next_marker='' 2024-03-09T14:53:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7494590672314839681', time=2024-03-09T16:48:23.831979+0000, chain.objs.size()=1 2024-03-09T14:53:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.8__shadow_.MT1sPPgci5eFBZW8SgfgB6jRilmn49r_0 2024-03-09T14:53:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5050093215078202002', time=2024-03-09T16:50:20.417928+0000, chain.objs.size()=1 2024-03-09T14:53:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.ibh6xXNom9O6d_6i3oe33G4Zb3cDaEk_0 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6679872064223326676', time=2024-03-09T16:50:24.806231+0000, chain.objs.size()=1 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.SAnamgaLPKVlwfNJrBNewRvLfYlNXpq_0 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3516280865447395523', time=2024-03-09T16:50:24.808059+0000, chain.objs.size()=1 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.uc7DhcYIYLcbVBZ4uqzJ744YQI82nxF_0 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16181953676377121360', time=2024-03-09T16:50:59.167735+0000, chain.objs.size()=1 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.Sjfo0mFLDiJOmgdslm1-2-XO-OhttGX_0 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10507475696696128188', time=2024-03-09T16:50:59.171754+0000, chain.objs.size()=1 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.1u44w3h2FRtaCEGJaWQGc_e1T0hV9_G_0 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.84748274977241000', time=2024-03-09T16:51:38.948878+0000, chain.objs.size()=1 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.BUZnkFXpP3o2vCzZ6dQ8oMOlwbLhXMY_0 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14064533395237628544', time=2024-03-09T16:51:38.978417+0000, chain.objs.size()=1 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.kytGuqgh8aTdOp6ElOFRd-OnjBboyYC_0 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14285120984590029109', time=2024-03-09T16:51:44.046883+0000, chain.objs.size()=1 2024-03-09T14:53:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.26eoe2Y8c6MiPj15ef6pWrTOPvjA70O_0 2024-03-09T14:53:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7654221613777636477', time=2024-03-09T16:52:22.865027+0000, chain.objs.size()=1 2024-03-09T14:53:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.SjnpD4tMmvamoXTPvMTe1QcuTMSADHk_0 2024-03-09T14:53:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14943146464051140979', time=2024-03-09T16:52:27.123068+0000, chain.objs.size()=1 2024-03-09T14:53:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.LUMz3WgRA3V2WesF7mvLVeWq90NkQg0_0 2024-03-09T14:53:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13631846220663815905', time=2024-03-09T16:53:02.344910+0000, chain.objs.size()=1 2024-03-09T14:53:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.961+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.jvNdvmuoekutAl9q4czgpW_dg3gkvmr_0 2024-03-09T14:53:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.969+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:42.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.973+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T14:53:42.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.977+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2024-03-09T14:53:42.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='_wOUf_3YSVi9eBYX04y7Qtu7Lvc6pOF7', time=2024-03-09T16:48:23.454686+0000, chain.objs.size()=1 2024-03-09T14:53:42.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.K19KTQwDT_7HP7rCPnwozXs0sIJ_JGe_0 2024-03-09T14:53:42.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6128150659680309893', time=2024-03-09T16:50:24.692945+0000, chain.objs.size()=1 2024-03-09T14:53:42.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.RyCdeJwIMueCIb4o8nxzLBcUsB-nMHI_0 2024-03-09T14:53:42.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12008423549802368528', time=2024-03-09T16:51:03.818240+0000, chain.objs.size()=1 2024-03-09T14:53:42.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.EZ6RL_S1_AgvfhJLyNCJTXcECq6cNk__0 2024-03-09T14:53:42.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16251574520072396812', time=2024-03-09T16:51:39.023812+0000, chain.objs.size()=1 2024-03-09T14:53:42.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.Ux82UkQbiVbOP1YZmB6vt2okRxRnAK8_0 2024-03-09T14:53:42.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.160436763090329575', time=2024-03-09T16:51:43.852147+0000, chain.objs.size()=1 2024-03-09T14:53:42.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.U71WSf5P7mWFnTcuYpSr1Pb9OseJiG6_0 2024-03-09T14:53:42.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2216360508283799451', time=2024-03-09T16:52:22.858774+0000, chain.objs.size()=1 2024-03-09T14:53:42.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.981+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.-k9luYWCF8YdhbKWQSO-6PxZ9zHXBz__0 2024-03-09T14:53:42.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.985+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:42.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.989+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T14:53:42.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.993+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:42.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.993+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:42.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker='' 2024-03-09T14:53:42.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17513661504900508715', time=2024-03-09T16:48:22.910776+0000, chain.objs.size()=1 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.ewxgXrsmdbuUM7dEAoL48fSKuywN4v6_0 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5914922296680739730', time=2024-03-09T16:48:23.808768+0000, chain.objs.size()=1 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.8__shadow_.Kbi0c8qCk-Xqmtvgp8RB1wsa-6e1A6U_0 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3576867788921491741', time=2024-03-09T16:50:59.079508+0000, chain.objs.size()=1 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.lhovyMfW4t0yWetFLH74RDV6ZRyCscn_0 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10084700248186262541', time=2024-03-09T16:51:03.917589+0000, chain.objs.size()=1 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.CCKK-xoNDW91SlSQMaQwGM_yiQzazkt_0 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2573037673124167195', time=2024-03-09T16:51:47.816182+0000, chain.objs.size()=1 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.I8RUewTJ90-OKBhobXwzUB1ut7HNrNu_0 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17943194793736313032', time=2024-03-09T16:52:26.376567+0000, chain.objs.size()=1 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.ynYdGuBddGGpVh37Vkx9ybdw4lNnCnU_0 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14173478386755474330', time=2024-03-09T16:53:04.024181+0000, chain.objs.size()=1 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17__shadow_.gZu-QUa5gnd0TApjCJakNjiifJL9o-Z_0 2024-03-09T14:53:42.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16555836047907847484', time=2024-03-09T16:53:04.035601+0000, chain.objs.size()=1 2024-03-09T14:53:42.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:42.997+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17__shadow_.P-Iu7uZElUXA3EX1ejUwky289bGAUvA_0 2024-03-09T14:53:43.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.001+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.005+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T14:53:43.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.009+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.013+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.013+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2024-03-09T14:53:43.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.013+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13215593658750293492', time=2024-03-09T16:50:24.690549+0000, chain.objs.size()=1 2024-03-09T14:53:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.013+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.cLgkRbtzh-nbUmfvTgFRr4Zf4VaRlkP_0 2024-03-09T14:53:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.013+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7524901039175462209', time=2024-03-09T16:50:24.895782+0000, chain.objs.size()=1 2024-03-09T14:53:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.013+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.Uf0YfiKO1tQFk_CM2ng7HwKVpg17tjV_0 2024-03-09T14:53:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.013+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12111984603064191384', time=2024-03-09T16:52:22.792585+0000, chain.objs.size()=1 2024-03-09T14:53:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.013+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.QOeV5RrswlOmiQmwhJn8CR_vBza9tGq_0 2024-03-09T14:53:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.013+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12705831967058452060', time=2024-03-09T16:52:27.162553+0000, chain.objs.size()=1 2024-03-09T14:53:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.013+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.PlHR28WMZFXteVFykyRAIN_SFVGTj64_0 2024-03-09T14:53:43.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.017+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.021+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T14:53:43.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.025+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker='' 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12030105678604887599', time=2024-03-09T16:48:22.919710+0000, chain.objs.size()=1 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.ucvhn6aR6ALNwPvWkCpM8s3tr-FpxGF_0 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.726303276373013898', time=2024-03-09T16:50:24.690532+0000, chain.objs.size()=1 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.pV1kKhdR-QT6wuRzhnfTvER5Tj97E1S_0 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16237466326865516342', time=2024-03-09T16:51:03.834556+0000, chain.objs.size()=1 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.9kKC1DOApkuip16P7w-hXaNBlWLSW72_0 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18354965234439094847', time=2024-03-09T16:51:03.868701+0000, chain.objs.size()=1 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.HegzZDKEvxTr1fTpI3awUFAgOc_PNA-_0 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18124148426113291218', time=2024-03-09T16:51:07.573955+0000, chain.objs.size()=1 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.7ypOvP_-esS550LcTshNKt3T0WLs7rc_0 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7127704206567770448', time=2024-03-09T16:51:38.948747+0000, chain.objs.size()=1 2024-03-09T14:53:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.zHslgjSYpy6syUvYNKzPi9Bhj82-CH7_0 2024-03-09T14:53:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15251208971051116550', time=2024-03-09T16:52:27.076023+0000, chain.objs.size()=1 2024-03-09T14:53:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.8THgWP21wyhHkBogjyVwyV-RV8L7jP3_0 2024-03-09T14:53:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12999505658377318227', time=2024-03-09T16:53:02.105185+0000, chain.objs.size()=1 2024-03-09T14:53:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.029+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.Tz09azJjCSOVfyJP13Gt8rGLIz3h-gq_0 2024-03-09T14:53:43.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.033+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.041+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T14:53:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=13, truncated=0, next_marker='' 2024-03-09T14:53:43.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2270428550249880474', time=2024-03-09T16:48:22.900736+0000, chain.objs.size()=1 2024-03-09T14:53:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.2u5tvqMBdf_LqBM7do0Az_QQ__vd4rE_0 2024-03-09T14:53:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1986235792109021368', time=2024-03-09T16:48:22.937832+0000, chain.objs.size()=1 2024-03-09T14:53:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.0566Emuqp471bh3hg3aCRaQD7K6bALR_0 2024-03-09T14:53:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='_Vg3UWQoIHbBn5LughNpupS2b_VFdF8O', time=2024-03-09T16:48:23.401593+0000, chain.objs.size()=2 2024-03-09T14:53:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.TgcdGeKwhJZuhCE33iNomtBXECVBLz6_0 2024-03-09T14:53:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.TgcdGeKwhJZuhCE33iNomtBXECVBLz6_1 2024-03-09T14:53:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5579231311581051923', time=2024-03-09T16:50:24.690823+0000, chain.objs.size()=1 2024-03-09T14:53:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.hRcRT-bQ0875xWRx6cIJXISL8jsaY8p_0 2024-03-09T14:53:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10257422577898884784', time=2024-03-09T16:51:03.834555+0000, chain.objs.size()=1 2024-03-09T14:53:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.fXkl6Of2QBhSBW_SDlzqAP9QFZcbMGH_0 2024-03-09T14:53:43.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2803474863499876302', time=2024-03-09T16:51:03.993305+0000, chain.objs.size()=1 2024-03-09T14:53:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.GCEYuAP84WpLWRxR1A5W3yMqV1Ch-zA_0 2024-03-09T14:53:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12251111064170452926', time=2024-03-09T16:51:39.160010+0000, chain.objs.size()=1 2024-03-09T14:53:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.m-ZwfZxRKc2xWPDabLkG62KxI4H8Sro_0 2024-03-09T14:53:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9675350959624356864', time=2024-03-09T16:51:43.847117+0000, chain.objs.size()=1 2024-03-09T14:53:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_._50XMTXPIRK1KhTszMS8p3v01MS_83R_0 2024-03-09T14:53:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.045+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11798826290091489152', time=2024-03-09T16:51:43.859914+0000, chain.objs.size()=1 2024-03-09T14:53:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.049+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.g7yBHIG7X1MtEAlzUvRpWGwRjsPLfr3_0 2024-03-09T14:53:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.049+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3330605805146696684', time=2024-03-09T16:52:22.865031+0000, chain.objs.size()=1 2024-03-09T14:53:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.049+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.p7NuVWt0x8zGDGIbA2Q352K0ijvhhi5_0 2024-03-09T14:53:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.049+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7683127906186978615', time=2024-03-09T16:52:27.041375+0000, chain.objs.size()=1 2024-03-09T14:53:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.049+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.bhevHX_U5GvgMvY-UU3cKWy-mO0zfEZ_0 2024-03-09T14:53:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.053+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8892002097546968522', time=2024-03-09T16:52:27.123227+0000, chain.objs.size()=1 2024-03-09T14:53:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.053+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.abKn0cAdKupL4tac_wtgVhurgUZtg3k_0 2024-03-09T14:53:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.053+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17436627442395996729', time=2024-03-09T16:52:27.125070+0000, chain.objs.size()=1 2024-03-09T14:53:43.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.053+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.fB7IEtPIs7TN5QGB8wjivn4kwMvN1Ao_0 2024-03-09T14:53:43.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.057+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.065+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T14:53:43.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.069+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.069+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=12, truncated=0, next_marker='' 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16367796950388320495', time=2024-03-09T16:48:23.040434+0000, chain.objs.size()=4 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.12__multipart_foo.2~YojN6LMtFT97TQR8YUGgGk0T-dv6dC2.1 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.12__multipart_foo.2~YojN6LMtFT97TQR8YUGgGk0T-dv6dC2.2 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.12__multipart_foo.2~YojN6LMtFT97TQR8YUGgGk0T-dv6dC2.3 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.12__multipart_foo.2~YojN6LMtFT97TQR8YUGgGk0T-dv6dC2.4 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11763471558483991125', time=2024-03-09T16:48:23.269113+0000, chain.objs.size()=1 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.K7A44frPkoX30Kh2NETJKndjf8Z7j22_0 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='_JE_5-O7EiDLbAYq56EPirG7OZCTG0Pk', time=2024-03-09T16:48:23.289399+0000, chain.objs.size()=1 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.zQ7D3ZyR40Q77MUVa_M_FDgOaqpc006_0 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9921280845339081383', time=2024-03-09T16:48:23.540332+0000, chain.objs.size()=3 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-STANDARD.2~LR-QRm1F_-76gwLwW4Kv2FQGX0mscth.1 2024-03-09T14:53:43.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-STANDARD.2~LR-QRm1F_-76gwLwW4Kv2FQGX0mscth.2 2024-03-09T14:53:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-STANDARD.2~LR-QRm1F_-76gwLwW4Kv2FQGX0mscth.3 2024-03-09T14:53:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13332728813266300294', time=2024-03-09T16:50:24.690307+0000, chain.objs.size()=1 2024-03-09T14:53:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.kqwu9Env7EHcXXupqQ7HUvgRAOr5X11_0 2024-03-09T14:53:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13808754280017157045', time=2024-03-09T16:50:24.692773+0000, chain.objs.size()=1 2024-03-09T14:53:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.mBsIU8QCveEtYF4CZNy20IiNFokId3w_0 2024-03-09T14:53:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1314369219104888615', time=2024-03-09T16:50:59.252874+0000, chain.objs.size()=1 2024-03-09T14:53:43.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.073+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.vYqoiQvxI3FMtcfGwbBOZCbwXmo8PL0_0 2024-03-09T14:53:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.077+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7569294595895174122', time=2024-03-09T16:51:03.993541+0000, chain.objs.size()=1 2024-03-09T14:53:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.077+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.INXaVUigJ61clFJ9kFa71aS7PkhpWA5_0 2024-03-09T14:53:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.077+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17291021315287509031', time=2024-03-09T16:51:39.011375+0000, chain.objs.size()=1 2024-03-09T14:53:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.077+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.1efD2T-vaIjvQ40r8VdWzNKGLGrT5Oo_0 2024-03-09T14:53:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.077+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17990968851179336091', time=2024-03-09T16:51:43.946408+0000, chain.objs.size()=1 2024-03-09T14:53:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.077+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.0djQVVM9hLaN6nCTOZj-JMdkkXLbozh_0 2024-03-09T14:53:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.077+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16410151771110357857', time=2024-03-09T16:52:22.976947+0000, chain.objs.size()=1 2024-03-09T14:53:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.077+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.RzN84LxDzemH9rUCbY1LXywVUo39G5g_0 2024-03-09T14:53:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.077+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.82753829028909121', time=2024-03-09T16:52:27.166316+0000, chain.objs.size()=1 2024-03-09T14:53:43.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.077+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.Zm258ZqAXysvxtiR2zyK4Jk9jnOy02P_0 2024-03-09T14:53:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.081+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.089+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T14:53:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.089+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2024-03-09T14:53:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2726321958320278490', time=2024-03-09T16:50:27.851013+0000, chain.objs.size()=1 2024-03-09T14:53:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.buF-reXsDDQJbz5g7fKpqHfVZvBU6y9_0 2024-03-09T14:53:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1212475719272425172', time=2024-03-09T16:50:59.066455+0000, chain.objs.size()=1 2024-03-09T14:53:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.r2JuqovLoF3ZanDKUqNnLHLXXmX-vmg_0 2024-03-09T14:53:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17015689066706249409', time=2024-03-09T16:51:39.034672+0000, chain.objs.size()=1 2024-03-09T14:53:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.H6yuR_osxTnxRjpYFYF9JXOwy4WtrNH_0 2024-03-09T14:53:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7122008807731013504', time=2024-03-09T16:51:47.713668+0000, chain.objs.size()=1 2024-03-09T14:53:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.v5YSJCwHFR9xxjF5BjbXKbSK4OkCKO3_0 2024-03-09T14:53:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17728976458797864887', time=2024-03-09T16:51:47.928501+0000, chain.objs.size()=1 2024-03-09T14:53:43.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.mUL3GWolXfAMXSRbYhLUC7DBxH7BJLX_0 2024-03-09T14:53:43.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12387406335841827063', time=2024-03-09T16:51:47.966783+0000, chain.objs.size()=1 2024-03-09T14:53:43.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.yvnor1Zj9k1MUSe6V9jibh_O5UB4Qif_0 2024-03-09T14:53:43.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.888440264977401', time=2024-03-09T16:52:27.229146+0000, chain.objs.size()=1 2024-03-09T14:53:43.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.093+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.4sU3jKWXW9CMLAXQDSuf0scSKaFuF8k_0 2024-03-09T14:53:43.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.097+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.105+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T14:53:43.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2024-03-09T14:53:43.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='_lKU5NSfyeT97Uul9hdQgAJuOZM-VvN2', time=2024-03-09T16:48:23.279378+0000, chain.objs.size()=1 2024-03-09T14:53:43.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.hQfaoR55zzlk6TfGoc_SdciF006tP75_0 2024-03-09T14:53:43.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9258072135161784416', time=2024-03-09T16:50:24.689085+0000, chain.objs.size()=1 2024-03-09T14:53:43.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.W3nf1oZZC6_IWhwZ4DdmeajjR7s4C-4_0 2024-03-09T14:53:43.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15082852352992162328', time=2024-03-09T16:51:43.843734+0000, chain.objs.size()=1 2024-03-09T14:53:43.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.LZiQRHW14Q5BctR-L3RgoLCTjsXf_Y0_0 2024-03-09T14:53:43.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14217459238985612350', time=2024-03-09T16:53:02.193950+0000, chain.objs.size()=1 2024-03-09T14:53:43.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.I1pX1TZo6utTsBPA8ON83eZmClIa9Yq_0 2024-03-09T14:53:43.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2220003452889971274', time=2024-03-09T16:53:02.272641+0000, chain.objs.size()=1 2024-03-09T14:53:43.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.109+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.047Dby7CNRrnKGsG8IOwBGazc4rbGE7_0 2024-03-09T14:53:43.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.113+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.121+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T14:53:43.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=12, truncated=0, next_marker='' 2024-03-09T14:53:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2996683048297604930', time=2024-03-09T16:48:22.954390+0000, chain.objs.size()=1 2024-03-09T14:53:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.QBQcztc3-0WgNyDnoVJLPQAotXq7zZa_0 2024-03-09T14:53:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10413128474800969159', time=2024-03-09T16:50:20.420874+0000, chain.objs.size()=1 2024-03-09T14:53:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.Q_NIjc01VhfRwnF3cRWXfMxAPxanzT2_0 2024-03-09T14:53:43.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8559713657262725100', time=2024-03-09T16:50:20.611662+0000, chain.objs.size()=1 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.xIO0FU4rli6JT91I_sOSZSPPQW9s_xx_0 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5488068231557749920', time=2024-03-09T16:50:24.693643+0000, chain.objs.size()=1 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.OXi6uAHEBzET6Q77_Sfbgkiw8sTYy_o_0 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2286774280013904676', time=2024-03-09T16:51:38.960973+0000, chain.objs.size()=1 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.JNCzu1NEQtuzj30obFUC2WZxgttF0nd_0 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1823724446943455766', time=2024-03-09T16:51:43.851912+0000, chain.objs.size()=1 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.iVzEWHpwZA-Y9JJMKrE-OZVr3DPnIsx_0 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14108900114644196122', time=2024-03-09T16:51:47.802289+0000, chain.objs.size()=1 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.d0z-aiGgwE3BQd_edGvYUNyQHa4Jn7i_0 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3886615166794479717', time=2024-03-09T16:52:27.249050+0000, chain.objs.size()=1 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.Tr39FXwEIzsUCqQ-XpbqKjonCcOK-8j_0 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14507146462935826196', time=2024-03-09T16:53:02.208296+0000, chain.objs.size()=1 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.ExYiA9rw8eh3ZuXu8WfD6WK_6bzIppO_0 2024-03-09T14:53:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10122478882882700193', time=2024-03-09T16:53:02.325746+0000, chain.objs.size()=1 2024-03-09T14:53:43.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.kN7FqQMOTf0USHXTLd5hS1qXzJzu5qN_0 2024-03-09T14:53:43.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15218880769814728520', time=2024-03-09T16:53:04.023366+0000, chain.objs.size()=1 2024-03-09T14:53:43.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17__shadow_.kCq9oNWnGT_YnQcN2txTVUZgUgnfHZO_0 2024-03-09T14:53:43.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15418577807621638044', time=2024-03-09T16:53:18.053667+0000, chain.objs.size()=1 2024-03-09T14:53:43.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.125+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.18__shadow:siboStXayI4Ad3c-zSk9b9Sf8rHx.DR_.omeERkLkw1CVbC4btK-CGV4hJ6kbC3Y_0 2024-03-09T14:53:43.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.137+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.141+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T14:53:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2024-03-09T14:53:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15211718419860944605', time=2024-03-09T16:48:23.220511+0000, chain.objs.size()=1 2024-03-09T14:53:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.JeUygVC1yQfDqC_IPNqgVkOubTC1qT-_0 2024-03-09T14:53:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.2_642486375c9191b85f4f634810046761', time=2024-03-09T16:48:23.465517+0000, chain.objs.size()=3 2024-03-09T14:53:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.1 2024-03-09T14:53:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.2 2024-03-09T14:53:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.3 2024-03-09T14:53:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9142723243605177960', time=2024-03-09T16:50:59.264074+0000, chain.objs.size()=1 2024-03-09T14:53:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.U-UK6vsRtF9uU4hTbnqWflh2VLwxh8D_0 2024-03-09T14:53:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5930262203805025551', time=2024-03-09T16:51:47.727507+0000, chain.objs.size()=1 2024-03-09T14:53:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.Lzc95tYQIGCrX7QPkf0EY95T2GahSaw_0 2024-03-09T14:53:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14301309790053434436', time=2024-03-09T16:51:47.803116+0000, chain.objs.size()=1 2024-03-09T14:53:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.J_ny7TfUV2KiAUYeBYCuw8ot_ZkU4KB_0 2024-03-09T14:53:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2016624907811255406', time=2024-03-09T16:52:27.317694+0000, chain.objs.size()=1 2024-03-09T14:53:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.145+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.PxzfUuFCQCeJp-Oy1N9CxXUIu1wARyd_0 2024-03-09T14:53:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.149+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.157+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T14:53:43.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.161+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.161+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.161+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2024-03-09T14:53:43.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.161+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='_XgO7GyB48tqkeyPWpuw2F4lss2fWUMv', time=2024-03-09T16:48:23.368011+0000, chain.objs.size()=2 2024-03-09T14:53:43.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.161+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.HhU-pMKO4BONFlReICRhhHT3RHFhB4s_0 2024-03-09T14:53:43.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.161+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.HhU-pMKO4BONFlReICRhhHT3RHFhB4s_1 2024-03-09T14:53:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.161+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11339083919106427599', time=2024-03-09T16:48:23.528798+0000, chain.objs.size()=3 2024-03-09T14:53:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.161+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-LUKEWARM.2~BjQJOEOSmNwAKFK8SlNi5IzXPjPVv6d.1 2024-03-09T14:53:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.161+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-LUKEWARM.2~BjQJOEOSmNwAKFK8SlNi5IzXPjPVv6d.2 2024-03-09T14:53:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.165+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.14__multipart_foo-LUKEWARM.2~BjQJOEOSmNwAKFK8SlNi5IzXPjPVv6d.3 2024-03-09T14:53:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.165+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5075647205592698143', time=2024-03-09T16:50:20.399180+0000, chain.objs.size()=1 2024-03-09T14:53:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.165+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.dD669uhWixdzQuUpBRt-sLsF5sOeMgY_0 2024-03-09T14:53:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.165+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11189899480712885213', time=2024-03-09T16:50:20.430416+0000, chain.objs.size()=1 2024-03-09T14:53:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.165+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.qBdWb81uHMLCoYJNOA5YcNfVoIEAbQy_0 2024-03-09T14:53:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.165+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10010305285496101210', time=2024-03-09T16:50:24.814323+0000, chain.objs.size()=1 2024-03-09T14:53:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.165+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.gBb8F106G5PRxHgTfy9xIJAr0OfdpnH_0 2024-03-09T14:53:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.165+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18320785292954477694', time=2024-03-09T16:51:02.679710+0000, chain.objs.size()=1 2024-03-09T14:53:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.165+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.OXPwgCKZhz8UyNN-r8DtbGqn9aJxanN_0 2024-03-09T14:53:43.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.173+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.177+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T14:53:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.181+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.185+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.185+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2024-03-09T14:53:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.185+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14106023454034700089', time=2024-03-09T16:48:22.947879+0000, chain.objs.size()=1 2024-03-09T14:53:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.185+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.FuRQbdMmReskxAGawS4Bsu2hFfu8rns_0 2024-03-09T14:53:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.185+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='_ncer9XPVEkGlCSGLfui_KFDOQmvzhpr', time=2024-03-09T16:48:23.447321+0000, chain.objs.size()=1 2024-03-09T14:53:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.185+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.uZ8j_bwWmyYzcQhApZhEFEZJ-scwUQv_0 2024-03-09T14:53:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.185+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15129702534265249839', time=2024-03-09T16:50:59.165682+0000, chain.objs.size()=1 2024-03-09T14:53:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.185+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.cabXGqc5ItT4v4_jad2bpYdRADTIs8F_0 2024-03-09T14:53:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.185+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7799009992041092240', time=2024-03-09T16:52:22.778366+0000, chain.objs.size()=1 2024-03-09T14:53:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.185+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.IzPfxbiy0balBBmiCZciQQ-AFh43_vf_0 2024-03-09T14:53:43.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.185+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11294048697841310680', time=2024-03-09T16:53:02.272755+0000, chain.objs.size()=1 2024-03-09T14:53:43.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.185+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.ckWMHS3SCvhTiAU_D_KmB7BX1ieB6bz_0 2024-03-09T14:53:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.189+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.197+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T14:53:43.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker='' 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3728002694601589019', time=2024-03-09T16:48:22.924739+0000, chain.objs.size()=1 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.QWVRSSM-4q-ufjt8IUHezXzGBNgxdwA_0 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16829480728355508927', time=2024-03-09T16:51:03.759125+0000, chain.objs.size()=1 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.9132Y9_zOhrchnPBtbSzwhsA3hzRodl_0 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16638500083500298510', time=2024-03-09T16:51:03.919993+0000, chain.objs.size()=1 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.iUMy0lA95WR8oK4wby4ihKQEHLetXw3_0 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12714432740733245294', time=2024-03-09T16:51:38.960893+0000, chain.objs.size()=1 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.1ncNooPSkRpDm8aAU1jj810nAMCJuQX_0 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6437278588900095954', time=2024-03-09T16:51:43.772481+0000, chain.objs.size()=1 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.Suc8yOcP7HrSS8v6_cHhxFRlfh872C6_0 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4237550329890759964', time=2024-03-09T16:51:43.851826+0000, chain.objs.size()=1 2024-03-09T14:53:43.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.ObkhsipAyjzz-8dTitejabsbQB2DWfw_0 2024-03-09T14:53:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5406736835961493886', time=2024-03-09T16:52:22.775283+0000, chain.objs.size()=1 2024-03-09T14:53:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.syDF44XhejbJL2m8RfqeiYVuc474zVs_0 2024-03-09T14:53:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5513686028605722403', time=2024-03-09T16:52:27.249299+0000, chain.objs.size()=1 2024-03-09T14:53:43.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.201+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.Y-4u9T-81SEQXnw-yrCfD42pSWfr7fN_0 2024-03-09T14:53:43.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.209+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.213+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T14:53:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=12, truncated=0, next_marker='' 2024-03-09T14:53:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1373402221988595462', time=2024-03-09T16:50:20.460473+0000, chain.objs.size()=1 2024-03-09T14:53:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.6kvNqw3_KkuhTGn7OmyHzBq9UY8LR3o_0 2024-03-09T14:53:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2502101891278914024', time=2024-03-09T16:50:24.716294+0000, chain.objs.size()=1 2024-03-09T14:53:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.JFBZXm5rHssM2Z_oci1cV0AmyVOpkIR_0 2024-03-09T14:53:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17991354613128243452', time=2024-03-09T16:50:24.807077+0000, chain.objs.size()=1 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.RpWsQFcdHJba3JmZAD1bTEHwg7CYWWm_0 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2080756190588227576', time=2024-03-09T16:50:59.023210+0000, chain.objs.size()=1 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.f-LWlAtxpsTBl-UX0yw1bXBtji1tJOJ_0 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14358800928414295054', time=2024-03-09T16:51:03.927447+0000, chain.objs.size()=1 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.3osn2Oi8zMvwf-MF7mF4Tb3JdhrXZA1_0 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8376481735212523407', time=2024-03-09T16:51:38.963154+0000, chain.objs.size()=1 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.Xck2QcN9fLdBXJ26fzRA6l4UfH66Aev_0 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11326693069126909347', time=2024-03-09T16:51:42.831126+0000, chain.objs.size()=1 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.7_4MvTlXhhfaPgmYnto-0NdbxHwWrwu_0 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1176882174877051280', time=2024-03-09T16:51:43.948591+0000, chain.objs.size()=1 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.a1fvLPOQ2cCHy748r0TF2NroSDjyGYC_0 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3037300272715507831', time=2024-03-09T16:51:44.032260+0000, chain.objs.size()=1 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.217+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.1OFJkzF6KO0qsXepTna51YZvmCm-sEY_0 2024-03-09T14:53:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.221+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12656068050728064730', time=2024-03-09T16:51:47.728769+0000, chain.objs.size()=1 2024-03-09T14:53:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.221+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.KelPlxSegtBRsQaXTmy0OvMbMbKFjx2_0 2024-03-09T14:53:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.221+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11190269265258422254', time=2024-03-09T16:51:47.742678+0000, chain.objs.size()=1 2024-03-09T14:53:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.221+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.y2EhFcLl_hEmhnyuF_NBwxzyukwOReu_0 2024-03-09T14:53:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.221+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17509872974796109857', time=2024-03-09T16:52:21.953897+0000, chain.objs.size()=1 2024-03-09T14:53:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.221+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.Ona1p7R-G6f2I2w5_r1Xig6Jzmn9z25_0 2024-03-09T14:53:43.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.229+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.233+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T14:53:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.237+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.237+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=10, truncated=0, next_marker='' 2024-03-09T14:53:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2694798983853814564', time=2024-03-09T16:48:22.877359+0000, chain.objs.size()=1 2024-03-09T14:53:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.4__shadow_.GagB3SqRbMBKp1DUcA8oirsIulWiX6m_0 2024-03-09T14:53:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4002354929203554414', time=2024-03-09T16:48:22.995263+0000, chain.objs.size()=4 2024-03-09T14:53:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.11__multipart_foo.2~6QP15A_WX6Wb-S4ld-DKSzM7SOVvjTQ.1 2024-03-09T14:53:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.11__multipart_foo.2~6QP15A_WX6Wb-S4ld-DKSzM7SOVvjTQ.2 2024-03-09T14:53:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.11__multipart_foo.2~6QP15A_WX6Wb-S4ld-DKSzM7SOVvjTQ.3 2024-03-09T14:53:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.11__multipart_foo.2~6QP15A_WX6Wb-S4ld-DKSzM7SOVvjTQ.4 2024-03-09T14:53:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='_w4wX0ZjCwcErA_sEIOl6KEdhp2gNRK8', time=2024-03-09T16:48:23.375117+0000, chain.objs.size()=2 2024-03-09T14:53:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.PRJj2nODkvBDgmQUCXF8EPXaGcYcZ_I_0 2024-03-09T14:53:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__shadow_.PRJj2nODkvBDgmQUCXF8EPXaGcYcZ_I_1 2024-03-09T14:53:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.2_d15e3a387bdcc02287b7f0a248c8905b', time=2024-03-09T16:48:23.409023+0000, chain.objs.size()=3 2024-03-09T14:53:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.1 2024-03-09T14:53:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.2 2024-03-09T14:53:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-LUKEWARM.2~PNXQ7H8R4MHNjVBnu2fMPvCrMzac2cE.3 2024-03-09T14:53:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3738874394170569229', time=2024-03-09T16:50:59.168498+0000, chain.objs.size()=1 2024-03-09T14:53:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.nf0v9b8g9TFnGeJj7ubFcK0NIY2v4rP_0 2024-03-09T14:53:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6576608937994383028', time=2024-03-09T16:51:03.795087+0000, chain.objs.size()=1 2024-03-09T14:53:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.241+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.lWPIIm6zJN8o4bbaZF_PaTk9RWhVDzg_0 2024-03-09T14:53:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.245+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13258060396458455755', time=2024-03-09T16:51:38.945660+0000, chain.objs.size()=1 2024-03-09T14:53:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.245+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.q6FcMRyW6b5NfsNudM7GMlZNL-XiYjR_0 2024-03-09T14:53:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.245+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6504590766528240214', time=2024-03-09T16:52:27.077217+0000, chain.objs.size()=1 2024-03-09T14:53:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.245+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.OhXw0YnsxyCGnmRyvpzPkJO51b2i3yw_0 2024-03-09T14:53:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.245+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14826887521932515390', time=2024-03-09T16:53:02.362307+0000, chain.objs.size()=1 2024-03-09T14:53:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.245+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.cNa0QaSbA-sOqYWxyJls8zOa6fYCpH8_0 2024-03-09T14:53:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.245+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6861167692489963277', time=2024-03-09T16:53:04.067601+0000, chain.objs.size()=1 2024-03-09T14:53:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.245+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17__shadow_.nnUeRvnFgK1T1T-LA7mbjiIyNkfFdDI_0 2024-03-09T14:53:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.249+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.257+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T14:53:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.261+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.261+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=10, truncated=0, next_marker='' 2024-03-09T14:53:43.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11924295039160912840', time=2024-03-09T16:48:23.720874+0000, chain.objs.size()=1 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.7__shadow_.06poVVlwvJuNBLPkHB6EV9G0wwCl5F9_0 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.674452567279814783', time=2024-03-09T16:50:20.619982+0000, chain.objs.size()=1 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.EKpDdn_HKt0upNeQGpquaYUBIjapWrf_0 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15606777697090082955', time=2024-03-09T16:50:24.811089+0000, chain.objs.size()=1 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.qBTvyI7hrYnd-I5vdtrLoTzS7V600jt_0 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9012610524805839264', time=2024-03-09T16:51:03.761068+0000, chain.objs.size()=1 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.QBSkEv9aEMTW2m14dSrI2VYMUqwpM9x_0 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13692147855145215626', time=2024-03-09T16:51:38.962595+0000, chain.objs.size()=1 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.bDPBHa-tjUwKbMT09jhVRvF78Pnj94q_0 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15216255562180791516', time=2024-03-09T16:51:47.726017+0000, chain.objs.size()=1 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.NQu4HMq66SI6Ff2fin4pOYevcAuVUrT_0 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12414164285977857177', time=2024-03-09T16:52:22.881178+0000, chain.objs.size()=1 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.IvORH3_-cB80GsBotzjsYGUgQrcUSFa_0 2024-03-09T14:53:43.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11849695486250094652', time=2024-03-09T16:53:02.152004+0000, chain.objs.size()=1 2024-03-09T14:53:43.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.k2_fwJs9UgEFi8np4oTg1lC2YRpNaZ3_0 2024-03-09T14:53:43.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6450626010743946679', time=2024-03-09T16:53:02.169040+0000, chain.objs.size()=1 2024-03-09T14:53:43.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.s4N2-waBcbDFDaa3wIX-5Wp8bFEDIBR_0 2024-03-09T14:53:43.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10609377075707753688', time=2024-03-09T16:53:04.001020+0000, chain.objs.size()=1 2024-03-09T14:53:43.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.265+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.17__shadow_.VH9bAqQqk3CGfbF3RdSn6qOj71BCVZ3_0 2024-03-09T14:53:43.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.269+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.277+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T14:53:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker='' 2024-03-09T14:53:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13120652936449906147', time=2024-03-09T16:50:19.590006+0000, chain.objs.size()=1 2024-03-09T14:53:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.7OE9YZekD-rUdTfdTDkjqCV1FCMPT1__0 2024-03-09T14:53:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1909170292345377915', time=2024-03-09T16:50:20.399163+0000, chain.objs.size()=1 2024-03-09T14:53:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.7QUikexgSkbF7faS7vqylzC9BnW9BLV_0 2024-03-09T14:53:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16639238157931040498', time=2024-03-09T16:51:39.017979+0000, chain.objs.size()=1 2024-03-09T14:53:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.D7Aj6-7yZ4VVKBASve5AvB3Ofdga7e__0 2024-03-09T14:53:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.917417412003900054', time=2024-03-09T16:51:47.728459+0000, chain.objs.size()=1 2024-03-09T14:53:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.GsxHXb0ndsctqtzY5v2acip4z5h-dK6_0 2024-03-09T14:53:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11497178289634994811', time=2024-03-09T16:51:47.809650+0000, chain.objs.size()=1 2024-03-09T14:53:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.vFhq41bb0WzM_3P7YijnPOg84vTGRFc_0 2024-03-09T14:53:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13386398719385863770', time=2024-03-09T16:52:22.774580+0000, chain.objs.size()=1 2024-03-09T14:53:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.rZgBOAS6ZuYC1tIvjnr3olLujBCUYxv_0 2024-03-09T14:53:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3424688347611006657', time=2024-03-09T16:52:27.156884+0000, chain.objs.size()=1 2024-03-09T14:53:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.DaB1ZgYocgIl0jWCnU1AUsacuGVPg4w_0 2024-03-09T14:53:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9549555326625945716', time=2024-03-09T16:53:02.148325+0000, chain.objs.size()=1 2024-03-09T14:53:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.281+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.HwlH0cEfkHdKGSyjmvz7wAFZ0rHW732_0 2024-03-09T14:53:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.289+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.293+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T14:53:43.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.297+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.297+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2024-03-09T14:53:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5773990313830998531', time=2024-03-09T16:48:23.179290+0000, chain.objs.size()=1 2024-03-09T14:53:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.ehEWaBBPx_0LAoLFpfGo1Zeknu9diW8_0 2024-03-09T14:53:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.ehEWaBBPx_0LAoLFpfGo1Zeknu9diW8_1_2646a65bd3b80e7732a4f9cee15a21d7', time=2024-03-09T16:48:23.189765+0000, chain.objs.size()=1 2024-03-09T14:53:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.15__shadow_.ehEWaBBPx_0LAoLFpfGo1Zeknu9diW8_0 2024-03-09T14:53:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15357775618702116300', time=2024-03-09T16:48:23.431622+0000, chain.objs.size()=3 2024-03-09T14:53:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.1 2024-03-09T14:53:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.2 2024-03-09T14:53:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-STANDARD.2~o3-Pw9G2sPOJDbGgchJzx9f-PtJyZIG.3 2024-03-09T14:53:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12764550389043283490', time=2024-03-09T16:48:23.793444+0000, chain.objs.size()=1 2024-03-09T14:53:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.8__shadow_.Pev7LwXAuL1O0-LElHQp_ngbPn4VO3F_0 2024-03-09T14:53:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15940397791667813442', time=2024-03-09T16:48:23.815770+0000, chain.objs.size()=1 2024-03-09T14:53:43.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.8__shadow_.5adlYQ7CWuxeRTCex7tU7N8s5keeJWY_0 2024-03-09T14:53:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15146865383656673247', time=2024-03-09T16:50:20.590897+0000, chain.objs.size()=1 2024-03-09T14:53:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.WG0YyK3PQ2bCgWsObZBxAIMGl57tRD6_0 2024-03-09T14:53:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9620606313280500525', time=2024-03-09T16:50:59.066190+0000, chain.objs.size()=1 2024-03-09T14:53:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.LfrVf-r7_wgTaZllgPPhyPSkHBzFzFl_0 2024-03-09T14:53:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9604248014196341720', time=2024-03-09T16:51:47.713763+0000, chain.objs.size()=1 2024-03-09T14:53:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.OJYkZj0b65hMFKAn2Kcahtl4O6CGVGk_0 2024-03-09T14:53:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17215871512630585947', time=2024-03-09T16:52:22.779063+0000, chain.objs.size()=1 2024-03-09T14:53:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.DjjfUvDtVvRxfNO8bjUIPQTVNMHNx55_0 2024-03-09T14:53:43.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18134182357837983015', time=2024-03-09T16:52:27.294070+0000, chain.objs.size()=1 2024-03-09T14:53:43.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.301+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.flWH8tJnLv0y7K4D8Cy84a9f_65TUkd_0 2024-03-09T14:53:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.305+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9104220151778353987', time=2024-03-09T16:53:02.270900+0000, chain.objs.size()=1 2024-03-09T14:53:43.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.305+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.79fhaUjpOzZpLr95Japv1dxEXIJGVKY_0 2024-03-09T14:53:43.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.309+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.313+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T14:53:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.321+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.321+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.321+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2024-03-09T14:53:43.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.321+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15572770197744358055', time=2024-03-09T16:50:24.923820+0000, chain.objs.size()=1 2024-03-09T14:53:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.321+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.20__shadow_.YyuSUEs8mCQGl8mSp9VnpTkF2VZf2gD_0 2024-03-09T14:53:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.321+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.119212936366934605', time=2024-03-09T16:50:59.166588+0000, chain.objs.size()=1 2024-03-09T14:53:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.321+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.f1D9PSPgEJAUN2XGwWqAXNlqSg67dzg_0 2024-03-09T14:53:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.321+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8105688952546617648', time=2024-03-09T16:50:59.168619+0000, chain.objs.size()=1 2024-03-09T14:53:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.321+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.KpfhsETE3fMf3SJbdOdySmVY2mnux8I_0 2024-03-09T14:53:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.321+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1538984711847182136', time=2024-03-09T16:51:43.944514+0000, chain.objs.size()=1 2024-03-09T14:53:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.321+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.pL7DTfdxZ9EUev-5g7Goa3T61YAmGxY_0 2024-03-09T14:53:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.325+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9362660655720394634', time=2024-03-09T16:51:43.946406+0000, chain.objs.size()=1 2024-03-09T14:53:43.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.325+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.24__shadow_.KvbVS_1mRRFcMX-iw8L3ihtTxPpovGI_0 2024-03-09T14:53:43.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.329+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.333+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T14:53:43.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.337+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2024-03-09T14:53:43.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9475011269421850760', time=2024-03-09T16:48:23.341244+0000, chain.objs.size()=3 2024-03-09T14:53:43.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.1 2024-03-09T14:53:43.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.2 2024-03-09T14:53:43.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.16__multipart_foo-FROZEN.2~0IjxVCFHW4zHNQgqVaGPyhcElD_p7Rb.3 2024-03-09T14:53:43.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13815806843305265435', time=2024-03-09T16:50:59.029584+0000, chain.objs.size()=1 2024-03-09T14:53:43.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.21__shadow_.2NLVGiOck7lKY7jmZejEJ-HRDuGfuuc_0 2024-03-09T14:53:43.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4881836002667733503', time=2024-03-09T16:51:47.818389+0000, chain.objs.size()=1 2024-03-09T14:53:43.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.25__shadow_.rbERwgPKPSOQlw1rnEQkiJKXX3WLW7R_0 2024-03-09T14:53:43.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4049790817602726304', time=2024-03-09T16:52:27.229078+0000, chain.objs.size()=1 2024-03-09T14:53:43.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.6HxlM-ys6CsCA5JqXeOhDkEgmB_Happ_0 2024-03-09T14:53:43.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10326261825919254556', time=2024-03-09T16:53:02.271028+0000, chain.objs.size()=1 2024-03-09T14:53:43.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.341+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.Py3N28DV_fAflcnIqzgiv8h1z8xIvaz_0 2024-03-09T14:53:43.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.345+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.353+0000 7f2422a8d980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T14:53:43.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2024-03-09T14:53:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14802124099349834369', time=2024-03-09T16:50:20.421111+0000, chain.objs.size()=1 2024-03-09T14:53:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.19__shadow_.D_eqzG7ofRd0fxl1NwQvHJbsfWeKuVO_0 2024-03-09T14:53:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3131823048477698704', time=2024-03-09T16:51:03.759117+0000, chain.objs.size()=1 2024-03-09T14:53:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.22__shadow_.5CJqsKf2VfmwbtJPQnC1sn_bqpiz5c3_0 2024-03-09T14:53:43.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7906652597549124898', time=2024-03-09T16:51:39.021187+0000, chain.objs.size()=1 2024-03-09T14:53:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.23__shadow_.mR-GXdiQRVWbKoskkxSMPYQrsHmlZ6Q_0 2024-03-09T14:53:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6105678660883622927', time=2024-03-09T16:52:22.970490+0000, chain.objs.size()=1 2024-03-09T14:53:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.26__shadow_.v7S53rt433IZCRgmxzCvunXaGo9Wd75_0 2024-03-09T14:53:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13553928869747930071', time=2024-03-09T16:52:27.249340+0000, chain.objs.size()=1 2024-03-09T14:53:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.27__shadow_.X8wMaokCfPLHZlpF3fShFr-CvPlle1__0 2024-03-09T14:53:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18108915189631506813', time=2024-03-09T16:53:02.247448+0000, chain.objs.size()=1 2024-03-09T14:53:43.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.357+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.28__shadow_.T_F7UB_Ygu5R9OFoBrvVysi2CYpvaH0_0 2024-03-09T14:53:43.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.365+0000 7f2422a8d980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T14:53:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.369+0000 7f22d67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.377+0000 7f2422a8d980 20 remove_watcher() i=0 2024-03-09T14:53:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.377+0000 7f2422a8d980 2 removed watcher, disabling cache 2024-03-09T14:53:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.377+0000 7f2422a8d980 20 remove_watcher() i=1 2024-03-09T14:53:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.381+0000 7f2422a8d980 20 remove_watcher() i=2 2024-03-09T14:53:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.385+0000 7f2422a8d980 20 remove_watcher() i=3 2024-03-09T14:53:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.389+0000 7f2422a8d980 20 remove_watcher() i=4 2024-03-09T14:53:43.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.389+0000 7f2422a8d980 20 remove_watcher() i=5 2024-03-09T14:53:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.389+0000 7f2422a8d980 20 remove_watcher() i=6 2024-03-09T14:53:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.393+0000 7f2422a8d980 20 remove_watcher() i=7 2024-03-09T14:53:43.416 DEBUG:teuthology.orchestra.run.smithi001:> radosgw-admin gc process --include-all 2024-03-09T14:53:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T14:53:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T14:53:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.537+0000 7fe084bd1980 20 rados->read ofs=0 len=0 2024-03-09T14:53:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.537+0000 7fe084bd1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.537+0000 7fe084bd1980 20 realm 2024-03-09T14:53:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.537+0000 7fe084bd1980 20 rados->read ofs=0 len=0 2024-03-09T14:53:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.541+0000 7fe084bd1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.541+0000 7fe084bd1980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T14:53:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.541+0000 7fe084bd1980 20 rados->read ofs=0 len=0 2024-03-09T14:53:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.541+0000 7fe084bd1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.541+0000 7fe084bd1980 20 rados->read ofs=0 len=0 2024-03-09T14:53:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.541+0000 7fe084bd1980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.541+0000 7fe084bd1980 20 rados->read ofs=0 len=0 2024-03-09T14:53:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.541+0000 7fe084bd1980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T14:53:43.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.541+0000 7fe084bd1980 20 searching for the correct realm 2024-03-09T14:53:43.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.545+0000 7fe0826e9640 0 -- 172.21.15.1:0/3816520546 >> [v2:172.21.15.1:6824/1666056338,v1:172.21.15.1:6825/1666056338] conn(0x7fe074008040 msgr2=0x7fe074057d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T14:53:43.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.757+0000 7fe082eea640 0 -- 172.21.15.1:0/3816520546 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7fe06c0b0070 msgr2=0x7fe06c0eec60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T14:53:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.773+0000 7fe084bd1980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T14:53:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.773+0000 7fe084bd1980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T14:53:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.773+0000 7fe084bd1980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T14:53:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.773+0000 7fe084bd1980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T14:53:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.773+0000 7fe084bd1980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T14:53:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.773+0000 7fe084bd1980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T14:53:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.773+0000 7fe084bd1980 20 rados->read ofs=0 len=0 2024-03-09T14:53:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.773+0000 7fe084bd1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:43.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.773+0000 7fe084bd1980 20 rados->read ofs=0 len=0 2024-03-09T14:53:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T14:53:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 rados->read ofs=0 len=0 2024-03-09T14:53:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T14:53:43.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 zone default found 2024-03-09T14:53:43.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 4 Realm: () 2024-03-09T14:53:43.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T14:53:43.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T14:53:43.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T14:53:43.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 zonegroup default 2024-03-09T14:53:43.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 rados->read ofs=0 len=0 2024-03-09T14:53:43.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:43.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 rados->read ofs=0 len=0 2024-03-09T14:53:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 rados->read ofs=0 len=0 2024-03-09T14:53:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T14:53:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 started sync module instance, tier type = 2024-03-09T14:53:43.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.777+0000 7fe084bd1980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T14:53:43.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.801+0000 7fe084bd1980 20 add_watcher() i=0 2024-03-09T14:53:43.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.801+0000 7fe084bd1980 20 add_watcher() i=1 2024-03-09T14:53:43.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.801+0000 7fe084bd1980 20 add_watcher() i=2 2024-03-09T14:53:43.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.801+0000 7fe084bd1980 20 add_watcher() i=3 2024-03-09T14:53:43.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.801+0000 7fe084bd1980 20 add_watcher() i=4 2024-03-09T14:53:43.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.801+0000 7fe084bd1980 20 add_watcher() i=5 2024-03-09T14:53:43.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.801+0000 7fe084bd1980 20 add_watcher() i=6 2024-03-09T14:53:43.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.805+0000 7fe084bd1980 20 add_watcher() i=7 2024-03-09T14:53:43.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.805+0000 7fe084bd1980 2 all 8 watchers are set, enabling cache 2024-03-09T14:53:43.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.809+0000 7fdf397e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:43.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.809+0000 7fe084bd1980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T14:53:43.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.809+0000 7fe084bd1980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T14:53:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.905+0000 7fdf337fe640 20 reqs_thread_entry: start 2024-03-09T14:53:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.905+0000 7fe084bd1980 20 init_complete bucket index max shards: 1 2024-03-09T14:53:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.905+0000 7fe084bd1980 20 Filter name: none 2024-03-09T14:53:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.905+0000 7fdf317fa640 20 reqs_thread_entry: start 2024-03-09T14:53:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.905+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T14:53:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.909+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.909+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.909+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.909+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T14:53:43.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.913+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.913+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.917+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.917+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T14:53:43.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.921+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.925+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.925+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.929+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T14:53:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.933+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.933+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.937+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.941+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T14:53:43.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.945+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.945+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.945+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.949+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T14:53:43.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.953+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.957+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.957+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.961+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T14:53:43.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.961+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.965+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.965+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.965+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T14:53:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.969+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.973+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.973+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.977+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T14:53:43.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.981+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.981+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.981+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.985+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T14:53:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.985+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.989+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.989+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.993+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T14:53:43.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.997+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.997+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:43.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:43.997+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.001+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T14:53:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.005+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.009+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.009+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.009+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T14:53:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.013+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.017+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.017+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.021+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T14:53:44.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.029+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.029+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.029+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.037+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T14:53:44.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.045+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.045+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.045+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.049+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T14:53:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.053+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.053+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.057+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.057+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T14:53:44.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.061+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.065+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.065+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.069+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T14:53:44.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.073+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.073+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.073+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.077+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T14:53:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.081+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.085+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.085+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.089+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T14:53:44.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.093+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.097+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.097+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.101+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T14:53:44.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.105+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.105+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.105+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.109+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T14:53:44.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.113+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.117+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.117+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.121+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T14:53:44.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.125+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.125+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.125+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.129+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T14:53:44.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.133+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.137+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.137+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.137+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T14:53:44.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.145+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.145+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.145+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.149+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T14:53:44.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.153+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.157+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.157+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.157+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T14:53:44.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.161+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.165+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.165+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.169+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T14:53:44.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.173+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.173+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.177+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.177+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T14:53:44.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.181+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.185+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.185+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.185+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T14:53:44.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.189+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.193+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.193+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.197+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T14:53:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.201+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.201+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.201+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.205+0000 7fe084bd1980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T14:53:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.209+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.213+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T14:53:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.213+0000 7fe084bd1980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T14:53:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.217+0000 7fdf397e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T14:53:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.225+0000 7fe084bd1980 20 remove_watcher() i=0 2024-03-09T14:53:44.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.225+0000 7fe084bd1980 2 removed watcher, disabling cache 2024-03-09T14:53:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.225+0000 7fe084bd1980 20 remove_watcher() i=1 2024-03-09T14:53:44.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.229+0000 7fe084bd1980 20 remove_watcher() i=2 2024-03-09T14:53:44.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.233+0000 7fe084bd1980 20 remove_watcher() i=3 2024-03-09T14:53:44.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.237+0000 7fe084bd1980 20 remove_watcher() i=4 2024-03-09T14:53:44.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.237+0000 7fe084bd1980 20 remove_watcher() i=5 2024-03-09T14:53:44.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.241+0000 7fe084bd1980 20 remove_watcher() i=6 2024-03-09T14:53:44.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T14:53:44.241+0000 7fe084bd1980 20 remove_watcher() i=7 2024-03-09T14:53:44.264 DEBUG:teuthology.orchestra.run.smithi001:> cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test 2024-03-09T14:53:44.927 INFO:teuthology.orchestra.run.smithi001.stdout:Starting a Gradle Daemon (subsequent builds will be faster) 2024-03-09T14:53:49.628 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :clean UP-TO-DATE 2024-03-09T14:54:08.727 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :compileJava 2024-03-09T14:54:08.727 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :processResources 2024-03-09T14:54:08.727 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :classes 2024-03-09T14:54:09.426 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:09.426 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :compileTestJava 2024-03-09T14:54:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:Note: /home/ubuntu/cephtest/s3-tests-java/src/test/java/BucketTest.java uses or overrides a deprecated API. 2024-03-09T14:54:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:Note: Recompile with -Xlint:deprecation for details. 2024-03-09T14:54:09.428 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:09.428 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :processTestResources NO-SOURCE 2024-03-09T14:54:09.428 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :testClasses 2024-03-09T14:54:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:16.427 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :test 2024-03-09T14:54:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:16.427 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadLLAPIAWS4 STARTED 2024-03-09T14:54:29.226 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:29.227 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadLLAPIAWS4 PASSED 2024-03-09T14:54:29.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:29.829 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadNotFoundLLAPIAWS4 STARTED 2024-03-09T14:54:30.026 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:30.026 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadNotFoundLLAPIAWS4 PASSED 2024-03-09T14:54:30.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:30.626 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testDownloadHLAPIAWS4 STARTED 2024-03-09T14:54:52.926 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:52.927 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testDownloadHLAPIAWS4 PASSED 2024-03-09T14:54:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:53.627 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoBucketHLAPIAWS4 STARTED 2024-03-09T14:54:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:53.627 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoBucketHLAPIAWS4 PASSED 2024-03-09T14:54:53.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:53.826 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoKeyHLAPIAWS4 STARTED 2024-03-09T14:54:54.026 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:54.026 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoKeyHLAPIAWS4 PASSED 2024-03-09T14:54:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:54:54.626 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyHLAPIAWS4 STARTED 2024-03-09T14:55:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:55:08.127 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyHLAPIAWS4 PASSED 2024-03-09T14:55:09.326 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:55:09.326 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyMultipleSizesLLAPIAWS4 STARTED 2024-03-09T14:55:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:55:09.726 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyMultipleSizesLLAPIAWS4 STANDARD_OUT 2024-03-09T14:55:09.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09 14:55:09,709 [Test worker] WARN com.amazonaws.util.Base64 - JAXB is unavailable. Will fallback to SDK implementation which may be less performant 2024-03-09T14:56:13.627 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:13.628 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyMultipleSizesLLAPIAWS4 PASSED 2024-03-09T14:56:14.726 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:14.726 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCBucketHLAPIAWS4 STARTED 2024-03-09T14:56:14.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:14.826 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCBucketHLAPIAWS4 PASSED 2024-03-09T14:56:15.426 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:15.426 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCKeyHLAPIAWS4 STARTED 2024-03-09T14:56:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:15.629 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCKeyHLAPIAWS4 PASSED 2024-03-09T14:56:16.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:16.626 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadHLAPIAWS4 STARTED 2024-03-09T14:56:30.526 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:30.526 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadHLAPIAWS4 PASSED 2024-03-09T14:56:31.226 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:31.226 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadNoBucketHLAPIAWS4 STARTED 2024-03-09T14:56:31.326 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:31.326 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadNoBucketHLAPIAWS4 PASSED 2024-03-09T14:56:31.526 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:31.526 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadWithPauseHLAPIAWS4 STARTED 2024-03-09T14:56:50.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:50.427 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadWithPauseHLAPIAWS4 PASSED 2024-03-09T14:56:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:55.527 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests STANDARD_OUT 2024-03-09T14:56:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09 14:56:55,488 [s3-transfer-manager-worker-10] WARN com.amazonaws.services.s3.internal.S3AbortableInputStream - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use. 2024-03-09T14:56:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09 14:56:55,498 [s3-transfer-manager-worker-9] WARN com.amazonaws.services.s3.internal.S3AbortableInputStream - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use. 2024-03-09T14:56:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:56:56.826 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadHLAPIAWS4 STARTED 2024-03-09T14:57:23.726 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:57:23.727 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadHLAPIAWS4 PASSED 2024-03-09T14:57:24.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:57:24.829 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadIncorrectMissingPartLLAPIAWS4 STARTED 2024-03-09T14:57:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:57:31.426 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadIncorrectMissingPartLLAPIAWS4 PASSED 2024-03-09T14:57:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T14:57:32.026 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadMultipleSizesLLAPIAWS4 STARTED 2024-03-09T15:01:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:01:08.827 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadMultipleSizesLLAPIAWS4 PASSED 2024-03-09T15:01:09.525 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:01:09.526 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadOverwriteExistingObjectLLAPIAWS4 STARTED 2024-03-09T15:01:27.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:01:27.626 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadOverwriteExistingObjectLLAPIAWS4 PASSED 2024-03-09T15:01:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:01:28.126 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadSmallLLAPIAWS4 STARTED 2024-03-09T15:01:50.326 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:01:50.326 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadSmallLLAPIAWS4 PASSED 2024-03-09T15:01:50.925 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:01:50.925 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadWithPauseAWS4 STARTED 2024-03-09T15:01:55.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:01:55.626 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadWithPauseAWS4 PASSED 2024-03-09T15:02:06.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:06.826 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateAfterToday4AWS4 STARTED 2024-03-09T15:02:08.225 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:08.226 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateAfterToday4AWS4 PASSED 2024-03-09T15:02:11.425 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:11.426 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeEpochAWS4 STARTED 2024-03-09T15:02:11.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:11.626 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeEpochAWS4 PASSED 2024-03-09T15:02:12.325 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:12.326 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeToday4AWS4 STARTED 2024-03-09T15:02:12.525 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:12.526 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeToday4AWS4 PASSED 2024-03-09T15:02:13.225 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:13.226 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateEmptyAWS4 STARTED 2024-03-09T15:02:13.426 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:13.426 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateEmptyAWS4 PASSED 2024-03-09T15:02:14.125 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:14.126 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateInvalidAWS4 STARTED 2024-03-09T15:02:14.325 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:14.326 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateInvalidAWS4 PASSED 2024-03-09T15:02:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:15.025 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateNoneAWS4 STARTED 2024-03-09T15:02:15.325 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:15.326 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateNoneAWS4 PASSED 2024-03-09T15:02:16.025 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:16.026 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadMd5InvalidGarbageAWS4 STARTED 2024-03-09T15:02:16.225 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:16.225 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadMd5InvalidGarbageAWS4 PASSED 2024-03-09T15:02:16.825 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:16.826 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testUploadFileHLAPIBigFileAWS4 STARTED 2024-03-09T15:02:47.126 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:47.126 INFO:teuthology.orchestra.run.smithi001.stdout:suite > AWS4 tests > AWS4Test.testUploadFileHLAPIBigFileAWS4 PASSED 2024-03-09T15:02:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:47.809 INFO:teuthology.orchestra.run.smithi001.stdout:---------------------------------------------------------------------- 2024-03-09T15:02:47.809 INFO:teuthology.orchestra.run.smithi001.stdout:| Results: SUCCESS (26 tests, 26 successes, 0 failures, 0 skipped) | 2024-03-09T15:02:47.809 INFO:teuthology.orchestra.run.smithi001.stdout:---------------------------------------------------------------------- 2024-03-09T15:02:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:47.809 INFO:teuthology.orchestra.run.smithi001.stdout:Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. 2024-03-09T15:02:47.809 INFO:teuthology.orchestra.run.smithi001.stdout:Use '--warning-mode all' to show the individual deprecation warnings. 2024-03-09T15:02:47.809 INFO:teuthology.orchestra.run.smithi001.stdout:See https://docs.gradle.org/6.0.1/userguide/command_line_interface.html#sec:command_line_warnings 2024-03-09T15:02:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:47.809 INFO:teuthology.orchestra.run.smithi001.stdout:BUILD SUCCESSFUL in 9m 3s 2024-03-09T15:02:47.809 INFO:teuthology.orchestra.run.smithi001.stdout:5 actionable tasks: 4 executed, 1 up-to-date 2024-03-09T15:02:48.139 DEBUG:teuthology.orchestra.run.smithi001:> radosgw-admin gc process --include-all 2024-03-09T15:02:48.170 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:02:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:02:48.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.210+0000 7ff5d69dc980 20 rados->read ofs=0 len=0 2024-03-09T15:02:48.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.214+0000 7ff5d69dc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:48.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.214+0000 7ff5d69dc980 20 realm 2024-03-09T15:02:48.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.214+0000 7ff5d69dc980 20 rados->read ofs=0 len=0 2024-03-09T15:02:48.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.214+0000 7ff5d69dc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:48.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.214+0000 7ff5d69dc980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:02:48.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.214+0000 7ff5d69dc980 20 rados->read ofs=0 len=0 2024-03-09T15:02:48.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.214+0000 7ff5d69dc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:48.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.214+0000 7ff5d69dc980 20 rados->read ofs=0 len=0 2024-03-09T15:02:48.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.214+0000 7ff5d69dc980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:02:48.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.214+0000 7ff5d69dc980 20 rados->read ofs=0 len=0 2024-03-09T15:02:48.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.218+0000 7ff5d69dc980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:02:48.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.218+0000 7ff5d69dc980 20 searching for the correct realm 2024-03-09T15:02:48.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.222+0000 7ff5d3cf3640 0 -- 172.21.15.1:0/2389873443 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55912b7bc2a0 msgr2=0x55912b7be760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:02:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.230+0000 7ff5d3cf3640 0 -- 172.21.15.1:0/2389873443 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7ff5c407c2e0 msgr2=0x7ff5c405d940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T15:02:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:02:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:02:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:02:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:02:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:02:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:02:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 rados->read ofs=0 len=0 2024-03-09T15:02:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 rados->read ofs=0 len=0 2024-03-09T15:02:48.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:02:48.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 rados->read ofs=0 len=0 2024-03-09T15:02:48.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:02:48.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 zone default found 2024-03-09T15:02:48.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 4 Realm: () 2024-03-09T15:02:48.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:02:48.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:02:48.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:02:48.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 zonegroup default 2024-03-09T15:02:48.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 rados->read ofs=0 len=0 2024-03-09T15:02:48.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:48.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 rados->read ofs=0 len=0 2024-03-09T15:02:48.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:48.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.238+0000 7ff5d69dc980 20 rados->read ofs=0 len=0 2024-03-09T15:02:48.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.242+0000 7ff5d69dc980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:48.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.242+0000 7ff5d69dc980 20 started sync module instance, tier type = 2024-03-09T15:02:48.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.242+0000 7ff5d69dc980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:02:48.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.254+0000 7ff5d69dc980 20 add_watcher() i=0 2024-03-09T15:02:48.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.254+0000 7ff5d69dc980 20 add_watcher() i=1 2024-03-09T15:02:48.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.254+0000 7ff5d69dc980 20 add_watcher() i=2 2024-03-09T15:02:48.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.254+0000 7ff5d69dc980 20 add_watcher() i=3 2024-03-09T15:02:48.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.254+0000 7ff5d69dc980 20 add_watcher() i=4 2024-03-09T15:02:48.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.254+0000 7ff5d69dc980 20 add_watcher() i=5 2024-03-09T15:02:48.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.254+0000 7ff5d69dc980 20 add_watcher() i=6 2024-03-09T15:02:48.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.258+0000 7ff5d69dc980 20 add_watcher() i=7 2024-03-09T15:02:48.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.258+0000 7ff5d69dc980 2 all 8 watchers are set, enabling cache 2024-03-09T15:02:48.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.258+0000 7ff48afdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:02:48.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.258+0000 7ff5d69dc980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:02:48.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.258+0000 7ff5d69dc980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:02:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.334+0000 7ff489fdb640 20 reqs_thread_entry: start 2024-03-09T15:02:48.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.334+0000 7ff5d69dc980 20 init_complete bucket index max shards: 1 2024-03-09T15:02:48.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.334+0000 7ff5d69dc980 20 Filter name: none 2024-03-09T15:02:48.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.334+0000 7ff3fb7fe640 20 reqs_thread_entry: start 2024-03-09T15:02:48.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.334+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T15:02:48.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.338+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.338+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.338+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.338+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5226870970546556461', time=2024-03-09T17:01:22.424220+0000, chain.objs.size()=1 2024-03-09T15:02:48.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.338+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.45__shadow_.zEw_QhlkSylVPH2BX09CaXS1DUx1rkI_0 2024-03-09T15:02:48.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.342+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.346+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T15:02:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.346+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.350+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.350+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.354+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T15:02:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.358+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.358+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4055718034818383692', time=2024-03-09T16:58:51.059488+0000, chain.objs.size()=11 2024-03-09T15:02:48.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~SG7WKubrZ8Fhh-5Bo1SK_hs8jU0DbJq.1 2024-03-09T15:02:48.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~SG7WKubrZ8Fhh-5Bo1SK_hs8jU0DbJq.2 2024-03-09T15:02:48.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~SG7WKubrZ8Fhh-5Bo1SK_hs8jU0DbJq.3 2024-03-09T15:02:48.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~SG7WKubrZ8Fhh-5Bo1SK_hs8jU0DbJq.4 2024-03-09T15:02:48.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~SG7WKubrZ8Fhh-5Bo1SK_hs8jU0DbJq.5 2024-03-09T15:02:48.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~SG7WKubrZ8Fhh-5Bo1SK_hs8jU0DbJq.6 2024-03-09T15:02:48.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~SG7WKubrZ8Fhh-5Bo1SK_hs8jU0DbJq.7 2024-03-09T15:02:48.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~SG7WKubrZ8Fhh-5Bo1SK_hs8jU0DbJq.8 2024-03-09T15:02:48.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~SG7WKubrZ8Fhh-5Bo1SK_hs8jU0DbJq.9 2024-03-09T15:02:48.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~SG7WKubrZ8Fhh-5Bo1SK_hs8jU0DbJq.10 2024-03-09T15:02:48.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.362+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~SG7WKubrZ8Fhh-5Bo1SK_hs8jU0DbJq.11 2024-03-09T15:02:48.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.374+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.378+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T15:02:48.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.382+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.382+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.382+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.382+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14199960024450701487', time=2024-03-09T16:55:59.062904+0000, chain.objs.size()=5 2024-03-09T15:02:48.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.382+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~oZpdeQGpnT5Sl5lWDfJqJZH6Dn5r5Q3.1 2024-03-09T15:02:48.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.382+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__shadow_key1.2~oZpdeQGpnT5Sl5lWDfJqJZH6Dn5r5Q3.1_1 2024-03-09T15:02:48.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.382+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~oZpdeQGpnT5Sl5lWDfJqJZH6Dn5r5Q3.2 2024-03-09T15:02:48.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.382+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__shadow_key1.2~oZpdeQGpnT5Sl5lWDfJqJZH6Dn5r5Q3.2_1 2024-03-09T15:02:48.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.382+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~oZpdeQGpnT5Sl5lWDfJqJZH6Dn5r5Q3.3 2024-03-09T15:02:48.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.390+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.398+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T15:02:48.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11168912656722086182', time=2024-03-09T17:01:08.694343+0000, chain.objs.size()=11 2024-03-09T15:02:48.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~evoxtnw-5gvAar8pp1E1AeWLrGdrNI-.1 2024-03-09T15:02:48.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~evoxtnw-5gvAar8pp1E1AeWLrGdrNI-.1_1 2024-03-09T15:02:48.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~evoxtnw-5gvAar8pp1E1AeWLrGdrNI-.2 2024-03-09T15:02:48.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~evoxtnw-5gvAar8pp1E1AeWLrGdrNI-.2_1 2024-03-09T15:02:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~evoxtnw-5gvAar8pp1E1AeWLrGdrNI-.3 2024-03-09T15:02:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~evoxtnw-5gvAar8pp1E1AeWLrGdrNI-.3_1 2024-03-09T15:02:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~evoxtnw-5gvAar8pp1E1AeWLrGdrNI-.4 2024-03-09T15:02:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~evoxtnw-5gvAar8pp1E1AeWLrGdrNI-.4_1 2024-03-09T15:02:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~evoxtnw-5gvAar8pp1E1AeWLrGdrNI-.5 2024-03-09T15:02:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~evoxtnw-5gvAar8pp1E1AeWLrGdrNI-.5_1 2024-03-09T15:02:48.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.402+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~evoxtnw-5gvAar8pp1E1AeWLrGdrNI-.6 2024-03-09T15:02:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.418+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.422+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T15:02:48.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.426+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.426+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.426+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.426+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5807259064430856007', time=2024-03-09T17:02:14.628200+0000, chain.objs.size()=1 2024-03-09T15:02:48.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.426+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.52__shadow_.lTEkGGvC0DIwADDJ33bgOw8fSdmbrbg_0 2024-03-09T15:02:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.430+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.434+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T15:02:48.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.438+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.442+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.442+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.442+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6253653629169468942', time=2024-03-09T16:57:24.201851+0000, chain.objs.size()=5 2024-03-09T15:02:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.442+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__multipart_file2.mpg.2~UsrbKxIcUh4j7S_GXHEOMCRzxCrMebv.1 2024-03-09T15:02:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.442+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__multipart_file2.mpg.2~UsrbKxIcUh4j7S_GXHEOMCRzxCrMebv.2 2024-03-09T15:02:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.442+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__multipart_file2.mpg.2~UsrbKxIcUh4j7S_GXHEOMCRzxCrMebv.3 2024-03-09T15:02:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.442+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__multipart_file2.mpg.2~UsrbKxIcUh4j7S_GXHEOMCRzxCrMebv.4 2024-03-09T15:02:48.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.442+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__multipart_file2.mpg.2~UsrbKxIcUh4j7S_GXHEOMCRzxCrMebv.5 2024-03-09T15:02:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.450+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.454+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T15:02:48.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.458+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.462+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.462+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.466+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T15:02:48.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2024-03-09T15:02:48.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8175646341775418607', time=2024-03-09T16:54:53.164962+0000, chain.objs.size()=5 2024-03-09T15:02:48.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.31__multipart_key1.2~-yBq3A0sSBMga2qz3g8ClGuTNdves3L.1 2024-03-09T15:02:48.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.31__multipart_key1.2~-yBq3A0sSBMga2qz3g8ClGuTNdves3L.2 2024-03-09T15:02:48.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.31__multipart_key1.2~-yBq3A0sSBMga2qz3g8ClGuTNdves3L.3 2024-03-09T15:02:48.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.31__multipart_key1.2~-yBq3A0sSBMga2qz3g8ClGuTNdves3L.4 2024-03-09T15:02:48.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.31__multipart_key1.2~-yBq3A0sSBMga2qz3g8ClGuTNdves3L.5 2024-03-09T15:02:48.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.33__multipart_key1.2~4sLScN7r62_RbKjhTFw4OGFHKihk8at.2_d6e8edb9e878e5184ee4a2ea1249ab35', time=2024-03-09T16:55:08.860264+0000, chain.objs.size()=5 2024-03-09T15:02:48.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.33__multipart_key1.2~4sLScN7r62_RbKjhTFw4OGFHKihk8at.1 2024-03-09T15:02:48.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.33__multipart_key1.2~4sLScN7r62_RbKjhTFw4OGFHKihk8at.2 2024-03-09T15:02:48.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.33__multipart_key1.2~4sLScN7r62_RbKjhTFw4OGFHKihk8at.3 2024-03-09T15:02:48.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.33__multipart_key1.2~4sLScN7r62_RbKjhTFw4OGFHKihk8at.4 2024-03-09T15:02:48.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.33__multipart_key1.2~4sLScN7r62_RbKjhTFw4OGFHKihk8at.5 2024-03-09T15:02:48.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3245443872189663853', time=2024-03-09T16:59:43.269296+0000, chain.objs.size()=11 2024-03-09T15:02:48.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~Y0sCW746gCD_hF9rcU5IVZeQpg7pha7.1 2024-03-09T15:02:48.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.470+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~Y0sCW746gCD_hF9rcU5IVZeQpg7pha7.2 2024-03-09T15:02:48.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.482+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~Y0sCW746gCD_hF9rcU5IVZeQpg7pha7.3 2024-03-09T15:02:48.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.482+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~Y0sCW746gCD_hF9rcU5IVZeQpg7pha7.4 2024-03-09T15:02:48.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.482+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~Y0sCW746gCD_hF9rcU5IVZeQpg7pha7.5 2024-03-09T15:02:48.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.482+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~Y0sCW746gCD_hF9rcU5IVZeQpg7pha7.6 2024-03-09T15:02:48.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.482+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~Y0sCW746gCD_hF9rcU5IVZeQpg7pha7.7 2024-03-09T15:02:48.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.482+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~Y0sCW746gCD_hF9rcU5IVZeQpg7pha7.8 2024-03-09T15:02:48.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.482+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~Y0sCW746gCD_hF9rcU5IVZeQpg7pha7.9 2024-03-09T15:02:48.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.482+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~Y0sCW746gCD_hF9rcU5IVZeQpg7pha7.10 2024-03-09T15:02:48.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.482+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~Y0sCW746gCD_hF9rcU5IVZeQpg7pha7.11 2024-03-09T15:02:48.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.498+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.502+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T15:02:48.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.506+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.510+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.510+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.510+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13942476616363137458', time=2024-03-09T16:56:30.793386+0000, chain.objs.size()=5 2024-03-09T15:02:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.510+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.40__multipart_key1.2~7pn4eKv9nYGiMBxOgb_c8HScoD4TOUH.1 2024-03-09T15:02:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.510+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.40__multipart_key1.2~7pn4eKv9nYGiMBxOgb_c8HScoD4TOUH.2 2024-03-09T15:02:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.510+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.40__multipart_key1.2~7pn4eKv9nYGiMBxOgb_c8HScoD4TOUH.3 2024-03-09T15:02:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.510+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.40__multipart_key1.2~7pn4eKv9nYGiMBxOgb_c8HScoD4TOUH.4 2024-03-09T15:02:48.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.510+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.40__multipart_key1.2~7pn4eKv9nYGiMBxOgb_c8HScoD4TOUH.5 2024-03-09T15:02:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.518+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.526+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T15:02:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.526+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=10, truncated=0, next_marker='' 2024-03-09T15:02:48.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2631226871539829052', time=2024-03-09T16:57:24.108596+0000, chain.objs.size()=1 2024-03-09T15:02:48.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__shadow_.ECAi00ZcxWTpsBCDJ2KaJQ1JMHbtt00_0 2024-03-09T15:02:48.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.358838553178931743', time=2024-03-09T17:00:12.796996+0000, chain.objs.size()=10 2024-03-09T15:02:48.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~l14bTGa7-XdqNZtXQgDvgDOqN2tOizU.1 2024-03-09T15:02:48.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~l14bTGa7-XdqNZtXQgDvgDOqN2tOizU.2 2024-03-09T15:02:48.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~l14bTGa7-XdqNZtXQgDvgDOqN2tOizU.3 2024-03-09T15:02:48.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~l14bTGa7-XdqNZtXQgDvgDOqN2tOizU.4 2024-03-09T15:02:48.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~l14bTGa7-XdqNZtXQgDvgDOqN2tOizU.5 2024-03-09T15:02:48.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~l14bTGa7-XdqNZtXQgDvgDOqN2tOizU.6 2024-03-09T15:02:48.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~l14bTGa7-XdqNZtXQgDvgDOqN2tOizU.7 2024-03-09T15:02:48.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~l14bTGa7-XdqNZtXQgDvgDOqN2tOizU.8 2024-03-09T15:02:48.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~l14bTGa7-XdqNZtXQgDvgDOqN2tOizU.9 2024-03-09T15:02:48.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~l14bTGa7-XdqNZtXQgDvgDOqN2tOizU.10 2024-03-09T15:02:48.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e', time=2024-03-09T17:02:01.997117+0000, chain.objs.size()=1 2024-03-09T15:02:48.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.530+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.13 2024-03-09T15:02:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.542+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e', time=2024-03-09T17:02:02.122883+0000, chain.objs.size()=1 2024-03-09T15:02:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.542+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.14 2024-03-09T15:02:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.542+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e', time=2024-03-09T17:02:02.225087+0000, chain.objs.size()=1 2024-03-09T15:02:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.542+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.15 2024-03-09T15:02:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.542+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e', time=2024-03-09T17:02:02.422321+0000, chain.objs.size()=1 2024-03-09T15:02:48.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.542+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.16 2024-03-09T15:02:48.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.542+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e', time=2024-03-09T17:02:02.575639+0000, chain.objs.size()=1 2024-03-09T15:02:48.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.542+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.17 2024-03-09T15:02:48.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.542+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e', time=2024-03-09T17:02:02.838624+0000, chain.objs.size()=1 2024-03-09T15:02:48.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.542+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.18 2024-03-09T15:02:48.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.542+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e', time=2024-03-09T17:02:03.080741+0000, chain.objs.size()=1 2024-03-09T15:02:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.542+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.20 2024-03-09T15:02:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.546+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e', time=2024-03-09T17:02:03.289704+0000, chain.objs.size()=21 2024-03-09T15:02:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.546+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.1 2024-03-09T15:02:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.546+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.2 2024-03-09T15:02:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.546+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.3 2024-03-09T15:02:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.546+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.4 2024-03-09T15:02:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.546+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.5 2024-03-09T15:02:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.546+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.6 2024-03-09T15:02:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.546+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.7 2024-03-09T15:02:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.546+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.8 2024-03-09T15:02:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.546+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.9 2024-03-09T15:02:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.546+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.10 2024-03-09T15:02:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.550+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.11 2024-03-09T15:02:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.550+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.12 2024-03-09T15:02:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.550+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.IP5E2v6uJH_KqmnS_TO2BVZkOr1GfPhV.13 2024-03-09T15:02:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.550+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.d7u8TnxlVhi8d3NosHORtOxkG9dNQaJm.14 2024-03-09T15:02:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.550+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.77FyJhK-HYlc5jipBjwm_h9ATqJ5eW_y.15 2024-03-09T15:02:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.550+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.-izsUnh1KM8gz3QE2TPveBhD0q4bn3ee.16 2024-03-09T15:02:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.550+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.95osog89hLe5IaCpnqvkD1aRWGJzRwj5.17 2024-03-09T15:02:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.550+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.4tXIGXI1EZurMPtzvdYgQi--NgZfwcBz.18 2024-03-09T15:02:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.554+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.19 2024-03-09T15:02:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.554+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.KHesYxY8hPtrV93eetzuyYhw8o0Y4Q50.20 2024-03-09T15:02:48.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.554+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.47__multipart_key1.2~dy229vc_WP3P3sA4CFZC2kVLxFliA9e.21 2024-03-09T15:02:48.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.558+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.562+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T15:02:48.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.862442126258891968', time=2024-03-09T16:56:54.392529+0000, chain.objs.size()=5 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.41__multipart_key1.2~ok3WuA9CewjEY3y1Ta8zMbRkqbbfXuM.1 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.41__multipart_key1.2~ok3WuA9CewjEY3y1Ta8zMbRkqbbfXuM.2 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.41__multipart_key1.2~ok3WuA9CewjEY3y1Ta8zMbRkqbbfXuM.3 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.41__multipart_key1.2~ok3WuA9CewjEY3y1Ta8zMbRkqbbfXuM.4 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.41__multipart_key1.2~ok3WuA9CewjEY3y1Ta8zMbRkqbbfXuM.5 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8877970866461322666', time=2024-03-09T17:02:47.296652+0000, chain.objs.size()=11 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.55__multipart_key1.2~Nvo8vMs6BOllJxj6Fygb1Rmadyl3Tt-.1 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.55__multipart_key1.2~Nvo8vMs6BOllJxj6Fygb1Rmadyl3Tt-.2 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.55__multipart_key1.2~Nvo8vMs6BOllJxj6Fygb1Rmadyl3Tt-.3 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.55__multipart_key1.2~Nvo8vMs6BOllJxj6Fygb1Rmadyl3Tt-.4 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.55__multipart_key1.2~Nvo8vMs6BOllJxj6Fygb1Rmadyl3Tt-.5 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.55__multipart_key1.2~Nvo8vMs6BOllJxj6Fygb1Rmadyl3Tt-.6 2024-03-09T15:02:48.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.566+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.55__multipart_key1.2~Nvo8vMs6BOllJxj6Fygb1Rmadyl3Tt-.7 2024-03-09T15:02:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.578+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.55__multipart_key1.2~Nvo8vMs6BOllJxj6Fygb1Rmadyl3Tt-.8 2024-03-09T15:02:48.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.578+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.55__multipart_key1.2~Nvo8vMs6BOllJxj6Fygb1Rmadyl3Tt-.9 2024-03-09T15:02:48.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.582+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.55__multipart_key1.2~Nvo8vMs6BOllJxj6Fygb1Rmadyl3Tt-.10 2024-03-09T15:02:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.582+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.55__multipart_key1.2~Nvo8vMs6BOllJxj6Fygb1Rmadyl3Tt-.11 2024-03-09T15:02:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.590+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.594+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T15:02:48.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.598+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.598+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.598+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.598+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3044124515902953714', time=2024-03-09T16:56:13.444269+0000, chain.objs.size()=5 2024-03-09T15:02:48.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.598+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~ZD20Ql8YiysKbQF9xBC5QXhrzBII84X.1 2024-03-09T15:02:48.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.598+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__shadow_key1.2~ZD20Ql8YiysKbQF9xBC5QXhrzBII84X.1_1 2024-03-09T15:02:48.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.598+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~ZD20Ql8YiysKbQF9xBC5QXhrzBII84X.2 2024-03-09T15:02:48.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.598+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__shadow_key1.2~ZD20Ql8YiysKbQF9xBC5QXhrzBII84X.2_1 2024-03-09T15:02:48.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.598+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~ZD20Ql8YiysKbQF9xBC5QXhrzBII84X.3 2024-03-09T15:02:48.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.606+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.610+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T15:02:48.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.614+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.618+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.618+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.618+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12248537381514901557', time=2024-03-09T17:01:50.546742+0000, chain.objs.size()=5 2024-03-09T15:02:48.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.618+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.46__multipart_key1.2~XVtfrIvtDcr1QKeYGrD-8PRKPRAr2AM.1 2024-03-09T15:02:48.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.618+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.46__multipart_key1.2~XVtfrIvtDcr1QKeYGrD-8PRKPRAr2AM.2 2024-03-09T15:02:48.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.618+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.46__multipart_key1.2~XVtfrIvtDcr1QKeYGrD-8PRKPRAr2AM.3 2024-03-09T15:02:48.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.618+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.46__multipart_key1.2~XVtfrIvtDcr1QKeYGrD-8PRKPRAr2AM.4 2024-03-09T15:02:48.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.618+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.46__multipart_key1.2~XVtfrIvtDcr1QKeYGrD-8PRKPRAr2AM.5 2024-03-09T15:02:48.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.626+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.630+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T15:02:48.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.634+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.638+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.638+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.642+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T15:02:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.646+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.646+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.646+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.646+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9733289585539645422', time=2024-03-09T16:57:23.985339+0000, chain.objs.size()=5 2024-03-09T15:02:48.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.646+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__multipart_file.mpg.2~pmlI7C4CpfTIgISx42z2D_AT64ODklZ.1 2024-03-09T15:02:48.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.646+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__multipart_file.mpg.2~pmlI7C4CpfTIgISx42z2D_AT64ODklZ.2 2024-03-09T15:02:48.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.646+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__multipart_file.mpg.2~pmlI7C4CpfTIgISx42z2D_AT64ODklZ.3 2024-03-09T15:02:48.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.646+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__multipart_file.mpg.2~pmlI7C4CpfTIgISx42z2D_AT64ODklZ.4 2024-03-09T15:02:48.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.646+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__multipart_file.mpg.2~pmlI7C4CpfTIgISx42z2D_AT64ODklZ.5 2024-03-09T15:02:48.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.658+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.662+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T15:02:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2024-03-09T15:02:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='2~zsstf4-05KJjuAG39-C0wZdtZLcKm1P', time=2024-03-09T16:54:29.179574+0000, chain.objs.size()=5 2024-03-09T15:02:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.29__multipart_key1.2~zsstf4-05KJjuAG39-C0wZdtZLcKm1P.1 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.29__multipart_key1.2~zsstf4-05KJjuAG39-C0wZdtZLcKm1P.2 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.29__multipart_key1.2~zsstf4-05KJjuAG39-C0wZdtZLcKm1P.3 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.29__multipart_key1.2~zsstf4-05KJjuAG39-C0wZdtZLcKm1P.4 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.29__multipart_key1.2~zsstf4-05KJjuAG39-C0wZdtZLcKm1P.5 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15487470616450806089', time=2024-03-09T16:56:13.793691+0000, chain.objs.size()=5 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~KG2yeVXaYhmfGaFVcnud8RQayCQPULa.1 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__shadow_key1.2~KG2yeVXaYhmfGaFVcnud8RQayCQPULa.1_1 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~KG2yeVXaYhmfGaFVcnud8RQayCQPULa.2 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__shadow_key1.2~KG2yeVXaYhmfGaFVcnud8RQayCQPULa.2_1 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~KG2yeVXaYhmfGaFVcnud8RQayCQPULa.3 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9132138136561669218', time=2024-03-09T17:02:09.149933+0000, chain.objs.size()=1 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.48__shadow_.n1ai06b_1c3rj6-INYbp-TSq9w79cls_0 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4511325939146056717', time=2024-03-09T17:02:15.550969+0000, chain.objs.size()=1 2024-03-09T15:02:48.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.666+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.53__shadow_.gl8sIyXAp200QsRxerNqTC-liuG5nRN_0 2024-03-09T15:02:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.678+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.686+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T15:02:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.690+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.690+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.690+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.690+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10521611571183864634', time=2024-03-09T16:57:24.400833+0000, chain.objs.size()=1 2024-03-09T15:02:48.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.690+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__shadow_.TQMvw7qNyDjjXm-VpwD2yiydg1I_jm9_0 2024-03-09T15:02:48.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.694+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.698+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T15:02:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.702+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.702+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.706+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.706+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16569362777449555509', time=2024-03-09T16:55:08.380495+0000, chain.objs.size()=5 2024-03-09T15:02:48.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.706+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.33__multipart_key1.2~4sLScN7r62_RbKjhTFw4OGFHKihk8at.1 2024-03-09T15:02:48.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.706+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.33__multipart_key1.2~4sLScN7r62_RbKjhTFw4OGFHKihk8at.2 2024-03-09T15:02:48.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.706+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.33__multipart_key1.2~4sLScN7r62_RbKjhTFw4OGFHKihk8at.3 2024-03-09T15:02:48.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.706+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.33__multipart_key1.2~4sLScN7r62_RbKjhTFw4OGFHKihk8at.4 2024-03-09T15:02:48.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.706+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.33__multipart_key1.2~4sLScN7r62_RbKjhTFw4OGFHKihk8at.5 2024-03-09T15:02:48.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.710+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.714+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T15:02:48.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.718+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.718+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2024-03-09T15:02:48.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15867293506815012289', time=2024-03-09T16:56:14.297115+0000, chain.objs.size()=4 2024-03-09T15:02:48.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.35__shadow_.19gdGt-kLGxF_KdTkmjYDWHZN7Rt4Hb_0 2024-03-09T15:02:48.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.35__shadow_.19gdGt-kLGxF_KdTkmjYDWHZN7Rt4Hb_1 2024-03-09T15:02:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.35__shadow_.19gdGt-kLGxF_KdTkmjYDWHZN7Rt4Hb_2 2024-03-09T15:02:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.35__shadow_.19gdGt-kLGxF_KdTkmjYDWHZN7Rt4Hb_3 2024-03-09T15:02:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.237023183420381005', time=2024-03-09T17:01:09.048975+0000, chain.objs.size()=11 2024-03-09T15:02:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~V8HeKBrp5Dtu3eNTXayzpx0SleuP3D0.1 2024-03-09T15:02:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~V8HeKBrp5Dtu3eNTXayzpx0SleuP3D0.1_1 2024-03-09T15:02:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~V8HeKBrp5Dtu3eNTXayzpx0SleuP3D0.2 2024-03-09T15:02:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~V8HeKBrp5Dtu3eNTXayzpx0SleuP3D0.2_1 2024-03-09T15:02:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~V8HeKBrp5Dtu3eNTXayzpx0SleuP3D0.3 2024-03-09T15:02:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~V8HeKBrp5Dtu3eNTXayzpx0SleuP3D0.3_1 2024-03-09T15:02:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~V8HeKBrp5Dtu3eNTXayzpx0SleuP3D0.4 2024-03-09T15:02:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.722+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~V8HeKBrp5Dtu3eNTXayzpx0SleuP3D0.4_1 2024-03-09T15:02:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.734+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~V8HeKBrp5Dtu3eNTXayzpx0SleuP3D0.5 2024-03-09T15:02:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.738+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~V8HeKBrp5Dtu3eNTXayzpx0SleuP3D0.5_1 2024-03-09T15:02:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.738+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~V8HeKBrp5Dtu3eNTXayzpx0SleuP3D0.6 2024-03-09T15:02:48.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.742+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.750+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T15:02:48.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2024-03-09T15:02:48.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5562067865184485775', time=2024-03-09T17:00:40.747617+0000, chain.objs.size()=11 2024-03-09T15:02:48.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~9orJaI0WfJvt5gvu7gcQnL321TaKyxq.1 2024-03-09T15:02:48.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~9orJaI0WfJvt5gvu7gcQnL321TaKyxq.1_1 2024-03-09T15:02:48.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~9orJaI0WfJvt5gvu7gcQnL321TaKyxq.2 2024-03-09T15:02:48.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~9orJaI0WfJvt5gvu7gcQnL321TaKyxq.2_1 2024-03-09T15:02:48.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~9orJaI0WfJvt5gvu7gcQnL321TaKyxq.3 2024-03-09T15:02:48.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~9orJaI0WfJvt5gvu7gcQnL321TaKyxq.3_1 2024-03-09T15:02:48.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~9orJaI0WfJvt5gvu7gcQnL321TaKyxq.4 2024-03-09T15:02:48.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~9orJaI0WfJvt5gvu7gcQnL321TaKyxq.4_1 2024-03-09T15:02:48.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~9orJaI0WfJvt5gvu7gcQnL321TaKyxq.5 2024-03-09T15:02:48.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__shadow_key1.2~9orJaI0WfJvt5gvu7gcQnL321TaKyxq.5_1 2024-03-09T15:02:48.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.44__multipart_key1.2~9orJaI0WfJvt5gvu7gcQnL321TaKyxq.6 2024-03-09T15:02:48.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12547846985907959762', time=2024-03-09T17:01:27.771756+0000, chain.objs.size()=5 2024-03-09T15:02:48.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.754+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.45__multipart_key1.2~Te4b4jLUcwAv--cZWungNbX_1rH8aw_.1 2024-03-09T15:02:48.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.770+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.45__multipart_key1.2~Te4b4jLUcwAv--cZWungNbX_1rH8aw_.2 2024-03-09T15:02:48.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.770+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.45__multipart_key1.2~Te4b4jLUcwAv--cZWungNbX_1rH8aw_.3 2024-03-09T15:02:48.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.770+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.45__multipart_key1.2~Te4b4jLUcwAv--cZWungNbX_1rH8aw_.4 2024-03-09T15:02:48.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.770+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.45__multipart_key1.2~Te4b4jLUcwAv--cZWungNbX_1rH8aw_.5 2024-03-09T15:02:48.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.770+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18102996412092629715', time=2024-03-09T17:02:13.708012+0000, chain.objs.size()=1 2024-03-09T15:02:48.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.770+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.51__shadow_.xGOo7cEXutElR6fEyZVZ6q-NuQ2zjpV_0 2024-03-09T15:02:48.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.782+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.790+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T15:02:48.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.794+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.794+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.794+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.794+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15139676902917377228', time=2024-03-09T17:02:11.839252+0000, chain.objs.size()=1 2024-03-09T15:02:48.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.794+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.49__shadow_.z0lD_DBX7IBawkEl_hD5a2G6MOIp0LL_0 2024-03-09T15:02:48.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.798+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.802+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T15:02:48.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.806+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.810+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.810+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.814+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T15:02:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.818+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.818+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.818+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.818+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9562099628397850342', time=2024-03-09T16:55:48.865204+0000, chain.objs.size()=5 2024-03-09T15:02:48.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.818+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~GnTeokRM71GANjP9TyfOzLvCQLDoDe-.1 2024-03-09T15:02:48.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.818+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~GnTeokRM71GANjP9TyfOzLvCQLDoDe-.2 2024-03-09T15:02:48.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.818+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~GnTeokRM71GANjP9TyfOzLvCQLDoDe-.3 2024-03-09T15:02:48.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.818+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~GnTeokRM71GANjP9TyfOzLvCQLDoDe-.4 2024-03-09T15:02:48.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.818+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~GnTeokRM71GANjP9TyfOzLvCQLDoDe-.5 2024-03-09T15:02:48.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.826+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.834+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T15:02:48.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.838+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.842+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.842+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.842+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T15:02:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.846+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.850+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.850+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2024-03-09T15:02:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.850+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18037574930370915192', time=2024-03-09T16:55:43.052150+0000, chain.objs.size()=5 2024-03-09T15:02:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.850+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~Jxhda87j0RCiiIboRQ4HD7OolG6Tlqf.1 2024-03-09T15:02:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.850+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~Jxhda87j0RCiiIboRQ4HD7OolG6Tlqf.2 2024-03-09T15:02:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.850+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~Jxhda87j0RCiiIboRQ4HD7OolG6Tlqf.3 2024-03-09T15:02:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.850+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~Jxhda87j0RCiiIboRQ4HD7OolG6Tlqf.4 2024-03-09T15:02:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.850+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~Jxhda87j0RCiiIboRQ4HD7OolG6Tlqf.5 2024-03-09T15:02:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.850+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1605102853453385736', time=2024-03-09T16:57:24.294991+0000, chain.objs.size()=1 2024-03-09T15:02:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.850+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.42__shadow_.5SF1OYK7G4MdbmKnNaGlVcxgXqQ79_U_0 2024-03-09T15:02:48.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.858+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.862+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T15:02:48.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.866+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.870+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.870+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.874+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T15:02:48.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.878+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.878+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.882+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.882+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T15:02:48.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.886+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.890+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.890+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.890+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.646374610892737870', time=2024-03-09T17:02:12.753089+0000, chain.objs.size()=1 2024-03-09T15:02:48.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.890+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.50__shadow_.CRmnECna67wOJCMc5c_S5mh2sl8ikwZ_0 2024-03-09T15:02:48.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.894+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.898+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T15:02:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.902+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.902+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.906+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.906+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T15:02:48.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.910+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.914+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.914+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:02:48.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.914+0000 7ff5d69dc980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14828658814666274680', time=2024-03-09T16:55:54.180536+0000, chain.objs.size()=5 2024-03-09T15:02:48.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.914+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~JuwWNNwViLa7tmYWGHz0MuLnioVYap8.1 2024-03-09T15:02:48.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.914+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~JuwWNNwViLa7tmYWGHz0MuLnioVYap8.2 2024-03-09T15:02:48.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.914+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~JuwWNNwViLa7tmYWGHz0MuLnioVYap8.3 2024-03-09T15:02:48.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.914+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~JuwWNNwViLa7tmYWGHz0MuLnioVYap8.4 2024-03-09T15:02:48.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.914+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.36__multipart_key1.2~JuwWNNwViLa7tmYWGHz0MuLnioVYap8.5 2024-03-09T15:02:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.926+0000 7ff5d69dc980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:02:48.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.934+0000 7ff5d69dc980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T15:02:48.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.934+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.938+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:48.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.938+0000 7ff5d69dc980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:48.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.942+0000 7ff48afdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:02:48.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.950+0000 7ff5d69dc980 20 remove_watcher() i=0 2024-03-09T15:02:48.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.950+0000 7ff5d69dc980 2 removed watcher, disabling cache 2024-03-09T15:02:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.950+0000 7ff5d69dc980 20 remove_watcher() i=1 2024-03-09T15:02:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.954+0000 7ff5d69dc980 20 remove_watcher() i=2 2024-03-09T15:02:48.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.958+0000 7ff5d69dc980 20 remove_watcher() i=3 2024-03-09T15:02:48.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.958+0000 7ff5d69dc980 20 remove_watcher() i=4 2024-03-09T15:02:48.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.962+0000 7ff5d69dc980 20 remove_watcher() i=5 2024-03-09T15:02:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.962+0000 7ff5d69dc980 20 remove_watcher() i=6 2024-03-09T15:02:48.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:48.966+0000 7ff5d69dc980 20 remove_watcher() i=7 2024-03-09T15:02:48.990 DEBUG:teuthology.orchestra.run.smithi001:> radosgw-admin gc process --include-all 2024-03-09T15:02:49.064 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:02:49.064 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:02:49.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.106+0000 7f132319f980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.106+0000 7f132319f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.106+0000 7f132319f980 20 realm 2024-03-09T15:02:49.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.106+0000 7f132319f980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.106+0000 7f132319f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.106+0000 7f132319f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:02:49.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.106+0000 7f132319f980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.106+0000 7f132319f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.106+0000 7f132319f980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.110+0000 7f132319f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:02:49.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.110+0000 7f132319f980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.110+0000 7f132319f980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:02:49.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.110+0000 7f132319f980 20 searching for the correct realm 2024-03-09T15:02:49.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.118+0000 7f13214b8640 0 -- 172.21.15.1:0/1853731883 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x555708e75d50 msgr2=0x555708e781c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:02:49.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.122+0000 7f1320cb7640 0 -- 172.21.15.1:0/1853731883 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f130c0536e0 msgr2=0x7f130c0692e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:02:49.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.130+0000 7f132319f980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:02:49.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.130+0000 7f132319f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:02:49.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.130+0000 7f132319f980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:02:49.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.130+0000 7f132319f980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:02:49.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.130+0000 7f132319f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:02:49.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.130+0000 7f132319f980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:02:49.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.130+0000 7f132319f980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.130+0000 7f132319f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.130+0000 7f132319f980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:02:49.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:02:49.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 zone default found 2024-03-09T15:02:49.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 4 Realm: () 2024-03-09T15:02:49.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:02:49.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:02:49.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:02:49.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 zonegroup default 2024-03-09T15:02:49.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 started sync module instance, tier type = 2024-03-09T15:02:49.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.134+0000 7f132319f980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:02:49.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.150+0000 7f132319f980 20 add_watcher() i=0 2024-03-09T15:02:49.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.150+0000 7f132319f980 20 add_watcher() i=1 2024-03-09T15:02:49.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.150+0000 7f132319f980 20 add_watcher() i=2 2024-03-09T15:02:49.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.150+0000 7f132319f980 20 add_watcher() i=3 2024-03-09T15:02:49.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.150+0000 7f132319f980 20 add_watcher() i=4 2024-03-09T15:02:49.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.150+0000 7f132319f980 20 add_watcher() i=5 2024-03-09T15:02:49.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.150+0000 7f132319f980 20 add_watcher() i=6 2024-03-09T15:02:49.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.150+0000 7f132319f980 20 add_watcher() i=7 2024-03-09T15:02:49.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.150+0000 7f132319f980 2 all 8 watchers are set, enabling cache 2024-03-09T15:02:49.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.154+0000 7f11d6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:02:49.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.154+0000 7f132319f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:02:49.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.154+0000 7f132319f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:02:49.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.238+0000 7f11d5fe3640 20 reqs_thread_entry: start 2024-03-09T15:02:49.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.238+0000 7f132319f980 20 init_complete bucket index max shards: 1 2024-03-09T15:02:49.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.238+0000 7f132319f980 20 Filter name: none 2024-03-09T15:02:49.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.238+0000 7f11ceffd640 20 reqs_thread_entry: start 2024-03-09T15:02:49.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.238+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T15:02:49.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.242+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.242+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.242+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.246+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T15:02:49.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.246+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.250+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.250+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.254+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T15:02:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.254+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.258+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.258+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.262+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T15:02:49.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.266+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.266+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.266+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.270+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T15:02:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.274+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.274+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.274+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.278+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T15:02:49.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.278+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.282+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.282+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.286+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T15:02:49.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.290+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.290+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.290+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.294+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T15:02:49.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.294+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.298+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.298+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.298+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T15:02:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.302+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.306+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.306+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.310+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T15:02:49.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.314+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.314+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.314+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.318+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T15:02:49.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.322+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.322+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.322+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.326+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T15:02:49.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.330+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.330+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.334+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.334+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T15:02:49.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.338+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.342+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.342+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.346+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T15:02:49.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.350+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.350+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.350+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.354+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T15:02:49.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.358+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.362+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.362+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.366+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T15:02:49.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.370+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.370+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.370+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.374+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T15:02:49.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.378+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.378+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.378+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.382+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T15:02:49.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.386+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.390+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.390+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.394+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T15:02:49.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.398+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.398+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.398+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.402+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T15:02:49.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.406+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.406+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.410+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.410+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T15:02:49.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.414+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.418+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.418+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.422+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T15:02:49.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.422+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.426+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.426+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.430+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T15:02:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.434+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.434+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.438+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.442+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T15:02:49.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.442+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.446+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.446+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.450+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T15:02:49.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.450+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.454+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.454+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.458+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T15:02:49.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.462+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.462+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.462+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.466+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T15:02:49.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.470+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.470+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.474+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.474+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T15:02:49.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.478+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.478+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.482+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.482+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T15:02:49.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.486+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.490+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.490+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.494+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T15:02:49.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.498+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.498+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.498+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.502+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T15:02:49.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.506+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.510+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.510+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.514+0000 7f132319f980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T15:02:49.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.518+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.518+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:49.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.518+0000 7f132319f980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:49.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.526+0000 7f11d6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:02:49.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.530+0000 7f132319f980 20 remove_watcher() i=0 2024-03-09T15:02:49.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.530+0000 7f132319f980 2 removed watcher, disabling cache 2024-03-09T15:02:49.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.534+0000 7f132319f980 20 remove_watcher() i=1 2024-03-09T15:02:49.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.538+0000 7f132319f980 20 remove_watcher() i=2 2024-03-09T15:02:49.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.538+0000 7f132319f980 20 remove_watcher() i=3 2024-03-09T15:02:49.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.542+0000 7f132319f980 20 remove_watcher() i=4 2024-03-09T15:02:49.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.542+0000 7f132319f980 20 remove_watcher() i=5 2024-03-09T15:02:49.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.546+0000 7f132319f980 20 remove_watcher() i=6 2024-03-09T15:02:49.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.550+0000 7f132319f980 20 remove_watcher() i=7 2024-03-09T15:02:49.574 DEBUG:teuthology.orchestra.run.smithi001:> radosgw-admin gc process --include-all 2024-03-09T15:02:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:02:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:02:49.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.690+0000 7f85cb178980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.690+0000 7f85cb178980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.690+0000 7f85cb178980 20 realm 2024-03-09T15:02:49.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.690+0000 7f85cb178980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.690+0000 7f85cb178980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.690+0000 7f85cb178980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:02:49.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.690+0000 7f85cb178980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.690+0000 7f85cb178980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.690+0000 7f85cb178980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.694+0000 7f85cb178980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:02:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.694+0000 7f85cb178980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.694+0000 7f85cb178980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:02:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.694+0000 7f85cb178980 20 searching for the correct realm 2024-03-09T15:02:49.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.698+0000 7f85c9491640 0 -- 172.21.15.1:0/3683409320 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f85b003c060 msgr2=0x7f85b0083860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:02:49.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.706+0000 7f85c8c90640 0 -- 172.21.15.1:0/3683409320 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f85b0059820 msgr2=0x7f85b007a720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T15:02:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.922+0000 7f85cb178980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:02:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.922+0000 7f85cb178980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:02:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.922+0000 7f85cb178980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:02:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.922+0000 7f85cb178980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:02:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.922+0000 7f85cb178980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:02:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.922+0000 7f85cb178980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:02:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.922+0000 7f85cb178980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.922+0000 7f85cb178980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.922+0000 7f85cb178980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:02:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:02:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 zone default found 2024-03-09T15:02:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 4 Realm: () 2024-03-09T15:02:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:02:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:02:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:02:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 zonegroup default 2024-03-09T15:02:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 rados->read ofs=0 len=0 2024-03-09T15:02:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 started sync module instance, tier type = 2024-03-09T15:02:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.926+0000 7f85cb178980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:02:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.950+0000 7f85cb178980 20 add_watcher() i=0 2024-03-09T15:02:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.950+0000 7f85cb178980 20 add_watcher() i=1 2024-03-09T15:02:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.950+0000 7f85cb178980 20 add_watcher() i=2 2024-03-09T15:02:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.950+0000 7f85cb178980 20 add_watcher() i=3 2024-03-09T15:02:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.950+0000 7f85cb178980 20 add_watcher() i=4 2024-03-09T15:02:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.950+0000 7f85cb178980 20 add_watcher() i=5 2024-03-09T15:02:49.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.950+0000 7f85cb178980 20 add_watcher() i=6 2024-03-09T15:02:49.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.954+0000 7f85cb178980 20 add_watcher() i=7 2024-03-09T15:02:49.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.954+0000 7f85cb178980 2 all 8 watchers are set, enabling cache 2024-03-09T15:02:49.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.958+0000 7f847efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:02:49.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.958+0000 7f85cb178980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:02:49.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:49.958+0000 7f85cb178980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:02:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.054+0000 7f847dfe3640 20 reqs_thread_entry: start 2024-03-09T15:02:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.054+0000 7f85cb178980 20 init_complete bucket index max shards: 1 2024-03-09T15:02:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.054+0000 7f85cb178980 20 Filter name: none 2024-03-09T15:02:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.054+0000 7f83ef7fe640 20 reqs_thread_entry: start 2024-03-09T15:02:50.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.054+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T15:02:50.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.058+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.058+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.058+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.058+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T15:02:50.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.062+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.062+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.062+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.066+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T15:02:50.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.070+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.070+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.070+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.074+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T15:02:50.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.078+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.078+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.078+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.082+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T15:02:50.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.086+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.086+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.086+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.086+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T15:02:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.090+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.094+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.094+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.094+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T15:02:50.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.098+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.098+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.098+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.102+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T15:02:50.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.106+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.106+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.106+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.110+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T15:02:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.110+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.114+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.114+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.114+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T15:02:50.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.118+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.122+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.122+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.122+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T15:02:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.126+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.126+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.126+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.130+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T15:02:50.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.134+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.134+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.138+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.138+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T15:02:50.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.142+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.142+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.146+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.146+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T15:02:50.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.150+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.150+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.154+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.154+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T15:02:50.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.162+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.162+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.166+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.170+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T15:02:50.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.170+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.174+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.174+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.178+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T15:02:50.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.182+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.186+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.186+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.190+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T15:02:50.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.194+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.198+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.198+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.202+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T15:02:50.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.206+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.206+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.206+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.210+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T15:02:50.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.214+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.218+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.218+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.222+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T15:02:50.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.226+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.226+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.230+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.230+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T15:02:50.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.234+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.238+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.238+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.242+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T15:02:50.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.246+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.246+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.246+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.250+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T15:02:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.254+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.254+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.258+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.258+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T15:02:50.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.262+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.266+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.266+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.270+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T15:02:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.270+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.274+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.274+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.278+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T15:02:50.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.282+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.282+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.286+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.286+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T15:02:50.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.290+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.294+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.294+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.298+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T15:02:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.302+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.302+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.306+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.306+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T15:02:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.310+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.314+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.314+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.318+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T15:02:50.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.322+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.322+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.326+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.326+0000 7f85cb178980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T15:02:50.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.330+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.334+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.334+0000 7f85cb178980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.342+0000 7f847efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:02:50.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.346+0000 7f85cb178980 20 remove_watcher() i=0 2024-03-09T15:02:50.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.346+0000 7f85cb178980 2 removed watcher, disabling cache 2024-03-09T15:02:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.350+0000 7f85cb178980 20 remove_watcher() i=1 2024-03-09T15:02:50.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.354+0000 7f85cb178980 20 remove_watcher() i=2 2024-03-09T15:02:50.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.354+0000 7f85cb178980 20 remove_watcher() i=3 2024-03-09T15:02:50.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.358+0000 7f85cb178980 20 remove_watcher() i=4 2024-03-09T15:02:50.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.362+0000 7f85cb178980 20 remove_watcher() i=5 2024-03-09T15:02:50.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.362+0000 7f85cb178980 20 remove_watcher() i=6 2024-03-09T15:02:50.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.366+0000 7f85cb178980 20 remove_watcher() i=7 2024-03-09T15:02:50.395 DEBUG:teuthology.orchestra.run.smithi001:> radosgw-admin gc process --include-all 2024-03-09T15:02:50.472 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:02:50.473 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:02:50.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.518+0000 7f4d9dd96980 20 rados->read ofs=0 len=0 2024-03-09T15:02:50.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.518+0000 7f4d9dd96980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.518+0000 7f4d9dd96980 20 realm 2024-03-09T15:02:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.518+0000 7f4d9dd96980 20 rados->read ofs=0 len=0 2024-03-09T15:02:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.518+0000 7f4d9dd96980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.518+0000 7f4d9dd96980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:02:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.518+0000 7f4d9dd96980 20 rados->read ofs=0 len=0 2024-03-09T15:02:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.518+0000 7f4d9dd96980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.518+0000 7f4d9dd96980 20 rados->read ofs=0 len=0 2024-03-09T15:02:50.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.522+0000 7f4d9dd96980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:02:50.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.522+0000 7f4d9dd96980 20 rados->read ofs=0 len=0 2024-03-09T15:02:50.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.522+0000 7f4d9dd96980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:02:50.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.522+0000 7f4d9dd96980 20 searching for the correct realm 2024-03-09T15:02:50.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.530+0000 7f4d9b0ad640 0 -- 172.21.15.1:0/119602649 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x55562a822ae0 msgr2=0x55562a824fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:02:50.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.534+0000 7f4d9b8ae640 0 -- 172.21.15.1:0/119602649 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f4d9006eba0 msgr2=0x7f4d90074220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:02:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:02:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:02:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:02:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:02:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:02:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:02:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 rados->read ofs=0 len=0 2024-03-09T15:02:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 rados->read ofs=0 len=0 2024-03-09T15:02:50.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:02:50.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 rados->read ofs=0 len=0 2024-03-09T15:02:50.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:02:50.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 zone default found 2024-03-09T15:02:50.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 4 Realm: () 2024-03-09T15:02:50.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:02:50.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:02:50.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:02:50.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 zonegroup default 2024-03-09T15:02:50.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 rados->read ofs=0 len=0 2024-03-09T15:02:50.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:50.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 rados->read ofs=0 len=0 2024-03-09T15:02:50.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:50.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 rados->read ofs=0 len=0 2024-03-09T15:02:50.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:02:50.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 started sync module instance, tier type = 2024-03-09T15:02:50.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.546+0000 7f4d9dd96980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:02:50.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.562+0000 7f4d9dd96980 20 add_watcher() i=0 2024-03-09T15:02:50.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.562+0000 7f4d9dd96980 20 add_watcher() i=1 2024-03-09T15:02:50.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.562+0000 7f4d9dd96980 20 add_watcher() i=2 2024-03-09T15:02:50.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.562+0000 7f4d9dd96980 20 add_watcher() i=3 2024-03-09T15:02:50.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.562+0000 7f4d9dd96980 20 add_watcher() i=4 2024-03-09T15:02:50.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.562+0000 7f4d9dd96980 20 add_watcher() i=5 2024-03-09T15:02:50.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.562+0000 7f4d9dd96980 20 add_watcher() i=6 2024-03-09T15:02:50.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.566+0000 7f4d9dd96980 20 add_watcher() i=7 2024-03-09T15:02:50.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.566+0000 7f4d9dd96980 2 all 8 watchers are set, enabling cache 2024-03-09T15:02:50.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.566+0000 7f4c51fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:02:50.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.570+0000 7f4d9dd96980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:02:50.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.570+0000 7f4d9dd96980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:02:50.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.650+0000 7f4c50fe1640 20 reqs_thread_entry: start 2024-03-09T15:02:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.650+0000 7f4d9dd96980 20 init_complete bucket index max shards: 1 2024-03-09T15:02:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.650+0000 7f4d9dd96980 20 Filter name: none 2024-03-09T15:02:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.650+0000 7f4bca7fc640 20 reqs_thread_entry: start 2024-03-09T15:02:50.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.650+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T15:02:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.654+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.654+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.654+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.658+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T15:02:50.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.658+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.662+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.662+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.666+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T15:02:50.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.666+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.666+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.670+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.670+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T15:02:50.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.674+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.674+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.674+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.678+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T15:02:50.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.678+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.678+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.682+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.682+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T15:02:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.686+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.690+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.690+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.694+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T15:02:50.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.694+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.698+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.698+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.698+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T15:02:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.702+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.706+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.706+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.710+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T15:02:50.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.710+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.710+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.710+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.714+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T15:02:50.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.714+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.714+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.714+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.718+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T15:02:50.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.722+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.722+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.722+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.726+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T15:02:50.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.726+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.730+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.730+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.730+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T15:02:50.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.734+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.734+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.734+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.738+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T15:02:50.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.738+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.738+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.742+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.742+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T15:02:50.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.746+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.746+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.750+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.750+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T15:02:50.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.754+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.754+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.754+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.758+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T15:02:50.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.762+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.762+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.766+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.766+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T15:02:50.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.770+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.770+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.774+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.774+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T15:02:50.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.778+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.782+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.782+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.786+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T15:02:50.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.790+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.790+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.790+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.794+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T15:02:50.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.798+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.798+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.802+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.802+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T15:02:50.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.806+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.810+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.810+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.814+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T15:02:50.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.818+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.818+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.818+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.822+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T15:02:50.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.826+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.826+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.826+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.830+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T15:02:50.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.834+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.834+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.838+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.838+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T15:02:50.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.842+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.846+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.846+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.850+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T15:02:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.854+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.854+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.854+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.858+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T15:02:50.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.862+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.862+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.862+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.866+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T15:02:50.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.870+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.870+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.874+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.874+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T15:02:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.878+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.882+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.882+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.886+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T15:02:50.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.890+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.890+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.890+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.894+0000 7f4d9dd96980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T15:02:50.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.898+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.898+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:02:50.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.898+0000 7f4d9dd96980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:02:50.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.906+0000 7f4c51fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:02:50.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.910+0000 7f4d9dd96980 20 remove_watcher() i=0 2024-03-09T15:02:50.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.910+0000 7f4d9dd96980 2 removed watcher, disabling cache 2024-03-09T15:02:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.914+0000 7f4d9dd96980 20 remove_watcher() i=1 2024-03-09T15:02:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.918+0000 7f4d9dd96980 20 remove_watcher() i=2 2024-03-09T15:02:50.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.918+0000 7f4d9dd96980 20 remove_watcher() i=3 2024-03-09T15:02:50.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.922+0000 7f4d9dd96980 20 remove_watcher() i=4 2024-03-09T15:02:50.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.926+0000 7f4d9dd96980 20 remove_watcher() i=5 2024-03-09T15:02:50.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.926+0000 7f4d9dd96980 20 remove_watcher() i=6 2024-03-09T15:02:50.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:02:50.930+0000 7f4d9dd96980 20 remove_watcher() i=7 2024-03-09T15:02:50.959 DEBUG:teuthology.orchestra.run.smithi001:> cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests BucketTest 2024-03-09T15:02:51.689 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :clean 2024-03-09T15:02:52.588 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :compileJava 2024-03-09T15:02:52.588 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :processResources 2024-03-09T15:02:52.588 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :classes 2024-03-09T15:02:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:53.187 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :compileTestJava 2024-03-09T15:02:53.187 INFO:teuthology.orchestra.run.smithi001.stderr:Note: /home/ubuntu/cephtest/s3-tests-java/src/test/java/BucketTest.java uses or overrides a deprecated API. 2024-03-09T15:02:53.188 INFO:teuthology.orchestra.run.smithi001.stderr:Note: Recompile with -Xlint:deprecation for details. 2024-03-09T15:02:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:53.188 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :processTestResources NO-SOURCE 2024-03-09T15:02:53.188 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :testClasses 2024-03-09T15:02:55.188 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:55.188 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :test 2024-03-09T15:02:55.189 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:55.189 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests STANDARD_OUT 2024-03-09T15:02:55.189 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09 15:02:54,548 [Test worker] WARN com.amazonaws.util.Base64 - JAXB is unavailable. Will fallback to SDK implementation which may be less performant 2024-03-09T15:02:55.189 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:55.189 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectEmpty STARTED 2024-03-09T15:02:55.189 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:55.189 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectEmpty PASSED 2024-03-09T15:02:55.587 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:55.587 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectMismatch STARTED 2024-03-09T15:02:55.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:55.687 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectMismatch PASSED 2024-03-09T15:02:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:56.187 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectUnreadable STARTED 2024-03-09T15:02:56.290 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:56.290 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectUnreadable PASSED 2024-03-09T15:02:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:56.787 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketCreateReadDelete STARTED 2024-03-09T15:02:57.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:57.087 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketCreateReadDelete PASSED 2024-03-09T15:02:57.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:57.187 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNonEmpty STARTED 2024-03-09T15:02:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:57.487 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNonEmpty PASSED 2024-03-09T15:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:58.087 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNotExist STARTED 2024-03-09T15:02:58.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:58.187 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNotExist PASSED 2024-03-09T15:02:58.286 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:58.287 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketListDistinct STARTED 2024-03-09T15:02:58.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:58.687 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketListDistinct PASSED 2024-03-09T15:02:59.587 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:59.587 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketListEmpty STARTED 2024-03-09T15:02:59.787 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:02:59.787 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketListEmpty PASSED 2024-03-09T15:03:00.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:00.187 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketNotExist STARTED 2024-03-09T15:03:00.287 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:00.287 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Bucket tests > BucketTest.testBucketNotExist PASSED 2024-03-09T15:03:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:00.456 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------- 2024-03-09T15:03:00.456 INFO:teuthology.orchestra.run.smithi001.stdout:| Results: SUCCESS (9 tests, 9 successes, 0 failures, 0 skipped) | 2024-03-09T15:03:00.456 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------- 2024-03-09T15:03:00.456 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:00.457 INFO:teuthology.orchestra.run.smithi001.stdout:Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. 2024-03-09T15:03:00.457 INFO:teuthology.orchestra.run.smithi001.stdout:Use '--warning-mode all' to show the individual deprecation warnings. 2024-03-09T15:03:00.457 INFO:teuthology.orchestra.run.smithi001.stdout:See https://docs.gradle.org/6.0.1/userguide/command_line_interface.html#sec:command_line_warnings 2024-03-09T15:03:00.457 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:00.457 INFO:teuthology.orchestra.run.smithi001.stdout:BUILD SUCCESSFUL in 9s 2024-03-09T15:03:00.457 INFO:teuthology.orchestra.run.smithi001.stdout:5 actionable tasks: 5 executed 2024-03-09T15:03:00.469 DEBUG:teuthology.orchestra.run.smithi001:> radosgw-admin gc process --include-all 2024-03-09T15:03:00.496 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:03:00.496 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:03:00.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.554+0000 7f4e14037980 20 rados->read ofs=0 len=0 2024-03-09T15:03:00.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.554+0000 7f4e14037980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:00.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.554+0000 7f4e14037980 20 realm 2024-03-09T15:03:00.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.554+0000 7f4e14037980 20 rados->read ofs=0 len=0 2024-03-09T15:03:00.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.554+0000 7f4e14037980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:00.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.554+0000 7f4e14037980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:03:00.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.554+0000 7f4e14037980 20 rados->read ofs=0 len=0 2024-03-09T15:03:00.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.554+0000 7f4e14037980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:00.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.554+0000 7f4e14037980 20 rados->read ofs=0 len=0 2024-03-09T15:03:00.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.558+0000 7f4e14037980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:03:00.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.558+0000 7f4e14037980 20 rados->read ofs=0 len=0 2024-03-09T15:03:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.558+0000 7f4e14037980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:03:00.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.558+0000 7f4e14037980 20 searching for the correct realm 2024-03-09T15:03:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.566+0000 7f4e1134e640 0 -- 172.21.15.1:0/2935378945 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55b4c02c79a0 msgr2=0x55b4c02c9e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:03:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.570+0000 7f4e1134e640 0 -- 172.21.15.1:0/2935378945 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f4e04008040 msgr2=0x7f4e0405b320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T15:03:00.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:03:00.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:03:00.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:03:00.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:03:00.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:03:00.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:03:00.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 rados->read ofs=0 len=0 2024-03-09T15:03:00.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:00.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 rados->read ofs=0 len=0 2024-03-09T15:03:00.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:03:00.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 rados->read ofs=0 len=0 2024-03-09T15:03:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:03:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 zone default found 2024-03-09T15:03:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 4 Realm: () 2024-03-09T15:03:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:03:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:03:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:03:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 zonegroup default 2024-03-09T15:03:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.582+0000 7f4e14037980 20 rados->read ofs=0 len=0 2024-03-09T15:03:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.586+0000 7f4e14037980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.586+0000 7f4e14037980 20 rados->read ofs=0 len=0 2024-03-09T15:03:00.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.586+0000 7f4e14037980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:00.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.586+0000 7f4e14037980 20 rados->read ofs=0 len=0 2024-03-09T15:03:00.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.586+0000 7f4e14037980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:00.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.586+0000 7f4e14037980 20 started sync module instance, tier type = 2024-03-09T15:03:00.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.586+0000 7f4e14037980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:03:00.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.602+0000 7f4e14037980 20 add_watcher() i=0 2024-03-09T15:03:00.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.602+0000 7f4e14037980 20 add_watcher() i=1 2024-03-09T15:03:00.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.602+0000 7f4e14037980 20 add_watcher() i=2 2024-03-09T15:03:00.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.602+0000 7f4e14037980 20 add_watcher() i=3 2024-03-09T15:03:00.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.602+0000 7f4e14037980 20 add_watcher() i=4 2024-03-09T15:03:00.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.602+0000 7f4e14037980 20 add_watcher() i=5 2024-03-09T15:03:00.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.602+0000 7f4e14037980 20 add_watcher() i=6 2024-03-09T15:03:00.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.602+0000 7f4e14037980 20 add_watcher() i=7 2024-03-09T15:03:00.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.602+0000 7f4e14037980 2 all 8 watchers are set, enabling cache 2024-03-09T15:03:00.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.606+0000 7f4cc37fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:03:00.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.606+0000 7f4e14037980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:03:00.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.606+0000 7f4e14037980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:03:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.678+0000 7f4cc27fc640 20 reqs_thread_entry: start 2024-03-09T15:03:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.678+0000 7f4e14037980 20 init_complete bucket index max shards: 1 2024-03-09T15:03:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.678+0000 7f4e14037980 20 Filter name: none 2024-03-09T15:03:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.678+0000 7f4c33fff640 20 reqs_thread_entry: start 2024-03-09T15:03:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.678+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T15:03:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.682+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.686+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.686+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.686+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T15:03:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.690+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.690+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.690+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.690+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T15:03:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.694+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.694+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.694+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.698+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T15:03:00.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.702+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.702+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.702+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.706+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T15:03:00.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.710+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.710+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.714+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.714+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T15:03:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.718+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.718+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.718+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.722+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T15:03:00.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.722+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.726+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.726+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.730+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T15:03:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.730+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.730+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.734+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.734+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T15:03:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.738+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.738+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.742+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.742+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T15:03:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.746+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.750+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.750+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.754+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T15:03:00.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.758+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.758+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.758+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.762+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T15:03:00.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.762+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.766+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.766+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.770+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T15:03:00.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.770+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.774+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.774+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.778+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T15:03:00.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.782+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.782+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.782+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.786+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T15:03:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.790+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.790+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.794+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.794+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T15:03:00.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.798+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.802+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.802+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.806+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T15:03:00.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.810+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.810+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.810+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.814+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T15:03:00.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.818+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.818+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.818+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.822+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T15:03:00.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.826+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.826+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.830+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.830+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T15:03:00.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.834+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.834+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.834+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.838+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T15:03:00.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.842+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.846+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.846+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.850+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T15:03:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.854+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.854+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.854+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.858+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T15:03:00.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.862+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.862+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.866+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.866+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T15:03:00.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.870+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.874+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.874+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.878+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T15:03:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.878+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.882+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.882+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.886+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T15:03:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.890+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.890+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.890+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.894+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T15:03:00.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.898+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.898+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.902+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2024-03-09T15:03:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.902+0000 7f4e14037980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17375410790184030085', time=2024-03-09T17:02:57.709166+0000, chain.objs.size()=1 2024-03-09T15:03:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.902+0000 7f4e14037980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.60__shadow_.UWCJe0qFrFjFTyARscctxBsseNK1NzE_0 2024-03-09T15:03:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.902+0000 7f4e14037980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10675759218134663196', time=2024-03-09T17:02:58.863102+0000, chain.objs.size()=1 2024-03-09T15:03:00.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.902+0000 7f4e14037980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.61__shadow_.nm8sZ9hSmOn3ZQyXgLRNMkdO9bwjGU__0 2024-03-09T15:03:00.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.902+0000 7f4e14037980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:03:00.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.910+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T15:03:00.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.914+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.914+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.918+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.918+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T15:03:00.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.922+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.926+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.926+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.926+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T15:03:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.938+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.938+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.938+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.942+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T15:03:00.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.946+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.946+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.950+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.950+0000 7f4e14037980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T15:03:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.958+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.958+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:00.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.962+0000 7f4e14037980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:00.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.966+0000 7f4cc37fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:03:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.970+0000 7f4e14037980 20 remove_watcher() i=0 2024-03-09T15:03:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.970+0000 7f4e14037980 2 removed watcher, disabling cache 2024-03-09T15:03:00.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.974+0000 7f4e14037980 20 remove_watcher() i=1 2024-03-09T15:03:00.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.978+0000 7f4e14037980 20 remove_watcher() i=2 2024-03-09T15:03:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.978+0000 7f4e14037980 20 remove_watcher() i=3 2024-03-09T15:03:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.982+0000 7f4e14037980 20 remove_watcher() i=4 2024-03-09T15:03:00.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.986+0000 7f4e14037980 20 remove_watcher() i=5 2024-03-09T15:03:00.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.986+0000 7f4e14037980 20 remove_watcher() i=6 2024-03-09T15:03:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:00.990+0000 7f4e14037980 20 remove_watcher() i=7 2024-03-09T15:03:01.019 DEBUG:teuthology.orchestra.run.smithi001:> radosgw-admin gc process --include-all 2024-03-09T15:03:01.090 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:03:01.091 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:03:01.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.134+0000 7f15b354c980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.134+0000 7f15b354c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:01.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.134+0000 7f15b354c980 20 realm 2024-03-09T15:03:01.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.134+0000 7f15b354c980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.134+0000 7f15b354c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:01.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.134+0000 7f15b354c980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:03:01.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.134+0000 7f15b354c980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.134+0000 7f15b354c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:01.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.134+0000 7f15b354c980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.138+0000 7f15b354c980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:03:01.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.138+0000 7f15b354c980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.138+0000 7f15b354c980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:03:01.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.138+0000 7f15b354c980 20 searching for the correct realm 2024-03-09T15:03:01.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.142+0000 7f15b1865640 0 -- 172.21.15.1:0/3849115319 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7f15a8010bd0 msgr2=0x7f15a801d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:03:01.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.150+0000 7f15b1064640 0 -- 172.21.15.1:0/3849115319 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f15a8040920 msgr2=0x7f15a8050400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T15:03:01.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.366+0000 7f15b354c980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:03:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.366+0000 7f15b354c980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:03:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.366+0000 7f15b354c980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:03:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.366+0000 7f15b354c980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:03:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.366+0000 7f15b354c980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:03:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.366+0000 7f15b354c980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:03:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.366+0000 7f15b354c980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:03:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:03:01.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 20 zone default found 2024-03-09T15:03:01.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 4 Realm: () 2024-03-09T15:03:01.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:03:01.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:03:01.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:03:01.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 20 zonegroup default 2024-03-09T15:03:01.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:01.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.370+0000 7f15b354c980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.374+0000 7f15b354c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:01.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.374+0000 7f15b354c980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.374+0000 7f15b354c980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.374+0000 7f15b354c980 20 started sync module instance, tier type = 2024-03-09T15:03:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.374+0000 7f15b354c980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:03:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.398+0000 7f15b354c980 20 add_watcher() i=0 2024-03-09T15:03:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.398+0000 7f15b354c980 20 add_watcher() i=1 2024-03-09T15:03:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.398+0000 7f15b354c980 20 add_watcher() i=2 2024-03-09T15:03:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.398+0000 7f15b354c980 20 add_watcher() i=3 2024-03-09T15:03:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.398+0000 7f15b354c980 20 add_watcher() i=4 2024-03-09T15:03:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.398+0000 7f15b354c980 20 add_watcher() i=5 2024-03-09T15:03:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.398+0000 7f15b354c980 20 add_watcher() i=6 2024-03-09T15:03:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.398+0000 7f15b354c980 20 add_watcher() i=7 2024-03-09T15:03:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.398+0000 7f15b354c980 2 all 8 watchers are set, enabling cache 2024-03-09T15:03:01.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.402+0000 7f14677e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:03:01.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.406+0000 7f15b354c980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:03:01.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.406+0000 7f15b354c980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:03:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.498+0000 7f14667e4640 20 reqs_thread_entry: start 2024-03-09T15:03:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.498+0000 7f15b354c980 20 init_complete bucket index max shards: 1 2024-03-09T15:03:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.498+0000 7f15b354c980 20 Filter name: none 2024-03-09T15:03:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.498+0000 7f13d7fff640 20 reqs_thread_entry: start 2024-03-09T15:03:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.498+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T15:03:01.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.502+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.502+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.506+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.506+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T15:03:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.510+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.510+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.510+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.514+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T15:03:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.514+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.514+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.518+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.518+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T15:03:01.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.522+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.522+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.522+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.526+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T15:03:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.530+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.530+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.534+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.534+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T15:03:01.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.538+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.538+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.538+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.542+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T15:03:01.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.546+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.546+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.550+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.550+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T15:03:01.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.554+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.558+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.558+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.558+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T15:03:01.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.562+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.566+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.566+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.570+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T15:03:01.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.574+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.574+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.578+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.582+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T15:03:01.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.582+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.586+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.586+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.590+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T15:03:01.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.594+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.594+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.594+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.598+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T15:03:01.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.602+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.602+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.602+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.606+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T15:03:01.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.610+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.610+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.610+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.614+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T15:03:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.618+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.618+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.622+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.622+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T15:03:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.626+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.630+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.630+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.638+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T15:03:01.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.642+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.642+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.646+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.646+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T15:03:01.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.650+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.654+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.654+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.658+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T15:03:01.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.662+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.662+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.666+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.666+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T15:03:01.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.670+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.674+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.674+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.674+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T15:03:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.678+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.682+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.682+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.686+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T15:03:01.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.690+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.690+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.694+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.694+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T15:03:01.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.698+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.702+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.702+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.706+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T15:03:01.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.706+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.710+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.710+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.714+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T15:03:01.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.718+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.718+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.722+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.722+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T15:03:01.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.726+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.726+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.726+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.730+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T15:03:01.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.734+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.734+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.738+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.738+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T15:03:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.742+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.746+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.746+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.750+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T15:03:01.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.754+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.758+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.758+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.762+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T15:03:01.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.766+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.766+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.766+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.770+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T15:03:01.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.774+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.774+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.778+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.778+0000 7f15b354c980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T15:03:01.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.782+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.786+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:01.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.786+0000 7f15b354c980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:01.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.790+0000 7f14677e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:03:01.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.794+0000 7f15b354c980 20 remove_watcher() i=0 2024-03-09T15:03:01.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.794+0000 7f15b354c980 2 removed watcher, disabling cache 2024-03-09T15:03:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.802+0000 7f15b354c980 20 remove_watcher() i=1 2024-03-09T15:03:01.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.802+0000 7f15b354c980 20 remove_watcher() i=2 2024-03-09T15:03:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.806+0000 7f15b354c980 20 remove_watcher() i=3 2024-03-09T15:03:01.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.810+0000 7f15b354c980 20 remove_watcher() i=4 2024-03-09T15:03:01.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.810+0000 7f15b354c980 20 remove_watcher() i=5 2024-03-09T15:03:01.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.814+0000 7f15b354c980 20 remove_watcher() i=6 2024-03-09T15:03:01.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.814+0000 7f15b354c980 20 remove_watcher() i=7 2024-03-09T15:03:01.842 DEBUG:teuthology.orchestra.run.smithi001:> radosgw-admin gc process --include-all 2024-03-09T15:03:01.915 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:03:01.915 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:03:01.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.962+0000 7fd040cc0980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.962+0000 7fd040cc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:01.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.962+0000 7fd040cc0980 20 realm 2024-03-09T15:03:01.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.962+0000 7fd040cc0980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.962+0000 7fd040cc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:01.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.962+0000 7fd040cc0980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:03:01.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.962+0000 7fd040cc0980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.962+0000 7fd040cc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:01.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.962+0000 7fd040cc0980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.966+0000 7fd040cc0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:03:01.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.966+0000 7fd040cc0980 20 rados->read ofs=0 len=0 2024-03-09T15:03:01.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.966+0000 7fd040cc0980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:03:01.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.966+0000 7fd040cc0980 20 searching for the correct realm 2024-03-09T15:03:01.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:01.974+0000 7fd03efd9640 0 -- 172.21.15.1:0/1752985047 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7fd034050950 msgr2=0x7fd034042400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T15:03:02.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.190+0000 7fd03e7d8640 0 -- 172.21.15.1:0/1752985047 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x55f05b8c95a0 msgr2=0x55f05b8cb990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:03:02.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:03:02.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:03:02.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:03:02.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:03:02.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:03:02.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:03:02.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:02.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:03:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:03:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 zone default found 2024-03-09T15:03:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 4 Realm: () 2024-03-09T15:03:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:03:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:03:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:03:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 zonegroup default 2024-03-09T15:03:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.198+0000 7fd040cc0980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.202+0000 7fd040cc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:02.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.202+0000 7fd040cc0980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.202+0000 7fd040cc0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:02.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.202+0000 7fd040cc0980 20 started sync module instance, tier type = 2024-03-09T15:03:02.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.202+0000 7fd040cc0980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:03:02.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.234+0000 7fd040cc0980 20 add_watcher() i=0 2024-03-09T15:03:02.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.234+0000 7fd040cc0980 20 add_watcher() i=1 2024-03-09T15:03:02.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.234+0000 7fd040cc0980 20 add_watcher() i=2 2024-03-09T15:03:02.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.234+0000 7fd040cc0980 20 add_watcher() i=3 2024-03-09T15:03:02.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.234+0000 7fd040cc0980 20 add_watcher() i=4 2024-03-09T15:03:02.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.234+0000 7fd040cc0980 20 add_watcher() i=5 2024-03-09T15:03:02.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.234+0000 7fd040cc0980 20 add_watcher() i=6 2024-03-09T15:03:02.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.238+0000 7fd040cc0980 20 add_watcher() i=7 2024-03-09T15:03:02.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.238+0000 7fd040cc0980 2 all 8 watchers are set, enabling cache 2024-03-09T15:03:02.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.242+0000 7fcef4fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:03:02.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.242+0000 7fd040cc0980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:03:02.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.242+0000 7fd040cc0980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:03:02.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.338+0000 7fce6f7fe640 20 reqs_thread_entry: start 2024-03-09T15:03:02.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.338+0000 7fd040cc0980 20 init_complete bucket index max shards: 1 2024-03-09T15:03:02.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.338+0000 7fd040cc0980 20 Filter name: none 2024-03-09T15:03:02.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.338+0000 7fce6d7fa640 20 reqs_thread_entry: start 2024-03-09T15:03:02.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.338+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T15:03:02.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.338+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.342+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.342+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.346+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T15:03:02.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.346+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.350+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.350+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.350+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T15:03:02.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.354+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.358+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.358+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.358+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T15:03:02.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.362+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.366+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.366+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.370+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T15:03:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.374+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.374+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.374+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.378+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T15:03:02.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.382+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.382+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.382+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.386+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T15:03:02.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.390+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.390+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.390+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.394+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T15:03:02.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.398+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.398+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.398+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.402+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T15:03:02.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.402+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.406+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.406+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.406+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T15:03:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.410+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.414+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.414+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.414+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T15:03:02.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.418+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.422+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.422+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.426+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T15:03:02.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.430+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.430+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.430+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.434+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T15:03:02.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.434+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.438+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.438+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.438+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T15:03:02.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.442+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.446+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.446+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.450+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T15:03:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.454+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.458+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.458+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.462+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T15:03:02.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.466+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.466+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.466+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.470+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T15:03:02.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.474+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.474+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.474+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.478+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T15:03:02.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.482+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.482+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.486+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.486+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T15:03:02.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.490+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.490+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.490+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.494+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T15:03:02.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.498+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.498+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.502+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.502+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T15:03:02.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.506+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.510+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.510+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.514+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T15:03:02.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.518+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.518+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.518+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.522+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T15:03:02.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.526+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.530+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.530+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.534+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T15:03:02.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.538+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.538+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.538+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.542+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T15:03:02.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.546+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.546+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.546+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.550+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T15:03:02.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.554+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.554+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.558+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.558+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T15:03:02.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.562+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.566+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.566+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.570+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T15:03:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.574+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.574+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.578+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.578+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T15:03:02.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.582+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.582+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.586+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.586+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T15:03:02.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.594+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.594+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.594+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.598+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T15:03:02.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.602+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.606+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.606+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.606+0000 7fd040cc0980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T15:03:02.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.610+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.614+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.614+0000 7fd040cc0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.618+0000 7fcef4fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:03:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.630+0000 7fd040cc0980 20 remove_watcher() i=0 2024-03-09T15:03:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.630+0000 7fd040cc0980 2 removed watcher, disabling cache 2024-03-09T15:03:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.630+0000 7fd040cc0980 20 remove_watcher() i=1 2024-03-09T15:03:02.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.634+0000 7fd040cc0980 20 remove_watcher() i=2 2024-03-09T15:03:02.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.634+0000 7fd040cc0980 20 remove_watcher() i=3 2024-03-09T15:03:02.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.642+0000 7fd040cc0980 20 remove_watcher() i=4 2024-03-09T15:03:02.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.642+0000 7fd040cc0980 20 remove_watcher() i=5 2024-03-09T15:03:02.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.646+0000 7fd040cc0980 20 remove_watcher() i=6 2024-03-09T15:03:02.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.646+0000 7fd040cc0980 20 remove_watcher() i=7 2024-03-09T15:03:02.672 DEBUG:teuthology.orchestra.run.smithi001:> radosgw-admin gc process --include-all 2024-03-09T15:03:02.751 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:03:02.751 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:03:02.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.798+0000 7f047937a980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.798+0000 7f047937a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.798+0000 7f047937a980 20 realm 2024-03-09T15:03:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.798+0000 7f047937a980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.798+0000 7f047937a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.798+0000 7f047937a980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:03:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.798+0000 7f047937a980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.798+0000 7f047937a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:02.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.798+0000 7f047937a980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.802+0000 7f047937a980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:03:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.802+0000 7f047937a980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.802+0000 7f047937a980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:03:02.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.802+0000 7f047937a980 20 searching for the correct realm 2024-03-09T15:03:02.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.810+0000 7f0476691640 0 -- 172.21.15.1:0/940961865 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x558093685a10 msgr2=0x558093687ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:03:02.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.814+0000 7f0476e92640 0 -- 172.21.15.1:0/940961865 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f04600857e0 msgr2=0x7f046007c790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:03:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.822+0000 7f047937a980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:03:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.822+0000 7f047937a980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:03:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.822+0000 7f047937a980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:03:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.822+0000 7f047937a980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:03:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.822+0000 7f047937a980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:03:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.822+0000 7f047937a980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:03:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.822+0000 7f047937a980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.822+0000 7f047937a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.822+0000 7f047937a980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:03:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:03:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 zone default found 2024-03-09T15:03:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 4 Realm: () 2024-03-09T15:03:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:03:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:03:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:03:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 zonegroup default 2024-03-09T15:03:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:02.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 rados->read ofs=0 len=0 2024-03-09T15:03:02.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:03:02.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 started sync module instance, tier type = 2024-03-09T15:03:02.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.826+0000 7f047937a980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:03:02.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.842+0000 7f047937a980 20 add_watcher() i=0 2024-03-09T15:03:02.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.842+0000 7f047937a980 20 add_watcher() i=1 2024-03-09T15:03:02.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.842+0000 7f047937a980 20 add_watcher() i=2 2024-03-09T15:03:02.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.842+0000 7f047937a980 20 add_watcher() i=3 2024-03-09T15:03:02.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.842+0000 7f047937a980 20 add_watcher() i=4 2024-03-09T15:03:02.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.842+0000 7f047937a980 20 add_watcher() i=5 2024-03-09T15:03:02.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.842+0000 7f047937a980 20 add_watcher() i=6 2024-03-09T15:03:02.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.846+0000 7f047937a980 20 add_watcher() i=7 2024-03-09T15:03:02.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.846+0000 7f047937a980 2 all 8 watchers are set, enabling cache 2024-03-09T15:03:02.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.846+0000 7f032dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:03:02.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.846+0000 7f047937a980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:03:02.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.846+0000 7f047937a980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:03:02.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.922+0000 7f032cfe1640 20 reqs_thread_entry: start 2024-03-09T15:03:02.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.922+0000 7f047937a980 20 init_complete bucket index max shards: 1 2024-03-09T15:03:02.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.922+0000 7f047937a980 20 Filter name: none 2024-03-09T15:03:02.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.922+0000 7f0325ffb640 20 reqs_thread_entry: start 2024-03-09T15:03:02.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.922+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T15:03:02.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.922+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.926+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.926+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.926+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T15:03:02.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.930+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.930+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.930+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.934+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T15:03:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.938+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.938+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.938+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.942+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T15:03:02.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.946+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.946+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.946+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.950+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T15:03:02.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.954+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.954+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.954+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.958+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T15:03:02.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.958+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.962+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.962+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.962+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T15:03:02.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.966+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.966+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.970+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.970+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T15:03:02.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.974+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.978+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.978+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.982+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T15:03:02.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.982+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.986+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.986+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.990+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T15:03:02.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.990+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.990+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.994+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.994+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T15:03:03.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.998+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.998+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:02.998+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.002+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T15:03:03.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.006+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.006+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.006+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.006+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T15:03:03.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.010+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.010+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.010+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.014+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T15:03:03.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.018+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.018+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.018+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.022+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T15:03:03.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.026+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.030+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.030+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.034+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T15:03:03.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.034+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.034+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.038+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.038+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T15:03:03.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.042+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.042+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.046+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.046+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T15:03:03.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.050+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.050+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.054+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.054+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T15:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.058+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.062+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.062+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.066+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T15:03:03.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.070+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.070+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.074+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.074+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T15:03:03.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.078+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.078+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.082+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.082+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T15:03:03.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.090+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.090+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.090+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.094+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T15:03:03.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.098+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.098+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.102+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.102+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T15:03:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.106+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.106+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.110+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.110+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T15:03:03.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.114+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.118+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.118+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.122+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T15:03:03.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.126+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.126+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.126+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.130+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T15:03:03.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.134+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.134+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.138+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.138+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T15:03:03.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.142+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.146+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.146+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.150+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T15:03:03.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.154+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.154+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.154+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.162+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T15:03:03.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.162+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.166+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.166+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.166+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T15:03:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.170+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.174+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.174+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.178+0000 7f047937a980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T15:03:03.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.182+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.182+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:03:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.182+0000 7f047937a980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:03:03.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.186+0000 7f032dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:03:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.194+0000 7f047937a980 20 remove_watcher() i=0 2024-03-09T15:03:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.194+0000 7f047937a980 2 removed watcher, disabling cache 2024-03-09T15:03:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.194+0000 7f047937a980 20 remove_watcher() i=1 2024-03-09T15:03:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.198+0000 7f047937a980 20 remove_watcher() i=2 2024-03-09T15:03:03.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.202+0000 7f047937a980 20 remove_watcher() i=3 2024-03-09T15:03:03.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.206+0000 7f047937a980 20 remove_watcher() i=4 2024-03-09T15:03:03.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.206+0000 7f047937a980 20 remove_watcher() i=5 2024-03-09T15:03:03.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.210+0000 7f047937a980 20 remove_watcher() i=6 2024-03-09T15:03:03.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:03:03.214+0000 7f047937a980 20 remove_watcher() i=7 2024-03-09T15:03:03.236 DEBUG:teuthology.orchestra.run.smithi001:> cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest 2024-03-09T15:03:03.962 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :clean 2024-03-09T15:03:04.860 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :compileJava 2024-03-09T15:03:04.860 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :processResources 2024-03-09T15:03:04.860 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :classes 2024-03-09T15:03:05.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:05.360 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :compileTestJava 2024-03-09T15:03:05.360 INFO:teuthology.orchestra.run.smithi001.stderr:Note: /home/ubuntu/cephtest/s3-tests-java/src/test/java/BucketTest.java uses or overrides a deprecated API. 2024-03-09T15:03:05.360 INFO:teuthology.orchestra.run.smithi001.stderr:Note: Recompile with -Xlint:deprecation for details. 2024-03-09T15:03:05.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:05.459 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :processTestResources NO-SOURCE 2024-03-09T15:03:05.460 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :testClasses 2024-03-09T15:03:07.360 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:07.360 INFO:teuthology.orchestra.run.smithi001.stdout:> Task :test 2024-03-09T15:03:07.360 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:07.360 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests STANDARD_OUT 2024-03-09T15:03:07.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09 15:03:06,809 [Test worker] WARN com.amazonaws.util.Base64 - JAXB is unavailable. Will fallback to SDK implementation which may be less performant 2024-03-09T15:03:07.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:07.361 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.TestObjectCopyKeyNotFound STARTED 2024-03-09T15:03:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:07.560 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.TestObjectCopyKeyNotFound PASSED 2024-03-09T15:03:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:08.360 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.TestObjectCreateBadExpectNone STARTED 2024-03-09T15:03:08.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:08.560 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.TestObjectCreateBadExpectNone PASSED 2024-03-09T15:03:09.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:09.060 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.TestObjectCreateBadMd5Empty STARTED 2024-03-09T15:03:09.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:09.260 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.TestObjectCreateBadMd5Empty PASSED 2024-03-09T15:03:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:09.760 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testAbortMultipartUploadLLAPI STARTED 2024-03-09T15:03:22.260 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:22.260 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testAbortMultipartUploadLLAPI PASSED 2024-03-09T15:03:22.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:22.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testAbortMultipartUploadNotFoundLLAPI STARTED 2024-03-09T15:03:22.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:22.960 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testAbortMultipartUploadNotFoundLLAPI PASSED 2024-03-09T15:03:23.460 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:23.460 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testDownloadHLAPI STARTED 2024-03-09T15:03:23.859 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:23.860 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testDownloadHLAPI PASSED 2024-03-09T15:03:24.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:24.460 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testDownloadNoBucketHLAPI STARTED 2024-03-09T15:03:24.460 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:24.460 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testDownloadNoBucketHLAPI PASSED 2024-03-09T15:03:24.659 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:24.660 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testDownloadNoKeyHLAPI STARTED 2024-03-09T15:03:24.760 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:24.760 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testDownloadNoKeyHLAPI PASSED 2024-03-09T15:03:25.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:25.260 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1MB STARTED 2024-03-09T15:03:34.860 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:34.860 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1MB PASSED 2024-03-09T15:03:35.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:35.360 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1b STARTED 2024-03-09T15:03:35.659 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:35.660 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1b PASSED 2024-03-09T15:03:36.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:36.159 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1kb STARTED 2024-03-09T15:03:36.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:36.560 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1kb PASSED 2024-03-09T15:03:37.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:37.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptionKeyNoSSEC STARTED 2024-03-09T15:03:37.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:37.260 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptionKeyNoSSEC PASSED 2024-03-09T15:03:37.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:37.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECInvalidMd5 STARTED 2024-03-09T15:03:37.960 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:37.960 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECInvalidMd5 PASSED 2024-03-09T15:03:38.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:38.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECNoKey STARTED 2024-03-09T15:03:38.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:38.560 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECNoKey PASSED 2024-03-09T15:03:39.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:39.060 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECNoMd5 STARTED 2024-03-09T15:03:39.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:39.260 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECNoMd5 PASSED 2024-03-09T15:03:39.760 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:39.760 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultiObjectDelete STARTED 2024-03-09T15:03:40.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:40.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultiObjectDelete PASSED 2024-03-09T15:03:40.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:40.559 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartCopyHLAPIA STARTED 2024-03-09T15:03:54.060 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:54.060 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartCopyHLAPIA PASSED 2024-03-09T15:03:55.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:03:55.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartCopyMultipleSizesLLAPI STARTED 2024-03-09T15:04:39.860 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:04:39.860 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartCopyMultipleSizesLLAPI PASSED 2024-03-09T15:04:40.859 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:04:40.859 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartCopyNoSRCBucketHLAPI STARTED 2024-03-09T15:04:40.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:04:40.959 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartCopyNoSRCBucketHLAPI PASSED 2024-03-09T15:04:41.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:04:41.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartCopyNoSRCKeyHLAPI STARTED 2024-03-09T15:04:41.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:04:41.559 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartCopyNoSRCKeyHLAPI PASSED 2024-03-09T15:04:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:04:42.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartDownloadHLAPI STARTED 2024-03-09T15:04:55.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:04:55.460 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartDownloadHLAPI PASSED 2024-03-09T15:04:55.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:04:55.960 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartDownloadNoBucketHLAPI STARTED 2024-03-09T15:04:56.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:04:56.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartDownloadNoBucketHLAPI PASSED 2024-03-09T15:04:56.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:04:56.159 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartDownloadWithPauseHLAPI STARTED 2024-03-09T15:05:24.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:05:24.259 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartDownloadWithPauseHLAPI PASSED 2024-03-09T15:05:28.860 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:05:28.860 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests STANDARD_OUT 2024-03-09T15:05:28.860 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09 15:05:28,837 [s3-transfer-manager-worker-9] WARN com.amazonaws.services.s3.internal.S3AbortableInputStream - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use. 2024-03-09T15:05:28.860 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09 15:05:28,842 [s3-transfer-manager-worker-10] WARN com.amazonaws.services.s3.internal.S3AbortableInputStream - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use. 2024-03-09T15:05:29.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09 15:05:28,962 [s3-transfer-manager-worker-1] WARN com.amazonaws.services.s3.internal.S3AbortableInputStream - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use. 2024-03-09T15:05:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:05:30.959 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartUploadHLAPIAWS4 STARTED 2024-03-09T15:05:57.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:05:57.460 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartUploadHLAPIAWS4 PASSED 2024-03-09T15:05:58.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:05:58.259 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartUploadIncorrectMissingPartLLAPI STARTED 2024-03-09T15:06:10.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:06:10.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartUploadIncorrectMissingPartLLAPI PASSED 2024-03-09T15:06:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:06:11.259 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartUploadMultipleSizesLLAPI STARTED 2024-03-09T15:07:37.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:07:37.360 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartUploadMultipleSizesLLAPI PASSED 2024-03-09T15:07:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:07:37.859 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartUploadOverwriteExistingObjectLLAPI STARTED 2024-03-09T15:07:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:07:55.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartUploadOverwriteExistingObjectLLAPI PASSED 2024-03-09T15:07:55.859 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:07:55.859 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartUploadSmallLLAPI STARTED 2024-03-09T15:08:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:08.360 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartUploadSmallLLAPI PASSED 2024-03-09T15:08:08.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:08.959 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartUploadWithPause STARTED 2024-03-09T15:08:13.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:13.559 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testMultipartUploadWithPause PASSED 2024-03-09T15:08:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:23.060 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectCopyBucketNotFound STARTED 2024-03-09T15:08:23.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:23.259 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectCopyBucketNotFound PASSED 2024-03-09T15:08:24.858 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:24.859 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectCreateBadExpectMismatch STARTED 2024-03-09T15:08:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:26.659 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectCreateBadExpectMismatch PASSED 2024-03-09T15:08:29.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:29.159 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5InvalidShort STARTED 2024-03-09T15:08:29.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:29.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5InvalidShort PASSED 2024-03-09T15:08:29.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:29.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5Ivalid STARTED 2024-03-09T15:08:29.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:29.959 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5Ivalid PASSED 2024-03-09T15:08:30.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:30.459 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5None STARTED 2024-03-09T15:08:30.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:30.559 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5None PASSED 2024-03-09T15:08:31.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:31.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5Unreadable STARTED 2024-03-09T15:08:31.258 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:31.259 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5Unreadable PASSED 2024-03-09T15:08:31.659 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:31.659 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectHeadZeroBytes STARTED 2024-03-09T15:08:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:31.959 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectHeadZeroBytes PASSED 2024-03-09T15:08:32.458 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:32.459 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterDot STARTED 2024-03-09T15:08:33.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:33.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterDot PASSED 2024-03-09T15:08:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:33.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterNotExist STARTED 2024-03-09T15:08:34.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:34.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterNotExist PASSED 2024-03-09T15:08:35.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:35.159 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterPercentage STARTED 2024-03-09T15:08:35.659 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:35.659 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterPercentage PASSED 2024-03-09T15:08:36.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:36.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMarkerAfterList STARTED 2024-03-09T15:08:37.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:37.159 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMarkerAfterList PASSED 2024-03-09T15:08:37.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:37.959 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMarkerBeforeList STARTED 2024-03-09T15:08:38.458 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:38.459 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMarkerBeforeList PASSED 2024-03-09T15:08:39.258 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:39.259 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMarkerNotInList STARTED 2024-03-09T15:08:39.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:39.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMarkerNotInList PASSED 2024-03-09T15:08:40.458 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:40.459 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMarkerUnreadable STARTED 2024-03-09T15:08:40.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:40.959 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMarkerUnreadable PASSED 2024-03-09T15:08:41.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:41.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysNegative STARTED 2024-03-09T15:08:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:42.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysNegative PASSED 2024-03-09T15:08:43.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:43.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysNone STARTED 2024-03-09T15:08:43.659 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:43.659 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysNone PASSED 2024-03-09T15:08:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:44.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysOne STARTED 2024-03-09T15:08:44.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:44.959 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysOne PASSED 2024-03-09T15:08:45.658 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:45.659 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysZero STARTED 2024-03-09T15:08:46.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:46.159 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysZero PASSED 2024-03-09T15:08:46.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:46.959 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixAlt STARTED 2024-03-09T15:08:47.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:47.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixAlt PASSED 2024-03-09T15:08:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:48.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixBasic STARTED 2024-03-09T15:08:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:48.459 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixBasic PASSED 2024-03-09T15:08:49.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterAlt STARTED 2024-03-09T15:08:49.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:49.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterAlt PASSED 2024-03-09T15:08:50.458 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:50.459 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterBasic STARTED 2024-03-09T15:08:50.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:50.959 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterBasic PASSED 2024-03-09T15:08:51.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:51.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterDelimiterNotExist STARTED 2024-03-09T15:08:52.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:52.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterDelimiterNotExist PASSED 2024-03-09T15:08:53.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:53.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterPrefixDelimiterNotExist STARTED 2024-03-09T15:08:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:53.559 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterPrefixDelimiterNotExist PASSED 2024-03-09T15:08:54.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:54.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterPrefixNotExist STARTED 2024-03-09T15:08:54.859 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:54.859 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterPrefixNotExist PASSED 2024-03-09T15:08:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:55.659 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixEmpty STARTED 2024-03-09T15:08:56.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:56.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixEmpty PASSED 2024-03-09T15:08:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:56.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixNone STARTED 2024-03-09T15:08:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:57.159 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixNone PASSED 2024-03-09T15:08:57.858 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:57.859 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixNotExist STARTED 2024-03-09T15:08:58.358 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:58.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixNotExist PASSED 2024-03-09T15:08:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:59.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixUnreadable STARTED 2024-03-09T15:08:59.458 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:08:59.459 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectListPrefixUnreadable PASSED 2024-03-09T15:09:00.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:00.159 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectReadFromNonExistantBucket STARTED 2024-03-09T15:09:00.358 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:00.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectReadFromNonExistantBucket PASSED 2024-03-09T15:09:00.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:00.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectReadNotExist STARTED 2024-03-09T15:09:00.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:00.959 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectReadNotExist PASSED 2024-03-09T15:09:01.358 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:01.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectWriteCacheControl STARTED 2024-03-09T15:09:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:01.659 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectWriteCacheControl PASSED 2024-03-09T15:09:02.258 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:02.259 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectWriteCheckEtag STARTED 2024-03-09T15:09:02.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:02.559 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectWriteCheckEtag PASSED 2024-03-09T15:09:03.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:03.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectWriteReadUpdateReadDelete STARTED 2024-03-09T15:09:03.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:03.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectWriteReadUpdateReadDelete PASSED 2024-03-09T15:09:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectWriteToNonExistantBucket STARTED 2024-03-09T15:09:04.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:04.259 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testObjectWriteToNonExistantBucket PASSED 2024-03-09T15:09:04.458 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:04.459 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testRangedSkipLeadingBytesResponseCode STARTED 2024-03-09T15:09:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:04.659 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testRangedSkipLeadingBytesResponseCode PASSED 2024-03-09T15:09:05.258 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:05.259 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testRangedrequestResponseCode STARTED 2024-03-09T15:09:05.558 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:05.559 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testRangedrequestResponseCode PASSED 2024-03-09T15:09:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:06.059 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testSSEKMSNoKey STARTED 2024-03-09T15:09:06.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:06.359 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testSSEKMSNoKey PASSED 2024-03-09T15:09:06.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:06.759 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testSSEKMSNotDeclared STARTED 2024-03-09T15:09:06.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:06.959 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testSSEKMSNotDeclared PASSED 2024-03-09T15:09:07.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:07.459 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testUploadFileHLAPIBigFile STARTED 2024-03-09T15:09:21.158 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:21.159 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testUploadFileHLAPIBigFile PASSED 2024-03-09T15:09:21.658 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:21.659 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testUploadFileList STARTED 2024-03-09T15:09:45.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:45.559 INFO:teuthology.orchestra.run.smithi001.stdout:suite > Object tests > ObjectTest.testUploadFileList PASSED 2024-03-09T15:09:46.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:46.159 INFO:teuthology.orchestra.run.smithi001.stdout:---------------------------------------------------------------------- 2024-03-09T15:09:46.159 INFO:teuthology.orchestra.run.smithi001.stdout:| Results: SUCCESS (70 tests, 70 successes, 0 failures, 0 skipped) | 2024-03-09T15:09:46.159 INFO:teuthology.orchestra.run.smithi001.stdout:---------------------------------------------------------------------- 2024-03-09T15:09:46.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:46.159 INFO:teuthology.orchestra.run.smithi001.stdout:Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. 2024-03-09T15:09:46.159 INFO:teuthology.orchestra.run.smithi001.stdout:Use '--warning-mode all' to show the individual deprecation warnings. 2024-03-09T15:09:46.159 INFO:teuthology.orchestra.run.smithi001.stdout:See https://docs.gradle.org/6.0.1/userguide/command_line_interface.html#sec:command_line_warnings 2024-03-09T15:09:46.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:46.159 INFO:teuthology.orchestra.run.smithi001.stdout:BUILD SUCCESSFUL in 6m 42s 2024-03-09T15:09:46.160 INFO:teuthology.orchestra.run.smithi001.stdout:5 actionable tasks: 5 executed 2024-03-09T15:09:46.168 DEBUG:teuthology.orchestra.run.smithi001:> radosgw-admin gc process --include-all 2024-03-09T15:09:46.195 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:46.195 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:46.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.234+0000 7ffa5b6f0980 20 rados->read ofs=0 len=0 2024-03-09T15:09:46.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.238+0000 7ffa5b6f0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:46.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.238+0000 7ffa5b6f0980 20 realm 2024-03-09T15:09:46.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.238+0000 7ffa5b6f0980 20 rados->read ofs=0 len=0 2024-03-09T15:09:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.238+0000 7ffa5b6f0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.238+0000 7ffa5b6f0980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.238+0000 7ffa5b6f0980 20 rados->read ofs=0 len=0 2024-03-09T15:09:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.238+0000 7ffa5b6f0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:46.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.238+0000 7ffa5b6f0980 20 rados->read ofs=0 len=0 2024-03-09T15:09:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.238+0000 7ffa5b6f0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.238+0000 7ffa5b6f0980 20 rados->read ofs=0 len=0 2024-03-09T15:09:46.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.242+0000 7ffa5b6f0980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:46.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.242+0000 7ffa5b6f0980 20 searching for the correct realm 2024-03-09T15:09:46.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.246+0000 7ffa58a07640 0 -- 172.21.15.1:0/2904926133 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x55df9310cd90 msgr2=0x55df9310f250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:46.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.254+0000 7ffa59208640 0 -- 172.21.15.1:0/2904926133 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7ffa500b0980 msgr2=0x7ffa500e5420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:46.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 rados->read ofs=0 len=0 2024-03-09T15:09:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:46.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 rados->read ofs=0 len=0 2024-03-09T15:09:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 rados->read ofs=0 len=0 2024-03-09T15:09:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 zone default found 2024-03-09T15:09:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 4 Realm: () 2024-03-09T15:09:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:46.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 zonegroup default 2024-03-09T15:09:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 rados->read ofs=0 len=0 2024-03-09T15:09:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:46.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 rados->read ofs=0 len=0 2024-03-09T15:09:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 rados->read ofs=0 len=0 2024-03-09T15:09:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 started sync module instance, tier type = 2024-03-09T15:09:46.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.262+0000 7ffa5b6f0980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:46.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.278+0000 7ffa5b6f0980 20 add_watcher() i=0 2024-03-09T15:09:46.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.278+0000 7ffa5b6f0980 20 add_watcher() i=1 2024-03-09T15:09:46.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.278+0000 7ffa5b6f0980 20 add_watcher() i=2 2024-03-09T15:09:46.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.278+0000 7ffa5b6f0980 20 add_watcher() i=3 2024-03-09T15:09:46.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.278+0000 7ffa5b6f0980 20 add_watcher() i=4 2024-03-09T15:09:46.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.278+0000 7ffa5b6f0980 20 add_watcher() i=5 2024-03-09T15:09:46.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.278+0000 7ffa5b6f0980 20 add_watcher() i=6 2024-03-09T15:09:46.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.282+0000 7ffa5b6f0980 20 add_watcher() i=7 2024-03-09T15:09:46.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.282+0000 7ffa5b6f0980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:46.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.282+0000 7ff90f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:46.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.282+0000 7ffa5b6f0980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:46.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.282+0000 7ffa5b6f0980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:46.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.354+0000 7ff90e7e4640 20 reqs_thread_entry: start 2024-03-09T15:09:46.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.354+0000 7ffa5b6f0980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:46.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.354+0000 7ffa5b6f0980 20 Filter name: none 2024-03-09T15:09:46.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.354+0000 7ff87ffff640 20 reqs_thread_entry: start 2024-03-09T15:09:46.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.354+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T15:09:46.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.358+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.358+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.358+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2024-03-09T15:09:46.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.358+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2794119866190917714', time=2024-03-09T17:08:33.287580+0000, chain.objs.size()=1 2024-03-09T15:09:46.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.358+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.101__shadow_.RwRSg8YElF4TZhtSVZC7BKnUR7Xi05G_0 2024-03-09T15:09:46.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.358+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12017500632635186221', time=2024-03-09T17:08:37.486221+0000, chain.objs.size()=1 2024-03-09T15:09:46.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.358+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.104__shadow_.NAF02NY79uyKEzTag5ZRIsGafIWInaa_0 2024-03-09T15:09:46.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.358+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9560357445032748273', time=2024-03-09T17:08:55.058960+0000, chain.objs.size()=1 2024-03-09T15:09:46.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.358+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.118__shadow_.bwe5Q5-0jZAbHXQw8bXzFblaRpFD2Vs_0 2024-03-09T15:09:46.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.358+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17842405349656826355', time=2024-03-09T17:09:04.866753+0000, chain.objs.size()=1 2024-03-09T15:09:46.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.358+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.128__shadow_.9PciCDXDkkAGfuM5jcN4vSr_qdS_8e8_0 2024-03-09T15:09:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.362+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.370+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T15:09:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.370+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.374+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.374+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2024-03-09T15:09:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.374+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15813107064558938758', time=2024-03-09T17:08:50.081862+0000, chain.objs.size()=1 2024-03-09T15:09:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.374+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.114__shadow_.7ReoPvRXgx6qlxRaW4ZFjuXaG_Mx1R9_0 2024-03-09T15:09:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.374+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7993219238470038142', time=2024-03-09T17:08:52.482644+0000, chain.objs.size()=1 2024-03-09T15:09:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.374+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.116__shadow_.Qp8PlIbi_aplO453oseuxMTzp9CInYH_0 2024-03-09T15:09:46.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.374+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.382+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T15:09:46.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.382+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.386+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.386+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2024-03-09T15:09:46.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.386+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12710339534002077061', time=2024-03-09T17:08:51.153524+0000, chain.objs.size()=1 2024-03-09T15:09:46.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.386+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.115__shadow_.9SVO0C91WdvhjwNQJmYuJkjTl4C52SP_0 2024-03-09T15:09:46.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.386+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18312831191796129056', time=2024-03-09T17:08:59.645854+0000, chain.objs.size()=1 2024-03-09T15:09:46.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.386+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.122__shadow_.NAkMxSL9yFgGBpR1eMmFSZaVfUQ5_y__0 2024-03-09T15:09:46.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.390+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.394+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T15:09:46.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.398+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.398+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.402+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2024-03-09T15:09:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.402+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8527991920891800147', time=2024-03-09T17:08:33.367727+0000, chain.objs.size()=1 2024-03-09T15:09:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.402+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.101__shadow_.GQjlcWDHmN-tGsPGk49qo0XVB5q2a40_0 2024-03-09T15:09:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.402+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8191575104261452905', time=2024-03-09T17:08:43.950360+0000, chain.objs.size()=1 2024-03-09T15:09:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.402+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.109__shadow_.CWcLaiQ8AyKDR0a1Jn5eUgD4B8Hbo2Z_0 2024-03-09T15:09:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.402+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11648549437544942519', time=2024-03-09T17:08:52.544299+0000, chain.objs.size()=1 2024-03-09T15:09:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.402+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.116__shadow_.vyLF3HyEW2Ce56BWssFb5j8nhLMOprL_0 2024-03-09T15:09:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.402+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6576084490466220927', time=2024-03-09T17:08:53.901784+0000, chain.objs.size()=1 2024-03-09T15:09:46.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.402+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.117__shadow_.NXLymc5ZdBx5ENVN2ZaROFlLT0KTEMT_0 2024-03-09T15:09:46.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.406+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.410+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T15:09:46.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2024-03-09T15:09:46.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8415522786640342951', time=2024-03-09T17:03:24.077092+0000, chain.objs.size()=1 2024-03-09T15:09:46.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.70__shadow_.hEyltmlGd-2RwWd5cAqF6mFG56zpXDM_0 2024-03-09T15:09:46.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3630792605492360382', time=2024-03-09T17:05:57.669949+0000, chain.objs.size()=1 2024-03-09T15:09:46.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__shadow_.k9IsI49-HwKKHWZ_CUoYI-Hs2KH-r5Q_0 2024-03-09T15:09:46.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8720096041346567003', time=2024-03-09T17:05:57.739836+0000, chain.objs.size()=5 2024-03-09T15:09:46.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__multipart_file2.mpg.2~1hwF8MLnEPiyDlM5XYKEmDb0FZ1aYF8.1 2024-03-09T15:09:46.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__multipart_file2.mpg.2~1hwF8MLnEPiyDlM5XYKEmDb0FZ1aYF8.2 2024-03-09T15:09:46.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__multipart_file2.mpg.2~1hwF8MLnEPiyDlM5XYKEmDb0FZ1aYF8.3 2024-03-09T15:09:46.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__multipart_file2.mpg.2~1hwF8MLnEPiyDlM5XYKEmDb0FZ1aYF8.4 2024-03-09T15:09:46.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.414+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__multipart_file2.mpg.2~1hwF8MLnEPiyDlM5XYKEmDb0FZ1aYF8.5 2024-03-09T15:09:46.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.418+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4722107740779407653', time=2024-03-09T17:07:12.775166+0000, chain.objs.size()=5 2024-03-09T15:09:46.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.418+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~e2qLnNfwTtTK3hqLAauhTga2s15drom.1 2024-03-09T15:09:46.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.418+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~e2qLnNfwTtTK3hqLAauhTga2s15drom.2 2024-03-09T15:09:46.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.418+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~e2qLnNfwTtTK3hqLAauhTga2s15drom.3 2024-03-09T15:09:46.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.418+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~e2qLnNfwTtTK3hqLAauhTga2s15drom.4 2024-03-09T15:09:46.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.418+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~e2qLnNfwTtTK3hqLAauhTga2s15drom.5 2024-03-09T15:09:46.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.422+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4146374169442215544', time=2024-03-09T17:09:03.579011+0000, chain.objs.size()=1 2024-03-09T15:09:46.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.422+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.127__shadow_.iPyTD7ru29EVsCAA0rZa7YNeTWl63Sn_0 2024-03-09T15:09:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.426+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.430+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T15:09:46.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2024-03-09T15:09:46.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11043028624015027742', time=2024-03-09T17:08:34.595187+0000, chain.objs.size()=1 2024-03-09T15:09:46.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.102__shadow_.w0VCkoV7SIUO4DFs2uiTO5OkxrC5yKR_0 2024-03-09T15:09:46.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8902286047716916744', time=2024-03-09T17:08:34.672449+0000, chain.objs.size()=1 2024-03-09T15:09:46.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.102__shadow_.gt2DCepgSZoWj4O_dRBrfAEWRJBx1lD_0 2024-03-09T15:09:46.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15045874332234642124', time=2024-03-09T17:08:48.674764+0000, chain.objs.size()=1 2024-03-09T15:09:46.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.113__shadow_.N618dzlIP4TGkPO8Z5ObEpa964IuwP6_0 2024-03-09T15:09:46.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4458819574541984495', time=2024-03-09T17:08:50.157360+0000, chain.objs.size()=1 2024-03-09T15:09:46.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.114__shadow_.jqPafeqMCmOHCkUfkBi1TytC6ixtcXq_0 2024-03-09T15:09:46.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13297006146111172065', time=2024-03-09T17:08:52.694796+0000, chain.objs.size()=1 2024-03-09T15:09:46.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.116__shadow_.srf8nf-V97shKJBtqpAc9yIQ3Zp0vzm_0 2024-03-09T15:09:46.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15780202781293496695', time=2024-03-09T17:08:53.824286+0000, chain.objs.size()=1 2024-03-09T15:09:46.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.117__shadow_.wEpSQgSYA0q8s0vDnOrU0tqllhyhSt-_0 2024-03-09T15:09:46.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1864419705085319328', time=2024-03-09T17:08:58.508421+0000, chain.objs.size()=1 2024-03-09T15:09:46.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.434+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.121__shadow_.Skv7IhBkIpZmb-XN3T-dBSEdoND5rmC_0 2024-03-09T15:09:46.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.442+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.446+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T15:09:46.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.450+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2024-03-09T15:09:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4959277622103233541', time=2024-03-09T17:04:35.405400+0000, chain.objs.size()=5 2024-03-09T15:09:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~kYMuP6yUPyadHoDMoMBy3p7xdXrmngn.1 2024-03-09T15:09:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__shadow_key1.2~kYMuP6yUPyadHoDMoMBy3p7xdXrmngn.1_1 2024-03-09T15:09:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~kYMuP6yUPyadHoDMoMBy3p7xdXrmngn.2 2024-03-09T15:09:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__shadow_key1.2~kYMuP6yUPyadHoDMoMBy3p7xdXrmngn.2_1 2024-03-09T15:09:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~kYMuP6yUPyadHoDMoMBy3p7xdXrmngn.3 2024-03-09T15:09:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5634890831772944834', time=2024-03-09T17:08:33.209294+0000, chain.objs.size()=1 2024-03-09T15:09:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.101__shadow_.enFaIYHiYDA6-8U-gO30H2q4OGpVr2L_0 2024-03-09T15:09:46.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8060317799933121586', time=2024-03-09T17:08:33.444335+0000, chain.objs.size()=1 2024-03-09T15:09:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.101__shadow_.o5jpChpMvjJBygPey9bK84-kxLySaT2_0 2024-03-09T15:09:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1894052782515522033', time=2024-03-09T17:08:35.953058+0000, chain.objs.size()=1 2024-03-09T15:09:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.103__shadow_.VGMur8fWHaTXotMQzjaQ46g9vbyOGxD_0 2024-03-09T15:09:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8115156785971878215', time=2024-03-09T17:08:53.747766+0000, chain.objs.size()=1 2024-03-09T15:09:46.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.454+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.117__shadow_.uuROYikX5FW7Meh9415ZqZ3pnVZNGqD_0 2024-03-09T15:09:46.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.466+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.470+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T15:09:46.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=16, truncated=0, next_marker='' 2024-03-09T15:09:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16268720436422765121', time=2024-03-09T17:03:54.267944+0000, chain.objs.size()=5 2024-03-09T15:09:46.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.80__multipart_key1.2~8Z-u_58hALBsHcFa4a2iY4D0kC-9bXi.1 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.80__multipart_key1.2~8Z-u_58hALBsHcFa4a2iY4D0kC-9bXi.2 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.80__multipart_key1.2~8Z-u_58hALBsHcFa4a2iY4D0kC-9bXi.3 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.80__multipart_key1.2~8Z-u_58hALBsHcFa4a2iY4D0kC-9bXi.4 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.80__multipart_key1.2~8Z-u_58hALBsHcFa4a2iY4D0kC-9bXi.5 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17744767852122405328', time=2024-03-09T17:05:57.593682+0000, chain.objs.size()=5 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__multipart_file.mpg.2~UyTPbxBbs7M4qLK38VzlvfEVqu5kB8J.1 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__multipart_file.mpg.2~UyTPbxBbs7M4qLK38VzlvfEVqu5kB8J.2 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__multipart_file.mpg.2~UyTPbxBbs7M4qLK38VzlvfEVqu5kB8J.3 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__multipart_file.mpg.2~UyTPbxBbs7M4qLK38VzlvfEVqu5kB8J.4 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__multipart_file.mpg.2~UyTPbxBbs7M4qLK38VzlvfEVqu5kB8J.5 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2617090655599520579', time=2024-03-09T17:07:24.949762+0000, chain.objs.size()=5 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~usP32apftPuM3AU7IJE-qGbQzeKH-ZR.1 2024-03-09T15:09:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.474+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__shadow_key1.2~usP32apftPuM3AU7IJE-qGbQzeKH-ZR.1_1 2024-03-09T15:09:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~usP32apftPuM3AU7IJE-qGbQzeKH-ZR.2 2024-03-09T15:09:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__shadow_key1.2~usP32apftPuM3AU7IJE-qGbQzeKH-ZR.2_1 2024-03-09T15:09:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~usP32apftPuM3AU7IJE-qGbQzeKH-ZR.3 2024-03-09T15:09:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7', time=2024-03-09T17:08:18.950151+0000, chain.objs.size()=1 2024-03-09T15:09:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.12 2024-03-09T15:09:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7', time=2024-03-09T17:08:18.980899+0000, chain.objs.size()=1 2024-03-09T15:09:46.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.13 2024-03-09T15:09:46.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7', time=2024-03-09T17:08:19.142055+0000, chain.objs.size()=1 2024-03-09T15:09:46.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.14 2024-03-09T15:09:46.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7', time=2024-03-09T17:08:19.261510+0000, chain.objs.size()=1 2024-03-09T15:09:46.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.15 2024-03-09T15:09:46.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7', time=2024-03-09T17:08:19.449166+0000, chain.objs.size()=1 2024-03-09T15:09:46.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.16 2024-03-09T15:09:46.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7', time=2024-03-09T17:08:19.627973+0000, chain.objs.size()=1 2024-03-09T15:09:46.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.17 2024-03-09T15:09:46.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7', time=2024-03-09T17:08:19.860483+0000, chain.objs.size()=1 2024-03-09T15:09:46.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.18 2024-03-09T15:09:46.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7', time=2024-03-09T17:08:19.913705+0000, chain.objs.size()=1 2024-03-09T15:09:46.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.19 2024-03-09T15:09:46.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7', time=2024-03-09T17:08:20.053176+0000, chain.objs.size()=1 2024-03-09T15:09:46.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.486+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.20 2024-03-09T15:09:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.490+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7', time=2024-03-09T17:08:20.289275+0000, chain.objs.size()=21 2024-03-09T15:09:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.490+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.1 2024-03-09T15:09:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.490+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.2 2024-03-09T15:09:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.490+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.3 2024-03-09T15:09:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.490+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.4 2024-03-09T15:09:46.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.490+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.5 2024-03-09T15:09:46.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.490+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.6 2024-03-09T15:09:46.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.490+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.7 2024-03-09T15:09:46.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.494+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.8 2024-03-09T15:09:46.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.494+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.9 2024-03-09T15:09:46.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.494+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.10 2024-03-09T15:09:46.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.494+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.11 2024-03-09T15:09:46.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.494+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.oR8lU9jW_N0s8-O72QdmajoISsbML_m8.12 2024-03-09T15:09:46.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.494+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.Qjfat9gL37zSaHyvMvUUH5UtbAYyibZ6.13 2024-03-09T15:09:46.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.494+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.7SO8Y0GzfXtN3vUenbjflX_7oBHe4IED.14 2024-03-09T15:09:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.494+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.cTixspz_ccoEHUEBx2_mSVbq_6MtpP6s.15 2024-03-09T15:09:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.498+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.sL9BwR4mCooHk7VAyrZIaKTdT7QBUtrB.16 2024-03-09T15:09:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.498+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2aMVm1OFR8X8Dc8gbpbdoQ6JMOuOwiPb.17 2024-03-09T15:09:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.498+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.tYljEthyaTisGSfAm2uJELXLyIIDqIv6.18 2024-03-09T15:09:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.498+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.z6PMnwAaGZK17zuhwh-X5b_vPGd8S7Pw.19 2024-03-09T15:09:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.498+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.PhKP-ATM_i4ErVCMB5zy7tocClKd0CnR.20 2024-03-09T15:09:46.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.498+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.94__multipart_key1.2~neLAtrT82lMkNmCkAnzXtONWX0V6fc7.21 2024-03-09T15:09:46.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.502+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8036827293211520368', time=2024-03-09T17:08:37.423233+0000, chain.objs.size()=1 2024-03-09T15:09:46.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.502+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.104__shadow_.hz9nNLPE6a_VVOE1ufGv32IoOeEvbFr_0 2024-03-09T15:09:46.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.502+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14406772285313953841', time=2024-03-09T17:08:47.627786+0000, chain.objs.size()=1 2024-03-09T15:09:46.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.502+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.112__shadow_.EcmSWwqqF3dwaPyPtCFIEeaAv4grFFP_0 2024-03-09T15:09:46.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.502+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1375390665673161206', time=2024-03-09T17:08:47.706492+0000, chain.objs.size()=1 2024-03-09T15:09:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.502+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.112__shadow_.fZL2SGpIhakfAiILE5EwN8DnkDYqxrX_0 2024-03-09T15:09:46.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.502+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.510+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T15:09:46.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2024-03-09T15:09:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18104572389660155043', time=2024-03-09T17:04:19.824500+0000, chain.objs.size()=5 2024-03-09T15:09:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~gW9ZqrORrgwTiq3HVW3xD41w2XYrrQ6.1 2024-03-09T15:09:46.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~gW9ZqrORrgwTiq3HVW3xD41w2XYrrQ6.2 2024-03-09T15:09:46.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~gW9ZqrORrgwTiq3HVW3xD41w2XYrrQ6.3 2024-03-09T15:09:46.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~gW9ZqrORrgwTiq3HVW3xD41w2XYrrQ6.4 2024-03-09T15:09:46.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~gW9ZqrORrgwTiq3HVW3xD41w2XYrrQ6.5 2024-03-09T15:09:46.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.915465267519491369', time=2024-03-09T17:08:42.652889+0000, chain.objs.size()=1 2024-03-09T15:09:46.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.108__shadow_.ASv4_VGzO29j5whf389lJ0x_jdAF_R8_0 2024-03-09T15:09:46.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11147206808777063726', time=2024-03-09T17:08:48.754021+0000, chain.objs.size()=1 2024-03-09T15:09:46.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.514+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.113__shadow_.Ciz0xvY_hKG0Qr_XcekoO11TBy394oF_0 2024-03-09T15:09:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.522+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.530+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T15:09:46.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.530+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.534+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.534+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2024-03-09T15:09:46.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.534+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5923918667856515065', time=2024-03-09T17:07:51.256361+0000, chain.objs.size()=1 2024-03-09T15:09:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.534+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.92__shadow_.oydKuhQpS9QSbjrJYJcSdrwbuqwV8zK_0 2024-03-09T15:09:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.534+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1340079139627277086', time=2024-03-09T17:08:40.158018+0000, chain.objs.size()=1 2024-03-09T15:09:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.534+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.106__shadow_.QMJmetCfAwl4b90ONpmOK3Ms8l-F_PZ_0 2024-03-09T15:09:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.534+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1270738385826101418', time=2024-03-09T17:08:59.723566+0000, chain.objs.size()=1 2024-03-09T15:09:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.534+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.122__shadow_.DevXAlyz65jhSjcuuATHJMUwobGkMmI_0 2024-03-09T15:09:46.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.534+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.538+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T15:09:46.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.542+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.546+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.546+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2024-03-09T15:09:46.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.546+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17756349332803642445', time=2024-03-09T17:08:45.162717+0000, chain.objs.size()=1 2024-03-09T15:09:46.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.546+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.110__shadow_.B0wH5bm22yCJ5uFHlAvJrjv_vHgVaLH_0 2024-03-09T15:09:46.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.546+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5009569050444946209', time=2024-03-09T17:08:51.316184+0000, chain.objs.size()=1 2024-03-09T15:09:46.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.546+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.115__shadow_.DSYVFqxU_-6PrLWpMakL3KgmfmXofkL_0 2024-03-09T15:09:46.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.546+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.900542503768215792', time=2024-03-09T17:08:55.210405+0000, chain.objs.size()=1 2024-03-09T15:09:46.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.546+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.118__shadow_.4L9SeFf5nyrv0enG84YTHsp7g8Z0RVu_0 2024-03-09T15:09:46.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.546+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13132708653951213772', time=2024-03-09T17:08:55.269261+0000, chain.objs.size()=1 2024-03-09T15:09:46.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.546+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.118__shadow_.5U08DwCRlXQVpzVF73oxRJ-PFtN13Td_0 2024-03-09T15:09:46.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.546+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.554+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T15:09:46.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.554+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.558+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.558+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2024-03-09T15:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.558+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4830400755454689933', time=2024-03-09T17:05:57.892662+0000, chain.objs.size()=1 2024-03-09T15:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.558+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__shadow_.VaPM8dXZWFVDHuYV7nU3D3CGPvjSPZI_0 2024-03-09T15:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.558+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15665453555615935925', time=2024-03-09T17:08:48.830709+0000, chain.objs.size()=1 2024-03-09T15:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.558+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.113__shadow_.FjKHXyqiFRZyg18OxPEqYUp7epVKeyQ_0 2024-03-09T15:09:46.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.558+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13815584456748354567', time=2024-03-09T17:08:57.461922+0000, chain.objs.size()=1 2024-03-09T15:09:46.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.558+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.120__shadow_.P0b7CSag9NCmjs8HZSawlpm9kNZYW_A_0 2024-03-09T15:09:46.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.558+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.562+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T15:09:46.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.566+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.566+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker='' 2024-03-09T15:09:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15159860046338611927', time=2024-03-09T17:03:39.945171+0000, chain.objs.size()=1 2024-03-09T15:09:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.79__shadow_.XTmKsdjkrK0pYkz3REXjYnfodkaJO7P_0 2024-03-09T15:09:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14093469078338170440', time=2024-03-09T17:04:39.752069+0000, chain.objs.size()=5 2024-03-09T15:09:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~D7K8k7lZ45JzWFhlpa4HlAYZ4Y3RVuk.1 2024-03-09T15:09:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__shadow_key1.2~D7K8k7lZ45JzWFhlpa4HlAYZ4Y3RVuk.1_1 2024-03-09T15:09:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~D7K8k7lZ45JzWFhlpa4HlAYZ4Y3RVuk.2 2024-03-09T15:09:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__shadow_key1.2~D7K8k7lZ45JzWFhlpa4HlAYZ4Y3RVuk.2_1 2024-03-09T15:09:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~D7K8k7lZ45JzWFhlpa4HlAYZ4Y3RVuk.3 2024-03-09T15:09:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8592576772944483153', time=2024-03-09T17:05:28.849979+0000, chain.objs.size()=5 2024-03-09T15:09:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.88__multipart_key1.2~YkgTabhSvuu4a73MC7Ly-nGAq-8YNFJ.1 2024-03-09T15:09:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.88__multipart_key1.2~YkgTabhSvuu4a73MC7Ly-nGAq-8YNFJ.2 2024-03-09T15:09:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.88__multipart_key1.2~YkgTabhSvuu4a73MC7Ly-nGAq-8YNFJ.3 2024-03-09T15:09:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.88__multipart_key1.2~YkgTabhSvuu4a73MC7Ly-nGAq-8YNFJ.4 2024-03-09T15:09:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.88__multipart_key1.2~YkgTabhSvuu4a73MC7Ly-nGAq-8YNFJ.5 2024-03-09T15:09:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7579727266914010759', time=2024-03-09T17:08:34.515679+0000, chain.objs.size()=1 2024-03-09T15:09:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.570+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.102__shadow_.mLDSb3wiNX86tUOJRFcpg7T0xV1XG3M_0 2024-03-09T15:09:46.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.578+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10379404802000149069', time=2024-03-09T17:08:38.692038+0000, chain.objs.size()=1 2024-03-09T15:09:46.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.578+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.105__shadow_.XJaYTWKqS5F9qKlz4KVKuOgPSYyeJ8d_0 2024-03-09T15:09:46.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.578+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10389485379582180479', time=2024-03-09T17:08:39.927975+0000, chain.objs.size()=1 2024-03-09T15:09:46.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.578+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.106__shadow_.MZ63fur0o7VucoZfwTjbMQdv6twjrUq_0 2024-03-09T15:09:46.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.578+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8244734972580910716', time=2024-03-09T17:08:46.508510+0000, chain.objs.size()=1 2024-03-09T15:09:46.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.578+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.111__shadow_.z0AtOlYhQElT3Ixjg31h8hBmY6wSA9d_0 2024-03-09T15:09:46.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.578+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.84941704638060422', time=2024-03-09T17:08:51.401733+0000, chain.objs.size()=1 2024-03-09T15:09:46.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.578+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.115__shadow_.QHnCttSEsfDghMqj8nJ_obdEl_jzHch_0 2024-03-09T15:09:46.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.582+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.590+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T15:09:46.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.594+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.594+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.594+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2024-03-09T15:09:46.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.594+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9313569824014523347', time=2024-03-09T17:08:45.089784+0000, chain.objs.size()=1 2024-03-09T15:09:46.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.594+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.110__shadow_.UwzAa0tyNziTRogbPcNABDqc3Pk8pZE_0 2024-03-09T15:09:46.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.594+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.256634726646007082', time=2024-03-09T17:08:46.352875+0000, chain.objs.size()=1 2024-03-09T15:09:46.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.594+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.111__shadow_.i-aqjaXHbxrhc1iCwJWMdqnhUmHe5wo_0 2024-03-09T15:09:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.598+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.606+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T15:09:46.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.610+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.610+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.610+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:09:46.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.610+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15824995992142563232', time=2024-03-09T17:08:38.846077+0000, chain.objs.size()=1 2024-03-09T15:09:46.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.610+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.105__shadow_.98I3DSCa98QMdy-3s_cPGUAGBbt36ux_0 2024-03-09T15:09:46.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.614+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.622+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T15:09:46.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.622+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2024-03-09T15:09:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9643214809353497430', time=2024-03-09T17:04:40.443764+0000, chain.objs.size()=4 2024-03-09T15:09:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.82__shadow_.ycddVoe2ke1doAlCQhJyttHhy3jKQqJ_0 2024-03-09T15:09:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.82__shadow_.ycddVoe2ke1doAlCQhJyttHhy3jKQqJ_1 2024-03-09T15:09:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.82__shadow_.ycddVoe2ke1doAlCQhJyttHhy3jKQqJ_2 2024-03-09T15:09:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.82__shadow_.ycddVoe2ke1doAlCQhJyttHhy3jKQqJ_3 2024-03-09T15:09:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13886566435623352675', time=2024-03-09T17:04:55.598583+0000, chain.objs.size()=5 2024-03-09T15:09:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.87__multipart_key1.2~CJFrAfcrQ8_HSstuQOsSmOprKrV9WlV.1 2024-03-09T15:09:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.87__multipart_key1.2~CJFrAfcrQ8_HSstuQOsSmOprKrV9WlV.2 2024-03-09T15:09:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.87__multipart_key1.2~CJFrAfcrQ8_HSstuQOsSmOprKrV9WlV.3 2024-03-09T15:09:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.87__multipart_key1.2~CJFrAfcrQ8_HSstuQOsSmOprKrV9WlV.4 2024-03-09T15:09:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.87__multipart_key1.2~CJFrAfcrQ8_HSstuQOsSmOprKrV9WlV.5 2024-03-09T15:09:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13993977950755138456', time=2024-03-09T17:08:53.981110+0000, chain.objs.size()=1 2024-03-09T15:09:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.117__shadow_.JYTIfW0Pziao-j8tM6HAOLkB1CwBZA3_0 2024-03-09T15:09:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15190836353089743367', time=2024-03-09T17:08:59.799338+0000, chain.objs.size()=1 2024-03-09T15:09:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.626+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.122__shadow_.mY3wHfWha5YdUpxyTDChQ7WBJClOsUY_0 2024-03-09T15:09:46.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.638+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.642+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T15:09:46.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.646+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2024-03-09T15:09:46.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14024761495718091357', time=2024-03-09T17:07:37.467320+0000, chain.objs.size()=5 2024-03-09T15:09:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~Nlb83R1tdjJGYdsw7rJeOWpGO6hhgTt.1 2024-03-09T15:09:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__shadow_key1.2~Nlb83R1tdjJGYdsw7rJeOWpGO6hhgTt.1_1 2024-03-09T15:09:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~Nlb83R1tdjJGYdsw7rJeOWpGO6hhgTt.2 2024-03-09T15:09:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__shadow_key1.2~Nlb83R1tdjJGYdsw7rJeOWpGO6hhgTt.2_1 2024-03-09T15:09:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~Nlb83R1tdjJGYdsw7rJeOWpGO6hhgTt.3 2024-03-09T15:09:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16072117188701153554', time=2024-03-09T17:08:35.800719+0000, chain.objs.size()=1 2024-03-09T15:09:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.103__shadow_.coHB3v2zpu2MUnnKQ7aU4CLH3szRXL0_0 2024-03-09T15:09:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7206248524535944150', time=2024-03-09T17:08:35.876522+0000, chain.objs.size()=1 2024-03-09T15:09:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.103__shadow_.ZsEcwQXlXcXg66p2_PlruUrlSUYfN2E_0 2024-03-09T15:09:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7074848258954939362', time=2024-03-09T17:08:41.406821+0000, chain.objs.size()=1 2024-03-09T15:09:46.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.650+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.107__shadow_.Okr9QaWnl6phvW7KWrswMhQUWWxCHpK_0 2024-03-09T15:09:46.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.658+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.666+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T15:09:46.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.670+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.670+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2024-03-09T15:09:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7592643350406856910', time=2024-03-09T17:04:30.590412+0000, chain.objs.size()=5 2024-03-09T15:09:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~gZqsAmUCHtxT1Nc0cJ95QOyMdH63IrT.1 2024-03-09T15:09:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~gZqsAmUCHtxT1Nc0cJ95QOyMdH63IrT.2 2024-03-09T15:09:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~gZqsAmUCHtxT1Nc0cJ95QOyMdH63IrT.3 2024-03-09T15:09:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~gZqsAmUCHtxT1Nc0cJ95QOyMdH63IrT.4 2024-03-09T15:09:46.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~gZqsAmUCHtxT1Nc0cJ95QOyMdH63IrT.5 2024-03-09T15:09:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16638973403337675587', time=2024-03-09T17:08:38.766540+0000, chain.objs.size()=1 2024-03-09T15:09:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.105__shadow_.Z0D-atPr1Ul6Yvca1bVipCmTv5RESJ0_0 2024-03-09T15:09:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9429899363063040858', time=2024-03-09T17:08:40.080593+0000, chain.objs.size()=1 2024-03-09T15:09:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.106__shadow_.DfV9fjIkk4jGEMb4DN1h5ug4s3xc6LN_0 2024-03-09T15:09:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17330872657291304047', time=2024-03-09T17:08:41.169816+0000, chain.objs.size()=1 2024-03-09T15:09:46.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.107__shadow_.02d2qbn41800DAPMi1pfgNa0paEBNdw_0 2024-03-09T15:09:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8253588814151603635', time=2024-03-09T17:08:41.321571+0000, chain.objs.size()=1 2024-03-09T15:09:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.674+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.107__shadow_.BcAp3qG47mXPNMWcuDqAQWvV0iFlbeV_0 2024-03-09T15:09:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.682+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.690+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T15:09:46.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2024-03-09T15:09:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1969390714802146535', time=2024-03-09T17:04:25.763225+0000, chain.objs.size()=5 2024-03-09T15:09:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~qRq_l0mHg1ETzeD0liAoGgvpbtFNQSN.1 2024-03-09T15:09:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~qRq_l0mHg1ETzeD0liAoGgvpbtFNQSN.2 2024-03-09T15:09:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~qRq_l0mHg1ETzeD0liAoGgvpbtFNQSN.3 2024-03-09T15:09:46.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~qRq_l0mHg1ETzeD0liAoGgvpbtFNQSN.4 2024-03-09T15:09:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~qRq_l0mHg1ETzeD0liAoGgvpbtFNQSN.5 2024-03-09T15:09:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12677846696790943433', time=2024-03-09T17:04:40.059448+0000, chain.objs.size()=5 2024-03-09T15:09:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~a85bEfqUBPM2slbQcdxjKbalgm7Zh7M.1 2024-03-09T15:09:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__shadow_key1.2~a85bEfqUBPM2slbQcdxjKbalgm7Zh7M.1_1 2024-03-09T15:09:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~a85bEfqUBPM2slbQcdxjKbalgm7Zh7M.2 2024-03-09T15:09:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__shadow_key1.2~a85bEfqUBPM2slbQcdxjKbalgm7Zh7M.2_1 2024-03-09T15:09:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.83__multipart_key1.2~a85bEfqUBPM2slbQcdxjKbalgm7Zh7M.3 2024-03-09T15:09:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9358392178592237547', time=2024-03-09T17:08:36.031941+0000, chain.objs.size()=1 2024-03-09T15:09:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.103__shadow_.eidtpxFzokwS3tPSpnhKv5patqCHwLa_0 2024-03-09T15:09:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12859033281427061247', time=2024-03-09T17:09:21.292404+0000, chain.objs.size()=5 2024-03-09T15:09:46.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.694+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.132__multipart_key1.2~3duLSsvB50PI-t0UqcK7H_C_KXgUcrk.1 2024-03-09T15:09:46.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.706+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.132__multipart_key1.2~3duLSsvB50PI-t0UqcK7H_C_KXgUcrk.2 2024-03-09T15:09:46.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.706+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.132__multipart_key1.2~3duLSsvB50PI-t0UqcK7H_C_KXgUcrk.3 2024-03-09T15:09:46.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.706+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.132__multipart_key1.2~3duLSsvB50PI-t0UqcK7H_C_KXgUcrk.4 2024-03-09T15:09:46.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.706+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.132__multipart_key1.2~3duLSsvB50PI-t0UqcK7H_C_KXgUcrk.5 2024-03-09T15:09:46.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.714+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.722+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T15:09:46.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.726+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.726+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2024-03-09T15:09:46.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.288626320071356468', time=2024-03-09T17:03:36.687567+0000, chain.objs.size()=1 2024-03-09T15:09:46.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.74__shadow_._wcyWVo77fYOAi1fGmdfgcrhTCM4fGq_0 2024-03-09T15:09:46.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3727372461201112138', time=2024-03-09T17:07:37.179985+0000, chain.objs.size()=5 2024-03-09T15:09:46.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~8VDwRSobjozaXIsF4TJIJiZgU9wP3Uf.1 2024-03-09T15:09:46.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__shadow_key1.2~8VDwRSobjozaXIsF4TJIJiZgU9wP3Uf.1_1 2024-03-09T15:09:46.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~8VDwRSobjozaXIsF4TJIJiZgU9wP3Uf.2 2024-03-09T15:09:46.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__shadow_key1.2~8VDwRSobjozaXIsF4TJIJiZgU9wP3Uf.2_1 2024-03-09T15:09:46.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~8VDwRSobjozaXIsF4TJIJiZgU9wP3Uf.3 2024-03-09T15:09:46.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17841670747734316868', time=2024-03-09T17:08:44.028834+0000, chain.objs.size()=1 2024-03-09T15:09:46.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.109__shadow_.8NBDZfTx9d-1e9o1Kq2XX20wr4n45hm_0 2024-03-09T15:09:46.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18403678063115212268', time=2024-03-09T17:09:02.708072+0000, chain.objs.size()=1 2024-03-09T15:09:46.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.730+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.126__shadow_.AL1VVgnOlESllXGPqUuYLPXKbn0Dh-t_0 2024-03-09T15:09:46.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.738+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.746+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T15:09:46.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.746+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.750+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.750+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2024-03-09T15:09:46.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.750+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10868778126537448504', time=2024-03-09T17:08:37.545303+0000, chain.objs.size()=1 2024-03-09T15:09:46.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.750+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.104__shadow_.o5z5KZJ1imq7NOBuBJCkvQwIrvp-etB_0 2024-03-09T15:09:46.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.750+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14436271163293348123', time=2024-03-09T17:08:46.431798+0000, chain.objs.size()=1 2024-03-09T15:09:46.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.750+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.111__shadow_.L0fD2ydc6GUaXSAkXlBtCRdJTJGdaGU_0 2024-03-09T15:09:46.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.750+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3602804255709202153', time=2024-03-09T17:08:49.910235+0000, chain.objs.size()=1 2024-03-09T15:09:46.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.750+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.114__shadow_.Mb1grsnvAH7vOtSvCIjwGDpUUW8QNMP_0 2024-03-09T15:09:46.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.750+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9831509959794553774', time=2024-03-09T17:08:57.539065+0000, chain.objs.size()=1 2024-03-09T15:09:46.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.750+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.120__shadow_.yaLt30UHE2zwzxf-GegVQ7221EkUHVZ_0 2024-03-09T15:09:46.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.754+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.758+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T15:09:46.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.762+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.766+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.766+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2024-03-09T15:09:46.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.766+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7491256287016273063', time=2024-03-09T17:08:42.498603+0000, chain.objs.size()=1 2024-03-09T15:09:46.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.766+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.108__shadow_.GZG1eZMHj7nfruV-F5D4TVLC1sbrRMK_0 2024-03-09T15:09:46.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.766+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17773474278812087209', time=2024-03-09T17:08:56.209119+0000, chain.objs.size()=1 2024-03-09T15:09:46.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.766+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.119__shadow_.jo5eGtDoBa5Qc7DaphN4sW9jQkh-Swu_0 2024-03-09T15:09:46.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.766+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1182489307150172212', time=2024-03-09T17:08:57.366610+0000, chain.objs.size()=1 2024-03-09T15:09:46.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.766+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.120__shadow_.ihN7XEnZuLQSMcgoJiNLU3NsOBm6-ql_0 2024-03-09T15:09:46.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.770+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.774+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T15:09:46.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.778+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.782+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.782+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:09:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.782+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10365054579049680528', time=2024-03-09T17:09:45.714245+0000, chain.objs.size()=5 2024-03-09T15:09:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.782+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.133__multipart_key1/data/file.mpg.2~v1CMY-8hIEB5vIPsSfU8-vDdA801PeN.1 2024-03-09T15:09:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.782+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.133__multipart_key1/data/file.mpg.2~v1CMY-8hIEB5vIPsSfU8-vDdA801PeN.2 2024-03-09T15:09:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.782+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.133__multipart_key1/data/file.mpg.2~v1CMY-8hIEB5vIPsSfU8-vDdA801PeN.3 2024-03-09T15:09:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.782+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.133__multipart_key1/data/file.mpg.2~v1CMY-8hIEB5vIPsSfU8-vDdA801PeN.4 2024-03-09T15:09:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.782+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.133__multipart_key1/data/file.mpg.2~v1CMY-8hIEB5vIPsSfU8-vDdA801PeN.5 2024-03-09T15:09:46.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.794+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.798+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T15:09:46.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.802+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2024-03-09T15:09:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15550821106718008389', time=2024-03-09T17:07:00.353191+0000, chain.objs.size()=5 2024-03-09T15:09:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~FbWHXnvcORPd_zfiYhaxbpZyEt0X_wa.1 2024-03-09T15:09:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~FbWHXnvcORPd_zfiYhaxbpZyEt0X_wa.2 2024-03-09T15:09:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~FbWHXnvcORPd_zfiYhaxbpZyEt0X_wa.3 2024-03-09T15:09:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~FbWHXnvcORPd_zfiYhaxbpZyEt0X_wa.4 2024-03-09T15:09:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~FbWHXnvcORPd_zfiYhaxbpZyEt0X_wa.5 2024-03-09T15:09:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6563987139373630991', time=2024-03-09T17:08:41.246170+0000, chain.objs.size()=1 2024-03-09T15:09:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.107__shadow_.THO4VRLoAF7B7zdytv6Vpav5ueAkIRY_0 2024-03-09T15:09:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14201741859264184554', time=2024-03-09T17:08:45.303011+0000, chain.objs.size()=1 2024-03-09T15:09:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.110__shadow_.6Ggi0TZiKw91K-bueJPbg3L5MICe8EP_0 2024-03-09T15:09:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13194592723534000992', time=2024-03-09T17:08:56.283762+0000, chain.objs.size()=1 2024-03-09T15:09:46.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.806+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.119__shadow_.NukNaAz7Mmj07Wr14rDXnZUuXSE_VY8_0 2024-03-09T15:09:46.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.814+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.822+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T15:09:46.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.826+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.826+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2024-03-09T15:09:46.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2581362084965647053', time=2024-03-09T17:03:08.717420+0000, chain.objs.size()=1 2024-03-09T15:09:46.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.66__shadow_.ewxg84hsfQy-iQMIK-1yROGybhkozcA_0 2024-03-09T15:09:46.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13304378414234236730', time=2024-03-09T17:03:35.012027+0000, chain.objs.size()=2 2024-03-09T15:09:46.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.72__shadow_.OQJxm-zVmyuiThqhMLLBOMTeUqtn63J_0 2024-03-09T15:09:46.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.72__shadow_.OQJxm-zVmyuiThqhMLLBOMTeUqtn63J_1 2024-03-09T15:09:46.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.80__multipart_key1.2~8Z-u_58hALBsHcFa4a2iY4D0kC-9bXi.2_16735866ca3ffa39039e034a5a846999', time=2024-03-09T17:03:54.660764+0000, chain.objs.size()=5 2024-03-09T15:09:46.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.80__multipart_key1.2~8Z-u_58hALBsHcFa4a2iY4D0kC-9bXi.1 2024-03-09T15:09:46.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.80__multipart_key1.2~8Z-u_58hALBsHcFa4a2iY4D0kC-9bXi.2 2024-03-09T15:09:46.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.80__multipart_key1.2~8Z-u_58hALBsHcFa4a2iY4D0kC-9bXi.3 2024-03-09T15:09:46.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.80__multipart_key1.2~8Z-u_58hALBsHcFa4a2iY4D0kC-9bXi.4 2024-03-09T15:09:46.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.80__multipart_key1.2~8Z-u_58hALBsHcFa4a2iY4D0kC-9bXi.5 2024-03-09T15:09:46.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1451223233886642457', time=2024-03-09T17:08:42.729690+0000, chain.objs.size()=1 2024-03-09T15:09:46.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.108__shadow_.RYALluPxSWQaIYWZkBY21xGF7DyhkIs_0 2024-03-09T15:09:46.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17377655716290215784', time=2024-03-09T17:09:05.674398+0000, chain.objs.size()=1 2024-03-09T15:09:46.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.830+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.129__shadow_.ULPEl5FD-vm0N2-fkEem2a899hI35pT_0 2024-03-09T15:09:46.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.834+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.842+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T15:09:46.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.846+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.846+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.850+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2024-03-09T15:09:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.850+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16903603132435659145', time=2024-03-09T17:03:35.827914+0000, chain.objs.size()=1 2024-03-09T15:09:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.850+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.73__shadow_.IHm9rHlC7q-GNSQuH-_fFKMCy4zXKbY_0 2024-03-09T15:09:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.850+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2252266780285174627', time=2024-03-09T17:08:28.699753+0000, chain.objs.size()=1 2024-03-09T15:09:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.850+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.95__shadow_.R-KVfPbtbeK_jB9mmsTP5rgDRY_a5op_0 2024-03-09T15:09:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.850+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5352026311252352070', time=2024-03-09T17:08:52.618498+0000, chain.objs.size()=1 2024-03-09T15:09:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.850+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.116__shadow_.R0zPbWVtkjvTIf1t8kE2droUNkRHqKZ_0 2024-03-09T15:09:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.850+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1335816035545253405', time=2024-03-09T17:08:56.359739+0000, chain.objs.size()=1 2024-03-09T15:09:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.850+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.119__shadow_.jkJ8AgwtrhTHWiy9N5MU-_yVz05rC1t_0 2024-03-09T15:09:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.850+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7709981946230795020', time=2024-03-09T17:09:03.405111+0000, chain.objs.size()=1 2024-03-09T15:09:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.850+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.127__shadow_.O_zRz_r0FpQkiCgUvJDxsoLO5E-8Qzg_0 2024-03-09T15:09:46.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.854+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.858+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T15:09:46.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.862+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.866+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.866+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:09:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.866+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12621120850281549319', time=2024-03-09T17:08:43.871191+0000, chain.objs.size()=1 2024-03-09T15:09:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.866+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.109__shadow_.ZQHAOC4azK6iGStSWpYF662ms8FJ4Ki_0 2024-03-09T15:09:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.870+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.874+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T15:09:46.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.878+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.878+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=10, truncated=0, next_marker='' 2024-03-09T15:09:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='2~HHulnbAar9O3enNKqA6BcBx1RYVAbM9', time=2024-03-09T17:03:22.223209+0000, chain.objs.size()=5 2024-03-09T15:09:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.68__multipart_key1.2~HHulnbAar9O3enNKqA6BcBx1RYVAbM9.1 2024-03-09T15:09:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.68__multipart_key1.2~HHulnbAar9O3enNKqA6BcBx1RYVAbM9.2 2024-03-09T15:09:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.68__multipart_key1.2~HHulnbAar9O3enNKqA6BcBx1RYVAbM9.3 2024-03-09T15:09:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.68__multipart_key1.2~HHulnbAar9O3enNKqA6BcBx1RYVAbM9.4 2024-03-09T15:09:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.68__multipart_key1.2~HHulnbAar9O3enNKqA6BcBx1RYVAbM9.5 2024-03-09T15:09:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4285238156600593261', time=2024-03-09T17:08:08.554494+0000, chain.objs.size()=5 2024-03-09T15:09:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.93__multipart_key1.2~ZzmRAYBoOXlMMC9IX3EolnjOLSid6w0.1 2024-03-09T15:09:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.93__multipart_key1.2~ZzmRAYBoOXlMMC9IX3EolnjOLSid6w0.2 2024-03-09T15:09:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.93__multipart_key1.2~ZzmRAYBoOXlMMC9IX3EolnjOLSid6w0.3 2024-03-09T15:09:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.93__multipart_key1.2~ZzmRAYBoOXlMMC9IX3EolnjOLSid6w0.4 2024-03-09T15:09:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.93__multipart_key1.2~ZzmRAYBoOXlMMC9IX3EolnjOLSid6w0.5 2024-03-09T15:09:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12923644590685325213', time=2024-03-09T17:08:38.615115+0000, chain.objs.size()=1 2024-03-09T15:09:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.105__shadow_.di2Ws6foLx1tuOMxr6kleNRNqkHpD1p_0 2024-03-09T15:09:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10535830414351602735', time=2024-03-09T17:08:40.003026+0000, chain.objs.size()=1 2024-03-09T15:09:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.882+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.106__shadow_.crAHbEduXTENa25ZUJyRMBqxEvtjfm2_0 2024-03-09T15:09:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.894+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15846625565166496034', time=2024-03-09T17:08:42.573535+0000, chain.objs.size()=1 2024-03-09T15:09:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.894+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.108__shadow_.xjxZFrfKmTDHc8ENEk2XDmwsXbC1vm4_0 2024-03-09T15:09:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.894+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8696210246364065722', time=2024-03-09T17:08:45.224395+0000, chain.objs.size()=1 2024-03-09T15:09:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.894+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.110__shadow_.cgQX2MrBqLVZBQzduCntLYpAt6yufl-_0 2024-03-09T15:09:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.894+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3217586869976729269', time=2024-03-09T17:08:46.574937+0000, chain.objs.size()=1 2024-03-09T15:09:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.894+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.111__shadow_.BLna8Hg9UVMVVWUkP77EOlSTedFduu5_0 2024-03-09T15:09:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.894+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9743892177531842490', time=2024-03-09T17:08:47.550626+0000, chain.objs.size()=1 2024-03-09T15:09:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.894+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.112__shadow_.7_jrtF138RUxNRTui3Q4vxiZDPQiKgG_0 2024-03-09T15:09:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.894+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7684218748369651266', time=2024-03-09T17:08:55.133903+0000, chain.objs.size()=1 2024-03-09T15:09:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.894+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.118__shadow_.EfAovG53T5kin0vgi9L-CL2KqIixsIP_0 2024-03-09T15:09:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.894+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3213468649160267686', time=2024-03-09T17:09:45.792695+0000, chain.objs.size()=1 2024-03-09T15:09:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.894+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.133__shadow_.bvlG6rpDIeJZCXyLR2-k6l9Gw5Du8SL_0 2024-03-09T15:09:46.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.898+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.906+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T15:09:46.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.910+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.910+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.910+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-03-09T15:09:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.910+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5018611146200366820', time=2024-03-09T17:08:37.361374+0000, chain.objs.size()=1 2024-03-09T15:09:46.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.910+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.104__shadow_.PbaeQNGfuSWsGmfkibuRpLFAgGcx4N4_0 2024-03-09T15:09:46.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.914+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.922+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T15:09:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.926+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.926+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.926+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2024-03-09T15:09:46.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.926+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10022482057597961251', time=2024-03-09T17:08:34.750330+0000, chain.objs.size()=1 2024-03-09T15:09:46.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.926+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.102__shadow_.ftynbJmjen6FO9adoK5bg9ixRZMwUHO_0 2024-03-09T15:09:46.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.926+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9401051493403522585', time=2024-03-09T17:08:58.666389+0000, chain.objs.size()=1 2024-03-09T15:09:46.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.926+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.121__shadow_.J0dP6W_Oe9zwdzfU2aLW4jXAhF9eSow_0 2024-03-09T15:09:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.930+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.934+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T15:09:46.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.938+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2024-03-09T15:09:46.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5712860414194848534', time=2024-03-09T17:05:57.816427+0000, chain.objs.size()=1 2024-03-09T15:09:46.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.89__shadow_.ZH_ueaP-XjYqTrbTkXTYcQE0Z7BfUq0_0 2024-03-09T15:09:46.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11179786325548981793', time=2024-03-09T17:06:47.139870+0000, chain.objs.size()=5 2024-03-09T15:09:46.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~vI_yBU1tn18GPYilIKVrPLwVpxdopgh.1 2024-03-09T15:09:46.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~vI_yBU1tn18GPYilIKVrPLwVpxdopgh.2 2024-03-09T15:09:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~vI_yBU1tn18GPYilIKVrPLwVpxdopgh.3 2024-03-09T15:09:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~vI_yBU1tn18GPYilIKVrPLwVpxdopgh.4 2024-03-09T15:09:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.91__multipart_key1.2~vI_yBU1tn18GPYilIKVrPLwVpxdopgh.5 2024-03-09T15:09:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17278342423219696730', time=2024-03-09T17:08:43.794501+0000, chain.objs.size()=1 2024-03-09T15:09:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.109__shadow_.pJDm6PLW8lb1xGimDwca_eWL8zQJb6F_0 2024-03-09T15:09:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14320497678684659967', time=2024-03-09T17:08:51.229152+0000, chain.objs.size()=1 2024-03-09T15:09:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.115__shadow_.utgiGz7sjnoM3f6cotnwkeGR03qcRV6_0 2024-03-09T15:09:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3803314972847785177', time=2024-03-09T17:08:58.588494+0000, chain.objs.size()=1 2024-03-09T15:09:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.121__shadow_.BIKltEAQOIPYIWwM-InXs13p0IU9rJ5_0 2024-03-09T15:09:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5340560939478107304', time=2024-03-09T17:09:01.862408+0000, chain.objs.size()=1 2024-03-09T15:09:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.942+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.125__shadow_.6nuxXsDT5JWsPL0L399ZfGft1a1fZnx_0 2024-03-09T15:09:46.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.950+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.958+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T15:09:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.962+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.962+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.962+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2024-03-09T15:09:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.962+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3865422341675209402', time=2024-03-09T17:07:55.483250+0000, chain.objs.size()=5 2024-03-09T15:09:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.962+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.92__multipart_key1.2~EIxeT1AvZZ5Www3i-DvgT1c4sluZwHB.1 2024-03-09T15:09:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.962+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.92__multipart_key1.2~EIxeT1AvZZ5Www3i-DvgT1c4sluZwHB.2 2024-03-09T15:09:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.962+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.92__multipart_key1.2~EIxeT1AvZZ5Www3i-DvgT1c4sluZwHB.3 2024-03-09T15:09:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.962+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.92__multipart_key1.2~EIxeT1AvZZ5Www3i-DvgT1c4sluZwHB.4 2024-03-09T15:09:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.962+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.92__multipart_key1.2~EIxeT1AvZZ5Www3i-DvgT1c4sluZwHB.5 2024-03-09T15:09:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.962+0000 7ffa5b6f0980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9657372010668215813', time=2024-03-09T17:08:49.985290+0000, chain.objs.size()=1 2024-03-09T15:09:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.962+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.114__shadow_.Q7dh1HsA-OvXU8PpOg1fioqUcMrjE46_0 2024-03-09T15:09:46.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.974+0000 7ffa5b6f0980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:09:46.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.986+0000 7ff90f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:46.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.990+0000 7ffa5b6f0980 20 remove_watcher() i=0 2024-03-09T15:09:46.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.990+0000 7ffa5b6f0980 2 removed watcher, disabling cache 2024-03-09T15:09:46.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.994+0000 7ffa5b6f0980 20 remove_watcher() i=1 2024-03-09T15:09:46.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.994+0000 7ffa5b6f0980 20 remove_watcher() i=2 2024-03-09T15:09:47.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:46.998+0000 7ffa5b6f0980 20 remove_watcher() i=3 2024-03-09T15:09:47.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.002+0000 7ffa5b6f0980 20 remove_watcher() i=4 2024-03-09T15:09:47.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.002+0000 7ffa5b6f0980 20 remove_watcher() i=5 2024-03-09T15:09:47.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.006+0000 7ffa5b6f0980 20 remove_watcher() i=6 2024-03-09T15:09:47.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.006+0000 7ffa5b6f0980 20 remove_watcher() i=7 2024-03-09T15:09:47.036 DEBUG:teuthology.orchestra.run.smithi001:> radosgw-admin gc process --include-all 2024-03-09T15:09:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:47.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.154+0000 7fd4ab1e6980 20 rados->read ofs=0 len=0 2024-03-09T15:09:47.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.154+0000 7fd4ab1e6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:47.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.154+0000 7fd4ab1e6980 20 realm 2024-03-09T15:09:47.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.154+0000 7fd4ab1e6980 20 rados->read ofs=0 len=0 2024-03-09T15:09:47.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.158+0000 7fd4ab1e6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:47.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.158+0000 7fd4ab1e6980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:47.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.158+0000 7fd4ab1e6980 20 rados->read ofs=0 len=0 2024-03-09T15:09:47.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.158+0000 7fd4ab1e6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:47.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.158+0000 7fd4ab1e6980 20 rados->read ofs=0 len=0 2024-03-09T15:09:47.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.158+0000 7fd4ab1e6980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:47.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.158+0000 7fd4ab1e6980 20 rados->read ofs=0 len=0 2024-03-09T15:09:47.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.158+0000 7fd4ab1e6980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:47.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.158+0000 7fd4ab1e6980 20 searching for the correct realm 2024-03-09T15:09:47.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.162+0000 7fd4a8cfe640 0 -- 172.21.15.1:0/91913079 >> [v2:172.21.15.1:6824/1666056338,v1:172.21.15.1:6825/1666056338] conn(0x5585b54253d0 msgr2=0x7fd4a00f00e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:47.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.170+0000 7fd4a94ff640 0 -- 172.21.15.1:0/91913079 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7fd49c018070 msgr2=0x7fd49c03a970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:47.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 rados->read ofs=0 len=0 2024-03-09T15:09:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 rados->read ofs=0 len=0 2024-03-09T15:09:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 rados->read ofs=0 len=0 2024-03-09T15:09:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 zone default found 2024-03-09T15:09:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 4 Realm: () 2024-03-09T15:09:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 zonegroup default 2024-03-09T15:09:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 rados->read ofs=0 len=0 2024-03-09T15:09:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 rados->read ofs=0 len=0 2024-03-09T15:09:47.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:47.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 rados->read ofs=0 len=0 2024-03-09T15:09:47.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:47.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 started sync module instance, tier type = 2024-03-09T15:09:47.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.182+0000 7fd4ab1e6980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:47.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.198+0000 7fd4ab1e6980 20 add_watcher() i=0 2024-03-09T15:09:47.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.198+0000 7fd4ab1e6980 20 add_watcher() i=1 2024-03-09T15:09:47.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.198+0000 7fd4ab1e6980 20 add_watcher() i=2 2024-03-09T15:09:47.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.198+0000 7fd4ab1e6980 20 add_watcher() i=3 2024-03-09T15:09:47.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.198+0000 7fd4ab1e6980 20 add_watcher() i=4 2024-03-09T15:09:47.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.198+0000 7fd4ab1e6980 20 add_watcher() i=5 2024-03-09T15:09:47.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.198+0000 7fd4ab1e6980 20 add_watcher() i=6 2024-03-09T15:09:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.202+0000 7fd4ab1e6980 20 add_watcher() i=7 2024-03-09T15:09:47.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.202+0000 7fd4ab1e6980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.202+0000 7fd35efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.202+0000 7fd4ab1e6980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:47.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.202+0000 7fd4ab1e6980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:47.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.286+0000 7fd35dfe3640 20 reqs_thread_entry: start 2024-03-09T15:09:47.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.286+0000 7fd4ab1e6980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:47.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.286+0000 7fd4ab1e6980 20 Filter name: none 2024-03-09T15:09:47.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.286+0000 7fd2cf7fe640 20 reqs_thread_entry: start 2024-03-09T15:09:47.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.286+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T15:09:47.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.290+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.290+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.290+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.290+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T15:09:47.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.294+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.294+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.298+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.298+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T15:09:47.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.302+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.302+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.302+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.306+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T15:09:47.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.306+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.306+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.310+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.310+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T15:09:47.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.314+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.314+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.314+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.318+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T15:09:47.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.322+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.326+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.326+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.330+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T15:09:47.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.330+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.334+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.334+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.334+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T15:09:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.338+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.342+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.342+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.346+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T15:09:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.350+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.350+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.350+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.354+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T15:09:47.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.354+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.358+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.358+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.362+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T15:09:47.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.366+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.366+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.366+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.370+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T15:09:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.374+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.374+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.378+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.378+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T15:09:47.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.382+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.386+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.386+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.390+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T15:09:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.394+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.394+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.394+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.398+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T15:09:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.402+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.402+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.402+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.406+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T15:09:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.410+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.410+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.410+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.414+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T15:09:47.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.418+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.422+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.422+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.426+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T15:09:47.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.430+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.430+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.430+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.434+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T15:09:47.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.438+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.438+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.438+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.442+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T15:09:47.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.446+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.446+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.450+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.450+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T15:09:47.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.454+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.458+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.458+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.458+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T15:09:47.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.462+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.466+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.466+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.470+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T15:09:47.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.474+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.474+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.474+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.478+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T15:09:47.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.482+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.482+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.486+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.486+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T15:09:47.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.490+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.490+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.494+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.494+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T15:09:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.498+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.502+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.502+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.506+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T15:09:47.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.506+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.510+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.510+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.514+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T15:09:47.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.518+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.518+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.518+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.522+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T15:09:47.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.526+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.526+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.530+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.530+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T15:09:47.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.534+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.538+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.538+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.542+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T15:09:47.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.546+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.546+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.546+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.550+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T15:09:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.554+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.558+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:09:47.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.558+0000 7fd4ab1e6980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:09:47.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.562+0000 7fd35efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.566+0000 7fd4ab1e6980 20 remove_watcher() i=0 2024-03-09T15:09:47.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.566+0000 7fd4ab1e6980 2 removed watcher, disabling cache 2024-03-09T15:09:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.570+0000 7fd4ab1e6980 20 remove_watcher() i=1 2024-03-09T15:09:47.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.574+0000 7fd4ab1e6980 20 remove_watcher() i=2 2024-03-09T15:09:47.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.574+0000 7fd4ab1e6980 20 remove_watcher() i=3 2024-03-09T15:09:47.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.578+0000 7fd4ab1e6980 20 remove_watcher() i=4 2024-03-09T15:09:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.582+0000 7fd4ab1e6980 20 remove_watcher() i=5 2024-03-09T15:09:47.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.582+0000 7fd4ab1e6980 20 remove_watcher() i=6 2024-03-09T15:09:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:47.586+0000 7fd4ab1e6980 20 remove_watcher() i=7 2024-03-09T15:09:47.608 INFO:teuthology.run_tasks:Running task s3tests... 2024-03-09T15:09:47.633 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'rgw_server': 'client.0', 'accounts': {'iam alt root': 'RGW99999999999999999', 'iam root': 'RGW88888888888888888'}, 'force-branch': 'wip-iam-user-apis', 'git_remote': 'https://github.com/cbodley/', 'storage classes': 'LUKEWARM, FROZEN'}} 2024-03-09T15:09:47.634 INFO:tasks.s3tests:Downloading s3-tests... 2024-03-09T15:09:47.634 INFO:tasks.s3tests:Using branch 'wip-iam-user-apis' for s3tests 2024-03-09T15:09:47.634 DEBUG:teuthology.orchestra.run.smithi001:> git clone -b wip-iam-user-apis https://github.com/cbodley/s3-tests.git /home/ubuntu/cephtest/s3-tests-client.0 2024-03-09T15:09:47.656 INFO:teuthology.orchestra.run.smithi001.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests-client.0'... 2024-03-09T15:09:48.347 INFO:tasks.s3tests:Creating rgw users... 2024-03-09T15:09:48.348 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2024-03-09T15:09:48.348 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name Mr.foo.client.0 --email foo+test@test.test --access-key HQZECAXQMQNDUDILWUVJ --secret zSz2Akynfrjps1iEV8VTD0Q0tTKicqSO4bj0ZQCjFciYIo4WN2Bwlw== --cluster ceph 2024-03-09T15:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:48.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.474+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.478+0000 7f7372da2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.478+0000 7f7372da2980 20 realm 2024-03-09T15:09:48.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.478+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.478+0000 7f7372da2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.478+0000 7f7372da2980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.478+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.478+0000 7f7372da2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.478+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.478+0000 7f7372da2980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:48.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.478+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.482+0000 7f7372da2980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:48.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.482+0000 7f7372da2980 20 searching for the correct realm 2024-03-09T15:09:48.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.486+0000 7f73700b9640 0 -- 172.21.15.1:0/2549977619 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x5613b86fad00 msgr2=0x5613b86fd1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:48.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.494+0000 7f73700b9640 0 -- 172.21.15.1:0/2549977619 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f7360063110 msgr2=0x7f736005bfa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:48.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:48.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:48.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:48.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 zone default found 2024-03-09T15:09:48.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 4 Realm: () 2024-03-09T15:09:48.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:48.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:48.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:48.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 zonegroup default 2024-03-09T15:09:48.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.502+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.506+0000 7f7372da2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.506+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.506+0000 7f7372da2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.506+0000 7f7372da2980 20 started sync module instance, tier type = 2024-03-09T15:09:48.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.506+0000 7f7372da2980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.522+0000 7f7372da2980 20 add_watcher() i=0 2024-03-09T15:09:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.522+0000 7f7372da2980 20 add_watcher() i=1 2024-03-09T15:09:48.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.522+0000 7f7372da2980 20 add_watcher() i=2 2024-03-09T15:09:48.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.522+0000 7f7372da2980 20 add_watcher() i=3 2024-03-09T15:09:48.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.522+0000 7f7372da2980 20 add_watcher() i=4 2024-03-09T15:09:48.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.522+0000 7f7372da2980 20 add_watcher() i=5 2024-03-09T15:09:48.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.522+0000 7f7372da2980 20 add_watcher() i=6 2024-03-09T15:09:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.522+0000 7f7372da2980 20 add_watcher() i=7 2024-03-09T15:09:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.522+0000 7f7372da2980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7226fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:48.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 5 note: GC not initialized 2024-03-09T15:09:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7225fe3640 20 reqs_thread_entry: start 2024-03-09T15:09:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 20 Filter name: none 2024-03-09T15:09:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f71977fe640 20 reqs_thread_entry: start 2024-03-09T15:09:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-03-09T15:09:48.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2024-03-09T15:09:48.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-03-09T15:09:48.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 10 cache get: name=default.rgw.meta+users.email+foo+test@test.test : miss 2024-03-09T15:09:48.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 10 cache put: name=default.rgw.meta+users.email+foo+test@test.test info.flags=0x0 2024-03-09T15:09:48.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 10 adding default.rgw.meta+users.email+foo+test@test.test to cache LRU end 2024-03-09T15:09:48.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 10 cache get: name=default.rgw.meta+users.keys+HQZECAXQMQNDUDILWUVJ : miss 2024-03-09T15:09:48.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 10 cache put: name=default.rgw.meta+users.keys+HQZECAXQMQNDUDILWUVJ info.flags=0x0 2024-03-09T15:09:48.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 10 adding default.rgw.meta+users.keys+HQZECAXQMQNDUDILWUVJ to cache LRU end 2024-03-09T15:09:48.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 10 cache get: name=default.rgw.meta+users.keys+HQZECAXQMQNDUDILWUVJ : hit (negative entry) 2024-03-09T15:09:48.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.526+0000 7f7372da2980 10 cache get: name=default.rgw.meta+users.keys+HQZECAXQMQNDUDILWUVJ : hit (negative entry) 2024-03-09T15:09:48.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.530+0000 7f7372da2980 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-03-09T15:09:48.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.530+0000 7f7372da2980 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-03-09T15:09:48.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.530+0000 7f7372da2980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-03-09T15:09:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.530+0000 7f7247fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562512 cookie 94642698699584 notifier 7127 bl.length()=645 2024-03-09T15:09:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.530+0000 7f7247fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-03-09T15:09:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.530+0000 7f7247fff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-03-09T15:09:48.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.534+0000 7f7372da2980 10 cache put: name=default.rgw.meta+users.email+foo+test@test.test info.flags=0x7 2024-03-09T15:09:48.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.534+0000 7f7372da2980 10 moving default.rgw.meta+users.email+foo+test@test.test to cache LRU end 2024-03-09T15:09:48.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.534+0000 7f7372da2980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.email:foo+test@test.test, ofs0, ns] 2024-03-09T15:09:48.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.534+0000 7f72477fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562513 cookie 94642698701792 notifier 7127 bl.length()=185 2024-03-09T15:09:48.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.534+0000 7f72477fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo+test@test.test info.flags=0x7 2024-03-09T15:09:48.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.534+0000 7f72477fe640 10 rgw watcher librados: moving default.rgw.meta+users.email+foo+test@test.test to cache LRU end 2024-03-09T15:09:48.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.538+0000 7f7372da2980 10 cache put: name=default.rgw.meta+users.keys+HQZECAXQMQNDUDILWUVJ info.flags=0x7 2024-03-09T15:09:48.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.538+0000 7f7372da2980 10 moving default.rgw.meta+users.keys+HQZECAXQMQNDUDILWUVJ to cache LRU end 2024-03-09T15:09:48.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.538+0000 7f7372da2980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.keys:HQZECAXQMQNDUDILWUVJ, ofs0, ns] 2024-03-09T15:09:48.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.538+0000 7f72477fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1108101562526 cookie 94642698762992 notifier 7127 bl.length()=186 2024-03-09T15:09:48.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.538+0000 7f72477fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+HQZECAXQMQNDUDILWUVJ info.flags=0x7 2024-03-09T15:09:48.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.538+0000 7f72477fe640 10 rgw watcher librados: moving default.rgw.meta+users.keys+HQZECAXQMQNDUDILWUVJ to cache LRU end 2024-03-09T15:09:48.541 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:09:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "user_id": "foo.client.0", 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "display_name": "Mr.foo.client.0", 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "email": "foo+test@test.test", 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "suspended": 0, 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "max_buckets": 1000, 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "subusers": [], 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "keys": [ 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "user": "foo.client.0", 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "access_key": "HQZECAXQMQNDUDILWUVJ", 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "secret_key": "zSz2Akynfrjps1iEV8VTD0Q0tTKicqSO4bj0ZQCjFciYIo4WN2Bwlw==", 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "active": true, 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:48.532543Z" 2024-03-09T15:09:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "swift_keys": [], 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": [], 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "default_placement": "", 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "default_storage_class": "", 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "placement_tags": [], 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "bucket_quota": { 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "user_quota": { 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "temp_url_keys": [], 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "rgw", 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "mfa_ids": [], 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "account_id": "", 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "path": "/", 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:48.532519Z", 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-03-09T15:09:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "group_ids": [] 2024-03-09T15:09:48.545 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:09:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:48.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.542+0000 7f7226fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:48.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.546+0000 7f7372da2980 20 remove_watcher() i=0 2024-03-09T15:09:48.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.546+0000 7f7372da2980 2 removed watcher, disabling cache 2024-03-09T15:09:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.546+0000 7f7372da2980 20 remove_watcher() i=1 2024-03-09T15:09:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.550+0000 7f7372da2980 20 remove_watcher() i=2 2024-03-09T15:09:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.554+0000 7f7372da2980 20 remove_watcher() i=3 2024-03-09T15:09:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.554+0000 7f7372da2980 20 remove_watcher() i=4 2024-03-09T15:09:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.554+0000 7f7372da2980 20 remove_watcher() i=5 2024-03-09T15:09:48.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.558+0000 7f7372da2980 20 remove_watcher() i=6 2024-03-09T15:09:48.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.558+0000 7f7372da2980 20 remove_watcher() i=7 2024-03-09T15:09:48.577 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 5097961610 --totp-seed SZTDBN6UNU5A5GCGPY5QNTTVIG2PBMWQJVLHBI4C6O6I2NH5QOK54E6WSAP7NGPE --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-03-09T15:09:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:48.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.654+0000 7f5c56818980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.658+0000 7f5c56818980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.658+0000 7f5c56818980 20 realm 2024-03-09T15:09:48.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.658+0000 7f5c56818980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.658+0000 7f5c56818980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.658+0000 7f5c56818980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:48.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.658+0000 7f5c56818980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.658+0000 7f5c56818980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.658+0000 7f5c56818980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.658+0000 7f5c56818980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:48.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.658+0000 7f5c56818980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.658+0000 7f5c56818980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:48.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.658+0000 7f5c56818980 20 searching for the correct realm 2024-03-09T15:09:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.666+0000 7f5c53b2f640 0 -- 172.21.15.1:0/3177577622 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x559ef349c360 msgr2=0x559ef349e820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:48.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.674+0000 7f5c54330640 0 -- 172.21.15.1:0/3177577622 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f5c3c03b920 msgr2=0x7f5c3c019410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:48.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:48.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 zone default found 2024-03-09T15:09:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 4 Realm: () 2024-03-09T15:09:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 zonegroup default 2024-03-09T15:09:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 started sync module instance, tier type = 2024-03-09T15:09:48.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.682+0000 7f5c56818980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.698+0000 7f5c56818980 20 add_watcher() i=0 2024-03-09T15:09:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.698+0000 7f5c56818980 20 add_watcher() i=1 2024-03-09T15:09:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.698+0000 7f5c56818980 20 add_watcher() i=2 2024-03-09T15:09:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.698+0000 7f5c56818980 20 add_watcher() i=3 2024-03-09T15:09:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.698+0000 7f5c56818980 20 add_watcher() i=4 2024-03-09T15:09:48.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.698+0000 7f5c56818980 20 add_watcher() i=5 2024-03-09T15:09:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.698+0000 7f5c56818980 20 add_watcher() i=6 2024-03-09T15:09:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.698+0000 7f5c56818980 20 add_watcher() i=7 2024-03-09T15:09:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.698+0000 7f5c56818980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5b0afdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5c56818980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5c56818980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:48.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5c56818980 5 note: GC not initialized 2024-03-09T15:09:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5b09fdb640 20 reqs_thread_entry: start 2024-03-09T15:09:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5c56818980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5c56818980 20 Filter name: none 2024-03-09T15:09:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5a7b7fe640 20 reqs_thread_entry: start 2024-03-09T15:09:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5c56818980 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-03-09T15:09:48.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5c56818980 20 rados->read ofs=0 len=0 2024-03-09T15:09:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5c56818980 20 rados_obj.operate() r=0 bl.length=460 2024-03-09T15:09:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5c56818980 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2024-03-09T15:09:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5c56818980 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-03-09T15:09:48.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:48.702+0000 7f5c56818980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T15:09:50.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.078+0000 7f5c56818980 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-03-09T15:09:50.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.078+0000 7f5c56818980 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-03-09T15:09:50.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.078+0000 7f5c56818980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-03-09T15:09:50.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.078+0000 7f5b2affd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1116691497106 cookie 94141174896128 notifier 7142 bl.length()=659 2024-03-09T15:09:50.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.078+0000 7f5b2affd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-03-09T15:09:50.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.078+0000 7f5b2affd640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-03-09T15:09:50.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.086+0000 7f5b0afdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:50.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.090+0000 7f5c56818980 20 remove_watcher() i=0 2024-03-09T15:09:50.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.090+0000 7f5c56818980 2 removed watcher, disabling cache 2024-03-09T15:09:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.090+0000 7f5c56818980 20 remove_watcher() i=1 2024-03-09T15:09:50.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.094+0000 7f5c56818980 20 remove_watcher() i=2 2024-03-09T15:09:50.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.094+0000 7f5c56818980 20 remove_watcher() i=3 2024-03-09T15:09:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.098+0000 7f5c56818980 20 remove_watcher() i=4 2024-03-09T15:09:50.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.102+0000 7f5c56818980 20 remove_watcher() i=5 2024-03-09T15:09:50.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.102+0000 7f5c56818980 20 remove_watcher() i=6 2024-03-09T15:09:50.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.102+0000 7f5c56818980 20 remove_watcher() i=7 2024-03-09T15:09:50.127 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2024-03-09T15:09:50.127 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name Mr.bar.client.0 --email bar+test@test.test --access-key QZCRYDLYNRJGDQDOWSWS --secret iDKeUTCHf+X/MmWabQakOYudIlgzDO0IavleN4g1FmIAKoipRJxiCw== --cluster ceph 2024-03-09T15:09:50.213 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:50.214 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:50.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.250+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.254+0000 7f0ab7c19980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.254+0000 7f0ab7c19980 20 realm 2024-03-09T15:09:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.254+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.254+0000 7f0ab7c19980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.254+0000 7f0ab7c19980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.254+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.254+0000 7f0ab7c19980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.254+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.254+0000 7f0ab7c19980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.254+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.258+0000 7f0ab7c19980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:50.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.258+0000 7f0ab7c19980 20 searching for the correct realm 2024-03-09T15:09:50.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.262+0000 7f0aacf30640 0 -- 172.21.15.1:0/1135393246 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f0aa801a040 msgr2=0x7f0aa802fad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T15:09:50.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.270+0000 7f0aacf30640 0 -- 172.21.15.1:0/1135393246 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f0980076bc0 msgr2=0x7f0980078ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:50.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:50.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:50.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:50.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:50.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:50.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:50.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:50.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:50.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 zone default found 2024-03-09T15:09:50.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 4 Realm: () 2024-03-09T15:09:50.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:50.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:50.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:50.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 zonegroup default 2024-03-09T15:09:50.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 started sync module instance, tier type = 2024-03-09T15:09:50.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.278+0000 7f0ab7c19980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:50.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.294+0000 7f0ab7c19980 20 add_watcher() i=0 2024-03-09T15:09:50.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.294+0000 7f0ab7c19980 20 add_watcher() i=1 2024-03-09T15:09:50.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.294+0000 7f0ab7c19980 20 add_watcher() i=2 2024-03-09T15:09:50.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.294+0000 7f0ab7c19980 20 add_watcher() i=3 2024-03-09T15:09:50.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.294+0000 7f0ab7c19980 20 add_watcher() i=4 2024-03-09T15:09:50.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.294+0000 7f0ab7c19980 20 add_watcher() i=5 2024-03-09T15:09:50.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.294+0000 7f0ab7c19980 20 add_watcher() i=6 2024-03-09T15:09:50.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.298+0000 7f0ab7c19980 20 add_watcher() i=7 2024-03-09T15:09:50.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.298+0000 7f0ab7c19980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:50.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.298+0000 7f096bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:50.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.298+0000 7f0ab7c19980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:50.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.298+0000 7f0ab7c19980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:50.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.298+0000 7f0ab7c19980 5 note: GC not initialized 2024-03-09T15:09:50.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.298+0000 7f096afe5640 20 reqs_thread_entry: start 2024-03-09T15:09:50.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.298+0000 7f0ab7c19980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:50.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.298+0000 7f0ab7c19980 20 Filter name: none 2024-03-09T15:09:50.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.298+0000 7f0968fe1640 20 reqs_thread_entry: start 2024-03-09T15:09:50.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.298+0000 7f0ab7c19980 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-03-09T15:09:50.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.298+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2024-03-09T15:09:50.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-03-09T15:09:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 cache get: name=default.rgw.meta+users.email+bar+test@test.test : miss 2024-03-09T15:09:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 cache put: name=default.rgw.meta+users.email+bar+test@test.test info.flags=0x0 2024-03-09T15:09:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 adding default.rgw.meta+users.email+bar+test@test.test to cache LRU end 2024-03-09T15:09:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 cache get: name=default.rgw.meta+users.keys+QZCRYDLYNRJGDQDOWSWS : miss 2024-03-09T15:09:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 cache put: name=default.rgw.meta+users.keys+QZCRYDLYNRJGDQDOWSWS info.flags=0x0 2024-03-09T15:09:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 adding default.rgw.meta+users.keys+QZCRYDLYNRJGDQDOWSWS to cache LRU end 2024-03-09T15:09:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 cache get: name=default.rgw.meta+users.keys+QZCRYDLYNRJGDQDOWSWS : hit (negative entry) 2024-03-09T15:09:50.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 cache get: name=default.rgw.meta+users.keys+QZCRYDLYNRJGDQDOWSWS : hit (negative entry) 2024-03-09T15:09:50.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-03-09T15:09:50.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-03-09T15:09:50.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.302+0000 7f0ab7c19980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-03-09T15:09:50.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.306+0000 7f0998ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1116691497076 cookie 94210821700816 notifier 7710 bl.length()=645 2024-03-09T15:09:50.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.306+0000 7f0998ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-03-09T15:09:50.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.306+0000 7f0998ff9640 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-03-09T15:09:50.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.310+0000 7f0ab7c19980 10 cache put: name=default.rgw.meta+users.email+bar+test@test.test info.flags=0x7 2024-03-09T15:09:50.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.310+0000 7f0ab7c19980 10 moving default.rgw.meta+users.email+bar+test@test.test to cache LRU end 2024-03-09T15:09:50.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.310+0000 7f0ab7c19980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.email:bar+test@test.test, ofs0, ns] 2024-03-09T15:09:50.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.310+0000 7f0998ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1116691497119 cookie 94210821712144 notifier 7710 bl.length()=185 2024-03-09T15:09:50.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.310+0000 7f0998ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar+test@test.test info.flags=0x7 2024-03-09T15:09:50.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.310+0000 7f0998ff9640 10 rgw watcher librados: moving default.rgw.meta+users.email+bar+test@test.test to cache LRU end 2024-03-09T15:09:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.310+0000 7f0ab7c19980 10 cache put: name=default.rgw.meta+users.keys+QZCRYDLYNRJGDQDOWSWS info.flags=0x7 2024-03-09T15:09:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.310+0000 7f0ab7c19980 10 moving default.rgw.meta+users.keys+QZCRYDLYNRJGDQDOWSWS to cache LRU end 2024-03-09T15:09:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.310+0000 7f0ab7c19980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.keys:QZCRYDLYNRJGDQDOWSWS, ofs0, ns] 2024-03-09T15:09:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.314+0000 7f0987fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1116691497120 cookie 94210821715568 notifier 7710 bl.length()=186 2024-03-09T15:09:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.314+0000 7f0987fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+QZCRYDLYNRJGDQDOWSWS info.flags=0x7 2024-03-09T15:09:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.314+0000 7f0987fff640 10 rgw watcher librados: moving default.rgw.meta+users.keys+QZCRYDLYNRJGDQDOWSWS to cache LRU end 2024-03-09T15:09:50.316 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:09:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "user_id": "bar.client.0", 2024-03-09T15:09:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "display_name": "Mr.bar.client.0", 2024-03-09T15:09:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "email": "bar+test@test.test", 2024-03-09T15:09:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "suspended": 0, 2024-03-09T15:09:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "max_buckets": 1000, 2024-03-09T15:09:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "subusers": [], 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: "keys": [ 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: "user": "bar.client.0", 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: "access_key": "QZCRYDLYNRJGDQDOWSWS", 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: "secret_key": "iDKeUTCHf+X/MmWabQakOYudIlgzDO0IavleN4g1FmIAKoipRJxiCw==", 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: "active": true, 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:50.306744Z" 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: "swift_keys": [], 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": [], 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: "default_placement": "", 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: "default_storage_class": "", 2024-03-09T15:09:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: "placement_tags": [], 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "bucket_quota": { 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "user_quota": { 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:50.318 INFO:teuthology.orchestra.run.smithi001.stdout: "temp_url_keys": [], 2024-03-09T15:09:50.319 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "rgw", 2024-03-09T15:09:50.319 INFO:teuthology.orchestra.run.smithi001.stdout: "mfa_ids": [], 2024-03-09T15:09:50.319 INFO:teuthology.orchestra.run.smithi001.stdout: "account_id": "", 2024-03-09T15:09:50.319 INFO:teuthology.orchestra.run.smithi001.stdout: "path": "/", 2024-03-09T15:09:50.319 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:50.306730Z", 2024-03-09T15:09:50.319 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-03-09T15:09:50.319 INFO:teuthology.orchestra.run.smithi001.stdout: "group_ids": [] 2024-03-09T15:09:50.319 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:09:50.319 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.314+0000 7f096bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.318+0000 7f0ab7c19980 20 remove_watcher() i=0 2024-03-09T15:09:50.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.318+0000 7f0ab7c19980 2 removed watcher, disabling cache 2024-03-09T15:09:50.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.322+0000 7f0ab7c19980 20 remove_watcher() i=1 2024-03-09T15:09:50.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.326+0000 7f0ab7c19980 20 remove_watcher() i=2 2024-03-09T15:09:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.326+0000 7f0ab7c19980 20 remove_watcher() i=3 2024-03-09T15:09:50.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.326+0000 7f0ab7c19980 20 remove_watcher() i=4 2024-03-09T15:09:50.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.330+0000 7f0ab7c19980 20 remove_watcher() i=5 2024-03-09T15:09:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.330+0000 7f0ab7c19980 20 remove_watcher() i=6 2024-03-09T15:09:50.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.330+0000 7f0ab7c19980 20 remove_watcher() i=7 2024-03-09T15:09:50.349 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 8218033936 --totp-seed YWMFPPKISO7V4FYTF6VFBALRSVG2ZIIJSOR6KTTJIY6DOBUB4ZGDDSE7KBZ36DEZ --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-03-09T15:09:50.437 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:50.437 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:50.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.474+0000 7f5a28c1e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.478+0000 7f5a28c1e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.478+0000 7f5a28c1e980 20 realm 2024-03-09T15:09:50.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.478+0000 7f5a28c1e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.478+0000 7f5a28c1e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.478+0000 7f5a28c1e980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:50.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.478+0000 7f5a28c1e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.478+0000 7f5a28c1e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.478+0000 7f5a28c1e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.478+0000 7f5a28c1e980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:50.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.478+0000 7f5a28c1e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.482+0000 7f5a28c1e980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:50.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.482+0000 7f5a28c1e980 20 searching for the correct realm 2024-03-09T15:09:50.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.486+0000 7f5a25f35640 0 -- 172.21.15.1:0/3874775030 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x556dd5461220 msgr2=0x556dd54636e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:50.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.494+0000 7f5a26736640 0 -- 172.21.15.1:0/3874775030 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f5a1c0b83e0 msgr2=0x7f5a1c0b8840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T15:09:50.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:50.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:50.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:50.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:50.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:50.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:50.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:50.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:50.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 zone default found 2024-03-09T15:09:50.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 4 Realm: () 2024-03-09T15:09:50.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:50.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:50.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:50.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 zonegroup default 2024-03-09T15:09:50.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.502+0000 7f5a28c1e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.506+0000 7f5a28c1e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.506+0000 7f5a28c1e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.506+0000 7f5a28c1e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.506+0000 7f5a28c1e980 20 started sync module instance, tier type = 2024-03-09T15:09:50.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.506+0000 7f5a28c1e980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:50.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f5a28c1e980 20 add_watcher() i=0 2024-03-09T15:09:50.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f5a28c1e980 20 add_watcher() i=1 2024-03-09T15:09:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f5a28c1e980 20 add_watcher() i=2 2024-03-09T15:09:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f5a28c1e980 20 add_watcher() i=3 2024-03-09T15:09:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f5a28c1e980 20 add_watcher() i=4 2024-03-09T15:09:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f5a28c1e980 20 add_watcher() i=5 2024-03-09T15:09:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f5a28c1e980 20 add_watcher() i=6 2024-03-09T15:09:50.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f5a28c1e980 20 add_watcher() i=7 2024-03-09T15:09:50.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f5a28c1e980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:50.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f58dd7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:50.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f5a28c1e980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:50.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f5a28c1e980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:50.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.522+0000 7f5a28c1e980 5 note: GC not initialized 2024-03-09T15:09:50.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.526+0000 7f58d77fe640 20 reqs_thread_entry: start 2024-03-09T15:09:50.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.526+0000 7f5a28c1e980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:50.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.526+0000 7f5a28c1e980 20 Filter name: none 2024-03-09T15:09:50.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.526+0000 7f58d57fa640 20 reqs_thread_entry: start 2024-03-09T15:09:50.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.526+0000 7f5a28c1e980 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-03-09T15:09:50.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.526+0000 7f5a28c1e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.526+0000 7f5a28c1e980 20 rados_obj.operate() r=0 bl.length=460 2024-03-09T15:09:50.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.526+0000 7f5a28c1e980 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2024-03-09T15:09:50.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.526+0000 7f5a28c1e980 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-03-09T15:09:50.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.526+0000 7f5a28c1e980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T15:09:50.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.530+0000 7f5a28c1e980 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-03-09T15:09:50.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.530+0000 7f5a28c1e980 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-03-09T15:09:50.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.530+0000 7f5a28c1e980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-03-09T15:09:50.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.530+0000 7f5909ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1116691497077 cookie 93930217948480 notifier 7728 bl.length()=659 2024-03-09T15:09:50.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.534+0000 7f5909ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-03-09T15:09:50.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.534+0000 7f5909ffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-03-09T15:09:50.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.534+0000 7f58dd7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:50.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.542+0000 7f5a28c1e980 20 remove_watcher() i=0 2024-03-09T15:09:50.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.542+0000 7f5a28c1e980 2 removed watcher, disabling cache 2024-03-09T15:09:50.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.542+0000 7f5a28c1e980 20 remove_watcher() i=1 2024-03-09T15:09:50.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.546+0000 7f5a28c1e980 20 remove_watcher() i=2 2024-03-09T15:09:50.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.546+0000 7f5a28c1e980 20 remove_watcher() i=3 2024-03-09T15:09:50.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.550+0000 7f5a28c1e980 20 remove_watcher() i=4 2024-03-09T15:09:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.550+0000 7f5a28c1e980 20 remove_watcher() i=5 2024-03-09T15:09:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.554+0000 7f5a28c1e980 20 remove_watcher() i=6 2024-03-09T15:09:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.554+0000 7f5a28c1e980 20 remove_watcher() i=7 2024-03-09T15:09:50.570 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2024-03-09T15:09:50.571 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr.testx$tenanteduser.client.0' --email 'testx$tenanteduser+test@test.test' --access-key XTTJPSJZYMIRBRXBMONC --secret 3qNNzhEOaft0l9xnCypk72tnvwy1l/DefhgaA8kSpVh/lZETh2F1eA== --cluster ceph 2024-03-09T15:09:50.609 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:50.610 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:50.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.646+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.650+0000 7fa0d0ce4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.650+0000 7fa0d0ce4980 20 realm 2024-03-09T15:09:50.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.650+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.650+0000 7fa0d0ce4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.650+0000 7fa0d0ce4980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:50.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.650+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.650+0000 7fa0d0ce4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.650+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.650+0000 7fa0d0ce4980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:50.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.650+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.650+0000 7fa0d0ce4980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:50.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.650+0000 7fa0d0ce4980 20 searching for the correct realm 2024-03-09T15:09:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.658+0000 7fa0ceffd640 0 -- 172.21.15.1:0/274152873 >> [v2:172.21.15.53:6801/315236776,v1:172.21.15.53:6802/315236776] conn(0x7fa0c4018020 msgr2=0x7fa0c4055720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:50.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.662+0000 7fa0ce7fc640 0 -- 172.21.15.1:0/274152873 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fa0c407cbe0 msgr2=0x7fa0c40821e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T15:09:50.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.878+0000 7fa0d0ce4980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:50.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.878+0000 7fa0d0ce4980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.878+0000 7fa0d0ce4980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.878+0000 7fa0d0ce4980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.878+0000 7fa0d0ce4980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.878+0000 7fa0d0ce4980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.878+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.878+0000 7fa0d0ce4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.878+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:50.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 zone default found 2024-03-09T15:09:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 4 Realm: () 2024-03-09T15:09:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 zonegroup default 2024-03-09T15:09:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 started sync module instance, tier type = 2024-03-09T15:09:50.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.882+0000 7fa0d0ce4980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:50.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.906+0000 7fa0d0ce4980 20 add_watcher() i=0 2024-03-09T15:09:50.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.906+0000 7fa0d0ce4980 20 add_watcher() i=1 2024-03-09T15:09:50.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.906+0000 7fa0d0ce4980 20 add_watcher() i=2 2024-03-09T15:09:50.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.906+0000 7fa0d0ce4980 20 add_watcher() i=3 2024-03-09T15:09:50.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.906+0000 7fa0d0ce4980 20 add_watcher() i=4 2024-03-09T15:09:50.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.906+0000 7fa0d0ce4980 20 add_watcher() i=5 2024-03-09T15:09:50.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.906+0000 7fa0d0ce4980 20 add_watcher() i=6 2024-03-09T15:09:50.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.910+0000 7fa0d0ce4980 20 add_watcher() i=7 2024-03-09T15:09:50.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.910+0000 7fa0d0ce4980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:50.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7f9f84fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 5 note: GC not initialized 2024-03-09T15:09:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7f9eff7fe640 20 reqs_thread_entry: start 2024-03-09T15:09:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 20 Filter name: none 2024-03-09T15:09:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7f9efd7fa640 20 reqs_thread_entry: start 2024-03-09T15:09:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-03-09T15:09:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2024-03-09T15:09:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-03-09T15:09:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser+test@test.test : miss 2024-03-09T15:09:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser+test@test.test info.flags=0x0 2024-03-09T15:09:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 10 adding default.rgw.meta+users.email+testx$tenanteduser+test@test.test to cache LRU end 2024-03-09T15:09:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 10 cache get: name=default.rgw.meta+users.keys+XTTJPSJZYMIRBRXBMONC : miss 2024-03-09T15:09:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.914+0000 7fa0d0ce4980 20 rados->read ofs=0 len=0 2024-03-09T15:09:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.918+0000 7fa0d0ce4980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.918+0000 7fa0d0ce4980 10 cache put: name=default.rgw.meta+users.keys+XTTJPSJZYMIRBRXBMONC info.flags=0x0 2024-03-09T15:09:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.918+0000 7fa0d0ce4980 10 adding default.rgw.meta+users.keys+XTTJPSJZYMIRBRXBMONC to cache LRU end 2024-03-09T15:09:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.918+0000 7fa0d0ce4980 10 cache get: name=default.rgw.meta+users.keys+XTTJPSJZYMIRBRXBMONC : hit (negative entry) 2024-03-09T15:09:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.918+0000 7fa0d0ce4980 10 cache get: name=default.rgw.meta+users.keys+XTTJPSJZYMIRBRXBMONC : hit (negative entry) 2024-03-09T15:09:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.918+0000 7fa0d0ce4980 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-03-09T15:09:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.918+0000 7fa0d0ce4980 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-03-09T15:09:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.918+0000 7fa0d0ce4980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-03-09T15:09:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.918+0000 7f9fb1ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1116691497121 cookie 93855333446832 notifier 7196 bl.length()=719 2024-03-09T15:09:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.918+0000 7f9fb1ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-03-09T15:09:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.918+0000 7f9fb1ffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-03-09T15:09:50.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.926+0000 7fa0d0ce4980 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser+test@test.test info.flags=0x7 2024-03-09T15:09:50.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.926+0000 7fa0d0ce4980 10 moving default.rgw.meta+users.email+testx$tenanteduser+test@test.test to cache LRU end 2024-03-09T15:09:50.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.926+0000 7fa0d0ce4980 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser+test@test.test, ofs0, ns] 2024-03-09T15:09:50.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.926+0000 7f9fb17fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1116691497033 cookie 93855333452736 notifier 7196 bl.length()=215 2024-03-09T15:09:50.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.926+0000 7f9fb17fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser+test@test.test info.flags=0x7 2024-03-09T15:09:50.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.926+0000 7f9fb17fa640 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser+test@test.test to cache LRU end 2024-03-09T15:09:50.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.930+0000 7fa0d0ce4980 10 cache put: name=default.rgw.meta+users.keys+XTTJPSJZYMIRBRXBMONC info.flags=0x7 2024-03-09T15:09:50.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.930+0000 7fa0d0ce4980 10 moving default.rgw.meta+users.keys+XTTJPSJZYMIRBRXBMONC to cache LRU end 2024-03-09T15:09:50.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.930+0000 7fa0d0ce4980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:XTTJPSJZYMIRBRXBMONC, ofs0, ns] 2024-03-09T15:09:50.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.930+0000 7f9fb17fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1116691497078 cookie 93855333433904 notifier 7196 bl.length()=201 2024-03-09T15:09:50.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.930+0000 7f9fb17fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+XTTJPSJZYMIRBRXBMONC info.flags=0x7 2024-03-09T15:09:50.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.930+0000 7f9fb17fa640 10 rgw watcher librados: moving default.rgw.meta+users.keys+XTTJPSJZYMIRBRXBMONC to cache LRU end 2024-03-09T15:09:50.937 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:09:50.937 INFO:teuthology.orchestra.run.smithi001.stdout: "user_id": "testx$tenanteduser.client.0", 2024-03-09T15:09:50.937 INFO:teuthology.orchestra.run.smithi001.stdout: "display_name": "Mr.testx$tenanteduser.client.0", 2024-03-09T15:09:50.937 INFO:teuthology.orchestra.run.smithi001.stdout: "email": "testx$tenanteduser+test@test.test", 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: "suspended": 0, 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: "max_buckets": 1000, 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: "subusers": [], 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: "keys": [ 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: "user": "testx$tenanteduser.client.0", 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: "access_key": "XTTJPSJZYMIRBRXBMONC", 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: "secret_key": "3qNNzhEOaft0l9xnCypk72tnvwy1l/DefhgaA8kSpVh/lZETh2F1eA==", 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: "active": true, 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:50.922149Z" 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: "swift_keys": [], 2024-03-09T15:09:50.938 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": [], 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "default_placement": "", 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "default_storage_class": "", 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "placement_tags": [], 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "bucket_quota": { 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "user_quota": { 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:50.939 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: "temp_url_keys": [], 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "rgw", 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: "mfa_ids": [], 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: "account_id": "", 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: "path": "/", 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:50.922117Z", 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: "group_ids": [] 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:09:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:50.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.938+0000 7f9f84fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:50.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.942+0000 7fa0d0ce4980 20 remove_watcher() i=0 2024-03-09T15:09:50.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.942+0000 7fa0d0ce4980 2 removed watcher, disabling cache 2024-03-09T15:09:50.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.946+0000 7fa0d0ce4980 20 remove_watcher() i=1 2024-03-09T15:09:50.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.946+0000 7fa0d0ce4980 20 remove_watcher() i=2 2024-03-09T15:09:50.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.950+0000 7fa0d0ce4980 20 remove_watcher() i=3 2024-03-09T15:09:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.950+0000 7fa0d0ce4980 20 remove_watcher() i=4 2024-03-09T15:09:50.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.954+0000 7fa0d0ce4980 20 remove_watcher() i=5 2024-03-09T15:09:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.954+0000 7fa0d0ce4980 20 remove_watcher() i=6 2024-03-09T15:09:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:50.958+0000 7fa0d0ce4980 20 remove_watcher() i=7 2024-03-09T15:09:50.976 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 6636596494 --totp-seed ORNF2N4DLZTBQ3RFUZWEQMHTMUYSCNE6WNSJOCODGGX3V7NBJEPNXN3BXJYD3YQ6 --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-03-09T15:09:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:51.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.122+0000 7f3800bda980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.122+0000 7f3800bda980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.122+0000 7f3800bda980 20 realm 2024-03-09T15:09:51.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.122+0000 7f3800bda980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.122+0000 7f3800bda980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.122+0000 7f3800bda980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.122+0000 7f3800bda980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.122+0000 7f3800bda980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.122+0000 7f3800bda980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.126+0000 7f3800bda980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.126+0000 7f3800bda980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.126+0000 7f3800bda980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.126+0000 7f3800bda980 20 searching for the correct realm 2024-03-09T15:09:51.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.134+0000 7f37feef3640 0 -- 172.21.15.1:0/2969541727 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x560124866e30 msgr2=0x5601248692d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:51.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.146+0000 7f37fe6f2640 0 -- 172.21.15.1:0/2969541727 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x560124866e30 msgr2=0x7f36c807b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.362+0000 7f3800bda980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:51.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.362+0000 7f3800bda980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.362+0000 7f3800bda980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.362+0000 7f3800bda980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.362+0000 7f3800bda980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.362+0000 7f3800bda980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.362+0000 7f3800bda980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:51.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:51.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 20 zone default found 2024-03-09T15:09:51.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 4 Realm: () 2024-03-09T15:09:51.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:51.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:51.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:51.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 20 zonegroup default 2024-03-09T15:09:51.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.366+0000 7f3800bda980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.370+0000 7f3800bda980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.370+0000 7f3800bda980 20 started sync module instance, tier type = 2024-03-09T15:09:51.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.370+0000 7f3800bda980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:51.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.394+0000 7f3800bda980 20 add_watcher() i=0 2024-03-09T15:09:51.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.394+0000 7f3800bda980 20 add_watcher() i=1 2024-03-09T15:09:51.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.394+0000 7f3800bda980 20 add_watcher() i=2 2024-03-09T15:09:51.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.394+0000 7f3800bda980 20 add_watcher() i=3 2024-03-09T15:09:51.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.394+0000 7f3800bda980 20 add_watcher() i=4 2024-03-09T15:09:51.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.394+0000 7f3800bda980 20 add_watcher() i=5 2024-03-09T15:09:51.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.394+0000 7f3800bda980 20 add_watcher() i=6 2024-03-09T15:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.394+0000 7f3800bda980 20 add_watcher() i=7 2024-03-09T15:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.394+0000 7f3800bda980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.398+0000 7f36b57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f3800bda980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f3800bda980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f3800bda980 5 note: GC not initialized 2024-03-09T15:09:51.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f36af7fe640 20 reqs_thread_entry: start 2024-03-09T15:09:51.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f3800bda980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:51.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f3800bda980 20 Filter name: none 2024-03-09T15:09:51.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f36ad7fa640 20 reqs_thread_entry: start 2024-03-09T15:09:51.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f3800bda980 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-03-09T15:09:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f3800bda980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f3800bda980 20 rados_obj.operate() r=0 bl.length=519 2024-03-09T15:09:51.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f3800bda980 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2024-03-09T15:09:51.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f3800bda980 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-03-09T15:09:51.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.402+0000 7f3800bda980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T15:09:51.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.406+0000 7f3800bda980 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-03-09T15:09:51.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.406+0000 7f3800bda980 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-03-09T15:09:51.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.406+0000 7f3800bda980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-03-09T15:09:51.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.410+0000 7f36e1ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464418 cookie 94562907806160 notifier 7211 bl.length()=733 2024-03-09T15:09:51.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.410+0000 7f36e1ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-03-09T15:09:51.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.410+0000 7f36e1ffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-03-09T15:09:51.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.410+0000 7f36b57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.418+0000 7f3800bda980 20 remove_watcher() i=0 2024-03-09T15:09:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.418+0000 7f3800bda980 2 removed watcher, disabling cache 2024-03-09T15:09:51.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.418+0000 7f3800bda980 20 remove_watcher() i=1 2024-03-09T15:09:51.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.422+0000 7f3800bda980 20 remove_watcher() i=2 2024-03-09T15:09:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.422+0000 7f3800bda980 20 remove_watcher() i=3 2024-03-09T15:09:51.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.426+0000 7f3800bda980 20 remove_watcher() i=4 2024-03-09T15:09:51.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.426+0000 7f3800bda980 20 remove_watcher() i=5 2024-03-09T15:09:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.430+0000 7f3800bda980 20 remove_watcher() i=6 2024-03-09T15:09:51.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.430+0000 7f3800bda980 20 remove_watcher() i=7 2024-03-09T15:09:51.449 DEBUG:tasks.s3tests:Creating user foobar.client.0 on client.0 2024-03-09T15:09:51.449 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foobar.client.0 --display-name Mr.foobar.client.0 --email foobar+test@test.test --access-key MOPVLEGSQVHOITBKGVDB --secret fIYwXvxGme9V0Zq5kpdUouNb3kMrVOH/PL1fhTbrEI/Pu7T+w3D33w== --cluster ceph 2024-03-09T15:09:51.490 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:51.490 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:51.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.526+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.530+0000 7fcab3fb5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.530+0000 7fcab3fb5980 20 realm 2024-03-09T15:09:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.530+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.530+0000 7fcab3fb5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.530+0000 7fcab3fb5980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.530+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.530+0000 7fcab3fb5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.530+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.530+0000 7fcab3fb5980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.530+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.534+0000 7fcab3fb5980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:51.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.534+0000 7fcab3fb5980 20 searching for the correct realm 2024-03-09T15:09:51.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.538+0000 7fcab12cc640 0 -- 172.21.15.1:0/2504082780 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x558aa8f74480 msgr2=0x558aa8f76940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:51.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.546+0000 7fcab1acd640 0 -- 172.21.15.1:0/2504082780 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fca9c10c840 msgr2=0x7fca9c0d9040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:51.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:51.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:51.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:51.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:51.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:51.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:51.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:51.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:51.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 zone default found 2024-03-09T15:09:51.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 4 Realm: () 2024-03-09T15:09:51.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:51.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:51.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:51.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 zonegroup default 2024-03-09T15:09:51.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 started sync module instance, tier type = 2024-03-09T15:09:51.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.554+0000 7fcab3fb5980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.570+0000 7fcab3fb5980 20 add_watcher() i=0 2024-03-09T15:09:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.570+0000 7fcab3fb5980 20 add_watcher() i=1 2024-03-09T15:09:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.570+0000 7fcab3fb5980 20 add_watcher() i=2 2024-03-09T15:09:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.570+0000 7fcab3fb5980 20 add_watcher() i=3 2024-03-09T15:09:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.570+0000 7fcab3fb5980 20 add_watcher() i=4 2024-03-09T15:09:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.570+0000 7fcab3fb5980 20 add_watcher() i=5 2024-03-09T15:09:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.570+0000 7fcab3fb5980 20 add_watcher() i=6 2024-03-09T15:09:51.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.574+0000 7fcab3fb5980 20 add_watcher() i=7 2024-03-09T15:09:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.574+0000 7fcab3fb5980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.574+0000 7fc963fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.574+0000 7fcab3fb5980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.574+0000 7fcab3fb5980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.574+0000 7fcab3fb5980 5 note: GC not initialized 2024-03-09T15:09:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.574+0000 7fc962ffd640 20 reqs_thread_entry: start 2024-03-09T15:09:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.574+0000 7fcab3fb5980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.574+0000 7fcab3fb5980 20 Filter name: none 2024-03-09T15:09:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.574+0000 7fc960ff9640 20 reqs_thread_entry: start 2024-03-09T15:09:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2024-03-09T15:09:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x0 2024-03-09T15:09:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-03-09T15:09:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 cache get: name=default.rgw.meta+users.email+foobar+test@test.test : miss 2024-03-09T15:09:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 cache put: name=default.rgw.meta+users.email+foobar+test@test.test info.flags=0x0 2024-03-09T15:09:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 adding default.rgw.meta+users.email+foobar+test@test.test to cache LRU end 2024-03-09T15:09:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 cache get: name=default.rgw.meta+users.keys+MOPVLEGSQVHOITBKGVDB : miss 2024-03-09T15:09:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 cache put: name=default.rgw.meta+users.keys+MOPVLEGSQVHOITBKGVDB info.flags=0x0 2024-03-09T15:09:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 adding default.rgw.meta+users.keys+MOPVLEGSQVHOITBKGVDB to cache LRU end 2024-03-09T15:09:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 cache get: name=default.rgw.meta+users.keys+MOPVLEGSQVHOITBKGVDB : hit (negative entry) 2024-03-09T15:09:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 cache get: name=default.rgw.meta+users.keys+MOPVLEGSQVHOITBKGVDB : hit (negative entry) 2024-03-09T15:09:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-03-09T15:09:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-03-09T15:09:51.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fcab3fb5980 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2024-03-09T15:09:51.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fc9957fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464330 cookie 94054028650160 notifier 7791 bl.length()=660 2024-03-09T15:09:51.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fc9957fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-03-09T15:09:51.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.578+0000 7fc9957fa640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-03-09T15:09:51.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.586+0000 7fcab3fb5980 10 cache put: name=default.rgw.meta+users.email+foobar+test@test.test info.flags=0x7 2024-03-09T15:09:51.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.586+0000 7fcab3fb5980 10 moving default.rgw.meta+users.email+foobar+test@test.test to cache LRU end 2024-03-09T15:09:51.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.586+0000 7fcab3fb5980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:foobar+test@test.test, ofs0, ns] 2024-03-09T15:09:51.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.586+0000 7fc994ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464375 cookie 94054028638112 notifier 7791 bl.length()=191 2024-03-09T15:09:51.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.586+0000 7fc994ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foobar+test@test.test info.flags=0x7 2024-03-09T15:09:51.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.586+0000 7fc994ff9640 10 rgw watcher librados: moving default.rgw.meta+users.email+foobar+test@test.test to cache LRU end 2024-03-09T15:09:51.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.590+0000 7fcab3fb5980 10 cache put: name=default.rgw.meta+users.keys+MOPVLEGSQVHOITBKGVDB info.flags=0x7 2024-03-09T15:09:51.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.590+0000 7fcab3fb5980 10 moving default.rgw.meta+users.keys+MOPVLEGSQVHOITBKGVDB to cache LRU end 2024-03-09T15:09:51.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.590+0000 7fcab3fb5980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.keys:MOPVLEGSQVHOITBKGVDB, ofs0, ns] 2024-03-09T15:09:51.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.590+0000 7fc994ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464419 cookie 94054028630992 notifier 7791 bl.length()=189 2024-03-09T15:09:51.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.590+0000 7fc994ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+MOPVLEGSQVHOITBKGVDB info.flags=0x7 2024-03-09T15:09:51.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.590+0000 7fc994ff9640 10 rgw watcher librados: moving default.rgw.meta+users.keys+MOPVLEGSQVHOITBKGVDB to cache LRU end 2024-03-09T15:09:51.595 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:09:51.595 INFO:teuthology.orchestra.run.smithi001.stdout: "user_id": "foobar.client.0", 2024-03-09T15:09:51.595 INFO:teuthology.orchestra.run.smithi001.stdout: "display_name": "Mr.foobar.client.0", 2024-03-09T15:09:51.595 INFO:teuthology.orchestra.run.smithi001.stdout: "email": "foobar+test@test.test", 2024-03-09T15:09:51.596 INFO:teuthology.orchestra.run.smithi001.stdout: "suspended": 0, 2024-03-09T15:09:51.596 INFO:teuthology.orchestra.run.smithi001.stdout: "max_buckets": 1000, 2024-03-09T15:09:51.596 INFO:teuthology.orchestra.run.smithi001.stdout: "subusers": [], 2024-03-09T15:09:51.596 INFO:teuthology.orchestra.run.smithi001.stdout: "keys": [ 2024-03-09T15:09:51.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:09:51.596 INFO:teuthology.orchestra.run.smithi001.stdout: "user": "foobar.client.0", 2024-03-09T15:09:51.597 INFO:teuthology.orchestra.run.smithi001.stdout: "access_key": "MOPVLEGSQVHOITBKGVDB", 2024-03-09T15:09:51.597 INFO:teuthology.orchestra.run.smithi001.stdout: "secret_key": "fIYwXvxGme9V0Zq5kpdUouNb3kMrVOH/PL1fhTbrEI/Pu7T+w3D33w==", 2024-03-09T15:09:51.597 INFO:teuthology.orchestra.run.smithi001.stdout: "active": true, 2024-03-09T15:09:51.597 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:51.583878Z" 2024-03-09T15:09:51.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:09:51.597 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-03-09T15:09:51.598 INFO:teuthology.orchestra.run.smithi001.stdout: "swift_keys": [], 2024-03-09T15:09:51.598 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": [], 2024-03-09T15:09:51.598 INFO:teuthology.orchestra.run.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T15:09:51.598 INFO:teuthology.orchestra.run.smithi001.stdout: "default_placement": "", 2024-03-09T15:09:51.598 INFO:teuthology.orchestra.run.smithi001.stdout: "default_storage_class": "", 2024-03-09T15:09:51.598 INFO:teuthology.orchestra.run.smithi001.stdout: "placement_tags": [], 2024-03-09T15:09:51.598 INFO:teuthology.orchestra.run.smithi001.stdout: "bucket_quota": { 2024-03-09T15:09:51.599 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:51.599 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:51.599 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:51.599 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:51.599 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:51.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:51.599 INFO:teuthology.orchestra.run.smithi001.stdout: "user_quota": { 2024-03-09T15:09:51.600 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:51.600 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:51.600 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:51.600 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:51.600 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:51.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:51.600 INFO:teuthology.orchestra.run.smithi001.stdout: "temp_url_keys": [], 2024-03-09T15:09:51.601 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "rgw", 2024-03-09T15:09:51.601 INFO:teuthology.orchestra.run.smithi001.stdout: "mfa_ids": [], 2024-03-09T15:09:51.601 INFO:teuthology.orchestra.run.smithi001.stdout: "account_id": "", 2024-03-09T15:09:51.601 INFO:teuthology.orchestra.run.smithi001.stdout: "path": "/", 2024-03-09T15:09:51.601 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:51.583833Z", 2024-03-09T15:09:51.601 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-03-09T15:09:51.601 INFO:teuthology.orchestra.run.smithi001.stdout: "group_ids": [] 2024-03-09T15:09:51.601 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:09:51.602 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.594+0000 7fc963fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.598+0000 7fcab3fb5980 20 remove_watcher() i=0 2024-03-09T15:09:51.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.598+0000 7fcab3fb5980 2 removed watcher, disabling cache 2024-03-09T15:09:51.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.602+0000 7fcab3fb5980 20 remove_watcher() i=1 2024-03-09T15:09:51.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.602+0000 7fcab3fb5980 20 remove_watcher() i=2 2024-03-09T15:09:51.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.606+0000 7fcab3fb5980 20 remove_watcher() i=3 2024-03-09T15:09:51.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.606+0000 7fcab3fb5980 20 remove_watcher() i=4 2024-03-09T15:09:51.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.610+0000 7fcab3fb5980 20 remove_watcher() i=5 2024-03-09T15:09:51.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.610+0000 7fcab3fb5980 20 remove_watcher() i=6 2024-03-09T15:09:51.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.610+0000 7fcab3fb5980 20 remove_watcher() i=7 2024-03-09T15:09:51.628 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foobar.client.0 --totp-serial 1565387454 --totp-seed 56AWL4KJDPAUL2KPDANLYLIP3U2R66QOL47PP2QIFCPZXDWG4ULYG6FRLROJEYRV --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-03-09T15:09:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:51.711 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:51.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.750+0000 7fb549657980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.750+0000 7fb549657980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.750+0000 7fb549657980 20 realm 2024-03-09T15:09:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.750+0000 7fb549657980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.750+0000 7fb549657980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.750+0000 7fb549657980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.750+0000 7fb549657980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.750+0000 7fb549657980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.750+0000 7fb549657980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.754+0000 7fb549657980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.754+0000 7fb549657980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.754+0000 7fb549657980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.754+0000 7fb549657980 20 searching for the correct realm 2024-03-09T15:09:51.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.762+0000 7fb54696e640 0 -- 172.21.15.1:0/3611458689 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fb538049430 msgr2=0x7fb538049850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:51.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.766+0000 7fb54696e640 0 -- 172.21.15.1:0/3611458689 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fb410076b40 msgr2=0x7fb410078f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.774+0000 7fb549657980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.774+0000 7fb549657980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.774+0000 7fb549657980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.774+0000 7fb549657980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.774+0000 7fb549657980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.774+0000 7fb549657980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.774+0000 7fb549657980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.774+0000 7fb549657980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.774+0000 7fb549657980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.774+0000 7fb549657980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.774+0000 7fb549657980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 20 zone default found 2024-03-09T15:09:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 4 Realm: () 2024-03-09T15:09:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 20 zonegroup default 2024-03-09T15:09:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 20 started sync module instance, tier type = 2024-03-09T15:09:51.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.778+0000 7fb549657980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:51.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.794+0000 7fb549657980 20 add_watcher() i=0 2024-03-09T15:09:51.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.794+0000 7fb549657980 20 add_watcher() i=1 2024-03-09T15:09:51.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.794+0000 7fb549657980 20 add_watcher() i=2 2024-03-09T15:09:51.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.794+0000 7fb549657980 20 add_watcher() i=3 2024-03-09T15:09:51.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.794+0000 7fb549657980 20 add_watcher() i=4 2024-03-09T15:09:51.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.794+0000 7fb549657980 20 add_watcher() i=5 2024-03-09T15:09:51.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.794+0000 7fb549657980 20 add_watcher() i=6 2024-03-09T15:09:51.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.794+0000 7fb549657980 20 add_watcher() i=7 2024-03-09T15:09:51.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.794+0000 7fb549657980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb3fdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb549657980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb549657980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb549657980 5 note: GC not initialized 2024-03-09T15:09:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb3fcfe1640 20 reqs_thread_entry: start 2024-03-09T15:09:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb549657980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:51.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb549657980 20 Filter name: none 2024-03-09T15:09:51.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb3767fc640 20 reqs_thread_entry: start 2024-03-09T15:09:51.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb549657980 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2024-03-09T15:09:51.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb549657980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb549657980 20 rados_obj.operate() r=0 bl.length=472 2024-03-09T15:09:51.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb549657980 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2024-03-09T15:09:51.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb549657980 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-03-09T15:09:51.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.798+0000 7fb549657980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T15:09:51.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.806+0000 7fb549657980 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-03-09T15:09:51.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.806+0000 7fb549657980 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-03-09T15:09:51.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.806+0000 7fb549657980 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2024-03-09T15:09:51.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.806+0000 7fb42affd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464331 cookie 94182272143952 notifier 7812 bl.length()=674 2024-03-09T15:09:51.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.806+0000 7fb42affd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-03-09T15:09:51.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.806+0000 7fb42affd640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-03-09T15:09:51.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.806+0000 7fb3fdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:51.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.814+0000 7fb549657980 20 remove_watcher() i=0 2024-03-09T15:09:51.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.814+0000 7fb549657980 2 removed watcher, disabling cache 2024-03-09T15:09:51.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.814+0000 7fb549657980 20 remove_watcher() i=1 2024-03-09T15:09:51.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.818+0000 7fb549657980 20 remove_watcher() i=2 2024-03-09T15:09:51.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.818+0000 7fb549657980 20 remove_watcher() i=3 2024-03-09T15:09:51.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.822+0000 7fb549657980 20 remove_watcher() i=4 2024-03-09T15:09:51.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.822+0000 7fb549657980 20 remove_watcher() i=5 2024-03-09T15:09:51.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.826+0000 7fb549657980 20 remove_watcher() i=6 2024-03-09T15:09:51.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.826+0000 7fb549657980 20 remove_watcher() i=7 2024-03-09T15:09:51.847 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 caps add --uid foobar.client.0 --caps 'oidc-provider=*;roles=*;user-policy=*' --cluster ceph 2024-03-09T15:09:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:51.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.962+0000 7f2b589d7980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.962+0000 7f2b589d7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.962+0000 7f2b589d7980 20 realm 2024-03-09T15:09:51.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.962+0000 7f2b589d7980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.966+0000 7f2b589d7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.966+0000 7f2b589d7980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:51.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.966+0000 7f2b589d7980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.966+0000 7f2b589d7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:51.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.966+0000 7f2b589d7980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.966+0000 7f2b589d7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.966+0000 7f2b589d7980 20 rados->read ofs=0 len=0 2024-03-09T15:09:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.966+0000 7f2b589d7980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.966+0000 7f2b589d7980 20 searching for the correct realm 2024-03-09T15:09:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:51.974+0000 7f2b56cf0640 0 -- 172.21.15.1:0/1935771379 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7f2b40047040 msgr2=0x7f2b40047930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T15:09:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.190+0000 7f2b564ef640 0 -- 172.21.15.1:0/1935771379 >> [v2:172.21.15.53:6824/256097109,v1:172.21.15.53:6825/256097109] conn(0x564c240f5a80 msgr2=0x564c240f7e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.198+0000 7f2b589d7980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.198+0000 7f2b589d7980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.198+0000 7f2b589d7980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.198+0000 7f2b589d7980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.198+0000 7f2b589d7980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:52.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.198+0000 7f2b589d7980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:52.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.198+0000 7f2b589d7980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.198+0000 7f2b589d7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.198+0000 7f2b589d7980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 zone default found 2024-03-09T15:09:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 4 Realm: () 2024-03-09T15:09:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 zonegroup default 2024-03-09T15:09:52.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 started sync module instance, tier type = 2024-03-09T15:09:52.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.202+0000 7f2b589d7980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.226+0000 7f2b589d7980 20 add_watcher() i=0 2024-03-09T15:09:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.226+0000 7f2b589d7980 20 add_watcher() i=1 2024-03-09T15:09:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.226+0000 7f2b589d7980 20 add_watcher() i=2 2024-03-09T15:09:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.226+0000 7f2b589d7980 20 add_watcher() i=3 2024-03-09T15:09:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.226+0000 7f2b589d7980 20 add_watcher() i=4 2024-03-09T15:09:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.226+0000 7f2b589d7980 20 add_watcher() i=5 2024-03-09T15:09:52.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.226+0000 7f2b589d7980 20 add_watcher() i=6 2024-03-09T15:09:52.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.226+0000 7f2b589d7980 20 add_watcher() i=7 2024-03-09T15:09:52.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.226+0000 7f2b589d7980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:52.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.230+0000 7f2a0cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:52.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.230+0000 7f2b589d7980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:52.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.230+0000 7f2b589d7980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:52.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.230+0000 7f2b589d7980 5 note: GC not initialized 2024-03-09T15:09:52.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.230+0000 7f29877fe640 20 reqs_thread_entry: start 2024-03-09T15:09:52.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.234+0000 7f2b589d7980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:52.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.234+0000 7f2b589d7980 20 Filter name: none 2024-03-09T15:09:52.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.234+0000 7f29857fa640 20 reqs_thread_entry: start 2024-03-09T15:09:52.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.234+0000 7f2b589d7980 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2024-03-09T15:09:52.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.234+0000 7f2b589d7980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.234+0000 7f2b589d7980 20 rados_obj.operate() r=0 bl.length=486 2024-03-09T15:09:52.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.234+0000 7f2b589d7980 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2024-03-09T15:09:52.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.234+0000 7f2b589d7980 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-03-09T15:09:52.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.234+0000 7f2b589d7980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T15:09:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.234+0000 7f2b589d7980 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-03-09T15:09:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.234+0000 7f2b589d7980 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-03-09T15:09:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.234+0000 7f2b589d7980 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2024-03-09T15:09:52.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.238+0000 7f2a39ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464332 cookie 94885022540336 notifier 7265 bl.length()=727 2024-03-09T15:09:52.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.238+0000 7f2a39ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-03-09T15:09:52.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.238+0000 7f2a39ffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-03-09T15:09:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:09:52.241 INFO:teuthology.orchestra.run.smithi001.stdout: "user_id": "foobar.client.0", 2024-03-09T15:09:52.241 INFO:teuthology.orchestra.run.smithi001.stdout: "display_name": "Mr.foobar.client.0", 2024-03-09T15:09:52.241 INFO:teuthology.orchestra.run.smithi001.stdout: "email": "foobar+test@test.test", 2024-03-09T15:09:52.241 INFO:teuthology.orchestra.run.smithi001.stdout: "suspended": 0, 2024-03-09T15:09:52.241 INFO:teuthology.orchestra.run.smithi001.stdout: "max_buckets": 1000, 2024-03-09T15:09:52.241 INFO:teuthology.orchestra.run.smithi001.stdout: "subusers": [], 2024-03-09T15:09:52.241 INFO:teuthology.orchestra.run.smithi001.stdout: "keys": [ 2024-03-09T15:09:52.241 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:09:52.241 INFO:teuthology.orchestra.run.smithi001.stdout: "user": "foobar.client.0", 2024-03-09T15:09:52.241 INFO:teuthology.orchestra.run.smithi001.stdout: "access_key": "MOPVLEGSQVHOITBKGVDB", 2024-03-09T15:09:52.241 INFO:teuthology.orchestra.run.smithi001.stdout: "secret_key": "fIYwXvxGme9V0Zq5kpdUouNb3kMrVOH/PL1fhTbrEI/Pu7T+w3D33w==", 2024-03-09T15:09:52.241 INFO:teuthology.orchestra.run.smithi001.stdout: "active": true, 2024-03-09T15:09:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:51.583878Z" 2024-03-09T15:09:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:09:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-03-09T15:09:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: "swift_keys": [], 2024-03-09T15:09:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": [ 2024-03-09T15:09:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:09:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "oidc-provider", 2024-03-09T15:09:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: "perm": "*" 2024-03-09T15:09:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:09:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "roles", 2024-03-09T15:09:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: "perm": "*" 2024-03-09T15:09:52.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:52.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:09:52.243 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "user-policy", 2024-03-09T15:09:52.243 INFO:teuthology.orchestra.run.smithi001.stdout: "perm": "*" 2024-03-09T15:09:52.243 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:09:52.243 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-03-09T15:09:52.243 INFO:teuthology.orchestra.run.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T15:09:52.243 INFO:teuthology.orchestra.run.smithi001.stdout: "default_placement": "", 2024-03-09T15:09:52.243 INFO:teuthology.orchestra.run.smithi001.stdout: "default_storage_class": "", 2024-03-09T15:09:52.243 INFO:teuthology.orchestra.run.smithi001.stdout: "placement_tags": [], 2024-03-09T15:09:52.243 INFO:teuthology.orchestra.run.smithi001.stdout: "bucket_quota": { 2024-03-09T15:09:52.243 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:52.244 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:52.244 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:52.244 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:52.244 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:52.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:52.244 INFO:teuthology.orchestra.run.smithi001.stdout: "user_quota": { 2024-03-09T15:09:52.244 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:52.244 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:52.244 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:52.244 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:52.244 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:52.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:52.245 INFO:teuthology.orchestra.run.smithi001.stdout: "temp_url_keys": [], 2024-03-09T15:09:52.245 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "rgw", 2024-03-09T15:09:52.245 INFO:teuthology.orchestra.run.smithi001.stdout: "mfa_ids": [ 2024-03-09T15:09:52.245 INFO:teuthology.orchestra.run.smithi001.stdout: "1565387454" 2024-03-09T15:09:52.245 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-03-09T15:09:52.245 INFO:teuthology.orchestra.run.smithi001.stdout: "account_id": "", 2024-03-09T15:09:52.245 INFO:teuthology.orchestra.run.smithi001.stdout: "path": "/", 2024-03-09T15:09:52.245 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:51.583833Z", 2024-03-09T15:09:52.245 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-03-09T15:09:52.245 INFO:teuthology.orchestra.run.smithi001.stdout: "group_ids": [] 2024-03-09T15:09:52.245 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:09:52.245 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:52.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.238+0000 7f2a0cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.246+0000 7f2b589d7980 20 remove_watcher() i=0 2024-03-09T15:09:52.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.246+0000 7f2b589d7980 2 removed watcher, disabling cache 2024-03-09T15:09:52.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.246+0000 7f2b589d7980 20 remove_watcher() i=1 2024-03-09T15:09:52.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.250+0000 7f2b589d7980 20 remove_watcher() i=2 2024-03-09T15:09:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.254+0000 7f2b589d7980 20 remove_watcher() i=3 2024-03-09T15:09:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.254+0000 7f2b589d7980 20 remove_watcher() i=4 2024-03-09T15:09:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.254+0000 7f2b589d7980 20 remove_watcher() i=5 2024-03-09T15:09:52.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.258+0000 7f2b589d7980 20 remove_watcher() i=6 2024-03-09T15:09:52.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.258+0000 7f2b589d7980 20 remove_watcher() i=7 2024-03-09T15:09:52.277 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 --cluster ceph account create --account-id RGW88888888888888888 --account-name 'Mr. RGW88888888888888888' --email RGW88888888888888888+test@test.test 2024-03-09T15:09:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:52.355 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:52.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.394+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.398+0000 7f4ef3ec2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.398+0000 7f4ef3ec2980 20 realm 2024-03-09T15:09:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.398+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.398+0000 7f4ef3ec2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.398+0000 7f4ef3ec2980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.398+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.398+0000 7f4ef3ec2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.398+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.398+0000 7f4ef3ec2980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:52.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.398+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.402+0000 7f4ef3ec2980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:52.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.402+0000 7f4ef3ec2980 20 searching for the correct realm 2024-03-09T15:09:52.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.406+0000 7f4ef11d9640 0 -- 172.21.15.1:0/3933640305 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f4ee404c040 msgr2=0x7f4ee404c930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T15:09:52.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.414+0000 7f4ef11d9640 0 -- 172.21.15.1:0/3933640305 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f4dbc076ce0 msgr2=0x7f4dbc079110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:52.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 zone default found 2024-03-09T15:09:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 4 Realm: () 2024-03-09T15:09:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 zonegroup default 2024-03-09T15:09:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.422+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.426+0000 7f4ef3ec2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.426+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.426+0000 7f4ef3ec2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.426+0000 7f4ef3ec2980 20 started sync module instance, tier type = 2024-03-09T15:09:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.426+0000 7f4ef3ec2980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.442+0000 7f4ef3ec2980 20 add_watcher() i=0 2024-03-09T15:09:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.442+0000 7f4ef3ec2980 20 add_watcher() i=1 2024-03-09T15:09:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.442+0000 7f4ef3ec2980 20 add_watcher() i=2 2024-03-09T15:09:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.442+0000 7f4ef3ec2980 20 add_watcher() i=3 2024-03-09T15:09:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.442+0000 7f4ef3ec2980 20 add_watcher() i=4 2024-03-09T15:09:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.442+0000 7f4ef3ec2980 20 add_watcher() i=5 2024-03-09T15:09:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.442+0000 7f4ef3ec2980 20 add_watcher() i=6 2024-03-09T15:09:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.442+0000 7f4ef3ec2980 20 add_watcher() i=7 2024-03-09T15:09:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.442+0000 7f4ef3ec2980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.442+0000 7f4da37fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 5 note: GC not initialized 2024-03-09T15:09:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4da27fc640 20 reqs_thread_entry: start 2024-03-09T15:09:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 20 Filter name: none 2024-03-09T15:09:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4d13fff640 20 reqs_thread_entry: start 2024-03-09T15:09:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 10 cache get: name=default.rgw.meta+accounts+name.$Mr. RGW88888888888888888 : miss 2024-03-09T15:09:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 10 cache put: name=default.rgw.meta+accounts+name.$Mr. RGW88888888888888888 info.flags=0x0 2024-03-09T15:09:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 10 adding default.rgw.meta+accounts+name.$Mr. RGW88888888888888888 to cache LRU end 2024-03-09T15:09:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 20 failed to read name.$Mr. RGW88888888888888888 with: (2) No such file or directory 2024-03-09T15:09:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 10 cache get: name=default.rgw.meta+users.email+rgw88888888888888888+test@test.test : miss 2024-03-09T15:09:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 10 cache put: name=default.rgw.meta+users.email+rgw88888888888888888+test@test.test info.flags=0x0 2024-03-09T15:09:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 10 adding default.rgw.meta+users.email+rgw88888888888888888+test@test.test to cache LRU end 2024-03-09T15:09:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.446+0000 7f4ef3ec2980 20 failed to read rgw88888888888888888+test@test.test with: (2) No such file or directory 2024-03-09T15:09:52.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.450+0000 7f4ef3ec2980 10 cache put: name=default.rgw.meta+accounts+account.RGW88888888888888888 info.flags=0x17 2024-03-09T15:09:52.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.450+0000 7f4ef3ec2980 10 adding default.rgw.meta+accounts+account.RGW88888888888888888 to cache LRU end 2024-03-09T15:09:52.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.450+0000 7f4ef3ec2980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:accounts:account.RGW88888888888888888, ofs0, ns] 2024-03-09T15:09:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.450+0000 7f4dd57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464376 cookie 94419093786576 notifier 7845 bl.length()=353 2024-03-09T15:09:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.450+0000 7f4dd57fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+accounts+account.RGW88888888888888888 info.flags=0x17 2024-03-09T15:09:52.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.450+0000 7f4dd57fa640 10 rgw watcher librados: moving default.rgw.meta+accounts+account.RGW88888888888888888 to cache LRU end 2024-03-09T15:09:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.454+0000 7f4ef3ec2980 10 cache put: name=default.rgw.meta+accounts+name.$Mr. RGW88888888888888888 info.flags=0x17 2024-03-09T15:09:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.454+0000 7f4ef3ec2980 10 moving default.rgw.meta+accounts+name.$Mr. RGW88888888888888888 to cache LRU end 2024-03-09T15:09:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.454+0000 7f4ef3ec2980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:accounts:name.$Mr. RGW88888888888888888, ofs0, ns] 2024-03-09T15:09:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.454+0000 7f4dd57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464403 cookie 94419093779664 notifier 7845 bl.length()=226 2024-03-09T15:09:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.454+0000 7f4dd57fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+accounts+name.$Mr. RGW88888888888888888 info.flags=0x17 2024-03-09T15:09:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.454+0000 7f4dd57fa640 10 rgw watcher librados: moving default.rgw.meta+accounts+name.$Mr. RGW88888888888888888 to cache LRU end 2024-03-09T15:09:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.458+0000 7f4ef3ec2980 10 cache put: name=default.rgw.meta+users.email+rgw88888888888888888+test@test.test info.flags=0x17 2024-03-09T15:09:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.458+0000 7f4ef3ec2980 10 moving default.rgw.meta+users.email+rgw88888888888888888+test@test.test to cache LRU end 2024-03-09T15:09:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.458+0000 7f4ef3ec2980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.email:rgw88888888888888888+test@test.test, ofs0, ns] 2024-03-09T15:09:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.458+0000 7f4dd57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464357 cookie 94419093792496 notifier 7845 bl.length()=234 2024-03-09T15:09:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.458+0000 7f4dd57fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+rgw88888888888888888+test@test.test info.flags=0x17 2024-03-09T15:09:52.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.458+0000 7f4dd57fa640 10 rgw watcher librados: moving default.rgw.meta+users.email+rgw88888888888888888+test@test.test to cache LRU end 2024-03-09T15:09:52.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.462+0000 7f4da37fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.466+0000 7f4ef3ec2980 20 remove_watcher() i=0 2024-03-09T15:09:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.466+0000 7f4ef3ec2980 2 removed watcher, disabling cache 2024-03-09T15:09:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.470+0000 7f4ef3ec2980 20 remove_watcher() i=1 2024-03-09T15:09:52.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.470+0000 7f4ef3ec2980 20 remove_watcher() i=2 2024-03-09T15:09:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.474+0000 7f4ef3ec2980 20 remove_watcher() i=3 2024-03-09T15:09:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.474+0000 7f4ef3ec2980 20 remove_watcher() i=4 2024-03-09T15:09:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.478+0000 7f4ef3ec2980 20 remove_watcher() i=5 2024-03-09T15:09:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.478+0000 7f4ef3ec2980 20 remove_watcher() i=6 2024-03-09T15:09:52.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.478+0000 7f4ef3ec2980 20 remove_watcher() i=7 2024-03-09T15:09:52.494 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:09:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "id": "RGW88888888888888888", 2024-03-09T15:09:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "tenant": "", 2024-03-09T15:09:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "Mr. RGW88888888888888888", 2024-03-09T15:09:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "email": "RGW88888888888888888+test@test.test", 2024-03-09T15:09:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "quota": { 2024-03-09T15:09:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "max_users": 1000, 2024-03-09T15:09:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "max_roles": 1000, 2024-03-09T15:09:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "max_groups": 1000, 2024-03-09T15:09:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "max_buckets": 1000, 2024-03-09T15:09:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "max_access_keys": 4 2024-03-09T15:09:52.495 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:09:52.497 DEBUG:tasks.s3tests:Creating user RGW88888888888888888 on client.0 2024-03-09T15:09:52.497 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid root1.client.0 --display-name Mr.RGW88888888888888888 --email root1+test@test.test --access-key CYOOHHIKEHQKRVCYAGZI --secret 4Y9PZU2ErHiwpVuouNZnBLdzsvFKKKs4LfMAu0n5TksHw0SuJJf5tg== --cluster ceph --account-id RGW88888888888888888 --account-root 2024-03-09T15:09:52.587 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:52.587 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:52.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.626+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.626+0000 7fe48c571980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.626+0000 7fe48c571980 20 realm 2024-03-09T15:09:52.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.626+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.626+0000 7fe48c571980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.626+0000 7fe48c571980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.626+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.626+0000 7fe48c571980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.626+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.626+0000 7fe48c571980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.630+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.630+0000 7fe48c571980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.630+0000 7fe48c571980 20 searching for the correct realm 2024-03-09T15:09:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.634+0000 7fe48a88a640 0 -- 172.21.15.1:0/195897007 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x5566044e67d0 msgr2=0x5566044e8c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.642+0000 7fe489888640 0 -- 172.21.15.1:0/195897007 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fe47c04c030 msgr2=0x7fe47c01e840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:52.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:52.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:52.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:52.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:52.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:52.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:52.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:52.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:52.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 zone default found 2024-03-09T15:09:52.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 4 Realm: () 2024-03-09T15:09:52.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:52.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:52.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:52.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 zonegroup default 2024-03-09T15:09:52.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.650+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.654+0000 7fe48c571980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.654+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.654+0000 7fe48c571980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.654+0000 7fe48c571980 20 started sync module instance, tier type = 2024-03-09T15:09:52.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.654+0000 7fe48c571980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:52.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.670+0000 7fe48c571980 20 add_watcher() i=0 2024-03-09T15:09:52.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.670+0000 7fe48c571980 20 add_watcher() i=1 2024-03-09T15:09:52.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.670+0000 7fe48c571980 20 add_watcher() i=2 2024-03-09T15:09:52.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.670+0000 7fe48c571980 20 add_watcher() i=3 2024-03-09T15:09:52.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.670+0000 7fe48c571980 20 add_watcher() i=4 2024-03-09T15:09:52.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.670+0000 7fe48c571980 20 add_watcher() i=5 2024-03-09T15:09:52.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.670+0000 7fe48c571980 20 add_watcher() i=6 2024-03-09T15:09:52.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.670+0000 7fe48c571980 20 add_watcher() i=7 2024-03-09T15:09:52.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.670+0000 7fe48c571980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:52.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe340fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:52.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:52.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:52.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 5 note: GC not initialized 2024-03-09T15:09:52.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe2bb7fe640 20 reqs_thread_entry: start 2024-03-09T15:09:52.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:52.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 20 Filter name: none 2024-03-09T15:09:52.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe2b97fa640 20 reqs_thread_entry: start 2024-03-09T15:09:52.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 cache get: name=default.rgw.meta+users.uid+root1.client.0 : miss 2024-03-09T15:09:52.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x0 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 adding default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 cache get: name=default.rgw.meta+users.email+root1+test@test.test : miss 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 cache put: name=default.rgw.meta+users.email+root1+test@test.test info.flags=0x0 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 adding default.rgw.meta+users.email+root1+test@test.test to cache LRU end 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 cache get: name=default.rgw.meta+users.keys+CYOOHHIKEHQKRVCYAGZI : miss 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 cache put: name=default.rgw.meta+users.keys+CYOOHHIKEHQKRVCYAGZI info.flags=0x0 2024-03-09T15:09:52.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 adding default.rgw.meta+users.keys+CYOOHHIKEHQKRVCYAGZI to cache LRU end 2024-03-09T15:09:52.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 cache get: name=default.rgw.meta+accounts+account.RGW88888888888888888 : miss 2024-03-09T15:09:52.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 20 rados_obj.operate() r=0 bl.length=153 2024-03-09T15:09:52.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 cache put: name=default.rgw.meta+accounts+account.RGW88888888888888888 info.flags=0x17 2024-03-09T15:09:52.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 adding default.rgw.meta+accounts+account.RGW88888888888888888 to cache LRU end 2024-03-09T15:09:52.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T15:09:52.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 cache get: name=default.rgw.meta+users.keys+CYOOHHIKEHQKRVCYAGZI : hit (negative entry) 2024-03-09T15:09:52.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.674+0000 7fe48c571980 10 cache get: name=default.rgw.meta+users.keys+CYOOHHIKEHQKRVCYAGZI : hit (negative entry) 2024-03-09T15:09:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.678+0000 7fe48c571980 10 cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x17 2024-03-09T15:09:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.678+0000 7fe48c571980 10 moving default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2024-03-09T15:09:52.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.678+0000 7fe48c571980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:root1.client.0, ofs0, ns] 2024-03-09T15:09:52.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.678+0000 7fe36dffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464377 cookie 93896647324000 notifier 7863 bl.length()=681 2024-03-09T15:09:52.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.678+0000 7fe36dffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x17 2024-03-09T15:09:52.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.678+0000 7fe36dffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2024-03-09T15:09:52.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.682+0000 7fe48c571980 10 cache put: name=default.rgw.meta+users.email+root1+test@test.test info.flags=0x7 2024-03-09T15:09:52.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.682+0000 7fe48c571980 10 moving default.rgw.meta+users.email+root1+test@test.test to cache LRU end 2024-03-09T15:09:52.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.682+0000 7fe48c571980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.email:root1+test@test.test, ofs0, ns] 2024-03-09T15:09:52.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.682+0000 7fe36dffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464420 cookie 93896647325056 notifier 7863 bl.length()=189 2024-03-09T15:09:52.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.682+0000 7fe36dffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+root1+test@test.test info.flags=0x7 2024-03-09T15:09:52.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.682+0000 7fe36dffb640 10 rgw watcher librados: moving default.rgw.meta+users.email+root1+test@test.test to cache LRU end 2024-03-09T15:09:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.690+0000 7fe48c571980 10 cache put: name=default.rgw.meta+users.keys+CYOOHHIKEHQKRVCYAGZI info.flags=0x7 2024-03-09T15:09:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.690+0000 7fe48c571980 10 moving default.rgw.meta+users.keys+CYOOHHIKEHQKRVCYAGZI to cache LRU end 2024-03-09T15:09:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.690+0000 7fe48c571980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:CYOOHHIKEHQKRVCYAGZI, ofs0, ns] 2024-03-09T15:09:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.690+0000 7fe36dffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464378 cookie 93896647324000 notifier 7863 bl.length()=188 2024-03-09T15:09:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.690+0000 7fe36dffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+CYOOHHIKEHQKRVCYAGZI info.flags=0x7 2024-03-09T15:09:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.690+0000 7fe36dffb640 10 rgw watcher librados: moving default.rgw.meta+users.keys+CYOOHHIKEHQKRVCYAGZI to cache LRU end 2024-03-09T15:09:52.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.694+0000 7fe48c571980 20 linked user root1.client.0 to account RGW88888888888888888 2024-03-09T15:09:52.696 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:09:52.696 INFO:teuthology.orchestra.run.smithi001.stdout: "user_id": "root1.client.0", 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: "display_name": "Mr.RGW88888888888888888", 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: "email": "root1+test@test.test", 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: "suspended": 0, 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: "max_buckets": 1000, 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: "subusers": [], 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: "keys": [ 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: "user": "root1.client.0", 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: "access_key": "CYOOHHIKEHQKRVCYAGZI", 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: "secret_key": "4Y9PZU2ErHiwpVuouNZnBLdzsvFKKKs4LfMAu0n5TksHw0SuJJf5tg==", 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: "active": true, 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:52.681504Z" 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:09:52.697 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "swift_keys": [], 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": [], 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "default_placement": "", 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "default_storage_class": "", 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "placement_tags": [], 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "bucket_quota": { 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "user_quota": { 2024-03-09T15:09:52.698 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: "temp_url_keys": [], 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "root", 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: "mfa_ids": [], 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: "account_id": "RGW88888888888888888", 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: "path": "/", 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:52.681490Z", 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: "group_ids": [] 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:09:52.699 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:52.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.694+0000 7fe340fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:52.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.698+0000 7fe48c571980 20 remove_watcher() i=0 2024-03-09T15:09:52.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.702+0000 7fe48c571980 2 removed watcher, disabling cache 2024-03-09T15:09:52.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.702+0000 7fe48c571980 20 remove_watcher() i=1 2024-03-09T15:09:52.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.706+0000 7fe48c571980 20 remove_watcher() i=2 2024-03-09T15:09:52.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.706+0000 7fe48c571980 20 remove_watcher() i=3 2024-03-09T15:09:52.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.710+0000 7fe48c571980 20 remove_watcher() i=4 2024-03-09T15:09:52.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.710+0000 7fe48c571980 20 remove_watcher() i=5 2024-03-09T15:09:52.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.710+0000 7fe48c571980 20 remove_watcher() i=6 2024-03-09T15:09:52.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.714+0000 7fe48c571980 20 remove_watcher() i=7 2024-03-09T15:09:52.731 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid root1.client.0 --totp-serial 7292394606 --totp-seed XU5NDCNI5RSOSX5PA27ASE2UOEZYFF4PBSRADTRRW4PV4BWMXHMVHC36GY3NJBKU --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-03-09T15:09:52.770 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:52.770 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:52.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.810+0000 7fb9ffe20980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.810+0000 7fb9ffe20980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.810+0000 7fb9ffe20980 20 realm 2024-03-09T15:09:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.810+0000 7fb9ffe20980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.810+0000 7fb9ffe20980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.810+0000 7fb9ffe20980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.810+0000 7fb9ffe20980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.810+0000 7fb9ffe20980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.810+0000 7fb9ffe20980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.810+0000 7fb9ffe20980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:52.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.810+0000 7fb9ffe20980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.814+0000 7fb9ffe20980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:52.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.814+0000 7fb9ffe20980 20 searching for the correct realm 2024-03-09T15:09:52.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.818+0000 7fb9fe139640 0 -- 172.21.15.1:0/3537974815 >> [v2:172.21.15.1:6805/3585289272,v1:172.21.15.1:6808/3585289272] conn(0x56473049e690 msgr2=0x5647304a0b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:52.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.826+0000 7fb9fd137640 0 -- 172.21.15.1:0/3537974815 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fb9f000fbf0 msgr2=0x7fb9f001d8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:52.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:52.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:52.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:52.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:52.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:52.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:52.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:52.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:52.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 zone default found 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 4 Realm: () 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 zonegroup default 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.834+0000 7fb9ffe20980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.838+0000 7fb9ffe20980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.838+0000 7fb9ffe20980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.838+0000 7fb9ffe20980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.838+0000 7fb9ffe20980 20 started sync module instance, tier type = 2024-03-09T15:09:52.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.838+0000 7fb9ffe20980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:52.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.854+0000 7fb9ffe20980 20 add_watcher() i=0 2024-03-09T15:09:52.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.854+0000 7fb9ffe20980 20 add_watcher() i=1 2024-03-09T15:09:52.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.854+0000 7fb9ffe20980 20 add_watcher() i=2 2024-03-09T15:09:52.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.854+0000 7fb9ffe20980 20 add_watcher() i=3 2024-03-09T15:09:52.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.854+0000 7fb9ffe20980 20 add_watcher() i=4 2024-03-09T15:09:52.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.854+0000 7fb9ffe20980 20 add_watcher() i=5 2024-03-09T15:09:52.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.854+0000 7fb9ffe20980 20 add_watcher() i=6 2024-03-09T15:09:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.854+0000 7fb9ffe20980 20 add_watcher() i=7 2024-03-09T15:09:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.854+0000 7fb9ffe20980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:52.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb8af7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:52.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb9ffe20980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:52.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb9ffe20980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:52.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb9ffe20980 5 note: GC not initialized 2024-03-09T15:09:52.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb8ae7fc640 20 reqs_thread_entry: start 2024-03-09T15:09:52.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb9ffe20980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:52.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb9ffe20980 20 Filter name: none 2024-03-09T15:09:52.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb823fff640 20 reqs_thread_entry: start 2024-03-09T15:09:52.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb9ffe20980 10 cache get: name=default.rgw.meta+users.uid+root1.client.0 : miss 2024-03-09T15:09:52.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb9ffe20980 20 rados->read ofs=0 len=0 2024-03-09T15:09:52.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb9ffe20980 20 rados_obj.operate() r=0 bl.length=494 2024-03-09T15:09:52.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb9ffe20980 10 cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x13 2024-03-09T15:09:52.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb9ffe20980 10 adding default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2024-03-09T15:09:52.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.858+0000 7fb9ffe20980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T15:09:52.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.862+0000 7fb9ffe20980 10 cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x17 2024-03-09T15:09:52.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.862+0000 7fb9ffe20980 10 moving default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2024-03-09T15:09:52.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.862+0000 7fb9ffe20980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:root1.client.0, ofs0, ns] 2024-03-09T15:09:52.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.862+0000 7fb8e17fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464379 cookie 94863752868080 notifier 7310 bl.length()=695 2024-03-09T15:09:52.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.862+0000 7fb8e17fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+root1.client.0 info.flags=0x17 2024-03-09T15:09:52.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.862+0000 7fb8e17fa640 10 rgw watcher librados: moving default.rgw.meta+users.uid+root1.client.0 to cache LRU end 2024-03-09T15:09:52.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.866+0000 7fb8af7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:52.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.870+0000 7fb9ffe20980 20 remove_watcher() i=0 2024-03-09T15:09:52.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.870+0000 7fb9ffe20980 2 removed watcher, disabling cache 2024-03-09T15:09:52.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.874+0000 7fb9ffe20980 20 remove_watcher() i=1 2024-03-09T15:09:52.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.874+0000 7fb9ffe20980 20 remove_watcher() i=2 2024-03-09T15:09:52.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.878+0000 7fb9ffe20980 20 remove_watcher() i=3 2024-03-09T15:09:52.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.878+0000 7fb9ffe20980 20 remove_watcher() i=4 2024-03-09T15:09:52.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.878+0000 7fb9ffe20980 20 remove_watcher() i=5 2024-03-09T15:09:52.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.882+0000 7fb9ffe20980 20 remove_watcher() i=6 2024-03-09T15:09:52.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:52.882+0000 7fb9ffe20980 20 remove_watcher() i=7 2024-03-09T15:09:52.899 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 --cluster ceph account create --account-id RGW99999999999999999 --account-name 'Mr. RGW99999999999999999' --email RGW99999999999999999+test@test.test 2024-03-09T15:09:52.982 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:52.983 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:53.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.022+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.026+0000 7fe5506b2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.026+0000 7fe5506b2980 20 realm 2024-03-09T15:09:53.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.026+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.026+0000 7fe5506b2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.026+0000 7fe5506b2980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:53.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.026+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.026+0000 7fe5506b2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.026+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.026+0000 7fe5506b2980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:53.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.026+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.026+0000 7fe5506b2980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:53.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.026+0000 7fe5506b2980 20 searching for the correct realm 2024-03-09T15:09:53.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.034+0000 7fe54d9c9640 0 -- 172.21.15.1:0/1210104052 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fe5400e2030 msgr2=0x7fe5400cd410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:53.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.038+0000 7fe54e1ca640 0 -- 172.21.15.1:0/1210104052 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7fe54406ec30 msgr2=0x7fe544045120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:53.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.254+0000 7fe5506b2980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:53.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.254+0000 7fe5506b2980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:53.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.254+0000 7fe5506b2980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:53.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.254+0000 7fe5506b2980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:53.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.254+0000 7fe5506b2980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:53.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.254+0000 7fe5506b2980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:53.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.254+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.254+0000 7fe5506b2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.254+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:53.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 zone default found 2024-03-09T15:09:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 4 Realm: () 2024-03-09T15:09:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 zonegroup default 2024-03-09T15:09:53.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 started sync module instance, tier type = 2024-03-09T15:09:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.258+0000 7fe5506b2980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:53.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.282+0000 7fe5506b2980 20 add_watcher() i=0 2024-03-09T15:09:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.282+0000 7fe5506b2980 20 add_watcher() i=1 2024-03-09T15:09:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.282+0000 7fe5506b2980 20 add_watcher() i=2 2024-03-09T15:09:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.282+0000 7fe5506b2980 20 add_watcher() i=3 2024-03-09T15:09:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.282+0000 7fe5506b2980 20 add_watcher() i=4 2024-03-09T15:09:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.282+0000 7fe5506b2980 20 add_watcher() i=5 2024-03-09T15:09:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.282+0000 7fe5506b2980 20 add_watcher() i=6 2024-03-09T15:09:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.286+0000 7fe5506b2980 20 add_watcher() i=7 2024-03-09T15:09:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.286+0000 7fe5506b2980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.290+0000 7fe5506b2980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.290+0000 7fe5506b2980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.290+0000 7fe5506b2980 5 note: GC not initialized 2024-03-09T15:09:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.290+0000 7fe37f7fe640 20 reqs_thread_entry: start 2024-03-09T15:09:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.290+0000 7fe404fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.290+0000 7fe5506b2980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.290+0000 7fe5506b2980 20 Filter name: none 2024-03-09T15:09:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.290+0000 7fe37d7fa640 20 reqs_thread_entry: start 2024-03-09T15:09:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.290+0000 7fe5506b2980 10 cache get: name=default.rgw.meta+accounts+name.$Mr. RGW99999999999999999 : miss 2024-03-09T15:09:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.290+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.294+0000 7fe5506b2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.294+0000 7fe5506b2980 10 cache put: name=default.rgw.meta+accounts+name.$Mr. RGW99999999999999999 info.flags=0x0 2024-03-09T15:09:53.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.294+0000 7fe5506b2980 10 adding default.rgw.meta+accounts+name.$Mr. RGW99999999999999999 to cache LRU end 2024-03-09T15:09:53.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.294+0000 7fe5506b2980 20 failed to read name.$Mr. RGW99999999999999999 with: (2) No such file or directory 2024-03-09T15:09:53.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.294+0000 7fe5506b2980 10 cache get: name=default.rgw.meta+users.email+rgw99999999999999999+test@test.test : miss 2024-03-09T15:09:53.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.294+0000 7fe5506b2980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.294+0000 7fe5506b2980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.294+0000 7fe5506b2980 10 cache put: name=default.rgw.meta+users.email+rgw99999999999999999+test@test.test info.flags=0x0 2024-03-09T15:09:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.294+0000 7fe5506b2980 10 adding default.rgw.meta+users.email+rgw99999999999999999+test@test.test to cache LRU end 2024-03-09T15:09:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.294+0000 7fe5506b2980 20 failed to read rgw99999999999999999+test@test.test with: (2) No such file or directory 2024-03-09T15:09:53.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.298+0000 7fe5506b2980 10 cache put: name=default.rgw.meta+accounts+account.RGW99999999999999999 info.flags=0x17 2024-03-09T15:09:53.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.298+0000 7fe5506b2980 10 adding default.rgw.meta+accounts+account.RGW99999999999999999 to cache LRU end 2024-03-09T15:09:53.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.298+0000 7fe5506b2980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:accounts:account.RGW99999999999999999, ofs0, ns] 2024-03-09T15:09:53.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.298+0000 7fe431ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464380 cookie 94183088812912 notifier 7902 bl.length()=353 2024-03-09T15:09:53.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.298+0000 7fe431ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+accounts+account.RGW99999999999999999 info.flags=0x17 2024-03-09T15:09:53.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.298+0000 7fe431ffb640 10 rgw watcher librados: moving default.rgw.meta+accounts+account.RGW99999999999999999 to cache LRU end 2024-03-09T15:09:53.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.302+0000 7fe5506b2980 10 cache put: name=default.rgw.meta+accounts+name.$Mr. RGW99999999999999999 info.flags=0x17 2024-03-09T15:09:53.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.302+0000 7fe5506b2980 10 moving default.rgw.meta+accounts+name.$Mr. RGW99999999999999999 to cache LRU end 2024-03-09T15:09:53.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.302+0000 7fe5506b2980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:accounts:name.$Mr. RGW99999999999999999, ofs0, ns] 2024-03-09T15:09:53.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.306+0000 7fe431ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464404 cookie 94183088818112 notifier 7902 bl.length()=226 2024-03-09T15:09:53.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.306+0000 7fe431ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+accounts+name.$Mr. RGW99999999999999999 info.flags=0x17 2024-03-09T15:09:53.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.306+0000 7fe431ffb640 10 rgw watcher librados: moving default.rgw.meta+accounts+name.$Mr. RGW99999999999999999 to cache LRU end 2024-03-09T15:09:53.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.310+0000 7fe5506b2980 10 cache put: name=default.rgw.meta+users.email+rgw99999999999999999+test@test.test info.flags=0x17 2024-03-09T15:09:53.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.310+0000 7fe5506b2980 10 moving default.rgw.meta+users.email+rgw99999999999999999+test@test.test to cache LRU end 2024-03-09T15:09:53.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.310+0000 7fe5506b2980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.email:rgw99999999999999999+test@test.test, ofs0, ns] 2024-03-09T15:09:53.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.310+0000 7fe431ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464358 cookie 94183088824256 notifier 7902 bl.length()=234 2024-03-09T15:09:53.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.310+0000 7fe431ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+rgw99999999999999999+test@test.test info.flags=0x17 2024-03-09T15:09:53.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.310+0000 7fe431ffb640 10 rgw watcher librados: moving default.rgw.meta+users.email+rgw99999999999999999+test@test.test to cache LRU end 2024-03-09T15:09:53.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.314+0000 7fe404fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:53.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.318+0000 7fe5506b2980 20 remove_watcher() i=0 2024-03-09T15:09:53.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.318+0000 7fe5506b2980 2 removed watcher, disabling cache 2024-03-09T15:09:53.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.322+0000 7fe5506b2980 20 remove_watcher() i=1 2024-03-09T15:09:53.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.322+0000 7fe5506b2980 20 remove_watcher() i=2 2024-03-09T15:09:53.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.326+0000 7fe5506b2980 20 remove_watcher() i=3 2024-03-09T15:09:53.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.330+0000 7fe5506b2980 20 remove_watcher() i=4 2024-03-09T15:09:53.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.330+0000 7fe5506b2980 20 remove_watcher() i=5 2024-03-09T15:09:53.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.330+0000 7fe5506b2980 20 remove_watcher() i=6 2024-03-09T15:09:53.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.334+0000 7fe5506b2980 20 remove_watcher() i=7 2024-03-09T15:09:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:09:53.348 INFO:teuthology.orchestra.run.smithi001.stdout: "id": "RGW99999999999999999", 2024-03-09T15:09:53.348 INFO:teuthology.orchestra.run.smithi001.stdout: "tenant": "", 2024-03-09T15:09:53.348 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "Mr. RGW99999999999999999", 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: "email": "RGW99999999999999999+test@test.test", 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: "quota": { 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: "max_users": 1000, 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: "max_roles": 1000, 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: "max_groups": 1000, 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: "max_buckets": 1000, 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout: "max_access_keys": 4 2024-03-09T15:09:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:09:53.351 DEBUG:tasks.s3tests:Creating user RGW99999999999999999 on client.0 2024-03-09T15:09:53.352 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid root2.client.0 --display-name Mr.RGW99999999999999999 --email root2+test@test.test --access-key HWKQTGEHYDQRCKBUKXNY --secret wX1u3FW1I9+Xicv/TIrSkUigy9DKf12UaJqoNKABOMs18uUqwpGqpQ== --cluster ceph --account-id RGW99999999999999999 --account-root 2024-03-09T15:09:53.444 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:53.444 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.482+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.486+0000 7fdd01de1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.486+0000 7fdd01de1980 20 realm 2024-03-09T15:09:53.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.486+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.486+0000 7fdd01de1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.486+0000 7fdd01de1980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:53.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.486+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.486+0000 7fdd01de1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.486+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.486+0000 7fdd01de1980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:53.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.486+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.486+0000 7fdd01de1980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:53.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.486+0000 7fdd01de1980 20 searching for the correct realm 2024-03-09T15:09:53.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.494+0000 7fdcff0f8640 0 -- 172.21.15.1:0/1950076493 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fdcf00c1c30 msgr2=0x7fdcf00ef660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:53.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.502+0000 7fdd000fa640 0 -- 172.21.15.1:0/1950076493 >> [v2:172.21.15.53:6800/1545189291,v1:172.21.15.53:6805/1545189291] conn(0x7fdcf401a040 msgr2=0x7fdcf4010070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:53.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.506+0000 7fdd01de1980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:53.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.506+0000 7fdd01de1980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:53.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.506+0000 7fdd01de1980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:53.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.506+0000 7fdd01de1980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:53.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.506+0000 7fdd01de1980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:53.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.506+0000 7fdd01de1980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:53.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.506+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.506+0000 7fdd01de1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.506+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:53.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:53.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 zone default found 2024-03-09T15:09:53.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 4 Realm: () 2024-03-09T15:09:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 zonegroup default 2024-03-09T15:09:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 started sync module instance, tier type = 2024-03-09T15:09:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.510+0000 7fdd01de1980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:53.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.526+0000 7fdd01de1980 20 add_watcher() i=0 2024-03-09T15:09:53.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.526+0000 7fdd01de1980 20 add_watcher() i=1 2024-03-09T15:09:53.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.526+0000 7fdd01de1980 20 add_watcher() i=2 2024-03-09T15:09:53.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.526+0000 7fdd01de1980 20 add_watcher() i=3 2024-03-09T15:09:53.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.526+0000 7fdd01de1980 20 add_watcher() i=4 2024-03-09T15:09:53.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.526+0000 7fdd01de1980 20 add_watcher() i=5 2024-03-09T15:09:53.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.526+0000 7fdd01de1980 20 add_watcher() i=6 2024-03-09T15:09:53.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.526+0000 7fdd01de1980 20 add_watcher() i=7 2024-03-09T15:09:53.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.526+0000 7fdd01de1980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:53.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdbb67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:53.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:53.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:53.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 5 note: GC not initialized 2024-03-09T15:09:53.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdbb57e2640 20 reqs_thread_entry: start 2024-03-09T15:09:53.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:53.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 20 Filter name: none 2024-03-09T15:09:53.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdbae7fc640 20 reqs_thread_entry: start 2024-03-09T15:09:53.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 10 cache get: name=default.rgw.meta+users.uid+root2.client.0 : miss 2024-03-09T15:09:53.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 10 cache put: name=default.rgw.meta+users.uid+root2.client.0 info.flags=0x0 2024-03-09T15:09:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 10 adding default.rgw.meta+users.uid+root2.client.0 to cache LRU end 2024-03-09T15:09:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 10 cache get: name=default.rgw.meta+users.email+root2+test@test.test : miss 2024-03-09T15:09:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 10 cache put: name=default.rgw.meta+users.email+root2+test@test.test info.flags=0x0 2024-03-09T15:09:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 10 adding default.rgw.meta+users.email+root2+test@test.test to cache LRU end 2024-03-09T15:09:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 10 cache get: name=default.rgw.meta+users.keys+HWKQTGEHYDQRCKBUKXNY : miss 2024-03-09T15:09:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 10 cache put: name=default.rgw.meta+users.keys+HWKQTGEHYDQRCKBUKXNY info.flags=0x0 2024-03-09T15:09:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 10 adding default.rgw.meta+users.keys+HWKQTGEHYDQRCKBUKXNY to cache LRU end 2024-03-09T15:09:53.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 10 cache get: name=default.rgw.meta+accounts+account.RGW99999999999999999 : miss 2024-03-09T15:09:53.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.530+0000 7fdd01de1980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.534+0000 7fdd01de1980 20 rados_obj.operate() r=0 bl.length=153 2024-03-09T15:09:53.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.534+0000 7fdd01de1980 10 cache put: name=default.rgw.meta+accounts+account.RGW99999999999999999 info.flags=0x17 2024-03-09T15:09:53.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.534+0000 7fdd01de1980 10 adding default.rgw.meta+accounts+account.RGW99999999999999999 to cache LRU end 2024-03-09T15:09:53.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.534+0000 7fdd01de1980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T15:09:53.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.534+0000 7fdd01de1980 10 cache get: name=default.rgw.meta+users.keys+HWKQTGEHYDQRCKBUKXNY : hit (negative entry) 2024-03-09T15:09:53.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.534+0000 7fdd01de1980 10 cache get: name=default.rgw.meta+users.keys+HWKQTGEHYDQRCKBUKXNY : hit (negative entry) 2024-03-09T15:09:53.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.534+0000 7fdd01de1980 10 cache put: name=default.rgw.meta+users.uid+root2.client.0 info.flags=0x17 2024-03-09T15:09:53.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.534+0000 7fdd01de1980 10 moving default.rgw.meta+users.uid+root2.client.0 to cache LRU end 2024-03-09T15:09:53.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.534+0000 7fdd01de1980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:root2.client.0, ofs0, ns] 2024-03-09T15:09:53.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.534+0000 7fdbdeffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464381 cookie 94417658557472 notifier 7920 bl.length()=681 2024-03-09T15:09:53.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.534+0000 7fdbdeffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+root2.client.0 info.flags=0x17 2024-03-09T15:09:53.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.534+0000 7fdbdeffd640 10 rgw watcher librados: moving default.rgw.meta+users.uid+root2.client.0 to cache LRU end 2024-03-09T15:09:53.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.546+0000 7fdd01de1980 10 cache put: name=default.rgw.meta+users.email+root2+test@test.test info.flags=0x7 2024-03-09T15:09:53.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.546+0000 7fdd01de1980 10 moving default.rgw.meta+users.email+root2+test@test.test to cache LRU end 2024-03-09T15:09:53.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.546+0000 7fdd01de1980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.email:root2+test@test.test, ofs0, ns] 2024-03-09T15:09:53.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.546+0000 7fdbdeffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464421 cookie 94417658567408 notifier 7920 bl.length()=189 2024-03-09T15:09:53.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.546+0000 7fdbdeffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+root2+test@test.test info.flags=0x7 2024-03-09T15:09:53.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.546+0000 7fdbdeffd640 10 rgw watcher librados: moving default.rgw.meta+users.email+root2+test@test.test to cache LRU end 2024-03-09T15:09:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.550+0000 7fdd01de1980 10 cache put: name=default.rgw.meta+users.keys+HWKQTGEHYDQRCKBUKXNY info.flags=0x7 2024-03-09T15:09:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.550+0000 7fdd01de1980 10 moving default.rgw.meta+users.keys+HWKQTGEHYDQRCKBUKXNY to cache LRU end 2024-03-09T15:09:53.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.550+0000 7fdd01de1980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:HWKQTGEHYDQRCKBUKXNY, ofs0, ns] 2024-03-09T15:09:53.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.550+0000 7fdbde7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464402 cookie 94417658561072 notifier 7920 bl.length()=188 2024-03-09T15:09:53.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.550+0000 7fdbde7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+HWKQTGEHYDQRCKBUKXNY info.flags=0x7 2024-03-09T15:09:53.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.550+0000 7fdbde7fc640 10 rgw watcher librados: moving default.rgw.meta+users.keys+HWKQTGEHYDQRCKBUKXNY to cache LRU end 2024-03-09T15:09:53.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.558+0000 7fdd01de1980 20 linked user root2.client.0 to account RGW99999999999999999 2024-03-09T15:09:53.559 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:09:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "user_id": "root2.client.0", 2024-03-09T15:09:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "display_name": "Mr.RGW99999999999999999", 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "email": "root2+test@test.test", 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "suspended": 0, 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "max_buckets": 1000, 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "subusers": [], 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "keys": [ 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "user": "root2.client.0", 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "access_key": "HWKQTGEHYDQRCKBUKXNY", 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "secret_key": "wX1u3FW1I9+Xicv/TIrSkUigy9DKf12UaJqoNKABOMs18uUqwpGqpQ==", 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "active": true, 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:53.538077Z" 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-03-09T15:09:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "swift_keys": [], 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": [], 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "op_mask": "read, write, delete", 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "default_placement": "", 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "default_storage_class": "", 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "placement_tags": [], 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "bucket_quota": { 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "user_quota": { 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": false, 2024-03-09T15:09:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "check_on_raw": false, 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size": -1, 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "max_size_kb": 0, 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "max_objects": -1 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "temp_url_keys": [], 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "root", 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "mfa_ids": [], 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "account_id": "RGW99999999999999999", 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "path": "/", 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "create_date": "2024-03-09T15:09:53.538063Z", 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "group_ids": [] 2024-03-09T15:09:53.562 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:09:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:09:53.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.558+0000 7fdbb67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:53.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.562+0000 7fdd01de1980 20 remove_watcher() i=0 2024-03-09T15:09:53.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.562+0000 7fdd01de1980 2 removed watcher, disabling cache 2024-03-09T15:09:53.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.566+0000 7fdd01de1980 20 remove_watcher() i=1 2024-03-09T15:09:53.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.566+0000 7fdd01de1980 20 remove_watcher() i=2 2024-03-09T15:09:53.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.570+0000 7fdd01de1980 20 remove_watcher() i=3 2024-03-09T15:09:53.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.570+0000 7fdd01de1980 20 remove_watcher() i=4 2024-03-09T15:09:53.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.574+0000 7fdd01de1980 20 remove_watcher() i=5 2024-03-09T15:09:53.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.574+0000 7fdd01de1980 20 remove_watcher() i=6 2024-03-09T15:09:53.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.578+0000 7fdd01de1980 20 remove_watcher() i=7 2024-03-09T15:09:53.594 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid root2.client.0 --totp-serial 6085624223 --totp-seed 6QJSEWS6JWVR45JJS6ZS5DDYOBCA3HG2LHX5JPYY4UPOCXR266N4IXMYPFYSEXWA --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-03-09T15:09:53.639 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:09:53.639 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:09:53.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.678+0000 7fae85c0e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.678+0000 7fae85c0e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.678+0000 7fae85c0e980 20 realm 2024-03-09T15:09:53.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.678+0000 7fae85c0e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.678+0000 7fae85c0e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.678+0000 7fae85c0e980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:09:53.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.678+0000 7fae85c0e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.678+0000 7fae85c0e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.678+0000 7fae85c0e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.682+0000 7fae85c0e980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:53.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.682+0000 7fae85c0e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.682+0000 7fae85c0e980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:09:53.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.682+0000 7fae85c0e980 20 searching for the correct realm 2024-03-09T15:09:53.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.690+0000 7fae82f25640 0 -- 172.21.15.1:0/60761892 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fae74030030 msgr2=0x7fae74017410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-03-09T15:09:53.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.698+0000 7fae82f25640 0 -- 172.21.15.1:0/60761892 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7fad4c076bc0 msgr2=0x7fad4c078ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:09:53.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.702+0000 7fae85c0e980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:09:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.702+0000 7fae85c0e980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:09:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.702+0000 7fae85c0e980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:09:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.702+0000 7fae85c0e980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:09:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.702+0000 7fae85c0e980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:09:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.702+0000 7fae85c0e980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:09:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.702+0000 7fae85c0e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.702+0000 7fae85c0e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.702+0000 7fae85c0e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:09:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:09:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 zone default found 2024-03-09T15:09:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 4 Realm: () 2024-03-09T15:09:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:09:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:09:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:09:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 zonegroup default 2024-03-09T15:09:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:09:53.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 started sync module instance, tier type = 2024-03-09T15:09:53.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.706+0000 7fae85c0e980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:09:53.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.722+0000 7fae85c0e980 20 add_watcher() i=0 2024-03-09T15:09:53.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.722+0000 7fae85c0e980 20 add_watcher() i=1 2024-03-09T15:09:53.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.722+0000 7fae85c0e980 20 add_watcher() i=2 2024-03-09T15:09:53.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.722+0000 7fae85c0e980 20 add_watcher() i=3 2024-03-09T15:09:53.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.722+0000 7fae85c0e980 20 add_watcher() i=4 2024-03-09T15:09:53.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.722+0000 7fae85c0e980 20 add_watcher() i=5 2024-03-09T15:09:53.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.722+0000 7fae85c0e980 20 add_watcher() i=6 2024-03-09T15:09:53.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.726+0000 7fae85c0e980 20 add_watcher() i=7 2024-03-09T15:09:53.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.726+0000 7fae85c0e980 2 all 8 watchers are set, enabling cache 2024-03-09T15:09:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.726+0000 7fad39fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.726+0000 7fae85c0e980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:09:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.726+0000 7fae85c0e980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:09:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.726+0000 7fae85c0e980 5 note: GC not initialized 2024-03-09T15:09:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.726+0000 7fad38fe1640 20 reqs_thread_entry: start 2024-03-09T15:09:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.726+0000 7fae85c0e980 20 init_complete bucket index max shards: 1 2024-03-09T15:09:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.726+0000 7fae85c0e980 20 Filter name: none 2024-03-09T15:09:53.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.726+0000 7facb27fc640 20 reqs_thread_entry: start 2024-03-09T15:09:53.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.726+0000 7fae85c0e980 10 cache get: name=default.rgw.meta+users.uid+root2.client.0 : miss 2024-03-09T15:09:53.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.726+0000 7fae85c0e980 20 rados->read ofs=0 len=0 2024-03-09T15:09:53.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.730+0000 7fae85c0e980 20 rados_obj.operate() r=0 bl.length=494 2024-03-09T15:09:53.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.730+0000 7fae85c0e980 10 cache put: name=default.rgw.meta+users.uid+root2.client.0 info.flags=0x13 2024-03-09T15:09:53.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.730+0000 7fae85c0e980 10 adding default.rgw.meta+users.uid+root2.client.0 to cache LRU end 2024-03-09T15:09:53.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.730+0000 7fae85c0e980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T15:09:53.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.734+0000 7fae85c0e980 10 cache put: name=default.rgw.meta+users.uid+root2.client.0 info.flags=0x17 2024-03-09T15:09:53.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.734+0000 7fae85c0e980 10 moving default.rgw.meta+users.uid+root2.client.0 to cache LRU end 2024-03-09T15:09:53.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.734+0000 7fae85c0e980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:root2.client.0, ofs0, ns] 2024-03-09T15:09:53.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.734+0000 7fad66ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464382 cookie 94438372116320 notifier 7361 bl.length()=695 2024-03-09T15:09:53.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.734+0000 7fad66ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+root2.client.0 info.flags=0x17 2024-03-09T15:09:53.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.734+0000 7fad66ffd640 10 rgw watcher librados: moving default.rgw.meta+users.uid+root2.client.0 to cache LRU end 2024-03-09T15:09:53.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.738+0000 7fad39fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:09:53.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.746+0000 7fae85c0e980 20 remove_watcher() i=0 2024-03-09T15:09:53.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.746+0000 7fae85c0e980 2 removed watcher, disabling cache 2024-03-09T15:09:53.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.746+0000 7fae85c0e980 20 remove_watcher() i=1 2024-03-09T15:09:53.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.750+0000 7fae85c0e980 20 remove_watcher() i=2 2024-03-09T15:09:53.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.750+0000 7fae85c0e980 20 remove_watcher() i=3 2024-03-09T15:09:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.754+0000 7fae85c0e980 20 remove_watcher() i=4 2024-03-09T15:09:53.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.754+0000 7fae85c0e980 20 remove_watcher() i=5 2024-03-09T15:09:53.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.758+0000 7fae85c0e980 20 remove_watcher() i=6 2024-03-09T15:09:53.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:09:53.758+0000 7fae85c0e980 20 remove_watcher() i=7 2024-03-09T15:09:53.774 INFO:tasks.s3tests:Configuring s3-tests... 2024-03-09T15:09:53.775 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T15:09:53.775 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2024-03-09T15:09:53.780 INFO:tasks.s3tests:Configuring boto... 2024-03-09T15:09:53.780 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-09T15:09:53.780 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/boto-client.0.cfg 2024-03-09T15:09:53.827 DEBUG:teuthology.orchestra.run.smithi001:s3 tests against rgw> source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3' 2024-03-09T15:09:53.992 INFO:teuthology.orchestra.run.smithi001.stdout:GLOB sdist-make: /home/ubuntu/cephtest/s3-tests-client.0/setup.py 2024-03-09T15:09:54.191 INFO:teuthology.orchestra.run.smithi001.stdout:py create: /home/ubuntu/cephtest/s3-tests-client.0/.tox/py 2024-03-09T15:09:54.466 INFO:teuthology.orchestra.run.smithi001.stdout:py installdeps: -rrequirements.txt 2024-03-09T15:11:35.077 INFO:teuthology.orchestra.run.smithi001.stdout:py inst: /home/ubuntu/cephtest/s3-tests-client.0/.tox/.tmp/package/1/s3tests-0.0.1.zip 2024-03-09T15:11:37.072 INFO:teuthology.orchestra.run.smithi001.stdout:py installed: boto==2.49.0,boto3==1.24.96,botocore==1.27.96,cachetools==5.3.3,certifi==2024.2.2,chardet==5.2.0,charset-normalizer==3.3.2,colorama==0.4.6,distlib==0.3.8,exceptiongroup==1.2.0,filelock==3.13.1,gevent==24.2.1,greenlet==3.0.3,httplib2==0.22.0,idna==3.6,iniconfig==2.0.0,isodate==0.6.1,jmespath==1.0.1,lxml==5.1.0,munch==4.0.0,packaging==23.2,platformdirs==4.2.0,pluggy==1.4.0,pyparsing==3.1.2,pyproject-api==1.6.1,pytest==8.1.1,python-dateutil==2.9.0.post0,pytz==2024.1,PyYAML==6.0.1,requests==2.31.0,s3tests @ file:///home/ubuntu/cephtest/s3-tests-client.0/.tox/.tmp/package/1/s3tests-0.0.1.zip#sha256=d83d80d906e167a6adb7980acbac098dd734bc23783801d99647486444dcb482,s3transfer==0.6.2,six==1.16.0,tomli==2.0.1,tox==4.14.1,urllib3==1.26.18,virtualenv==20.25.1,zope.event==5.0,zope.interface==6.2 2024-03-09T15:11:37.072 INFO:teuthology.orchestra.run.smithi001.stdout:py run-test-pre: PYTHONHASHSEED='872521737' 2024-03-09T15:11:37.072 INFO:teuthology.orchestra.run.smithi001.stdout:py run-test: commands[0] | pytest -v -m 'not fails_on_rgw and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3' 2024-03-09T15:11:37.269 INFO:teuthology.orchestra.run.smithi001.stdout:============================= test session starts ============================== 2024-03-09T15:11:37.269 INFO:teuthology.orchestra.run.smithi001.stdout:platform linux -- Python 3.10.6, pytest-8.1.1, pluggy-1.4.0 -- /home/ubuntu/cephtest/s3-tests-client.0/.tox/py/bin/python 2024-03-09T15:11:37.269 INFO:teuthology.orchestra.run.smithi001.stdout:cachedir: .tox/py/.pytest_cache 2024-03-09T15:11:37.270 INFO:teuthology.orchestra.run.smithi001.stdout:rootdir: /home/ubuntu/cephtest/s3-tests-client.0 2024-03-09T15:11:37.270 INFO:teuthology.orchestra.run.smithi001.stdout:configfile: pytest.ini 2024-03-09T15:11:39.005 INFO:teuthology.orchestra.run.smithi001.stdout:collecting ... collected 850 items / 124 deselected / 726 selected 2024-03-09T15:11:39.006 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:11:40.265 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_contentlength_none FAILED [ 0%] 2024-03-09T15:11:40.931 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_authorization_empty FAILED [ 0%] 2024-03-09T15:11:41.577 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_date_and_amz_date PASSED [ 0%] 2024-03-09T15:11:42.234 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_amz_date_and_no_date PASSED [ 0%] 2024-03-09T15:11:42.910 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_authorization_none FAILED [ 0%] 2024-03-09T15:11:43.394 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_contentlength_none PASSED [ 0%] 2024-03-09T15:11:44.239 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_acl_create_contentlength_none PASSED [ 0%] 2024-03-09T15:11:44.769 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_contentlength_none PASSED [ 1%] 2024-03-09T15:11:45.289 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_authorization_empty FAILED [ 1%] 2024-03-09T15:11:45.811 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_authorization_none FAILED [ 1%] 2024-03-09T15:11:46.487 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_contentlength_mismatch_below_aws2 FAILED [ 1%] 2024-03-09T15:11:47.162 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_authorization_incorrect_aws2 FAILED [ 1%] 2024-03-09T15:11:47.834 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_authorization_invalid_aws2 FAILED [ 1%] 2024-03-09T15:11:48.496 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_none_aws2 FAILED [ 1%] 2024-03-09T15:11:49.009 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_authorization_invalid_aws2 FAILED [ 2%] 2024-03-09T15:11:49.543 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_date_none_aws2 FAILED [ 2%] 2024-03-09T15:11:49.680 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_md5_invalid_garbage_aws4 SKIPPED [ 2%] 2024-03-09T15:11:49.839 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_contentlength_mismatch_below_aws4 SKIPPED [ 2%] 2024-03-09T15:11:49.974 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_authorization_incorrect_aws4 SKIPPED [ 2%] 2024-03-09T15:11:50.113 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_authorization_invalid_aws4 SKIPPED [ 2%] 2024-03-09T15:11:50.238 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_ua_empty_aws4 SKIPPED [ 2%] 2024-03-09T15:11:50.380 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_ua_none_aws4 SKIPPED [ 3%] 2024-03-09T15:11:50.518 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_invalid_aws4 SKIPPED [ 3%] 2024-03-09T15:11:50.656 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_amz_date_invalid_aws4 SKIPPED [ 3%] 2024-03-09T15:11:50.794 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_empty_aws4 SKIPPED [ 3%] 2024-03-09T15:11:50.931 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_amz_date_empty_aws4 SKIPPED [ 3%] 2024-03-09T15:11:51.068 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_none_aws4 SKIPPED [ 3%] 2024-03-09T15:11:51.204 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_amz_date_none_aws4 SKIPPED [ 3%] 2024-03-09T15:11:51.343 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_before_today_aws4 SKIPPED [ 3%] 2024-03-09T15:11:51.480 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_amz_date_before_today_aws4 SKIPPED [ 4%] 2024-03-09T15:11:51.619 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_after_today_aws4 SKIPPED [ 4%] 2024-03-09T15:11:51.752 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_amz_date_after_today_aws4 SKIPPED [ 4%] 2024-03-09T15:11:51.888 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_before_epoch_aws4 SKIPPED [ 4%] 2024-03-09T15:11:52.023 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_amz_date_before_epoch_aws4 SKIPPED [ 4%] 2024-03-09T15:11:52.163 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_date_after_end_aws4 SKIPPED [ 4%] 2024-03-09T15:11:52.302 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_bad_amz_date_after_end_aws4 SKIPPED [ 4%] 2024-03-09T15:11:52.438 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_missing_signed_custom_header_aws4 SKIPPED [ 5%] 2024-03-09T15:11:52.574 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_object_create_missing_signed_header_aws4 SKIPPED [ 5%] 2024-03-09T15:11:52.711 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_authorization_invalid_aws4 SKIPPED [ 5%] 2024-03-09T15:11:52.848 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_ua_empty_aws4 SKIPPED [ 5%] 2024-03-09T15:11:52.982 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_ua_none_aws4 SKIPPED [ 5%] 2024-03-09T15:11:53.125 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_date_invalid_aws4 SKIPPED [ 5%] 2024-03-09T15:11:53.260 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_invalid_aws4 SKIPPED [ 5%] 2024-03-09T15:11:53.396 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_date_empty_aws4 SKIPPED [ 6%] 2024-03-09T15:11:53.532 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_empty_aws4 SKIPPED [ 6%] 2024-03-09T15:11:53.669 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_date_none_aws4 SKIPPED [ 6%] 2024-03-09T15:11:53.806 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_none_aws4 SKIPPED [ 6%] 2024-03-09T15:11:53.942 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_date_before_today_aws4 SKIPPED [ 6%] 2024-03-09T15:11:54.080 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_before_today_aws4 SKIPPED [ 6%] 2024-03-09T15:11:54.215 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_date_after_today_aws4 SKIPPED [ 6%] 2024-03-09T15:11:54.353 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_after_today_aws4 SKIPPED [ 7%] 2024-03-09T15:11:54.508 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_date_before_epoch_aws4 SKIPPED [ 7%] 2024-03-09T15:11:54.645 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_before_epoch_aws4 SKIPPED [ 7%] 2024-03-09T15:11:56.174 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_versioning_obj_read_not_exist_null PASSED [ 7%] 2024-03-09T15:12:15.047 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_object_storage_class PASSED [ 7%] 2024-03-09T15:12:35.129 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_object_storage_class_multipart PASSED [ 7%] 2024-03-09T15:13:03.748 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_object_modify_storage_class PASSED [ 7%] 2024-03-09T15:13:40.519 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_object_modify_storage_class_multipart PASSED [ 7%] 2024-03-09T15:13:51.891 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_object_storage_class_copy PASSED [ 8%] 2024-03-09T15:14:04.025 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_object_storage_class_copy_multipart PASSED [ 8%] 2024-03-09T15:14:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_atomic_dual_conditional_write_1mb PASSED [ 8%] 2024-03-09T15:14:08.505 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_atomic_write_bucket_gone PASSED [ 8%] 2024-03-09T15:14:12.469 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_multipart_invalid_chunks_1 PASSED [ 8%] 2024-03-09T15:14:17.800 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_encryption_sse_c_multipart_invalid_chunks_2 PASSED [ 8%] 2024-03-09T15:14:20.248 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_bucket_policy_set_condition_operator_end_with_IfExists PASSED [ 8%] 2024-03-09T15:14:21.588 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py::test_bucket_policy_put_obj_request_obj_tag PASSED [ 9%] 2024-03-09T15:14:22.579 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_nonexistant_bucket_rgw SKIPPED [ 9%] 2024-03-09T15:14:22.581 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_public_bucket_list_public_index SKIPPED [ 9%] 2024-03-09T15:14:22.588 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_private_bucket_list_public_index SKIPPED [ 9%] 2024-03-09T15:14:22.591 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_private_bucket_list_empty SKIPPED [ 9%] 2024-03-09T15:14:22.594 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_public_bucket_list_empty SKIPPED [ 9%] 2024-03-09T15:14:22.596 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_public_bucket_list_private_index SKIPPED [ 9%] 2024-03-09T15:14:22.599 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_private_bucket_list_private_index SKIPPED [ 10%] 2024-03-09T15:14:22.602 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_private_bucket_list_empty_missingerrordoc SKIPPED [ 10%] 2024-03-09T15:14:22.605 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_public_bucket_list_empty_missingerrordoc SKIPPED [ 10%] 2024-03-09T15:14:22.608 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_public_bucket_list_private_index_missingerrordoc SKIPPED [ 10%] 2024-03-09T15:14:22.611 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_private_bucket_list_private_index_missingerrordoc SKIPPED [ 10%] 2024-03-09T15:14:22.613 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_private_bucket_list_empty_blockederrordoc SKIPPED [ 10%] 2024-03-09T15:14:22.616 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_public_bucket_list_pubilc_errordoc SKIPPED [ 10%] 2024-03-09T15:14:22.618 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_public_bucket_list_empty_blockederrordoc SKIPPED [ 11%] 2024-03-09T15:14:22.620 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_public_bucket_list_private_index_blockederrordoc SKIPPED [ 11%] 2024-03-09T15:14:22.623 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_private_bucket_list_private_index_blockederrordoc SKIPPED [ 11%] 2024-03-09T15:14:22.625 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_private_bucket_list_empty_gooderrordoc SKIPPED [ 11%] 2024-03-09T15:14:22.627 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_public_bucket_list_empty_gooderrordoc SKIPPED [ 11%] 2024-03-09T15:14:22.630 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_public_bucket_list_private_index_gooderrordoc SKIPPED [ 11%] 2024-03-09T15:14:22.633 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_private_bucket_list_private_index_gooderrordoc SKIPPED [ 11%] 2024-03-09T15:14:22.635 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_bucket_private_redirectall_base SKIPPED [ 11%] 2024-03-09T15:14:22.638 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_bucket_private_redirectall_path SKIPPED [ 12%] 2024-03-09T15:14:22.640 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_bucket_private_redirectall_path_upgrade SKIPPED [ 12%] 2024-03-09T15:14:22.643 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_xredirect_nonwebsite SKIPPED [ 12%] 2024-03-09T15:14:22.645 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_xredirect_public_relative SKIPPED [ 12%] 2024-03-09T15:14:22.647 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_xredirect_public_abs SKIPPED [ 12%] 2024-03-09T15:14:22.650 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_xredirect_private_relative SKIPPED [ 12%] 2024-03-09T15:14:22.652 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_website_xredirect_private_abs SKIPPED [ 12%] 2024-03-09T15:14:22.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py::test_routing_generator XFAIL [ 13%] 2024-03-09T15:14:22.762 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_utils.py::test_generate PASSED [ 13%] 2024-03-09T15:14:25.634 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_md5_invalid_short PASSED [ 13%] 2024-03-09T15:14:38.339 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_md5_bad PASSED [ 13%] 2024-03-09T15:14:40.667 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_md5_empty PASSED [ 13%] 2024-03-09T15:14:43.211 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_md5_none PASSED [ 13%] 2024-03-09T15:14:45.775 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_expect_mismatch PASSED [ 13%] 2024-03-09T15:14:48.319 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_expect_empty PASSED [ 14%] 2024-03-09T15:14:50.903 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_expect_none PASSED [ 14%] 2024-03-09T15:14:53.199 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_contentlength_negative PASSED [ 14%] 2024-03-09T15:14:55.891 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_contenttype_invalid PASSED [ 14%] 2024-03-09T15:14:58.263 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_contenttype_empty PASSED [ 14%] 2024-03-09T15:15:00.699 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_contenttype_none PASSED [ 14%] 2024-03-09T15:15:02.895 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_bucket_put_bad_canned_acl PASSED [ 14%] 2024-03-09T15:15:04.983 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_bucket_create_bad_expect_mismatch PASSED [ 15%] 2024-03-09T15:15:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_bucket_create_bad_expect_empty PASSED [ 15%] 2024-03-09T15:15:08.867 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_bucket_create_bad_contentlength_negative PASSED [ 15%] 2024-03-09T15:15:11.119 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_md5_invalid_garbage_aws2 PASSED [ 15%] 2024-03-09T15:15:13.631 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_ua_empty_aws2 PASSED [ 15%] 2024-03-09T15:15:16.115 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_ua_none_aws2 PASSED [ 15%] 2024-03-09T15:15:18.371 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_date_invalid_aws2 PASSED [ 15%] 2024-03-09T15:15:20.587 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_date_empty_aws2 PASSED [ 15%] 2024-03-09T15:15:22.918 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_date_before_today_aws2 PASSED [ 16%] 2024-03-09T15:15:25.171 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_date_before_epoch_aws2 PASSED [ 16%] 2024-03-09T15:15:27.391 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_object_create_bad_date_after_end_aws2 PASSED [ 16%] 2024-03-09T15:15:29.427 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_bucket_create_bad_ua_empty_aws2 PASSED [ 16%] 2024-03-09T15:15:31.415 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_bucket_create_bad_ua_none_aws2 PASSED [ 16%] 2024-03-09T15:15:33.135 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_bucket_create_bad_date_invalid_aws2 PASSED [ 16%] 2024-03-09T15:15:34.851 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_bucket_create_bad_date_empty_aws2 PASSED [ 16%] 2024-03-09T15:15:36.528 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_bucket_create_bad_date_before_today_aws2 PASSED [ 17%] 2024-03-09T15:15:38.246 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_bucket_create_bad_date_after_today_aws2 PASSED [ 17%] 2024-03-09T15:15:39.919 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py::test_bucket_create_bad_date_before_epoch_aws2 PASSED [ 17%] 2024-03-09T15:15:41.943 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_put_user_policy PASSED [ 17%] 2024-03-09T15:15:43.679 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_put_user_policy_invalid_user PASSED [ 17%] 2024-03-09T15:15:45.579 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_put_user_policy_parameter_limit PASSED [ 17%] 2024-03-09T15:15:47.734 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_put_existing_user_policy PASSED [ 17%] 2024-03-09T15:15:49.731 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_list_user_policy PASSED [ 18%] 2024-03-09T15:15:51.451 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_list_user_policy_invalid_user PASSED [ 18%] 2024-03-09T15:15:53.515 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_get_user_policy PASSED [ 18%] 2024-03-09T15:15:55.563 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_get_user_policy_invalid_user PASSED [ 18%] 2024-03-09T15:15:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_get_user_policy_from_multiple_policies PASSED [ 18%] 2024-03-09T15:15:59.967 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_delete_user_policy PASSED [ 18%] 2024-03-09T15:16:02.031 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_delete_user_policy_invalid_user PASSED [ 18%] 2024-03-09T15:16:04.071 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_delete_user_policy_invalid_policy_name PASSED [ 19%] 2024-03-09T15:16:06.763 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_delete_user_policy_from_multiple_policies PASSED [ 19%] 2024-03-09T15:16:09.587 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_allow_bucket_actions_in_user_policy PASSED [ 19%] 2024-03-09T15:16:12.079 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_deny_bucket_actions_in_user_policy PASSED [ 19%] 2024-03-09T15:16:14.915 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_allow_object_actions_in_user_policy PASSED [ 19%] 2024-03-09T15:16:17.867 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_deny_object_actions_in_user_policy PASSED [ 19%] 2024-03-09T15:16:23.375 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_allow_multipart_actions_in_user_policy PASSED [ 19%] 2024-03-09T15:16:28.807 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_deny_multipart_actions_in_user_policy PASSED [ 19%] 2024-03-09T15:16:31.975 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_allow_tagging_actions_in_user_policy PASSED [ 20%] 2024-03-09T15:16:35.067 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_deny_tagging_actions_in_user_policy PASSED [ 20%] 2024-03-09T15:16:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_verify_conflicting_user_policy_statements PASSED [ 20%] 2024-03-09T15:16:40.395 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_verify_conflicting_user_policies PASSED [ 20%] 2024-03-09T15:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_verify_allow_iam_actions PASSED [ 20%] 2024-03-09T15:16:46.523 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_create PASSED [ 20%] 2024-03-09T15:16:49.315 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_case_insensitive_name PASSED [ 20%] 2024-03-09T15:16:52.447 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_delete PASSED [ 21%] 2024-03-09T15:16:57.584 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_list PASSED [ 21%] 2024-03-09T15:17:03.996 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_list_path_prefix PASSED [ 21%] 2024-03-09T15:17:08.119 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_update_name PASSED [ 21%] 2024-03-09T15:17:12.008 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_update_path PASSED [ 21%] 2024-03-09T15:17:15.372 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_access_key_create PASSED [ 21%] 2024-03-09T15:17:17.871 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_current_user_access_key_create PASSED [ 21%] 2024-03-09T15:17:21.568 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_access_key_update PASSED [ 22%] 2024-03-09T15:17:24.411 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_current_user_access_key_update PASSED [ 22%] 2024-03-09T15:17:27.860 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_access_key_delete PASSED [ 22%] 2024-03-09T15:17:30.675 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_current_user_access_key_delete PASSED [ 22%] 2024-03-09T15:17:34.868 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_access_key_list PASSED [ 22%] 2024-03-09T15:17:38.811 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_bucket_policy_allow PASSED [ 22%] 2024-03-09T15:17:43.139 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_policy PASSED [ 22%] 2024-03-09T15:17:47.632 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_policy_managed PASSED [ 23%] 2024-03-09T15:17:51.212 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_user_policy_allow PASSED [ 23%] 2024-03-09T15:17:54.551 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_group_create PASSED [ 23%] 2024-03-09T15:17:57.344 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_group_case_insensitive_name PASSED [ 23%] 2024-03-09T15:18:02.372 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_group_list PASSED [ 23%] 2024-03-09T15:18:06.411 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_group_update PASSED [ 23%] 2024-03-09T15:18:10.823 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_inline_group_policy PASSED [ 23%] 2024-03-09T15:18:15.091 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_managed_group_policy PASSED [ 23%] 2024-03-09T15:18:19.428 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_inline_group_policy_allow PASSED [ 24%] 2024-03-09T15:18:23.689 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_managed_group_policy_allow PASSED [ 24%] 2024-03-09T15:18:27.496 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_role_create PASSED [ 24%] 2024-03-09T15:18:30.404 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_role_case_insensitive_name PASSED [ 24%] 2024-03-09T15:18:33.575 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_role_delete PASSED [ 24%] 2024-03-09T15:18:38.532 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_role_list PASSED [ 24%] 2024-03-09T15:18:44.488 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_role_list_path_prefix PASSED [ 24%] 2024-03-09T15:18:47.671 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_role_update PASSED [ 25%] 2024-03-09T15:18:51.939 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_role_policy PASSED [ 25%] 2024-03-09T15:18:56.079 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_role_policy_managed PASSED [ 25%] 2024-03-09T15:19:00.592 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_role_policy_allow PASSED [ 25%] 2024-03-09T15:19:05.340 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_same_account_role_policy_allow PASSED [ 25%] 2024-03-09T15:19:10.304 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_role_policy_allow PASSED [ 25%] 2024-03-09T15:19:15.448 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_role_policy_allow_create_bucket PASSED [ 25%] 2024-03-09T15:19:20.204 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_role_policy_allow_get_role PASSED [ 26%] 2024-03-09T15:19:23.423 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_account_oidc_provider PASSED [ 26%] 2024-03-09T15:19:27.852 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_bucket_user_policy_allow_user_arn PASSED [ 26%] 2024-03-09T15:19:32.296 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_user_bucket_policy_allow_user_arn PASSED [ 26%] 2024-03-09T15:19:36.812 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_user_bucket_policy_allow_account_arn PASSED [ 26%] 2024-03-09T15:19:41.224 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_bucket_user_policy_allow_account_arn PASSED [ 26%] 2024-03-09T15:19:45.656 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_user_bucket_policy_allow_account_id PASSED [ 26%] 2024-03-09T15:19:50.128 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_bucket_user_policy_allow_account_id PASSED [ 26%] 2024-03-09T15:19:54.548 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_bucket_acl_user_policy_grant_user_id PASSED [ 27%] 2024-03-09T15:19:59.060 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_user_policy_bucket_acl_grant_user_id PASSED [ 27%] 2024-03-09T15:20:03.448 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_bucket_acl_user_policy_grant_canonical_id PASSED [ 27%] 2024-03-09T15:20:07.892 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_user_policy_bucket_acl_grant_canonical_id PASSED [ 27%] 2024-03-09T15:20:12.344 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_bucket_acl_user_policy_grant_account_email PASSED [ 27%] 2024-03-09T15:20:16.912 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_user_policy_bucket_acl_grant_account_email PASSED [ 27%] 2024-03-09T15:20:20.316 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_root_bucket_policy_allow_account_arn PASSED [ 27%] 2024-03-09T15:20:23.712 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_root_bucket_policy_allow_account_id PASSED [ 28%] 2024-03-09T15:20:27.024 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_root_bucket_acl_grant_canonical_id PASSED [ 28%] 2024-03-09T15:20:30.356 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py::test_cross_account_root_bucket_acl_grant_account_email PASSED [ 28%] 2024-03-09T15:20:32.595 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_empty PASSED [ 28%] 2024-03-09T15:20:35.427 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_distinct PASSED [ 28%] 2024-03-09T15:20:38.315 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_many PASSED [ 28%] 2024-03-09T15:20:41.199 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_many PASSED [ 28%] 2024-03-09T15:20:43.987 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_basic_key_count PASSED [ 29%] 2024-03-09T15:20:46.923 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_basic PASSED [ 29%] 2024-03-09T15:20:49.771 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_basic PASSED [ 29%] 2024-03-09T15:20:52.735 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_encoding_basic PASSED [ 29%] 2024-03-09T15:20:55.703 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_encoding_basic PASSED [ 29%] 2024-03-09T15:21:00.183 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_prefix PASSED [ 29%] 2024-03-09T15:21:04.579 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_prefix PASSED [ 29%] 2024-03-09T15:21:07.011 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_prefix_ends_with_delimiter PASSED [ 30%] 2024-03-09T15:21:09.512 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_prefix_ends_with_delimiter PASSED [ 30%] 2024-03-09T15:21:12.479 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_alt PASSED [ 30%] 2024-03-09T15:21:15.251 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_alt PASSED [ 30%] 2024-03-09T15:21:19.711 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_prefix_underscore PASSED [ 30%] 2024-03-09T15:21:24.199 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_prefix_underscore PASSED [ 30%] 2024-03-09T15:21:27.111 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_percentage PASSED [ 30%] 2024-03-09T15:21:29.968 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_percentage PASSED [ 30%] 2024-03-09T15:21:32.939 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_whitespace PASSED [ 31%] 2024-03-09T15:21:35.815 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_whitespace PASSED [ 31%] 2024-03-09T15:21:38.715 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_dot PASSED [ 31%] 2024-03-09T15:21:41.615 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_dot PASSED [ 31%] 2024-03-09T15:21:44.419 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_unreadable PASSED [ 31%] 2024-03-09T15:21:47.319 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_unreadable PASSED [ 31%] 2024-03-09T15:21:50.096 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_empty PASSED [ 31%] 2024-03-09T15:21:53.003 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_empty PASSED [ 32%] 2024-03-09T15:21:55.857 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_none PASSED [ 32%] 2024-03-09T15:21:58.735 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_none PASSED [ 32%] 2024-03-09T15:22:01.543 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_fetchowner_notempty PASSED [ 32%] 2024-03-09T15:22:04.259 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_fetchowner_defaultempty PASSED [ 32%] 2024-03-09T15:22:07.623 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_fetchowner_empty PASSED [ 32%] 2024-03-09T15:22:10.487 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_not_exist PASSED [ 32%] 2024-03-09T15:22:13.355 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_not_exist PASSED [ 33%] 2024-03-09T15:24:32.496 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_delimiter_not_skip_special PASSED [ 33%] 2024-03-09T15:24:35.335 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_prefix_basic PASSED [ 33%] 2024-03-09T15:24:38.119 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_basic PASSED [ 33%] 2024-03-09T15:24:40.904 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_prefix_alt PASSED [ 33%] 2024-03-09T15:24:43.675 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_alt PASSED [ 33%] 2024-03-09T15:24:46.399 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_prefix_empty PASSED [ 33%] 2024-03-09T15:24:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_empty PASSED [ 34%] 2024-03-09T15:24:51.903 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_prefix_none PASSED [ 34%] 2024-03-09T15:24:54.683 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_none PASSED [ 34%] 2024-03-09T15:24:57.427 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_prefix_not_exist PASSED [ 34%] 2024-03-09T15:25:00.227 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_not_exist PASSED [ 34%] 2024-03-09T15:25:02.907 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_prefix_unreadable PASSED [ 34%] 2024-03-09T15:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_unreadable PASSED [ 34%] 2024-03-09T15:25:08.591 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_prefix_delimiter_basic PASSED [ 34%] 2024-03-09T15:25:11.539 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_basic PASSED [ 35%] 2024-03-09T15:25:14.435 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_prefix_delimiter_alt PASSED [ 35%] 2024-03-09T15:25:17.308 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_alt PASSED [ 35%] 2024-03-09T15:25:20.275 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_prefix_delimiter_prefix_not_exist PASSED [ 35%] 2024-03-09T15:25:23.108 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_prefix_not_exist PASSED [ 35%] 2024-03-09T15:25:26.107 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_prefix_delimiter_delimiter_not_exist PASSED [ 35%] 2024-03-09T15:25:29.067 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_delimiter_not_exist PASSED [ 35%] 2024-03-09T15:25:31.887 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist PASSED [ 36%] 2024-03-09T15:25:34.683 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist PASSED [ 36%] 2024-03-09T15:25:37.699 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_maxkeys_one PASSED [ 36%] 2024-03-09T15:25:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_maxkeys_one PASSED [ 36%] 2024-03-09T15:25:43.563 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_maxkeys_zero PASSED [ 36%] 2024-03-09T15:25:46.476 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_maxkeys_zero PASSED [ 36%] 2024-03-09T15:25:49.423 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_maxkeys_none PASSED [ 36%] 2024-03-09T15:25:52.323 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_maxkeys_none PASSED [ 37%] 2024-03-09T15:25:54.135 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_account_usage PASSED [ 37%] 2024-03-09T15:25:56.543 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_head_bucket_usage PASSED [ 37%] 2024-03-09T15:26:02.855 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_unordered PASSED [ 37%] 2024-03-09T15:26:09.235 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_unordered PASSED [ 37%] 2024-03-09T15:26:12.035 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_maxkeys_invalid PASSED [ 37%] 2024-03-09T15:26:15.023 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_marker_none PASSED [ 37%] 2024-03-09T15:26:17.795 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_marker_empty PASSED [ 38%] 2024-03-09T15:26:20.859 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_continuationtoken_empty PASSED [ 38%] 2024-03-09T15:26:23.947 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_continuationtoken PASSED [ 38%] 2024-03-09T15:26:26.996 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_both_continuationtoken_startafter PASSED [ 38%] 2024-03-09T15:26:29.943 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_marker_unreadable PASSED [ 38%] 2024-03-09T15:26:32.791 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_startafter_unreadable PASSED [ 38%] 2024-03-09T15:26:35.660 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_marker_not_in_list PASSED [ 38%] 2024-03-09T15:26:38.619 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_startafter_not_in_list PASSED [ 38%] 2024-03-09T15:26:41.595 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_marker_after_list PASSED [ 39%] 2024-03-09T15:26:44.456 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_startafter_after_list PASSED [ 39%] 2024-03-09T15:26:47.784 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_return_data PASSED [ 39%] 2024-03-09T15:26:52.047 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_return_data_versioning PASSED [ 39%] 2024-03-09T15:26:54.496 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_objects_anonymous PASSED [ 39%] 2024-03-09T15:26:56.920 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_objects_anonymous PASSED [ 39%] 2024-03-09T15:26:59.120 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_objects_anonymous_fail PASSED [ 39%] 2024-03-09T15:27:01.432 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_listv2_objects_anonymous_fail PASSED [ 40%] 2024-03-09T15:27:03.124 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_notexist PASSED [ 40%] 2024-03-09T15:27:04.981 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucketv2_notexist PASSED [ 40%] 2024-03-09T15:27:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_delete_notexist PASSED [ 40%] 2024-03-09T15:27:09.151 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_delete_nonempty PASSED [ 40%] 2024-03-09T15:27:19.116 INFO:tasks.rgw.client.0.smithi001.stdout:2024-03-09T15:27:19.111+0000 90cf4640 -1 req 4032889334048936167 0.115998261s s3:delete_bucket ERROR: could not remove bucket test-client.0-8p45c4thhbi3i01-156 2024-03-09T15:27:19.924 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_concurrent_set_canned_acl PASSED [ 40%] 2024-03-09T15:27:20.377 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_write_to_nonexist_bucket ERROR [ 40%] 2024-03-09T15:27:20.797 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_write_with_chunked_transfer_encoding ERROR [ 41%] 2024-03-09T15:27:21.176 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_delete ERROR [ 41%] 2024-03-09T15:27:21.512 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_read_not_exist ERROR [ 41%] 2024-03-09T15:27:21.887 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_requestid_matches_header_on_error ERROR [ 41%] 2024-03-09T15:27:22.219 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_concurrent_multi_object_delete ERROR [ 41%] 2024-03-09T15:27:22.591 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multi_object_delete ERROR [ 41%] 2024-03-09T15:27:22.957 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multi_objectv2_delete ERROR [ 41%] 2024-03-09T15:27:23.282 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multi_object_delete_key_limit ERROR [ 42%] 2024-03-09T15:27:23.656 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multi_objectv2_delete_key_limit ERROR [ 42%] 2024-03-09T15:27:23.983 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_head_zero_bytes ERROR [ 42%] 2024-03-09T15:27:24.376 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_write_check_etag ERROR [ 42%] 2024-03-09T15:27:24.728 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_write_cache_control ERROR [ 42%] 2024-03-09T15:27:25.116 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_write_expires ERROR [ 42%] 2024-03-09T15:27:25.427 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_write_read_update_read_delete ERROR [ 42%] 2024-03-09T15:27:25.848 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_set_get_metadata_none_to_good ERROR [ 42%] 2024-03-09T15:27:26.213 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_set_get_metadata_none_to_empty ERROR [ 43%] 2024-03-09T15:27:26.607 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_set_get_metadata_overwrite_to_empty ERROR [ 43%] 2024-03-09T15:27:26.980 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_metadata_replaced_on_put ERROR [ 43%] 2024-03-09T15:27:27.351 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_write_file ERROR [ 43%] 2024-03-09T15:27:27.783 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_anonymous_request ERROR [ 43%] 2024-03-09T15:27:28.152 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_authenticated_request ERROR [ 43%] 2024-03-09T15:27:28.518 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_authenticated_no_content_type ERROR [ 43%] 2024-03-09T15:27:28.883 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_authenticated_request_bad_access_key ERROR [ 44%] 2024-03-09T15:27:29.260 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_set_success_code ERROR [ 44%] 2024-03-09T15:27:29.680 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_set_invalid_success_code ERROR [ 44%] 2024-03-09T15:27:29.967 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_upload_larger_than_chunk ERROR [ 44%] 2024-03-09T15:27:30.399 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_set_key_from_filename ERROR [ 44%] 2024-03-09T15:27:30.776 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_ignored_header ERROR [ 44%] 2024-03-09T15:27:31.111 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_case_insensitive_condition_fields ERROR [ 44%] 2024-03-09T15:27:31.540 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_escaped_field_values ERROR [ 45%] 2024-03-09T15:27:31.892 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_success_redirect_action ERROR [ 45%] 2024-03-09T15:27:32.252 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_invalid_signature ERROR [ 45%] 2024-03-09T15:27:32.587 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_invalid_access_key ERROR [ 45%] 2024-03-09T15:27:33.021 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_invalid_date_format ERROR [ 45%] 2024-03-09T15:27:33.330 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_no_key_specified ERROR [ 45%] 2024-03-09T15:27:33.700 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_missing_signature ERROR [ 45%] 2024-03-09T15:27:34.076 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_missing_policy_condition ERROR [ 46%] 2024-03-09T15:27:34.449 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_user_specified_header ERROR [ 46%] 2024-03-09T15:27:34.881 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_request_missing_policy_specified_field ERROR [ 46%] 2024-03-09T15:27:35.241 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_condition_is_case_sensitive ERROR [ 46%] 2024-03-09T15:27:35.614 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_expires_is_case_sensitive ERROR [ 46%] 2024-03-09T15:27:35.973 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_expired_policy ERROR [ 46%] 2024-03-09T15:27:36.290 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_wrong_bucket ERROR [ 46%] 2024-03-09T15:27:36.660 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_invalid_request_field_value ERROR [ 46%] 2024-03-09T15:27:37.005 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_missing_expires_condition ERROR [ 47%] 2024-03-09T15:27:37.314 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_missing_conditions_list ERROR [ 47%] 2024-03-09T15:27:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_upload_size_limit_exceeded ERROR [ 47%] 2024-03-09T15:27:38.004 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_missing_content_length_argument ERROR [ 47%] 2024-03-09T15:27:38.440 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_invalid_content_length_argument ERROR [ 47%] 2024-03-09T15:27:38.818 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_upload_size_below_minimum ERROR [ 47%] 2024-03-09T15:27:39.126 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_upload_size_rgw_chunk_size_bug ERROR [ 47%] 2024-03-09T15:27:39.432 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_empty_conditions ERROR [ 48%] 2024-03-09T15:27:39.830 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_object_ifmatch_good ERROR [ 48%] 2024-03-09T15:27:40.200 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_object_ifmatch_failed ERROR [ 48%] 2024-03-09T15:27:40.584 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_object_ifnonematch_good ERROR [ 48%] 2024-03-09T15:27:40.957 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_object_ifnonematch_failed ERROR [ 48%] 2024-03-09T15:27:41.319 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_object_ifmodifiedsince_good ERROR [ 48%] 2024-03-09T15:27:41.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_object_ifmodifiedsince_failed ERROR [ 48%] 2024-03-09T15:27:42.060 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_object_ifunmodifiedsince_good ERROR [ 49%] 2024-03-09T15:27:42.424 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_object_ifunmodifiedsince_failed ERROR [ 49%] 2024-03-09T15:27:42.804 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_object_ifmatch_good ERROR [ 49%] 2024-03-09T15:27:43.139 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_object_ifmatch_failed ERROR [ 49%] 2024-03-09T15:27:43.557 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_object_ifmatch_overwrite_existed_good ERROR [ 49%] 2024-03-09T15:27:43.864 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_object_ifmatch_nonexisted_failed ERROR [ 49%] 2024-03-09T15:27:44.232 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_object_ifnonmatch_good ERROR [ 49%] 2024-03-09T15:27:44.588 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_object_ifnonmatch_failed ERROR [ 50%] 2024-03-09T15:27:44.915 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_object_ifnonmatch_nonexisted_good ERROR [ 50%] 2024-03-09T15:27:45.300 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_object_ifnonmatch_overwrite_existed_failed ERROR [ 50%] 2024-03-09T15:27:45.654 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_get ERROR [ 50%] 2024-03-09T15:27:46.027 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_get_bucket_gone ERROR [ 50%] 2024-03-09T15:27:46.377 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_delete_key_bucket_gone ERROR [ 50%] 2024-03-09T15:27:46.814 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_get_object_gone ERROR [ 50%] 2024-03-09T15:27:47.189 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_head ERROR [ 50%] 2024-03-09T15:27:47.564 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_head_notexist ERROR [ 51%] 2024-03-09T15:27:47.932 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_head_extended ERROR [ 51%] 2024-03-09T15:27:48.305 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_get_bucket_acl ERROR [ 51%] 2024-03-09T15:27:48.746 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_get_object_acl ERROR [ 51%] 2024-03-09T15:27:49.066 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_put_acl_mtime ERROR [ 51%] 2024-03-09T15:27:49.444 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_authenticated ERROR [ 51%] 2024-03-09T15:27:49.824 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_response_headers ERROR [ 51%] 2024-03-09T15:27:50.215 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_authenticated_bucket_acl ERROR [ 52%] 2024-03-09T15:27:50.637 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_authenticated_object_acl ERROR [ 52%] 2024-03-09T15:27:51.013 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_authenticated_bucket_gone ERROR [ 52%] 2024-03-09T15:27:51.388 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_authenticated_object_gone ERROR [ 52%] 2024-03-09T15:27:51.761 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_get_x_amz_expires_not_expired ERROR [ 52%] 2024-03-09T15:27:52.140 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_get_x_amz_expires_not_expired_tenant ERROR [ 52%] 2024-03-09T15:27:52.526 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_get_x_amz_expires_out_range_zero ERROR [ 52%] 2024-03-09T15:27:52.877 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_get_x_amz_expires_out_max_range ERROR [ 53%] 2024-03-09T15:27:53.245 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_get_x_amz_expires_out_positive_range ERROR [ 53%] 2024-03-09T15:27:53.609 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_anon_put ERROR [ 53%] 2024-03-09T15:27:54.035 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_anon_put_write_access ERROR [ 53%] 2024-03-09T15:27:54.380 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_put_authenticated ERROR [ 53%] 2024-03-09T15:27:54.752 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_presigned_put_object_with_acl ERROR [ 53%] 2024-03-09T15:27:55.085 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_presigned_put_object_with_acl_tenant ERROR [ 53%] 2024-03-09T15:27:55.396 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_raw_put_authenticated_expired ERROR [ 53%] 2024-03-09T15:27:55.829 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_bad_starts_nonalpha ERROR [ 54%] 2024-03-09T15:27:56.197 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_bad_short_one ERROR [ 54%] 2024-03-09T15:27:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_bad_short_two ERROR [ 54%] 2024-03-09T15:27:56.884 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_long_60 ERROR [ 54%] 2024-03-09T15:27:57.225 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_long_61 ERROR [ 54%] 2024-03-09T15:27:57.602 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_long_62 ERROR [ 54%] 2024-03-09T15:27:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_long_63 ERROR [ 54%] 2024-03-09T15:27:58.317 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_long_name ERROR [ 55%] 2024-03-09T15:27:58.669 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_bad_ip ERROR [ 55%] 2024-03-09T15:27:59.023 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_dns_underscore ERROR [ 55%] 2024-03-09T15:27:59.452 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_dns_long ERROR [ 55%] 2024-03-09T15:27:59.785 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_dns_dash_at_end ERROR [ 55%] 2024-03-09T15:28:00.161 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_dns_dot_dot ERROR [ 55%] 2024-03-09T15:28:00.541 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_dns_dot_dash ERROR [ 55%] 2024-03-09T15:28:00.977 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_dns_dash_dot ERROR [ 56%] 2024-03-09T15:28:01.342 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_exists ERROR [ 56%] 2024-03-09T15:28:01.710 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_get_location ERROR [ 56%] 2024-03-09T15:28:02.016 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_exists_nonowner ERROR [ 56%] 2024-03-09T15:28:02.381 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_recreate_overwrite_acl ERROR [ 56%] 2024-03-09T15:28:02.818 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_recreate_new_acl ERROR [ 56%] 2024-03-09T15:28:03.142 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_default ERROR [ 56%] 2024-03-09T15:28:03.502 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_canned_during_create ERROR [ 57%] 2024-03-09T15:28:03.867 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_canned ERROR [ 57%] 2024-03-09T15:28:04.243 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_canned_publicreadwrite ERROR [ 57%] 2024-03-09T15:28:04.664 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_canned_authenticatedread ERROR [ 57%] 2024-03-09T15:28:05.020 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_default ERROR [ 57%] 2024-03-09T15:28:05.396 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_canned_during_create ERROR [ 57%] 2024-03-09T15:28:05.802 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_canned ERROR [ 57%] 2024-03-09T15:28:06.176 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_canned_publicreadwrite ERROR [ 57%] 2024-03-09T15:28:06.593 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_canned_authenticatedread ERROR [ 58%] 2024-03-09T15:28:06.965 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_canned_bucketownerread ERROR [ 58%] 2024-03-09T15:28:07.300 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_canned_bucketownerfullcontrol ERROR [ 58%] 2024-03-09T15:28:07.612 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_full_control_verify_owner ERROR [ 58%] 2024-03-09T15:28:08.051 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_full_control_verify_attributes ERROR [ 58%] 2024-03-09T15:28:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_canned_private_to_private ERROR [ 58%] 2024-03-09T15:28:08.727 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl ERROR [ 58%] 2024-03-09T15:28:09.105 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_write ERROR [ 59%] 2024-03-09T15:28:09.480 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_writeacp ERROR [ 59%] 2024-03-09T15:28:09.922 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_read ERROR [ 59%] 2024-03-09T15:28:10.288 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_acl_readacp ERROR [ 59%] 2024-03-09T15:28:10.658 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_userid_fullcontrol ERROR [ 59%] 2024-03-09T15:28:11.014 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_userid_read ERROR [ 59%] 2024-03-09T15:28:11.369 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_userid_readacp ERROR [ 59%] 2024-03-09T15:28:11.739 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_userid_write ERROR [ 60%] 2024-03-09T15:28:12.074 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_userid_writeacp ERROR [ 60%] 2024-03-09T15:28:12.448 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_nonexist_user ERROR [ 60%] 2024-03-09T15:28:12.756 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_header_acl_grants ERROR [ 60%] 2024-03-09T15:28:13.132 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_header_acl_grants ERROR [ 60%] 2024-03-09T15:28:13.527 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_email ERROR [ 60%] 2024-03-09T15:28:13.872 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_email_not_exist ERROR [ 60%] 2024-03-09T15:28:14.244 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_acl_revoke_all ERROR [ 61%] 2024-03-09T15:28:14.615 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_access_bucket_private_object_private ERROR [ 61%] 2024-03-09T15:28:15.007 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_access_bucket_private_objectv2_private ERROR [ 61%] 2024-03-09T15:28:15.380 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_access_bucket_private_object_publicread ERROR [ 61%] 2024-03-09T15:28:15.808 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_access_bucket_private_objectv2_publicread ERROR [ 61%] 2024-03-09T15:28:16.133 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_access_bucket_private_object_publicreadwrite ERROR [ 61%] 2024-03-09T15:28:16.489 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_access_bucket_private_objectv2_publicreadwrite ERROR [ 61%] 2024-03-09T15:28:16.929 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_access_bucket_publicread_object_private ERROR [ 61%] 2024-03-09T15:28:17.297 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_access_bucket_publicread_object_publicread ERROR [ 62%] 2024-03-09T15:28:17.617 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_access_bucket_publicread_object_publicreadwrite ERROR [ 62%] 2024-03-09T15:28:17.959 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_access_bucket_publicreadwrite_object_private ERROR [ 62%] 2024-03-09T15:28:18.328 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_access_bucket_publicreadwrite_object_publicread ERROR [ 62%] 2024-03-09T15:28:18.739 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_access_bucket_publicreadwrite_object_publicreadwrite ERROR [ 62%] 2024-03-09T15:28:19.112 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_buckets_create_then_list ERROR [ 62%] 2024-03-09T15:28:19.463 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_buckets_list_ctime ERROR [ 62%] 2024-03-09T15:28:19.788 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_list_buckets_anonymous ERROR [ 63%] 2024-03-09T15:28:20.155 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_list_buckets_invalid_auth ERROR [ 63%] 2024-03-09T15:28:20.591 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_list_buckets_bad_auth ERROR [ 63%] 2024-03-09T15:28:21.005 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_starts_alpha ERROR [ 63%] 2024-03-09T15:28:21.357 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_starts_digit ERROR [ 63%] 2024-03-09T15:28:21.730 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_contains_period ERROR [ 63%] 2024-03-09T15:28:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_contains_hyphen ERROR [ 63%] 2024-03-09T15:28:22.534 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_recreate_not_overriding ERROR [ 64%] 2024-03-09T15:28:22.904 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_create_special_key_names ERROR [ 64%] 2024-03-09T15:28:23.268 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_list_special_prefix ERROR [ 64%] 2024-03-09T15:28:23.643 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_zero_size ERROR [ 64%] 2024-03-09T15:28:24.080 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_16m ERROR [ 64%] 2024-03-09T15:28:24.432 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_same_bucket ERROR [ 64%] 2024-03-09T15:28:24.804 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_verify_contenttype ERROR [ 64%] 2024-03-09T15:28:25.173 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_to_itself ERROR [ 65%] 2024-03-09T15:28:25.548 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_to_itself_with_metadata ERROR [ 65%] 2024-03-09T15:28:26.054 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_diff_bucket ERROR [ 65%] 2024-03-09T15:28:26.427 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_not_owned_bucket ERROR [ 65%] 2024-03-09T15:28:26.728 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_not_owned_object_bucket ERROR [ 65%] 2024-03-09T15:28:27.100 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_canned_acl ERROR [ 65%] 2024-03-09T15:28:27.438 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_retaining_metadata ERROR [ 65%] 2024-03-09T15:28:27.823 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_replacing_metadata ERROR [ 65%] 2024-03-09T15:28:28.098 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_bucket_not_found ERROR [ 66%] 2024-03-09T15:28:28.437 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_key_not_found ERROR [ 66%] 2024-03-09T15:28:28.810 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_versioned_bucket ERROR [ 66%] 2024-03-09T15:28:29.247 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_versioned_url_encoding ERROR [ 66%] 2024-03-09T15:28:29.613 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_copy_versioning_multipart_upload ERROR [ 66%] 2024-03-09T15:28:29.983 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_upload_empty ERROR [ 66%] 2024-03-09T15:28:30.335 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_upload_small ERROR [ 66%] 2024-03-09T15:28:30.699 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_copy_small ERROR [ 67%] 2024-03-09T15:28:31.205 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_copy_invalid_range ERROR [ 67%] 2024-03-09T15:28:31.536 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_copy_without_range ERROR [ 67%] 2024-03-09T15:28:31.917 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_copy_special_names ERROR [ 67%] 2024-03-09T15:28:32.281 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_upload ERROR [ 67%] 2024-03-09T15:28:32.663 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_copy_versioned ERROR [ 67%] 2024-03-09T15:28:33.087 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_upload_resend_part ERROR [ 67%] 2024-03-09T15:28:33.457 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_upload_multiple_sizes ERROR [ 68%] 2024-03-09T15:28:33.782 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_copy_multiple_sizes ERROR [ 68%] 2024-03-09T15:28:34.154 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_upload_size_too_small ERROR [ 68%] 2024-03-09T15:28:34.535 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_upload_contents ERROR [ 68%] 2024-03-09T15:28:34.861 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_upload_overwrite_existing_object ERROR [ 68%] 2024-03-09T15:28:35.235 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_abort_multipart_upload ERROR [ 68%] 2024-03-09T15:28:35.533 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_abort_multipart_upload_not_found ERROR [ 68%] 2024-03-09T15:28:35.835 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_list_multipart_upload ERROR [ 69%] 2024-03-09T15:28:36.338 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_list_multipart_upload_owner ERROR [ 69%] 2024-03-09T15:28:36.650 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_upload_missing_part ERROR [ 69%] 2024-03-09T15:28:37.033 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_upload_incorrect_etag ERROR [ 69%] 2024-03-09T15:28:37.402 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_get_part ERROR [ 69%] 2024-03-09T15:28:37.791 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_non_multipart_get_part ERROR [ 69%] 2024-03-09T15:28:38.195 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_100_continue ERROR [ 69%] 2024-03-09T15:28:38.518 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_set_cors ERROR [ 69%] 2024-03-09T15:28:38.880 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_cors_origin_response ERROR [ 70%] 2024-03-09T15:28:39.217 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_cors_origin_wildcard ERROR [ 70%] 2024-03-09T15:28:39.595 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_cors_header_option ERROR [ 70%] 2024-03-09T15:28:40.021 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_cors_presigned_get_object ERROR [ 70%] 2024-03-09T15:28:40.367 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_cors_presigned_get_object_tenant ERROR [ 70%] 2024-03-09T15:28:40.738 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_cors_presigned_put_object ERROR [ 70%] 2024-03-09T15:28:41.106 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_cors_presigned_put_object_with_acl ERROR [ 70%] 2024-03-09T15:28:41.478 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_cors_presigned_put_object_tenant ERROR [ 71%] 2024-03-09T15:28:41.847 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_cors_presigned_put_object_tenant_with_acl ERROR [ 71%] 2024-03-09T15:28:42.203 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_set_bucket_tagging ERROR [ 71%] 2024-03-09T15:28:42.561 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_atomic_read_1mb ERROR [ 71%] 2024-03-09T15:28:42.935 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_atomic_read_4mb ERROR [ 71%] 2024-03-09T15:28:43.377 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_atomic_read_8mb ERROR [ 71%] 2024-03-09T15:28:43.752 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_atomic_write_1mb ERROR [ 71%] 2024-03-09T15:28:44.083 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_atomic_write_4mb ERROR [ 72%] 2024-03-09T15:28:44.456 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_atomic_write_8mb ERROR [ 72%] 2024-03-09T15:28:44.819 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_atomic_dual_write_1mb ERROR [ 72%] 2024-03-09T15:28:45.259 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_atomic_dual_write_4mb ERROR [ 72%] 2024-03-09T15:28:45.628 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_atomic_dual_write_8mb ERROR [ 72%] 2024-03-09T15:28:46.009 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_atomic_conditional_write_1mb ERROR [ 72%] 2024-03-09T15:28:46.451 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_atomic_multipart_upload_write ERROR [ 72%] 2024-03-09T15:28:46.815 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_resend_first_finishes_last ERROR [ 73%] 2024-03-09T15:28:47.240 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_ranged_request_response_code ERROR [ 73%] 2024-03-09T15:28:47.616 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_ranged_big_request_response_code ERROR [ 73%] 2024-03-09T15:28:47.980 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_ranged_request_skip_leading_bytes_response_code ERROR [ 73%] 2024-03-09T15:28:48.325 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_ranged_request_return_trailing_bytes_response_code ERROR [ 73%] 2024-03-09T15:28:48.663 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_ranged_request_invalid_range ERROR [ 73%] 2024-03-09T15:28:49.047 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_ranged_request_empty_object ERROR [ 73%] 2024-03-09T15:28:49.418 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_bucket_create_suspend ERROR [ 73%] 2024-03-09T15:28:49.788 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_obj_create_read_remove ERROR [ 74%] 2024-03-09T15:28:50.162 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_obj_create_read_remove_head ERROR [ 74%] 2024-03-09T15:28:50.602 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_obj_plain_null_version_removal ERROR [ 74%] 2024-03-09T15:28:50.965 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_obj_plain_null_version_overwrite ERROR [ 74%] 2024-03-09T15:28:51.340 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_obj_plain_null_version_overwrite_suspended ERROR [ 74%] 2024-03-09T15:28:51.716 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_obj_suspend_versions ERROR [ 74%] 2024-03-09T15:28:52.090 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_obj_suspended_copy ERROR [ 74%] 2024-03-09T15:28:52.538 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_obj_create_versions_remove_all ERROR [ 75%] 2024-03-09T15:28:52.896 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_obj_create_versions_remove_special_names ERROR [ 75%] 2024-03-09T15:28:53.274 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_obj_create_overwrite_multipart ERROR [ 75%] 2024-03-09T15:28:53.643 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_obj_list_marker ERROR [ 75%] 2024-03-09T15:28:53.999 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_copy_obj_version ERROR [ 75%] 2024-03-09T15:28:54.435 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_multi_object_delete ERROR [ 75%] 2024-03-09T15:28:54.791 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_multi_object_delete_with_marker ERROR [ 75%] 2024-03-09T15:28:55.165 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_multi_object_delete_with_marker_create ERROR [ 76%] 2024-03-09T15:28:55.568 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioned_object_acl ERROR [ 76%] 2024-03-09T15:28:55.930 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioned_object_acl_no_version_specified ERROR [ 76%] 2024-03-09T15:28:56.357 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioned_concurrent_object_create_and_remove ERROR [ 76%] 2024-03-09T15:28:56.704 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set ERROR [ 76%] 2024-03-09T15:28:57.080 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_get ERROR [ 76%] 2024-03-09T15:28:57.429 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_get_no_id ERROR [ 76%] 2024-03-09T15:28:57.862 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_id_too_long ERROR [ 76%] 2024-03-09T15:28:58.234 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_same_id ERROR [ 77%] 2024-03-09T15:28:58.601 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_invalid_status ERROR [ 77%] 2024-03-09T15:28:58.954 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_date ERROR [ 77%] 2024-03-09T15:28:59.323 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_invalid_date ERROR [ 77%] 2024-03-09T15:28:59.773 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_noncurrent ERROR [ 77%] 2024-03-09T15:29:00.145 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_deletemarker ERROR [ 77%] 2024-03-09T15:29:00.513 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_filter ERROR [ 77%] 2024-03-09T15:29:00.887 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_empty_filter ERROR [ 78%] 2024-03-09T15:29:01.216 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_multipart ERROR [ 78%] 2024-03-09T15:29:01.656 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_transition_set_invalid_date ERROR [ 78%] 2024-03-09T15:29:02.028 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encrypted_transfer_1b ERROR [ 78%] 2024-03-09T15:29:02.418 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encrypted_transfer_1kb ERROR [ 78%] 2024-03-09T15:29:02.752 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encrypted_transfer_1MB ERROR [ 78%] 2024-03-09T15:29:03.123 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encrypted_transfer_13b ERROR [ 78%] 2024-03-09T15:29:03.563 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_method_head ERROR [ 79%] 2024-03-09T15:29:03.936 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_present ERROR [ 79%] 2024-03-09T15:29:04.278 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_other_key ERROR [ 79%] 2024-03-09T15:29:04.663 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_invalid_md5 ERROR [ 79%] 2024-03-09T15:29:05.102 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_no_md5 ERROR [ 79%] 2024-03-09T15:29:05.388 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_no_key ERROR [ 79%] 2024-03-09T15:29:05.760 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encryption_key_no_sse_c ERROR [ 79%] 2024-03-09T15:29:06.112 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_multipart_upload ERROR [ 80%] 2024-03-09T15:29:06.517 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_unaligned_multipart_upload ERROR [ 80%] 2024-03-09T15:29:06.922 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_multipart_bad_download ERROR [ 80%] 2024-03-09T15:29:07.290 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_post_object_authenticated_request ERROR [ 80%] 2024-03-09T15:29:07.668 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_method_head ERROR [ 80%] 2024-03-09T15:29:08.044 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_present ERROR [ 80%] 2024-03-09T15:29:08.421 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_no_key ERROR [ 80%] 2024-03-09T15:29:08.844 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_not_declared ERROR [ 80%] 2024-03-09T15:29:09.217 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_multipart_upload ERROR [ 81%] 2024-03-09T15:29:09.539 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_multipart_invalid_chunks_1 ERROR [ 81%] 2024-03-09T15:29:09.916 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_multipart_invalid_chunks_2 ERROR [ 81%] 2024-03-09T15:29:10.284 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_post_object_authenticated_request ERROR [ 81%] 2024-03-09T15:29:10.732 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_transfer_1b ERROR [ 81%] 2024-03-09T15:29:11.113 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_transfer_1kb ERROR [ 81%] 2024-03-09T15:29:11.457 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_transfer_1MB ERROR [ 81%] 2024-03-09T15:29:11.848 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_transfer_13b ERROR [ 82%] 2024-03-09T15:29:12.294 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_read_declare ERROR [ 82%] 2024-03-09T15:29:12.667 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy ERROR [ 82%] 2024-03-09T15:29:13.023 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucketv2_policy ERROR [ 82%] 2024-03-09T15:29:13.413 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy_acl ERROR [ 82%] 2024-03-09T15:29:13.809 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucketv2_policy_acl ERROR [ 82%] 2024-03-09T15:29:14.228 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy_another_bucket ERROR [ 82%] 2024-03-09T15:29:14.659 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucketv2_policy_another_bucket ERROR [ 83%] 2024-03-09T15:29:15.036 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_obj_tagging ERROR [ 83%] 2024-03-09T15:29:15.373 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_obj_head_tagging ERROR [ 83%] 2024-03-09T15:29:15.763 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_max_tags ERROR [ 83%] 2024-03-09T15:29:16.206 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_excess_tags ERROR [ 83%] 2024-03-09T15:29:16.629 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_max_kvsize_tags ERROR [ 83%] 2024-03-09T15:29:16.988 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_excess_key_tags ERROR [ 83%] 2024-03-09T15:29:17.329 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_excess_val_tags ERROR [ 84%] 2024-03-09T15:29:17.695 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_modify_tags ERROR [ 84%] 2024-03-09T15:29:18.099 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_delete_tags ERROR [ 84%] 2024-03-09T15:29:18.481 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_tags_anonymous_request ERROR [ 84%] 2024-03-09T15:29:18.804 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_post_object_tags_authenticated_request ERROR [ 84%] 2024-03-09T15:29:19.179 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_obj_with_tags ERROR [ 84%] 2024-03-09T15:29:19.624 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_tags_acl_public ERROR [ 84%] 2024-03-09T15:29:19.951 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_tags_acl_public ERROR [ 84%] 2024-03-09T15:29:20.331 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_delete_tags_obj_public ERROR [ 85%] 2024-03-09T15:29:20.700 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_bucket_atomic_upload_return_version_id ERROR [ 85%] 2024-03-09T15:29:21.057 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_versioning_bucket_multipart_upload_return_version_id ERROR [ 85%] 2024-03-09T15:29:21.535 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy_get_obj_existing_tag ERROR [ 85%] 2024-03-09T15:29:21.921 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy_get_obj_tagging_existing_tag ERROR [ 85%] 2024-03-09T15:29:22.290 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_tagging_existing_tag ERROR [ 85%] 2024-03-09T15:29:22.641 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_copy_source ERROR [ 85%] 2024-03-09T15:29:23.001 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_copy_source_meta ERROR [ 86%] 2024-03-09T15:29:23.442 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_acl ERROR [ 86%] 2024-03-09T15:29:23.812 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_grant ERROR [ 86%] 2024-03-09T15:29:24.181 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_obj_enc_conflict_c_s3 ERROR [ 86%] 2024-03-09T15:29:24.568 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_obj_enc_conflict_c_kms ERROR [ 86%] 2024-03-09T15:29:24.942 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_obj_enc_conflict_s3_kms ERROR [ 86%] 2024-03-09T15:29:25.383 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_obj_enc_conflict_bad_enc_kms ERROR [ 86%] 2024-03-09T15:29:25.759 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_kms_noenc ERROR [ 87%] 2024-03-09T15:29:26.119 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_kms_s3 ERROR [ 87%] 2024-03-09T15:29:26.502 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_bucket_policy_get_obj_acl_existing_tag ERROR [ 87%] 2024-03-09T15:29:26.934 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock ERROR [ 87%] 2024-03-09T15:29:27.256 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_bucket ERROR [ 87%] 2024-03-09T15:29:27.584 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock_with_days_and_years ERROR [ 87%] 2024-03-09T15:29:27.944 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_days ERROR [ 87%] 2024-03-09T15:29:28.316 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_years ERROR [ 88%] 2024-03-09T15:29:28.753 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_mode ERROR [ 88%] 2024-03-09T15:29:29.120 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_status ERROR [ 88%] 2024-03-09T15:29:29.493 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_suspend_versioning ERROR [ 88%] 2024-03-09T15:29:29.851 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_get_obj_lock ERROR [ 88%] 2024-03-09T15:29:30.203 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_get_obj_lock_invalid_bucket ERROR [ 88%] 2024-03-09T15:29:30.642 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention ERROR [ 88%] 2024-03-09T15:29:31.016 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_invalid_bucket ERROR [ 88%] 2024-03-09T15:29:31.391 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_invalid_mode ERROR [ 89%] 2024-03-09T15:29:31.730 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_get_obj_retention ERROR [ 89%] 2024-03-09T15:29:32.068 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_get_obj_retention_iso8601 ERROR [ 89%] 2024-03-09T15:29:32.508 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_get_obj_retention_invalid_bucket ERROR [ 89%] 2024-03-09T15:29:32.821 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_versionid ERROR [ 89%] 2024-03-09T15:29:33.192 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_override_default_retention ERROR [ 89%] 2024-03-09T15:29:33.551 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_increase_period ERROR [ 89%] 2024-03-09T15:29:33.936 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_shorten_period ERROR [ 90%] 2024-03-09T15:29:34.261 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_shorten_period_bypass ERROR [ 90%] 2024-03-09T15:29:34.624 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_delete_object_with_retention ERROR [ 90%] 2024-03-09T15:29:34.949 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_delete_object_with_retention_and_marker ERROR [ 90%] 2024-03-09T15:29:35.279 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_multi_delete_object_with_retention ERROR [ 90%] 2024-03-09T15:29:35.732 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_legal_hold ERROR [ 90%] 2024-03-09T15:29:36.109 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_legal_hold_invalid_bucket ERROR [ 90%] 2024-03-09T15:29:36.433 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_put_legal_hold_invalid_status ERROR [ 91%] 2024-03-09T15:29:36.791 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_get_legal_hold ERROR [ 91%] 2024-03-09T15:29:37.156 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_get_legal_hold_invalid_bucket ERROR [ 91%] 2024-03-09T15:29:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_delete_object_with_legal_hold_on ERROR [ 91%] 2024-03-09T15:29:37.943 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_delete_object_with_legal_hold_off ERROR [ 91%] 2024-03-09T15:29:38.322 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_get_obj_metadata ERROR [ 91%] 2024-03-09T15:29:38.701 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_uploading_obj ERROR [ 91%] 2024-03-09T15:29:39.055 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_changing_mode_from_governance_with_bypass ERROR [ 92%] 2024-03-09T15:29:39.491 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_changing_mode_from_governance_without_bypass ERROR [ 92%] 2024-03-09T15:29:39.816 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_object_lock_changing_mode_from_compliance ERROR [ 92%] 2024-03-09T15:29:40.172 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_copy_object_ifmatch_good ERROR [ 92%] 2024-03-09T15:29:40.508 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_copy_object_ifnonematch_failed ERROR [ 92%] 2024-03-09T15:29:40.953 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_bucket_policy_status ERROR [ 92%] 2024-03-09T15:29:41.328 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_public_acl_bucket_policy_status ERROR [ 92%] 2024-03-09T15:29:41.720 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_authpublic_acl_bucket_policy_status ERROR [ 92%] 2024-03-09T15:29:42.080 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_publicpolicy_acl_bucket_policy_status ERROR [ 93%] 2024-03-09T15:29:42.435 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_nonpublicpolicy_acl_bucket_policy_status ERROR [ 93%] 2024-03-09T15:29:42.885 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_nonpublicpolicy_deny_bucket_policy_status ERROR [ 93%] 2024-03-09T15:29:43.234 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_undefined_public_block ERROR [ 93%] 2024-03-09T15:29:43.611 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_public_block_deny_bucket_policy ERROR [ 93%] 2024-03-09T15:29:43.977 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_public_block ERROR [ 93%] 2024-03-09T15:29:44.321 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_block_public_put_bucket_acls ERROR [ 93%] 2024-03-09T15:29:44.724 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_block_public_object_canned_acls ERROR [ 94%] 2024-03-09T15:29:45.099 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_block_public_policy ERROR [ 94%] 2024-03-09T15:29:45.469 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_ignore_public_acls ERROR [ 94%] 2024-03-09T15:29:45.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_multipart_upload_on_a_bucket_with_policy ERROR [ 94%] 2024-03-09T15:29:46.156 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_put_bucket_encryption_kms ERROR [ 94%] 2024-03-09T15:29:46.516 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_bucket_encryption_kms ERROR [ 94%] 2024-03-09T15:29:46.874 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_delete_bucket_encryption_kms ERROR [ 94%] 2024-03-09T15:29:47.223 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_sse_kms_default_post_object_authenticated_request ERROR [ 95%] 2024-03-09T15:29:47.603 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py::test_get_object_torrent ERROR [ 95%] 2024-03-09T15:30:14.340 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_generate_where_clause PASSED [ 95%] 2024-03-09T15:30:39.328 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_generate_projection PASSED [ 95%] 2024-03-09T15:30:42.192 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_count_operation PASSED [ 95%] 2024-03-09T15:30:46.036 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_count_json_operation PASSED [ 95%] 2024-03-09T15:31:06.112 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_json_column_sum_min_max PASSED [ 95%] 2024-03-09T15:31:29.568 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_json_nullif_expressions PASSED [ 96%] 2024-03-09T15:31:37.704 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_column_sum_min_max PASSED [ 96%] 2024-03-09T15:31:42.916 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_nullif_expressions PASSED [ 96%] 2024-03-09T15:31:48.396 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_nulliftrue_expressions PASSED [ 96%] 2024-03-09T15:31:53.200 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_is_not_null_expressions PASSED [ 96%] 2024-03-09T15:31:56.192 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_lowerupper_expressions PASSED [ 96%] 2024-03-09T15:32:18.777 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_in_expressions PASSED [ 96%] 2024-03-09T15:32:28.164 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_true_false_in_expressions PASSED [ 96%] 2024-03-09T15:32:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_like_expressions PASSED [ 97%] 2024-03-09T15:33:01.172 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_truefalselike_expressions PASSED [ 97%] 2024-03-09T15:33:06.200 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_complex_expressions PASSED [ 97%] 2024-03-09T15:33:10.864 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_alias PASSED [ 97%] 2024-03-09T15:33:14.048 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_alias_cyclic_refernce PASSED [ 97%] 2024-03-09T15:33:32.288 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_datetime PASSED [ 97%] 2024-03-09T15:33:45.420 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_true_false_datetime PASSED [ 97%] 2024-03-09T15:33:49.400 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_csv_parser PASSED [ 98%] 2024-03-09T15:33:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_csv_definition PASSED [ 98%] 2024-03-09T15:34:03.580 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_schema_definition PASSED [ 98%] 2024-03-09T15:34:10.433 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_when_then_else_expressions PASSED [ 98%] 2024-03-09T15:34:18.688 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_coalesce_expressions PASSED [ 98%] 2024-03-09T15:34:23.284 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_cast_expressions PASSED [ 98%] 2024-03-09T15:34:24.840 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_version PASSED [ 98%] 2024-03-09T15:34:32.680 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_trim_expressions PASSED [ 99%] 2024-03-09T15:34:40.648 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_truefalse_trim_expressions PASSED [ 99%] 2024-03-09T15:34:46.032 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_escape_expressions PASSED [ 99%] 2024-03-09T15:34:53.772 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_case_value_expressions PASSED [ 99%] 2024-03-09T15:34:57.700 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_bool_cast_expressions PASSED [ 99%] 2024-03-09T15:35:53.820 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_progress_expressions PASSED [ 99%] 2024-03-09T15:35:55.272 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py::test_output_serial_expressions PASSED [ 99%] 2024-03-09T15:35:55.328 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_utils.py::test_generate PASSED [100%] 2024-03-09T15:35:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.328 INFO:teuthology.orchestra.run.smithi001.stdout:==================================== ERRORS ==================================== 2024-03-09T15:35:55.328 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_object_write_to_nonexist_bucket ____________ 2024-03-09T15:35:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.328 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.328 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.328 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.329 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.330 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.330 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.330 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.330 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.330 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.331 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.332 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.333 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.334 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.335 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.335 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.336 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.336 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_object_write_with_chunked_transfer_encoding ______ 2024-03-09T15:35:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.336 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.336 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.336 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.336 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.336 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.337 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.338 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.338 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.338 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.338 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.338 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.338 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.338 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.338 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.338 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.339 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.340 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.341 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.342 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.343 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.343 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.343 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.343 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.343 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.343 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_bucket_create_delete __________________ 2024-03-09T15:35:55.343 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.343 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.343 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.343 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.343 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.343 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.344 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.344 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.344 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.344 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.344 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.344 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.344 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.347 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.347 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.347 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.348 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.348 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.348 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.348 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.348 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.348 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.348 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.348 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.348 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.348 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.348 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.348 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.350 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.350 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.350 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.350 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.350 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.350 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.350 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.350 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.350 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.350 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.350 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.350 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.351 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.352 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.352 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.352 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.352 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.352 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.352 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.352 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.352 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.352 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.352 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.352 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.352 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_object_read_not_exist _________________ 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.353 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.354 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.354 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.354 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.354 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.354 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.354 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.354 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.354 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.354 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.354 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.354 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.354 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.355 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.355 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.355 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.355 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.355 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.355 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.357 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.358 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.359 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.360 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.360 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.360 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.360 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.360 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.360 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.360 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.360 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.360 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_object_requestid_matches_header_on_error ________ 2024-03-09T15:35:55.360 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.360 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.361 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.362 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.362 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.362 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.362 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.362 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.362 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.362 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.362 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.362 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.362 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.362 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.362 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.363 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.364 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.365 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.366 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.366 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.366 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.366 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.366 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.366 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.366 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.366 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.366 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.366 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.366 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.366 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.367 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.367 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.367 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.367 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.367 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.367 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.367 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.367 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.367 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.367 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_versioning_concurrent_multi_object_delete _______ 2024-03-09T15:35:55.367 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.367 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.368 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.368 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.368 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.368 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.368 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.368 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.368 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.368 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.368 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.368 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.368 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.368 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.369 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.370 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.371 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.372 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.373 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.374 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.374 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.374 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.374 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.374 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.374 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.374 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.374 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.374 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.374 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_multi_object_delete __________________ 2024-03-09T15:35:55.374 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.374 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.375 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.375 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.375 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.375 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.375 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.375 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.375 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.375 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.375 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.375 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.375 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.376 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.377 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.379 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.379 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.379 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.379 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.379 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.379 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.379 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.379 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.379 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.379 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.380 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.380 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.380 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.380 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.380 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.380 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.380 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.380 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.380 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.380 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.381 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.382 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.382 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.382 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.382 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_multi_objectv2_delete _________________ 2024-03-09T15:35:55.382 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.382 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.382 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.382 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.382 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.382 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.382 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.383 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.384 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.384 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.384 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.384 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.384 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.384 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.386 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.386 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.386 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.386 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.386 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.386 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.388 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.388 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.388 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.388 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.388 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.388 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.388 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.388 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.388 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.388 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.388 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.388 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.389 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.391 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.391 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.391 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.391 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.391 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.391 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.391 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_multi_object_delete_key_limit _____________ 2024-03-09T15:35:55.391 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.391 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.391 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.391 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.391 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.392 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.393 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.393 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.393 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.393 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.393 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.393 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.393 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.393 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.393 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.393 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.393 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.393 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.394 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.395 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.395 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.395 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.395 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.395 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.395 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.395 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.395 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.395 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.395 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.395 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.396 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.397 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.398 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.398 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.398 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.398 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.398 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.398 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.398 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.398 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_multi_objectv2_delete_key_limit ____________ 2024-03-09T15:35:55.398 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.398 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.398 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.398 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.399 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.400 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.401 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.402 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.402 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.402 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.402 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.402 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.402 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.402 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.402 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.402 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.402 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.402 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.403 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.404 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_object_head_zero_bytes _________________ 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.405 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.406 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.407 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.408 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.409 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.409 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.409 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.409 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.409 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.409 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.409 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.409 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.409 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.409 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.409 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.410 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.411 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.412 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.412 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.412 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.412 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.413 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.413 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.413 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.413 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.413 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.413 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.414 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.414 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.414 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.414 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.414 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.414 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.414 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_object_write_check_etag ________________ 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.415 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.416 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.417 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.418 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.419 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.419 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.419 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.419 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.419 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.419 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.419 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.419 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.419 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.419 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.419 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_object_write_cache_control _______________ 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.421 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.422 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.422 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.422 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.422 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.422 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.422 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.422 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.422 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.422 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.422 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.422 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.422 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.423 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.423 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.423 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.423 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.423 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.423 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.423 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.423 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.423 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.423 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.423 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.423 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.424 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.425 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.426 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.427 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_object_write_expires __________________ 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.428 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.429 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.430 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.431 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.432 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.432 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.432 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.432 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.432 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.432 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.432 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.432 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.432 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.432 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.432 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.433 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_object_write_read_update_read_delete __________ 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.434 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.435 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.436 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.437 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.438 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.438 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.438 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.438 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.438 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.438 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.438 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.438 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.438 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.438 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.439 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.439 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.439 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.439 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.439 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.439 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.439 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.439 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.439 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.439 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.440 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_object_set_get_metadata_none_to_good __________ 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.441 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.442 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.443 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.445 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.446 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.447 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.448 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_object_set_get_metadata_none_to_empty _________ 2024-03-09T15:35:55.448 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.448 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.448 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.448 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.448 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.448 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.448 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.448 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.448 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.448 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.450 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.453 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.453 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.453 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.453 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.453 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.453 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.453 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.453 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.453 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.453 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.453 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.454 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.454 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.454 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.454 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.454 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.455 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.455 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.455 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.455 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.455 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.455 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.455 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.456 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.456 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.456 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.456 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_object_set_get_metadata_overwrite_to_empty _______ 2024-03-09T15:35:55.456 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.456 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.457 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.457 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.457 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.457 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.457 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.457 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.457 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.458 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.458 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.458 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.458 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.458 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.458 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.459 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.459 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.459 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.459 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.459 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.459 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.460 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.460 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.460 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.460 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.460 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.460 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.461 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.461 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.461 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.461 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.461 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.461 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.462 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.462 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.463 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.463 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.463 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.463 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.463 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.463 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.464 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.464 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.464 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.464 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.464 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.464 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.464 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.465 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.465 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.465 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.465 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.465 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.465 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.465 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.466 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.466 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.466 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.466 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.466 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.466 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.467 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.467 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.467 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.467 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.467 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.467 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.467 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.468 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.468 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.468 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.468 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.468 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.468 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.468 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.468 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.468 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.468 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.468 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.468 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_object_metadata_replaced_on_put ____________ 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.469 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.470 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.471 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.473 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.474 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout:___________________ ERROR at setup of test_object_write_file ___________________ 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.475 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.476 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.478 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.479 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.480 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_post_object_anonymous_request _____________ 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.481 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.482 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.483 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.484 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.485 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.486 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_post_object_authenticated_request ___________ 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.487 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.488 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.489 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.490 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.491 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.492 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.493 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_post_object_authenticated_no_content_type _______ 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.494 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.495 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.496 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.497 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.498 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.499 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout:___ ERROR at setup of test_post_object_authenticated_request_bad_access_key ____ 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.500 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.501 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.502 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.503 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.504 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.505 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_post_object_set_success_code ______________ 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.506 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.507 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.508 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.509 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.510 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.511 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_post_object_set_invalid_success_code __________ 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.512 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.513 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.514 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.515 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.516 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.517 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_post_object_upload_larger_than_chunk __________ 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.518 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.519 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.520 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.521 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.523 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_post_object_set_key_from_filename ___________ 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.525 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.526 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.527 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.528 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.529 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_post_object_ignored_header _______________ 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.530 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.531 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.532 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.533 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.534 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.535 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_post_object_case_insensitive_condition_fields _____ 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.536 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.537 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.538 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.539 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.540 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.541 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_post_object_escaped_field_values ____________ 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.542 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.543 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.544 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.545 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.546 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.547 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.548 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_post_object_success_redirect_action __________ 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.549 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.550 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.551 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.552 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.553 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.554 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_post_object_invalid_signature _____________ 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.555 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.556 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.556 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.556 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.556 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.556 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.556 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.556 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.556 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.556 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.556 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.556 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.556 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.580 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.582 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.583 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.584 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_post_object_invalid_access_key _____________ 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.585 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.586 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.587 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.587 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.587 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.587 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.587 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.587 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.587 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.587 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.587 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.587 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.587 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.589 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.589 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.589 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.589 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.589 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.589 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.589 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.589 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.589 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.589 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.590 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.591 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.592 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_post_object_invalid_date_format ____________ 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.593 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.594 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.595 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.595 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.595 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.595 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.595 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.595 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.595 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.595 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.595 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.595 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.596 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.597 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.598 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_post_object_no_key_specified ______________ 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.599 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.600 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.601 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.603 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.604 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_post_object_missing_signature _____________ 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.605 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.606 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.607 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.608 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.609 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.609 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.609 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.609 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.609 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.609 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.609 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.609 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.610 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.611 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_post_object_missing_policy_condition __________ 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.612 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.613 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.614 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.615 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.616 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.617 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_post_object_user_specified_header ___________ 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.619 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.621 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.622 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.622 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.622 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.622 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.622 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.622 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.622 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.622 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.622 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.622 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout:__ ERROR at setup of test_post_object_request_missing_policy_specified_field ___ 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.624 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.625 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.626 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.628 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.629 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_post_object_condition_is_case_sensitive ________ 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.630 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.631 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.632 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.634 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.635 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.636 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.636 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.636 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.636 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.636 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.636 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.636 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.636 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_post_object_expires_is_case_sensitive _________ 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.638 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.639 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.640 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.641 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.642 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_post_object_expired_policy _______________ 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.643 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.644 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.645 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.646 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.647 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.648 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.649 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.649 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.649 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.649 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.649 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.649 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.649 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.649 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_post_object_wrong_bucket ________________ 2024-03-09T15:35:55.649 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.649 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.649 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.649 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.650 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.651 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.652 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.653 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.654 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_post_object_invalid_request_field_value ________ 2024-03-09T15:35:55.655 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.656 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.657 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.658 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.659 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.660 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.661 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_post_object_missing_expires_condition _________ 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.662 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.663 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.663 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.663 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.663 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.663 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.663 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.663 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.663 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.664 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.665 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.666 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.667 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_post_object_missing_conditions_list __________ 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.668 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.669 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.670 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.671 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.672 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.673 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_post_object_upload_size_limit_exceeded _________ 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.675 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.676 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.676 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.676 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.677 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.678 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.679 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.680 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_post_object_missing_content_length_argument ______ 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.681 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.682 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.683 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.684 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.685 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.686 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_post_object_invalid_content_length_argument ______ 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.687 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.688 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.689 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.690 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.690 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.690 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.690 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.690 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.690 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.690 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.690 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.690 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.690 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.690 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.691 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.692 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_post_object_upload_size_below_minimum _________ 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.693 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.694 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.696 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.697 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.698 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_post_object_upload_size_rgw_chunk_size_bug _______ 2024-03-09T15:35:55.699 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.700 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.701 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.702 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.703 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.703 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.703 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.703 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.703 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.703 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.703 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.703 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.703 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.703 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.703 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.704 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.705 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_post_object_empty_conditions ______________ 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.706 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.707 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.708 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.710 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.710 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.710 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.710 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.710 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.710 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.710 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.710 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.710 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.710 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.710 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.710 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.711 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_get_object_ifmatch_good ________________ 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.712 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.713 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.715 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.716 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.717 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.717 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.717 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.717 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.717 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.717 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.717 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.717 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.717 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.717 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.718 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_get_object_ifmatch_failed _______________ 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.719 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.720 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.721 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.722 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.723 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.724 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.725 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.725 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_get_object_ifnonematch_good ______________ 2024-03-09T15:35:55.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.725 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.725 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.725 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.725 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.725 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.726 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.728 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.729 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.730 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.731 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.731 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.731 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.731 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.731 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.731 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.731 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.731 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.731 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_get_object_ifnonematch_failed _____________ 2024-03-09T15:35:55.731 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.731 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.731 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.732 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.733 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.734 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.735 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.736 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.737 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_get_object_ifmodifiedsince_good ____________ 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.738 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.739 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.740 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.741 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.742 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.743 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_get_object_ifmodifiedsince_failed ___________ 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.744 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.745 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.746 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.747 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_get_object_ifunmodifiedsince_good ___________ 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.750 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.751 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.752 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.753 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.754 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.755 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_get_object_ifunmodifiedsince_failed __________ 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.756 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.757 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.758 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.758 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.758 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.758 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.758 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.758 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.758 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.758 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.758 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.758 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.759 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.760 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.761 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.762 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_put_object_ifmatch_good ________________ 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.763 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.764 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.765 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.766 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.767 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.768 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_put_object_ifmatch_failed _______________ 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.769 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.770 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.771 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.772 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.772 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.772 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.772 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.772 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.772 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.772 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.772 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.772 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.772 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.773 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.774 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_put_object_ifmatch_overwrite_existed_good _______ 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.775 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.776 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.777 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.778 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.779 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.780 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.781 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_put_object_ifmatch_nonexisted_failed __________ 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.782 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.783 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.784 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.785 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.786 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.787 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_put_object_ifnonmatch_good _______________ 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.788 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.789 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.790 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.791 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.792 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.792 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.792 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.792 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.793 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.793 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.793 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.793 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.793 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.793 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.793 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.793 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.793 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.793 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.793 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.794 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_put_object_ifnonmatch_failed ______________ 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.795 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.796 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.798 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.799 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.800 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.801 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_put_object_ifnonmatch_nonexisted_good _________ 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.802 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.803 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.804 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.805 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.806 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.807 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout:____ ERROR at setup of test_put_object_ifnonmatch_overwrite_existed_failed _____ 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.808 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.809 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.810 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.811 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.812 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.813 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_object_raw_get _____________________ 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.814 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.815 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.816 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.817 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.818 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.819 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_object_raw_get_bucket_gone _______________ 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.820 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.821 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.823 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.824 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.825 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_object_delete_key_bucket_gone _____________ 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.826 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.827 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.827 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.827 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.827 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.827 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.827 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.827 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.827 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.829 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.830 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.831 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_object_raw_get_object_gone _______________ 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.832 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.833 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.834 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.835 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.836 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.837 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.838 INFO:teuthology.orchestra.run.smithi001.stdout:______________________ ERROR at setup of test_bucket_head ______________________ 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.839 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.840 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.840 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.840 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.840 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.840 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.840 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.840 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.840 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.841 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.842 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.843 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.844 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_bucket_head_notexist __________________ 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.845 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.846 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.847 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.848 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.849 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.850 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_bucket_head_extended __________________ 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.851 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.852 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.853 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.853 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.853 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.853 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.853 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.853 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.853 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.853 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.853 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.853 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.854 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.855 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.856 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_object_raw_get_bucket_acl _______________ 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.857 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.858 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.859 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.860 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.861 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.862 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_object_raw_get_object_acl _______________ 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.863 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.864 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.865 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.866 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.867 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.867 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.867 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.867 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.867 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.867 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.867 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.867 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.867 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.867 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.867 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.867 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.868 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.869 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_object_put_acl_mtime __________________ 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.870 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.871 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.871 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.871 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.871 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.871 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.871 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.872 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.872 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.872 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.872 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.872 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.873 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.874 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.876 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_object_raw_authenticated ________________ 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.877 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.878 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.879 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.880 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.881 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.881 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.881 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.881 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.881 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.881 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.881 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.881 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.881 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.881 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.881 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.882 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_object_raw_response_headers ______________ 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.883 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.884 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.885 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.886 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.887 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.888 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_object_raw_authenticated_bucket_acl __________ 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.889 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.890 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.892 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.893 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.894 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.894 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.894 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.894 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.894 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.894 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.894 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.894 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.894 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.894 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.894 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.894 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.895 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.895 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.895 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.895 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.895 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.895 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.895 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.895 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.895 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.895 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.895 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.895 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_object_raw_authenticated_object_acl __________ 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.896 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.897 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.898 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.899 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.900 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.901 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_object_raw_authenticated_bucket_gone __________ 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.902 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.903 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.904 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.905 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.906 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.907 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_object_raw_authenticated_object_gone __________ 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.908 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.909 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.911 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.912 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.913 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_object_raw_get_x_amz_expires_not_expired ________ 2024-03-09T15:35:55.914 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.915 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.916 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.917 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.918 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.919 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.920 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.920 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.920 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.920 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.920 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.920 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.920 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.920 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.920 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.920 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.920 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.920 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout:____ ERROR at setup of test_object_raw_get_x_amz_expires_not_expired_tenant ____ 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.921 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.922 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.922 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.922 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.922 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.922 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.922 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.922 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.922 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.922 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.922 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.922 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.922 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.923 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.924 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.925 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.926 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_object_raw_get_x_amz_expires_out_range_zero ______ 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.928 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.930 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.931 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.932 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_object_raw_get_x_amz_expires_out_max_range _______ 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.934 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.935 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.935 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.935 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.935 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.935 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.935 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.935 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.935 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.935 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.935 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.935 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.935 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.936 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.937 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.938 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.939 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout:____ ERROR at setup of test_object_raw_get_x_amz_expires_out_positive_range ____ 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.940 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.941 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.942 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.943 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.944 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.945 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_object_anon_put ____________________ 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.946 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.947 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.948 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.949 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.949 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.949 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.949 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.949 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.949 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.949 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.949 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.949 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.949 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.949 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.949 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.950 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.951 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_object_anon_put_write_access ______________ 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.952 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.953 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.955 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.956 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.957 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_object_put_authenticated ________________ 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.958 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.959 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.960 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.961 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.962 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.962 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.962 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.962 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.962 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.962 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.962 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.962 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.962 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.963 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.964 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_object_presigned_put_object_with_acl __________ 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.965 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.966 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.967 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.968 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.969 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.970 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_object_presigned_put_object_with_acl_tenant ______ 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.971 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.972 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.973 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.974 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.975 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.976 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.976 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.976 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.976 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.976 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.976 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.976 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.976 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.976 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.976 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.976 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.976 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_object_raw_put_authenticated_expired __________ 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.977 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.978 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.979 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.980 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.981 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.982 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_bucket_create_naming_bad_starts_nonalpha ________ 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.983 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.984 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.985 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.986 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.987 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.988 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.989 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_bucket_create_naming_bad_short_one ___________ 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.990 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.991 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.992 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.993 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:55.994 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:55.995 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_bucket_create_naming_bad_short_two ___________ 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:55.996 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.997 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.998 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:55.999 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.000 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.001 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_bucket_create_naming_good_long_60 ___________ 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.002 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.003 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.005 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.006 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_bucket_create_naming_good_long_61 ___________ 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.008 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.009 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.013 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_bucket_create_naming_good_long_62 ___________ 2024-03-09T15:35:56.014 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.015 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.016 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.016 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.016 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.016 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.016 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.016 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.016 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.017 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.018 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.019 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.020 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_bucket_create_naming_good_long_63 ___________ 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.021 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.022 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.023 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.024 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.025 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.026 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_bucket_list_long_name _________________ 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.027 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.028 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.029 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.030 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.030 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.030 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.030 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.030 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.030 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.030 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.030 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.030 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.030 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.030 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.031 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.032 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_bucket_create_naming_bad_ip ______________ 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.033 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.034 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.036 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.037 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.039 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_bucket_create_naming_dns_underscore __________ 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.040 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.041 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.042 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.043 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.044 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.045 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_bucket_create_naming_dns_long _____________ 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.046 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.047 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.048 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.049 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.050 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.051 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_bucket_create_naming_dns_dash_at_end __________ 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.052 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.053 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.054 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.055 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.056 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.057 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.057 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.057 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.057 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.057 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.057 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.057 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.057 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.057 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.057 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.057 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.057 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_bucket_create_naming_dns_dot_dot ____________ 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.058 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.059 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.060 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.063 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_bucket_create_naming_dns_dot_dash ___________ 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.064 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.065 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.067 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.068 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.069 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.070 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.070 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.070 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.070 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.070 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.070 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.070 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.070 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.070 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_bucket_create_naming_dns_dash_dot ___________ 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.072 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.073 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.074 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.075 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.076 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_bucket_create_exists __________________ 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.077 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.078 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.079 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.080 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.081 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.082 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_bucket_get_location __________________ 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.083 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.084 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.085 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.086 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.088 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_bucket_create_exists_nonowner _____________ 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.089 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.090 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.094 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_bucket_recreate_overwrite_acl _____________ 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.095 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.096 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.097 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.097 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.097 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.097 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.097 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.097 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.097 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.097 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.097 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.097 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.097 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.098 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.099 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.100 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.101 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_bucket_recreate_new_acl ________________ 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.102 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.103 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.104 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.106 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.107 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_bucket_acl_default ___________________ 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.108 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.109 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.110 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.110 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.110 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.110 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.110 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.110 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.110 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.110 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.110 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.110 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.111 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.112 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.113 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_bucket_acl_canned_during_create ____________ 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.114 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.115 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.118 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.119 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout:___________________ ERROR at setup of test_bucket_acl_canned ___________________ 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.120 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.121 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.122 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.123 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.124 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.125 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.126 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_bucket_acl_canned_publicreadwrite ___________ 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.127 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.128 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.129 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.130 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.131 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.132 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_bucket_acl_canned_authenticatedread __________ 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.133 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_object_acl_default ___________________ 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.140 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_object_acl_canned_during_create ____________ 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.146 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.147 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.148 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.149 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.150 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout:___________________ ERROR at setup of test_object_acl_canned ___________________ 2024-03-09T15:35:56.151 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.152 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.153 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.154 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.155 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.156 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.157 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_object_acl_canned_publicreadwrite ___________ 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.158 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.159 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.160 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.161 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.162 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.163 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.164 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.164 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.164 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_object_acl_canned_authenticatedread __________ 2024-03-09T15:35:56.164 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.164 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.164 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.164 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.164 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.164 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.164 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.164 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.164 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.165 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.166 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.167 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.168 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.169 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_object_acl_canned_bucketownerread ___________ 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.170 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.171 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.173 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.174 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.175 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_object_acl_canned_bucketownerfullcontrol ________ 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.176 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.177 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.177 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.177 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.177 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.177 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.177 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.177 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.177 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.177 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.177 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.177 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.177 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.178 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.179 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.180 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.181 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_object_acl_full_control_verify_owner __________ 2024-03-09T15:35:56.182 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.183 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.184 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.185 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.186 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.187 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.188 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_object_acl_full_control_verify_attributes _______ 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.189 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.190 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.191 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.191 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.191 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.191 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.191 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.191 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.191 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.191 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.191 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.191 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.192 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.193 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.194 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_bucket_acl_canned_private_to_private __________ 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.195 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.196 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.197 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.198 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.199 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.201 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.201 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.201 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.201 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.201 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.201 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.201 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.201 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.201 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.201 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout:______________________ ERROR at setup of test_object_acl _______________________ 2024-03-09T15:35:56.202 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.203 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.204 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.205 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.206 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.207 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.208 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout:___________________ ERROR at setup of test_object_acl_write ____________________ 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.209 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.210 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.211 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.212 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.213 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.214 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_object_acl_writeacp __________________ 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.215 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.216 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.217 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.219 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.219 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.219 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.219 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.219 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.219 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.219 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.219 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.219 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.219 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.219 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.219 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.220 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_object_acl_read ____________________ 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.221 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.222 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.224 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.225 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.226 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.226 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.226 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.226 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.226 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.226 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.226 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.226 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.226 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.226 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_object_acl_readacp ___________________ 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.228 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.229 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.230 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.231 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.232 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.233 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_bucket_acl_grant_userid_fullcontrol __________ 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.234 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.235 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.236 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.237 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.238 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_bucket_acl_grant_userid_read ______________ 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.243 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.244 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.245 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.246 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.246 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.246 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.246 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.246 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.246 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.246 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.246 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.246 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.246 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.246 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_bucket_acl_grant_userid_readacp ____________ 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.247 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.248 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.249 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.250 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.251 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.252 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_bucket_acl_grant_userid_write _____________ 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.253 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.254 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.255 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.256 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.257 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.258 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_bucket_acl_grant_userid_writeacp ____________ 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.259 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.260 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.261 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.262 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.263 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.264 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_bucket_acl_grant_nonexist_user _____________ 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.265 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.266 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.268 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.270 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_object_header_acl_grants ________________ 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.271 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.272 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.274 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.275 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.276 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_bucket_header_acl_grants ________________ 2024-03-09T15:35:56.277 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.278 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.279 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.280 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.281 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.282 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.283 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_bucket_acl_grant_email _________________ 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.284 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.285 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.286 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.287 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.288 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.289 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_bucket_acl_grant_email_not_exist ____________ 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.290 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.291 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.293 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.294 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.295 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_bucket_acl_revoke_all _________________ 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.296 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.297 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.298 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_access_bucket_private_object_private __________ 2024-03-09T15:35:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.308 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_access_bucket_private_objectv2_private _________ 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.309 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.313 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.313 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.313 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.313 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.313 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.313 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.313 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.313 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.313 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.314 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_access_bucket_private_object_publicread ________ 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.315 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.316 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.317 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.318 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.319 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.320 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_access_bucket_private_objectv2_publicread _______ 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.321 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.322 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.323 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.324 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.325 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.326 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.327 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.327 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.327 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.327 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.327 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.327 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.327 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.327 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.327 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.327 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.327 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.327 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_access_bucket_private_object_publicreadwrite ______ 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.328 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.329 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.330 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.331 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.332 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.333 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout:____ ERROR at setup of test_access_bucket_private_objectv2_publicreadwrite _____ 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.334 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.335 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.336 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.337 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.338 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.339 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_access_bucket_publicread_object_private ________ 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.340 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.341 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.341 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.341 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.341 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.341 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.343 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.344 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.345 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_access_bucket_publicread_object_publicread _______ 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.346 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.348 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.350 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.351 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout:____ ERROR at setup of test_access_bucket_publicread_object_publicreadwrite ____ 2024-03-09T15:35:56.352 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.353 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.354 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.354 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.354 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.354 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.354 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.354 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.354 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.354 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.354 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.354 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.354 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.354 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.355 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.356 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.357 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.358 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_access_bucket_publicreadwrite_object_private ______ 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.359 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.360 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.361 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.362 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.363 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.364 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout:____ ERROR at setup of test_access_bucket_publicreadwrite_object_publicread ____ 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.365 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.366 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.368 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.369 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.370 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout:_ ERROR at setup of test_access_bucket_publicreadwrite_object_publicreadwrite __ 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.371 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.372 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.373 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.374 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.375 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.376 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_buckets_create_then_list ________________ 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.377 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.378 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.380 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.381 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.383 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_buckets_list_ctime ___________________ 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.384 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_list_buckets_anonymous _________________ 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.390 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.391 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.392 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.393 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.394 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.394 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.394 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.394 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.394 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.394 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.394 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.394 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.394 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.394 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.394 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.394 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_list_buckets_invalid_auth _______________ 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.396 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.397 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.398 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.399 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.400 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.401 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_list_buckets_bad_auth _________________ 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.402 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.403 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.405 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.405 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.405 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.405 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.405 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.405 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.405 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.405 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.405 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.405 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.406 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.407 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.408 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.409 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_bucket_create_naming_good_starts_alpha _________ 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.410 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.411 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.412 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.413 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.414 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.415 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_bucket_create_naming_good_starts_digit _________ 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.416 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.417 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.418 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.419 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.420 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.421 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.422 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.422 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.422 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.422 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.422 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.422 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.422 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.422 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.422 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.422 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.422 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.422 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_bucket_create_naming_good_contains_period _______ 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.423 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.424 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.427 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_bucket_create_naming_good_contains_hyphen _______ 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.430 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.431 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.432 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.433 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.434 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_bucket_recreate_not_overriding _____________ 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.435 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.436 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.436 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.436 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.436 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.436 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.436 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.436 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.436 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.436 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.436 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.436 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.436 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.437 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.438 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.439 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.440 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_bucket_create_special_key_names ____________ 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.441 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.442 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.443 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.444 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.445 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.446 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_bucket_list_special_prefix _______________ 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.447 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.448 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.449 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.450 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_object_copy_zero_size _________________ 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.453 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.454 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.455 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.456 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.457 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.458 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_object_copy_16m ____________________ 2024-03-09T15:35:56.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.460 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.461 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.463 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.464 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.465 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_object_copy_same_bucket ________________ 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.466 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.467 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.468 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.469 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.470 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.471 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_object_copy_verify_contenttype _____________ 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.472 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.473 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.474 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.475 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.476 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.477 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_object_copy_to_itself _________________ 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.478 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.479 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.480 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.481 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.482 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.483 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_object_copy_to_itself_with_metadata __________ 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.484 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.485 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.486 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.487 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.488 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.489 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.490 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_object_copy_diff_bucket ________________ 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.491 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.492 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.493 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.494 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.495 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.496 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_object_copy_not_owned_bucket ______________ 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.497 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.498 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.499 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.500 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.501 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.502 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_object_copy_not_owned_object_bucket __________ 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.503 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.504 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.505 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.506 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.507 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.508 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_object_copy_canned_acl _________________ 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.509 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.510 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.511 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.512 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.513 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.514 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_object_copy_retaining_metadata _____________ 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.515 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.516 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.517 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.518 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.519 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.520 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.521 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_object_copy_replacing_metadata _____________ 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.522 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.523 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.524 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.525 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.526 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.527 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_object_copy_bucket_not_found ______________ 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.528 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.529 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.530 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.530 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.530 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.530 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.530 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.530 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.530 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.530 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.530 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.530 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.530 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.532 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_object_copy_key_not_found _______________ 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.534 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.535 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.536 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.537 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.538 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.539 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.540 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_object_copy_versioned_bucket ______________ 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.541 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.542 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.543 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.544 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.545 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.546 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_object_copy_versioned_url_encoding ___________ 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.547 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.548 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.549 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.550 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.551 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.552 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_object_copy_versioning_multipart_upload ________ 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.553 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.554 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.555 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.556 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.557 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.558 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_multipart_upload_empty _________________ 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.559 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.560 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.561 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.562 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.564 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_multipart_upload_small _________________ 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.566 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.567 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.569 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.570 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.571 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_multipart_copy_small __________________ 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.572 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.573 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.574 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.575 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.576 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.577 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_multipart_copy_invalid_range ______________ 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.578 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.579 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.580 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.581 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.582 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.583 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.584 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.584 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.584 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.584 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.584 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.584 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.584 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_multipart_copy_without_range ______________ 2024-03-09T15:35:56.584 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.584 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.584 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.584 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.584 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.585 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.586 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.587 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.588 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.589 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_multipart_copy_special_names ______________ 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.590 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.591 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.592 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.593 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.594 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.595 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.596 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout:___________________ ERROR at setup of test_multipart_upload ____________________ 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.597 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.598 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.599 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.600 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.601 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.602 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_multipart_copy_versioned ________________ 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.604 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.606 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.607 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.609 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.609 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.609 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.609 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.609 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.609 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.609 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.609 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.609 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.610 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_multipart_upload_resend_part ______________ 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.611 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.612 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.613 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.614 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.615 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.616 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_multipart_upload_multiple_sizes ____________ 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.617 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.618 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.619 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.620 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.621 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.622 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_multipart_copy_multiple_sizes _____________ 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.623 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.624 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.626 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.627 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.628 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_multipart_upload_size_too_small ____________ 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.629 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.630 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.631 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.632 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.633 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.634 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_multipart_upload_contents _______________ 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.635 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.636 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.637 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.638 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.638 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.638 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.638 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.638 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.638 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.638 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.638 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.638 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.638 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.638 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.639 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.640 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.641 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_multipart_upload_overwrite_existing_object _______ 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.642 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.643 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.644 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.645 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.646 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.647 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_abort_multipart_upload _________________ 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.648 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_abort_multipart_upload_not_found ____________ 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.654 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.655 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_list_multipart_upload _________________ 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.660 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.661 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.662 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.663 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.664 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.665 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.665 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.665 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.665 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.665 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.665 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.665 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.665 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.665 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.665 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_list_multipart_upload_owner ______________ 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.666 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.667 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.668 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.669 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.670 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.671 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.672 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_multipart_upload_missing_part _____________ 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.673 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.674 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.675 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.676 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.677 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.678 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.679 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.679 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_multipart_upload_incorrect_etag ____________ 2024-03-09T15:35:56.679 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.679 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.679 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.679 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.679 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.679 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.679 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.679 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.679 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.679 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.680 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.681 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.682 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.683 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.684 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_multipart_get_part ___________________ 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.685 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.686 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.687 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.688 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.689 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.690 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_non_multipart_get_part _________________ 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.692 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.693 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.694 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.696 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout:_____________________ ERROR at setup of test_100_continue ______________________ 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.697 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.698 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.699 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.700 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.701 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.702 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.703 INFO:teuthology.orchestra.run.smithi001.stdout:_______________________ ERROR at setup of test_set_cors ________________________ 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.704 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.705 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.706 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.707 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.708 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.709 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_cors_origin_response __________________ 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.710 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.713 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.714 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.715 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_cors_origin_wildcard __________________ 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.716 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.717 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.718 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.719 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.720 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.721 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_cors_header_option ___________________ 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.722 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.723 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.724 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.725 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.726 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.727 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_cors_presigned_get_object _______________ 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.728 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.729 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.730 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.731 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.732 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.733 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.734 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_cors_presigned_get_object_tenant ____________ 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.735 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.736 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.737 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.738 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.739 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.740 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_cors_presigned_put_object _______________ 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.741 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.742 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.744 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.745 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.746 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_cors_presigned_put_object_with_acl ___________ 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.747 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.748 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.752 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_cors_presigned_put_object_tenant ____________ 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.753 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.754 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.755 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.757 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.758 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_cors_presigned_put_object_tenant_with_acl _______ 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.760 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.761 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.762 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.763 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.764 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.765 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_set_bucket_tagging ___________________ 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.766 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.767 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.768 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_atomic_read_1mb ____________________ 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.772 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.773 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.773 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.773 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.773 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.773 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.773 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.773 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.773 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.773 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.773 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.773 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.773 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.774 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.777 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.778 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_atomic_read_4mb ____________________ 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.779 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.780 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.781 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.782 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.783 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.784 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_atomic_read_8mb ____________________ 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.785 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.786 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.786 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.786 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.786 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.786 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.786 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.786 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.786 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.786 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.786 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.786 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.786 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.787 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.788 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.789 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.790 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout:___________________ ERROR at setup of test_atomic_write_1mb ____________________ 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.791 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.792 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.793 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.794 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.795 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.796 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout:___________________ ERROR at setup of test_atomic_write_4mb ____________________ 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.797 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.798 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.799 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.801 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.802 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout:___________________ ERROR at setup of test_atomic_write_8mb ____________________ 2024-03-09T15:35:56.803 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.804 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.805 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.806 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.807 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.808 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.809 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_atomic_dual_write_1mb _________________ 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.810 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.811 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.812 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.812 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.812 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.813 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.813 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.813 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.813 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.813 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.813 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.813 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.813 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.814 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.814 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.814 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.814 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.814 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.814 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.814 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.814 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.814 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.814 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.815 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.815 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.815 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.815 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.815 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.815 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.815 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.815 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.815 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.816 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.817 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_atomic_dual_write_4mb _________________ 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.818 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.819 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.820 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.821 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.822 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.823 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_atomic_dual_write_8mb _________________ 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.824 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.825 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.828 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.829 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_atomic_conditional_write_1mb ______________ 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.830 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.831 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.833 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.834 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.835 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_atomic_multipart_upload_write _____________ 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.836 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.837 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.838 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.839 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.840 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.841 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.842 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.842 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.842 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.842 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.842 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.842 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_multipart_resend_first_finishes_last __________ 2024-03-09T15:35:56.842 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.843 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.844 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.845 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.846 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.847 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.848 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_ranged_request_response_code ______________ 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.849 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.851 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.852 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.853 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.854 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_ranged_big_request_response_code ____________ 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.855 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.856 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.857 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.858 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.859 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.860 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout:____ ERROR at setup of test_ranged_request_skip_leading_bytes_response_code ____ 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.861 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.862 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.863 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.864 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.865 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.866 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout:__ ERROR at setup of test_ranged_request_return_trailing_bytes_response_code ___ 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.867 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.868 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.870 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.871 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.872 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_ranged_request_invalid_range ______________ 2024-03-09T15:35:56.873 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.874 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.875 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.876 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.877 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.879 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_ranged_request_empty_object ______________ 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.880 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.881 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.882 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.882 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.882 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.882 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.882 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.882 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.882 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.882 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.882 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.882 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.883 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.884 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_versioning_bucket_create_suspend ____________ 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.886 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.887 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.888 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.889 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.890 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.891 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_versioning_obj_create_read_remove ___________ 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.892 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.893 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.894 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.895 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.896 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.897 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_versioning_obj_create_read_remove_head _________ 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.898 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.899 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.900 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.901 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.903 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.904 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_versioning_obj_plain_null_version_removal _______ 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.905 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.906 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.907 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.908 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.909 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.910 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_versioning_obj_plain_null_version_overwrite ______ 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.911 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.912 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.913 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.914 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.915 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.916 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout:_ ERROR at setup of test_versioning_obj_plain_null_version_overwrite_suspended _ 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.917 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.918 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.919 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.920 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.921 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.922 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_versioning_obj_suspend_versions ____________ 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.923 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.924 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.925 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.926 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.928 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_versioning_obj_suspended_copy _____________ 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.929 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.930 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.933 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.934 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_versioning_obj_create_versions_remove_all _______ 2024-03-09T15:35:56.935 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.936 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.937 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.938 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.939 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.940 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.941 INFO:teuthology.orchestra.run.smithi001.stdout:__ ERROR at setup of test_versioning_obj_create_versions_remove_special_names __ 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.942 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.943 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.944 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.945 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.946 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.947 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_versioning_obj_create_overwrite_multipart _______ 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.948 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.949 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.950 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.951 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.952 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.953 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_versioning_obj_list_marker _______________ 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.954 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.955 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.956 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.957 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.958 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.959 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_versioning_copy_obj_version ______________ 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.960 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.961 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.962 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.963 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.964 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.965 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_versioning_multi_object_delete _____________ 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.966 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.967 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.968 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.969 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.970 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.971 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_versioning_multi_object_delete_with_marker _______ 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.972 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.973 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.974 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.977 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.978 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout:___ ERROR at setup of test_versioning_multi_object_delete_with_marker_create ___ 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.979 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.980 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.981 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.982 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.983 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.984 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_versioned_object_acl __________________ 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.985 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.986 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.987 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.988 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.989 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.990 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_versioned_object_acl_no_version_specified _______ 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.991 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.992 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:56.993 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:56.994 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:56.995 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:56.996 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_versioned_concurrent_object_create_and_remove _____ 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:56.998 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:56.999 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.000 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.001 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.002 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.003 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout:_____________________ ERROR at setup of test_lifecycle_set _____________________ 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.004 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.005 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.006 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.007 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.008 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.009 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout:_____________________ ERROR at setup of test_lifecycle_get _____________________ 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.010 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.011 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.013 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.014 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.015 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.015 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.015 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.015 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.015 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.015 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.015 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.015 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.015 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.015 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.015 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.015 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.017 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.017 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.017 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.017 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.017 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.017 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.017 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.017 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_lifecycle_get_no_id __________________ 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.018 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.019 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.020 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.021 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.022 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.023 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_lifecycle_id_too_long _________________ 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.024 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.025 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.026 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.027 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.028 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.029 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout:___________________ ERROR at setup of test_lifecycle_same_id ___________________ 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.030 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.031 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.032 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.033 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.034 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.035 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.036 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_lifecycle_invalid_status ________________ 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.037 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.038 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.039 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.040 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.041 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.042 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_lifecycle_set_date ___________________ 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.043 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.044 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.046 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.047 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.048 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_lifecycle_set_invalid_date _______________ 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.049 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.050 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.051 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.052 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.053 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.054 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_lifecycle_set_noncurrent ________________ 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.055 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.056 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.058 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.058 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.058 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.058 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.058 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.058 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.058 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.058 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.059 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.060 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_lifecycle_set_deletemarker _______________ 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.062 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.063 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.064 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.065 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.066 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.067 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_lifecycle_set_filter __________________ 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.068 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.069 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.070 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.071 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.072 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.073 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_lifecycle_set_empty_filter _______________ 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.074 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.075 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.077 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.078 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.079 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_lifecycle_set_multipart ________________ 2024-03-09T15:35:57.080 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.081 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.082 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.083 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.084 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.085 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.086 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_lifecycle_transition_set_invalid_date _________ 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.087 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.088 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.089 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.090 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.091 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.092 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_encrypted_transfer_1b _________________ 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.093 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.094 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.096 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.097 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.098 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_encrypted_transfer_1kb _________________ 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.099 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.100 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.101 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.102 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.103 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.104 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_encrypted_transfer_1MB _________________ 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.105 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.106 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.107 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.108 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.109 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.110 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.111 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_encrypted_transfer_13b _________________ 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.112 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.113 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.114 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.115 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.116 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.117 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_encryption_sse_c_method_head ______________ 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.118 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.119 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.122 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.123 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_encryption_sse_c_present ________________ 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.124 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.125 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.126 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.127 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.128 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.129 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_encryption_sse_c_other_key _______________ 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.130 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.131 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.132 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.133 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.134 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.135 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_encryption_sse_c_invalid_md5 ______________ 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.136 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.137 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.138 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.138 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.138 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.138 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.138 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.138 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.138 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.138 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.138 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.138 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.138 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.138 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.139 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.140 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.141 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.142 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_encryption_sse_c_no_md5 ________________ 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.143 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.144 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.145 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.146 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.147 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.148 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_encryption_sse_c_no_key ________________ 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.149 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.150 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.151 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.152 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.153 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.154 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_encryption_key_no_sse_c ________________ 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.155 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.156 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.157 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.158 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.159 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.160 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_encryption_sse_c_multipart_upload ___________ 2024-03-09T15:35:57.161 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.162 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.163 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.164 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.165 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.166 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.167 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_encryption_sse_c_unaligned_multipart_upload ______ 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.168 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.169 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.170 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.171 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.172 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.173 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_encryption_sse_c_multipart_bad_download ________ 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.174 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.175 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.177 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.178 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.179 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout:__ ERROR at setup of test_encryption_sse_c_post_object_authenticated_request ___ 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.180 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.181 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.182 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.183 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.184 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.185 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_sse_kms_method_head __________________ 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.186 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.187 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.188 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.189 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.190 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.191 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.192 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_sse_kms_present ____________________ 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.193 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.194 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.195 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.196 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.197 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.198 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_sse_kms_no_key _____________________ 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.199 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.200 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.201 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.202 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.203 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.204 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_sse_kms_not_declared __________________ 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.205 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.206 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.206 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.206 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.206 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.206 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.206 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.206 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.206 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.206 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.206 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.206 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.207 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.208 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.209 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.210 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_sse_kms_multipart_upload ________________ 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.211 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.212 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.213 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.214 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.215 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.216 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_sse_kms_multipart_invalid_chunks_1 ___________ 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.217 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.218 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.219 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.219 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.219 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.219 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.219 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.219 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.219 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.219 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.219 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.219 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.221 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.221 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.221 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.221 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.221 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.221 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.221 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.221 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.222 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.223 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.224 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_sse_kms_multipart_invalid_chunks_2 ___________ 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.225 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.226 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.227 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.228 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.229 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.230 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_sse_kms_post_object_authenticated_request _______ 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.231 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.232 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.233 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.235 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.236 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_sse_kms_transfer_1b __________________ 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.237 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.238 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.239 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.240 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.241 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.242 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_sse_kms_transfer_1kb __________________ 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.243 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.244 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.249 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_sse_kms_transfer_1MB __________________ 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.250 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.251 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.252 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.253 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.254 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.255 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_sse_kms_transfer_13b __________________ 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.256 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.257 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.258 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.259 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.260 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.261 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_sse_kms_read_declare __________________ 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.262 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.266 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout:_____________________ ERROR at setup of test_bucket_policy _____________________ 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.268 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.269 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.270 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.271 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.272 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.273 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_bucketv2_policy ____________________ 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.274 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.275 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.277 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.278 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.279 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout:___________________ ERROR at setup of test_bucket_policy_acl ___________________ 2024-03-09T15:35:57.280 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.281 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.282 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.283 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.284 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.285 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.286 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_bucketv2_policy_acl __________________ 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.287 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.288 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.289 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.290 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.290 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.290 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.290 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.290 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.290 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.290 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.290 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.290 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.291 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.292 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.292 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.292 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.292 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.292 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.292 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.292 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.292 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.292 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.292 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.292 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.293 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.294 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.294 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_bucket_policy_another_bucket ______________ 2024-03-09T15:35:57.294 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.294 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.294 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.294 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.294 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.294 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.294 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.294 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.295 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.295 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.295 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.295 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.295 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.295 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.295 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.295 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.295 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.295 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.295 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.295 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.296 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.296 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.296 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.296 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.296 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.296 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.296 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.296 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.296 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.296 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.296 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.296 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.297 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.298 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.298 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.298 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.298 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.298 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.298 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.298 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.298 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.298 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.298 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.298 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.298 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.299 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.300 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.300 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.300 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.300 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.300 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.300 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.300 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.300 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.300 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.300 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.300 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.301 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.301 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.301 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.301 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.301 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.301 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.301 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.301 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.301 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_bucketv2_policy_another_bucket _____________ 2024-03-09T15:35:57.301 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.301 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.301 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.302 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.302 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.302 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.302 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.302 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.302 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.302 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.302 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.302 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.302 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.302 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.302 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.303 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.303 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.303 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.303 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.303 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.303 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.303 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.303 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.303 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.303 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.303 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.303 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.305 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.306 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.306 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.306 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.306 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.306 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.306 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.306 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.306 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.306 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.306 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.306 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.306 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.308 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.309 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_get_obj_tagging ____________________ 2024-03-09T15:35:57.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.309 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.309 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.309 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.309 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.309 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.309 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.309 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.310 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.310 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.310 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.310 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.310 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.310 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.310 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.310 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.310 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.310 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.310 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.310 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.311 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.312 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.312 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.312 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.312 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.312 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.312 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.312 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.312 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.312 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.312 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.312 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.313 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.313 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.313 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.313 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.313 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.313 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.313 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.313 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.313 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.313 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.313 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.313 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.314 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.316 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.316 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.316 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.316 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.316 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.316 INFO:teuthology.orchestra.run.smithi001.stdout:_________________ ERROR at setup of test_get_obj_head_tagging __________________ 2024-03-09T15:35:57.316 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.316 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.316 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.316 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.316 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.316 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.317 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.317 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.317 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.317 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.317 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.317 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.317 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.317 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.317 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.317 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.317 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.318 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.318 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.318 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.318 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.318 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.318 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.318 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.318 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.318 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.318 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.318 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.318 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.319 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.319 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.319 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.319 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.319 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.319 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.319 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.319 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.319 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.321 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.321 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.321 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.321 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.321 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.321 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.321 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.321 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.321 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.321 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.321 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.321 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.322 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.322 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.322 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.322 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.322 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.322 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.322 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.322 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.322 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.322 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.322 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.322 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.323 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.323 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.323 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.323 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.323 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.323 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.323 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.323 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.323 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.323 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.323 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.323 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.324 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.324 INFO:teuthology.orchestra.run.smithi001.stdout:_____________________ ERROR at setup of test_put_max_tags ______________________ 2024-03-09T15:35:57.324 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.324 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.324 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.324 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.324 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.324 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.324 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.324 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.324 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.324 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.325 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.326 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.326 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.326 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.326 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.326 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.326 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.326 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.326 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.326 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.326 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.326 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.327 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.327 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.327 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.327 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.327 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.327 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.327 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.327 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.327 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.327 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.327 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.328 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.329 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.329 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.329 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.329 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.329 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.329 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.329 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.329 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.329 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.329 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.329 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.329 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.330 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.330 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.330 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.330 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.330 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.330 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.330 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.330 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.330 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.330 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.330 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.330 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.331 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.331 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.331 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.331 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.331 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.331 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_put_excess_tags ____________________ 2024-03-09T15:35:57.331 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.331 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.331 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.331 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.331 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.331 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.332 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.332 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.332 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.332 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.332 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.332 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.332 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.332 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.332 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.332 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.332 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.332 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.333 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.333 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.333 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.333 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.333 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.333 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.333 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.333 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.333 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.333 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.333 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.333 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.334 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.334 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.334 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.334 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.334 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.334 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.334 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.334 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.334 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.335 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.335 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.335 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.335 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.335 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.335 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.335 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.335 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.335 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.335 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.335 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.336 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.336 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.336 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.336 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.336 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.336 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.336 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.336 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.336 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.336 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.336 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.337 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.337 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.337 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.337 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.337 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.337 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.337 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.337 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.337 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.337 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.337 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.337 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.338 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.338 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.338 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.338 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.338 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.338 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.338 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.338 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.338 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.338 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.338 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_put_max_kvsize_tags __________________ 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.339 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.340 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.341 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.341 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.341 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.341 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.341 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.341 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.341 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.341 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.341 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.341 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.341 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.343 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.344 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.344 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.344 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.344 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.344 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.344 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.344 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.344 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.344 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.344 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.344 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.344 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.345 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.346 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.346 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.346 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.346 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.346 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.346 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.346 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_put_excess_key_tags __________________ 2024-03-09T15:35:57.346 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.346 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.346 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.346 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.346 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.347 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.347 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.347 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.347 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.347 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.347 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.347 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.347 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.348 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.348 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.348 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.348 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.348 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.348 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.348 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.348 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.348 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.348 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.348 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.348 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.349 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.350 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.351 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.351 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.351 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.351 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.351 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.351 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.351 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.351 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.351 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.351 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.351 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.352 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.352 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.352 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.352 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.352 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.352 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.352 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.352 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.352 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.352 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.352 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.353 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_put_excess_val_tags __________________ 2024-03-09T15:35:57.354 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.354 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.354 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.354 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.354 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.354 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.354 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.354 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.354 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.354 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.354 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.354 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.355 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.355 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.355 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.355 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.355 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.355 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.355 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.355 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.355 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.355 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.356 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.356 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.356 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.356 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.356 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.356 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.356 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.356 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.356 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.356 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.356 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.357 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.358 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.358 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.358 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.358 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.358 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.358 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.358 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.358 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.358 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.358 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.359 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.360 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.360 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.360 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.360 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.360 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.360 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.360 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.360 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.360 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.360 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.360 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.360 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.361 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.361 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.361 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.361 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.361 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.361 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_put_modify_tags ____________________ 2024-03-09T15:35:57.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.361 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.361 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.361 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.362 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.362 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.362 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.362 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.362 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.362 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.362 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.362 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.362 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.362 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.362 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.362 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.363 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.363 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.363 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.363 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.363 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.363 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.363 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.363 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.363 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.363 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.363 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.363 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.364 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.365 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.365 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.365 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.365 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.365 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.365 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.365 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.365 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.365 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.365 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.365 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.366 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.366 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.366 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.366 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.366 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.366 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.366 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.366 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.366 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.366 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.366 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.366 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.367 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.367 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.367 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.367 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.367 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.367 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.367 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.367 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.367 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.367 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.367 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.367 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.368 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.368 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.368 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.368 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.368 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.368 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.368 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.368 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.368 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.368 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.368 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ ERROR at setup of test_put_delete_tags ____________________ 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.369 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.370 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.370 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.370 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.370 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.370 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.370 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.370 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.370 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.370 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.370 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.370 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.370 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.371 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.371 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.371 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.371 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.371 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.371 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.371 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.371 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.371 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.371 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.372 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.372 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.372 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.372 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.372 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.372 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.372 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.372 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.372 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.372 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.373 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.373 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.373 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.373 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.373 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.373 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.373 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.373 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.373 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.373 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.373 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.373 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.374 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.374 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.374 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.374 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.374 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.374 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.374 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.374 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.374 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.374 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.374 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.374 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.375 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.375 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.375 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.375 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.375 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.375 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.375 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.375 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.375 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.375 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.375 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.376 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.376 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.376 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.376 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.376 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.376 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.376 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.376 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_post_object_tags_anonymous_request ___________ 2024-03-09T15:35:57.376 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.376 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.376 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.376 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.377 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.377 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.377 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.377 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.377 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.377 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.377 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.377 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.377 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.377 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.377 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.377 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.378 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.378 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.378 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.378 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.378 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.378 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.378 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.378 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.378 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.378 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.378 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.378 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.379 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.380 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.380 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.380 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.380 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.380 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.380 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.380 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.380 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.380 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.380 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.380 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.380 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.381 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.382 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.382 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.382 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.382 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.382 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.382 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.382 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.382 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.382 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.382 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.382 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.383 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_post_object_tags_authenticated_request _________ 2024-03-09T15:35:57.384 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.384 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.384 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.384 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.384 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.384 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.384 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.384 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.384 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.385 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.385 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.385 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.385 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.385 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.385 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.385 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.385 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.385 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.385 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.385 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.385 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.386 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.386 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.386 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.386 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.386 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.386 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.386 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.386 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.386 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.386 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.386 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.387 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.388 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.388 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.388 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.388 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.388 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.388 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.388 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.388 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.388 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.388 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.388 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.388 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.389 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.389 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.389 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.389 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.389 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.389 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.389 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.389 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.389 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.389 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.389 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.389 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.390 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.390 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.390 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.390 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.390 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.390 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.390 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.390 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.390 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.390 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.390 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.390 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.391 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.391 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.391 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.391 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.391 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.391 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.391 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.391 INFO:teuthology.orchestra.run.smithi001.stdout:___________________ ERROR at setup of test_put_obj_with_tags ___________________ 2024-03-09T15:35:57.391 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.391 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.391 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.391 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.392 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.393 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.393 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.393 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.393 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.393 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.393 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.393 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.393 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.393 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.393 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.393 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.393 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.398 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.398 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.398 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.398 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.398 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.398 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.398 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.398 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.398 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.398 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.398 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_get_tags_acl_public __________________ 2024-03-09T15:35:57.398 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.399 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.399 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.399 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.399 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.399 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.399 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.399 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.399 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.399 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.399 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.399 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.399 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.400 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.400 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.400 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.400 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.400 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.400 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.400 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.400 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.400 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.400 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.400 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.400 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.401 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.401 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.401 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.401 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.401 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.401 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.401 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.401 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.401 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.401 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.402 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.402 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.402 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.402 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.402 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.402 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.402 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.402 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.402 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.402 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.403 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.403 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.403 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.403 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.403 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.403 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.403 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.403 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.403 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.403 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.403 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.403 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.404 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.404 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.404 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.404 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.404 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.404 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.404 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.404 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.404 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.404 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.404 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.404 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.406 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.406 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.406 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.406 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.406 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.406 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_put_tags_acl_public __________________ 2024-03-09T15:35:57.406 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.406 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.406 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.406 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.406 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.406 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.407 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.407 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.407 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.407 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.407 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.407 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.407 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.407 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.407 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.407 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.407 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.407 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.408 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.410 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.410 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.410 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.410 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.410 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.410 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.410 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.410 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.410 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.410 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.410 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.411 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.411 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.411 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.411 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.411 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.411 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.411 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.411 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.411 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.411 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.411 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.411 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.412 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.412 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.412 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.412 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.412 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.412 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.412 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.412 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.412 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.412 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.412 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.413 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.413 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.413 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.413 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.413 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.413 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.413 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.413 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.413 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.413 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.413 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.413 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.414 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.414 INFO:teuthology.orchestra.run.smithi001.stdout:________________ ERROR at setup of test_delete_tags_obj_public _________________ 2024-03-09T15:35:57.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.414 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.414 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.414 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.414 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.414 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.414 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.414 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.415 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.415 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.415 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.415 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.415 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.415 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.415 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.415 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.415 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.415 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.415 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.415 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.416 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.416 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.416 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.416 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.416 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.416 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.416 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.416 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.416 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.416 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.417 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.417 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.417 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.417 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.417 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.417 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.417 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.417 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.417 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.417 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.418 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.418 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.418 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.418 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.418 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.418 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.418 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.418 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.418 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.418 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.419 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.420 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.420 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.420 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.420 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.420 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.420 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.420 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.420 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.420 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.420 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.420 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.420 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout:___ ERROR at setup of test_versioning_bucket_atomic_upload_return_version_id ___ 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.421 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.422 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.422 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.422 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.422 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.422 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.422 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.422 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.422 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.422 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.422 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.422 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.422 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.423 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.423 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.423 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.423 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.423 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.423 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.423 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.423 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.423 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.423 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.423 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.423 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.424 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.425 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.425 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.425 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.425 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.425 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.425 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.426 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.426 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.426 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.426 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.426 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.426 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.426 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.426 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.426 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.426 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.426 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.427 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.427 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.427 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.427 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.427 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.427 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.427 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.427 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.427 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.427 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.428 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.428 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.428 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.428 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.428 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.428 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.428 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.428 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.428 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.428 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.428 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.428 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.429 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.429 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.429 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.429 INFO:teuthology.orchestra.run.smithi001.stdout:_ ERROR at setup of test_versioning_bucket_multipart_upload_return_version_id __ 2024-03-09T15:35:57.429 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.429 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.429 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.429 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.429 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.429 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.429 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.429 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.430 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.430 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.430 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.430 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.430 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.430 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.430 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.430 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.430 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.430 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.430 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.430 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.431 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.431 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.431 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.431 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.431 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.431 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.431 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.431 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.431 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.431 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.431 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.432 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.433 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.433 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.433 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.433 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.433 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.433 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.433 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.433 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.433 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.433 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.434 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.434 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.434 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.434 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.434 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.434 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.434 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.434 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.434 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.434 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.434 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.435 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.435 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.435 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.435 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.435 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.435 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.435 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.435 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.435 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.435 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.435 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.435 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.436 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.436 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.436 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.436 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.436 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.436 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.436 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.436 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.436 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.436 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.436 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.436 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.437 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_bucket_policy_get_obj_existing_tag ___________ 2024-03-09T15:35:57.437 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.437 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.437 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.437 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.437 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.437 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.437 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.437 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.437 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.437 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.437 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.438 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.438 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.438 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.438 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.438 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.438 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.439 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.439 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.439 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.439 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.439 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.439 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.439 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.439 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.439 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.439 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.444 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.444 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.444 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.444 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_bucket_policy_get_obj_tagging_existing_tag _______ 2024-03-09T15:35:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.444 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.445 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.445 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.445 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.445 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.445 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.445 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.445 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.446 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.446 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.446 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.446 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.446 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.446 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.451 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.451 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.452 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.452 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_bucket_policy_put_obj_tagging_existing_tag _______ 2024-03-09T15:35:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.452 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.452 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.452 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.452 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.453 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.453 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.453 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.453 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.453 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.453 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.453 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.453 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.453 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.453 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.453 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.453 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.454 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.454 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.454 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.454 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.454 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.454 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.454 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.454 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.454 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.454 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.454 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.455 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.456 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.456 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.456 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.456 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.456 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.456 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.456 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.456 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.456 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.456 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.456 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.456 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.457 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.457 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.457 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.457 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.457 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.457 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.457 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.457 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.457 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.457 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.457 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.457 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.458 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.459 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.459 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.459 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.459 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.459 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.459 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.459 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_bucket_policy_put_obj_copy_source ___________ 2024-03-09T15:35:57.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.459 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.459 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.459 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.460 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.461 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.461 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.461 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.461 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.461 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.461 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.466 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.466 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.467 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.467 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_bucket_policy_put_obj_copy_source_meta _________ 2024-03-09T15:35:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.467 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.467 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.467 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.467 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.468 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.468 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.468 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.468 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.468 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.468 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.469 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.469 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.469 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.469 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.474 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.474 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.474 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.474 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_bucket_policy_put_obj_acl _______________ 2024-03-09T15:35:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.474 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.476 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.482 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.482 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_bucket_policy_put_obj_grant ______________ 2024-03-09T15:35:57.482 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.482 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.482 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.482 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.482 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.482 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.482 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.482 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.483 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.483 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.483 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.483 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.483 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.483 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.483 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.483 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.483 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.483 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.483 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.483 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.484 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.484 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.484 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.484 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.489 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.489 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.490 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.490 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_put_obj_enc_conflict_c_s3 _______________ 2024-03-09T15:35:57.490 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.490 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.490 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.490 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.490 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.490 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.490 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.490 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.490 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.491 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.492 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.492 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.492 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.497 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.497 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.497 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.497 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_put_obj_enc_conflict_c_kms _______________ 2024-03-09T15:35:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.498 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.498 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.498 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.498 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.498 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.498 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.498 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.498 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.498 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.498 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.498 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.498 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.499 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.499 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.499 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.499 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.499 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.499 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.500 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.500 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.505 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.505 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.505 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.505 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.505 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.505 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.505 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_put_obj_enc_conflict_s3_kms ______________ 2024-03-09T15:35:57.505 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.505 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.505 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.505 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.506 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.506 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.506 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.506 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.506 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.506 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.506 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.506 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.506 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.506 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.506 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.506 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.507 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.507 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.507 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.507 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.507 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.507 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.512 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.512 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout:___________ ERROR at setup of test_put_obj_enc_conflict_bad_enc_kms ____________ 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.513 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.514 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.514 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.514 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.514 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.514 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.514 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.514 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.514 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.514 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.514 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.514 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.514 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.515 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.515 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.515 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.520 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.520 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.520 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.520 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_bucket_policy_put_obj_kms_noenc ____________ 2024-03-09T15:35:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.520 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.521 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.521 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.521 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.521 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.521 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.521 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.521 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.522 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.522 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.522 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.522 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.522 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.522 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.527 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.527 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.527 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.528 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_bucket_policy_put_obj_kms_s3 ______________ 2024-03-09T15:35:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.528 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.528 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.528 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.528 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.528 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.529 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.530 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.530 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_bucket_policy_get_obj_acl_existing_tag _________ 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.536 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.536 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.536 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.536 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.536 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.536 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.536 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.537 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.537 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.537 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.537 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.537 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.542 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.542 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.542 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.543 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_object_lock_put_obj_lock ________________ 2024-03-09T15:35:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.543 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.543 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.543 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.543 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.543 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.544 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.545 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.545 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.550 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.550 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.550 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.550 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_object_lock_put_obj_lock_invalid_bucket ________ 2024-03-09T15:35:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.550 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.551 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.551 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.551 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.551 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.551 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.551 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.551 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.557 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.557 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.557 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.557 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_object_lock_put_obj_lock_with_days_and_years ______ 2024-03-09T15:35:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.558 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.559 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.559 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.559 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.559 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.559 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.559 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.559 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.564 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.564 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.565 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.565 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_object_lock_put_obj_lock_invalid_days _________ 2024-03-09T15:35:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.565 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.565 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.565 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.565 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.565 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.566 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.566 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.566 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.566 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.566 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.566 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.567 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.567 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.567 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_object_lock_put_obj_lock_invalid_years _________ 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.573 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.574 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.574 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.574 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.574 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.579 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.579 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.579 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.579 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_object_lock_put_obj_lock_invalid_mode _________ 2024-03-09T15:35:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.579 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.580 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.580 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.580 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.580 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.580 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.580 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.586 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.586 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.587 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.587 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_object_lock_put_obj_lock_invalid_status ________ 2024-03-09T15:35:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.587 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.587 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.587 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.587 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.588 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.589 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.589 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.589 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.594 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.594 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.594 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.594 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_object_lock_suspend_versioning _____________ 2024-03-09T15:35:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.594 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.594 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.595 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.595 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.595 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.595 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.595 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.595 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.596 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.596 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.596 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.596 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.596 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.596 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.601 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.601 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.601 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.601 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_object_lock_get_obj_lock ________________ 2024-03-09T15:35:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.602 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.602 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.602 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.602 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.602 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.602 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.603 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.604 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.608 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.608 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_object_lock_get_obj_lock_invalid_bucket ________ 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.614 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.614 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.614 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.614 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.614 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.614 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.614 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.614 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.614 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.614 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.614 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.615 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.615 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.615 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.615 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.615 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.615 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.615 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.615 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.615 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.615 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.615 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.615 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.616 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.616 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.616 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.616 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.616 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_object_lock_put_obj_retention _____________ 2024-03-09T15:35:57.616 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.617 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.617 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.617 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.617 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.617 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.617 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.617 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.617 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.617 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.617 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.617 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.617 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.618 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.618 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.618 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.618 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.618 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.618 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.618 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.618 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.618 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.618 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.618 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.619 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.619 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.619 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.619 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.619 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.619 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.619 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.619 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.619 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.621 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.621 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.621 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.621 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.621 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.621 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.621 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.621 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.621 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.621 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.621 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.621 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.622 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.623 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.623 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.623 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.623 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.623 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.623 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.623 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.623 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.623 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.623 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.623 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.623 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_object_lock_put_obj_retention_invalid_bucket ______ 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.624 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.625 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.625 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.625 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.625 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.625 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.625 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.625 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.625 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.625 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.625 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.625 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.625 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.626 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.626 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.626 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.626 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.626 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.626 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.626 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.626 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.627 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.627 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.627 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.627 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.627 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.627 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.629 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.629 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.629 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.629 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.629 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.629 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.631 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout:______ ERROR at setup of test_object_lock_put_obj_retention_invalid_mode _______ 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.633 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.634 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.635 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.636 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.637 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.638 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_object_lock_get_obj_retention _____________ 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.639 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.640 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.641 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.642 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.642 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.642 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.642 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.642 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.642 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.642 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.642 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.642 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.642 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.642 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.644 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout:_________ ERROR at setup of test_object_lock_get_obj_retention_iso8601 _________ 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.645 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.646 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.647 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.648 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.649 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.650 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.651 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_object_lock_get_obj_retention_invalid_bucket ______ 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.652 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.653 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.654 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.655 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.656 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.657 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_object_lock_put_obj_retention_versionid ________ 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.658 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.659 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.660 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.661 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.662 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.663 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout:_ ERROR at setup of test_object_lock_put_obj_retention_override_default_retention _ 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.664 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.665 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.666 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.667 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.668 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.669 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.670 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_object_lock_put_obj_retention_increase_period _____ 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.671 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.672 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_object_lock_put_obj_retention_shorten_period ______ 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout:__ ERROR at setup of test_object_lock_put_obj_retention_shorten_period_bypass __ 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.684 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.685 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.686 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.687 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.688 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.689 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_object_lock_delete_object_with_retention ________ 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.690 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.691 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.692 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.694 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout:__ ERROR at setup of test_object_lock_delete_object_with_retention_and_marker __ 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.696 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.697 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.698 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.700 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.701 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout:____ ERROR at setup of test_object_lock_multi_delete_object_with_retention _____ 2024-03-09T15:35:57.702 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.703 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.704 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.705 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.706 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.707 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.708 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_object_lock_put_legal_hold _______________ 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.709 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.710 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.711 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.713 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.714 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_object_lock_put_legal_hold_invalid_bucket _______ 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.715 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.716 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.717 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.718 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.719 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.720 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.721 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_object_lock_put_legal_hold_invalid_status _______ 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_object_lock_get_legal_hold _______________ 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.728 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.729 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_object_lock_get_legal_hold_invalid_bucket _______ 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.735 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.740 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_object_lock_delete_object_with_legal_hold_on ______ 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.746 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_object_lock_delete_object_with_legal_hold_off _____ 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.747 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.748 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.750 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.751 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.752 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_object_lock_get_obj_metadata ______________ 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.753 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.754 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.755 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.758 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_object_lock_uploading_obj _______________ 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.759 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.760 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.761 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.762 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.763 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.764 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.765 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout:_ ERROR at setup of test_object_lock_changing_mode_from_governance_with_bypass _ 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.766 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.767 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.768 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.769 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.771 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout:_ ERROR at setup of test_object_lock_changing_mode_from_governance_without_bypass _ 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.772 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.773 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.774 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.775 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.776 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.777 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_object_lock_changing_mode_from_compliance _______ 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.778 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.779 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.779 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.779 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.779 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.779 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.779 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.779 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.779 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.779 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.779 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.779 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.780 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.781 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.783 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.784 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_copy_object_ifmatch_good ________________ 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.785 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.786 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.787 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.788 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.789 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.790 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_copy_object_ifnonematch_failed _____________ 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.792 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.793 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.794 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.795 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.796 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.797 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_get_bucket_policy_status ________________ 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.798 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.799 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.800 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.801 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.802 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.803 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.804 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.804 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.804 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.804 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.804 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.804 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.804 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_get_public_acl_bucket_policy_status __________ 2024-03-09T15:35:57.804 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.804 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.804 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.805 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.806 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.807 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.807 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.807 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.807 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.807 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.807 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.807 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.807 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.807 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.808 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.808 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.808 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.808 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.808 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.808 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.808 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.808 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.808 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.808 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.808 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.809 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.810 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.810 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.810 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.810 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.810 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.810 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.810 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.810 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.810 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.810 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.811 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout:________ ERROR at setup of test_get_authpublic_acl_bucket_policy_status ________ 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.812 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.813 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.814 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.815 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.816 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.817 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_get_publicpolicy_acl_bucket_policy_status _______ 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.818 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.819 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.820 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.821 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.822 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.823 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.824 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_get_nonpublicpolicy_acl_bucket_policy_status ______ 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.825 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.826 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.827 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.828 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.829 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout:_____ ERROR at setup of test_get_nonpublicpolicy_deny_bucket_policy_status _____ 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.831 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.832 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.833 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.834 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.835 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.836 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.837 INFO:teuthology.orchestra.run.smithi001.stdout:______________ ERROR at setup of test_get_undefined_public_block _______________ 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.838 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.839 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.840 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.841 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.842 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.843 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout:__________ ERROR at setup of test_get_public_block_deny_bucket_policy __________ 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.844 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.845 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.846 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.847 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.848 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.849 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout:___________________ ERROR at setup of test_put_public_block ____________________ 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.851 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.852 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.853 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.854 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.855 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_block_public_put_bucket_acls ______________ 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.856 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.857 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.858 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.859 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.860 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.861 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.862 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout:____________ ERROR at setup of test_block_public_object_canned_acls ____________ 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.863 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.864 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.869 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.869 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.869 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.869 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_block_public_policy __________________ 2024-03-09T15:35:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.869 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.870 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.875 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_ignore_public_acls ___________________ 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.877 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.877 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.877 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.877 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.877 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.877 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.877 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout:_______ ERROR at setup of test_multipart_upload_on_a_bucket_with_policy ________ 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_put_bucket_encryption_kms _______________ 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.894 INFO:teuthology.orchestra.run.smithi001.stdout:_______________ ERROR at setup of test_get_bucket_encryption_kms _______________ 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.900 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ ERROR at setup of test_delete_bucket_encryption_kms ______________ 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.901 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.902 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout:___ ERROR at setup of test_sse_kms_default_post_object_authenticated_request ___ 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.907 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.908 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ ERROR at setup of test_get_object_torrent ___________________ 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.913 INFO:teuthology.orchestra.run.smithi001.stdout:configfile = 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.fixture(autouse=True) 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: def setup_teardown(configfile): 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout:> setup() 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:331: 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:295: in setup 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_prefixed_buckets(prefix=prefix) 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: raise err 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: nuke_bucket(client, bucket_name) 2024-03-09T15:35:57.914 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:103: in nuke_bucket 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: for objects in list_versions(client, bucket, batch_size): 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/__init__.py:88: in list_versions 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: listing = client.list_object_versions(**kwargs) 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:514: in _api_call 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: return self._make_api_call(operation_name, kwargs) 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout:self = 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout:operation_name = 'ListObjectVersions' 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout:api_params = {'Bucket': 'test-client.0-8p45c4thhbi3i01-156', 'EncodingType': 'url', 'MaxKeys': 128} 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: def _make_api_call(self, operation_name, api_params): 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model = self._service_model.operation_model(operation_name) 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: service_name = self._service_model.service_name 2024-03-09T15:35:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: history_recorder.record( 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: 'API_CALL', 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: 'service': service_name, 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: 'operation': operation_name, 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: 'params': api_params, 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: if operation_model.deprecated: 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: logger.debug( 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: 'Warning: %s.%s() is deprecated', service_name, operation_name 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: request_context = { 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_region': self.meta.region_name, 2024-03-09T15:35:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: 'client_config': self.meta.config, 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: 'has_streaming_input': operation_model.has_streaming_input, 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: 'auth_type': operation_model.auth_type, 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: request_dict = self._convert_to_request_dict( 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: api_params, operation_model, context=request_context 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: resolve_checksum_context(request_dict, operation_model, api_params) 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: service_id = self._service_model.service_id.hyphenize() 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: handler, event_response = self.meta.events.emit_until_response( 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: 'before-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: params=request_dict, 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: request_signer=self._request_signer, 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: if event_response is not None: 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = event_response 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: apply_request_checksum(request_dict) 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: http, parsed_response = self._make_request( 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: operation_model, request_dict, request_context 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: self.meta.events.emit( 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: 'after-call.{service_id}.{operation_name}'.format( 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: service_id=service_id, operation_name=operation_name 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: ), 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: http_response=http, 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: parsed=parsed_response, 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: model=operation_model, 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: context=request_context, 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: ) 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: if http.status_code >= 300: 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: error_code = parsed_response.get("Error", {}).get("Code") 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: error_class = self.exceptions.from_code(error_code) 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout:> raise error_class(parsed_response, operation_name) 2024-03-09T15:35:57.919 INFO:teuthology.orchestra.run.smithi001.stdout:E botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: None 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/botocore/client.py:938: NoSuchBucket 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout:=================================== FAILURES =================================== 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ test_object_create_bad_contentlength_none ___________________ 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.auth_common 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.fails_on_dbstore 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: def test_object_create_bad_contentlength_none(): 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: key = _setup_bad_object(remove=('Content-Length',)) 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout:> e = assert_raises(boto.exception.S3ResponseError, key.set_contents_from_string, 'bar') 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:180: 2024-03-09T15:35:57.920 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout:excClass = 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout:callableObj = > 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout:args = ('bar',), kwargs = {}, excName = 'S3ResponseError' 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: def assert_raises(excClass, callableObj, *args, **kwargs): 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: Like unittest.TestCase.assertRaises, but returns the exception. 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: try: 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: callableObj(*args, **kwargs) 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: except excClass as e: 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: return e 2024-03-09T15:35:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: if hasattr(excClass, '__name__'): 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: excName = excClass.__name__ 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: excName = str(excClass) 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout:> raise AssertionError("%s not raised" % excName) 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout:E AssertionError: S3ResponseError not raised 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/utils.py:19: AssertionError 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout:---------------------------- Captured stdout setup ----------------------------- 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on main 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection main prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on alt 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection alt prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.922 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on tenant 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection tenant prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:Done with cleanup of test buckets. 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:--------------------------- Captured stdout teardown --------------------------- 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on main 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection main prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:prefix= test-client.0-jjbc677ewdfcmzd- 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket key 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on alt 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection alt prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on tenant 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection tenant prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:Done with cleanup of test buckets. 2024-03-09T15:35:57.923 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ test_object_create_bad_authorization_empty __________________ 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.auth_common 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.fails_on_dbstore 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: def test_object_create_bad_authorization_empty(): 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: key = _setup_bad_object({'Authorization': ''}) 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout:> e = assert_raises(boto.exception.S3ResponseError, key.set_contents_from_string, 'bar') 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:210: 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout:excClass = 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout:callableObj = > 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout:args = ('bar',), kwargs = {}, excName = 'S3ResponseError' 2024-03-09T15:35:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: def assert_raises(excClass, callableObj, *args, **kwargs): 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: Like unittest.TestCase.assertRaises, but returns the exception. 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: try: 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: callableObj(*args, **kwargs) 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: except excClass as e: 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: return e 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: if hasattr(excClass, '__name__'): 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: excName = excClass.__name__ 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: excName = str(excClass) 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout:> raise AssertionError("%s not raised" % excName) 2024-03-09T15:35:57.925 INFO:teuthology.orchestra.run.smithi001.stdout:E AssertionError: S3ResponseError not raised 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/utils.py:19: AssertionError 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:--------------------------- Captured stdout teardown --------------------------- 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on main 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection main prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:prefix= test-client.0-jjbc677ewdfcmzd- 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket key 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on alt 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection alt prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on tenant 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection tenant prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:Done with cleanup of test buckets. 2024-03-09T15:35:57.926 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ test_object_create_bad_authorization_none ___________________ 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.auth_common 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.fails_on_dbstore 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: def test_object_create_bad_authorization_none(): 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: key = _setup_bad_object(remove=('Authorization',)) 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout:> e = assert_raises(boto.exception.S3ResponseError, key.set_contents_from_string, 'bar') 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:236: 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout:excClass = 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout:callableObj = > 2024-03-09T15:35:57.927 INFO:teuthology.orchestra.run.smithi001.stdout:args = ('bar',), kwargs = {}, excName = 'S3ResponseError' 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: def assert_raises(excClass, callableObj, *args, **kwargs): 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: Like unittest.TestCase.assertRaises, but returns the exception. 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: try: 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: callableObj(*args, **kwargs) 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: except excClass as e: 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: return e 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: if hasattr(excClass, '__name__'): 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: excName = excClass.__name__ 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: excName = str(excClass) 2024-03-09T15:35:57.928 INFO:teuthology.orchestra.run.smithi001.stdout:> raise AssertionError("%s not raised" % excName) 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:E AssertionError: S3ResponseError not raised 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/utils.py:19: AssertionError 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:--------------------------- Captured stdout teardown --------------------------- 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on main 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection main prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:prefix= test-client.0-jjbc677ewdfcmzd- 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket key 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on alt 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection alt prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on tenant 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection tenant prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:Done with cleanup of test buckets. 2024-03-09T15:35:57.929 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ test_bucket_create_bad_authorization_empty __________________ 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.auth_common 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.fails_on_dbstore 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: def test_bucket_create_bad_authorization_empty(): 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: _add_custom_headers({'Authorization': ''}) 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout:> e = assert_raises(boto.exception.S3ResponseError, get_new_bucket) 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:294: 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout:excClass = 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout:callableObj = , args = () 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout:kwargs = {}, excName = 'S3ResponseError' 2024-03-09T15:35:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: def assert_raises(excClass, callableObj, *args, **kwargs): 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: Like unittest.TestCase.assertRaises, but returns the exception. 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: try: 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: callableObj(*args, **kwargs) 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: except excClass as e: 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: return e 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: if hasattr(excClass, '__name__'): 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: excName = excClass.__name__ 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: excName = str(excClass) 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:> raise AssertionError("%s not raised" % excName) 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:E AssertionError: S3ResponseError not raised 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/utils.py:19: AssertionError 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:--------------------------- Captured stdout teardown --------------------------- 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on main 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection main prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:prefix= test-client.0-jjbc677ewdfcmzd- 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on alt 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection alt prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on tenant 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection tenant prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.932 INFO:teuthology.orchestra.run.smithi001.stdout:Done with cleanup of test buckets. 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout:__________________ test_bucket_create_bad_authorization_none ___________________ 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.auth_common 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.fails_on_dbstore 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: def test_bucket_create_bad_authorization_none(): 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: _add_custom_headers(remove=('Authorization',)) 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout:> e = assert_raises(boto.exception.S3ResponseError, get_new_bucket) 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:305: 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout:excClass = 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout:callableObj = , args = () 2024-03-09T15:35:57.933 INFO:teuthology.orchestra.run.smithi001.stdout:kwargs = {}, excName = 'S3ResponseError' 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: def assert_raises(excClass, callableObj, *args, **kwargs): 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: Like unittest.TestCase.assertRaises, but returns the exception. 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: try: 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: callableObj(*args, **kwargs) 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: except excClass as e: 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: return e 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: if hasattr(excClass, '__name__'): 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: excName = excClass.__name__ 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: excName = str(excClass) 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:> raise AssertionError("%s not raised" % excName) 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:E AssertionError: S3ResponseError not raised 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/utils.py:19: AssertionError 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:--------------------------- Captured stdout teardown --------------------------- 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on main 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection main prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:prefix= test-client.0-jjbc677ewdfcmzd- 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on alt 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection alt prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on tenant 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection tenant prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.935 INFO:teuthology.orchestra.run.smithi001.stdout:Done with cleanup of test buckets. 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout:___________ test_object_create_bad_contentlength_mismatch_below_aws2 ___________ 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.auth_aws2 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.fails_on_dbstore 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: def test_object_create_bad_contentlength_mismatch_below_aws2(): 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: check_aws2_support() 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: content = 'bar' 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: length = len(content) - 1 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: key = _setup_bad_object({'Content-Length': length}) 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout:> e = assert_raises(boto.exception.S3ResponseError, key.set_contents_from_string, content) 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:321: 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.936 INFO:teuthology.orchestra.run.smithi001.stdout:excClass = 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout:callableObj = > 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout:args = ('bar',), kwargs = {}, excName = 'S3ResponseError' 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: def assert_raises(excClass, callableObj, *args, **kwargs): 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: Like unittest.TestCase.assertRaises, but returns the exception. 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: try: 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: callableObj(*args, **kwargs) 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: except excClass as e: 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: return e 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: if hasattr(excClass, '__name__'): 2024-03-09T15:35:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: excName = excClass.__name__ 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: excName = str(excClass) 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout:> raise AssertionError("%s not raised" % excName) 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout:E AssertionError: S3ResponseError not raised 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/utils.py:19: AssertionError 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout:--------------------------- Captured stdout teardown --------------------------- 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on main 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection main prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout:prefix= test-client.0-jjbc677ewdfcmzd- 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket key 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on alt 2024-03-09T15:35:57.938 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection alt prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on tenant 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection tenant prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout:Done with cleanup of test buckets. 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout:_____________ test_object_create_bad_authorization_incorrect_aws2 ______________ 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.auth_aws2 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.fails_on_dbstore 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: def test_object_create_bad_authorization_incorrect_aws2(): 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: check_aws2_support() 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: key = _setup_bad_object({'Authorization': 'AWS AKIAIGR7ZNNBHC5BKSUB:FWeDfwojDSdS2Ztmpfeubhd9isU='}) 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout:> e = assert_raises(boto.exception.S3ResponseError, key.set_contents_from_string, 'bar') 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:332: 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout:excClass = 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout:callableObj = > 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout:args = ('bar',), kwargs = {}, excName = 'S3ResponseError' 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: def assert_raises(excClass, callableObj, *args, **kwargs): 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: Like unittest.TestCase.assertRaises, but returns the exception. 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: try: 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: callableObj(*args, **kwargs) 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: except excClass as e: 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: return e 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: if hasattr(excClass, '__name__'): 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: excName = excClass.__name__ 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: excName = str(excClass) 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout:> raise AssertionError("%s not raised" % excName) 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout:E AssertionError: S3ResponseError not raised 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/utils.py:19: AssertionError 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout:--------------------------- Captured stdout teardown --------------------------- 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on main 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection main prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout:prefix= test-client.0-jjbc677ewdfcmzd- 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket key 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on alt 2024-03-09T15:35:57.941 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection alt prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on tenant 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection tenant prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout:Done with cleanup of test buckets. 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout:______________ test_object_create_bad_authorization_invalid_aws2 _______________ 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.auth_aws2 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.fails_on_dbstore 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: def test_object_create_bad_authorization_invalid_aws2(): 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: check_aws2_support() 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: key = _setup_bad_object({'Authorization': 'AWS HAHAHA'}) 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout:> e = assert_raises(boto.exception.S3ResponseError, key.set_contents_from_string, 'bar') 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:343: 2024-03-09T15:35:57.942 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout:excClass = 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout:callableObj = > 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout:args = ('bar',), kwargs = {}, excName = 'S3ResponseError' 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: def assert_raises(excClass, callableObj, *args, **kwargs): 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: Like unittest.TestCase.assertRaises, but returns the exception. 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: try: 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: callableObj(*args, **kwargs) 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: except excClass as e: 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: return e 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: if hasattr(excClass, '__name__'): 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: excName = excClass.__name__ 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: excName = str(excClass) 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout:> raise AssertionError("%s not raised" % excName) 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout:E AssertionError: S3ResponseError not raised 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/utils.py:19: AssertionError 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout:--------------------------- Captured stdout teardown --------------------------- 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on main 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection main prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout:prefix= test-client.0-jjbc677ewdfcmzd- 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket key 2024-03-09T15:35:57.944 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on alt 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection alt prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on tenant 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection tenant prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout:Done with cleanup of test buckets. 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ test_object_create_bad_date_none_aws2 _____________________ 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.auth_aws2 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.fails_on_dbstore 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: def test_object_create_bad_date_none_aws2(): 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: check_aws2_support() 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: key = _setup_bad_object(remove=('Date',)) 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout:> e = assert_raises(boto.exception.S3ResponseError, key.set_contents_from_string, 'bar') 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.945 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:353: 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout:excClass = 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout:callableObj = > 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout:args = ('bar',), kwargs = {}, excName = 'S3ResponseError' 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: def assert_raises(excClass, callableObj, *args, **kwargs): 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: Like unittest.TestCase.assertRaises, but returns the exception. 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: try: 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: callableObj(*args, **kwargs) 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: except excClass as e: 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: return e 2024-03-09T15:35:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: if hasattr(excClass, '__name__'): 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: excName = excClass.__name__ 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: excName = str(excClass) 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout:> raise AssertionError("%s not raised" % excName) 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout:E AssertionError: S3ResponseError not raised 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/utils.py:19: AssertionError 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout:--------------------------- Captured stdout teardown --------------------------- 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on main 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection main prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout:prefix= test-client.0-jjbc677ewdfcmzd- 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket 2024-03-09T15:35:57.947 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket key 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on alt 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection alt prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on tenant 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection tenant prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout:Done with cleanup of test buckets. 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout:______________ test_bucket_create_bad_authorization_invalid_aws2 _______________ 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.auth_aws2 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: def test_bucket_create_bad_authorization_invalid_aws2(): 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: check_aws2_support() 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: _add_custom_headers({'Authorization': 'AWS HAHAHA'}) 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout:> e = assert_raises(boto.exception.S3ResponseError, get_new_bucket) 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.948 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:363: 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout:excClass = 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout:callableObj = , args = () 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout:kwargs = {}, excName = 'S3ResponseError' 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: def assert_raises(excClass, callableObj, *args, **kwargs): 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: Like unittest.TestCase.assertRaises, but returns the exception. 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: try: 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: callableObj(*args, **kwargs) 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: except excClass as e: 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: return e 2024-03-09T15:35:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: if hasattr(excClass, '__name__'): 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: excName = excClass.__name__ 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: excName = str(excClass) 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout:> raise AssertionError("%s not raised" % excName) 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout:E AssertionError: S3ResponseError not raised 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/utils.py:19: AssertionError 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout:--------------------------- Captured stdout teardown --------------------------- 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on main 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection main prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout:prefix= test-client.0-jjbc677ewdfcmzd- 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket 2024-03-09T15:35:57.950 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on alt 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection alt prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on tenant 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection tenant prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout:Done with cleanup of test buckets. 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout:____________________ test_bucket_create_bad_date_none_aws2 _____________________ 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.auth_aws2 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.fails_on_dbstore 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: def test_bucket_create_bad_date_none_aws2(): 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: check_aws2_support() 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: _add_custom_headers(remove=('Date',)) 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout:> e = assert_raises(boto.exception.S3ResponseError, get_new_bucket) 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:373: 2024-03-09T15:35:57.951 INFO:teuthology.orchestra.run.smithi001.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout:excClass = 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout:callableObj = , args = () 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout:kwargs = {}, excName = 'S3ResponseError' 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: def assert_raises(excClass, callableObj, *args, **kwargs): 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: Like unittest.TestCase.assertRaises, but returns the exception. 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: """ 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: try: 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: callableObj(*args, **kwargs) 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: except excClass as e: 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: return e 2024-03-09T15:35:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: if hasattr(excClass, '__name__'): 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: excName = excClass.__name__ 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: else: 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: excName = str(excClass) 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout:> raise AssertionError("%s not raised" % excName) 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout:E AssertionError: S3ResponseError not raised 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/utils.py:19: AssertionError 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout:--------------------------- Captured stdout teardown --------------------------- 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on main 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection main prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout:prefix= test-client.0-jjbc677ewdfcmzd- 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning bucket 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on alt 2024-03-09T15:35:57.953 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection alt prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting buckets on tenant 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:Cleaning buckets from connection tenant prefix 'test-client.0-jjbc677ewdfcmzd-'. 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:Done with cleanup of test buckets. 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:=============================== warnings summary =============================== 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:.tox/py/lib/python3.10/site-packages/boto/plugin.py:40 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/.tox/py/lib/python3.10/site-packages/boto/plugin.py:40: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: import imp 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:124 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests/functional/test_headers.py:124: SyntaxWarning: "is" with a literal. Did you mean "=="? 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: if boto_type is 'S3Connection': 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_headers.py:129 2024-03-09T15:35:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests/functional/test_headers.py:129: SyntaxWarning: "is" with a literal. Did you mean "=="? 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: elif boto_type is 'HTTPRequest': 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3.py:359 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests/functional/test_s3.py:359: DeprecationWarning: invalid escape sequence '\W' 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: extra_sc = re.split('\W+', config['main']['storage_classes']) 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py:176 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests/functional/test_s3_website.py:176: SyntaxWarning: "is" with a literal. Did you mean "=="? 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: if k is 'BucketName': 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py:253 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests/functional/test_s3_website.py:253: PytestUnknownMarkWarning: Unknown pytest.mark.timeout - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.timeout(10) 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.955 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests/functional/test_s3_website.py:1077 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: s3tests/functional/test_s3_website.py:1077: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_routing_generator will be ignored 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: @pytest.mark.s3website_routing_rules 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py:2212 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2212: DeprecationWarning: invalid escape sequence '\$' 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: ["starts-with", "$key", "\$foo"],\ 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py:2227 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2227: DeprecationWarning: invalid escape sequence '\$' 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: payload = OrderedDict([ ("key" , "\$foo.txt"),("AWSAccessKeyId" , aws_access_key_id),\ 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py:2233 2024-03-09T15:35:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2233: DeprecationWarning: invalid escape sequence '\$' 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: response = client.get_object(Bucket=bucket_name, Key='\$foo.txt') 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py:2290 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2290: DeprecationWarning: invalid escape sequence '\$' 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: ["starts-with", "$key", "\$foo"],\ 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py:2305 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2305: DeprecationWarning: invalid escape sequence '\$' 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: payload = OrderedDict([ ("key" , "\$foo.txt"),("AWSAccessKeyId" , aws_access_key_id),\ 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py:2323 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2323: DeprecationWarning: invalid escape sequence '\$' 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: ["starts-with", "$key", "\$foo"],\ 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.957 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py:2338 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2338: DeprecationWarning: invalid escape sequence '\$' 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: payload = OrderedDict([ ("key" , "\$foo.txt"),("AWSAccessKeyId" , aws_access_key_id[::-1]),\ 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py:2356 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2356: DeprecationWarning: invalid escape sequence '\$' 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: ["starts-with", "$key", "\$foo"],\ 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py:2371 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2371: DeprecationWarning: invalid escape sequence '\$' 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: payload = OrderedDict([ ("key" , "\$foo.txt"),("AWSAccessKeyId" , aws_access_key_id),\ 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py:2421 2024-03-09T15:35:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2421: DeprecationWarning: invalid escape sequence '\$' 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: ["starts-with", "$key", "\$foo"],\ 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py:2453 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2453: DeprecationWarning: invalid escape sequence '\$' 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: ["starts-with", "$key", "\$foo"],\ 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_headers.py: 29 warnings 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_iam.py: 79 warnings 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3.py: 484 warnings 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout:s3tests_boto3/functional/test_s3select.py: 33 warnings 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: /home/ubuntu/cephtest/s3-tests-client.0/.tox/py/lib/python3.10/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'smithi001.front.sepia.ceph.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: warnings.warn( 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:35:57.959 INFO:teuthology.orchestra.run.smithi001.stdout:-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:=========================== short test summary info ============================ 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:FAILED s3tests/functional/test_headers.py::test_object_create_bad_contentlength_none 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:FAILED s3tests/functional/test_headers.py::test_object_create_bad_authorization_empty 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:FAILED s3tests/functional/test_headers.py::test_object_create_bad_authorization_none 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:FAILED s3tests/functional/test_headers.py::test_bucket_create_bad_authorization_empty 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:FAILED s3tests/functional/test_headers.py::test_bucket_create_bad_authorization_none 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:FAILED s3tests/functional/test_headers.py::test_object_create_bad_contentlength_mismatch_below_aws2 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:FAILED s3tests/functional/test_headers.py::test_object_create_bad_authorization_incorrect_aws2 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:FAILED s3tests/functional/test_headers.py::test_object_create_bad_authorization_invalid_aws2 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:FAILED s3tests/functional/test_headers.py::test_object_create_bad_date_none_aws2 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:FAILED s3tests/functional/test_headers.py::test_bucket_create_bad_authorization_invalid_aws2 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:FAILED s3tests/functional/test_headers.py::test_bucket_create_bad_date_none_aws2 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_write_to_nonexist_bucket 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_write_with_chunked_transfer_encoding 2024-03-09T15:35:57.960 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_delete - botoco... 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_read_not_exist - botoc... 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_requestid_matches_header_on_error 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_concurrent_multi_object_delete 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multi_object_delete - botocor... 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multi_objectv2_delete - botoc... 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multi_object_delete_key_limit 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multi_objectv2_delete_key_limit 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_head_zero_bytes - boto... 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_write_check_etag - bot... 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_write_cache_control - ... 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_write_expires - botoco... 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_write_read_update_read_delete 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_set_get_metadata_none_to_good 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_set_get_metadata_none_to_empty 2024-03-09T15:35:57.961 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_set_get_metadata_overwrite_to_empty 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_metadata_replaced_on_put 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_write_file - botocore.... 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_anonymous_request 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_authenticated_request 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_authenticated_no_content_type 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_authenticated_request_bad_access_key 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_set_success_code 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_set_invalid_success_code 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_upload_larger_than_chunk 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_set_key_from_filename 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_ignored_header - ... 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_case_insensitive_condition_fields 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_escaped_field_values 2024-03-09T15:35:57.962 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_success_redirect_action 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_invalid_signature 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_invalid_access_key 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_invalid_date_format 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_no_key_specified 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_missing_signature 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_missing_policy_condition 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_user_specified_header 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_request_missing_policy_specified_field 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_condition_is_case_sensitive 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_expires_is_case_sensitive 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_expired_policy - ... 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_wrong_bucket - bo... 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_invalid_request_field_value 2024-03-09T15:35:57.963 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_missing_expires_condition 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_missing_conditions_list 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_upload_size_limit_exceeded 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_missing_content_length_argument 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_invalid_content_length_argument 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_upload_size_below_minimum 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_upload_size_rgw_chunk_size_bug 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_empty_conditions 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_object_ifmatch_good - bot... 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_object_ifmatch_failed - b... 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_object_ifnonematch_good 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_object_ifnonematch_failed 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_object_ifmodifiedsince_good 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_object_ifmodifiedsince_failed 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_object_ifunmodifiedsince_good 2024-03-09T15:35:57.964 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_object_ifunmodifiedsince_failed 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_object_ifmatch_good - bot... 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_object_ifmatch_failed - b... 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_object_ifmatch_overwrite_existed_good 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_object_ifmatch_nonexisted_failed 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_object_ifnonmatch_good - ... 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_object_ifnonmatch_failed 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_object_ifnonmatch_nonexisted_good 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_object_ifnonmatch_overwrite_existed_failed 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_get - botocore.err... 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_get_bucket_gone - ... 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_delete_key_bucket_gone 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_get_object_gone - ... 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_head - botocore.errorf... 2024-03-09T15:35:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_head_notexist - botoco... 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_head_extended - botoco... 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_get_bucket_acl - b... 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_get_object_acl - b... 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_put_acl_mtime - botoco... 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_authenticated - bo... 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_response_headers 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_authenticated_bucket_acl 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_authenticated_object_acl 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_authenticated_bucket_gone 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_authenticated_object_gone 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_get_x_amz_expires_not_expired 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_get_x_amz_expires_not_expired_tenant 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_get_x_amz_expires_out_range_zero 2024-03-09T15:35:57.966 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_get_x_amz_expires_out_max_range 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_get_x_amz_expires_out_positive_range 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_anon_put - botocore.er... 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_anon_put_write_access 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_put_authenticated - bo... 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_presigned_put_object_with_acl 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_presigned_put_object_with_acl_tenant 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_raw_put_authenticated_expired 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_bad_starts_nonalpha 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_bad_short_one 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_bad_short_two 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_long_60 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_long_61 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_long_62 2024-03-09T15:35:57.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_long_63 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_list_long_name - botoc... 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_bad_ip 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_dns_underscore 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_dns_long 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_dns_dash_at_end 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_dns_dot_dot 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_dns_dot_dash 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_dns_dash_dot 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_exists - botoco... 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_get_location - botocor... 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_exists_nonowner 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_recreate_overwrite_acl 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_recreate_new_acl - bot... 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_default - botocore... 2024-03-09T15:35:57.968 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_canned_during_create 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_canned - botocore.... 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_canned_publicreadwrite 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_canned_authenticatedread 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_default - botocore... 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_canned_during_create 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_canned - botocore.... 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_canned_publicreadwrite 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_canned_authenticatedread 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_canned_bucketownerread 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_canned_bucketownerfullcontrol 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_full_control_verify_owner 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_full_control_verify_attributes 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_canned_private_to_private 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl - botocore.errorfa... 2024-03-09T15:35:57.969 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_write - botocore.e... 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_writeacp - botocor... 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_read - botocore.er... 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_acl_readacp - botocore... 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_userid_fullcontrol 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_userid_read 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_userid_readacp 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_userid_write 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_userid_writeacp 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_nonexist_user 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_header_acl_grants - bo... 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_header_acl_grants - bo... 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_email - boto... 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_grant_email_not_exist 2024-03-09T15:35:57.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_acl_revoke_all - botoc... 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_access_bucket_private_object_private 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_access_bucket_private_objectv2_private 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_access_bucket_private_object_publicread 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_access_bucket_private_objectv2_publicread 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_access_bucket_private_object_publicreadwrite 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_access_bucket_private_objectv2_publicreadwrite 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_access_bucket_publicread_object_private 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_access_bucket_publicread_object_publicread 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_access_bucket_publicread_object_publicreadwrite 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_access_bucket_publicreadwrite_object_private 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_access_bucket_publicreadwrite_object_publicread 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_access_bucket_publicreadwrite_object_publicreadwrite 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_buckets_create_then_list - bo... 2024-03-09T15:35:57.971 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_buckets_list_ctime - botocore... 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_list_buckets_anonymous - boto... 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_list_buckets_invalid_auth - b... 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_list_buckets_bad_auth - botoc... 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_starts_alpha 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_starts_digit 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_contains_period 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_naming_good_contains_hyphen 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_recreate_not_overriding 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_create_special_key_names 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_list_special_prefix - ... 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_zero_size - botoc... 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_16m - botocore.er... 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_same_bucket - bot... 2024-03-09T15:35:57.972 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_verify_contenttype 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_to_itself - botoc... 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_to_itself_with_metadata 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_diff_bucket - bot... 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_not_owned_bucket 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_not_owned_object_bucket 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_canned_acl - boto... 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_retaining_metadata 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_replacing_metadata 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_bucket_not_found 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_key_not_found - b... 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_versioned_bucket 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_versioned_url_encoding 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_copy_versioning_multipart_upload 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_upload_empty - boto... 2024-03-09T15:35:57.973 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_upload_small - boto... 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_copy_small - botoco... 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_copy_invalid_range 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_copy_without_range 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_copy_special_names 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_upload - botocore.e... 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_copy_versioned - bo... 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_upload_resend_part 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_upload_multiple_sizes 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_copy_multiple_sizes 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_upload_size_too_small 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_upload_contents - b... 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_upload_overwrite_existing_object 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_abort_multipart_upload - boto... 2024-03-09T15:35:57.974 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_abort_multipart_upload_not_found 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_list_multipart_upload - botoc... 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_list_multipart_upload_owner 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_upload_missing_part 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_upload_incorrect_etag 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_get_part - botocore... 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_non_multipart_get_part - boto... 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_100_continue - botocore.error... 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_set_cors - botocore.errorfact... 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_cors_origin_response - botoco... 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_cors_origin_wildcard - botoco... 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_cors_header_option - botocore... 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_cors_presigned_get_object - b... 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_cors_presigned_get_object_tenant 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_cors_presigned_put_object - b... 2024-03-09T15:35:57.975 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_cors_presigned_put_object_with_acl 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_cors_presigned_put_object_tenant 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_cors_presigned_put_object_tenant_with_acl 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_set_bucket_tagging - botocore... 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_atomic_read_1mb - botocore.er... 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_atomic_read_4mb - botocore.er... 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_atomic_read_8mb - botocore.er... 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_atomic_write_1mb - botocore.e... 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_atomic_write_4mb - botocore.e... 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_atomic_write_8mb - botocore.e... 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_atomic_dual_write_1mb - botoc... 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_atomic_dual_write_4mb - botoc... 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_atomic_dual_write_8mb - botoc... 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_atomic_conditional_write_1mb 2024-03-09T15:35:57.976 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_atomic_multipart_upload_write 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_resend_first_finishes_last 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_ranged_request_response_code 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_ranged_big_request_response_code 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_ranged_request_skip_leading_bytes_response_code 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_ranged_request_return_trailing_bytes_response_code 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_ranged_request_invalid_range 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_ranged_request_empty_object 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_bucket_create_suspend 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_obj_create_read_remove 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_obj_create_read_remove_head 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_obj_plain_null_version_removal 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_obj_plain_null_version_overwrite 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_obj_plain_null_version_overwrite_suspended 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_obj_suspend_versions 2024-03-09T15:35:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_obj_suspended_copy 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_obj_create_versions_remove_all 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_obj_create_versions_remove_special_names 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_obj_create_overwrite_multipart 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_obj_list_marker - ... 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_copy_obj_version 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_multi_object_delete 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_multi_object_delete_with_marker 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_multi_object_delete_with_marker_create 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioned_object_acl - botoco... 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioned_object_acl_no_version_specified 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioned_concurrent_object_create_and_remove 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_set - botocore.erro... 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_get - botocore.erro... 2024-03-09T15:35:57.978 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_get_no_id - botocor... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_id_too_long - botoc... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_same_id - botocore.... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_invalid_status - bo... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_set_date - botocore... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_set_invalid_date - ... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_set_noncurrent - bo... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_set_deletemarker - ... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_set_filter - botoco... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_set_empty_filter - ... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_set_multipart - bot... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_lifecycle_transition_set_invalid_date 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encrypted_transfer_1b - botoc... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encrypted_transfer_1kb - boto... 2024-03-09T15:35:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encrypted_transfer_1MB - boto... 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encrypted_transfer_13b - boto... 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_method_head 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_present - bo... 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_other_key - ... 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_invalid_md5 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_no_md5 - bot... 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_no_key - bot... 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encryption_key_no_sse_c - bot... 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_multipart_upload 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_unaligned_multipart_upload 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_multipart_bad_download 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_post_object_authenticated_request 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_method_head - botocor... 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_present - botocore.er... 2024-03-09T15:35:57.980 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_no_key - botocore.err... 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_not_declared - botoco... 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_multipart_upload - bo... 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_multipart_invalid_chunks_1 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_multipart_invalid_chunks_2 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_post_object_authenticated_request 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_transfer_1b - botocor... 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_transfer_1kb - botoco... 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_transfer_1MB - botoco... 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_transfer_13b - botoco... 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_read_declare - botoco... 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy - botocore.erro... 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucketv2_policy - botocore.er... 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy_acl - botocore.... 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucketv2_policy_acl - botocor... 2024-03-09T15:35:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy_another_bucket 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucketv2_policy_another_bucket 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_obj_tagging - botocore.er... 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_obj_head_tagging - botoco... 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_max_tags - botocore.error... 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_excess_tags - botocore.er... 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_max_kvsize_tags - botocor... 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_excess_key_tags - botocor... 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_excess_val_tags - botocor... 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_modify_tags - botocore.er... 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_delete_tags - botocore.er... 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_tags_anonymous_request 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_post_object_tags_authenticated_request 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_obj_with_tags - botocore.... 2024-03-09T15:35:57.982 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_tags_acl_public - botocor... 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_tags_acl_public - botocor... 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_delete_tags_obj_public - boto... 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_bucket_atomic_upload_return_version_id 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_versioning_bucket_multipart_upload_return_version_id 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy_get_obj_existing_tag 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy_get_obj_tagging_existing_tag 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_tagging_existing_tag 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_copy_source 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_copy_source_meta 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_acl - b... 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_grant 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_obj_enc_conflict_c_s3 - b... 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_obj_enc_conflict_c_kms - ... 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_obj_enc_conflict_s3_kms 2024-03-09T15:35:57.983 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_obj_enc_conflict_bad_enc_kms 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_kms_noenc 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy_put_obj_kms_s3 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_bucket_policy_get_obj_acl_existing_tag 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock - bo... 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_bucket 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock_with_days_and_years 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_days 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_years 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_mode 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_lock_invalid_status 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_suspend_versioning 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_get_obj_lock - bo... 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_get_obj_lock_invalid_bucket 2024-03-09T15:35:57.984 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_invalid_bucket 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_invalid_mode 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_get_obj_retention 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_get_obj_retention_iso8601 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_get_obj_retention_invalid_bucket 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_versionid 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_override_default_retention 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_increase_period 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_shorten_period 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_obj_retention_shorten_period_bypass 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_delete_object_with_retention 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_delete_object_with_retention_and_marker 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_multi_delete_object_with_retention 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_legal_hold - ... 2024-03-09T15:35:57.985 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_legal_hold_invalid_bucket 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_put_legal_hold_invalid_status 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_get_legal_hold - ... 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_get_legal_hold_invalid_bucket 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_delete_object_with_legal_hold_on 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_delete_object_with_legal_hold_off 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_get_obj_metadata 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_uploading_obj - b... 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_changing_mode_from_governance_with_bypass 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_changing_mode_from_governance_without_bypass 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_object_lock_changing_mode_from_compliance 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_copy_object_ifmatch_good - bo... 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_copy_object_ifnonematch_failed 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_bucket_policy_status - bo... 2024-03-09T15:35:57.986 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_public_acl_bucket_policy_status 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_authpublic_acl_bucket_policy_status 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_publicpolicy_acl_bucket_policy_status 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_nonpublicpolicy_acl_bucket_policy_status 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_nonpublicpolicy_deny_bucket_policy_status 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_undefined_public_block - ... 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_public_block_deny_bucket_policy 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_public_block - botocore.e... 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_block_public_put_bucket_acls 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_block_public_object_canned_acls 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_block_public_policy - botocor... 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_ignore_public_acls - botocore... 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_multipart_upload_on_a_bucket_with_policy 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_put_bucket_encryption_kms - b... 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_bucket_encryption_kms - b... 2024-03-09T15:35:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_delete_bucket_encryption_kms 2024-03-09T15:35:57.988 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_sse_kms_default_post_object_authenticated_request 2024-03-09T15:35:57.988 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR s3tests_boto3/functional/test_s3.py::test_get_object_torrent - botocore... 2024-03-09T15:35:57.988 INFO:teuthology.orchestra.run.smithi001.stdout:= 11 failed, 254 passed, 65 skipped, 124 deselected, 1 xfailed, 643 warnings, 395 errors in 1458.23s (0:24:18) = 2024-03-09T15:35:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T15:35:57.989 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: InvocationError for command /home/ubuntu/cephtest/s3-tests-client.0/.tox/py/bin/pytest -v -m 'not fails_on_rgw and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3' (exited with code 1) 2024-03-09T15:35:57.989 INFO:teuthology.orchestra.run.smithi001.stdout:___________________________________ summary ____________________________________ 2024-03-09T15:35:57.989 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: py: commands failed 2024-03-09T15:35:57.990 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/s3tests.py", line 462, in run_tests toxvenv_sh(ctx, remote, args, label="s3 tests against rgw") File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/s3tests.py", line 405, in toxvenv_sh return remote.sh(['source', activate, run.Raw('&&')] + args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi001 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" 2024-03-09T15:35:57.991 INFO:tasks.s3tests:Cleaning up boto... 2024-03-09T15:35:57.992 DEBUG:teuthology.orchestra.run.smithi001:> rm /home/ubuntu/cephtest/boto-client.0.cfg 2024-03-09T15:35:57.998 INFO:tasks.s3tests:Removing s3-tests... 2024-03-09T15:35:57.998 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/s3-tests-client.0 2024-03-09T15:35:58.193 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/s3tests.py", line 736, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/s3tests.py", line 61, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/s3tests.py", line 462, in run_tests toxvenv_sh(ctx, remote, args, label="s3 tests against rgw") File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/s3tests.py", line 405, in toxvenv_sh return remote.sh(['source', activate, run.Raw('&&')] + args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi001 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" 2024-03-09T15:35:58.414 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=43cae2bbeecb47dd8b4ca966794a07c5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/s3tests.py", line 736, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/s3tests.py", line 61, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/s3tests.py", line 462, in run_tests toxvenv_sh(ctx, remote, args, label="s3 tests against rgw") File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/s3tests.py", line 405, in toxvenv_sh return remote.sh(['source', activate, run.Raw('&&')] + args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi001 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" 2024-03-09T15:35:58.418 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2024-03-09T15:35:58.429 DEBUG:teuthology.run_tasks:Unwinding manager s3tests-java 2024-03-09T15:35:58.466 DEBUG:tasks.s3tests_java:S3 Tests Java: END 2024-03-09T15:35:58.466 INFO:tasks.s3tests_java:S3 Tests Java: Cleaning up s3-tests-java... 2024-03-09T15:35:58.466 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/s3-tests-java 2024-03-09T15:35:58.525 INFO:tasks.s3tests_java:S3 Tests Java: Deleting S3 users... 2024-03-09T15:35:58.525 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph 2024-03-09T15:35:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:35:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:35:58.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.596+0000 7f7d577f9980 20 rados->read ofs=0 len=0 2024-03-09T15:35:58.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.596+0000 7f7d577f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:35:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.596+0000 7f7d577f9980 20 realm 2024-03-09T15:35:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.596+0000 7f7d577f9980 20 rados->read ofs=0 len=0 2024-03-09T15:35:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.596+0000 7f7d577f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:35:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.596+0000 7f7d577f9980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:35:58.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.596+0000 7f7d577f9980 20 rados->read ofs=0 len=0 2024-03-09T15:35:58.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.596+0000 7f7d577f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:35:58.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.596+0000 7f7d577f9980 20 rados->read ofs=0 len=0 2024-03-09T15:35:58.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.600+0000 7f7d577f9980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:35:58.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.600+0000 7f7d577f9980 20 rados->read ofs=0 len=0 2024-03-09T15:35:58.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.600+0000 7f7d577f9980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:35:58.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.600+0000 7f7d577f9980 20 searching for the correct realm 2024-03-09T15:35:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.608+0000 7f7d54b10640 0 -- 172.21.15.1:0/1782319522 >> [v2:172.21.15.1:6800/4091251076,v1:172.21.15.1:6801/4091251076] conn(0x55d7e3f426c0 msgr2=0x55d7e3f44b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:35:58.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.612+0000 7f7d55311640 0 -- 172.21.15.1:0/1782319522 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f7d4c07b280 msgr2=0x7f7d4c07c840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:35:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.620+0000 7f7d577f9980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:35:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.620+0000 7f7d577f9980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:35:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.620+0000 7f7d577f9980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:35:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.620+0000 7f7d577f9980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:35:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.620+0000 7f7d577f9980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:35:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.620+0000 7f7d577f9980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:35:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.620+0000 7f7d577f9980 20 rados->read ofs=0 len=0 2024-03-09T15:35:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.620+0000 7f7d577f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:35:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.620+0000 7f7d577f9980 20 rados->read ofs=0 len=0 2024-03-09T15:35:58.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:35:58.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 rados->read ofs=0 len=0 2024-03-09T15:35:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:35:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 zone default found 2024-03-09T15:35:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 4 Realm: () 2024-03-09T15:35:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:35:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:35:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:35:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 zonegroup default 2024-03-09T15:35:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 rados->read ofs=0 len=0 2024-03-09T15:35:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:35:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 rados->read ofs=0 len=0 2024-03-09T15:35:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:35:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 rados->read ofs=0 len=0 2024-03-09T15:35:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:35:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 started sync module instance, tier type = 2024-03-09T15:35:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.624+0000 7f7d577f9980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:35:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.640+0000 7f7d577f9980 20 add_watcher() i=0 2024-03-09T15:35:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.640+0000 7f7d577f9980 20 add_watcher() i=1 2024-03-09T15:35:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.640+0000 7f7d577f9980 20 add_watcher() i=2 2024-03-09T15:35:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.640+0000 7f7d577f9980 20 add_watcher() i=3 2024-03-09T15:35:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.640+0000 7f7d577f9980 20 add_watcher() i=4 2024-03-09T15:35:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.640+0000 7f7d577f9980 20 add_watcher() i=5 2024-03-09T15:35:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.640+0000 7f7d577f9980 20 add_watcher() i=6 2024-03-09T15:35:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.644+0000 7f7d577f9980 20 add_watcher() i=7 2024-03-09T15:35:58.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.644+0000 7f7d577f9980 2 all 8 watchers are set, enabling cache 2024-03-09T15:35:58.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.644+0000 7f7c0bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:35:58.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.644+0000 7f7d577f9980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:35:58.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.644+0000 7f7d577f9980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:35:58.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.724+0000 7f7c0afe5640 20 reqs_thread_entry: start 2024-03-09T15:35:58.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.724+0000 7f7d577f9980 20 init_complete bucket index max shards: 1 2024-03-09T15:35:58.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.724+0000 7f7d577f9980 20 Filter name: none 2024-03-09T15:35:58.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.724+0000 7f7c08fe1640 20 reqs_thread_entry: start 2024-03-09T15:35:58.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.724+0000 7f7d577f9980 10 cache get: name=default.rgw.meta+users.uid+tester.client.0 : miss 2024-03-09T15:35:58.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.724+0000 7f7d577f9980 20 rados->read ofs=0 len=0 2024-03-09T15:35:58.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.724+0000 7f7d577f9980 20 rados_obj.operate() r=0 bl.length=482 2024-03-09T15:35:58.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.724+0000 7f7d577f9980 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x13 2024-03-09T15:35:58.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.724+0000 7f7d577f9980 10 adding default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2024-03-09T15:35:58.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.724+0000 7f7d577f9980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-03-09T15:35:58.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.728+0000 7f7d577f9980 10 removing key index: KNSHDPEDHGBLMBSLSSNA 2024-03-09T15:35:58.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.728+0000 7f7d577f9980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.keys:KNSHDPEDHGBLMBSLSSNA, ofs0, ns] 2024-03-09T15:35:58.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.728+0000 7f7c38ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464924 cookie 94385730782736 notifier 7959 bl.length()=170 2024-03-09T15:35:58.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.732+0000 7f7d577f9980 10 removing email index: tester.client.0_test@test.test 2024-03-09T15:35:58.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.732+0000 7f7d577f9980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.email:tester.client.0_test@test.test, ofs0, ns] 2024-03-09T15:35:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.736+0000 7f7c27fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464541 cookie 94385730776416 notifier 7959 bl.length()=181 2024-03-09T15:35:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.736+0000 7f7d577f9980 10 removing user buckets index 2024-03-09T15:35:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.740+0000 7f7d577f9980 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.uid:tester.client.0.buckets, ofs0, ns] 2024-03-09T15:35:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.740+0000 7f7c27fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464605 cookie 94385730785232 notifier 7959 bl.length()=172 2024-03-09T15:35:58.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.740+0000 7f7d577f9980 10 removing user index: tester.client.0 2024-03-09T15:35:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.744+0000 7f7d577f9980 10 removing default.rgw.meta+users.uid+tester.client.0 from cache 2024-03-09T15:35:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.744+0000 7f7d577f9980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:tester.client.0, ofs0, ns] 2024-03-09T15:35:58.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.744+0000 7f7c27fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1120986464925 cookie 94385730782736 notifier 7959 bl.length()=164 2024-03-09T15:35:58.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.748+0000 7f7c0bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:35:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.752+0000 7f7d577f9980 20 remove_watcher() i=0 2024-03-09T15:35:58.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.752+0000 7f7d577f9980 2 removed watcher, disabling cache 2024-03-09T15:35:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.756+0000 7f7d577f9980 20 remove_watcher() i=1 2024-03-09T15:35:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.760+0000 7f7d577f9980 20 remove_watcher() i=2 2024-03-09T15:35:58.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.760+0000 7f7d577f9980 20 remove_watcher() i=3 2024-03-09T15:35:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.764+0000 7f7d577f9980 20 remove_watcher() i=4 2024-03-09T15:35:58.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.764+0000 7f7d577f9980 20 remove_watcher() i=5 2024-03-09T15:35:58.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.768+0000 7f7d577f9980 20 remove_watcher() i=6 2024-03-09T15:35:58.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:35:58.768+0000 7f7d577f9980 20 remove_watcher() i=7 2024-03-09T15:35:58.788 DEBUG:teuthology.run_tasks:Unwinding manager ragweed 2024-03-09T15:35:58.837 DEBUG:teuthology.run_tasks:Unwinding manager tox 2024-03-09T15:35:58.852 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/tox-venv 2024-03-09T15:35:58.899 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-03-09T15:35:58.913 DEBUG:tasks.rgw.client.0:waiting for process to exit 2024-03-09T15:35:58.913 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:35:58.920 INFO:tasks.rgw.client.0.smithi001.stdout:2024-03-09T15:35:58.916+0000 e528640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/ceph.client.0.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends beast ssl_certificate=/home/ubuntu/cephtest/ca/rgw.client.0.crt ssl_port=443 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground (PID: 37882) UID: 0 2024-03-09T15:35:58.923 INFO:tasks.rgw.client.0.smithi001.stdout:2024-03-09T15:35:58.920+0000 8e90e80 -1 shutting down 2024-03-09T15:36:03.840 INFO:tasks.rgw.client.0.smithi001.stderr:==00:01:07:18.111 37886== could not unlink /tmp/vgdb-pipe-from-vgdb-to-37886-by-root-on-??? 2024-03-09T15:36:03.841 INFO:tasks.rgw.client.0.smithi001.stderr:==00:01:07:18.111 37886== could not unlink /tmp/vgdb-pipe-to-vgdb-from-37886-by-root-on-??? 2024-03-09T15:36:03.841 INFO:tasks.rgw.client.0.smithi001.stderr:==00:01:07:18.111 37886== could not unlink /tmp/vgdb-pipe-shared-mem-vgdb-37886-by-root-on-??? 2024-03-09T15:36:05.014 INFO:tasks.rgw.client.0:Stopped 2024-03-09T15:36:05.015 DEBUG:teuthology.orchestra.run.smithi001:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2024-03-09T15:36:05.021 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f /etc/ceph/vault-root-token 2024-03-09T15:36:05.077 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f /home/ubuntu/cephtest/url_file 2024-03-09T15:36:05.137 INFO:tasks.util.rgw:rgwadmin: client.0 : ['gc', 'process', '--include-all'] 2024-03-09T15:36:05.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'gc', 'process', '--include-all'] 2024-03-09T15:36:05.137 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc process --include-all 2024-03-09T15:36:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setuser ceph since I am not root 2024-03-09T15:36:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:ignoring --setgroup ceph since I am not root 2024-03-09T15:36:05.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.260+0000 7f471c3a7980 20 rados->read ofs=0 len=0 2024-03-09T15:36:05.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.264+0000 7f471c3a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:36:05.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.264+0000 7f471c3a7980 20 realm 2024-03-09T15:36:05.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.264+0000 7f471c3a7980 20 rados->read ofs=0 len=0 2024-03-09T15:36:05.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.264+0000 7f471c3a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:36:05.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.264+0000 7f471c3a7980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-03-09T15:36:05.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.264+0000 7f471c3a7980 20 rados->read ofs=0 len=0 2024-03-09T15:36:05.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.264+0000 7f471c3a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:36:05.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.264+0000 7f471c3a7980 20 rados->read ofs=0 len=0 2024-03-09T15:36:05.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.264+0000 7f471c3a7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:36:05.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.264+0000 7f471c3a7980 20 rados->read ofs=0 len=0 2024-03-09T15:36:05.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.264+0000 7f471c3a7980 20 rados_obj.operate() r=0 bl.length=1103 2024-03-09T15:36:05.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.264+0000 7f471c3a7980 20 searching for the correct realm 2024-03-09T15:36:05.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.268+0000 7f47196be640 0 -- 172.21.15.1:0/4162714827 >> [v2:172.21.15.53:6816/859385596,v1:172.21.15.53:6817/859385596] conn(0x7f4714056040 msgr2=0x7f471403f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:36:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.484+0000 7f4719ebf640 0 -- 172.21.15.1:0/4162714827 >> [v2:172.21.15.1:6806/3097686011,v1:172.21.15.1:6809/3097686011] conn(0x7f47100fe5a0 msgr2=0x7f47100c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-03-09T15:36:05.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.700+0000 7f471c3a7980 20 RGWRados::pool_iterate: got zonegroup_info.ff50b5c7-a620-4021-9d53-f3ce5022ce92 2024-03-09T15:36:05.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.700+0000 7f471c3a7980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-03-09T15:36:05.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.700+0000 7f471c3a7980 20 RGWRados::pool_iterate: got default.zone. 2024-03-09T15:36:05.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.700+0000 7f471c3a7980 20 RGWRados::pool_iterate: got zone_names.default 2024-03-09T15:36:05.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.700+0000 7f471c3a7980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-03-09T15:36:05.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.700+0000 7f471c3a7980 20 RGWRados::pool_iterate: got zone_info.8c209108-e1cd-4d45-9608-6f491550b24c 2024-03-09T15:36:05.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.700+0000 7f471c3a7980 20 rados->read ofs=0 len=0 2024-03-09T15:36:05.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:36:05.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 20 rados->read ofs=0 len=0 2024-03-09T15:36:05.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 20 rados_obj.operate() r=0 bl.length=46 2024-03-09T15:36:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 20 rados->read ofs=0 len=0 2024-03-09T15:36:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 20 rados_obj.operate() r=0 bl.length=432 2024-03-09T15:36:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 20 zone default found 2024-03-09T15:36:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 4 Realm: () 2024-03-09T15:36:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 4 ZoneGroup: default (ff50b5c7-a620-4021-9d53-f3ce5022ce92) 2024-03-09T15:36:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 4 Zone: default (8c209108-e1cd-4d45-9608-6f491550b24c) 2024-03-09T15:36:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 10 cannot find current period zonegroup using local zonegroup configuration 2024-03-09T15:36:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 20 zonegroup default 2024-03-09T15:36:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 20 rados->read ofs=0 len=0 2024-03-09T15:36:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:36:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 20 rados->read ofs=0 len=0 2024-03-09T15:36:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:36:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.704+0000 7f471c3a7980 20 rados->read ofs=0 len=0 2024-03-09T15:36:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.708+0000 7f471c3a7980 20 rados_obj.operate() r=-2 bl.length=0 2024-03-09T15:36:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.708+0000 7f471c3a7980 20 started sync module instance, tier type = 2024-03-09T15:36:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.708+0000 7f471c3a7980 20 started zone id=8c209108-e1cd-4d45-9608-6f491550b24c (name=default) with tier type = 2024-03-09T15:36:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.732+0000 7f471c3a7980 20 add_watcher() i=0 2024-03-09T15:36:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.732+0000 7f471c3a7980 20 add_watcher() i=1 2024-03-09T15:36:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.732+0000 7f471c3a7980 20 add_watcher() i=2 2024-03-09T15:36:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.732+0000 7f471c3a7980 20 add_watcher() i=3 2024-03-09T15:36:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.732+0000 7f471c3a7980 20 add_watcher() i=4 2024-03-09T15:36:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.732+0000 7f471c3a7980 20 add_watcher() i=5 2024-03-09T15:36:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.732+0000 7f471c3a7980 20 add_watcher() i=6 2024-03-09T15:36:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.736+0000 7f471c3a7980 20 add_watcher() i=7 2024-03-09T15:36:05.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.736+0000 7f471c3a7980 2 all 8 watchers are set, enabling cache 2024-03-09T15:36:05.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.740+0000 7f45cffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:36:05.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.740+0000 7f471c3a7980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-03-09T15:36:05.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.740+0000 7f471c3a7980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-03-09T15:36:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.840+0000 7f45cefe5640 20 reqs_thread_entry: start 2024-03-09T15:36:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.840+0000 7f471c3a7980 20 init_complete bucket index max shards: 1 2024-03-09T15:36:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.840+0000 7f471c3a7980 20 Filter name: none 2024-03-09T15:36:05.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.840+0000 7f45ccfe1640 20 reqs_thread_entry: start 2024-03-09T15:36:05.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.840+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-03-09T15:36:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.844+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=51, truncated=0, next_marker='' 2024-03-09T15:36:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14521315387455534312', time=2024-03-09T17:11:41.996698+0000, chain.objs.size()=1 2024-03-09T15:36:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.137__shadow_.CzpUpxpYMqcYJcUP_2Ga_sZGpUtfn7F_0 2024-03-09T15:36:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15490611292913494395', time=2024-03-09T17:12:14.742641+0000, chain.objs.size()=2 2024-03-09T15:36:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.151__shadow_.duilujPXVVTN_iMdwN2-Cm52ukkCLxx_0 2024-03-09T15:36:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.151__shadow_.duilujPXVVTN_iMdwN2-Cm52ukkCLxx_1 2024-03-09T15:36:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14921943644464762402', time=2024-03-09T17:21:23.135293+0000, chain.objs.size()=1 2024-03-09T15:36:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.237__shadow_.kgCshYK-ADqoD43cOMresZMRPn2tMvw_0 2024-03-09T15:36:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12599716242962588244', time=2024-03-09T17:23:59.752933+0000, chain.objs.size()=1 2024-03-09T15:36:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fEc00fGwvee_51oIqJvSZleUa4ZANG6_0 2024-03-09T15:36:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16258246136107647079', time=2024-03-09T17:24:00.118081+0000, chain.objs.size()=1 2024-03-09T15:36:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ckqq6GXaFBxoMtOL-Qb1oxX5sr2jhVQ_0 2024-03-09T15:36:05.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10442601426007394372', time=2024-03-09T17:24:00.629376+0000, chain.objs.size()=1 2024-03-09T15:36:05.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Rs1V9SMg3Da8Xnq5Oiy7bQUU2olxTaI_0 2024-03-09T15:36:05.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17739774752506343845', time=2024-03-09T17:24:00.862604+0000, chain.objs.size()=1 2024-03-09T15:36:05.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.q6V43AA-ucikBHFbBdpJjUfHJBew7nP_0 2024-03-09T15:36:05.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6025487235837902825', time=2024-03-09T17:24:01.019916+0000, chain.objs.size()=1 2024-03-09T15:36:05.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.FGojt6dpyQfDjQbMcXXXDWXedBHRgBe_0 2024-03-09T15:36:05.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16045984648679582883', time=2024-03-09T17:24:01.489640+0000, chain.objs.size()=1 2024-03-09T15:36:05.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LYCT_AhkIY8yq_vd3UKVVfXlY0Ceudr_0 2024-03-09T15:36:05.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5298930842176478823', time=2024-03-09T17:24:05.415706+0000, chain.objs.size()=1 2024-03-09T15:36:05.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.rGeehdsEWpeTUCFQyDcBYQoY3R7pFEB_0 2024-03-09T15:36:05.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7791535034741332409', time=2024-03-09T17:24:07.885392+0000, chain.objs.size()=1 2024-03-09T15:36:05.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.848+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_._Y14wbMXF63hO-qqson7jvJM8-2MXp8_0 2024-03-09T15:36:05.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.852+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15373695761121332256', time=2024-03-09T17:24:07.937202+0000, chain.objs.size()=1 2024-03-09T15:36:05.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.852+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HY9ukqIuKxL5FKB7BLy6hKfG4SFRevq_0 2024-03-09T15:36:05.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6186176523506517476', time=2024-03-09T17:24:07.939076+0000, chain.objs.size()=1 2024-03-09T15:36:05.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.BeRjxascFMzblSTDZXbPqkDxqhVRSom_0 2024-03-09T15:36:05.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.265687083158592127', time=2024-03-09T17:24:08.316684+0000, chain.objs.size()=1 2024-03-09T15:36:05.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sBBTwNOTEmv5i6FGs9gJ08EudHjWLCj_0 2024-03-09T15:36:05.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9934525747658806341', time=2024-03-09T17:24:08.461612+0000, chain.objs.size()=1 2024-03-09T15:36:05.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Yhs5Jm5_0LfIHcauhQhiFLoeyXYNo8U_0 2024-03-09T15:36:05.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17124049025413153058', time=2024-03-09T17:24:08.794604+0000, chain.objs.size()=1 2024-03-09T15:36:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.BVMQdKifrhtgk34zHvSHtGH65Xp3jbY_0 2024-03-09T15:36:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10182280572816106543', time=2024-03-09T17:24:09.161902+0000, chain.objs.size()=1 2024-03-09T15:36:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.luwCd_z-L0VLZbrAv0L94BK5dyCmTi-_0 2024-03-09T15:36:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2167577351115346733', time=2024-03-09T17:24:11.130556+0000, chain.objs.size()=1 2024-03-09T15:36:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.yeo4Bu9ECfWrgg_XpOZb3NC7qgRRK6F_0 2024-03-09T15:36:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4473471113281030272', time=2024-03-09T17:24:12.155879+0000, chain.objs.size()=1 2024-03-09T15:36:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.H_DcFmerxU4wTwAMN38Ia4ZaRW6HKpc_0 2024-03-09T15:36:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2186620773096098048', time=2024-03-09T17:24:12.662153+0000, chain.objs.size()=1 2024-03-09T15:36:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.aDKXrmFI5IQ6sPkVWYEh3X96PerY465_0 2024-03-09T15:36:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5123059093526807305', time=2024-03-09T17:24:14.553387+0000, chain.objs.size()=1 2024-03-09T15:36:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-0kV9Jij31XesY7aR8SIv2EtuCvc82e_0 2024-03-09T15:36:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3189838028771610483', time=2024-03-09T17:24:14.722040+0000, chain.objs.size()=1 2024-03-09T15:36:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LUGiD3cXYKRFkGIDqvVGd94syzxPD4M_0 2024-03-09T15:36:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16771251689511443014', time=2024-03-09T17:24:15.239399+0000, chain.objs.size()=1 2024-03-09T15:36:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.856+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6TjYGS-EDCEweeGqEksTcBknMQnFHCY_0 2024-03-09T15:36:05.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.432045361850876375', time=2024-03-09T17:24:15.368976+0000, chain.objs.size()=1 2024-03-09T15:36:05.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tLXL9JfH-ts5KCX58LMrDq6Nqcky3cB_0 2024-03-09T15:36:05.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17620463248016116993', time=2024-03-09T17:24:16.721941+0000, chain.objs.size()=1 2024-03-09T15:36:05.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.F9wCaWtdEGXr_EaYsRfsXMDu8DD7lZC_0 2024-03-09T15:36:05.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4446994882991926963', time=2024-03-09T17:24:16.832214+0000, chain.objs.size()=1 2024-03-09T15:36:05.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ShCI7ququbn9qAyxZu9kDuQFZjNKd3H_0 2024-03-09T15:36:05.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10197171336663115161', time=2024-03-09T17:24:16.833710+0000, chain.objs.size()=1 2024-03-09T15:36:05.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.zv0fQ0MsRbLAfs5ah1hDeMSMX75aKeD_0 2024-03-09T15:36:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7174740949795211881', time=2024-03-09T17:24:17.807730+0000, chain.objs.size()=1 2024-03-09T15:36:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.DvtjHVPG97akOosCNdXFxuCG_qyZG-j_0 2024-03-09T15:36:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16285970130732524655', time=2024-03-09T17:24:19.530367+0000, chain.objs.size()=1 2024-03-09T15:36:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.NnfVzmqNntnN_qFF93VAH_cUMzNAsIm_0 2024-03-09T15:36:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4101250681613462613', time=2024-03-09T17:24:21.320898+0000, chain.objs.size()=1 2024-03-09T15:36:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.bSOOjDaIZOMSxjbCNeE5BZ7d9Nt57g4_0 2024-03-09T15:36:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.203609946842562042', time=2024-03-09T17:24:21.642640+0000, chain.objs.size()=1 2024-03-09T15:36:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ztWAeFOtNBmBTGGCV83v0QLdtDvfnC6_0 2024-03-09T15:36:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15364117634220306062', time=2024-03-09T17:24:22.362402+0000, chain.objs.size()=1 2024-03-09T15:36:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.4YFkp4n7-0aVoe3fRDk2qDXBnML4wSN_0 2024-03-09T15:36:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.270156845504225710', time=2024-03-09T17:24:23.171807+0000, chain.objs.size()=1 2024-03-09T15:36:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.XQ_cIXM7vgEMbxdB_HDGWIp2VTzq3rc_0 2024-03-09T15:36:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11185701375027072483', time=2024-03-09T17:24:23.173248+0000, chain.objs.size()=1 2024-03-09T15:36:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.860+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.D4R7ty-Gkq9G1wFLnPR-3wGxo55yDll_0 2024-03-09T15:36:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10131681611672022010', time=2024-03-09T17:24:23.488966+0000, chain.objs.size()=1 2024-03-09T15:36:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.IQvpUaBKL6a1p5sUW1CQoLi0OWhyiRm_0 2024-03-09T15:36:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9267670738479629965', time=2024-03-09T17:24:24.714196+0000, chain.objs.size()=1 2024-03-09T15:36:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.hjUNtpoYIVUIn0JYQBS6aSVs5LzKMZK_0 2024-03-09T15:36:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11743088620210594312', time=2024-03-09T17:24:25.571170+0000, chain.objs.size()=1 2024-03-09T15:36:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.YZm1fEDxIXgIGru0-Sp0d9GcVbMH9wE_0 2024-03-09T15:36:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11737527841734751385', time=2024-03-09T17:24:25.860442+0000, chain.objs.size()=1 2024-03-09T15:36:05.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.erDyc7j9KqHIoPfWPfeHrFLS_PItbEz_0 2024-03-09T15:36:05.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5789279997362190041', time=2024-03-09T17:24:26.788810+0000, chain.objs.size()=1 2024-03-09T15:36:05.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.yr8zWIwjupazZ5aMSwdSbA-YERZwsrZ_0 2024-03-09T15:36:05.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10168208150524271694', time=2024-03-09T17:24:27.161065+0000, chain.objs.size()=1 2024-03-09T15:36:05.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.U5x3mb4432W4TAhSwKiMVGPnMQuz5o9_0 2024-03-09T15:36:05.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5604097339587876831', time=2024-03-09T17:24:30.766967+0000, chain.objs.size()=1 2024-03-09T15:36:05.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.864+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.gngaCF-7a84mnLkFEv1JqMftps-qieA_0 2024-03-09T15:36:05.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6288185698450864958', time=2024-03-09T17:24:31.062003+0000, chain.objs.size()=1 2024-03-09T15:36:05.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-hcVmwNrKUMJCP8eWktbhh7sQyD9uEr_0 2024-03-09T15:36:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2903959682492587670', time=2024-03-09T17:24:31.213058+0000, chain.objs.size()=1 2024-03-09T15:36:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.EoJs2Y2Yh9y6M7YUsX3wyUWLJYqaJl4_0 2024-03-09T15:36:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8827369034109115666', time=2024-03-09T17:24:31.329711+0000, chain.objs.size()=1 2024-03-09T15:36:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.vX5g-P39ETxBA8ofGuP69YUJBmsBN0h_0 2024-03-09T15:36:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8219189640788241071', time=2024-03-09T17:24:59.250309+0000, chain.objs.size()=1 2024-03-09T15:36:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.265__shadow_.M9F2Jc4wVJeNv3UVD-JLFeTWJdlqNC8_0 2024-03-09T15:36:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2391166323608552665', time=2024-03-09T17:25:04.702926+0000, chain.objs.size()=1 2024-03-09T15:36:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.267__shadow_.crVHmf_hOmvMpN7X86eOiSYB938XADu_0 2024-03-09T15:36:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5002129462477647288', time=2024-03-09T17:25:19.292649+0000, chain.objs.size()=1 2024-03-09T15:36:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.272__shadow_.PsV-be_ngS5hgfhI2QiTDZS0-QDlSBX_0 2024-03-09T15:36:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5652199709047688595', time=2024-03-09T17:25:51.331758+0000, chain.objs.size()=1 2024-03-09T15:36:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.283__shadow_.A9NRwbeRjPBFSaluaTzVijq7ib2VLtw_0 2024-03-09T15:36:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15835435140022766117', time=2024-03-09T17:26:07.936411+0000, chain.objs.size()=1 2024-03-09T15:36:05.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.mXq3f5KDCYhkJQxNmmNRPafcprn7Nzl_0 2024-03-09T15:36:05.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17963599655841834522', time=2024-03-09T17:26:08.081152+0000, chain.objs.size()=1 2024-03-09T15:36:05.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.Uz5s9Ry0RoNV8dpDC16y2sS7OQAphe5_0 2024-03-09T15:36:05.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17486589755555751270', time=2024-03-09T17:33:13.085201+0000, chain.objs.size()=1 2024-03-09T15:36:05.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.868+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.328__shadow_.b_m997siFrYju-XEeSPYmqLllh4z7hM_0 2024-03-09T15:36:05.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.872+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:05.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.876+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-03-09T15:36:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=40, truncated=0, next_marker='' 2024-03-09T15:36:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10826316736105414211', time=2024-03-09T17:11:41.346056+0000, chain.objs.size()=1 2024-03-09T15:36:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.136__shadow_.CtJnFwlTDWWqhJAmHQkXpo95JFhyVGS_0 2024-03-09T15:36:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1439288167255802084', time=2024-03-09T17:14:07.613904+0000, chain.objs.size()=1 2024-03-09T15:36:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.159__shadow_.aQQ23soip07rLyOfZthyHYuHbEH1vCn_0 2024-03-09T15:36:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13034935591155696214', time=2024-03-09T17:20:37.345787+0000, chain.objs.size()=1 2024-03-09T15:36:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.223__shadow_.a8TO5dWEGW3w0NLT7o5k8rwjxm5gvxS_0 2024-03-09T15:36:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15185422315467354383', time=2024-03-09T17:20:59.179672+0000, chain.objs.size()=1 2024-03-09T15:36:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.230__shadow_.SEHwM9nPm3YA_oBAYPlVRVkdcF9Kmfn_0 2024-03-09T15:36:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5093136850865590438', time=2024-03-09T17:23:59.275341+0000, chain.objs.size()=1 2024-03-09T15:36:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.J_efmb5BPC_wkl_l0zQRtmPjmvK9GH2_0 2024-03-09T15:36:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13414853053144851628', time=2024-03-09T17:23:59.277599+0000, chain.objs.size()=1 2024-03-09T15:36:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tJPUnyM-e_iEw2DOrHYfEmXewwhnyvc_0 2024-03-09T15:36:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17213255249336035688', time=2024-03-09T17:23:59.389090+0000, chain.objs.size()=1 2024-03-09T15:36:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PCy709Pe38avs5wZl-V-WSizreXJ1CK_0 2024-03-09T15:36:05.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.56043385897900273', time=2024-03-09T17:24:00.345490+0000, chain.objs.size()=1 2024-03-09T15:36:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2YRKtjnCepy4gdUIpVBuJJHvXus1Li2_0 2024-03-09T15:36:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12250636384965289100', time=2024-03-09T17:24:00.870118+0000, chain.objs.size()=1 2024-03-09T15:36:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.eQO6rjjCdp3a_p-0DTxYiP6M6HbM7-l_0 2024-03-09T15:36:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15500154903039739520', time=2024-03-09T17:24:01.475775+0000, chain.objs.size()=1 2024-03-09T15:36:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.z1zErgnUSb0XVnscjFWlBFuu8JNbd4O_0 2024-03-09T15:36:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7502979968735023287', time=2024-03-09T17:24:05.118136+0000, chain.objs.size()=1 2024-03-09T15:36:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.xQhftTYwfOEgr041M7gx4TwkMBfUSme_0 2024-03-09T15:36:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14736244162788895241', time=2024-03-09T17:24:05.625647+0000, chain.objs.size()=1 2024-03-09T15:36:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.880+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Fgo0r0Hslsf09z2gNOL9vTj6rBYJQAo_0 2024-03-09T15:36:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.884+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4542150290920113', time=2024-03-09T17:24:05.645737+0000, chain.objs.size()=1 2024-03-09T15:36:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.884+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LURR8sAkAmfpZtu93cAHfGpTU3atBIk_0 2024-03-09T15:36:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.884+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12913364694996215449', time=2024-03-09T17:24:06.236537+0000, chain.objs.size()=1 2024-03-09T15:36:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.884+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.S77yNVxu4PUbTPNnT2FPZ4XkaEMzmNO_0 2024-03-09T15:36:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.884+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9215787428182020244', time=2024-03-09T17:24:06.361942+0000, chain.objs.size()=1 2024-03-09T15:36:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.884+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PUzf-reFGa3_MktZ9mI0MiluAvKZWyH_0 2024-03-09T15:36:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7730175867956803326', time=2024-03-09T17:24:06.778151+0000, chain.objs.size()=1 2024-03-09T15:36:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ETcJzDS-hWlIUAlBSSgixlDS9hD7HYD_0 2024-03-09T15:36:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16467727555349090729', time=2024-03-09T17:24:08.896637+0000, chain.objs.size()=1 2024-03-09T15:36:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.MuUqaaVHk_nKkduUiY_fhe_MpHram_9_0 2024-03-09T15:36:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9296914922905281641', time=2024-03-09T17:24:08.907815+0000, chain.objs.size()=1 2024-03-09T15:36:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.no33QaMXpWH_iRMruUdzYtmP2_0DVIE_0 2024-03-09T15:36:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10646484564972584356', time=2024-03-09T17:24:10.047001+0000, chain.objs.size()=1 2024-03-09T15:36:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.TTsX_4QYRkLpripKgbGIJt0jnRxzroq_0 2024-03-09T15:36:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1518784517272367423', time=2024-03-09T17:24:13.803885+0000, chain.objs.size()=1 2024-03-09T15:36:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.t9rTLhHHZwHc0RI5Rge55B4gwinAf5U_0 2024-03-09T15:36:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9175691634054142979', time=2024-03-09T17:24:14.716624+0000, chain.objs.size()=1 2024-03-09T15:36:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.706rVQxM2mX6h4XTSYfxsMH09eatQ4K_0 2024-03-09T15:36:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13411029271820516775', time=2024-03-09T17:24:14.796615+0000, chain.objs.size()=1 2024-03-09T15:36:05.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.KT3RlmEycKFRsgwSgvz03xQg8COfe53_0 2024-03-09T15:36:05.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4997654962746014387', time=2024-03-09T17:24:19.470505+0000, chain.objs.size()=1 2024-03-09T15:36:05.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.888+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.DLExpIWcQhMlSfUsHFgpkS1A_NCbnjG_0 2024-03-09T15:36:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16213157712666290732', time=2024-03-09T17:24:21.910347+0000, chain.objs.size()=1 2024-03-09T15:36:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fI1WNR6KqjEph_8xo8fGivb0RE9a_GG_0 2024-03-09T15:36:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11552277829756230283', time=2024-03-09T17:24:25.432059+0000, chain.objs.size()=1 2024-03-09T15:36:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.MkiZdCOfOYS3BSEQtie8X0Drr_gVeV2_0 2024-03-09T15:36:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7486384171133232241', time=2024-03-09T17:24:26.655572+0000, chain.objs.size()=1 2024-03-09T15:36:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.g1PF_lnT4SVqPCBP4hYI-2ZGR7lPGEW_0 2024-03-09T15:36:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17291775792326694227', time=2024-03-09T17:24:26.798840+0000, chain.objs.size()=1 2024-03-09T15:36:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.54tjbqXbo0Vnx9bK60wirBYNPEXgGai_0 2024-03-09T15:36:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7452202120803187048', time=2024-03-09T17:24:27.528161+0000, chain.objs.size()=1 2024-03-09T15:36:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Sg8TAmHzrku1o-5bVotaP0OdLwNPTKn_0 2024-03-09T15:36:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16375660308915027618', time=2024-03-09T17:24:27.725439+0000, chain.objs.size()=1 2024-03-09T15:36:05.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sASDJ85ZjR2iBBFd6AmMtSAkO_wGJ1k_0 2024-03-09T15:36:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16632232384382802072', time=2024-03-09T17:24:28.677108+0000, chain.objs.size()=1 2024-03-09T15:36:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.zSutHmbVe3bskcL-Cu7VnAIRYqBaMTv_0 2024-03-09T15:36:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7923347435724307851', time=2024-03-09T17:24:28.678604+0000, chain.objs.size()=1 2024-03-09T15:36:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.5-N0bcp-Oim-b-OL9gneOzT0GugX5sT_0 2024-03-09T15:36:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6190143611831081925', time=2024-03-09T17:24:29.154925+0000, chain.objs.size()=1 2024-03-09T15:36:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.892+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7ekfRbYcrFrfilonDT2E4x9ETL7EZDs_0 2024-03-09T15:36:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.896+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6577765246728365384', time=2024-03-09T17:24:30.297919+0000, chain.objs.size()=1 2024-03-09T15:36:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.896+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.xWX13YWThXLSWPvoXqP4O5HxqVx-OlK_0 2024-03-09T15:36:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.896+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10622636091400455762', time=2024-03-09T17:24:30.777042+0000, chain.objs.size()=1 2024-03-09T15:36:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.896+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.5xg02l6k7YlkG0uv1fqq4vZXwILmpJ1_0 2024-03-09T15:36:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.896+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16086467650024172378', time=2024-03-09T17:24:31.160615+0000, chain.objs.size()=1 2024-03-09T15:36:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.896+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.XHjc0U4gaNd6hvBCPu0aSASdTLFg1A2_0 2024-03-09T15:36:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.896+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15001185146307535713', time=2024-03-09T17:24:31.357771+0000, chain.objs.size()=1 2024-03-09T15:36:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.896+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9ihmIUxO6Dngf-6BEiaDpZ0OzSmlK0x_0 2024-03-09T15:36:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.896+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4539245603274709388', time=2024-03-09T17:25:27.976403+0000, chain.objs.size()=1 2024-03-09T15:36:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.896+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.275__shadow_.C5nNMYQ83TcJvugag-fp29cVQv_zlmL_0 2024-03-09T15:36:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.900+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7360187271232659878', time=2024-03-09T17:25:28.001439+0000, chain.objs.size()=1 2024-03-09T15:36:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.900+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.275__shadow_.9lnBM02I1kLqZsrM1GttojZE_yeWYRE_0 2024-03-09T15:36:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.900+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4311113509820395393', time=2024-03-09T17:25:42.653327+0000, chain.objs.size()=1 2024-03-09T15:36:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.900+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.280__shadow_.LR8U5dUSGdNZzvGXVEexRG0L31N-NwO_0 2024-03-09T15:36:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.900+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9399004777294164686', time=2024-03-09T17:26:01.407459+0000, chain.objs.size()=1 2024-03-09T15:36:05.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.900+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.A_zgsRpWIqzTi60aqguIPwuT_tmCaO2_0 2024-03-09T15:36:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.904+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:05.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.908+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-03-09T15:36:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:05.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=53, truncated=0, next_marker='' 2024-03-09T15:36:05.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_bzJDyJ4dnZV0LqBOE_PxtQObzMZnSpG', time=2024-03-09T17:13:40.211763+0000, chain.objs.size()=2 2024-03-09T15:36:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__shadow_.mVDOW6wsxVL_qRPW6HCY5cGUMmOfo3K_0 2024-03-09T15:36:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__shadow_.mVDOW6wsxVL_qRPW6HCY5cGUMmOfo3K_1 2024-03-09T15:36:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5302564816673258465', time=2024-03-09T17:14:49.961582+0000, chain.objs.size()=1 2024-03-09T15:36:05.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.172__shadow_.QghVSOHWy_zbA1pIgTjkVbH0-2KnzK7_0 2024-03-09T15:36:05.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4233224005542883610', time=2024-03-09T17:21:14.289305+0000, chain.objs.size()=1 2024-03-09T15:36:05.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.235__shadow_.Za5QzEUska_BLNI4rn_XxEW-0gFrEEU_0 2024-03-09T15:36:05.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3648543124825282211', time=2024-03-09T17:21:46.311645+0000, chain.objs.size()=1 2024-03-09T15:36:05.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.245__shadow_.sOUxIgthX6enMHoFz6vR29ItbAwO9LV_0 2024-03-09T15:36:05.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10814646200727891637', time=2024-03-09T17:23:59.392552+0000, chain.objs.size()=1 2024-03-09T15:36:05.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.rJy0oIEHJgmNybv9HEsuR0LXVhrvtp2_0 2024-03-09T15:36:05.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10887037002177654911', time=2024-03-09T17:23:59.660488+0000, chain.objs.size()=1 2024-03-09T15:36:05.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.AyJ33etU4YXccwG73SUvZXawGBIYyNG_0 2024-03-09T15:36:05.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9776448825571419754', time=2024-03-09T17:23:59.662234+0000, chain.objs.size()=1 2024-03-09T15:36:05.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qte0zmoTZeAcbCxfkRABfo8U6VAIIEd_0 2024-03-09T15:36:05.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2128629662269721364', time=2024-03-09T17:23:59.947946+0000, chain.objs.size()=1 2024-03-09T15:36:05.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Zof7_6-EhxhS2CcHogxlxvWwX5loqC4_0 2024-03-09T15:36:05.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5274045123737983552', time=2024-03-09T17:24:00.075631+0000, chain.objs.size()=1 2024-03-09T15:36:05.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.UIfvbMLdMhaHY_zlIRYB7QA7VTLqRAb_0 2024-03-09T15:36:05.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7728431541386232357', time=2024-03-09T17:24:00.081319+0000, chain.objs.size()=1 2024-03-09T15:36:05.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Qh9PR4tyK0GaNnvW2SuRyf8n5EksqLq_0 2024-03-09T15:36:05.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4385802267752530748', time=2024-03-09T17:24:00.526261+0000, chain.objs.size()=1 2024-03-09T15:36:05.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.912+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.RF-wYj2_txnNAboTyJrP57lg1ljyklk_0 2024-03-09T15:36:05.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7114853046308619393', time=2024-03-09T17:24:01.484542+0000, chain.objs.size()=1 2024-03-09T15:36:05.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.nh2WniQaP1jNWfOsTinjwY9jg6qruaM_0 2024-03-09T15:36:05.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12969923938420465823', time=2024-03-09T17:24:01.961097+0000, chain.objs.size()=1 2024-03-09T15:36:05.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.bvnXb7trwpDJzKVtSrwt9rJREl4bgr3_0 2024-03-09T15:36:05.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4444442525015873884', time=2024-03-09T17:24:03.904581+0000, chain.objs.size()=1 2024-03-09T15:36:05.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1HeBnBhsGSHT8zDDnwYenfY3Ef-cdkz_0 2024-03-09T15:36:05.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7165874311269319535', time=2024-03-09T17:24:03.971584+0000, chain.objs.size()=1 2024-03-09T15:36:05.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8-OwIymcltRqY-uNVRglLrezTM9wQ1F_0 2024-03-09T15:36:05.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13242554979326739965', time=2024-03-09T17:24:05.497293+0000, chain.objs.size()=1 2024-03-09T15:36:05.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.VpNKi1mMVWGoDF6GdWsG85Yr2IKlvaS_0 2024-03-09T15:36:05.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2267648054710686587', time=2024-03-09T17:24:06.733982+0000, chain.objs.size()=1 2024-03-09T15:36:05.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.xJMwiMpR3cPIfC8vzZs0QZpQw7NKEAm_0 2024-03-09T15:36:05.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13193163113154850785', time=2024-03-09T17:24:09.218516+0000, chain.objs.size()=1 2024-03-09T15:36:05.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.uZ5f9M9L92y54Q_T984diF70F07lxxf_0 2024-03-09T15:36:05.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17764087240800689', time=2024-03-09T17:24:09.305757+0000, chain.objs.size()=1 2024-03-09T15:36:05.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.SLRvgHmRnY-9zYulNakw6xE9WL8eaPD_0 2024-03-09T15:36:05.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9639322159837118514', time=2024-03-09T17:24:09.726751+0000, chain.objs.size()=1 2024-03-09T15:36:05.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.920+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Y758bgiD2PEy5mRTC7kQbWdmhyoZR3g_0 2024-03-09T15:36:05.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.924+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5361965596189578798', time=2024-03-09T17:24:10.336311+0000, chain.objs.size()=1 2024-03-09T15:36:05.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.924+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.T4Ffbv_7t2Vzdb_xwHh8aDhIgu36BHw_0 2024-03-09T15:36:05.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.924+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15704393421117411015', time=2024-03-09T17:24:10.829048+0000, chain.objs.size()=1 2024-03-09T15:36:05.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.924+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.OZ6nsvPTiD8OiCk9m441-quMRORTyiQ_0 2024-03-09T15:36:05.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.924+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2343617824580626361', time=2024-03-09T17:24:12.653479+0000, chain.objs.size()=1 2024-03-09T15:36:05.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.924+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fCahOlZ6ZhWbOXQHwmm8ckvDDIadS9a_0 2024-03-09T15:36:05.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.297752069559440047', time=2024-03-09T17:24:13.107119+0000, chain.objs.size()=1 2024-03-09T15:36:05.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.yWpNlUXOznZ4bsrVuYLjZHbLy0sULXu_0 2024-03-09T15:36:05.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12816732880514003358', time=2024-03-09T17:24:14.306875+0000, chain.objs.size()=1 2024-03-09T15:36:05.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.dsh4X8k5cmXkgb7U8dmmUhGonZAAXcK_0 2024-03-09T15:36:05.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5901849413442300978', time=2024-03-09T17:24:14.309189+0000, chain.objs.size()=1 2024-03-09T15:36:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.jN8uQqZASI87ThTA3o68TISLmBHq6qv_0 2024-03-09T15:36:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4868210333485154862', time=2024-03-09T17:24:16.697618+0000, chain.objs.size()=1 2024-03-09T15:36:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ABiHATy__0o0Ugcs4ndeGclSpaoHeJd_0 2024-03-09T15:36:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10913458758970250616', time=2024-03-09T17:24:19.209010+0000, chain.objs.size()=1 2024-03-09T15:36:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sXvakWS9FU_9gyvdpWo0NB2duVfdLoM_0 2024-03-09T15:36:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6084461360701689771', time=2024-03-09T17:24:19.423771+0000, chain.objs.size()=1 2024-03-09T15:36:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.lJj3IdiQiiTwvJE4Fl5Pxs0yBfHg4Ob_0 2024-03-09T15:36:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10816207504424821616', time=2024-03-09T17:24:20.848526+0000, chain.objs.size()=1 2024-03-09T15:36:05.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.zouAGzGDU9hYVau0CjHeRlDfbNMrDpo_0 2024-03-09T15:36:05.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1231066637137690339', time=2024-03-09T17:24:20.878586+0000, chain.objs.size()=1 2024-03-09T15:36:05.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.928+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.l2H0wSvlQ69IkUXgb1tuqizgxsK1NdG_0 2024-03-09T15:36:05.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16115054198943550087', time=2024-03-09T17:24:21.654265+0000, chain.objs.size()=1 2024-03-09T15:36:05.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.vpKFzo04JCQIPzdwK3r8h13GtEz7JtJ_0 2024-03-09T15:36:05.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2985299668090655704', time=2024-03-09T17:24:22.141758+0000, chain.objs.size()=1 2024-03-09T15:36:05.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tJ64_2JHwuqSq71-RA-ujx_e7HT339e_0 2024-03-09T15:36:05.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17013123488717407662', time=2024-03-09T17:24:22.157341+0000, chain.objs.size()=1 2024-03-09T15:36:05.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Y80fvJLmS6IcCx9l58xHcYXaGfQ9sVQ_0 2024-03-09T15:36:05.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1861397020887520654', time=2024-03-09T17:24:22.797820+0000, chain.objs.size()=1 2024-03-09T15:36:05.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iJagBVVc40iFSoboKh3kzTekKjWvanq_0 2024-03-09T15:36:05.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16019930153119475138', time=2024-03-09T17:24:25.022062+0000, chain.objs.size()=1 2024-03-09T15:36:05.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fQ9j2P397-9dKTk3kQ_IvJxSo7T2INP_0 2024-03-09T15:36:05.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2365306335754491748', time=2024-03-09T17:24:25.115218+0000, chain.objs.size()=1 2024-03-09T15:36:05.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tc2K5R9YGOeW2kIJX0qGgffW74D3hhA_0 2024-03-09T15:36:05.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16987202294952918986', time=2024-03-09T17:24:26.802548+0000, chain.objs.size()=1 2024-03-09T15:36:05.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.uUgkrPAWBP1226u0xyOVy52zSSP3hoh_0 2024-03-09T15:36:05.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7960633766679380712', time=2024-03-09T17:24:27.430572+0000, chain.objs.size()=1 2024-03-09T15:36:05.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.KtB7JakspJITg2xNCmXvU1XrtrhGGZZ_0 2024-03-09T15:36:05.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2038127372430858785', time=2024-03-09T17:24:27.521596+0000, chain.objs.size()=1 2024-03-09T15:36:05.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.V0u9LrHStGz3Te_zrY9M-PnGwIyzq4A_0 2024-03-09T15:36:05.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18001486191698549965', time=2024-03-09T17:24:29.214113+0000, chain.objs.size()=1 2024-03-09T15:36:05.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.4-DROxVL4NHMsk9bo0wxCZLwsbRNkPs_0 2024-03-09T15:36:05.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4939906867365933701', time=2024-03-09T17:24:29.618528+0000, chain.objs.size()=1 2024-03-09T15:36:05.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.932+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9mVhOt5pslf-yhIqaj1a_Y8Bf9GT0va_0 2024-03-09T15:36:05.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.936+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5099905370711281877', time=2024-03-09T17:24:30.304359+0000, chain.objs.size()=1 2024-03-09T15:36:05.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.936+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2oGvGXJNKfIZTaECWsbT8UUYca-1t8Z_0 2024-03-09T15:36:05.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.936+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12058401537567946322', time=2024-03-09T17:24:45.439368+0000, chain.objs.size()=1 2024-03-09T15:36:05.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.936+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.260__shadow_.vtlops0uD0uUCwzGs68o3v22bCj8XQE_0 2024-03-09T15:36:05.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.936+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11503916325711755915', time=2024-03-09T17:25:07.555306+0000, chain.objs.size()=1 2024-03-09T15:36:05.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.936+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.268__shadow_.MVjuK7fxl3uPRs9XVzHEEtc4PMrLCf1_0 2024-03-09T15:36:05.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.936+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13700691535372965402', time=2024-03-09T17:25:45.503564+0000, chain.objs.size()=1 2024-03-09T15:36:05.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.936+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.281__shadow_.28g_mUALzdCigda-G3MEupEGw5Bz22w_0 2024-03-09T15:36:05.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.936+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4349589985526588078', time=2024-03-09T17:25:55.606661+0000, chain.objs.size()=1 2024-03-09T15:36:05.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.936+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.284__shadow_.1miaEpRq8yB9f87p1MlovMUEq6Om5tK_0 2024-03-09T15:36:05.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.936+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14532422365026644749', time=2024-03-09T17:26:07.740063+0000, chain.objs.size()=1 2024-03-09T15:36:05.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.936+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.UGJn9zfS60Wv9O92KPCuATSj6hLldzc_0 2024-03-09T15:36:05.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.940+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9711936292744008782', time=2024-03-09T17:26:08.086635+0000, chain.objs.size()=1 2024-03-09T15:36:05.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.940+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.eLuFYNR4noIfvrEVkEw4ALs_8W5pxEO_0 2024-03-09T15:36:05.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.940+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16911547552661965659', time=2024-03-09T17:26:16.828523+0000, chain.objs.size()=1 2024-03-09T15:36:05.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.940+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.289__shadow_.3gIvkghCzw-ll2X-jtdqhpDKWFsaqmH_0 2024-03-09T15:36:05.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.940+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2168457265803508918', time=2024-03-09T17:26:46.763052+0000, chain.objs.size()=1 2024-03-09T15:36:05.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.940+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.299__shadow_.DMB44WJC78Kg777_8k-RcCLrAC4zr0C_0 2024-03-09T15:36:05.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.940+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12736661990373709240', time=2024-03-09T17:31:05.122415+0000, chain.objs.size()=1 2024-03-09T15:36:05.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.940+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.313__shadow_.lJLjJrQTXPou66LjcUNFJkWltX-MGSk_0 2024-03-09T15:36:05.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.940+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11637592368390454200', time=2024-03-09T17:33:00.196756+0000, chain.objs.size()=1 2024-03-09T15:36:05.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.940+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.325__shadow_.ES-qpwzkXfG2FtpEdw1g-A722taJU_q_0 2024-03-09T15:36:05.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.940+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:05.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.944+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-03-09T15:36:05.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=38, truncated=0, next_marker='' 2024-03-09T15:36:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10867519162986795036', time=2024-03-09T17:12:41.024090+0000, chain.objs.size()=1 2024-03-09T15:36:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.5hgh_rvyO_O8wivYWPqowZl-NNSfkYA_0 2024-03-09T15:36:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14212174193125450070', time=2024-03-09T17:14:44.817564+0000, chain.objs.size()=1 2024-03-09T15:36:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.170__shadow_.x71Klxe36UGwvwxUB8_MaO68aBNlsZX_0 2024-03-09T15:36:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14227995174711248751', time=2024-03-09T17:21:11.477165+0000, chain.objs.size()=1 2024-03-09T15:36:05.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.234__shadow_.ycZ049Mu6hFjcM2Ni0FGCKA1bO03XmX_0 2024-03-09T15:36:05.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12281237987247362482', time=2024-03-09T17:21:28.945968+0000, chain.objs.size()=1 2024-03-09T15:36:05.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.239__shadow_.KPBsDF6HNkhXBPcschkAacy1CHUWrbO_0 2024-03-09T15:36:05.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14158690866140244140', time=2024-03-09T17:21:34.863407+0000, chain.objs.size()=1 2024-03-09T15:36:05.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.241__shadow_.jwO7paXGKKTrg4jl26tD_86Z_L8PkBc_0 2024-03-09T15:36:05.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11429422925895829970', time=2024-03-09T17:21:40.677431+0000, chain.objs.size()=1 2024-03-09T15:36:05.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.243__shadow_.fHJ-Kded9ZB2Px2dDP6MHcorB26NOtJ_0 2024-03-09T15:36:05.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2712520872710696997', time=2024-03-09T17:23:59.380760+0000, chain.objs.size()=1 2024-03-09T15:36:05.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.mQ2gjtXWMBpn55FMRk4zwLezP0WDLw3_0 2024-03-09T15:36:05.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7607751772014795696', time=2024-03-09T17:24:01.096958+0000, chain.objs.size()=1 2024-03-09T15:36:05.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.bB94WOgMK6o3mnUxsZ5U0J7G4gqNBo4_0 2024-03-09T15:36:05.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13551120310672816380', time=2024-03-09T17:24:03.535350+0000, chain.objs.size()=1 2024-03-09T15:36:05.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.JFklEUA3f36WuWIpwLbngvmqHEalGTi_0 2024-03-09T15:36:05.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18031222218152969037', time=2024-03-09T17:24:04.476772+0000, chain.objs.size()=1 2024-03-09T15:36:05.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.BMAMdhNqNge6-Za1zkhEeOpRtrm4ox__0 2024-03-09T15:36:05.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3143673447755679099', time=2024-03-09T17:24:05.985456+0000, chain.objs.size()=1 2024-03-09T15:36:05.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.CzYIHL7zCskT8CrNxA3jKh1UIgne9iF_0 2024-03-09T15:36:05.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2344999777629723592', time=2024-03-09T17:24:07.939642+0000, chain.objs.size()=1 2024-03-09T15:36:05.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.948+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_._gkAxCyfqJEmNcsq-1rDcq2J5UFqyuK_0 2024-03-09T15:36:05.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11415097497985608573', time=2024-03-09T17:24:08.922888+0000, chain.objs.size()=1 2024-03-09T15:36:05.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.BtrWWOm8NHeuhAS7sh5vp-o114ZyMmj_0 2024-03-09T15:36:05.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14884796587504068403', time=2024-03-09T17:24:09.143403+0000, chain.objs.size()=1 2024-03-09T15:36:05.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.YaOBv9qoBISx_oVdzrgJT-HM57wyZGI_0 2024-03-09T15:36:05.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9063747041820867816', time=2024-03-09T17:24:09.512852+0000, chain.objs.size()=1 2024-03-09T15:36:05.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.pPk_Bt9EfAR1Xm8JTIUuO0h826TJ5Fh_0 2024-03-09T15:36:05.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5226724549670653209', time=2024-03-09T17:24:09.728046+0000, chain.objs.size()=1 2024-03-09T15:36:05.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ZwyTbjftwdfXo5OAktFP3BEvlYeQpK5_0 2024-03-09T15:36:05.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11062101507554493764', time=2024-03-09T17:24:10.047757+0000, chain.objs.size()=1 2024-03-09T15:36:05.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.asITB34Tvm55VJc01fHWmnoC3fANaap_0 2024-03-09T15:36:05.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9164076955163386129', time=2024-03-09T17:24:10.831441+0000, chain.objs.size()=1 2024-03-09T15:36:05.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.pn0si3rEGZihEHLmZABxpkLU0960fPk_0 2024-03-09T15:36:05.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8187527117447306687', time=2024-03-09T17:24:11.132658+0000, chain.objs.size()=1 2024-03-09T15:36:05.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fZ9HZa0Hm5IdaHdxLYbxldana1jM92M_0 2024-03-09T15:36:05.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.272846546871767210', time=2024-03-09T17:24:12.215326+0000, chain.objs.size()=1 2024-03-09T15:36:05.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.12pGX5iYaBREwn5aKxPqCky-YMbyCvG_0 2024-03-09T15:36:05.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6198310248961780326', time=2024-03-09T17:24:13.152638+0000, chain.objs.size()=1 2024-03-09T15:36:05.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_._-9JaGpNsP5Fzv9d36RmTMQiqGCH71T_0 2024-03-09T15:36:05.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13098499439887921776', time=2024-03-09T17:24:13.155205+0000, chain.objs.size()=1 2024-03-09T15:36:05.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.B5KCkmcK0c1oRTqaias2VWEMZTkfSt9_0 2024-03-09T15:36:05.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2151159884098690066', time=2024-03-09T17:24:14.720474+0000, chain.objs.size()=1 2024-03-09T15:36:05.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.952+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.u2s7VaSG0uCYpkh2A-nuaIfBvGkaHI1_0 2024-03-09T15:36:05.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1327551429443889144', time=2024-03-09T17:24:15.199719+0000, chain.objs.size()=1 2024-03-09T15:36:05.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fowFQXDwdWcWo5CwbrC2stsqnHrk5cv_0 2024-03-09T15:36:05.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6639366359503948065', time=2024-03-09T17:24:16.274771+0000, chain.objs.size()=1 2024-03-09T15:36:05.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.SxxgOcaBh5ysnvsohiXwuOqLkEs0toO_0 2024-03-09T15:36:05.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17001410791463474533', time=2024-03-09T17:24:18.170324+0000, chain.objs.size()=1 2024-03-09T15:36:05.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.pKuhns1xdV7NQHVIFoSE2LkHOkIGQWp_0 2024-03-09T15:36:05.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4440854936528879042', time=2024-03-09T17:24:18.427567+0000, chain.objs.size()=1 2024-03-09T15:36:05.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.puPdJdDyJVsAzMozXXxrwkkkop_JWtd_0 2024-03-09T15:36:05.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6177263808210888756', time=2024-03-09T17:24:20.587062+0000, chain.objs.size()=1 2024-03-09T15:36:05.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.cwyAv5ER1afMT2zDavD5OFHU4bw115f_0 2024-03-09T15:36:05.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10142084977899677009', time=2024-03-09T17:24:21.129434+0000, chain.objs.size()=1 2024-03-09T15:36:05.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.nb9XtPRhsG3Ot_A3hX9xvlpRC9vc-Cz_0 2024-03-09T15:36:05.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.125069097087603129', time=2024-03-09T17:24:21.415200+0000, chain.objs.size()=1 2024-03-09T15:36:05.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Cuugeemq4pj0Y4Mbg-yzxnad7UhzMWE_0 2024-03-09T15:36:05.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16240262537813025283', time=2024-03-09T17:24:25.516788+0000, chain.objs.size()=1 2024-03-09T15:36:05.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.j0QVj7BNpkeQ2jQnR04nNonaLxm7BY8_0 2024-03-09T15:36:05.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10020873386483522645', time=2024-03-09T17:24:26.378867+0000, chain.objs.size()=1 2024-03-09T15:36:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.nqsds3KMPLwOQbZ66_v5p4RzSNRTaiY_0 2024-03-09T15:36:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6887646263554543550', time=2024-03-09T17:24:27.560070+0000, chain.objs.size()=1 2024-03-09T15:36:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.b7eSZ8Vz_ARy1JAjUEzlIF7nq6NCJF7_0 2024-03-09T15:36:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16267506785131669953', time=2024-03-09T17:24:30.376596+0000, chain.objs.size()=1 2024-03-09T15:36:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9LOORZiCm2gLU-1Ef4YiU_dp4eYkB9z_0 2024-03-09T15:36:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15281380678102472968', time=2024-03-09T17:24:31.276533+0000, chain.objs.size()=1 2024-03-09T15:36:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.956+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iMSsDgxRarjvP900X1ZOigX6QbjWNQf_0 2024-03-09T15:36:05.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.960+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7810851833529767430', time=2024-03-09T17:25:10.544731+0000, chain.objs.size()=1 2024-03-09T15:36:05.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.960+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.269__shadow_.XKcgDIELyJOV0SJczB6-g5AEcVYx4yx_0 2024-03-09T15:36:05.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.960+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10530511727896827404', time=2024-03-09T17:25:51.344086+0000, chain.objs.size()=1 2024-03-09T15:36:05.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.960+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.283__shadow_.NLoIom-uBYZ6_MLgipR8r4aN5_VP36y_0 2024-03-09T15:36:05.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.960+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11639695117081648581', time=2024-03-09T17:26:01.464531+0000, chain.objs.size()=1 2024-03-09T15:36:05.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.960+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.JIdKg9CWkynJD-iIThZHuVDtBQOxlP7_0 2024-03-09T15:36:05.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.964+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:05.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.968+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-03-09T15:36:05.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.968+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:05.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:05.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=48, truncated=0, next_marker='' 2024-03-09T15:36:05.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_cicXv_JH63QTRrhYiKiV4ZAEtKcpTkn', time=2024-03-09T17:12:52.046923+0000, chain.objs.size()=2 2024-03-09T15:36:05.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.KA1lMyW80VwjedXhiVeV3M_6SbUuwxH_0 2024-03-09T15:36:05.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.KA1lMyW80VwjedXhiVeV3M_6SbUuwxH_1 2024-03-09T15:36:05.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13724775858182361527', time=2024-03-09T17:13:11.614440+0000, chain.objs.size()=3 2024-03-09T15:36:05.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__multipart_foo-STANDARD.2~_KGRWYXnNOpD1_SBpRdNSsiifJlum2B.1 2024-03-09T15:36:05.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__multipart_foo-STANDARD.2~_KGRWYXnNOpD1_SBpRdNSsiifJlum2B.2 2024-03-09T15:36:05.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__multipart_foo-STANDARD.2~_KGRWYXnNOpD1_SBpRdNSsiifJlum2B.3 2024-03-09T15:36:05.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14145977644130206987', time=2024-03-09T17:14:42.188922+0000, chain.objs.size()=1 2024-03-09T15:36:05.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.169__shadow_.kZZuB4a7uEGH15VcliwSdvlq3J5X-Uo_0 2024-03-09T15:36:05.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4540985484519342647', time=2024-03-09T17:21:11.481266+0000, chain.objs.size()=1 2024-03-09T15:36:05.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.234__shadow_.sN5gPGqKMcfX1PUpzAU8zJNzEN_NLY6_0 2024-03-09T15:36:05.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5363896639334536305', time=2024-03-09T17:21:18.711550+0000, chain.objs.size()=1 2024-03-09T15:36:05.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.236__shadow_.52k8kvQT5yiXHKN3lAisfsdj1tgImlM_0 2024-03-09T15:36:05.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12474145006581545584', time=2024-03-09T17:21:46.326149+0000, chain.objs.size()=1 2024-03-09T15:36:05.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.245__shadow_.i8yKTMw5QBBhH2G1CWdQDng_sAcVhgo_0 2024-03-09T15:36:05.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14978936485410477100', time=2024-03-09T17:22:09.488225+0000, chain.objs.size()=1 2024-03-09T15:36:05.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.253__shadow_.-NJfG9MwBavPRkv5PAAq0XTM_MjxxWW_0 2024-03-09T15:36:05.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17235368695014571609', time=2024-03-09T17:23:59.974218+0000, chain.objs.size()=1 2024-03-09T15:36:05.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fEVG-5n5GelY84oZXx35xGf6G9uAIbQ_0 2024-03-09T15:36:05.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15330835255582590654', time=2024-03-09T17:24:00.343955+0000, chain.objs.size()=1 2024-03-09T15:36:05.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.972+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.5IhijE4DnW7lga9NJ1YSIHrzWpj71Am_0 2024-03-09T15:36:05.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17643910117528513251', time=2024-03-09T17:24:00.633661+0000, chain.objs.size()=1 2024-03-09T15:36:05.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.D2MWVBDMDsGgjvXD0isYKFZUIr5mDeF_0 2024-03-09T15:36:05.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16787705125013186978', time=2024-03-09T17:24:01.475809+0000, chain.objs.size()=1 2024-03-09T15:36:05.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.SJZ3rVkUkpiiWFpDt6Y-4h4pO6vaF5N_0 2024-03-09T15:36:05.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15767744869169832159', time=2024-03-09T17:24:04.819138+0000, chain.objs.size()=1 2024-03-09T15:36:05.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.YtRU4syqQflAYgBngEcBcWDeJQ0cEgM_0 2024-03-09T15:36:05.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9338571359299949361', time=2024-03-09T17:24:07.880737+0000, chain.objs.size()=1 2024-03-09T15:36:05.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.T3iQLLW-8tUXtAMK9cZIndqlPU9aXK7_0 2024-03-09T15:36:05.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11660833853107009185', time=2024-03-09T17:24:08.329636+0000, chain.objs.size()=1 2024-03-09T15:36:05.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Rypf2guaFj_esWpqf7Vtf4ITYtECsXp_0 2024-03-09T15:36:05.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8233594635737746571', time=2024-03-09T17:24:10.577697+0000, chain.objs.size()=1 2024-03-09T15:36:05.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.POGLMm30JPD4Ltm3bLB0WAIchSq8nxb_0 2024-03-09T15:36:05.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7861697677768447008', time=2024-03-09T17:24:11.005951+0000, chain.objs.size()=1 2024-03-09T15:36:05.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.s9kNkyra78fv1YwXSobyBXVhcQlaWpq_0 2024-03-09T15:36:05.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6341001189302661977', time=2024-03-09T17:24:11.019739+0000, chain.objs.size()=1 2024-03-09T15:36:05.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Q9RjoiFf_NiN-YCOgyJt1nFHc-BMfLk_0 2024-03-09T15:36:05.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15957504243402356970', time=2024-03-09T17:24:12.213367+0000, chain.objs.size()=1 2024-03-09T15:36:05.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ILnP-g_00Ajd3LKhrBEfjTOSK2BN1bM_0 2024-03-09T15:36:05.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7034376489037735863', time=2024-03-09T17:24:12.663000+0000, chain.objs.size()=1 2024-03-09T15:36:05.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9ug9c8KeXG7xX3CcY21TQrdbe8nsj0-_0 2024-03-09T15:36:05.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6445161865493487846', time=2024-03-09T17:24:13.400872+0000, chain.objs.size()=1 2024-03-09T15:36:05.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.980+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.G6zJQzYQFyyrLK4R-__S1Pb_3pyuvsl_0 2024-03-09T15:36:05.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15552038024357292570', time=2024-03-09T17:24:13.530788+0000, chain.objs.size()=1 2024-03-09T15:36:05.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.omHIBRUE1XmRgTDYybQMVmjEJd-Ql2L_0 2024-03-09T15:36:05.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12495714163975217331', time=2024-03-09T17:24:14.552211+0000, chain.objs.size()=1 2024-03-09T15:36:05.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0gStfKVnOkGss9-JmsOGiornxYE160P_0 2024-03-09T15:36:05.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12105478308056331222', time=2024-03-09T17:24:15.238349+0000, chain.objs.size()=1 2024-03-09T15:36:05.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HMc8OXwzRO7mtjQqoZfrdQudI4q1wx__0 2024-03-09T15:36:05.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1485229343799574266', time=2024-03-09T17:24:16.340554+0000, chain.objs.size()=1 2024-03-09T15:36:05.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.KfsQRsxHfs9fi1-FSKZzOTddFLRyggc_0 2024-03-09T15:36:05.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8356126579152503861', time=2024-03-09T17:24:16.342191+0000, chain.objs.size()=1 2024-03-09T15:36:05.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.R-HV6JRfT8VP7i-uBonT9oj1QTcgY40_0 2024-03-09T15:36:05.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15238378738183879980', time=2024-03-09T17:24:17.270855+0000, chain.objs.size()=1 2024-03-09T15:36:05.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.R96XbJ-067ANCUPZUDGmGX7OQrUB016_0 2024-03-09T15:36:05.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11519504576230944821', time=2024-03-09T17:24:17.714827+0000, chain.objs.size()=1 2024-03-09T15:36:05.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.euE7jpFkBQL651jKlOJ3OjvQ2TZcSPR_0 2024-03-09T15:36:05.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11095594869521731416', time=2024-03-09T17:24:17.716923+0000, chain.objs.size()=1 2024-03-09T15:36:05.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.AyJlJ9yUnoeC10R69oGt9QLWrvWkim5_0 2024-03-09T15:36:05.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5095897037493000304', time=2024-03-09T17:24:22.138194+0000, chain.objs.size()=1 2024-03-09T15:36:05.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.gmSqqAk6hIqS-1dvPyTq7EexB-LOcjg_0 2024-03-09T15:36:05.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17315856144407752755', time=2024-03-09T17:24:22.265381+0000, chain.objs.size()=1 2024-03-09T15:36:05.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.cCC7Ti9Qyib6qIWy2OofxYj4QRxLkn9_0 2024-03-09T15:36:05.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10506004594460725568', time=2024-03-09T17:24:25.510339+0000, chain.objs.size()=1 2024-03-09T15:36:05.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.984+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9_Jd46BD-VVci1T04jNmXGx8R8iuyXG_0 2024-03-09T15:36:05.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14466707493940541819', time=2024-03-09T17:24:25.859448+0000, chain.objs.size()=1 2024-03-09T15:36:05.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.VjKKWV2n8XfjnbYdNhnCFDiFYatycPu_0 2024-03-09T15:36:05.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1784026447209507215', time=2024-03-09T17:24:26.947740+0000, chain.objs.size()=1 2024-03-09T15:36:05.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.k_x4cbw3y2b2RoBNQHoCIho0S98ifps_0 2024-03-09T15:36:05.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16299882207755519000', time=2024-03-09T17:24:27.162963+0000, chain.objs.size()=1 2024-03-09T15:36:05.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.dqW3-Szok1IzglhM4iYkNlemw0sN9cp_0 2024-03-09T15:36:05.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3828709698286185479', time=2024-03-09T17:24:27.865544+0000, chain.objs.size()=1 2024-03-09T15:36:05.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.FXOv03L4b2aDNh2e_Que63U-wAZ4tXf_0 2024-03-09T15:36:05.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12235214578616227218', time=2024-03-09T17:24:27.869241+0000, chain.objs.size()=1 2024-03-09T15:36:05.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.V1AlSTUElYKEjUJemUIlGk4oFNnVVfw_0 2024-03-09T15:36:05.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3442694700820411635', time=2024-03-09T17:24:27.870857+0000, chain.objs.size()=1 2024-03-09T15:36:05.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.pHvce3Ins3Vb6HnioqlQAMaCi5E9u1F_0 2024-03-09T15:36:05.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11747854713996270524', time=2024-03-09T17:24:28.638075+0000, chain.objs.size()=1 2024-03-09T15:36:05.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.SzpBP9eRSeMP7n_tuelb91hmT1IqVVD_0 2024-03-09T15:36:05.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16501722962119764786', time=2024-03-09T17:24:29.117375+0000, chain.objs.size()=1 2024-03-09T15:36:05.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.yQgc8BvWCejB9HlujDIN9_-balS7Xrp_0 2024-03-09T15:36:05.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5333732001854671163', time=2024-03-09T17:24:29.586602+0000, chain.objs.size()=1 2024-03-09T15:36:05.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.DF7jUAhWMdWo8kzRf__ynK8WL4P4LOa_0 2024-03-09T15:36:05.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17085134038704425973', time=2024-03-09T17:24:29.990855+0000, chain.objs.size()=1 2024-03-09T15:36:05.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.VNG2LK8jDlARQ_OG03skiUzkrDipg5I_0 2024-03-09T15:36:05.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10712407438657580766', time=2024-03-09T17:24:31.038828+0000, chain.objs.size()=1 2024-03-09T15:36:05.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.988+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.pXN_Oi-RkvZ4W_y0z9yEyjPWsCns41C_0 2024-03-09T15:36:05.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.992+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6912907871882346935', time=2024-03-09T17:24:39.904198+0000, chain.objs.size()=1 2024-03-09T15:36:05.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.992+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.258__shadow_.m6HH_Wq-GxmEY3Z1IOkV0rv1LHb0MLi_0 2024-03-09T15:36:05.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.992+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8896322413188577382', time=2024-03-09T17:24:50.883020+0000, chain.objs.size()=1 2024-03-09T15:36:05.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.992+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.262__shadow_.qAQ_f6dm8sy007D_8dw8LfTWN2mBjCH_0 2024-03-09T15:36:05.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.992+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8992156770557521129', time=2024-03-09T17:25:04.700232+0000, chain.objs.size()=1 2024-03-09T15:36:05.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.992+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.267__shadow_.Nek4YwHzmpiEi5RoM-Cmm0g9CyUZ7ME_0 2024-03-09T15:36:05.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.992+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1619745490803053828', time=2024-03-09T17:26:08.084589+0000, chain.objs.size()=1 2024-03-09T15:36:05.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.992+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.2GUYImoQed2leiN-FTwzbfr5btzl66X_0 2024-03-09T15:36:05.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.992+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8344787549124926797', time=2024-03-09T17:26:26.004127+0000, chain.objs.size()=1 2024-03-09T15:36:05.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.992+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.292__shadow_.QWKYr1hcihFABTtEx4K82PpaaLnMWuI_0 2024-03-09T15:36:05.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.992+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13269650411306493094', time=2024-03-09T17:31:36.741396+0000, chain.objs.size()=1 2024-03-09T15:36:05.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.992+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.316__shadow_.jy3wW-Agbq0bt-PbJub2wKlMNb9KGzn_0 2024-03-09T15:36:05.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:05.996+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.000+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-03-09T15:36:06.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=37, truncated=0, next_marker='' 2024-03-09T15:36:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_CWvY9q3RevCXXgXmXqCIz77NYfdAtH_', time=2024-03-09T17:13:40.136048+0000, chain.objs.size()=2 2024-03-09T15:36:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__shadow_.3fM2orVXq-TGPv-XbpUL3IzZ61CCwe0_0 2024-03-09T15:36:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__shadow_.3fM2orVXq-TGPv-XbpUL3IzZ61CCwe0_1 2024-03-09T15:36:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='2~NVsJdnCOH2AhdvXP4_MbbEGieM5zq-T', time=2024-03-09T17:16:21.991524+0000, chain.objs.size()=1 2024-03-09T15:36:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.194__multipart_mymultipart.2~NVsJdnCOH2AhdvXP4_MbbEGieM5zq-T.1 2024-03-09T15:36:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2986625917125502568', time=2024-03-09T17:21:31.908296+0000, chain.objs.size()=1 2024-03-09T15:36:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.240__shadow_.zCnKfgde00C7UiyZGgUQhqfOfFAVXEJ_0 2024-03-09T15:36:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1867686598739649062', time=2024-03-09T17:23:59.737633+0000, chain.objs.size()=1 2024-03-09T15:36:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.da2eAJLVZ7p78NlIbP89YcTCBW6Jbay_0 2024-03-09T15:36:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10453429185569588943', time=2024-03-09T17:24:00.630512+0000, chain.objs.size()=1 2024-03-09T15:36:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sDgaFHC_rK50i_MyyRv_ZEzaAyXhJgV_0 2024-03-09T15:36:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2583406404464516210', time=2024-03-09T17:24:01.484541+0000, chain.objs.size()=1 2024-03-09T15:36:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1drDBCTYPgosr4siClS-mnSQsSeoQN8_0 2024-03-09T15:36:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7923537145914325871', time=2024-03-09T17:24:01.845416+0000, chain.objs.size()=1 2024-03-09T15:36:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.wCSOZ7TEcWnmxaQltByvq1E4xIYPY9p_0 2024-03-09T15:36:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14545777163534568335', time=2024-03-09T17:24:03.460709+0000, chain.objs.size()=1 2024-03-09T15:36:06.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tDDSBPF8_eDNVAL78SjhNHcC8rXaulT_0 2024-03-09T15:36:06.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14230199227441312501', time=2024-03-09T17:24:04.255539+0000, chain.objs.size()=1 2024-03-09T15:36:06.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LWENOXd9L8xux2LAEt024a_LVWMHAvM_0 2024-03-09T15:36:06.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13587401555580806239', time=2024-03-09T17:24:05.304792+0000, chain.objs.size()=1 2024-03-09T15:36:06.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.R_JQwP9cZiRPm2cDKFjemmUdL9GR45G_0 2024-03-09T15:36:06.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1995893060844241971', time=2024-03-09T17:24:06.773995+0000, chain.objs.size()=1 2024-03-09T15:36:06.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.004+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iOMQXTbjZajJf4ngCwBeEmYubN-F3MZ_0 2024-03-09T15:36:06.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13884286357082251819', time=2024-03-09T17:24:07.882369+0000, chain.objs.size()=1 2024-03-09T15:36:06.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.GYtbgnuHkEktIB6_2b4ME03AcO6QXQo_0 2024-03-09T15:36:06.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12465515054818080633', time=2024-03-09T17:24:08.888717+0000, chain.objs.size()=1 2024-03-09T15:36:06.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.yZtWSSZSTgScw5ezw-z05ackai7Zvvl_0 2024-03-09T15:36:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7412968676957266881', time=2024-03-09T17:24:09.305527+0000, chain.objs.size()=1 2024-03-09T15:36:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ZHpf5RAkKTDwOOJOmIj2ntoVpZZCJLp_0 2024-03-09T15:36:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5272362677392982584', time=2024-03-09T17:24:09.797208+0000, chain.objs.size()=1 2024-03-09T15:36:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.JHOqlh3KtlZGG4QZzLK93SSaQpwFdb4_0 2024-03-09T15:36:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5067326646367030967', time=2024-03-09T17:24:10.041417+0000, chain.objs.size()=1 2024-03-09T15:36:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ujEDmFIcGj2o9PkaCw-mYkr3RUDJRj0_0 2024-03-09T15:36:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3687829553949896053', time=2024-03-09T17:24:12.085600+0000, chain.objs.size()=1 2024-03-09T15:36:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ghcvEqFMBhJ4HPRlVYWXnBW-Fugt7e5_0 2024-03-09T15:36:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5622198880754877840', time=2024-03-09T17:24:13.461615+0000, chain.objs.size()=1 2024-03-09T15:36:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.RtgS3j2tUefa13xPjJHv9UtVI2QmXr__0 2024-03-09T15:36:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6246874176398971280', time=2024-03-09T17:24:15.361970+0000, chain.objs.size()=1 2024-03-09T15:36:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.GHjU5vBdlCQqfte2xWnJw7TzYz404fM_0 2024-03-09T15:36:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11385970131142115544', time=2024-03-09T17:24:20.988611+0000, chain.objs.size()=1 2024-03-09T15:36:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Ape9OeFbdrf_fEgc8POcrviJrx9MNSZ_0 2024-03-09T15:36:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14998019387039556367', time=2024-03-09T17:24:21.413418+0000, chain.objs.size()=1 2024-03-09T15:36:06.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.y3MECwiVTcO5UTZmge9BNp-V-9c__tW_0 2024-03-09T15:36:06.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6431872829891953285', time=2024-03-09T17:24:21.484735+0000, chain.objs.size()=1 2024-03-09T15:36:06.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.012+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7wgYpkEAzP50V_S4RKBDKHxuRJNSZru_0 2024-03-09T15:36:06.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11862480449807781982', time=2024-03-09T17:24:21.864765+0000, chain.objs.size()=1 2024-03-09T15:36:06.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Cx1MwY6veAU5rh1qPLjEK9ovyjSH76C_0 2024-03-09T15:36:06.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.543241507226221560', time=2024-03-09T17:24:24.727539+0000, chain.objs.size()=1 2024-03-09T15:36:06.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.vNjCA8pKB24ZCeM0Zz6c1335hRguHjl_0 2024-03-09T15:36:06.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2650515677659842412', time=2024-03-09T17:24:26.508864+0000, chain.objs.size()=1 2024-03-09T15:36:06.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ZA244g4TLr2uRyWnDaBkHv1T3R0VEU2_0 2024-03-09T15:36:06.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17937134047089971751', time=2024-03-09T17:24:27.657872+0000, chain.objs.size()=1 2024-03-09T15:36:06.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.biain_CTYgRTXzLgswgNMjn9wtwDInO_0 2024-03-09T15:36:06.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7168012062070397363', time=2024-03-09T17:24:29.214113+0000, chain.objs.size()=1 2024-03-09T15:36:06.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.KqjnNbNFdyR0hXlEyN4MK46mQrxMuhT_0 2024-03-09T15:36:06.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14525870948886766604', time=2024-03-09T17:24:30.774190+0000, chain.objs.size()=1 2024-03-09T15:36:06.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Ta1IRE_D5V6BUI1CPhwGz-BZS4r_XK6_0 2024-03-09T15:36:06.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7916402367718858023', time=2024-03-09T17:24:31.213549+0000, chain.objs.size()=1 2024-03-09T15:36:06.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Ak3aME9IaJlVrqr7oXUFkct9Qr1O5gm_0 2024-03-09T15:36:06.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2905461846174542482', time=2024-03-09T17:24:31.215853+0000, chain.objs.size()=1 2024-03-09T15:36:06.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3SaiUAnmIHb9SBz9_mWqRSW_8znFdug_0 2024-03-09T15:36:06.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13023364336871599477', time=2024-03-09T17:25:04.698117+0000, chain.objs.size()=1 2024-03-09T15:36:06.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.016+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.267__shadow_.cFbFgEYC1Vb6BkM6OrWD21Ci2ff2JsY_0 2024-03-09T15:36:06.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16936076859040244720', time=2024-03-09T17:25:30.925157+0000, chain.objs.size()=1 2024-03-09T15:36:06.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.276__shadow_.Mxi-tQtoOcwHHU-yuVeSlw0scuJA1t5_0 2024-03-09T15:36:06.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10473550871822859688', time=2024-03-09T17:26:01.461930+0000, chain.objs.size()=1 2024-03-09T15:36:06.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.wG7soQMEYWPJtkIlN15Yu867vtKKnKP_0 2024-03-09T15:36:06.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12167258026297106266', time=2024-03-09T17:26:37.603441+0000, chain.objs.size()=1 2024-03-09T15:36:06.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.296__shadow_.CfRrcW2UzXqW8sJMqVszhM5EkJOqlmE_0 2024-03-09T15:36:06.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15741496694838852444', time=2024-03-09T17:34:02.611083+0000, chain.objs.size()=1 2024-03-09T15:36:06.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.333__shadow_.OK6Jhji-AxnnPalzJt1B6ns_eZY3vLJ_0 2024-03-09T15:36:06.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16975940905656719684', time=2024-03-09T17:34:17.729698+0000, chain.objs.size()=1 2024-03-09T15:36:06.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.335__shadow_.ZzUZuWC8SJYYovzrQr3SETxxU-KvDuE_0 2024-03-09T15:36:06.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6010792871644823322', time=2024-03-09T17:35:52.827314+0000, chain.objs.size()=5 2024-03-09T15:36:06.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.342__shadow_.DAabDFz9q1UogR2XU2J_2gZ55bcvn9C_0 2024-03-09T15:36:06.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.020+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.342__shadow_.DAabDFz9q1UogR2XU2J_2gZ55bcvn9C_1 2024-03-09T15:36:06.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.024+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.342__shadow_.DAabDFz9q1UogR2XU2J_2gZ55bcvn9C_2 2024-03-09T15:36:06.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.024+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.342__shadow_.DAabDFz9q1UogR2XU2J_2gZ55bcvn9C_3 2024-03-09T15:36:06.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.024+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.342__shadow_.DAabDFz9q1UogR2XU2J_2gZ55bcvn9C_4 2024-03-09T15:36:06.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.028+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.032+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-03-09T15:36:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=38, truncated=0, next_marker='' 2024-03-09T15:36:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9708512324464045269', time=2024-03-09T17:14:21.186258+0000, chain.objs.size()=1 2024-03-09T15:36:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.164__shadow_.2cU30zWadoIWGQMCwRSekjMRiLNZYyT_0 2024-03-09T15:36:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16632654851409435303', time=2024-03-09T17:20:51.711667+0000, chain.objs.size()=1 2024-03-09T15:36:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.228__shadow_.RDLABkfXDAekRDF19b5k9VoxcvZfO6u_0 2024-03-09T15:36:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12222966013426722934', time=2024-03-09T17:21:26.088136+0000, chain.objs.size()=1 2024-03-09T15:36:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.238__shadow_.3tDfEAhHkJj2jRuJPL1spZ5R359Ahcm_0 2024-03-09T15:36:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2141453680270646952', time=2024-03-09T17:21:31.918803+0000, chain.objs.size()=1 2024-03-09T15:36:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.240__shadow_.W5dRhK70UyJ47wvMbBI_Xp18678Cv67_0 2024-03-09T15:36:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3182492529241675657', time=2024-03-09T17:21:43.465060+0000, chain.objs.size()=1 2024-03-09T15:36:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.244__shadow_.WuS3Y4LM6kxAFvFCE-VlCtGqxpTtflc_0 2024-03-09T15:36:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15033711278365692653', time=2024-03-09T17:21:46.306537+0000, chain.objs.size()=1 2024-03-09T15:36:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.245__shadow_.3P7V_8f7qm7GDGMfcN_X4ZUKbjGrpS1_0 2024-03-09T15:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7850520537062519349', time=2024-03-09T17:21:54.835907+0000, chain.objs.size()=1 2024-03-09T15:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.248__shadow_.huAtTzv4F_koNqj-Lz32Ushur5j_yN3_0 2024-03-09T15:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10456864846989535847', time=2024-03-09T17:22:12.380798+0000, chain.objs.size()=1 2024-03-09T15:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.254__shadow_.m19ewMi6p0LTKNJ6LiamRey9pMyV93N_0 2024-03-09T15:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2804177046755530495', time=2024-03-09T17:22:12.392750+0000, chain.objs.size()=1 2024-03-09T15:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.254__shadow_.SEGQM2QVOK287HQQgf3qZg511KSo-pe_0 2024-03-09T15:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1856703602812191674', time=2024-03-09T17:23:58.865254+0000, chain.objs.size()=1 2024-03-09T15:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.VfEmtccz__qnUxi8zJie9WJ_Da28tdI_0 2024-03-09T15:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2285306658876719563', time=2024-03-09T17:23:58.871112+0000, chain.objs.size()=1 2024-03-09T15:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.D__6m86WRnhzWUoaf-z5sICkPKZNbrd_0 2024-03-09T15:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5253766925504214520', time=2024-03-09T17:24:02.134803+0000, chain.objs.size()=1 2024-03-09T15:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.036+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.jbxSzkjtJwJ2h7qBd4QhRMrhEOMt7me_0 2024-03-09T15:36:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.040+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13428112092460058685', time=2024-03-09T17:24:10.185919+0000, chain.objs.size()=1 2024-03-09T15:36:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.040+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.RQrbXopW2pwC6Sfbf28uulu1SqAJfRN_0 2024-03-09T15:36:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.040+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11800175749311215234', time=2024-03-09T17:24:10.783454+0000, chain.objs.size()=1 2024-03-09T15:36:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.040+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.AVWaympyuI97UHhsQtO797U703ibJjL_0 2024-03-09T15:36:06.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.040+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2420394077508221632', time=2024-03-09T17:24:12.072569+0000, chain.objs.size()=1 2024-03-09T15:36:06.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.040+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.MBTlD72LvqeRDBCjV6S0hfiSDhtk_HJ_0 2024-03-09T15:36:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1117647401440765559', time=2024-03-09T17:24:12.799565+0000, chain.objs.size()=1 2024-03-09T15:36:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.NAdw-TecIK2GUmgwWnjmhnmKpI0sFvJ_0 2024-03-09T15:36:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16506718610818863980', time=2024-03-09T17:24:14.886960+0000, chain.objs.size()=1 2024-03-09T15:36:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ytyQ2BLZG37wUrdcyUEPzA0ZL4dSw12_0 2024-03-09T15:36:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13128014220108170523', time=2024-03-09T17:24:15.268900+0000, chain.objs.size()=1 2024-03-09T15:36:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.mw-vOYFkMDkOxvJD5KjouIkzXMClbCY_0 2024-03-09T15:36:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14995791290894420829', time=2024-03-09T17:24:16.340778+0000, chain.objs.size()=1 2024-03-09T15:36:06.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.hQU8arIRMHHzX93-xht5n6pCN7NgZrk_0 2024-03-09T15:36:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9131624684235826477', time=2024-03-09T17:24:18.393353+0000, chain.objs.size()=1 2024-03-09T15:36:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.M0oS74uWtIZ9iO90X9QsCMzmYRmsUBB_0 2024-03-09T15:36:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1512007214149823214', time=2024-03-09T17:24:18.884658+0000, chain.objs.size()=1 2024-03-09T15:36:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6gvWouTIZOHvvT9aR3QA3Ela27ekYjd_0 2024-03-09T15:36:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14748091957148826147', time=2024-03-09T17:24:20.493843+0000, chain.objs.size()=1 2024-03-09T15:36:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.wjlbWY5jafWkmjFPodmlkrVdLouhABv_0 2024-03-09T15:36:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.602780008094093236', time=2024-03-09T17:24:23.173535+0000, chain.objs.size()=1 2024-03-09T15:36:06.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.eX8m6uHxtDpiOBbPP5W705s34p34dC2_0 2024-03-09T15:36:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11110079929730626115', time=2024-03-09T17:24:23.548477+0000, chain.objs.size()=1 2024-03-09T15:36:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.BGLkmyC_OggLQJdcV7Nq_kAOJ--S7jr_0 2024-03-09T15:36:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6232256297582042631', time=2024-03-09T17:24:26.082671+0000, chain.objs.size()=1 2024-03-09T15:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.bT8DPQLXP4m2ZZw_h52e6L-VGe-XXve_0 2024-03-09T15:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14491493420180469827', time=2024-03-09T17:24:26.269160+0000, chain.objs.size()=1 2024-03-09T15:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.jFSl1i6-3f57dmHfobIzZeHYRqB4sfh_0 2024-03-09T15:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11850245828338719027', time=2024-03-09T17:24:27.520734+0000, chain.objs.size()=1 2024-03-09T15:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2aRyzZV22mCF-4TJApSB0MUFuChS_YY_0 2024-03-09T15:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15713681122204657136', time=2024-03-09T17:24:27.870774+0000, chain.objs.size()=1 2024-03-09T15:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.044+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sKm70N8hdxY4sqU1TWn9CkMMzAy3TRu_0 2024-03-09T15:36:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15822760553693378887', time=2024-03-09T17:24:28.741988+0000, chain.objs.size()=1 2024-03-09T15:36:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.eAXu_OMzgKBhfxssB9ilzuGMePbg9uC_0 2024-03-09T15:36:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12241796420542523467', time=2024-03-09T17:24:29.197439+0000, chain.objs.size()=1 2024-03-09T15:36:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.NV5DlgqE86BHhTQvyvPzk5FWcBjq4vu_0 2024-03-09T15:36:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12306071643832358868', time=2024-03-09T17:24:29.306097+0000, chain.objs.size()=1 2024-03-09T15:36:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.SihOEOrYvuhZKnXf7JFsqVu-bhSB2Ze_0 2024-03-09T15:36:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12880950069751596887', time=2024-03-09T17:24:31.373247+0000, chain.objs.size()=1 2024-03-09T15:36:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.aRopTOEKuAeEAbV7ZNWUFsYZz682OJ1_0 2024-03-09T15:36:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2106838494131701218', time=2024-03-09T17:24:37.083970+0000, chain.objs.size()=1 2024-03-09T15:36:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.257__shadow_.1a81WUhKtLA9a_azPgEHiZp14B4qaK3_0 2024-03-09T15:36:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16709046824163469395', time=2024-03-09T17:25:30.904982+0000, chain.objs.size()=1 2024-03-09T15:36:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.276__shadow_.7xUZG9Fl-wWsUWvZN0zzUo1u5tuAi6v_0 2024-03-09T15:36:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1671641428403310382', time=2024-03-09T17:25:39.766795+0000, chain.objs.size()=1 2024-03-09T15:36:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.279__shadow_.FZVbgAVa_kKVFKJYg6T7-fM6UiGJ1mJ_0 2024-03-09T15:36:06.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7981149446250761555', time=2024-03-09T17:26:01.594332+0000, chain.objs.size()=1 2024-03-09T15:36:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.YvnqWeJo5DHLhTWbzzYTwRfS1GRtMUE_0 2024-03-09T15:36:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16569119472090715162', time=2024-03-09T17:26:08.085011+0000, chain.objs.size()=1 2024-03-09T15:36:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.fxVEv-gB15adzspUHj0syHAJpVZ8jYB_0 2024-03-09T15:36:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1942582854608071442', time=2024-03-09T17:26:46.775955+0000, chain.objs.size()=1 2024-03-09T15:36:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.048+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.299__shadow_.JJiAsjyJJ5X6UzYkOlDl_CAGycxHsaa_0 2024-03-09T15:36:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.052+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.056+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-03-09T15:36:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.056+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.056+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=31, truncated=0, next_marker='' 2024-03-09T15:36:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12365549310307397960', time=2024-03-09T17:20:54.643112+0000, chain.objs.size()=1 2024-03-09T15:36:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.229__shadow_.3vZNDv7dcXx1nKVk10dpGZgDCmpBh5X_0 2024-03-09T15:36:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1819742083448152322', time=2024-03-09T17:21:40.692850+0000, chain.objs.size()=1 2024-03-09T15:36:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.243__shadow_.hH72M2sy11gfuOSjeanMcJcgo83YLEi_0 2024-03-09T15:36:06.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3469863003933036413', time=2024-03-09T17:24:05.226983+0000, chain.objs.size()=1 2024-03-09T15:36:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.vet1AmGOgQDzUyC_wQ2srLIQlj6pDSN_0 2024-03-09T15:36:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4515640596123955116', time=2024-03-09T17:24:05.228897+0000, chain.objs.size()=1 2024-03-09T15:36:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.joGdqtR4HSacaAQU_9-Y11LX7qgyMkE_0 2024-03-09T15:36:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16145994150659512690', time=2024-03-09T17:24:05.924179+0000, chain.objs.size()=1 2024-03-09T15:36:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.4--SNwy1abufQGSYgUTf_6tJZ6Ko1IT_0 2024-03-09T15:36:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13477002530161158277', time=2024-03-09T17:24:10.198002+0000, chain.objs.size()=1 2024-03-09T15:36:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qXSBRSBcV_DPto0pEybrExyGCzxdrez_0 2024-03-09T15:36:06.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5218252389797990145', time=2024-03-09T17:24:12.625830+0000, chain.objs.size()=1 2024-03-09T15:36:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.m_US1ptWEDpb2njC9FzWhMSFUI02q1J_0 2024-03-09T15:36:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4792257125138725430', time=2024-03-09T17:24:13.295472+0000, chain.objs.size()=1 2024-03-09T15:36:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.OtoEWjDcYtU1Uy5j5Pjut5SkQaSNdGP_0 2024-03-09T15:36:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.598988654026598683', time=2024-03-09T17:24:13.464458+0000, chain.objs.size()=1 2024-03-09T15:36:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.CXP6KfUb97pSHaAeVu7aMnNKfiuFShm_0 2024-03-09T15:36:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3496007357354599037', time=2024-03-09T17:24:13.476775+0000, chain.objs.size()=1 2024-03-09T15:36:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.e2Bn2Y0SDFHr6JUff40F2X6U0TiooAm_0 2024-03-09T15:36:06.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10012768749838402807', time=2024-03-09T17:24:14.295454+0000, chain.objs.size()=1 2024-03-09T15:36:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iWR5lssScbR7knfyXtMtcK0GHl28dEA_0 2024-03-09T15:36:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6447514016297308641', time=2024-03-09T17:24:17.250359+0000, chain.objs.size()=1 2024-03-09T15:36:06.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Rt9ZMsfy9Y_g98VqH02lVHmUAq1c0Jc_0 2024-03-09T15:36:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7398828513911125794', time=2024-03-09T17:24:18.811541+0000, chain.objs.size()=1 2024-03-09T15:36:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7zsK9YRSIIii5gdAm2kDu19di3zC4CH_0 2024-03-09T15:36:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4620986973887497165', time=2024-03-09T17:24:23.489255+0000, chain.objs.size()=1 2024-03-09T15:36:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.lp2FUSnAfuQEZPJdmt5XUYg7ForNA0B_0 2024-03-09T15:36:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6933745511164176509', time=2024-03-09T17:24:24.617697+0000, chain.objs.size()=1 2024-03-09T15:36:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_._H5mIJN6-KH054_rcF4zP35wCtEw6SX_0 2024-03-09T15:36:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11411882874492304859', time=2024-03-09T17:24:25.126387+0000, chain.objs.size()=1 2024-03-09T15:36:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.060+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8Ecv4bqyBwT49E28Zo-mwAbecBh1fxP_0 2024-03-09T15:36:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.368206699633147244', time=2024-03-09T17:24:25.128225+0000, chain.objs.size()=1 2024-03-09T15:36:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PDeexalWkXysFXIsgFBV4MO-cXegTCc_0 2024-03-09T15:36:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6997585575699701449', time=2024-03-09T17:24:26.047099+0000, chain.objs.size()=1 2024-03-09T15:36:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qKxUO_ImGaonGvzoIdPxxjlXi6Ywzw0_0 2024-03-09T15:36:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16656750663346254978', time=2024-03-09T17:24:26.322421+0000, chain.objs.size()=1 2024-03-09T15:36:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.CLophCkDf2oE1Wu79V7Jxi9-iLRduPf_0 2024-03-09T15:36:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3477871164120288420', time=2024-03-09T17:24:26.699365+0000, chain.objs.size()=1 2024-03-09T15:36:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.owsPkExDYRs8HMMHDabs8BFKtwvm0eT_0 2024-03-09T15:36:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15931649712346930320', time=2024-03-09T17:24:28.701717+0000, chain.objs.size()=1 2024-03-09T15:36:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.oAgN3viFcmA218qDZcZLHPrgvS_uKMq_0 2024-03-09T15:36:06.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7781582645756341475', time=2024-03-09T17:24:30.320322+0000, chain.objs.size()=1 2024-03-09T15:36:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Mrh05O1FJr25JAkruhdp93YrsRf9IZt_0 2024-03-09T15:36:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12134547670912588721', time=2024-03-09T17:24:45.439722+0000, chain.objs.size()=1 2024-03-09T15:36:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.260__shadow_.hHuczdlwQZVjOajIAOSs-sqLrQ3du3M_0 2024-03-09T15:36:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17102280397439887600', time=2024-03-09T17:25:10.520840+0000, chain.objs.size()=1 2024-03-09T15:36:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.269__shadow_.yL_o6fK_L7hRBqaFVLcx7RIlVb5qxUF_0 2024-03-09T15:36:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1488545520441799897', time=2024-03-09T17:25:22.121468+0000, chain.objs.size()=1 2024-03-09T15:36:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.064+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.273__shadow_.jBuTDwsr5M2VpoY5OlaDJ9bb8QMsCP1_0 2024-03-09T15:36:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.068+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17518562545467150994', time=2024-03-09T17:25:25.162594+0000, chain.objs.size()=1 2024-03-09T15:36:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.068+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.274__shadow_.h2bPLiIp6ON1u2pwTUEeQ4lbtgk0rMq_0 2024-03-09T15:36:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.068+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10180899324147157896', time=2024-03-09T17:25:48.356283+0000, chain.objs.size()=1 2024-03-09T15:36:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.068+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.282__shadow_.yfVWkxSHQ7QGhATM7CLASe_EE5Ow2qZ_0 2024-03-09T15:36:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.068+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15011617742163856495', time=2024-03-09T17:26:07.936237+0000, chain.objs.size()=1 2024-03-09T15:36:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.068+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.xf4HTz2VXFV_Zu-v9WSIALZ6q9LaHOJ_0 2024-03-09T15:36:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.068+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10071395741991447762', time=2024-03-09T17:26:08.081509+0000, chain.objs.size()=1 2024-03-09T15:36:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.068+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.U5p4mv5dPGO5x4pv3E8iKBW811_7Cij_0 2024-03-09T15:36:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.068+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1742751492795327221', time=2024-03-09T17:26:22.898848+0000, chain.objs.size()=1 2024-03-09T15:36:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.068+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.291__shadow_.ZvReL5oMYrLIEGUNg7td4YYzUzkr5TI_0 2024-03-09T15:36:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.068+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3512635876420792444', time=2024-03-09T17:26:40.569664+0000, chain.objs.size()=1 2024-03-09T15:36:06.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.068+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.297__shadow_.ImbhVdsl7L9sehU3Z7UNM_kcyIAHU4K_0 2024-03-09T15:36:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.072+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.076+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-03-09T15:36:06.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.076+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.076+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=37, truncated=0, next_marker='' 2024-03-09T15:36:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2205601269018558751', time=2024-03-09T17:11:40.040402+0000, chain.objs.size()=1 2024-03-09T15:36:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.134__shadow_.XDlVtIkk-VWr7j5EGzgkEAmxKo9GmOr_0 2024-03-09T15:36:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7628002701287829340', time=2024-03-09T17:14:57.259355+0000, chain.objs.size()=1 2024-03-09T15:36:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.175__shadow_.hzylZ125Egh3101U22ZYFqtC8V-0Hm1_0 2024-03-09T15:36:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17700470318737948857', time=2024-03-09T17:20:54.659686+0000, chain.objs.size()=1 2024-03-09T15:36:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.229__shadow_.8l_YC4EYJ15elcz71MV67XfsEUXHwYz_0 2024-03-09T15:36:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3943621807578122199', time=2024-03-09T17:21:06.072960+0000, chain.objs.size()=1 2024-03-09T15:36:06.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.232__shadow_.U_ba3n2FOgMuXYCQKPBId8S61q6_oqe_0 2024-03-09T15:36:06.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15407694676907746515', time=2024-03-09T17:21:14.253682+0000, chain.objs.size()=1 2024-03-09T15:36:06.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.235__shadow_.2OphVetWRoe65AlDSYBF2aYmQjTnu7A_0 2024-03-09T15:36:06.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13121673538187762074', time=2024-03-09T17:21:28.953127+0000, chain.objs.size()=1 2024-03-09T15:36:06.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.239__shadow_.hNbWWKGVg68V6c7k7IO2415I1rlVn8N_0 2024-03-09T15:36:06.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6292768442942848130', time=2024-03-09T17:23:58.858195+0000, chain.objs.size()=1 2024-03-09T15:36:06.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.a6BY5G9NSTbxaQ5wxu45FzPSp_kakOL_0 2024-03-09T15:36:06.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9765309495905984872', time=2024-03-09T17:24:01.474175+0000, chain.objs.size()=1 2024-03-09T15:36:06.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sLUlrSEq2KfWH4nK1ZGqv9pmXHIUt88_0 2024-03-09T15:36:06.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13987258677389835213', time=2024-03-09T17:24:01.856348+0000, chain.objs.size()=1 2024-03-09T15:36:06.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_._eQyULrd1iiXOCWFDod6rxsqMy3rZ3C_0 2024-03-09T15:36:06.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15536225482448274179', time=2024-03-09T17:24:02.082036+0000, chain.objs.size()=1 2024-03-09T15:36:06.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tx2l83dU3M3mrzlFbApe2BK460ROQP0_0 2024-03-09T15:36:06.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11771164426949717772', time=2024-03-09T17:24:03.501812+0000, chain.objs.size()=1 2024-03-09T15:36:06.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.c_mY-M4kaP_SLglh5txS247SMgZBSfK_0 2024-03-09T15:36:06.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8701535897162810132', time=2024-03-09T17:24:03.542107+0000, chain.objs.size()=1 2024-03-09T15:36:06.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.JacU5Aptbdd0SEFXxSY0OUlEum6rn0K_0 2024-03-09T15:36:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6687263698505493301', time=2024-03-09T17:24:06.353200+0000, chain.objs.size()=1 2024-03-09T15:36:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.e4nuQizowfgfm-oBKXeGenrsH2HzSBu_0 2024-03-09T15:36:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10712062910864888264', time=2024-03-09T17:24:08.496157+0000, chain.objs.size()=1 2024-03-09T15:36:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.080+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tYxRDXSL2CT_4lVSYwFzJ60RmulISc8_0 2024-03-09T15:36:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18187677970673280558', time=2024-03-09T17:24:09.157206+0000, chain.objs.size()=1 2024-03-09T15:36:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.jjjczGTtz_3204xvOHMOc06MFumvlGt_0 2024-03-09T15:36:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3202990453279327902', time=2024-03-09T17:24:13.535311+0000, chain.objs.size()=1 2024-03-09T15:36:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.DTYPQYVGt-_VabbzuRxEfY9tFR4rft1_0 2024-03-09T15:36:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13328308241566043129', time=2024-03-09T17:24:13.815597+0000, chain.objs.size()=1 2024-03-09T15:36:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.pP3OhwaKhDbOs6sHbjH00lPmyTHyq5H_0 2024-03-09T15:36:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15740966923657067356', time=2024-03-09T17:24:19.425708+0000, chain.objs.size()=1 2024-03-09T15:36:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.5wAHZR6iCnM6FdRxghLgs88lg7OodVE_0 2024-03-09T15:36:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5423848499679620039', time=2024-03-09T17:24:20.611232+0000, chain.objs.size()=1 2024-03-09T15:36:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.d_zdHsumJ0SYCzZta6MXR0J9qKEWUDT_0 2024-03-09T15:36:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16420151252632799434', time=2024-03-09T17:24:21.359970+0000, chain.objs.size()=1 2024-03-09T15:36:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-qDIZ6xjolfTaGmkGl1l4Ng6Y4AhTBf_0 2024-03-09T15:36:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17164479522519150350', time=2024-03-09T17:24:22.257093+0000, chain.objs.size()=1 2024-03-09T15:36:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.h3QvUE5s82vwyXqpn8ERB_TRgpUmbCt_0 2024-03-09T15:36:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8911094463210527354', time=2024-03-09T17:24:22.439843+0000, chain.objs.size()=1 2024-03-09T15:36:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HMomq2lTorP32njr6SHzZFqb78R4s9I_0 2024-03-09T15:36:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2316730545524798422', time=2024-03-09T17:24:23.051849+0000, chain.objs.size()=1 2024-03-09T15:36:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.eIbKz-kmEe0A4hPhENBJApRM5PxtT8I_0 2024-03-09T15:36:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1091056284172427656', time=2024-03-09T17:24:24.975523+0000, chain.objs.size()=1 2024-03-09T15:36:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.084+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Tv9zm_U3xQW2PpDHygViUHEoiwlsccr_0 2024-03-09T15:36:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12588421895976936751', time=2024-03-09T17:24:25.007769+0000, chain.objs.size()=1 2024-03-09T15:36:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Ye1Dk7EalXUIV2Bu0c-lA9usoZVBTuy_0 2024-03-09T15:36:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17224849484022463172', time=2024-03-09T17:24:25.538152+0000, chain.objs.size()=1 2024-03-09T15:36:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.j3-YMSnGns8_If-18nAiVCKnVVkPZ5n_0 2024-03-09T15:36:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6508644500053598660', time=2024-03-09T17:24:25.928738+0000, chain.objs.size()=1 2024-03-09T15:36:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2MkRK3ws-Ub1Alhsb_YmTbT6C_z0dSq_0 2024-03-09T15:36:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7797870603463162265', time=2024-03-09T17:24:29.668659+0000, chain.objs.size()=1 2024-03-09T15:36:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.AzTr3OXWfsy-8G6zu8bdn7JRhsF6XRA_0 2024-03-09T15:36:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8282323366654718457', time=2024-03-09T17:24:30.010402+0000, chain.objs.size()=1 2024-03-09T15:36:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.VxNG-sOinnw-RVPU82Xys2QeCIVGIQa_0 2024-03-09T15:36:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17855260287369274237', time=2024-03-09T17:24:30.369957+0000, chain.objs.size()=1 2024-03-09T15:36:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.y0Trw_nj_z0mtkxDBKGt-JNUeaESNxD_0 2024-03-09T15:36:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5561794252151752239', time=2024-03-09T17:24:45.453943+0000, chain.objs.size()=1 2024-03-09T15:36:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.088+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.260__shadow_.Q5xHC74WmTuF9zSqTtkSUQ4FO9w_cYF_0 2024-03-09T15:36:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.092+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14700612807353028572', time=2024-03-09T17:24:50.906811+0000, chain.objs.size()=1 2024-03-09T15:36:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.092+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.262__shadow_.wsv5GF4088YmXw_bPvQWqx4dZs5wyJD_0 2024-03-09T15:36:06.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.092+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4492293119375041863', time=2024-03-09T17:24:53.715549+0000, chain.objs.size()=1 2024-03-09T15:36:06.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.092+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.263__shadow_.xSVj7plXNp3vanKSe_pueNH3iB2AZbA_0 2024-03-09T15:36:06.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.092+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1781446064352542486', time=2024-03-09T17:25:01.941565+0000, chain.objs.size()=1 2024-03-09T15:36:06.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.092+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.266__shadow_.ZsX2uU18dUsYkOytAQYxzkSWUM7S-5d_0 2024-03-09T15:36:06.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.092+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7786861489635195491', time=2024-03-09T17:26:07.794414+0000, chain.objs.size()=1 2024-03-09T15:36:06.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.092+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.QjBxuhsBeZVgQm6sXEB16szOP4UfqsJ_0 2024-03-09T15:36:06.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.092+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17305184980578590944', time=2024-03-09T17:32:27.185276+0000, chain.objs.size()=1 2024-03-09T15:36:06.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.092+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.323__shadow_.tt5eYWDBTMrQ9fZboDwYb2JJU_GkAmN_0 2024-03-09T15:36:06.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.092+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2697752589404524781', time=2024-03-09T17:32:52.371555+0000, chain.objs.size()=1 2024-03-09T15:36:06.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.092+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.324__shadow_.KsGJdfF8BpU3w3rzUZHbA9YykZb96qa_0 2024-03-09T15:36:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.096+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-03-09T15:36:06.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=48, truncated=0, next_marker='' 2024-03-09T15:36:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15459265916734493569', time=2024-03-09T17:11:55.916189+0000, chain.objs.size()=1 2024-03-09T15:36:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.150__shadow:1StSzNVD9h6jutLU5uNlZdKuslwVItt_.eHWu-bM799yz2_migWdLGyhCs5VjaPs_0 2024-03-09T15:36:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6375775588977709484', time=2024-03-09T17:12:34.814193+0000, chain.objs.size()=3 2024-03-09T15:36:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.152__multipart_mymultipart-LUKEWARM.2~SBO3GdczIn6l_JJAi4CRiPUZA98uox4.1 2024-03-09T15:36:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.152__multipart_mymultipart-LUKEWARM.2~SBO3GdczIn6l_JJAi4CRiPUZA98uox4.2 2024-03-09T15:36:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.152__multipart_mymultipart-LUKEWARM.2~SBO3GdczIn6l_JJAi4CRiPUZA98uox4.3 2024-03-09T15:36:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2210845048510016183', time=2024-03-09T17:14:05.741529+0000, chain.objs.size()=1 2024-03-09T15:36:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.159__shadow_.WkA66QW2NIZIa_c_GeN2hr_slfNmT9M_0 2024-03-09T15:36:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10897365478621995635', time=2024-03-09T17:15:15.157091+0000, chain.objs.size()=1 2024-03-09T15:36:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.100+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.182__shadow_.ZI7cc0hrE9xHAbiB23xVEIQkVCuM9OU_0 2024-03-09T15:36:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15336430132070626315', time=2024-03-09T17:16:33.729388+0000, chain.objs.size()=1 2024-03-09T15:36:06.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.197__shadow_.RSiFA4grphJ1GAhGNWm4qONzS9OF9O5_0 2024-03-09T15:36:06.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9742936481847678035', time=2024-03-09T17:20:45.914675+0000, chain.objs.size()=1 2024-03-09T15:36:06.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.226__shadow_.5-brXPLka4ZIfmGSnyX1vAcCUA6rVLq_0 2024-03-09T15:36:06.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7121948452230781939', time=2024-03-09T17:20:45.916707+0000, chain.objs.size()=1 2024-03-09T15:36:06.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.226__shadow_.dnfQmgN_DRLneFFXOXrd0qI4OoFpPqS_0 2024-03-09T15:36:06.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10111550774919321032', time=2024-03-09T17:21:37.751752+0000, chain.objs.size()=1 2024-03-09T15:36:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.242__shadow_.S9wv2yoD1icVlqBgbHpXU5Ayw8qzNle_0 2024-03-09T15:36:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6169563614621116926', time=2024-03-09T17:22:00.538750+0000, chain.objs.size()=1 2024-03-09T15:36:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.250__shadow_.Af2sddlWLKjL0O8Vx4u0sgdw9FQV8Rp_0 2024-03-09T15:36:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14054658223926426131', time=2024-03-09T17:22:09.506825+0000, chain.objs.size()=1 2024-03-09T15:36:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.253__shadow_.Wga4B97VfAZYr8UiiXe6wJs0Sez2NVd_0 2024-03-09T15:36:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9041529599137894464', time=2024-03-09T17:22:12.397247+0000, chain.objs.size()=1 2024-03-09T15:36:06.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.104+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.254__shadow_.EDDcOXh1q3qay886_RwVZhWGekb33Ql_0 2024-03-09T15:36:06.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8745789065335408076', time=2024-03-09T17:24:00.832482+0000, chain.objs.size()=1 2024-03-09T15:36:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.QxH3OOvUBdOIl48Re83zX5dN3K1DTCt_0 2024-03-09T15:36:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18276346973735306643', time=2024-03-09T17:24:01.961174+0000, chain.objs.size()=1 2024-03-09T15:36:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9Jjp4RfEtfHA8utfnGQLSN2X1ix_8zU_0 2024-03-09T15:36:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18008937876535883829', time=2024-03-09T17:24:05.677532+0000, chain.objs.size()=1 2024-03-09T15:36:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3_P2njMIfn0tr8Jvmm_kbF40NryTDaX_0 2024-03-09T15:36:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5248321695437745711', time=2024-03-09T17:24:06.669984+0000, chain.objs.size()=1 2024-03-09T15:36:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.glS-Ijyw-tM7ok9e_fXhpfJYRdpg3Jw_0 2024-03-09T15:36:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2173062726555283924', time=2024-03-09T17:24:06.840855+0000, chain.objs.size()=1 2024-03-09T15:36:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ZUVLsR1SvzjOn-HIzjTfTYMGFRKQRR7_0 2024-03-09T15:36:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1937339019281602232', time=2024-03-09T17:24:07.882269+0000, chain.objs.size()=1 2024-03-09T15:36:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qHabouR6hAF20KxtAWNuACaFsIqgJNN_0 2024-03-09T15:36:06.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14250941131152930529', time=2024-03-09T17:24:08.324233+0000, chain.objs.size()=1 2024-03-09T15:36:06.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.bl7U-psuBVo54VGd4dpkdHVqoIBF9Qx_0 2024-03-09T15:36:06.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1439177858747516480', time=2024-03-09T17:24:08.888593+0000, chain.objs.size()=1 2024-03-09T15:36:06.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0B6smYmuEa9qzva0uc53EqLSZfqINwD_0 2024-03-09T15:36:06.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3636042468852601163', time=2024-03-09T17:24:08.890651+0000, chain.objs.size()=1 2024-03-09T15:36:06.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.TMJIgthOuB3Mj4wZKG9eSwcB6FkVt7T_0 2024-03-09T15:36:06.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13979752106735923575', time=2024-03-09T17:24:10.041246+0000, chain.objs.size()=1 2024-03-09T15:36:06.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Phf8aUuFLFoahg-35RnbHHtSX4eGmOC_0 2024-03-09T15:36:06.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1630690132372403830', time=2024-03-09T17:24:10.043013+0000, chain.objs.size()=1 2024-03-09T15:36:06.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7XCe-Fqh6Nm_gWXqRoyPQo8cq0N_0Fp_0 2024-03-09T15:36:06.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10916447531988395967', time=2024-03-09T17:24:10.361438+0000, chain.objs.size()=1 2024-03-09T15:36:06.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.108+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2lxUC3jEHiSNO-coPWOoHnhrPnYQp3y_0 2024-03-09T15:36:06.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2390823977676320411', time=2024-03-09T17:24:12.112302+0000, chain.objs.size()=1 2024-03-09T15:36:06.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.nwEHzqJf4owMikYixjl4wkawavgXGNK_0 2024-03-09T15:36:06.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2475942264834129664', time=2024-03-09T17:24:16.729532+0000, chain.objs.size()=1 2024-03-09T15:36:06.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ndq1-e-FqyxS1-ckuLypn7e-OmlGRQN_0 2024-03-09T15:36:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1923864053090830125', time=2024-03-09T17:24:19.670928+0000, chain.objs.size()=1 2024-03-09T15:36:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1BlDd3KfWdH4K3GT-gL5B6mSlPNtmlr_0 2024-03-09T15:36:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13970410284342260948', time=2024-03-09T17:24:21.922743+0000, chain.objs.size()=1 2024-03-09T15:36:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.QGLwwDQuh6iLaXTHm_FXJkhSfif07FN_0 2024-03-09T15:36:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5142977901117265428', time=2024-03-09T17:24:22.261448+0000, chain.objs.size()=1 2024-03-09T15:36:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.yV_rnFeHYF195TD902Zys3Z32tSzDN2_0 2024-03-09T15:36:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6158324342432083071', time=2024-03-09T17:24:22.564317+0000, chain.objs.size()=1 2024-03-09T15:36:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.5uC55CnOblztloXBc1OGf1Aq7NedA3t_0 2024-03-09T15:36:06.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3409169491421455993', time=2024-03-09T17:24:22.843434+0000, chain.objs.size()=1 2024-03-09T15:36:06.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.E2R8ObkKnrYBIYxoe43FeaxJuQKrEdv_0 2024-03-09T15:36:06.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7309519446191548994', time=2024-03-09T17:24:23.352014+0000, chain.objs.size()=1 2024-03-09T15:36:06.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.wDSp9ipG7mFT-50KbzE4HRLWod0S54O_0 2024-03-09T15:36:06.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5949720127733229920', time=2024-03-09T17:24:23.426873+0000, chain.objs.size()=1 2024-03-09T15:36:06.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.BPWIa-kWDZHtXTJvN7F2jnJDkrQyvFD_0 2024-03-09T15:36:06.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4441496508731747763', time=2024-03-09T17:24:23.759856+0000, chain.objs.size()=1 2024-03-09T15:36:06.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.eZFHGtuZ__er20i0pLrmkCFNG6UM3_6_0 2024-03-09T15:36:06.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8526509155581437284', time=2024-03-09T17:24:26.441918+0000, chain.objs.size()=1 2024-03-09T15:36:06.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.GafRDTTNgka0ek2X7U_MwKFZuKQkPry_0 2024-03-09T15:36:06.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1249384901902924461', time=2024-03-09T17:24:26.908023+0000, chain.objs.size()=1 2024-03-09T15:36:06.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HMS3zUJuPhgfmzHE1Mvn-L7vUV2A2E__0 2024-03-09T15:36:06.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14422695887356399580', time=2024-03-09T17:24:29.618789+0000, chain.objs.size()=1 2024-03-09T15:36:06.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.xHjTBA1ZSytR9ImLL1-9DIE_00TH9vp_0 2024-03-09T15:36:06.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2292346996444765830', time=2024-03-09T17:24:30.586262+0000, chain.objs.size()=1 2024-03-09T15:36:06.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.mWvESnCpuhblcc_8j08FQ0BtBA62Dto_0 2024-03-09T15:36:06.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9878943191735665303', time=2024-03-09T17:24:31.037078+0000, chain.objs.size()=1 2024-03-09T15:36:06.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.rey1_gwaYs-7IzQOFvZIBs-3B-luKbh_0 2024-03-09T15:36:06.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14145298552946907097', time=2024-03-09T17:24:31.213868+0000, chain.objs.size()=1 2024-03-09T15:36:06.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.112+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.rkqRSjjsN3Iruv2pkoiR1scnwJOhEKX_0 2024-03-09T15:36:06.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2863138774099202272', time=2024-03-09T17:24:42.661195+0000, chain.objs.size()=1 2024-03-09T15:36:06.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.259__shadow_.WSK5UqYLBfQV8JdG9PeKWIiO-qHU5y1_0 2024-03-09T15:36:06.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12919358710213480208', time=2024-03-09T17:26:01.732568+0000, chain.objs.size()=1 2024-03-09T15:36:06.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.QOeiyBZOH-xYrgpikpRcZtFBhpYHjQT_0 2024-03-09T15:36:06.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3212147567011510740', time=2024-03-09T17:26:01.830672+0000, chain.objs.size()=1 2024-03-09T15:36:06.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.eOQNRDUcm0cVsfYBhlntZ1ST8x5DUGq_0 2024-03-09T15:36:06.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.899681476346537742', time=2024-03-09T17:26:07.806347+0000, chain.objs.size()=1 2024-03-09T15:36:06.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.tNoKLJApRWlwY_yHPFAIt-ckaYcD98n_0 2024-03-09T15:36:06.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15518200675767476572', time=2024-03-09T17:26:34.679834+0000, chain.objs.size()=1 2024-03-09T15:36:06.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.295__shadow_.w2xSnj4sM6PxkieUoxd3s-mfliNq19E_0 2024-03-09T15:36:06.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5871106973785518950', time=2024-03-09T17:26:34.682214+0000, chain.objs.size()=1 2024-03-09T15:36:06.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.295__shadow_.NTeN96-CeY-miXsuYCtkFb99xKqP9Bo_0 2024-03-09T15:36:06.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14455123459881249279', time=2024-03-09T17:26:34.683972+0000, chain.objs.size()=1 2024-03-09T15:36:06.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.295__shadow_.eEDLTZI_Akb4IPHHZ4qNH6-pwgPedF6_0 2024-03-09T15:36:06.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17310029501080810716', time=2024-03-09T17:33:48.428260+0000, chain.objs.size()=1 2024-03-09T15:36:06.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.331__shadow_.ryhq_mF0eCXIYtS3xqAy0xzfBaWO-Pb_0 2024-03-09T15:36:06.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12717034585709754111', time=2024-03-09T17:34:09.460328+0000, chain.objs.size()=1 2024-03-09T15:36:06.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.116+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.334__shadow_.eTHTKHILEl3FEp5222GGNbTeNObtxZQ_0 2024-03-09T15:36:06.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.120+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.124+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-03-09T15:36:06.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.124+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.124+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.124+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=38, truncated=0, next_marker='' 2024-03-09T15:36:06.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.124+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8484098639435134561', time=2024-03-09T17:12:59.918324+0000, chain.objs.size()=2 2024-03-09T15:36:06.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.124+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.zp_TqYXR8kkr9m5xccYH7CujST8Teue_0 2024-03-09T15:36:06.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.zp_TqYXR8kkr9m5xccYH7CujST8Teue_1 2024-03-09T15:36:06.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_nAAHCacC2mxhyMYY4s80EpW9WluWepP', time=2024-03-09T17:13:51.663756+0000, chain.objs.size()=2 2024-03-09T15:36:06.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.156__shadow_.RoOaMQjQ4gOD36ME4lY_jq-B19wl01i_0 2024-03-09T15:36:06.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.156__shadow_.RoOaMQjQ4gOD36ME4lY_jq-B19wl01i_1 2024-03-09T15:36:06.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12268438193051546734', time=2024-03-09T17:14:19.856201+0000, chain.objs.size()=1 2024-03-09T15:36:06.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.163__shadow_.n7encKZAzrQ23Fuway4kTO13E9IPWPm_0 2024-03-09T15:36:06.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18404584890279051449', time=2024-03-09T17:14:54.922912+0000, chain.objs.size()=1 2024-03-09T15:36:06.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.174__shadow_.GoSZvKWHNP7ppkdyd1_2APF8V7x8DQQ_0 2024-03-09T15:36:06.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6977948049866094525', time=2024-03-09T17:21:54.861844+0000, chain.objs.size()=1 2024-03-09T15:36:06.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.248__shadow_.rY52uSiEy-XU6wKvpaHlOoeQ_cHnTIU_0 2024-03-09T15:36:06.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11331897177842498510', time=2024-03-09T17:24:00.251042+0000, chain.objs.size()=1 2024-03-09T15:36:06.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1TsUeWafcoWnYBvD5cZPvj_1Sia7GFG_0 2024-03-09T15:36:06.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12509502991652229906', time=2024-03-09T17:24:05.740556+0000, chain.objs.size()=1 2024-03-09T15:36:06.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iS0G_AtmHoWn_4pxNesgOgl0kHAUS13_0 2024-03-09T15:36:06.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8938564541676843116', time=2024-03-09T17:24:06.208010+0000, chain.objs.size()=1 2024-03-09T15:36:06.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-Ww0wsVDrG0iQ9Z93IUIipX_U0yry7A_0 2024-03-09T15:36:06.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.25623868031753114', time=2024-03-09T17:24:06.354932+0000, chain.objs.size()=1 2024-03-09T15:36:06.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.u___jt1jcM0FvxaynK-hjoTHkoQxDZ6_0 2024-03-09T15:36:06.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8149521655163004557', time=2024-03-09T17:24:13.140893+0000, chain.objs.size()=1 2024-03-09T15:36:06.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.128+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.llupHt3RwSLjNwK9fSQgm6FWpU_2k7H_0 2024-03-09T15:36:06.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13401304741223607759', time=2024-03-09T17:24:15.441009+0000, chain.objs.size()=1 2024-03-09T15:36:06.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.uffhP-vLKQ5QAZN7CL2zYLX7D8zLd2v_0 2024-03-09T15:36:06.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17813967805056380051', time=2024-03-09T17:24:17.424149+0000, chain.objs.size()=1 2024-03-09T15:36:06.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.46zw9pUg04PBf8rGFj13RGkWPqJM-V1_0 2024-03-09T15:36:06.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9398510532212373577', time=2024-03-09T17:24:17.765504+0000, chain.objs.size()=1 2024-03-09T15:36:06.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.zlzLpmLGv9G5irvkC5j7Huc2ymZbnB9_0 2024-03-09T15:36:06.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12357020411325266183', time=2024-03-09T17:24:18.023404+0000, chain.objs.size()=1 2024-03-09T15:36:06.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ySuzVVvP4woENe1c0aXngDWWBXVXwIr_0 2024-03-09T15:36:06.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9458530038821743617', time=2024-03-09T17:24:19.425589+0000, chain.objs.size()=1 2024-03-09T15:36:06.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.hyE9ziIwmi8WweAfwPpr6zYG6foP9Ge_0 2024-03-09T15:36:06.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11599978260146222237', time=2024-03-09T17:24:21.025705+0000, chain.objs.size()=1 2024-03-09T15:36:06.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.z_brw1cbutXkbK8e3wgpPmRs-5jhjBI_0 2024-03-09T15:36:06.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16315096980028408511', time=2024-03-09T17:24:21.417398+0000, chain.objs.size()=1 2024-03-09T15:36:06.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.jmu5e-FyyVIktGbxFSQizPnddu-d7BC_0 2024-03-09T15:36:06.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11097432092666881453', time=2024-03-09T17:24:22.558131+0000, chain.objs.size()=1 2024-03-09T15:36:06.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1k5o7ZzQLTE8SRkNwqwD_v5fQBjiRcm_0 2024-03-09T15:36:06.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1188726801673814011', time=2024-03-09T17:24:22.842272+0000, chain.objs.size()=1 2024-03-09T15:36:06.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LtFFO0_1zCjKZGaTdfDnaE-Se8vnU7z_0 2024-03-09T15:36:06.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13313169660461742411', time=2024-03-09T17:24:23.108596+0000, chain.objs.size()=1 2024-03-09T15:36:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.WzwRpVZwzmiXDsIIZnDQOv697xMI58B_0 2024-03-09T15:36:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5611450849294501347', time=2024-03-09T17:24:23.534188+0000, chain.objs.size()=1 2024-03-09T15:36:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.yE2CD5EtEn22dd6F0PjIAWOHjK-qcJz_0 2024-03-09T15:36:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5779706833334710730', time=2024-03-09T17:24:23.696200+0000, chain.objs.size()=1 2024-03-09T15:36:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.MrpjA76FFy2Ts4tXtB4ZEjxRiZsaBWd_0 2024-03-09T15:36:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8950345767817715122', time=2024-03-09T17:24:23.698562+0000, chain.objs.size()=1 2024-03-09T15:36:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.132+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.X8-mwzg2P7-BTwyOoOWmH-szKHDRLS-_0 2024-03-09T15:36:06.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1621754063009365541', time=2024-03-09T17:24:23.760615+0000, chain.objs.size()=1 2024-03-09T15:36:06.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HEzeiqZBR4lVxaNR1RzcbfsTpKl-2bj_0 2024-03-09T15:36:06.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10515863010973393675', time=2024-03-09T17:24:25.575205+0000, chain.objs.size()=1 2024-03-09T15:36:06.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.YaQq8f3I4SG56yDSA_66aaPQvph0Lvg_0 2024-03-09T15:36:06.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8194228126321257165', time=2024-03-09T17:24:26.170401+0000, chain.objs.size()=1 2024-03-09T15:36:06.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.kynnpA0fxsBMPsv7LmhTLkHhD-V9-HS_0 2024-03-09T15:36:06.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18401509025380034683', time=2024-03-09T17:24:26.790102+0000, chain.objs.size()=1 2024-03-09T15:36:06.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6S75aro0esr4hWi43Shjkhhfjk1EtTl_0 2024-03-09T15:36:06.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15451164333835017897', time=2024-03-09T17:24:28.715592+0000, chain.objs.size()=1 2024-03-09T15:36:06.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.a37RfLk0r5H0--D8NDfM6k2Fx9NDNyg_0 2024-03-09T15:36:06.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6421961552655634583', time=2024-03-09T17:24:29.270693+0000, chain.objs.size()=1 2024-03-09T15:36:06.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.dlyEkN7VSlB2ZyPk-uWZfc1tYodoQAQ_0 2024-03-09T15:36:06.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14126051163520630295', time=2024-03-09T17:24:29.681645+0000, chain.objs.size()=1 2024-03-09T15:36:06.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.136+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.eZKFZttm-cdcjsATk4v-Q1DcfzP1-NO_0 2024-03-09T15:36:06.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.858534055541760059', time=2024-03-09T17:24:31.263267+0000, chain.objs.size()=1 2024-03-09T15:36:06.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.D614QkI4Yu6X8T07cx6RhTscGobX1Jx_0 2024-03-09T15:36:06.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16726363773892415983', time=2024-03-09T17:24:59.253195+0000, chain.objs.size()=1 2024-03-09T15:36:06.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.265__shadow_.G-NixcAgtzSBmPHuZWFg9prkEk3kGWD_0 2024-03-09T15:36:06.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10695497680784562026', time=2024-03-09T17:25:16.319724+0000, chain.objs.size()=1 2024-03-09T15:36:06.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.271__shadow_.7YKAGqtt3lZOXkW-liIx_ZnquXlOfDc_0 2024-03-09T15:36:06.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13911249393511560239', time=2024-03-09T17:25:33.677737+0000, chain.objs.size()=1 2024-03-09T15:36:06.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.277__shadow_.nAWarhE6Y9zSN4e2vaCRd4YucjPLtdh_0 2024-03-09T15:36:06.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16573703322545414823', time=2024-03-09T17:26:01.517854+0000, chain.objs.size()=1 2024-03-09T15:36:06.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.1N0gATkLz9_t5UCaA9t8MzL_90JXSOc_0 2024-03-09T15:36:06.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9741183356967767931', time=2024-03-09T17:26:11.106872+0000, chain.objs.size()=1 2024-03-09T15:36:06.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.287__shadow_.as9DXvaTBCZEUqggbyJDdYn5V3ABict_0 2024-03-09T15:36:06.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13688621101051053368', time=2024-03-09T17:26:40.572846+0000, chain.objs.size()=1 2024-03-09T15:36:06.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.297__shadow_._hLu_DyzM1w4LZZrxPxPJKzt2-3dBR2_0 2024-03-09T15:36:06.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8255107251592008087', time=2024-03-09T17:30:38.354050+0000, chain.objs.size()=1 2024-03-09T15:36:06.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.309__shadow_.pb4VaKqo4kcNPwM58C1dxQRw4wOAOQi_0 2024-03-09T15:36:06.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.140+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.144+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-03-09T15:36:06.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=54, truncated=0, next_marker='' 2024-03-09T15:36:06.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9540085103032697766', time=2024-03-09T17:12:50.159679+0000, chain.objs.size()=2 2024-03-09T15:36:06.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.YebXhBntru98RB7Og7PhdSTISOcVHFL_0 2024-03-09T15:36:06.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.YebXhBntru98RB7Og7PhdSTISOcVHFL_1 2024-03-09T15:36:06.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_wRGDdgoeglEQoQyOXNGZkuoMdIymvOe', time=2024-03-09T17:13:03.517987+0000, chain.objs.size()=1 2024-03-09T15:36:06.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.A90Xn9Rb7sS1LSn2JdNzCggBqtVSd0u_0 2024-03-09T15:36:06.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_-qnoRnQq6MBtIfBCSrnkPHjNLV2Ybvu', time=2024-03-09T17:13:40.287218+0000, chain.objs.size()=1 2024-03-09T15:36:06.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__shadow_.qJAbMrcqhdQ4xS74nAK2cOsNqEvCsQP_0 2024-03-09T15:36:06.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7388702102226909199', time=2024-03-09T17:21:03.557202+0000, chain.objs.size()=1 2024-03-09T15:36:06.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.231__shadow_.DGg_fQOkerXHb5BdhSiATm_O740CVkj_0 2024-03-09T15:36:06.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1771308550090983881', time=2024-03-09T17:21:28.968159+0000, chain.objs.size()=1 2024-03-09T15:36:06.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.239__shadow_.WD3sBYKYXUKMYK0-OFqq0fAUVBwrqrV_0 2024-03-09T15:36:06.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15341592474049905081', time=2024-03-09T17:23:59.754576+0000, chain.objs.size()=1 2024-03-09T15:36:06.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.uvTuIc0pcraMxF1VVxQcPRcbpMl3CZn_0 2024-03-09T15:36:06.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10051330051812505798', time=2024-03-09T17:23:59.756595+0000, chain.objs.size()=1 2024-03-09T15:36:06.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.s3xmNuHVFD4v7feI0if5Q6bB9xpzpLa_0 2024-03-09T15:36:06.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.895164108210862055', time=2024-03-09T17:24:01.487937+0000, chain.objs.size()=1 2024-03-09T15:36:06.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iJO_wdwbA9lLVRDLTCAyOqxQTHNz2he_0 2024-03-09T15:36:06.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5529941926278280356', time=2024-03-09T17:24:01.571314+0000, chain.objs.size()=1 2024-03-09T15:36:06.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.vjU-GECxQxHWVttl40cf03SZ-TN36ZJ_0 2024-03-09T15:36:06.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15663003631263990493', time=2024-03-09T17:24:01.868108+0000, chain.objs.size()=1 2024-03-09T15:36:06.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.FKR2-PwUHN_RoIj7sFBxrXkAIKe58W4_0 2024-03-09T15:36:06.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8157191921186463379', time=2024-03-09T17:24:02.165751+0000, chain.objs.size()=1 2024-03-09T15:36:06.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.148+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.WXb5LOJn_Oe69MfdAPWMFHWAwU2HNe8_0 2024-03-09T15:36:06.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16324384301008074451', time=2024-03-09T17:24:03.491330+0000, chain.objs.size()=1 2024-03-09T15:36:06.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ayU-VRRRgknXVJZPsaVWfBREaJfbNqk_0 2024-03-09T15:36:06.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5863084418902730484', time=2024-03-09T17:24:03.492829+0000, chain.objs.size()=1 2024-03-09T15:36:06.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.F6OTFSfKi52lT09J4v-_Ow21kxWytzA_0 2024-03-09T15:36:06.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4391143526584567562', time=2024-03-09T17:24:03.494767+0000, chain.objs.size()=1 2024-03-09T15:36:06.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.X4JtmRu-dnRXXz14bI62XoTmZN0Yaz2_0 2024-03-09T15:36:06.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9361023753565000890', time=2024-03-09T17:24:04.474396+0000, chain.objs.size()=1 2024-03-09T15:36:06.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HynUBMYQlhGznSE8GmepQj14q2cLEb2_0 2024-03-09T15:36:06.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8396899153753176782', time=2024-03-09T17:24:05.078706+0000, chain.objs.size()=1 2024-03-09T15:36:06.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.QYEpj4YcvokJgDXIvjGUMKP3rEGgvzK_0 2024-03-09T15:36:06.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1534641164545672722', time=2024-03-09T17:24:05.227514+0000, chain.objs.size()=1 2024-03-09T15:36:06.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ABhciNVjoXawAc_oGszg8CiIotpHilj_0 2024-03-09T15:36:06.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17814092370820173122', time=2024-03-09T17:24:05.758436+0000, chain.objs.size()=1 2024-03-09T15:36:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.80lDDE6w9wy2KGxb4dAWyen0yIbtThO_0 2024-03-09T15:36:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1347996443365363569', time=2024-03-09T17:24:08.792203+0000, chain.objs.size()=1 2024-03-09T15:36:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.F1V7DtKdZcJqXee8UPxqo-GuV0cfgmz_0 2024-03-09T15:36:06.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17779398049289517532', time=2024-03-09T17:24:09.536267+0000, chain.objs.size()=1 2024-03-09T15:36:06.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0diBj1PENehRMzJ2xtLTUWrEPqFwqRB_0 2024-03-09T15:36:06.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15726991269883161946', time=2024-03-09T17:24:10.571542+0000, chain.objs.size()=1 2024-03-09T15:36:06.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.YACzeqq1qRqqS0p_kCnisox3Qo8krv0_0 2024-03-09T15:36:06.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7348399151615316936', time=2024-03-09T17:24:11.076583+0000, chain.objs.size()=1 2024-03-09T15:36:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.152+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fKj-3SIK_R66MVN4uS1EZoX3lcJ6S5l_0 2024-03-09T15:36:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7762427920974611623', time=2024-03-09T17:24:12.625812+0000, chain.objs.size()=1 2024-03-09T15:36:06.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.GIvw-TyNKSuKz-4whxcKDZYZbxiaTKu_0 2024-03-09T15:36:06.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5672534592703431821', time=2024-03-09T17:24:12.627468+0000, chain.objs.size()=1 2024-03-09T15:36:06.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.NZhxeoP-DDbEQprnGm5v_G_fAU6R4e2_0 2024-03-09T15:36:06.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15615172068053837000', time=2024-03-09T17:24:12.791481+0000, chain.objs.size()=1 2024-03-09T15:36:06.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.TcuFn5hIWcvkwzrcqT81HtxZvrQGKp6_0 2024-03-09T15:36:06.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15590146059853248598', time=2024-03-09T17:24:13.727112+0000, chain.objs.size()=1 2024-03-09T15:36:06.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qShgW6Ykbq1-8fNYTZIjwEnrxhtp2L4_0 2024-03-09T15:36:06.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13284942244485638057', time=2024-03-09T17:24:14.967936+0000, chain.objs.size()=1 2024-03-09T15:36:06.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qIRj0zwAr3xJKho_tywtomhLDWFphki_0 2024-03-09T15:36:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3463129490492246133', time=2024-03-09T17:24:15.162993+0000, chain.objs.size()=1 2024-03-09T15:36:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.CDdxsu49ELmcfSyI4Qz8oGYp1EdezFX_0 2024-03-09T15:36:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14063331589559032124', time=2024-03-09T17:24:16.441326+0000, chain.objs.size()=1 2024-03-09T15:36:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.icwc7xV4GweHJrFLrHl3mDWV-oxVpsi_0 2024-03-09T15:36:06.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17770387730098316768', time=2024-03-09T17:24:16.723753+0000, chain.objs.size()=1 2024-03-09T15:36:06.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.dvNIuOTiQ3QKoLhlb6f-aGxdc4pM0lY_0 2024-03-09T15:36:06.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5304372457999685233', time=2024-03-09T17:24:16.725712+0000, chain.objs.size()=1 2024-03-09T15:36:06.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2gOI5mF_K8poV8AmAkj1K8BStueqNMt_0 2024-03-09T15:36:06.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7018132533615197043', time=2024-03-09T17:24:17.187219+0000, chain.objs.size()=1 2024-03-09T15:36:06.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.XYi_UieEoT8Y3m2tNtCyROCxUmts9GY_0 2024-03-09T15:36:06.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4005432151322857476', time=2024-03-09T17:24:18.552714+0000, chain.objs.size()=1 2024-03-09T15:36:06.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.156+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sqnMDeHzMdWkARagarQA84sAhR1lDEy_0 2024-03-09T15:36:06.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17934300415947069285', time=2024-03-09T17:24:18.889993+0000, chain.objs.size()=1 2024-03-09T15:36:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.UWLVGUMFHaheYQPZYkQJEdMAGqyK-C5_0 2024-03-09T15:36:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7319262060889772723', time=2024-03-09T17:24:19.026420+0000, chain.objs.size()=1 2024-03-09T15:36:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.5qEnfLay68n_7Vuopx8INavLA4rG7iF_0 2024-03-09T15:36:06.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17486163871855897316', time=2024-03-09T17:24:19.087178+0000, chain.objs.size()=1 2024-03-09T15:36:06.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.306tqoM5azt1ojQDv1Hoj1fr2cQSktW_0 2024-03-09T15:36:06.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17385194482270935604', time=2024-03-09T17:24:19.205477+0000, chain.objs.size()=1 2024-03-09T15:36:06.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.UfUP9GlSAswGzQCx3OyH_-oytGQI-s0_0 2024-03-09T15:36:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3311858100420841634', time=2024-03-09T17:24:23.001652+0000, chain.objs.size()=1 2024-03-09T15:36:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.zSZ-bHBcMQFPejTd8E5tfVexO4eBmI5_0 2024-03-09T15:36:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1706671994817394447', time=2024-03-09T17:24:25.613049+0000, chain.objs.size()=1 2024-03-09T15:36:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.VAdNmweYfPDL-C_jr8_JmWny69sscuZ_0 2024-03-09T15:36:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7375338855452203511', time=2024-03-09T17:24:25.614474+0000, chain.objs.size()=1 2024-03-09T15:36:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6E0poeWQ5zqNZ2KglYloVQdBAsIe1Iw_0 2024-03-09T15:36:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.366137037481449040', time=2024-03-09T17:24:25.993933+0000, chain.objs.size()=1 2024-03-09T15:36:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.c-QVw_ArkGagArOh25ua2QD-Xzg-e8-_0 2024-03-09T15:36:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6624876314946485147', time=2024-03-09T17:24:26.046969+0000, chain.objs.size()=1 2024-03-09T15:36:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.r4ttFNGeZH0cm_Hpm_gEIfWvJf1s5Go_0 2024-03-09T15:36:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13348610206187342318', time=2024-03-09T17:24:26.049097+0000, chain.objs.size()=1 2024-03-09T15:36:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Sclsyz-4uvIZmGxwHU-RSMLTMRPFidN_0 2024-03-09T15:36:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9807128678474999550', time=2024-03-09T17:24:26.283236+0000, chain.objs.size()=1 2024-03-09T15:36:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iAshYPEfLm1Q5lPxUYSVas4n7ZE8IfS_0 2024-03-09T15:36:06.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11204003481998898233', time=2024-03-09T17:24:26.979739+0000, chain.objs.size()=1 2024-03-09T15:36:06.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.4i40-XcoanINJqO-sVhcllwN0pvnsz1_0 2024-03-09T15:36:06.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9010540014391462125', time=2024-03-09T17:24:27.797768+0000, chain.objs.size()=1 2024-03-09T15:36:06.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.B391vCqRNCgDCrhgpnfLwTH0-HXfEsq_0 2024-03-09T15:36:06.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1247301689023351021', time=2024-03-09T17:24:29.799300+0000, chain.objs.size()=1 2024-03-09T15:36:06.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.160+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.69HfQvtGyMIahN4Pyoo2ikQ8yzB1i31_0 2024-03-09T15:36:06.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15738681147797264691', time=2024-03-09T17:24:30.581409+0000, chain.objs.size()=1 2024-03-09T15:36:06.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.P0sFy_nFhGUgW5_8GtYfdSkpDAUri-O_0 2024-03-09T15:36:06.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11982150900408742778', time=2024-03-09T17:24:30.739288+0000, chain.objs.size()=1 2024-03-09T15:36:06.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.mGgS_smRAGakUb5w5OIwEKneGkQlcXr_0 2024-03-09T15:36:06.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.286155268609954422', time=2024-03-09T17:24:56.416858+0000, chain.objs.size()=1 2024-03-09T15:36:06.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.264__shadow_.lpHQv5VlibI89UYsKAIwbOXti2v_KQn_0 2024-03-09T15:36:06.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9430933105356168692', time=2024-03-09T17:25:22.121534+0000, chain.objs.size()=1 2024-03-09T15:36:06.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.273__shadow_.a37qqMpBz3RVIz1HJanSr0qRHrtx7HF_0 2024-03-09T15:36:06.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13266775752167419257', time=2024-03-09T17:25:42.649646+0000, chain.objs.size()=1 2024-03-09T15:36:06.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.280__shadow_.tEFnBSVK46rwCDfjt8LYM7cQsxSP0Zh_0 2024-03-09T15:36:06.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11285360067775467957', time=2024-03-09T17:26:07.672309+0000, chain.objs.size()=1 2024-03-09T15:36:06.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.nEzS1QzGX0RfL3n3lQNQhlLSRKaL425_0 2024-03-09T15:36:06.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16444800076001306563', time=2024-03-09T17:33:44.448162+0000, chain.objs.size()=1 2024-03-09T15:36:06.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.164+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.330__shadow_.PmWJTlzKmXwmgkNus1P1SN_jLP2H3eb_0 2024-03-09T15:36:06.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.168+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.172+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-03-09T15:36:06.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.172+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=37, truncated=0, next_marker='' 2024-03-09T15:36:06.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_uJnTa0Z5-YI0aVy9hnliEw-Ex3MMiqV', time=2024-03-09T17:13:03.442270+0000, chain.objs.size()=2 2024-03-09T15:36:06.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.jrn1gtQIoqzYXee24UbBjEV7bxx87uU_0 2024-03-09T15:36:06.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.jrn1gtQIoqzYXee24UbBjEV7bxx87uU_1 2024-03-09T15:36:06.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9628559810991221870', time=2024-03-09T17:14:03.423558+0000, chain.objs.size()=2 2024-03-09T15:36:06.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.157__multipart_foo.2~0T3Ea-PMHuvPpZ0rGzg3tl7rDRk-Kdh.1 2024-03-09T15:36:06.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.157__multipart_foo.2~0T3Ea-PMHuvPpZ0rGzg3tl7rDRk-Kdh.2 2024-03-09T15:36:06.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_2MDL9I_lbs2pvgbrrd6MurecLtpJHGe', time=2024-03-09T17:14:03.714627+0000, chain.objs.size()=2 2024-03-09T15:36:06.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.158__shadow_.a6gsPu5lLqAHTeWA4X9n7-EZ9TTelpt_0 2024-03-09T15:36:06.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.158__shadow_.a6gsPu5lLqAHTeWA4X9n7-EZ9TTelpt_1 2024-03-09T15:36:06.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14552113045791464329', time=2024-03-09T17:24:01.225802+0000, chain.objs.size()=1 2024-03-09T15:36:06.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.atdASarMYQ96IJ5YqmbZghz0e5sC-mD_0 2024-03-09T15:36:06.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9320279678442839732', time=2024-03-09T17:24:01.639233+0000, chain.objs.size()=1 2024-03-09T15:36:06.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qJr54HM92rjAKRNWKpfHjMj0JHKMd0E_0 2024-03-09T15:36:06.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2145837834658222143', time=2024-03-09T17:24:01.949480+0000, chain.objs.size()=1 2024-03-09T15:36:06.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.a9imZ43QrVJOv3hd_flkLr8QXCF9B5Y_0 2024-03-09T15:36:06.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8863964071783331497', time=2024-03-09T17:24:03.893726+0000, chain.objs.size()=1 2024-03-09T15:36:06.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7RNqg12jXhR3ghJD_rXL_GKuOxLN_rQ_0 2024-03-09T15:36:06.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15478591999167370103', time=2024-03-09T17:24:09.874053+0000, chain.objs.size()=1 2024-03-09T15:36:06.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Il5tXSfGNrJqiFDjs9dOlQ-oGjlw_s8_0 2024-03-09T15:36:06.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10139901478896448922', time=2024-03-09T17:24:10.557308+0000, chain.objs.size()=1 2024-03-09T15:36:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.176+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.r47Bg7WfCWrSQaY8Ug6XbofpcvZWODi_0 2024-03-09T15:36:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11078763029407632069', time=2024-03-09T17:24:10.906415+0000, chain.objs.size()=1 2024-03-09T15:36:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.X_MPUAyrry9FoioT2TmInazmjYizoIM_0 2024-03-09T15:36:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17867220967238017541', time=2024-03-09T17:24:11.029123+0000, chain.objs.size()=1 2024-03-09T15:36:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.a1aO90ggppOopqyxZzEG5APaklBDWlN_0 2024-03-09T15:36:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4394623437251331725', time=2024-03-09T17:24:12.654952+0000, chain.objs.size()=1 2024-03-09T15:36:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.RSdscGbIffFxHotxqyf8xufpouNuqQb_0 2024-03-09T15:36:06.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16433164020891040551', time=2024-03-09T17:24:13.044549+0000, chain.objs.size()=1 2024-03-09T15:36:06.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.56wP_T4bwyB1E9NWorG2gtUKufanNqf_0 2024-03-09T15:36:06.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17822820852051922166', time=2024-03-09T17:24:14.766668+0000, chain.objs.size()=1 2024-03-09T15:36:06.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.EOjQOFajxTcLMqliDLi5z10JpAKFvRc_0 2024-03-09T15:36:06.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3438777657846403898', time=2024-03-09T17:24:15.300408+0000, chain.objs.size()=1 2024-03-09T15:36:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.OSBTslcbwlpWkWjHIeksF6cKbv_UtSB_0 2024-03-09T15:36:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11338225247492746490', time=2024-03-09T17:24:16.347327+0000, chain.objs.size()=1 2024-03-09T15:36:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8MncE4Lh82aFU0zEywQEDWGSfK8-bPJ_0 2024-03-09T15:36:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18238428109406771738', time=2024-03-09T17:24:17.171270+0000, chain.objs.size()=1 2024-03-09T15:36:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PBF3rg_DKHg-igKfvbpqu6DGd07b46q_0 2024-03-09T15:36:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3274171488266315485', time=2024-03-09T17:24:20.605897+0000, chain.objs.size()=1 2024-03-09T15:36:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.EAIshilYjyAINrTWM5mAm_qMGiRwEPW_0 2024-03-09T15:36:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14157489032544158637', time=2024-03-09T17:24:20.945305+0000, chain.objs.size()=1 2024-03-09T15:36:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PAq-Kf7_E6wRUhUvmkn5f0zda3m5D8I_0 2024-03-09T15:36:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9214407556060861970', time=2024-03-09T17:24:21.449938+0000, chain.objs.size()=1 2024-03-09T15:36:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.eMFST72rNN8iB1mG1Hi4gK-j8JgPagb_0 2024-03-09T15:36:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15048055593542398185', time=2024-03-09T17:24:22.983507+0000, chain.objs.size()=1 2024-03-09T15:36:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.180+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.I_ulvAIG8z5MvawA4H0oS7y1XEITW1L_0 2024-03-09T15:36:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5899766342297774242', time=2024-03-09T17:24:23.477305+0000, chain.objs.size()=1 2024-03-09T15:36:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-z8scHQagXbG07kLFMhjtwAJ8EaCjyS_0 2024-03-09T15:36:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14785125707690602154', time=2024-03-09T17:24:24.607306+0000, chain.objs.size()=1 2024-03-09T15:36:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.JB-B9OZTao41hKIHk26ei52l2T3cDYt_0 2024-03-09T15:36:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10254633505242881615', time=2024-03-09T17:24:25.523697+0000, chain.objs.size()=1 2024-03-09T15:36:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.haH9iNNQfLhBgvFWpsI_UVYCVyT4Pu__0 2024-03-09T15:36:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4763492456089257736', time=2024-03-09T17:24:27.362560+0000, chain.objs.size()=1 2024-03-09T15:36:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.vbemhROpiaFME9AaYjkv4KOAGV5asoW_0 2024-03-09T15:36:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1043636697449194489', time=2024-03-09T17:24:28.786769+0000, chain.objs.size()=1 2024-03-09T15:36:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.5jhGPp6H_TvTqNf22tUzmU9cTENaMmY_0 2024-03-09T15:36:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6275451673603939594', time=2024-03-09T17:24:29.800832+0000, chain.objs.size()=1 2024-03-09T15:36:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-ImpKxpAWKHDCRq5Fpu8Kp6yoOAiQWJ_0 2024-03-09T15:36:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6870544461158497573', time=2024-03-09T17:24:30.327730+0000, chain.objs.size()=1 2024-03-09T15:36:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.FnCgEv-qF_FSZBE16gzdzBZSdH_mO3O_0 2024-03-09T15:36:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5189704663391801770', time=2024-03-09T17:24:31.052100+0000, chain.objs.size()=1 2024-03-09T15:36:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.NMay0kM7TToeXJDOjvzwal5DVUG-nsO_0 2024-03-09T15:36:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1896594819604414800', time=2024-03-09T17:25:25.142369+0000, chain.objs.size()=1 2024-03-09T15:36:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.274__shadow_.EyFwS7raCjE610n6sl2cr-eFlS14Yp2_0 2024-03-09T15:36:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2531345270391895825', time=2024-03-09T17:25:25.151609+0000, chain.objs.size()=1 2024-03-09T15:36:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.274__shadow_.-hwR7aDtr7WjhAizWSCYCWHctjeq9M7_0 2024-03-09T15:36:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17895132281884925754', time=2024-03-09T17:25:48.375395+0000, chain.objs.size()=1 2024-03-09T15:36:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.282__shadow_.B9R1X1aNg54ICQ9ExY9NDLfDgX2iH48_0 2024-03-09T15:36:06.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17973159915539091756', time=2024-03-09T17:26:01.401799+0000, chain.objs.size()=1 2024-03-09T15:36:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.68IPz4jBuZlrQ5XCrQ0O29rz39YcIvk_0 2024-03-09T15:36:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4570747523646748908', time=2024-03-09T17:26:08.137708+0000, chain.objs.size()=1 2024-03-09T15:36:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.184+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.kd03-xa292cNijxw2Jm0D2dBDVaLSwI_0 2024-03-09T15:36:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.188+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12693931043537313934', time=2024-03-09T17:26:19.751877+0000, chain.objs.size()=1 2024-03-09T15:36:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.188+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.290__shadow_.Dss0e2JZC6Y6l4-TIA9BrrHaqxYOmcN_0 2024-03-09T15:36:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.188+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3705764713224283175', time=2024-03-09T17:26:34.676270+0000, chain.objs.size()=1 2024-03-09T15:36:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.188+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.295__shadow_.bMeNpaeYFVR90Vju32jMviH0l8uy3tb_0 2024-03-09T15:36:06.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.188+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13665614037483964669', time=2024-03-09T17:31:47.411261+0000, chain.objs.size()=1 2024-03-09T15:36:06.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.188+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.319__shadow_.2bmtnp-iVR_dI75_Km7vcsEL70xbXZf_0 2024-03-09T15:36:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.188+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.192+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-03-09T15:36:06.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.192+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=55, truncated=0, next_marker='' 2024-03-09T15:36:06.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6101490697842449716', time=2024-03-09T17:14:59.696673+0000, chain.objs.size()=1 2024-03-09T15:36:06.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.176__shadow_.GD18OXfu3GghWRFFBENSJzs-gkR920t_0 2024-03-09T15:36:06.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3114115547873851371', time=2024-03-09T17:16:08.269375+0000, chain.objs.size()=1 2024-03-09T15:36:06.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.190__shadow_.Ui75SwbXELa8AuqZxOeDlGepfWfPxfJ_0 2024-03-09T15:36:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15197010554332517091', time=2024-03-09T17:16:30.650706+0000, chain.objs.size()=1 2024-03-09T15:36:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.196__shadow_.N3HV7ZUKjivwMFYSg4Hmeyx17Zu0Kqc_0 2024-03-09T15:36:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4478051160512427670', time=2024-03-09T17:20:45.929923+0000, chain.objs.size()=1 2024-03-09T15:36:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.226__shadow_.YPndM9FPpm-oubFesNv3dy4uWglqSoD_0 2024-03-09T15:36:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4263422043097294261', time=2024-03-09T17:20:51.716787+0000, chain.objs.size()=1 2024-03-09T15:36:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.228__shadow_.Zffkw8ZKRRct7jP4_BWJsKj_2xjip7k_0 2024-03-09T15:36:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9478275515208105139', time=2024-03-09T17:21:03.542301+0000, chain.objs.size()=1 2024-03-09T15:36:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.231__shadow_.ULjH6GeNo08eOCnIasIdiRrAkaig0aR_0 2024-03-09T15:36:06.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3685082396207734538', time=2024-03-09T17:21:11.480489+0000, chain.objs.size()=1 2024-03-09T15:36:06.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.234__shadow_.WxXg1I1mI00kAX23DKmhArviFW9RwM3_0 2024-03-09T15:36:06.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12384669900569427377', time=2024-03-09T17:21:28.967241+0000, chain.objs.size()=1 2024-03-09T15:36:06.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.239__shadow_.iqr4NeAReS5sYlPHyGJKZfsO4SuyVyX_0 2024-03-09T15:36:06.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12096038873177077933', time=2024-03-09T17:21:31.941374+0000, chain.objs.size()=1 2024-03-09T15:36:06.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.240__shadow_.eMCL9KGhytYlJeM2_O41TnIHMM213Fr_0 2024-03-09T15:36:06.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11286652349795649720', time=2024-03-09T17:21:49.183103+0000, chain.objs.size()=1 2024-03-09T15:36:06.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.246__shadow_.DCvrSdWfO8HU9klWrLWZKfoq9REZf_T_0 2024-03-09T15:36:06.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14985611107946361567', time=2024-03-09T17:22:03.275009+0000, chain.objs.size()=1 2024-03-09T15:36:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.251__shadow_.u0Gz3VcZCxfw3kFeQJ83N8hHXkH102E_0 2024-03-09T15:36:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7051222549899818170', time=2024-03-09T17:23:58.901409+0000, chain.objs.size()=1 2024-03-09T15:36:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.196+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.D6T2Q4Vm3ZRurKhAw6_U6I8isSDJPmT_0 2024-03-09T15:36:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.200+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6970885988183465710', time=2024-03-09T17:24:00.091933+0000, chain.objs.size()=1 2024-03-09T15:36:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.200+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.vfMcD0eDuDpTkGadqEUkq9Ww9mwgIZH_0 2024-03-09T15:36:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.200+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8089556624197625768', time=2024-03-09T17:24:00.397409+0000, chain.objs.size()=1 2024-03-09T15:36:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.200+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.YR1HiyjgbtwYEgXSOo0DoWbl7BkAiAP_0 2024-03-09T15:36:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.200+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17621412348726390285', time=2024-03-09T17:24:01.024177+0000, chain.objs.size()=1 2024-03-09T15:36:06.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.200+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Ig3A3EDz-8e1FNrfuaDuJPNEPs3kuI4_0 2024-03-09T15:36:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.200+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18001599352087420347', time=2024-03-09T17:24:01.668876+0000, chain.objs.size()=1 2024-03-09T15:36:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.200+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.4vFVCGHIJycpE4rPLdWZ6rbv9zpn0-M_0 2024-03-09T15:36:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.200+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10118481811156810958', time=2024-03-09T17:24:01.915107+0000, chain.objs.size()=1 2024-03-09T15:36:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.200+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ycUkCJHgBoCnipKLxWZ2A2Lwb2sS4ov_0 2024-03-09T15:36:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2177890971151570409', time=2024-03-09T17:24:06.810284+0000, chain.objs.size()=1 2024-03-09T15:36:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.WH9oqZHGtoQk4NwBMMCoMq3I4Lj0Zwa_0 2024-03-09T15:36:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6840433620712480534', time=2024-03-09T17:24:06.814579+0000, chain.objs.size()=1 2024-03-09T15:36:06.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.hZtYQsH05gZsLgXgeJcxYRKCTB5BtrH_0 2024-03-09T15:36:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9762539705656532384', time=2024-03-09T17:24:07.922267+0000, chain.objs.size()=1 2024-03-09T15:36:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.XTNF3e4sXdb5MGKlsDpK3opOVvIpG-V_0 2024-03-09T15:36:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5403367199442991273', time=2024-03-09T17:24:08.306011+0000, chain.objs.size()=1 2024-03-09T15:36:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.D1UIWhMqZEv01I78jvFQb9r9IQFozr-_0 2024-03-09T15:36:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5483696588507813907', time=2024-03-09T17:24:09.320860+0000, chain.objs.size()=1 2024-03-09T15:36:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.hA04Awj0ZsYKQdU_g2Ch-ecfbzroupC_0 2024-03-09T15:36:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12031574427179810537', time=2024-03-09T17:24:10.445610+0000, chain.objs.size()=1 2024-03-09T15:36:06.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.OiG_1-REklsPNZ-hmyVVEIo7wbGo5Jt_0 2024-03-09T15:36:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1372807495360608483', time=2024-03-09T17:24:12.208772+0000, chain.objs.size()=1 2024-03-09T15:36:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.MFJ7-8nSMijUrUDHcMOt0wp-Vn3j0Ff_0 2024-03-09T15:36:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17219472267442858132', time=2024-03-09T17:24:12.261524+0000, chain.objs.size()=1 2024-03-09T15:36:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6DWQUx_d0fXgrr9tWp6nzQP0RsxB6FE_0 2024-03-09T15:36:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8291026333433585408', time=2024-03-09T17:24:13.097262+0000, chain.objs.size()=1 2024-03-09T15:36:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Bb4DvzU5sN4pV8vo5d6UzqxoR9VqxTs_0 2024-03-09T15:36:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7342196719331588074', time=2024-03-09T17:24:14.036459+0000, chain.objs.size()=1 2024-03-09T15:36:06.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8aml1dDpcrvg2Z3owE5DED2EjDcgg00_0 2024-03-09T15:36:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13160625459756780306', time=2024-03-09T17:24:14.128708+0000, chain.objs.size()=1 2024-03-09T15:36:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.204+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PNdFJLWt3GexBkn_wVVJbo4hoYaEIZz_0 2024-03-09T15:36:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7738543472474894373', time=2024-03-09T17:24:15.300693+0000, chain.objs.size()=1 2024-03-09T15:36:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.v8QNmYIOHaLPloOecxaesPXPRMB1s5T_0 2024-03-09T15:36:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17056293543561463302', time=2024-03-09T17:24:16.799722+0000, chain.objs.size()=1 2024-03-09T15:36:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0xmMGLhF60RQI84CmJDBf1yPsAkwjgi_0 2024-03-09T15:36:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7072585160699920377', time=2024-03-09T17:24:16.940178+0000, chain.objs.size()=1 2024-03-09T15:36:06.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.UzGOyglrrFL3uHoQJhaBo6-wD8DNBFP_0 2024-03-09T15:36:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6036271950901017464', time=2024-03-09T17:24:18.531546+0000, chain.objs.size()=1 2024-03-09T15:36:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HCNexkdwk8xqp35dmto1_y5Fy2bkfyG_0 2024-03-09T15:36:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4525016838654895233', time=2024-03-09T17:24:19.518788+0000, chain.objs.size()=1 2024-03-09T15:36:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tf2rL02FgseNJQF9aAMZ__ufY0PM27w_0 2024-03-09T15:36:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2455038274230803749', time=2024-03-09T17:24:20.926224+0000, chain.objs.size()=1 2024-03-09T15:36:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.zyz4stm7H8ST_HJ157xaY9r7l7CYYTS_0 2024-03-09T15:36:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7562217845309463593', time=2024-03-09T17:24:21.000962+0000, chain.objs.size()=1 2024-03-09T15:36:06.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.eLER8lzEezuTO39qOwt6hvr9SHpEuG9_0 2024-03-09T15:36:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2509798275202882232', time=2024-03-09T17:24:24.594196+0000, chain.objs.size()=1 2024-03-09T15:36:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7reDLCTGuaoOM4q7DBuZr6xMPhZyarM_0 2024-03-09T15:36:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4517139815582805671', time=2024-03-09T17:24:25.140482+0000, chain.objs.size()=1 2024-03-09T15:36:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.BLkZZ1nifFQrj1W_h3ifL7_TmDzyEXs_0 2024-03-09T15:36:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4939426772164627409', time=2024-03-09T17:24:26.732276+0000, chain.objs.size()=1 2024-03-09T15:36:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fEsO6BdAcwPDRPx1xjHAReSpyckVzP5_0 2024-03-09T15:36:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1722733782071506183', time=2024-03-09T17:24:28.718711+0000, chain.objs.size()=1 2024-03-09T15:36:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.208+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3-aeRK2_e-Ry60MW6kSkD05sN1bmp21_0 2024-03-09T15:36:06.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13043217963528308144', time=2024-03-09T17:24:29.097259+0000, chain.objs.size()=1 2024-03-09T15:36:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0xp3F3sau7rOIFOSWSi5cNg0apkRxu1_0 2024-03-09T15:36:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4014945060782626513', time=2024-03-09T17:24:29.670849+0000, chain.objs.size()=1 2024-03-09T15:36:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.m9XOcVwySPq3XBj_hkucLgaB7uiXgqQ_0 2024-03-09T15:36:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10615291289808168161', time=2024-03-09T17:24:29.993791+0000, chain.objs.size()=1 2024-03-09T15:36:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.w872rlaPtK3Dp0sB_OArJ4n9usBi2jU_0 2024-03-09T15:36:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5663949343309396518', time=2024-03-09T17:24:30.036516+0000, chain.objs.size()=1 2024-03-09T15:36:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9mRxvrJMr2wB13UMMOeFhVa5FsOYgXD_0 2024-03-09T15:36:06.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18209750542975694530', time=2024-03-09T17:24:30.323191+0000, chain.objs.size()=1 2024-03-09T15:36:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3qUjsKtqrgD0iZcLxb_O7wXTl_Q4cPk_0 2024-03-09T15:36:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7868659354542383532', time=2024-03-09T17:24:30.837412+0000, chain.objs.size()=1 2024-03-09T15:36:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LTvUK-SqIROaGhM0bDQlVo_AECGV2Zf_0 2024-03-09T15:36:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11484400846623910693', time=2024-03-09T17:24:30.839794+0000, chain.objs.size()=1 2024-03-09T15:36:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9-1FXV6HjJ8LIbgiu1fgK51zLQwtnOU_0 2024-03-09T15:36:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4061496185437596538', time=2024-03-09T17:24:31.251440+0000, chain.objs.size()=1 2024-03-09T15:36:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.CqSqKomqMU3bBUKFVsG8zEg4zNzaKp6_0 2024-03-09T15:36:06.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13395084993643709552', time=2024-03-09T17:24:31.264050+0000, chain.objs.size()=1 2024-03-09T15:36:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Evbq83do-HApNko718IjxGbk7pmJagK_0 2024-03-09T15:36:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1031743696057227480', time=2024-03-09T17:24:37.081912+0000, chain.objs.size()=1 2024-03-09T15:36:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.257__shadow_.APSfha0qSE5lKS0qKeyjmDJUt-3d9My_0 2024-03-09T15:36:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15265650445590817161', time=2024-03-09T17:25:07.568801+0000, chain.objs.size()=1 2024-03-09T15:36:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.268__shadow_.0jII0sJGoyNClXlHNM6pO9UQVa4PjkL_0 2024-03-09T15:36:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3855474653568710085', time=2024-03-09T17:25:27.988358+0000, chain.objs.size()=1 2024-03-09T15:36:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.275__shadow_.9kNDRf-D1j_ANK6VKt7xiAmGtEISTZO_0 2024-03-09T15:36:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8192863359718285293', time=2024-03-09T17:25:36.643640+0000, chain.objs.size()=1 2024-03-09T15:36:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.278__shadow_.a_986XtGsXaqkXoBnxGzPi8bRY184kC_0 2024-03-09T15:36:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7750962785425416687', time=2024-03-09T17:26:01.463828+0000, chain.objs.size()=1 2024-03-09T15:36:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.riZVVl3K3UPBuQybz3-vBnwHfzb6xdT_0 2024-03-09T15:36:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2910590276974167925', time=2024-03-09T17:26:07.795343+0000, chain.objs.size()=1 2024-03-09T15:36:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.Sljs0E_sbqQSF3Vu9B3Yg-AuURcaF0W_0 2024-03-09T15:36:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5328876666540170083', time=2024-03-09T17:31:52.220453+0000, chain.objs.size()=1 2024-03-09T15:36:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.212+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.320__shadow_.s3u8CbGsG28BJKciGEXlumE4ZryY46n_0 2024-03-09T15:36:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.216+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-03-09T15:36:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=51, truncated=0, next_marker='' 2024-03-09T15:36:06.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13419044806524324384', time=2024-03-09T17:11:46.923921+0000, chain.objs.size()=1 2024-03-09T15:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.145__shadow_.K_HKmyCWqbZVbEEIwv8A3QZH4EBdKUT_0 2024-03-09T15:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1259273649711990740', time=2024-03-09T17:14:47.354965+0000, chain.objs.size()=1 2024-03-09T15:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.171__shadow_.atGsFXX0D8HRXED9MYZk0L8vLWMcgVL_0 2024-03-09T15:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5366029508867084464', time=2024-03-09T17:20:59.171054+0000, chain.objs.size()=1 2024-03-09T15:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.230__shadow_.1Xt_dvhE0b0RsBMTcnsdGf04NB63j_X_0 2024-03-09T15:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18311495102914223352', time=2024-03-09T17:20:59.172744+0000, chain.objs.size()=1 2024-03-09T15:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.230__shadow_.gyRzYEYpLLAaX87YItYW48-IbRek8n8_0 2024-03-09T15:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9559047852011414877', time=2024-03-09T17:21:23.173227+0000, chain.objs.size()=1 2024-03-09T15:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.237__shadow_.wlyWp5Oq6q3umoK0qEnPMIplXKjISMs_0 2024-03-09T15:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5756836211208920192', time=2024-03-09T17:21:43.476769+0000, chain.objs.size()=1 2024-03-09T15:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.244__shadow_.hlHaZkWdAo1LxWx8kH3RDWywvsdizkE_0 2024-03-09T15:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2644868082183186054', time=2024-03-09T17:23:58.856158+0000, chain.objs.size()=1 2024-03-09T15:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.16WusbmsSNMW5h452KpPsgH9VVqbvA3_0 2024-03-09T15:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8823045063421752606', time=2024-03-09T17:24:03.962854+0000, chain.objs.size()=1 2024-03-09T15:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.JfFQJRiduC9FnrPnJ8DSt0RFfW3ToBL_0 2024-03-09T15:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14778466149870412288', time=2024-03-09T17:24:04.324181+0000, chain.objs.size()=1 2024-03-09T15:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.yKNxfEPljHUPMbaE8HzJhxyfrw9KTi1_0 2024-03-09T15:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4686144044615919257', time=2024-03-09T17:24:05.774483+0000, chain.objs.size()=1 2024-03-09T15:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.GZySTZiM2xHsPczb3Gq-uI6FqNVYfPZ_0 2024-03-09T15:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2628304606610713111', time=2024-03-09T17:24:07.939396+0000, chain.objs.size()=1 2024-03-09T15:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.UpiNugpvpOlSejpB78fr7xffXjPUGhF_0 2024-03-09T15:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6377803995498205105', time=2024-03-09T17:24:08.315966+0000, chain.objs.size()=1 2024-03-09T15:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.220+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ujoNqRQBawP9wf7ZPGwls3HDW9cSvC1_0 2024-03-09T15:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3613566885877658140', time=2024-03-09T17:24:08.717035+0000, chain.objs.size()=1 2024-03-09T15:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.kubhuge3NJp3cVumqaZUCtRT0bIyERg_0 2024-03-09T15:36:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12453854305600122380', time=2024-03-09T17:24:13.860687+0000, chain.objs.size()=1 2024-03-09T15:36:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.lFM3OhuTj7G2xz4pULcdbGd-fQZNekg_0 2024-03-09T15:36:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2468502062681717991', time=2024-03-09T17:24:14.303198+0000, chain.objs.size()=1 2024-03-09T15:36:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.lrahOZ5teDxDB4kX7nNnc1DeUIv2-8e_0 2024-03-09T15:36:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3530792634119440989', time=2024-03-09T17:24:14.581766+0000, chain.objs.size()=1 2024-03-09T15:36:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HAZQnYvVcCD0VA_d3lCUnszysGuldxy_0 2024-03-09T15:36:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8036638749114248327', time=2024-03-09T17:24:14.583638+0000, chain.objs.size()=1 2024-03-09T15:36:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.UQV13PGQ9IvgNMnX4gs3CCZrLyNpOoW_0 2024-03-09T15:36:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8889299498291706304', time=2024-03-09T17:24:17.815660+0000, chain.objs.size()=1 2024-03-09T15:36:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.CfaiznvfLYdJKAkM7nLx5PiKjgOs1vE_0 2024-03-09T15:36:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5490954366939244589', time=2024-03-09T17:24:18.488709+0000, chain.objs.size()=1 2024-03-09T15:36:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.GMM63enKHJ8seZ-_yv2HGe4Wo93IQDr_0 2024-03-09T15:36:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3642325170408279065', time=2024-03-09T17:24:18.490323+0000, chain.objs.size()=1 2024-03-09T15:36:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.t-UBch7GhMZKcacNWOJ_Pcu29LjIo5c_0 2024-03-09T15:36:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4246869740969695083', time=2024-03-09T17:24:18.537965+0000, chain.objs.size()=1 2024-03-09T15:36:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tOtHjCSj32gRddfxAo37PuRZQ8jOd0T_0 2024-03-09T15:36:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1774655751672254003', time=2024-03-09T17:24:18.900111+0000, chain.objs.size()=1 2024-03-09T15:36:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.74VUEZEijS69XHeKXd6MaO6elmq1PRX_0 2024-03-09T15:36:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17237999441052160094', time=2024-03-09T17:24:18.901828+0000, chain.objs.size()=1 2024-03-09T15:36:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9iKlVMXPW6diyptP5SM5jNTLGVnG3mZ_0 2024-03-09T15:36:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7700240658687851245', time=2024-03-09T17:24:19.621774+0000, chain.objs.size()=1 2024-03-09T15:36:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.224+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PoFrco69mIH9SYdkcyna98QOjvB8kFb_0 2024-03-09T15:36:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17047629111530741789', time=2024-03-09T17:24:20.421253+0000, chain.objs.size()=1 2024-03-09T15:36:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6SoHlJ_qkA1ZT-6HnT5jv-bDoU3Tqgf_0 2024-03-09T15:36:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12132221107576682227', time=2024-03-09T17:24:20.427976+0000, chain.objs.size()=1 2024-03-09T15:36:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-C8yHvZpUm4k9VQEGj7bDvfBGujRx5o_0 2024-03-09T15:36:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10343051117606861808', time=2024-03-09T17:24:20.432844+0000, chain.objs.size()=1 2024-03-09T15:36:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.hM2G-JELtvSJTSxfiU8QGw75rbQW9re_0 2024-03-09T15:36:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12872568905915645598', time=2024-03-09T17:24:21.020518+0000, chain.objs.size()=1 2024-03-09T15:36:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.86u5D2wFYWD_ttmkaL1fUxdYaeMoP7S_0 2024-03-09T15:36:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1445935924561002715', time=2024-03-09T17:24:21.362050+0000, chain.objs.size()=1 2024-03-09T15:36:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.72TA9Ds__BQcxSYaea64cZbhluy6K4j_0 2024-03-09T15:36:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6484234762205541733', time=2024-03-09T17:24:21.470103+0000, chain.objs.size()=1 2024-03-09T15:36:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.mwr-qZ5FYADc_AqG-_c0fuApxJPNj3U_0 2024-03-09T15:36:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18189502437665756581', time=2024-03-09T17:24:23.045175+0000, chain.objs.size()=1 2024-03-09T15:36:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HV9khgtoEbvyvRMfsIw-S96jXa0XSab_0 2024-03-09T15:36:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17861443701388991331', time=2024-03-09T17:24:23.172149+0000, chain.objs.size()=1 2024-03-09T15:36:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.DD9SVha9BJkKViS4WfGX7CAe3hoauPi_0 2024-03-09T15:36:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15030444522151391911', time=2024-03-09T17:24:23.180946+0000, chain.objs.size()=1 2024-03-09T15:36:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.XwPqAaQPYcAYKdYL3jCwBZuGZTXPGk__0 2024-03-09T15:36:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11775139430960951979', time=2024-03-09T17:24:23.487599+0000, chain.objs.size()=1 2024-03-09T15:36:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.z7ybKd9AY-wdwP7wHiroH1eoukKQaO9_0 2024-03-09T15:36:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13292991273754011107', time=2024-03-09T17:24:25.912422+0000, chain.objs.size()=1 2024-03-09T15:36:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.t4wwX-0Ep2AmsTpxtxbjGYcB9U-dexx_0 2024-03-09T15:36:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11232816467661057974', time=2024-03-09T17:24:26.409250+0000, chain.objs.size()=1 2024-03-09T15:36:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.CmYQ_jcEqzNozdW2Lf4Hhub_8KOOGjD_0 2024-03-09T15:36:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9559637871318408880', time=2024-03-09T17:24:26.752184+0000, chain.objs.size()=1 2024-03-09T15:36:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.228+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.dL5NfftIb3_pVhTLsHQuPyCU2OTANCT_0 2024-03-09T15:36:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4986253397505245963', time=2024-03-09T17:24:26.754481+0000, chain.objs.size()=1 2024-03-09T15:36:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.rLC-doFm_6acbv2z0GUmLOi0CqZ0xrG_0 2024-03-09T15:36:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5637297614988273143', time=2024-03-09T17:24:27.564449+0000, chain.objs.size()=1 2024-03-09T15:36:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.4zBGCGIuStXnjz9MCZlTZlRDC-UeYTO_0 2024-03-09T15:36:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6092147142415921518', time=2024-03-09T17:24:28.687528+0000, chain.objs.size()=1 2024-03-09T15:36:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.lr6SkknSZ2pNrDxI1TB-_TXKbAB33jh_0 2024-03-09T15:36:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14502004436310832810', time=2024-03-09T17:24:29.594882+0000, chain.objs.size()=1 2024-03-09T15:36:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.280HpneUdl_4-ubh2pSwt1gZtCr-Bya_0 2024-03-09T15:36:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12065404126373012949', time=2024-03-09T17:24:30.043617+0000, chain.objs.size()=1 2024-03-09T15:36:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.k17_i21KijkdSlzwleNiR2gJQsSUww3_0 2024-03-09T15:36:06.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11324865872661357561', time=2024-03-09T17:24:30.591498+0000, chain.objs.size()=1 2024-03-09T15:36:06.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fPybbqvV3VktsSnNncHSJ2eDIHC5LfA_0 2024-03-09T15:36:06.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15132808818508183350', time=2024-03-09T17:24:30.786840+0000, chain.objs.size()=1 2024-03-09T15:36:06.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.cgnFPdLZs03m2DTfnpntid92tropy0r_0 2024-03-09T15:36:06.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4966062392084852831', time=2024-03-09T17:24:31.378572+0000, chain.objs.size()=1 2024-03-09T15:36:06.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.JtuO6tFxxjO89yFycBgWm72dbVkxAEp_0 2024-03-09T15:36:06.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15636981002463104815', time=2024-03-09T17:25:13.479040+0000, chain.objs.size()=1 2024-03-09T15:36:06.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.270__shadow_.vqVAZVSFOKEwz-5IEZ0cMw1jToo_0t9_0 2024-03-09T15:36:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8387103229503707781', time=2024-03-09T17:25:19.297875+0000, chain.objs.size()=1 2024-03-09T15:36:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.272__shadow_.0d1YhoWBOe60TSUTPPONMsv0NX0Haeb_0 2024-03-09T15:36:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2926395457251633432', time=2024-03-09T17:26:07.798976+0000, chain.objs.size()=1 2024-03-09T15:36:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.qIGtPCZBaCPH76FSMFfKV5v2k-1QE0e_0 2024-03-09T15:36:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2837299979927112012', time=2024-03-09T17:26:25.980339+0000, chain.objs.size()=1 2024-03-09T15:36:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.292__shadow_.duTeen1XAC0TcKzBMmEHggNweAzeihL_0 2024-03-09T15:36:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14795821393827436772', time=2024-03-09T17:33:31.282902+0000, chain.objs.size()=1 2024-03-09T15:36:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.329__shadow_.aNcFlYt-W-JwtDHm-O6M2EEUtZwEbwl_0 2024-03-09T15:36:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14077869832616662174', time=2024-03-09T17:34:22.277276+0000, chain.objs.size()=1 2024-03-09T15:36:06.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.232+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.336__shadow_.VO91VtpVvDDRVdMLRyWRoP6Vom0zXo0_0 2024-03-09T15:36:06.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.236+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-03-09T15:36:06.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=49, truncated=0, next_marker='' 2024-03-09T15:36:06.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17215247221363845711', time=2024-03-09T17:11:46.251793+0000, chain.objs.size()=1 2024-03-09T15:36:06.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.144__shadow_.GP3mfj9B-mBN9vtDN67iIplYlZbYb7u_0 2024-03-09T15:36:06.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6505579538014381775', time=2024-03-09T17:16:13.575979+0000, chain.objs.size()=1 2024-03-09T15:36:06.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.192__shadow_.omxySZX3sa17M-F-ImdTVZlUbidQIG5_0 2024-03-09T15:36:06.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10318242664087181830', time=2024-03-09T17:20:48.810383+0000, chain.objs.size()=1 2024-03-09T15:36:06.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.227__shadow_.SUrFcD5BtZFzimPfQj0fC6SLpCOxGSC_0 2024-03-09T15:36:06.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6020275585100722726', time=2024-03-09T17:21:08.526552+0000, chain.objs.size()=1 2024-03-09T15:36:06.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.233__shadow_.ZgD9WVf7UewQn0THjvyxsosQG0J5Ip9_0 2024-03-09T15:36:06.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7648982404167143029', time=2024-03-09T17:23:58.865533+0000, chain.objs.size()=1 2024-03-09T15:36:06.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6zMOwMParobzsizWuEDWSOahRYZGbVX_0 2024-03-09T15:36:06.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11780942030872822230', time=2024-03-09T17:23:58.867857+0000, chain.objs.size()=1 2024-03-09T15:36:06.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.shjAz8O1-PX5SjEEHGq-t8ZSJ85B8QS_0 2024-03-09T15:36:06.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15377550148622123265', time=2024-03-09T17:23:59.321269+0000, chain.objs.size()=1 2024-03-09T15:36:06.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Ma7jwIlS1nQkd9_5gQITHNvxH43ce1X_0 2024-03-09T15:36:06.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1790466796773456653', time=2024-03-09T17:24:00.114599+0000, chain.objs.size()=1 2024-03-09T15:36:06.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.AeCSweeRhqkhdo5oDPzX1kE8z_stoow_0 2024-03-09T15:36:06.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10634978062122163281', time=2024-03-09T17:24:01.022663+0000, chain.objs.size()=1 2024-03-09T15:36:06.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.BC-0BnLR6Ve_S4hOLucb3nqAexvEiK7_0 2024-03-09T15:36:06.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16329783349077284689', time=2024-03-09T17:24:01.908963+0000, chain.objs.size()=1 2024-03-09T15:36:06.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.YLRIbYosFCcLTyhGH9wbDmUBrKpnDNH_0 2024-03-09T15:36:06.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7991562736078009867', time=2024-03-09T17:24:02.134790+0000, chain.objs.size()=1 2024-03-09T15:36:06.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8-7z8OGyl1Tmp98_EKzLrUxm4LOnEZa_0 2024-03-09T15:36:06.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17973689066093229121', time=2024-03-09T17:24:04.090317+0000, chain.objs.size()=1 2024-03-09T15:36:06.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.240+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.yf6HkqKTG_0hrbE7RZKwPgXXVVmr34z_0 2024-03-09T15:36:06.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5496147510517520734', time=2024-03-09T17:24:04.912744+0000, chain.objs.size()=1 2024-03-09T15:36:06.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.IGgwEvU5hgEK9Yb53FZOgNJv1rZEEQ-_0 2024-03-09T15:36:06.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10027386757505375442', time=2024-03-09T17:24:05.498556+0000, chain.objs.size()=1 2024-03-09T15:36:06.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.gDEOI2niUD00HxDlMZ_tpcprtVTFaJ0_0 2024-03-09T15:36:06.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9772444841041572312', time=2024-03-09T17:24:06.772574+0000, chain.objs.size()=1 2024-03-09T15:36:06.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.v187qlNCNiLQmsxA6jcfeBua3ueHsv0_0 2024-03-09T15:36:06.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7425127068602732889', time=2024-03-09T17:24:07.944758+0000, chain.objs.size()=1 2024-03-09T15:36:06.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HteQd52JCrz99rwaBlHZCofIgcsuFu3_0 2024-03-09T15:36:06.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1261664897909058853', time=2024-03-09T17:24:08.785715+0000, chain.objs.size()=1 2024-03-09T15:36:06.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Q8XaJnAFTifrIa-zwxjh8sAkG0ofhok_0 2024-03-09T15:36:06.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1078912972275142614', time=2024-03-09T17:24:10.186192+0000, chain.objs.size()=1 2024-03-09T15:36:06.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.l6n4U7NSHDCe0R353hvduc_Nd-l5EpJ_0 2024-03-09T15:36:06.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3177432681016517222', time=2024-03-09T17:24:10.783528+0000, chain.objs.size()=1 2024-03-09T15:36:06.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Ntiu5jWQUJzhlI3xaOfFPvs-ujDJITH_0 2024-03-09T15:36:06.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15416825752196936728', time=2024-03-09T17:24:10.827850+0000, chain.objs.size()=1 2024-03-09T15:36:06.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ViywFon0v_GAL-RXf52BzU2anrYvBOj_0 2024-03-09T15:36:06.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6957177472728411235', time=2024-03-09T17:24:12.219398+0000, chain.objs.size()=1 2024-03-09T15:36:06.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Kgoai0UT7CPXJ4KRN-RB_1Wqyg9QA8n_0 2024-03-09T15:36:06.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15601148965257405749', time=2024-03-09T17:24:13.791215+0000, chain.objs.size()=1 2024-03-09T15:36:06.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.19IAKnOzPEhhAnuGVVDyQB35yS6Dlj5_0 2024-03-09T15:36:06.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16917089119598756048', time=2024-03-09T17:24:14.300220+0000, chain.objs.size()=1 2024-03-09T15:36:06.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.H72ZSguQUgpQiEPBf52xcuLYGlFhjj-_0 2024-03-09T15:36:06.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14143148192023169253', time=2024-03-09T17:24:14.302788+0000, chain.objs.size()=1 2024-03-09T15:36:06.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.244+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.hzJU5uh_efjgKNL-nbeSwS0A4pbrf3Y_0 2024-03-09T15:36:06.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7643655432655620024', time=2024-03-09T17:24:14.886989+0000, chain.objs.size()=1 2024-03-09T15:36:06.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.OaNVk9Tvoe1_cBh_X4faA7HJ2Yq0KAV_0 2024-03-09T15:36:06.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7065408798305680637', time=2024-03-09T17:24:15.378470+0000, chain.objs.size()=1 2024-03-09T15:36:06.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.33LmevBze7cNRzrLE_fWp9naeZbOEjw_0 2024-03-09T15:36:06.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14884137534647512262', time=2024-03-09T17:24:16.266339+0000, chain.objs.size()=1 2024-03-09T15:36:06.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.W1cbm1tJdNZMELbkudeLUGtkFk_NTdz_0 2024-03-09T15:36:06.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8685719047367883023', time=2024-03-09T17:24:16.810317+0000, chain.objs.size()=1 2024-03-09T15:36:06.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.zkWRVAO47pxZMpZpd2GvrNs66cXvYrn_0 2024-03-09T15:36:06.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2500649888912540386', time=2024-03-09T17:24:16.958829+0000, chain.objs.size()=1 2024-03-09T15:36:06.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.o53cybEl4dZgMpsN-yB_AykjvFIHtq-_0 2024-03-09T15:36:06.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1034978413712253109', time=2024-03-09T17:24:19.113247+0000, chain.objs.size()=1 2024-03-09T15:36:06.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.wR5AMdQ_Urd7hhBtcWD6LhZGXEzOvG3_0 2024-03-09T15:36:06.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8529364667606455367', time=2024-03-09T17:24:19.331007+0000, chain.objs.size()=1 2024-03-09T15:36:06.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.UaJKpn4SETiSaPfcW5pCir3FECswISj_0 2024-03-09T15:36:06.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12512227396627091886', time=2024-03-09T17:24:19.631163+0000, chain.objs.size()=1 2024-03-09T15:36:06.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Z6YObngBaW9dAKudAhYNp87nERCahjq_0 2024-03-09T15:36:06.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2497076181868473736', time=2024-03-09T17:24:20.416400+0000, chain.objs.size()=1 2024-03-09T15:36:06.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_._Ec_uKJuB0-lVquU17G6K7Gojq6MRPV_0 2024-03-09T15:36:06.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1887996373820095812', time=2024-03-09T17:24:20.487964+0000, chain.objs.size()=1 2024-03-09T15:36:06.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.KNPKfCC1MgUhM4K6spaUGL1nfIlf515_0 2024-03-09T15:36:06.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7542194967699552637', time=2024-03-09T17:24:20.489787+0000, chain.objs.size()=1 2024-03-09T15:36:06.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.A2soS7CupYPCeulmx_feWYylaRyah6s_0 2024-03-09T15:36:06.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1633711968307523569', time=2024-03-09T17:24:20.491814+0000, chain.objs.size()=1 2024-03-09T15:36:06.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.afzy3lCgrnndoKifwry1Y5jlu3tDi_X_0 2024-03-09T15:36:06.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1208201269945508292', time=2024-03-09T17:24:21.130393+0000, chain.objs.size()=1 2024-03-09T15:36:06.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.116ZksN9MlXY3bhEatfu71Xga6zPKUR_0 2024-03-09T15:36:06.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14932377992681452832', time=2024-03-09T17:24:23.764188+0000, chain.objs.size()=1 2024-03-09T15:36:06.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2D85wGD0Lp8tfSI7rvQfx74-3a-ZERc_0 2024-03-09T15:36:06.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7673202957962776115', time=2024-03-09T17:24:25.513978+0000, chain.objs.size()=1 2024-03-09T15:36:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qsZeqrBcVC0doLJAZWsldB78wqVZz4J_0 2024-03-09T15:36:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7169167528984202143', time=2024-03-09T17:24:25.805456+0000, chain.objs.size()=1 2024-03-09T15:36:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iHw2A0YwcsCE74yefufq-ZGRwXKIa3X_0 2024-03-09T15:36:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14793524734435338303', time=2024-03-09T17:24:29.197429+0000, chain.objs.size()=1 2024-03-09T15:36:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.KcGl1yMLO6JWuk7Oan4SA8fU0JYF0zi_0 2024-03-09T15:36:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2401786327534422657', time=2024-03-09T17:24:31.194175+0000, chain.objs.size()=1 2024-03-09T15:36:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.248+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ysO6VTa9ZioSTHiG1qtXfjWTs-Xc7vm_0 2024-03-09T15:36:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3464322472091464370', time=2024-03-09T17:25:36.634361+0000, chain.objs.size()=1 2024-03-09T15:36:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.278__shadow_.MCJEFJo225Lv7jIwHb-XUOgpEB7w-hB_0 2024-03-09T15:36:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13000241862335165652', time=2024-03-09T17:25:51.346773+0000, chain.objs.size()=1 2024-03-09T15:36:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.283__shadow_.zCvjAmgBV7eUt7PfGuNezuZe8WlREYr_0 2024-03-09T15:36:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12493476687548103089', time=2024-03-09T17:26:08.123154+0000, chain.objs.size()=1 2024-03-09T15:36:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.ePZsdYN9hMEQuFRcvVgGRNPPigGzRuK_0 2024-03-09T15:36:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16492654254538730599', time=2024-03-09T17:26:11.124900+0000, chain.objs.size()=1 2024-03-09T15:36:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.287__shadow_.hDtCFM8g4dnyT1O_UZvXppE_BrRInAk_0 2024-03-09T15:36:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14417514432623089114', time=2024-03-09T17:26:43.467231+0000, chain.objs.size()=1 2024-03-09T15:36:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.298__shadow_.k_qf53bONc4cNsJeYbm6yQ1wgWj-rEf_0 2024-03-09T15:36:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.564696610870851283', time=2024-03-09T17:27:08.162626+0000, chain.objs.size()=1 2024-03-09T15:36:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.306__shadow_.zlCWK4cT-daF3WBj5Ap18pO1Ae-FYEz_0 2024-03-09T15:36:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3077490419423288104', time=2024-03-09T17:32:17.831187+0000, chain.objs.size()=1 2024-03-09T15:36:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.322__shadow_.6yYVOyvCFBTb-2BC7t8zESJddBfr5V8_0 2024-03-09T15:36:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.252+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.256+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-03-09T15:36:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.256+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=41, truncated=0, next_marker='' 2024-03-09T15:36:06.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5207924506063757643', time=2024-03-09T17:13:35.952141+0000, chain.objs.size()=3 2024-03-09T15:36:06.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__multipart_foo-FROZEN.2~FZsfRzRd_e9CsDcv4cXTrCcYJ3hj4cf.1 2024-03-09T15:36:06.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__multipart_foo-FROZEN.2~FZsfRzRd_e9CsDcv4cXTrCcYJ3hj4cf.2 2024-03-09T15:36:06.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__multipart_foo-FROZEN.2~FZsfRzRd_e9CsDcv4cXTrCcYJ3hj4cf.3 2024-03-09T15:36:06.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_DbcZbce-opICrpU9ePTw1ZIAX0CseU9', time=2024-03-09T17:13:38.009729+0000, chain.objs.size()=2 2024-03-09T15:36:06.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__shadow_.en7-CGdcxeu0BmgvrxJa8fj3lzmY5lA_0 2024-03-09T15:36:06.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__shadow_.en7-CGdcxeu0BmgvrxJa8fj3lzmY5lA_1 2024-03-09T15:36:06.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13078071966750147455', time=2024-03-09T17:20:34.203829+0000, chain.objs.size()=1 2024-03-09T15:36:06.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.221__shadow_.wG6vhlYjnhc9skd_Vh4-C_3H6EzF27O_0 2024-03-09T15:36:06.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3655333804728148157', time=2024-03-09T17:20:37.351070+0000, chain.objs.size()=1 2024-03-09T15:36:06.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.223__shadow_.IKGmgmgs5HvxKSwD8uTwK1vaLARf805_0 2024-03-09T15:36:06.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14268278641051214139', time=2024-03-09T17:20:51.699285+0000, chain.objs.size()=1 2024-03-09T15:36:06.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.228__shadow_.a7Yn8iOxRYk3Yv52EdZ0TNuBGmTo2Pe_0 2024-03-09T15:36:06.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18420732004168662234', time=2024-03-09T17:21:37.777599+0000, chain.objs.size()=1 2024-03-09T15:36:06.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.242__shadow_.im6o31vFxT14af6qctfhVYG9gdimpFc_0 2024-03-09T15:36:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3637646756918615196', time=2024-03-09T17:21:57.703338+0000, chain.objs.size()=1 2024-03-09T15:36:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.249__shadow_.PNGHto3Zbp47cIGYD_CClUsUO06Ho_Z_0 2024-03-09T15:36:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10979247832639522861', time=2024-03-09T17:21:57.724938+0000, chain.objs.size()=1 2024-03-09T15:36:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.249__shadow_.ncBB2oJ-sgb0f52Xs3ybQ1jn9aWB7nR_0 2024-03-09T15:36:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3937194012301820999', time=2024-03-09T17:23:59.500195+0000, chain.objs.size()=1 2024-03-09T15:36:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.260+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-cDHVFbMZCVAnH4Z69s1gOMiQnrfLds_0 2024-03-09T15:36:06.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9316110948415706783', time=2024-03-09T17:24:00.427830+0000, chain.objs.size()=1 2024-03-09T15:36:06.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.EezoFrbJQ-8Qj5bpVdoRn4777AN28le_0 2024-03-09T15:36:06.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2610878709888784927', time=2024-03-09T17:24:00.947677+0000, chain.objs.size()=1 2024-03-09T15:36:06.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.zac4tNEN2GPKhbbdOt30nOMhYlleMhy_0 2024-03-09T15:36:06.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4235429634538279862', time=2024-03-09T17:24:01.243713+0000, chain.objs.size()=1 2024-03-09T15:36:06.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iBJMSugczPEeoQiS20B6dc6wjibZe3N_0 2024-03-09T15:36:06.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7427934032085010001', time=2024-03-09T17:24:01.875860+0000, chain.objs.size()=1 2024-03-09T15:36:06.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LyAlo0nrgPLdiTz9_QPrMK6kliQAV4N_0 2024-03-09T15:36:06.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14884902807171491495', time=2024-03-09T17:24:01.948630+0000, chain.objs.size()=1 2024-03-09T15:36:06.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.BMO1PRS7ofTwfezJFsPRsb2-P44GoZG_0 2024-03-09T15:36:06.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1910475236028230566', time=2024-03-09T17:24:03.570220+0000, chain.objs.size()=1 2024-03-09T15:36:06.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Zj84WUQjwS22HAviEbjZL34IYVXLkfv_0 2024-03-09T15:36:06.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14597442762383719756', time=2024-03-09T17:24:03.980307+0000, chain.objs.size()=1 2024-03-09T15:36:06.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HGiDW4VijNui39RO9t4CcNEup6eAEjb_0 2024-03-09T15:36:06.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15085767151062203188', time=2024-03-09T17:24:04.594236+0000, chain.objs.size()=1 2024-03-09T15:36:06.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PzbCtALNpWLbVpjY7XcrSCh7zMJFXz4_0 2024-03-09T15:36:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11297556109398892767', time=2024-03-09T17:24:09.546985+0000, chain.objs.size()=1 2024-03-09T15:36:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.g3kdb1TZY3Mu_4McAw9HD4ZyUklh7Hl_0 2024-03-09T15:36:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.45131678395480673', time=2024-03-09T17:24:09.629272+0000, chain.objs.size()=1 2024-03-09T15:36:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.DMCyO0t554FMKEcQONPWaDr1Uu7CAve_0 2024-03-09T15:36:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14804579255012883309', time=2024-03-09T17:24:09.811687+0000, chain.objs.size()=1 2024-03-09T15:36:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ZMB0slj_TM_wUL0VhXbilspozMhU1vp_0 2024-03-09T15:36:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.63272523115772488', time=2024-03-09T17:24:10.757740+0000, chain.objs.size()=1 2024-03-09T15:36:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.264+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.q6_BZLG99-iTl3YTMfBqLAKUntgdTU9_0 2024-03-09T15:36:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16101189182245465978', time=2024-03-09T17:24:12.210943+0000, chain.objs.size()=1 2024-03-09T15:36:06.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.hnz5bG8BEzHY8BzdsI9Zzjv5QCKi0Kp_0 2024-03-09T15:36:06.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6866608412805463980', time=2024-03-09T17:24:12.654877+0000, chain.objs.size()=1 2024-03-09T15:36:06.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.N1mfue3o5Qbdul7mcHuC_OqIDHCQnEe_0 2024-03-09T15:36:06.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15111504106467794476', time=2024-03-09T17:24:14.981201+0000, chain.objs.size()=1 2024-03-09T15:36:06.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9kATIxxa8t0fJ9bdluNjLsJ5eOzi22X_0 2024-03-09T15:36:06.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13052467067589895690', time=2024-03-09T17:24:16.443503+0000, chain.objs.size()=1 2024-03-09T15:36:06.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.utcmE7nArU05vB_79r4_W701z9Eea-I_0 2024-03-09T15:36:06.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11888574891023815980', time=2024-03-09T17:24:16.836838+0000, chain.objs.size()=1 2024-03-09T15:36:06.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.e9E3i2IRiRk1AJbmGJ1oZrKLfx7zgJR_0 2024-03-09T15:36:06.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10100842010504727644', time=2024-03-09T17:24:17.236068+0000, chain.objs.size()=1 2024-03-09T15:36:06.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.BVKlPqoByJwLoyGIynZcb1wqODT8Y-t_0 2024-03-09T15:36:06.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4181747135476059994', time=2024-03-09T17:24:18.589302+0000, chain.objs.size()=1 2024-03-09T15:36:06.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.cH5tu1HEm5F0siZMsNX8Vy9QLV42TzL_0 2024-03-09T15:36:06.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4989013408037618683', time=2024-03-09T17:24:22.151079+0000, chain.objs.size()=1 2024-03-09T15:36:06.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Wcpkh9hdy36eg7iqgxUev_L83XGgsnr_0 2024-03-09T15:36:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2986424951885593569', time=2024-03-09T17:24:25.521157+0000, chain.objs.size()=1 2024-03-09T15:36:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.QE1eOhb5ugBvxgvvK_iK0qgqLrbwm7__0 2024-03-09T15:36:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4798405629391078556', time=2024-03-09T17:24:26.963895+0000, chain.objs.size()=1 2024-03-09T15:36:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.WQ4AZ4eW9VG3M521X1zxkVue_ktmuvQ_0 2024-03-09T15:36:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8809431692974268296', time=2024-03-09T17:24:27.186807+0000, chain.objs.size()=1 2024-03-09T15:36:06.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7KGSMGj3ivSx9Z2fTUqkiE08Q5Tpwtl_0 2024-03-09T15:36:06.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5245553747743698703', time=2024-03-09T17:24:27.257504+0000, chain.objs.size()=1 2024-03-09T15:36:06.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.vEVmt9IWgnIr22K5h-AvC4oaC7BL18B_0 2024-03-09T15:36:06.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15681768889658548365', time=2024-03-09T17:24:29.211201+0000, chain.objs.size()=1 2024-03-09T15:36:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.268+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.MvyRpNeknsmFuLKafX5R78vymHqFA08_0 2024-03-09T15:36:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17140566125713326660', time=2024-03-09T17:24:30.167982+0000, chain.objs.size()=1 2024-03-09T15:36:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.f5GirYkk248VCmXpuXFNkjDpypAsyfn_0 2024-03-09T15:36:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1870935270852635091', time=2024-03-09T17:24:37.066726+0000, chain.objs.size()=1 2024-03-09T15:36:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.257__shadow_.J2fHZXrlU3D-TuHBrJXQVQUCBCI86Ij_0 2024-03-09T15:36:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9311935582612190870', time=2024-03-09T17:24:53.713960+0000, chain.objs.size()=1 2024-03-09T15:36:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.263__shadow_.xljngsG_JPWJfGeRkkj5O7r0nywwr2T_0 2024-03-09T15:36:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11740384242824973816', time=2024-03-09T17:24:53.716050+0000, chain.objs.size()=1 2024-03-09T15:36:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.263__shadow_.uc0dtKJkH2rs_IbcsprfUPoUSu65cob_0 2024-03-09T15:36:06.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15563334576825909294', time=2024-03-09T17:25:19.297850+0000, chain.objs.size()=1 2024-03-09T15:36:06.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.272__shadow_.j5Y0MEOu_S8MZGB8AmR_uZcXRX4GDDj_0 2024-03-09T15:36:06.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7344027085507791682', time=2024-03-09T17:25:28.014938+0000, chain.objs.size()=1 2024-03-09T15:36:06.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.275__shadow_.twUaKtf9cAYbLjJRt0OfweRgV3RqaLl_0 2024-03-09T15:36:06.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12801299381024974891', time=2024-03-09T17:30:44.239237+0000, chain.objs.size()=1 2024-03-09T15:36:06.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.311__shadow_.UPuwq3euolqiy6nUCZrpvYzFCcOxfVK_0 2024-03-09T15:36:06.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.272+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.276+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-03-09T15:36:06.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.276+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=39, truncated=0, next_marker='' 2024-03-09T15:36:06.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6953490991845555743', time=2024-03-09T17:21:18.699904+0000, chain.objs.size()=1 2024-03-09T15:36:06.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.236__shadow_.jvIltLZ8kLMvIe1dYAbmZhAbHezy9ge_0 2024-03-09T15:36:06.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8736172253361221540', time=2024-03-09T17:23:58.785709+0000, chain.objs.size()=1 2024-03-09T15:36:06.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.XEv8ndsYEGtFERZWm5DrCPX2LU8MPAh_0 2024-03-09T15:36:06.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16984482290206485203', time=2024-03-09T17:23:59.950030+0000, chain.objs.size()=1 2024-03-09T15:36:06.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.mvGAKnnEPcvmrzCUXB7ZVh2wkaUbQEu_0 2024-03-09T15:36:06.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10619334588628825480', time=2024-03-09T17:24:00.630175+0000, chain.objs.size()=1 2024-03-09T15:36:06.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3kIZCIHAKQhibXc_qe8KIrLxLk_0SxH_0 2024-03-09T15:36:06.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3550549616813246491', time=2024-03-09T17:24:02.172882+0000, chain.objs.size()=1 2024-03-09T15:36:06.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fxau2CHO9a9CNTD477APYkIiZ272cs8_0 2024-03-09T15:36:06.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9525082005584476365', time=2024-03-09T17:24:03.460935+0000, chain.objs.size()=1 2024-03-09T15:36:06.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8_B3u18HiOAOutuLwmwIMvtvRwJD7x5_0 2024-03-09T15:36:06.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10261494326805543447', time=2024-03-09T17:24:03.543979+0000, chain.objs.size()=1 2024-03-09T15:36:06.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.XTgfCrVbVh0kWsvA-7yWGOLiWMec66e_0 2024-03-09T15:36:06.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10163499036368131737', time=2024-03-09T17:24:03.902252+0000, chain.objs.size()=1 2024-03-09T15:36:06.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Ib8CzqDryCokrAtDrb-3-K9gxDQ48gN_0 2024-03-09T15:36:06.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10731990263359648874', time=2024-03-09T17:24:04.475276+0000, chain.objs.size()=1 2024-03-09T15:36:06.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.G0_yVe9kjLVO2jatshic8TV-PPqkSr8_0 2024-03-09T15:36:06.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1704738127709341245', time=2024-03-09T17:24:06.365727+0000, chain.objs.size()=1 2024-03-09T15:36:06.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HjQgRBsXN45Gv_JEHw0ClPBE-zYxeek_0 2024-03-09T15:36:06.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.328676134067034550', time=2024-03-09T17:24:06.670179+0000, chain.objs.size()=1 2024-03-09T15:36:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1raU8xcA22fxrLvmBYUhH_YLFaKbPuL_0 2024-03-09T15:36:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3872563558077035344', time=2024-03-09T17:24:06.773981+0000, chain.objs.size()=1 2024-03-09T15:36:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.uF_n3MwbOk0X3Zc7YbCRtyHyvTYozxX_0 2024-03-09T15:36:06.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5665453132649211057', time=2024-03-09T17:24:10.632306+0000, chain.objs.size()=1 2024-03-09T15:36:06.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.N-tLd97WP06tTaBHmUzm-RfR9ELPWyv_0 2024-03-09T15:36:06.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12917237233170852020', time=2024-03-09T17:24:12.481311+0000, chain.objs.size()=1 2024-03-09T15:36:06.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.10PwhgJ80gZiFh0iXEnQGMh7Dewvj3J_0 2024-03-09T15:36:06.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15726505264760608998', time=2024-03-09T17:24:12.651760+0000, chain.objs.size()=1 2024-03-09T15:36:06.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.vssFs_ytvvxW3w8rdCqkSmtAeVMXV34_0 2024-03-09T15:36:06.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10022546675886592021', time=2024-03-09T17:24:13.109028+0000, chain.objs.size()=1 2024-03-09T15:36:06.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HHplX8o0oATYuypKIhqIYWj8MT3uvj4_0 2024-03-09T15:36:06.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13403759802396636767', time=2024-03-09T17:24:13.576543+0000, chain.objs.size()=1 2024-03-09T15:36:06.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.hwKopZ2YAlscCQixwqvdQK7P2nhVwEG_0 2024-03-09T15:36:06.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1546671558721638571', time=2024-03-09T17:24:16.447425+0000, chain.objs.size()=1 2024-03-09T15:36:06.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Wdo-WdnQkXUFWCP0t-XIK69Eue0L3W1_0 2024-03-09T15:36:06.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14863810336895890323', time=2024-03-09T17:24:18.524253+0000, chain.objs.size()=1 2024-03-09T15:36:06.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.IADjkZMXRbACFChA-Run9nApWfqXCFu_0 2024-03-09T15:36:06.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15214077934443801931', time=2024-03-09T17:24:19.595708+0000, chain.objs.size()=1 2024-03-09T15:36:06.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.RuFj-PKlBOQPshG3teI12HpUTFuH7_f_0 2024-03-09T15:36:06.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13850942441819038890', time=2024-03-09T17:24:20.479298+0000, chain.objs.size()=1 2024-03-09T15:36:06.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.hmDvlZIh5W0qCvQ04DHx0utQzOKG3qP_0 2024-03-09T15:36:06.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11886078941442161852', time=2024-03-09T17:24:22.563986+0000, chain.objs.size()=1 2024-03-09T15:36:06.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.OCF1vXqoqRvDqH3ipaqHiBgU_lVz-dk_0 2024-03-09T15:36:06.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13627538248854810889', time=2024-03-09T17:24:22.566198+0000, chain.objs.size()=1 2024-03-09T15:36:06.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.280+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8z0CYC3-QTmsyTK436WJhuo_8YkPeUu_0 2024-03-09T15:36:06.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17913398843429253448', time=2024-03-09T17:24:23.708993+0000, chain.objs.size()=1 2024-03-09T15:36:06.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.aDl0-Tcscbzby9-BJA0e-9hCYLFCKfH_0 2024-03-09T15:36:06.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14400104064476936573', time=2024-03-09T17:24:23.759924+0000, chain.objs.size()=1 2024-03-09T15:36:06.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.TvKejPmN6S_4YoZ1nb1hJrDZo2LCRqD_0 2024-03-09T15:36:06.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17911790329082347911', time=2024-03-09T17:24:24.584870+0000, chain.objs.size()=1 2024-03-09T15:36:06.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.svwY_hyXl2i_d-nl4rHpPSecu_8cAnn_0 2024-03-09T15:36:06.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15119416511757347666', time=2024-03-09T17:24:25.116938+0000, chain.objs.size()=1 2024-03-09T15:36:06.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7TDX9bQtWSxUjXFosZzWzhu1Hcl2RyI_0 2024-03-09T15:36:06.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2450466226438576970', time=2024-03-09T17:24:27.166453+0000, chain.objs.size()=1 2024-03-09T15:36:06.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.j6b3u3c2I6NJzzGeZ9ABKFaYfuz8GGY_0 2024-03-09T15:36:06.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7432459120484831029', time=2024-03-09T17:24:28.866616+0000, chain.objs.size()=1 2024-03-09T15:36:06.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9Z6QOhZRLbpoKLl65Of3aZ1B5KhMFwb_0 2024-03-09T15:36:06.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13252019254343749911', time=2024-03-09T17:24:30.379793+0000, chain.objs.size()=1 2024-03-09T15:36:06.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.riR1VU6ylBe5kUiaapI4ZTfLYDLHd4p_0 2024-03-09T15:36:06.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11886941272396614873', time=2024-03-09T17:24:30.771940+0000, chain.objs.size()=1 2024-03-09T15:36:06.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.17TmbYemT8GjNcuep67yc2SOy9bwHKK_0 2024-03-09T15:36:06.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6837305062594848507', time=2024-03-09T17:24:30.774273+0000, chain.objs.size()=1 2024-03-09T15:36:06.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.p90lGwNeg1x31IJFMoBvFLp2EoubLkZ_0 2024-03-09T15:36:06.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4911105701546267272', time=2024-03-09T17:25:13.457401+0000, chain.objs.size()=1 2024-03-09T15:36:06.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.270__shadow_.T0TnKR8uN37ZwHsd4kkFXhawF4LYan3_0 2024-03-09T15:36:06.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2737527555554708852', time=2024-03-09T17:25:36.644758+0000, chain.objs.size()=1 2024-03-09T15:36:06.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.278__shadow_.uKKgrMCyaMUrrpi9cOSuENoPeniAnzs_0 2024-03-09T15:36:06.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14023122897037339743', time=2024-03-09T17:25:42.675675+0000, chain.objs.size()=1 2024-03-09T15:36:06.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.280__shadow_.QkB0xrl9cba1pJys0JbqTwWn46ZORC9_0 2024-03-09T15:36:06.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10761839899030533005', time=2024-03-09T17:26:01.732560+0000, chain.objs.size()=1 2024-03-09T15:36:06.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.nHtOAcjI_gsRUcwkNWZ2A7uH-AttK99_0 2024-03-09T15:36:06.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17441489628236942879', time=2024-03-09T17:26:19.736153+0000, chain.objs.size()=1 2024-03-09T15:36:06.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.290__shadow_.SWfQ_OTgtDqKcDrCEtsKNy3ZPjxX0--_0 2024-03-09T15:36:06.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2452919872408807451', time=2024-03-09T17:26:43.467848+0000, chain.objs.size()=1 2024-03-09T15:36:06.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.284+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.298__shadow_.TPOV3WuSZf0grwk5_FLwWiqC-8plGoL_0 2024-03-09T15:36:06.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.288+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4507395289609161966', time=2024-03-09T17:33:05.222026+0000, chain.objs.size()=1 2024-03-09T15:36:06.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.288+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.326__shadow_.UO9flwajHiBN9kF5FqZztV8Gox4YzQI_0 2024-03-09T15:36:06.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.288+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-03-09T15:36:06.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=50, truncated=0, next_marker='' 2024-03-09T15:36:06.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15784875720804505867', time=2024-03-09T17:12:14.817743+0000, chain.objs.size()=1 2024-03-09T15:36:06.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.151__shadow_.jjNEi51dhzuYBhxCYuMfsSl0dai9_a4_0 2024-03-09T15:36:06.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12249815863697155536', time=2024-03-09T17:21:49.168161+0000, chain.objs.size()=1 2024-03-09T15:36:06.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.246__shadow_.M7TRD0zy1mrmC8C84zW_fqCCckN3yTu_0 2024-03-09T15:36:06.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13896322287514385502', time=2024-03-09T17:22:12.409065+0000, chain.objs.size()=1 2024-03-09T15:36:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.254__shadow_.ivcriy6Vnst0YhlT5cWWC9CgaBbq3XP_0 2024-03-09T15:36:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3553306504879960549', time=2024-03-09T17:23:58.905996+0000, chain.objs.size()=1 2024-03-09T15:36:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9EJ9PpCB7W3ZZf8cjnXwIy8I0CfdMTD_0 2024-03-09T15:36:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4762366929626768682', time=2024-03-09T17:23:59.310613+0000, chain.objs.size()=1 2024-03-09T15:36:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.eak9eOvPYtfNuvrgTuxsU1PasHNG7CI_0 2024-03-09T15:36:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3868776754563000698', time=2024-03-09T17:23:59.752963+0000, chain.objs.size()=1 2024-03-09T15:36:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.QEL76PviFwyKCsiBeSkS9b9OvheV0Kf_0 2024-03-09T15:36:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5960753796081376296', time=2024-03-09T17:24:01.101606+0000, chain.objs.size()=1 2024-03-09T15:36:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Ox8OF89JsrFNpgcBxeDnoJGcsWu2BFk_0 2024-03-09T15:36:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2149021353754947799', time=2024-03-09T17:24:01.103368+0000, chain.objs.size()=1 2024-03-09T15:36:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.70G4VbTGq4j67j3oHWcKCPtcm0-1zzE_0 2024-03-09T15:36:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5363342962177708421', time=2024-03-09T17:24:03.493647+0000, chain.objs.size()=1 2024-03-09T15:36:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.JTEZUp8oNx5-KyDtFqRXCleGjlknqXN_0 2024-03-09T15:36:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6740262147805884108', time=2024-03-09T17:24:04.764185+0000, chain.objs.size()=1 2024-03-09T15:36:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.xodPNfh9Qvyp5fwsZCetLH6QpBWX3Bb_0 2024-03-09T15:36:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11876103752444844611', time=2024-03-09T17:24:05.986231+0000, chain.objs.size()=1 2024-03-09T15:36:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.nY_EBwh0B2DWx7xKsaEORJsEDCF86hp_0 2024-03-09T15:36:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8209684423675094680', time=2024-03-09T17:24:06.714030+0000, chain.objs.size()=1 2024-03-09T15:36:06.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.292+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2fn5psNgNRTQaSK1GwGtHup9_YV2iNt_0 2024-03-09T15:36:06.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2625730717795118254', time=2024-03-09T17:24:07.937185+0000, chain.objs.size()=1 2024-03-09T15:36:06.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.nIFLH-e6hweYRqQdNILl63S_G0iGlCt_0 2024-03-09T15:36:06.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11525402806485487615', time=2024-03-09T17:24:08.461470+0000, chain.objs.size()=1 2024-03-09T15:36:06.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.yAzcGPsN4Q-Mk06mQXkilw9AHJ2Ahhx_0 2024-03-09T15:36:06.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12609444319082761969', time=2024-03-09T17:24:08.463842+0000, chain.objs.size()=1 2024-03-09T15:36:06.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.TnC5UK6ifKKRZk5CGQ0PWwHYCO7Xt2Y_0 2024-03-09T15:36:06.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7000912212816633643', time=2024-03-09T17:24:09.304967+0000, chain.objs.size()=1 2024-03-09T15:36:06.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.jq0ZHx6kPLhIlwTdl5Q2gVmwCokt66b_0 2024-03-09T15:36:06.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12398081152569259427', time=2024-03-09T17:24:10.907275+0000, chain.objs.size()=1 2024-03-09T15:36:06.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.aDFaQWGq9e0z0ghFUO_qrqV5V0Z8T-s_0 2024-03-09T15:36:06.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3570604324674871813', time=2024-03-09T17:24:12.155879+0000, chain.objs.size()=1 2024-03-09T15:36:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.aujbpmAwPC7LE88XTv3Mho0fAb0N-0U_0 2024-03-09T15:36:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.641873656103024462', time=2024-03-09T17:24:14.967830+0000, chain.objs.size()=1 2024-03-09T15:36:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.mzbzak9Lh0ux6B5F-QW4BCdY71R2gvR_0 2024-03-09T15:36:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16184326842620358407', time=2024-03-09T17:24:17.408490+0000, chain.objs.size()=1 2024-03-09T15:36:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.pHq1KKnnbeI0DMooRJnD9vQm9m_Qpq__0 2024-03-09T15:36:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9983582337469954543', time=2024-03-09T17:24:17.768858+0000, chain.objs.size()=1 2024-03-09T15:36:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.kRXzEWNqZWxYpLdod1xegV-qfarXhIK_0 2024-03-09T15:36:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8967246016558744784', time=2024-03-09T17:24:18.138013+0000, chain.objs.size()=1 2024-03-09T15:36:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iawR96nVAn7zes7e-1hMDybdRArGPDa_0 2024-03-09T15:36:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16647690775549581262', time=2024-03-09T17:24:18.552743+0000, chain.objs.size()=1 2024-03-09T15:36:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.296+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.jMdsN9MMfNXMbgFwLwT29-FnoOr553d_0 2024-03-09T15:36:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13896976272269872563', time=2024-03-09T17:24:19.422487+0000, chain.objs.size()=1 2024-03-09T15:36:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.k2kclSIBdTMM27sNyPuKpKGN72XAWMH_0 2024-03-09T15:36:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5924621050829711387', time=2024-03-09T17:24:19.530107+0000, chain.objs.size()=1 2024-03-09T15:36:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LY2i5lSQoKgdg1me7K3tDoLbOvUcbfO_0 2024-03-09T15:36:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2359491237389985037', time=2024-03-09T17:24:21.236411+0000, chain.objs.size()=1 2024-03-09T15:36:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3gn3GN7ba2zWjqOIpGRULpJdJl-scQW_0 2024-03-09T15:36:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14809226922738561577', time=2024-03-09T17:24:22.289456+0000, chain.objs.size()=1 2024-03-09T15:36:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.JULMfnLlaWAYmkdyFX_A7Y2CV6PHD0U_0 2024-03-09T15:36:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.976463587655385603', time=2024-03-09T17:24:22.565920+0000, chain.objs.size()=1 2024-03-09T15:36:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8NAgoodyrG8QwMUnmCd_BUjQrw0ApIU_0 2024-03-09T15:36:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10920645313401640294', time=2024-03-09T17:24:25.129876+0000, chain.objs.size()=1 2024-03-09T15:36:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.CJWYhws0Y77ptVDir_5QhzRQJ-Blj0q_0 2024-03-09T15:36:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14774449875195455521', time=2024-03-09T17:24:25.556863+0000, chain.objs.size()=1 2024-03-09T15:36:06.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.5bhbaxK2tC-nIf0LN_H7M2D7swP-Bag_0 2024-03-09T15:36:06.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2537644984446162393', time=2024-03-09T17:24:26.397624+0000, chain.objs.size()=1 2024-03-09T15:36:06.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.juiHUDsMy9c2rXCe3JKkwkSXN8l1kWS_0 2024-03-09T15:36:06.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3969258867705141286', time=2024-03-09T17:24:26.537514+0000, chain.objs.size()=1 2024-03-09T15:36:06.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.wfDSbT-aoIRHila_e1MtlUCAYJ4x8Ur_0 2024-03-09T15:36:06.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11610349903497185520', time=2024-03-09T17:24:26.696811+0000, chain.objs.size()=1 2024-03-09T15:36:06.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.b53K5Tdna_nte67y32eEqHdvZyKXReq_0 2024-03-09T15:36:06.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14936200227805461525', time=2024-03-09T17:24:28.699872+0000, chain.objs.size()=1 2024-03-09T15:36:06.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ytyBJ7e_-NZmRW41MEggLSY3MyZqjXN_0 2024-03-09T15:36:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10329828987381940462', time=2024-03-09T17:24:29.151260+0000, chain.objs.size()=1 2024-03-09T15:36:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LHUFzgeNxTADWkZtGcdxJknF1Cb8Tku_0 2024-03-09T15:36:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3869147844788796357', time=2024-03-09T17:24:29.192215+0000, chain.objs.size()=1 2024-03-09T15:36:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-jr0lcRXOPhJlrJDmTEk5MxtdvZvvr3_0 2024-03-09T15:36:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14929054765795240002', time=2024-03-09T17:24:29.938519+0000, chain.objs.size()=1 2024-03-09T15:36:06.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7IMxlmXZNCx2BoPHZka8FyE4EWyDrZ9_0 2024-03-09T15:36:06.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15065379114622730953', time=2024-03-09T17:24:30.004739+0000, chain.objs.size()=1 2024-03-09T15:36:06.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Vjf50myIa5LdTj05i7-_5aj2gQm19_T_0 2024-03-09T15:36:06.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7777455622335523045', time=2024-03-09T17:24:30.739286+0000, chain.objs.size()=1 2024-03-09T15:36:06.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.aqKLRBcZnFgjRfbkTB96b369VNgjf5a_0 2024-03-09T15:36:06.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9889187539849759752', time=2024-03-09T17:24:48.166153+0000, chain.objs.size()=1 2024-03-09T15:36:06.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.261__shadow_.bdU8OJpuF4j1yon9x116J8OfXp_GIQD_0 2024-03-09T15:36:06.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10409710401409749131', time=2024-03-09T17:25:07.565082+0000, chain.objs.size()=1 2024-03-09T15:36:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.268__shadow_.MhDEXoE3fiQCI0iadf-qRddRKU8Ecs6_0 2024-03-09T15:36:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5212330008753352913', time=2024-03-09T17:25:22.103358+0000, chain.objs.size()=1 2024-03-09T15:36:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.273__shadow_.BskciH0AZ-sontQRnxOHzIfmeyNg2XR_0 2024-03-09T15:36:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11306230415636697174', time=2024-03-09T17:25:33.700909+0000, chain.objs.size()=1 2024-03-09T15:36:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.300+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.277__shadow_.6lBBohKRgkM3_4Aoy2dYKgsbxQYs9sw_0 2024-03-09T15:36:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13356807750144560295', time=2024-03-09T17:25:39.747993+0000, chain.objs.size()=1 2024-03-09T15:36:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.279__shadow_.yKcEW2d3piL8lW1EHtPMAsLkO6lr-0s_0 2024-03-09T15:36:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.667674759494664119', time=2024-03-09T17:25:48.341868+0000, chain.objs.size()=1 2024-03-09T15:36:06.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.282__shadow_.ceV_6L3gXWF8s3q0AVhx_tUFXsxmsaY_0 2024-03-09T15:36:06.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9529041680737477058', time=2024-03-09T17:26:01.486375+0000, chain.objs.size()=1 2024-03-09T15:36:06.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.iRMgl3wbiOr79chXCooFtYHDompnnu1_0 2024-03-09T15:36:06.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2662755347902470370', time=2024-03-09T17:26:01.488215+0000, chain.objs.size()=1 2024-03-09T15:36:06.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.7FvjhFbGXNR_fU7m2wDBnylK21pNc4m_0 2024-03-09T15:36:06.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9042449341369457987', time=2024-03-09T17:26:07.940956+0000, chain.objs.size()=1 2024-03-09T15:36:06.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.izZf06okWBHvQ6_YjgJCSZV5UHnY617_0 2024-03-09T15:36:06.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5264984557331166688', time=2024-03-09T17:26:19.758348+0000, chain.objs.size()=1 2024-03-09T15:36:06.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.290__shadow_.G-diWHTCT8ZIRuIuN5fEmqHzpy3hyez_0 2024-03-09T15:36:06.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3693234479727846152', time=2024-03-09T17:26:37.636332+0000, chain.objs.size()=1 2024-03-09T15:36:06.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.304+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.296__shadow_.cnTLkDr-sdAJzEuId_g-V3VW2Xib_jx_0 2024-03-09T15:36:06.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.308+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.308+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-03-09T15:36:06.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=58, truncated=0, next_marker='' 2024-03-09T15:36:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11023496849693745156', time=2024-03-09T17:13:23.198993+0000, chain.objs.size()=3 2024-03-09T15:36:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__multipart_foo-LUKEWARM.2~VXYg8tfmHbwM5Rw7w1ESyLG-wjANPht.1 2024-03-09T15:36:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__multipart_foo-LUKEWARM.2~VXYg8tfmHbwM5Rw7w1ESyLG-wjANPht.2 2024-03-09T15:36:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__multipart_foo-LUKEWARM.2~VXYg8tfmHbwM5Rw7w1ESyLG-wjANPht.3 2024-03-09T15:36:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.403910838154383795', time=2024-03-09T17:20:40.204221+0000, chain.objs.size()=1 2024-03-09T15:36:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.224__shadow_.40zGn4AENutOdev3Jg4DcSDXl0rJLr__0 2024-03-09T15:36:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7814176462733776069', time=2024-03-09T17:20:59.178137+0000, chain.objs.size()=1 2024-03-09T15:36:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.230__shadow_.V6CDvTNVoKk_tm77WRMS4Pad0IX67E-_0 2024-03-09T15:36:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4415674958988295093', time=2024-03-09T17:21:03.581335+0000, chain.objs.size()=1 2024-03-09T15:36:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.231__shadow_.oesT8ZtMowmjWCriJJYcxip8onCoykB_0 2024-03-09T15:36:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9600512203080563559', time=2024-03-09T17:21:26.113804+0000, chain.objs.size()=1 2024-03-09T15:36:06.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.238__shadow_.eBjTztlP0MjfQq9qKb26xzW-ZcJaUhh_0 2024-03-09T15:36:06.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5539373082543287330', time=2024-03-09T17:21:34.846116+0000, chain.objs.size()=1 2024-03-09T15:36:06.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.241__shadow_.UKkc-49LcA7IKc9EusPEElTBJFyPQX1_0 2024-03-09T15:36:06.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12411571615123300382', time=2024-03-09T17:21:37.773709+0000, chain.objs.size()=1 2024-03-09T15:36:06.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.242__shadow_.wUVIZ3GumbISAcnIOATXBxj7vHhVsXF_0 2024-03-09T15:36:06.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13039115831785240437', time=2024-03-09T17:22:03.266863+0000, chain.objs.size()=1 2024-03-09T15:36:06.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.251__shadow_.5VWDfX0gjc5i8CU7J0RxPCNNPgw8Jn8_0 2024-03-09T15:36:06.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18271031283043680053', time=2024-03-09T17:22:06.010445+0000, chain.objs.size()=1 2024-03-09T15:36:06.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.252__shadow_.DV3IaYqkmxHbgtWasRudDL03siC-RER_0 2024-03-09T15:36:06.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11160938165046936309', time=2024-03-09T17:23:58.789268+0000, chain.objs.size()=1 2024-03-09T15:36:06.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.312+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Kt36TmyLVn3VtKxAg1fdiYdmUr6LlNC_0 2024-03-09T15:36:06.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2908568514820073886', time=2024-03-09T17:24:01.054557+0000, chain.objs.size()=1 2024-03-09T15:36:06.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.D3ez4V-drqnfNI4Fuz1LnyBlzJQZ_mB_0 2024-03-09T15:36:06.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13831560782074421909', time=2024-03-09T17:24:01.473120+0000, chain.objs.size()=1 2024-03-09T15:36:06.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.A7T1Rv6lOnD-0bo2G9vDoZiGav6clsz_0 2024-03-09T15:36:06.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6587129516944263220', time=2024-03-09T17:24:02.171428+0000, chain.objs.size()=1 2024-03-09T15:36:06.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7_3fIpzp4fjHtTs_OfZNXXVRdtx_qXZ_0 2024-03-09T15:36:06.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9125862272647433668', time=2024-03-09T17:24:03.499002+0000, chain.objs.size()=1 2024-03-09T15:36:06.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.L5_94uaPuWQdItIuXImQqgTzgSEd_tC_0 2024-03-09T15:36:06.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14330240971993026844', time=2024-03-09T17:24:03.501264+0000, chain.objs.size()=1 2024-03-09T15:36:06.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.dKybCXhzlhhYGMMZdr0KT7BZA1c0XN7_0 2024-03-09T15:36:06.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13071640512133074343', time=2024-03-09T17:24:04.311430+0000, chain.objs.size()=1 2024-03-09T15:36:06.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.p8Xe-FgKW1xfNv6XLkl6GI3FvXPdxF5_0 2024-03-09T15:36:06.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3663673536837876523', time=2024-03-09T17:24:05.321222+0000, chain.objs.size()=1 2024-03-09T15:36:06.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.T4Jse95Hgx6XWtmUdkQfqy_km41wZsx_0 2024-03-09T15:36:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3002992922176514641', time=2024-03-09T17:24:06.120079+0000, chain.objs.size()=1 2024-03-09T15:36:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HJHVQzroF9M9SYlUpK4XQaTlJ7UmN5V_0 2024-03-09T15:36:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14965091378189109560', time=2024-03-09T17:24:06.779507+0000, chain.objs.size()=1 2024-03-09T15:36:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.hikch6eqGuADOXZKG2dnf_Ag_DATsOe_0 2024-03-09T15:36:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2240000822489501496', time=2024-03-09T17:24:06.796446+0000, chain.objs.size()=1 2024-03-09T15:36:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0WD_nQ7eYvmOgE8o0xKpzEutojEAQDa_0 2024-03-09T15:36:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2670358498094254777', time=2024-03-09T17:24:08.495690+0000, chain.objs.size()=1 2024-03-09T15:36:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.aMFxwYlK2H_WvLj3-hW_gBXk4qJeEAH_0 2024-03-09T15:36:06.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7142956549361085194', time=2024-03-09T17:24:08.892663+0000, chain.objs.size()=1 2024-03-09T15:36:06.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.kdFjBERYPDuz41HH_7VNdgBLz1832Uu_0 2024-03-09T15:36:06.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9738842976613930029', time=2024-03-09T17:24:08.918249+0000, chain.objs.size()=1 2024-03-09T15:36:06.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Ea3Elo8oG4UyeBSvEgFRa71Nyt7eOMB_0 2024-03-09T15:36:06.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9862130077825516116', time=2024-03-09T17:24:09.576428+0000, chain.objs.size()=1 2024-03-09T15:36:06.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iYtBnVOMA1hk6MzLmN4InFj_sOnAGTa_0 2024-03-09T15:36:06.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8119522806865628274', time=2024-03-09T17:24:09.723446+0000, chain.objs.size()=1 2024-03-09T15:36:06.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.316+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9TDD6uN2e7XjeuhDGzHbBdnN1DR8u2o_0 2024-03-09T15:36:06.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4184031352250533781', time=2024-03-09T17:24:10.227502+0000, chain.objs.size()=1 2024-03-09T15:36:06.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.eXPifqZG25G65ubM9wfPkGKnSrW5tIU_0 2024-03-09T15:36:06.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15808522359399058803', time=2024-03-09T17:24:10.272986+0000, chain.objs.size()=1 2024-03-09T15:36:06.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.30dUa3-2uk1TOSZHwj9hAwYBdFuio8h_0 2024-03-09T15:36:06.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6529302161617446404', time=2024-03-09T17:24:10.694706+0000, chain.objs.size()=1 2024-03-09T15:36:06.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6_OU9vP47yKybtsx18-kM5bzOOvAKGg_0 2024-03-09T15:36:06.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15586225724761046412', time=2024-03-09T17:24:11.002075+0000, chain.objs.size()=1 2024-03-09T15:36:06.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0Xnjvbpo-pMM1EJHzaLmKcjTaifP0XD_0 2024-03-09T15:36:06.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15594087401794882030', time=2024-03-09T17:24:11.120210+0000, chain.objs.size()=1 2024-03-09T15:36:06.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.aaf-RdeK5ctcUkuyyIBjYH4M_kC7W4W_0 2024-03-09T15:36:06.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8256331850840395843', time=2024-03-09T17:24:12.111958+0000, chain.objs.size()=1 2024-03-09T15:36:06.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.GartSQcpEjEsGi2vbDVAUDA0YQJzZR5_0 2024-03-09T15:36:06.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1583941551493265008', time=2024-03-09T17:24:13.034535+0000, chain.objs.size()=1 2024-03-09T15:36:06.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.NbqGyO-bOzjOiWd_RLVJ0iaywaFbogL_0 2024-03-09T15:36:06.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17129040866593949083', time=2024-03-09T17:24:13.163206+0000, chain.objs.size()=1 2024-03-09T15:36:06.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.pFnG9DcxwpivX40_BqWYVUcvscjvbYI_0 2024-03-09T15:36:06.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3110535452015937046', time=2024-03-09T17:24:13.165379+0000, chain.objs.size()=1 2024-03-09T15:36:06.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.jN6abEKrTOR98SgdRy-mIx_3EZDJmZf_0 2024-03-09T15:36:06.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15464031220239986271', time=2024-03-09T17:24:15.165097+0000, chain.objs.size()=1 2024-03-09T15:36:06.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iznkJjhCQwwdj0mI7XA10ZH4Yb4TAZV_0 2024-03-09T15:36:06.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2443290699422013447', time=2024-03-09T17:24:15.277607+0000, chain.objs.size()=1 2024-03-09T15:36:06.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2gnq84FdcCWHhaXrVoXFDjP_rZUiCE1_0 2024-03-09T15:36:06.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.658149793967351940', time=2024-03-09T17:24:17.304126+0000, chain.objs.size()=1 2024-03-09T15:36:06.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.v2Yk9ZEtmCjaWs7oO63RTgxvHkjfEL5_0 2024-03-09T15:36:06.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15357228268081622954', time=2024-03-09T17:24:17.587167+0000, chain.objs.size()=1 2024-03-09T15:36:06.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qel2JX9EB76TFC8VD879Zxh24wFjqsC_0 2024-03-09T15:36:06.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1413496669633688696', time=2024-03-09T17:24:18.386864+0000, chain.objs.size()=1 2024-03-09T15:36:06.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.cU3jn3RxyxPL9AVd1vcOmtXxbhpejPN_0 2024-03-09T15:36:06.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12598519670742612591', time=2024-03-09T17:24:19.467454+0000, chain.objs.size()=1 2024-03-09T15:36:06.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.kM7HgtUMHH4tBpzeXigRMSkhE9TfvUj_0 2024-03-09T15:36:06.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1988007291002824146', time=2024-03-09T17:24:22.138280+0000, chain.objs.size()=1 2024-03-09T15:36:06.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.q9rdx5okf21rwequIGiRpBTG5d5my5G_0 2024-03-09T15:36:06.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15367679721139421529', time=2024-03-09T17:24:22.841378+0000, chain.objs.size()=1 2024-03-09T15:36:06.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.WFxWPQbaUwp8euzXBQ5sJKjq2oIDPZH_0 2024-03-09T15:36:06.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5737665509023820020', time=2024-03-09T17:24:24.737968+0000, chain.objs.size()=1 2024-03-09T15:36:06.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.h70EMY_Df3Ggkk_aCdGXew981DZdtSM_0 2024-03-09T15:36:06.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10192792063544069639', time=2024-03-09T17:24:25.130387+0000, chain.objs.size()=1 2024-03-09T15:36:06.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.D1XQjaRPCbFV1Vqewf8yKsChMORs5uG_0 2024-03-09T15:36:06.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12842595316518597501', time=2024-03-09T17:24:27.751776+0000, chain.objs.size()=1 2024-03-09T15:36:06.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.320+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.rtklrS3XRbfOq0TqV-8x8WpmKCiEN5E_0 2024-03-09T15:36:06.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1483038244792243012', time=2024-03-09T17:24:29.586965+0000, chain.objs.size()=1 2024-03-09T15:36:06.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6BzCk3mYMQFjjjhebsUGJQ2B1GQ2KM__0 2024-03-09T15:36:06.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4626333077480555947', time=2024-03-09T17:24:29.821298+0000, chain.objs.size()=1 2024-03-09T15:36:06.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1TEegvb0k-g0vWslKCcBLCF6pHbyoiF_0 2024-03-09T15:36:06.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16560241752462468530', time=2024-03-09T17:24:39.904269+0000, chain.objs.size()=1 2024-03-09T15:36:06.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.258__shadow_.erfq1LAm2Usf1EWWZ5efDl1lpehowNP_0 2024-03-09T15:36:06.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1791065008794678851', time=2024-03-09T17:25:25.131299+0000, chain.objs.size()=1 2024-03-09T15:36:06.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.274__shadow_.hhthGt1Vr1EjDwQpHRXF71XQrw_fRPk_0 2024-03-09T15:36:06.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5625777935075072460', time=2024-03-09T17:26:01.412166+0000, chain.objs.size()=1 2024-03-09T15:36:06.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.KXtGu1WHZvGXAaxim3jcY8Fdfss3g4t_0 2024-03-09T15:36:06.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1038855783762908930', time=2024-03-09T17:26:08.080599+0000, chain.objs.size()=1 2024-03-09T15:36:06.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.B_hswv10-uvoQ6ejPaOTyBMvXAd-qaf_0 2024-03-09T15:36:06.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17176678224895636427', time=2024-03-09T17:26:13.977141+0000, chain.objs.size()=1 2024-03-09T15:36:06.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.288__shadow_.Pcu6qURJYVmbHspivTUA247WF0QhTaV_0 2024-03-09T15:36:06.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3286447874324188384', time=2024-03-09T17:26:16.826611+0000, chain.objs.size()=1 2024-03-09T15:36:06.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.289__shadow_.95MAJbSskv6Age8n7TgBF7pYAyt0guD_0 2024-03-09T15:36:06.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8508578430895468801', time=2024-03-09T17:26:16.828326+0000, chain.objs.size()=1 2024-03-09T15:36:06.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.289__shadow_.-Vj94uBgKrl5qRNX57UFDpuMnaD5K3o_0 2024-03-09T15:36:06.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.430646701248254804', time=2024-03-09T17:26:37.621270+0000, chain.objs.size()=1 2024-03-09T15:36:06.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.296__shadow_.rFSkchImA-__-dc8GxrFnZF5oIjoQ8p_0 2024-03-09T15:36:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4632435876480767874', time=2024-03-09T17:26:43.492577+0000, chain.objs.size()=1 2024-03-09T15:36:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.298__shadow_.XF_6GwOX_Zojb4hu9oBwCn1OBKrWpxd_0 2024-03-09T15:36:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9951117767807300560', time=2024-03-09T17:26:50.961493+0000, chain.objs.size()=1 2024-03-09T15:36:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.300__shadow:5kmwLTyTtOX7HK88evcVYKVv7CFwhEe_.JVI0q1qbFrM1C_Hm7LVwHoW92S9sk9a_0 2024-03-09T15:36:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6417288462819593701', time=2024-03-09T17:33:52.434835+0000, chain.objs.size()=1 2024-03-09T15:36:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.324+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.332__shadow_.Um3b56g7WH6yUo1TaH1CxOhEpM8p8sL_0 2024-03-09T15:36:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.328+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.328+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-03-09T15:36:06.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=38, truncated=0, next_marker='' 2024-03-09T15:36:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9179179804007649380', time=2024-03-09T17:21:18.708806+0000, chain.objs.size()=1 2024-03-09T15:36:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.236__shadow_.Rfy1nh-ddpr7EqjAoHoNf8_I7-h8iFm_0 2024-03-09T15:36:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5466396189564858590', time=2024-03-09T17:21:18.726802+0000, chain.objs.size()=1 2024-03-09T15:36:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.236__shadow_.5mGwv40M-nBcv7QVHeXTwRnYeWQVvIb_0 2024-03-09T15:36:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15245164382198183641', time=2024-03-09T17:21:23.175064+0000, chain.objs.size()=1 2024-03-09T15:36:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.237__shadow_.yY2wE97O8Qsmz2D5jH0tDRsh01_o89P_0 2024-03-09T15:36:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11586140241823833244', time=2024-03-09T17:23:58.801957+0000, chain.objs.size()=1 2024-03-09T15:36:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.VXG0UQaLRrRKwO2fdIcKBW6K7fNFUXi_0 2024-03-09T15:36:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9247099730706561224', time=2024-03-09T17:23:59.218007+0000, chain.objs.size()=1 2024-03-09T15:36:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.lWUs9JsXohJkq7DJrE5MP1N5Xlhi8kO_0 2024-03-09T15:36:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5378957019209841028', time=2024-03-09T17:23:59.276225+0000, chain.objs.size()=1 2024-03-09T15:36:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.vEhfjqvXMqOGjRLbp3jJrkrJ8svSqZW_0 2024-03-09T15:36:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4101509614333099441', time=2024-03-09T17:24:00.382620+0000, chain.objs.size()=1 2024-03-09T15:36:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.S0_jiIvBTulncKHSqIdZQbZYJMHmrRy_0 2024-03-09T15:36:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6162544625355090738', time=2024-03-09T17:24:00.652962+0000, chain.objs.size()=1 2024-03-09T15:36:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.xlMHfERcZnyFjsq1n2f-a1-h4OAqxP5_0 2024-03-09T15:36:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15611029709332283697', time=2024-03-09T17:24:01.877276+0000, chain.objs.size()=1 2024-03-09T15:36:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.no4H3OzovXUUgJmuXuRYc2HqXSFNece_0 2024-03-09T15:36:06.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13310070690605335012', time=2024-03-09T17:24:04.481525+0000, chain.objs.size()=1 2024-03-09T15:36:06.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2gfwBz2E7ooY2RxsM8X_1kwULHC4_WB_0 2024-03-09T15:36:06.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17388348404516418014', time=2024-03-09T17:24:04.821637+0000, chain.objs.size()=1 2024-03-09T15:36:06.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3Upp_E_VqRskyzZFIHaJ9j9Lqgm4rvk_0 2024-03-09T15:36:06.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15402616053086127501', time=2024-03-09T17:24:04.832441+0000, chain.objs.size()=1 2024-03-09T15:36:06.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.wKIatn57wDh8xuDvviXD1h54S7l7ACB_0 2024-03-09T15:36:06.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4483109045922172063', time=2024-03-09T17:24:05.293567+0000, chain.objs.size()=1 2024-03-09T15:36:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1ex1tUkGRGZYu6Id_r5zMwgLRZhdER5_0 2024-03-09T15:36:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17771654306538059315', time=2024-03-09T17:24:07.892543+0000, chain.objs.size()=1 2024-03-09T15:36:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.332+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.lw3PGfzjHnDQHB3aTtemI35U9TNqIuN_0 2024-03-09T15:36:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3880629816741542813', time=2024-03-09T17:24:14.169209+0000, chain.objs.size()=1 2024-03-09T15:36:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sQRwFXYqJWVYHYgvElfgid9na3LgaqM_0 2024-03-09T15:36:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18258704297580133568', time=2024-03-09T17:24:14.723024+0000, chain.objs.size()=1 2024-03-09T15:36:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Zg5rnCjuzc3Xxf9GY6p1Clua0Oox-GE_0 2024-03-09T15:36:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13369430135032936001', time=2024-03-09T17:24:14.727637+0000, chain.objs.size()=1 2024-03-09T15:36:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Y5wU5MBnnfs8XFhxSavEULKQVd_vt64_0 2024-03-09T15:36:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10578801773863728197', time=2024-03-09T17:24:14.729284+0000, chain.objs.size()=1 2024-03-09T15:36:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0hr9bVSfiXikPSwPoP5c5NTyRSGQrQz_0 2024-03-09T15:36:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7838668840694265342', time=2024-03-09T17:24:15.393922+0000, chain.objs.size()=1 2024-03-09T15:36:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.n2YA5qN_ZVsKpHaGMrO36AyGRyC52-J_0 2024-03-09T15:36:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10509697988147539267', time=2024-03-09T17:24:16.258158+0000, chain.objs.size()=1 2024-03-09T15:36:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PwJylgMDgTqqDYSqpF1_OAfCpDq9oQW_0 2024-03-09T15:36:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1171907017627257806', time=2024-03-09T17:24:17.196890+0000, chain.objs.size()=1 2024-03-09T15:36:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2MfaEDTTYTdUw7IeMKHhW_galDCvtnf_0 2024-03-09T15:36:06.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4776704469439544999', time=2024-03-09T17:24:17.769921+0000, chain.objs.size()=1 2024-03-09T15:36:06.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.a9HavOuZgqC4Z2QEJNyWwXwCuPEVJ9H_0 2024-03-09T15:36:06.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2585408063866869716', time=2024-03-09T17:24:18.303336+0000, chain.objs.size()=1 2024-03-09T15:36:06.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LmHzZNrwKDN273WXyzwbPqvXwfQd39M_0 2024-03-09T15:36:06.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2069977136336290239', time=2024-03-09T17:24:20.626766+0000, chain.objs.size()=1 2024-03-09T15:36:06.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2hPCogt9jL_0jkvcHCSQ1__D0ZPl1Fu_0 2024-03-09T15:36:06.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17103302878240623096', time=2024-03-09T17:24:21.508429+0000, chain.objs.size()=1 2024-03-09T15:36:06.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.a_ItoO8rFl1ULUQjvsCSnKVR8n4-27f_0 2024-03-09T15:36:06.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11914119168393763202', time=2024-03-09T17:24:21.716840+0000, chain.objs.size()=1 2024-03-09T15:36:06.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3QmaqYxHNHnLCAqpl71EKVvbE1t-dDS_0 2024-03-09T15:36:06.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9178986254086821396', time=2024-03-09T17:24:21.795509+0000, chain.objs.size()=1 2024-03-09T15:36:06.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ZXYzb6HXqIAjSIZdFSlPLi4w0pPU4dt_0 2024-03-09T15:36:06.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1471947195231992476', time=2024-03-09T17:24:22.017657+0000, chain.objs.size()=1 2024-03-09T15:36:06.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.AYyXHoqeq-T9BFtPvR0kZZEEM61CDcN_0 2024-03-09T15:36:06.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15642934835799393491', time=2024-03-09T17:24:25.835440+0000, chain.objs.size()=1 2024-03-09T15:36:06.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.EFeqbP1FGue-FzOs3b1z63e2nJVyyHU_0 2024-03-09T15:36:06.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3123279406904957082', time=2024-03-09T17:24:27.422068+0000, chain.objs.size()=1 2024-03-09T15:36:06.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.x6OGe5FXKTSqb6Jl1FgBh_63LOboFy3_0 2024-03-09T15:36:06.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.850342966970920606', time=2024-03-09T17:24:27.640756+0000, chain.objs.size()=1 2024-03-09T15:36:06.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.336+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.bCDBMlej5ILCm6Ghbea8PNSS84CRtPZ_0 2024-03-09T15:36:06.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12926512650555490175', time=2024-03-09T17:24:31.404192+0000, chain.objs.size()=1 2024-03-09T15:36:06.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.umRXWxq5Df8uxZNL2KQ5KiA7hIKWsBz_0 2024-03-09T15:36:06.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6265089356847443690', time=2024-03-09T17:24:31.429219+0000, chain.objs.size()=1 2024-03-09T15:36:06.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.mbonBKMLUY_j2nJxMTH9bswb9Ezu__S_0 2024-03-09T15:36:06.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9245743023672789461', time=2024-03-09T17:26:01.463616+0000, chain.objs.size()=1 2024-03-09T15:36:06.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.bJblGDZSu_LqI3W9ySz_wRq4iiA9ie5_0 2024-03-09T15:36:06.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4287427647130085218', time=2024-03-09T17:26:07.796740+0000, chain.objs.size()=1 2024-03-09T15:36:06.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.qbaSBXgp9yYtWeJxhbhBxNSu7kU2lCQ_0 2024-03-09T15:36:06.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8988388686804935284', time=2024-03-09T17:26:31.782055+0000, chain.objs.size()=1 2024-03-09T15:36:06.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.294__shadow_.5j3H7p6jiYNqFmUwSNML-SNDBqp1Ubc_0 2024-03-09T15:36:06.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8136054417826362797', time=2024-03-09T17:30:45.073864+0000, chain.objs.size()=1 2024-03-09T15:36:06.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.311__shadow_.pzk9gowWYVPeP12HmfjKnfpprkBX5pN_0 2024-03-09T15:36:06.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11010207116154563370', time=2024-03-09T17:34:31.729678+0000, chain.objs.size()=1 2024-03-09T15:36:06.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.337__shadow_.jlG06XHSKGaFywatAQ9Gk7hpDOye_7T_0 2024-03-09T15:36:06.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.340+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.344+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-03-09T15:36:06.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.344+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.344+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=41, truncated=0, next_marker='' 2024-03-09T15:36:06.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5322794618836835495', time=2024-03-09T17:11:44.004776+0000, chain.objs.size()=1 2024-03-09T15:36:06.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.140__shadow_.bRNO0Pp_xkMdUJhM_IcDY7mpCtAIguF_0 2024-03-09T15:36:06.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_0vyh2Rf1IDKW1_nvSC6WarAhpBgSZsN', time=2024-03-09T17:14:03.791874+0000, chain.objs.size()=2 2024-03-09T15:36:06.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.158__shadow_.F1DIqGR0ABBHmOvXJfts9graJ51FuGT_0 2024-03-09T15:36:06.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.158__shadow_.F1DIqGR0ABBHmOvXJfts9graJ51FuGT_1 2024-03-09T15:36:06.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8697078835134838785', time=2024-03-09T17:15:12.649705+0000, chain.objs.size()=1 2024-03-09T15:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.181__shadow_.OHhStpIHN8ICo8Ox8dnmCm916nEH1D9_0 2024-03-09T15:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11969443695694958125', time=2024-03-09T17:20:48.798019+0000, chain.objs.size()=1 2024-03-09T15:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.227__shadow_.ZPB_RvyS5BpnHMLiTrOKhg0Vjcs6AXj_0 2024-03-09T15:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18213727728847388985', time=2024-03-09T17:20:54.663380+0000, chain.objs.size()=1 2024-03-09T15:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.229__shadow_.LSvFNcH0FLXh4dANheoB1QzAuQFFD4X_0 2024-03-09T15:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12970529567908252683', time=2024-03-09T17:21:23.134185+0000, chain.objs.size()=1 2024-03-09T15:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.237__shadow_.jG2TxAYvRCO3qqs7dN4mkLG7I85Mn2x_0 2024-03-09T15:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7564458096307163111', time=2024-03-09T17:21:34.879119+0000, chain.objs.size()=1 2024-03-09T15:36:06.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.241__shadow_.atbX1Dn78lphkQl6Gj-VNqzXyuK3WjQ_0 2024-03-09T15:36:06.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14004447599584275300', time=2024-03-09T17:22:00.517373+0000, chain.objs.size()=1 2024-03-09T15:36:06.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.250__shadow_.twSd5LBN8SJwEYhTQE97_mRDUy_SaMh_0 2024-03-09T15:36:06.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12830369547046401812', time=2024-03-09T17:22:09.479666+0000, chain.objs.size()=1 2024-03-09T15:36:06.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.253__shadow_.sD22OGuTpy8mBfIhJbQPjOLeUM3DIp6_0 2024-03-09T15:36:06.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18276273198504744281', time=2024-03-09T17:23:58.785535+0000, chain.objs.size()=1 2024-03-09T15:36:06.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.q9z9RKkD1WBDD2PiC550GsbnY8cXrm6_0 2024-03-09T15:36:06.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.895894084417395780', time=2024-03-09T17:23:59.737416+0000, chain.objs.size()=1 2024-03-09T15:36:06.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ZWSL733sZiT0IlQRCyVtZgymTNnkGKb_0 2024-03-09T15:36:06.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17000471025937687238', time=2024-03-09T17:23:59.948227+0000, chain.objs.size()=1 2024-03-09T15:36:06.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7NfLF7cRQ_6tSKfUn-6cRNZEPyKsz8Y_0 2024-03-09T15:36:06.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16910254261316570986', time=2024-03-09T17:24:04.834699+0000, chain.objs.size()=1 2024-03-09T15:36:06.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.o5Wv5tgT_VW5kCmV62lbdlmN9I9Zjsy_0 2024-03-09T15:36:06.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.348+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11987298108952312447', time=2024-03-09T17:24:06.579007+0000, chain.objs.size()=1 2024-03-09T15:36:06.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.TSIZgyYS2azHJKWf4qePwO8OETCxO9__0 2024-03-09T15:36:06.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8574236459792098080', time=2024-03-09T17:24:08.323990+0000, chain.objs.size()=1 2024-03-09T15:36:06.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6as8RGP5-rUIPbl7BIDxrb3Qarj6GC4_0 2024-03-09T15:36:06.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12821428224897238679', time=2024-03-09T17:24:09.468188+0000, chain.objs.size()=1 2024-03-09T15:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6KyOlaUB0xN9xnaadoYbczPcS1iQOI0_0 2024-03-09T15:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18377101950432539476', time=2024-03-09T17:24:10.041335+0000, chain.objs.size()=1 2024-03-09T15:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HwzGgYMxhd5CcmQJ6NQbSxZlBhoPM7M_0 2024-03-09T15:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.553366192320073943', time=2024-03-09T17:24:10.300099+0000, chain.objs.size()=1 2024-03-09T15:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.CT4gdS_s3C0mOd822taZwgEIH_TyyDf_0 2024-03-09T15:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.146229383280009588', time=2024-03-09T17:24:10.574803+0000, chain.objs.size()=1 2024-03-09T15:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HyeaOlhKXZs7Ibgf88f9cIkcs72Cthz_0 2024-03-09T15:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13399684256709177200', time=2024-03-09T17:24:11.165570+0000, chain.objs.size()=1 2024-03-09T15:36:06.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.YL3TqhuXdNui-hONe2elQO0BeuSO4pi_0 2024-03-09T15:36:06.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7170605012100013048', time=2024-03-09T17:24:13.810424+0000, chain.objs.size()=1 2024-03-09T15:36:06.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.gHDduN_-nq_M9dJsyuuamEyhWtu-FKi_0 2024-03-09T15:36:06.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.948720667109015738', time=2024-03-09T17:24:17.764522+0000, chain.objs.size()=1 2024-03-09T15:36:06.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3j3E0aySG2kdaf1VcyMKBhpxxxD6sqd_0 2024-03-09T15:36:06.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14246457500249371472', time=2024-03-09T17:24:20.988756+0000, chain.objs.size()=1 2024-03-09T15:36:06.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tvV0166rzwh09yp--yiElK-sYXTwzsj_0 2024-03-09T15:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6271041843700948335', time=2024-03-09T17:24:21.777193+0000, chain.objs.size()=1 2024-03-09T15:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.F42WeCVfnXEgyxygRcv5mAQ5MeEctHv_0 2024-03-09T15:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16680790401898746370', time=2024-03-09T17:24:22.797599+0000, chain.objs.size()=1 2024-03-09T15:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.eOPX_Xdxi-wYhrlDNZUteQKoA8tiBxe_0 2024-03-09T15:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5494913363482102134', time=2024-03-09T17:24:23.352159+0000, chain.objs.size()=1 2024-03-09T15:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PFTwi43CyPPs76O5GWXwNwo9YSL-5wx_0 2024-03-09T15:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2723050788992546833', time=2024-03-09T17:24:23.515257+0000, chain.objs.size()=1 2024-03-09T15:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.U-q6kZXjFdPQsKk-Enu13R5z9diLH1m_0 2024-03-09T15:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8733003272842545888', time=2024-03-09T17:24:24.727744+0000, chain.objs.size()=1 2024-03-09T15:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.XzoSXgWLu_GHQKM6D27L7_DC9YonDsU_0 2024-03-09T15:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2399144100959395628', time=2024-03-09T17:24:25.115218+0000, chain.objs.size()=1 2024-03-09T15:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.gqu2giD0-z5DlxRLQYNDCdN6Ka9PCg9_0 2024-03-09T15:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15228764400130215300', time=2024-03-09T17:24:26.619491+0000, chain.objs.size()=1 2024-03-09T15:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6YqSzj9M5VvpU8Gpj1kRKcuPYYYg-P__0 2024-03-09T15:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7673634081563752195', time=2024-03-09T17:24:27.430572+0000, chain.objs.size()=1 2024-03-09T15:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fuY9WnhGP_SkYVux6d0dJCaOGVTC5z2_0 2024-03-09T15:36:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6563347345301120548', time=2024-03-09T17:24:27.752286+0000, chain.objs.size()=1 2024-03-09T15:36:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_._SSUj7MY03Z77OsMDprkQgaWmZu63Ng_0 2024-03-09T15:36:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7150010759782041035', time=2024-03-09T17:24:42.661186+0000, chain.objs.size()=1 2024-03-09T15:36:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.259__shadow_.mjIfTOgn3Fu_RTqVcM14CcKXvGmIMnO_0 2024-03-09T15:36:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6095969914415043461', time=2024-03-09T17:25:01.940006+0000, chain.objs.size()=1 2024-03-09T15:36:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.266__shadow_.FoqLkP5Sr3trSluYNpDlCwS1pXXhB3I_0 2024-03-09T15:36:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6644082480954066161', time=2024-03-09T17:25:13.475145+0000, chain.objs.size()=1 2024-03-09T15:36:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.352+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.270__shadow_.TziPJYVJTGWaH77uYoPwBgGA7gRerM2_0 2024-03-09T15:36:06.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18238583526162072607', time=2024-03-09T17:25:48.345515+0000, chain.objs.size()=1 2024-03-09T15:36:06.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.282__shadow_.INpJ5wXVxyIImIbbo-qPiztt9f12D_8_0 2024-03-09T15:36:06.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12716039043154684241', time=2024-03-09T17:26:01.363524+0000, chain.objs.size()=1 2024-03-09T15:36:06.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.nSHTFeIbhnvkpPmrQIecCiffkx-HGNC_0 2024-03-09T15:36:06.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7226706895910253041', time=2024-03-09T17:26:19.750314+0000, chain.objs.size()=1 2024-03-09T15:36:06.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.290__shadow_.lcjUZYwjXPaLpnvo84KMMUpQKhdcVbv_0 2024-03-09T15:36:06.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3359691643626429599', time=2024-03-09T17:26:31.801902+0000, chain.objs.size()=1 2024-03-09T15:36:06.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.294__shadow_.DACmyyVHAOmSkdQ1HrOU3yOMphv5Aty_0 2024-03-09T15:36:06.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12317488312443830047', time=2024-03-09T17:26:43.492717+0000, chain.objs.size()=1 2024-03-09T15:36:06.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.298__shadow_.JyhXIByMtqA7dkBg1QVg1XR0VySzUVk_0 2024-03-09T15:36:06.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7914673954126660590', time=2024-03-09T17:34:45.055368+0000, chain.objs.size()=1 2024-03-09T15:36:06.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.339__shadow_.Dyc0ViNqQwJ_GVp8yYx8i1v_O9U97sl_0 2024-03-09T15:36:06.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.356+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.360+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-03-09T15:36:06.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.360+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=44, truncated=0, next_marker='' 2024-03-09T15:36:06.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16900085584675938898', time=2024-03-09T17:12:34.733737+0000, chain.objs.size()=3 2024-03-09T15:36:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.152__multipart_mymultipart-FROZEN.2~Nyi71UW0ynNr9nTBpCdiiNhysX8KLLB.1 2024-03-09T15:36:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.152__multipart_mymultipart-FROZEN.2~Nyi71UW0ynNr9nTBpCdiiNhysX8KLLB.2 2024-03-09T15:36:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.152__multipart_mymultipart-FROZEN.2~Nyi71UW0ynNr9nTBpCdiiNhysX8KLLB.3 2024-03-09T15:36:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14423431069104031377', time=2024-03-09T17:20:40.194868+0000, chain.objs.size()=1 2024-03-09T15:36:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.224__shadow_.D4bAclBsiH53Oz9arXv9VNEjnoLFkyo_0 2024-03-09T15:36:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.601733043640676281', time=2024-03-09T17:20:54.678740+0000, chain.objs.size()=1 2024-03-09T15:36:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.229__shadow_.sm0hN7-iEP8me7jUsQaZ4OudJS1w0SA_0 2024-03-09T15:36:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15530735268689816379', time=2024-03-09T17:21:18.711668+0000, chain.objs.size()=1 2024-03-09T15:36:06.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.236__shadow_.wYsrb9b4d7u_qOyfSyZfzvzeZ8EyA_b_0 2024-03-09T15:36:06.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10864220428343248671', time=2024-03-09T17:21:34.861165+0000, chain.objs.size()=1 2024-03-09T15:36:06.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.241__shadow_.YKZ2LpqjEBBgSoCBJUqqn2Qe4fKDhQ__0 2024-03-09T15:36:06.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10464647644184232322', time=2024-03-09T17:21:52.098704+0000, chain.objs.size()=1 2024-03-09T15:36:06.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.247__shadow_.WQBb0ibnlY_ZgsuQsKi5Upe0OPVMEkW_0 2024-03-09T15:36:06.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18404680733224268839', time=2024-03-09T17:23:59.654501+0000, chain.objs.size()=1 2024-03-09T15:36:06.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.R2L6xru4um9gk3kM6w1eDHQtu4c-lQ8_0 2024-03-09T15:36:06.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14335833634408112268', time=2024-03-09T17:23:59.754073+0000, chain.objs.size()=1 2024-03-09T15:36:06.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.esI2gS2PMTcE-EBUvzsLOfWGVX7BmAA_0 2024-03-09T15:36:06.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8799787620629331378', time=2024-03-09T17:24:02.171422+0000, chain.objs.size()=1 2024-03-09T15:36:06.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Clpd1wWdl4eezi-JBFGxL_rOrXNiYZk_0 2024-03-09T15:36:06.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7709190124158715348', time=2024-03-09T17:24:03.952954+0000, chain.objs.size()=1 2024-03-09T15:36:06.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.BA0MJ5FkcEzDQ0YH4lhLia7RVWUSAPz_0 2024-03-09T15:36:06.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11531689946924601868', time=2024-03-09T17:24:04.112514+0000, chain.objs.size()=1 2024-03-09T15:36:06.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.364+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ELrweo5Af09bzyAznMKO7JEsV66cAlx_0 2024-03-09T15:36:06.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3364578691080103324', time=2024-03-09T17:24:04.461002+0000, chain.objs.size()=1 2024-03-09T15:36:06.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.cpR4pzkVi3w_KILUQTpG3xI_6XFBz8C_0 2024-03-09T15:36:06.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.767184095734037090', time=2024-03-09T17:24:07.880752+0000, chain.objs.size()=1 2024-03-09T15:36:06.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.cn6l3UemQLQWIkdZqTgE27cIWJm3Hjf_0 2024-03-09T15:36:06.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2249669318357090383', time=2024-03-09T17:24:07.936115+0000, chain.objs.size()=1 2024-03-09T15:36:06.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ltz37ykCRD3_llGSV-KYzlPjgFqak-V_0 2024-03-09T15:36:06.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5080452521326647947', time=2024-03-09T17:24:08.820929+0000, chain.objs.size()=1 2024-03-09T15:36:06.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.wzV1JFSUAaSEEALYHEf_bIixqQbT4qg_0 2024-03-09T15:36:06.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13192664596694658659', time=2024-03-09T17:24:09.313359+0000, chain.objs.size()=1 2024-03-09T15:36:06.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tIrbm7yPL5S2JuHYFQZXZ5sxWYAetcU_0 2024-03-09T15:36:06.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5147284266595914794', time=2024-03-09T17:24:09.630712+0000, chain.objs.size()=1 2024-03-09T15:36:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Vr9wYY5X1mfH81DCSIPjb02EeVkULDt_0 2024-03-09T15:36:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8185958802161397663', time=2024-03-09T17:24:11.162708+0000, chain.objs.size()=1 2024-03-09T15:36:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qJW1h3gU6Lcx8KDpvc9bNbjVeM2gdmf_0 2024-03-09T15:36:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.869109848875804140', time=2024-03-09T17:24:12.071486+0000, chain.objs.size()=1 2024-03-09T15:36:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.uUaGarKcMs3GNtTU9-7oUeNYWsKvULr_0 2024-03-09T15:36:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15278153150278947177', time=2024-03-09T17:24:13.018151+0000, chain.objs.size()=1 2024-03-09T15:36:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6Zq3QOwgRDl859BEvonvWnCSG48sAYO_0 2024-03-09T15:36:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11363465518256048867', time=2024-03-09T17:24:13.487987+0000, chain.objs.size()=1 2024-03-09T15:36:06.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.NwDNns2oNW_bIf5uN-M8V6YL07sKJ1p_0 2024-03-09T15:36:06.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7617654731981876735', time=2024-03-09T17:24:13.804251+0000, chain.objs.size()=1 2024-03-09T15:36:06.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.WdumXu-UgJLBEmT_OOOLCVAthxlgWy2_0 2024-03-09T15:36:06.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6284088503811338698', time=2024-03-09T17:24:13.901966+0000, chain.objs.size()=1 2024-03-09T15:36:06.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-0KXq5E-k4VxgRQQCk1eO1hxP0nkigr_0 2024-03-09T15:36:06.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11665674804463459265', time=2024-03-09T17:24:13.903546+0000, chain.objs.size()=1 2024-03-09T15:36:06.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.4qo-m9SqmW-g9ejsJhfzi0mJ9LudMSM_0 2024-03-09T15:36:06.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8994432384445180198', time=2024-03-09T17:24:16.464267+0000, chain.objs.size()=1 2024-03-09T15:36:06.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2jHC3Ui4ddJFzIga0Zo7ZZlPOjd7T03_0 2024-03-09T15:36:06.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4608150296371401141', time=2024-03-09T17:24:18.127171+0000, chain.objs.size()=1 2024-03-09T15:36:06.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.A9aReUbODGpMSPZiQc5UEwg8mVSWLLS_0 2024-03-09T15:36:06.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15417766357395417952', time=2024-03-09T17:24:18.774706+0000, chain.objs.size()=1 2024-03-09T15:36:06.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.AgyIq5YCpOb6J73kpyhdnpvtZ0KgZR5_0 2024-03-09T15:36:06.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.166168873447330554', time=2024-03-09T17:24:18.933146+0000, chain.objs.size()=1 2024-03-09T15:36:06.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.IXtTMprCdh2FvLl77rQB7JD5RToGQ5m_0 2024-03-09T15:36:06.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7645159576333009308', time=2024-03-09T17:24:20.489485+0000, chain.objs.size()=1 2024-03-09T15:36:06.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.bPXnzZ2IDeD5QHUdAWNIP9MyllM0LjE_0 2024-03-09T15:36:06.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1661241528241386525', time=2024-03-09T17:24:20.837816+0000, chain.objs.size()=1 2024-03-09T15:36:06.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.jOHtJAcKF-NNAAGKqxfZ3ScNemEbUGY_0 2024-03-09T15:36:06.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.276897980424140493', time=2024-03-09T17:24:23.162450+0000, chain.objs.size()=1 2024-03-09T15:36:06.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.5uVujJ1rKp8NaCzDd0g_QHXGyJhbabz_0 2024-03-09T15:36:06.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17538454356829397648', time=2024-03-09T17:24:24.611456+0000, chain.objs.size()=1 2024-03-09T15:36:06.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0v-T304rC4SV8GyhhOeo0YK5dwRfSJ5_0 2024-03-09T15:36:06.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7707083860009279406', time=2024-03-09T17:24:24.738063+0000, chain.objs.size()=1 2024-03-09T15:36:06.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.368+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Z8UxTGJwsgJ9Q-jQQtCepkHvjm1F6tj_0 2024-03-09T15:36:06.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18066670643057731001', time=2024-03-09T17:24:25.015104+0000, chain.objs.size()=1 2024-03-09T15:36:06.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.uK4kbbOI5vEHqH6DDeA6K2ILmDjT-oU_0 2024-03-09T15:36:06.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14151181589635761722', time=2024-03-09T17:24:28.745807+0000, chain.objs.size()=1 2024-03-09T15:36:06.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2xDHOyQLcBvnSGMo_ZDUQHLIj00jcDz_0 2024-03-09T15:36:06.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4593030136525986257', time=2024-03-09T17:24:28.749322+0000, chain.objs.size()=1 2024-03-09T15:36:06.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1yufJgq9RmjoBoaCU4IwTG6DfC21ZRR_0 2024-03-09T15:36:06.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.32568276754547065', time=2024-03-09T17:24:31.216265+0000, chain.objs.size()=1 2024-03-09T15:36:06.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.34kMo_h6-J6WQTonseL0EUUnEhROcAh_0 2024-03-09T15:36:06.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3049678512572366067', time=2024-03-09T17:24:42.654594+0000, chain.objs.size()=1 2024-03-09T15:36:06.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.259__shadow_.OrUdvsixk05sgJTNnA-Zrh4AkmkD7Hv_0 2024-03-09T15:36:06.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12723850773623000791', time=2024-03-09T17:25:16.352022+0000, chain.objs.size()=1 2024-03-09T15:36:06.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.271__shadow_.YZ3BGI0O4jCazWopKul6CKlFkikZoOE_0 2024-03-09T15:36:06.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.637626810746913638', time=2024-03-09T17:26:01.737638+0000, chain.objs.size()=1 2024-03-09T15:36:06.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.YhZA5PXaH5oUriKdYtDNlTyIjWy3FgW_0 2024-03-09T15:36:06.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1212017713260795244', time=2024-03-09T17:26:11.103866+0000, chain.objs.size()=1 2024-03-09T15:36:06.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.287__shadow_.oi_mXGY_2-Npd9-OluybG41sJ4OiGZj_0 2024-03-09T15:36:06.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.751332031226698493', time=2024-03-09T17:26:16.804568+0000, chain.objs.size()=1 2024-03-09T15:36:06.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.289__shadow_.17wiCbEVDbLU9svTH-uHd3jnIKTh2QX_0 2024-03-09T15:36:06.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2987324376615388065', time=2024-03-09T17:31:41.972731+0000, chain.objs.size()=1 2024-03-09T15:36:06.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.318__shadow_.hMbMuVPGAO8AveL9ppIqMQRG0reo2h-_0 2024-03-09T15:36:06.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8778039202782829103', time=2024-03-09T17:31:55.227238+0000, chain.objs.size()=1 2024-03-09T15:36:06.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.372+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.321__shadow_.O9pR1e3hCNh4Bctx-GowktzJFqk8-gh_0 2024-03-09T15:36:06.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.376+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-03-09T15:36:06.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=47, truncated=0, next_marker='' 2024-03-09T15:36:06.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10062108253506746488', time=2024-03-09T17:11:48.263116+0000, chain.objs.size()=1 2024-03-09T15:36:06.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.147__shadow_.Rjb_VXeHctx1wxBI3hvs4mL3b_SS5um_0 2024-03-09T15:36:06.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12987312133624441139', time=2024-03-09T17:12:14.665486+0000, chain.objs.size()=2 2024-03-09T15:36:06.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.151__shadow_.Xy9M-4ySDaU9KOn94G_Tj4fli7PwURm_0 2024-03-09T15:36:06.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.151__shadow_.Xy9M-4ySDaU9KOn94G_Tj4fli7PwURm_1 2024-03-09T15:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3074930663308174440', time=2024-03-09T17:16:17.054962+0000, chain.objs.size()=1 2024-03-09T15:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.193__shadow_.o8tSWh7IRfDTwmLDxrNvJ3IImyPINzU_0 2024-03-09T15:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6693000009373380348', time=2024-03-09T17:20:45.915256+0000, chain.objs.size()=1 2024-03-09T15:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.226__shadow_.YiW1xKGqoG1YUA5jxbB2NuLEbh9IjeE_0 2024-03-09T15:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12677142655947397395', time=2024-03-09T17:21:52.067834+0000, chain.objs.size()=1 2024-03-09T15:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.247__shadow_.9AunROTMEVqrLNT_wUaALTPdQMxojwE_0 2024-03-09T15:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12585446014404010759', time=2024-03-09T17:21:52.094570+0000, chain.objs.size()=1 2024-03-09T15:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.247__shadow_.9f8wa0AqT9uE9TYcqb41j8YbrJqfbvv_0 2024-03-09T15:36:06.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17499845906482865848', time=2024-03-09T17:22:06.005542+0000, chain.objs.size()=1 2024-03-09T15:36:06.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.252__shadow_.dXLokoEBL8D2Cq4ya4zF3Aa-HEss5_Y_0 2024-03-09T15:36:06.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1115012207929722105', time=2024-03-09T17:23:59.440903+0000, chain.objs.size()=1 2024-03-09T15:36:06.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.rImY5rHee7gNK1qkGpZPlnh9tO1AzGF_0 2024-03-09T15:36:06.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17057564644784677126', time=2024-03-09T17:24:02.165562+0000, chain.objs.size()=1 2024-03-09T15:36:06.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.FYWNadSWWf9WVQnnI4wwxS4DEccuHXX_0 2024-03-09T15:36:06.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.284298702928224214', time=2024-03-09T17:24:03.491335+0000, chain.objs.size()=1 2024-03-09T15:36:06.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sE_Xc2M9Raz3zPE3BRmLdp5bMiAve1u_0 2024-03-09T15:36:06.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9312855348466276182', time=2024-03-09T17:24:03.959872+0000, chain.objs.size()=1 2024-03-09T15:36:06.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.380+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sbCXgQlvzDpBV_ZrytcjHIHeyGmXR1g_0 2024-03-09T15:36:06.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14196281192964459581', time=2024-03-09T17:24:04.823820+0000, chain.objs.size()=1 2024-03-09T15:36:06.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LErFKPaR0Dx30SDxrbJnC09DQIYHRge_0 2024-03-09T15:36:06.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7185811114913573798', time=2024-03-09T17:24:05.231248+0000, chain.objs.size()=1 2024-03-09T15:36:06.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.nUAwaS0oXPKSd3zRoTyFLM6jhRZvvpf_0 2024-03-09T15:36:06.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4589503992321952651', time=2024-03-09T17:24:05.644030+0000, chain.objs.size()=1 2024-03-09T15:36:06.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6La50D8-v6oOfbZq80eDEFIgmGMLqn6_0 2024-03-09T15:36:06.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15408628706459823847', time=2024-03-09T17:24:06.124303+0000, chain.objs.size()=1 2024-03-09T15:36:06.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LarLq5f8VDpmtKVfqQLvcsJ1mbaAAE8_0 2024-03-09T15:36:06.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16217132327979959752', time=2024-03-09T17:24:07.887122+0000, chain.objs.size()=1 2024-03-09T15:36:06.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.JxFORV5CmQd4wXiSBtncAaV9joRQvEb_0 2024-03-09T15:36:06.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5951748560023730977', time=2024-03-09T17:24:08.316674+0000, chain.objs.size()=1 2024-03-09T15:36:06.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tqlBmGomHzEg7J7dyFlChHxh8IBsPJN_0 2024-03-09T15:36:06.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11254097868584012727', time=2024-03-09T17:24:08.465875+0000, chain.objs.size()=1 2024-03-09T15:36:06.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.wu2Z3T0A9Kk2u230_6Chal6Rh1vecGc_0 2024-03-09T15:36:06.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7406882254795492726', time=2024-03-09T17:24:09.307270+0000, chain.objs.size()=1 2024-03-09T15:36:06.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.e9eHaRx_VZCsBN6TwQi0pjv83YYTmm5_0 2024-03-09T15:36:06.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14606279591455824009', time=2024-03-09T17:24:09.714734+0000, chain.objs.size()=1 2024-03-09T15:36:06.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LOI4aw0XcXz6OS5eAZF1EJtiCfacrCE_0 2024-03-09T15:36:06.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16291067152190507714', time=2024-03-09T17:24:09.716908+0000, chain.objs.size()=1 2024-03-09T15:36:06.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.CHb8Lb_uIfoenVCm_ScUQn5JlcaHPy1_0 2024-03-09T15:36:06.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14868674464077229179', time=2024-03-09T17:24:12.982518+0000, chain.objs.size()=1 2024-03-09T15:36:06.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.384+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.aAzyHbEcD-K99rkDRrq4g-t3E17nGJw_0 2024-03-09T15:36:06.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1908422953597698491', time=2024-03-09T17:24:14.091957+0000, chain.objs.size()=1 2024-03-09T15:36:06.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.n8j9nsoJdpCkpRYsBVdvh43s2_Sk5Qc_0 2024-03-09T15:36:06.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13548295196516693706', time=2024-03-09T17:24:15.391346+0000, chain.objs.size()=1 2024-03-09T15:36:06.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Z1CsLuJUU_k5Hd0Uv1eW1YhUREO7jeP_0 2024-03-09T15:36:06.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17245674680686668064', time=2024-03-09T17:24:17.187219+0000, chain.objs.size()=1 2024-03-09T15:36:06.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.dqr7kTlRvvMfKPvtTIFwbb14EOOmweZ_0 2024-03-09T15:36:06.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.656959406548680927', time=2024-03-09T17:24:18.140150+0000, chain.objs.size()=1 2024-03-09T15:36:06.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ZKR1PK4w_YDeTEAhMG0tta9RRIGi4VY_0 2024-03-09T15:36:06.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13564124426651732360', time=2024-03-09T17:24:18.485043+0000, chain.objs.size()=1 2024-03-09T15:36:06.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.imrzSBMgSwlQMGZfGjTEjqbtyYD56JC_0 2024-03-09T15:36:06.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3927514321514370623', time=2024-03-09T17:24:19.086925+0000, chain.objs.size()=1 2024-03-09T15:36:06.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.D6x23mfG-dhEmC51MlyouRRhEdTr-Hc_0 2024-03-09T15:36:06.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14909744611373206563', time=2024-03-09T17:24:20.808557+0000, chain.objs.size()=1 2024-03-09T15:36:06.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.pcR7QT0PgGlh4x3KZxT-RZhuwSzGJuv_0 2024-03-09T15:36:06.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15486365529897628689', time=2024-03-09T17:24:20.938175+0000, chain.objs.size()=1 2024-03-09T15:36:06.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.FNaIx3b4cG0wxtuhfWO7oWbbB8B4_2F_0 2024-03-09T15:36:06.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13598192111845153038', time=2024-03-09T17:24:23.491618+0000, chain.objs.size()=1 2024-03-09T15:36:06.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.XClRwYHZKzMiBdxFDG4C9neQymU8ygw_0 2024-03-09T15:36:06.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18113250421811262211', time=2024-03-09T17:24:23.503290+0000, chain.objs.size()=1 2024-03-09T15:36:06.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Dwgs0rBD9Ro_uKrj6xQss-xTg8JFrNZ_0 2024-03-09T15:36:06.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.702151043156792554', time=2024-03-09T17:24:23.618340+0000, chain.objs.size()=1 2024-03-09T15:36:06.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.KFHxEID1C_flKFwifafAgd1xUTELH40_0 2024-03-09T15:36:06.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9859407107380956740', time=2024-03-09T17:24:23.620664+0000, chain.objs.size()=1 2024-03-09T15:36:06.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.388+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9l5GfpKhO6HWLX4rua4Q_arf9TDYevs_0 2024-03-09T15:36:06.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1129381801334259900', time=2024-03-09T17:24:24.578179+0000, chain.objs.size()=1 2024-03-09T15:36:06.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qGwyr4js0qpo_ySVCYa_1FR9PjoxvBo_0 2024-03-09T15:36:06.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8796051266195938608', time=2024-03-09T17:24:24.714175+0000, chain.objs.size()=1 2024-03-09T15:36:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.AmZLq-w91A-_eFUHekBFMyA_UeIQVrb_0 2024-03-09T15:36:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13248037189351009434', time=2024-03-09T17:24:26.283151+0000, chain.objs.size()=1 2024-03-09T15:36:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.lVSC-xXj9NGBy4zrtohXOChdiFUXaeP_0 2024-03-09T15:36:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15527250663587061958', time=2024-03-09T17:24:27.523558+0000, chain.objs.size()=1 2024-03-09T15:36:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.cUuA3RY2EFzsUo5BNrCMbLQtm_V_pk0_0 2024-03-09T15:36:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4489141211484984367', time=2024-03-09T17:24:28.696998+0000, chain.objs.size()=1 2024-03-09T15:36:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.M0rofDJrTZtCyrinbUJTRFgyxptuI4x_0 2024-03-09T15:36:06.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14120391022346310705', time=2024-03-09T17:24:28.698564+0000, chain.objs.size()=1 2024-03-09T15:36:06.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.m42fb05tWsx7252YmJtb4W-OteplQtP_0 2024-03-09T15:36:06.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15290112339782376099', time=2024-03-09T17:24:34.329772+0000, chain.objs.size()=1 2024-03-09T15:36:06.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.256__shadow_.2H0WbyZFibn_r6O72QEsYG2J6wdBnKk_0 2024-03-09T15:36:06.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16423546940479297390', time=2024-03-09T17:24:50.891931+0000, chain.objs.size()=1 2024-03-09T15:36:06.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.262__shadow_.YLXZgvVh5dk1FWdXu3hu30e9JI-wXpV_0 2024-03-09T15:36:06.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2758898831519138914', time=2024-03-09T17:25:01.947233+0000, chain.objs.size()=1 2024-03-09T15:36:06.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.266__shadow_.VMm57KpgiS_vwkh4UofTLU2jZM63AnF_0 2024-03-09T15:36:06.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1037374111530259838', time=2024-03-09T17:26:01.783317+0000, chain.objs.size()=1 2024-03-09T15:36:06.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.lMgMrOaHFqfbl-MFeuOQ-lutX5hiTEl_0 2024-03-09T15:36:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14954549637769748700', time=2024-03-09T17:26:07.938570+0000, chain.objs.size()=1 2024-03-09T15:36:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.xOyDWTNpZFyaPJrfB-w_pYa_zgi-xVz_0 2024-03-09T15:36:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11794620681039076834', time=2024-03-09T17:26:22.900616+0000, chain.objs.size()=1 2024-03-09T15:36:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.291__shadow_.GQCMGAwaw3rH7UilL_j9g5i_k3IId5J_0 2024-03-09T15:36:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1952263505328224309', time=2024-03-09T17:26:28.905183+0000, chain.objs.size()=1 2024-03-09T15:36:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.293__shadow_.XAuGZ-0SDwW9L-PFU02SEgWjpIkOamG_0 2024-03-09T15:36:06.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.392+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.396+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-03-09T15:36:06.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.396+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=49, truncated=0, next_marker='' 2024-03-09T15:36:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2042016076988888734', time=2024-03-09T17:11:47.591797+0000, chain.objs.size()=1 2024-03-09T15:36:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.146__shadow_.aUz2I9fJECL1Q3gtlZ9zea-fA7srsqW_0 2024-03-09T15:36:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12334385123036617650', time=2024-03-09T17:12:34.896186+0000, chain.objs.size()=3 2024-03-09T15:36:06.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.152__multipart_mymultipart-STANDARD.2~m9xkB7eYn6WMBO74TmnPWsHpNlWZZLe.1 2024-03-09T15:36:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.152__multipart_mymultipart-STANDARD.2~m9xkB7eYn6WMBO74TmnPWsHpNlWZZLe.2 2024-03-09T15:36:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.152__multipart_mymultipart-STANDARD.2~m9xkB7eYn6WMBO74TmnPWsHpNlWZZLe.3 2024-03-09T15:36:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10135603155806979235', time=2024-03-09T17:20:48.796727+0000, chain.objs.size()=1 2024-03-09T15:36:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.227__shadow_.eTwl9PXql3aptt1jNI8F0C7QrXMSyPg_0 2024-03-09T15:36:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18132165388904900072', time=2024-03-09T17:21:40.693387+0000, chain.objs.size()=1 2024-03-09T15:36:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.243__shadow_.p8qXMC4YjKOeWd4caYyWGceGOf4S4bh_0 2024-03-09T15:36:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6905110675697113095', time=2024-03-09T17:21:54.812811+0000, chain.objs.size()=1 2024-03-09T15:36:06.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.248__shadow_.Dj1weY4CTFiT7M8kEtffumgUmvCQCwX_0 2024-03-09T15:36:06.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3634164030878740344', time=2024-03-09T17:24:00.632805+0000, chain.objs.size()=1 2024-03-09T15:36:06.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.a2cISJ0ABihrC6jjADWQhKsybapXBrH_0 2024-03-09T15:36:06.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14947313207640338097', time=2024-03-09T17:24:00.714700+0000, chain.objs.size()=1 2024-03-09T15:36:06.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0DUxqCi0AYB895w9WHnpBHbc6A_Vt9s_0 2024-03-09T15:36:06.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5183465766550364162', time=2024-03-09T17:24:01.835867+0000, chain.objs.size()=1 2024-03-09T15:36:06.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9YHXd_ElB8KVArWup2quo901pZNMt0j_0 2024-03-09T15:36:06.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6648257818840092991', time=2024-03-09T17:24:04.464662+0000, chain.objs.size()=1 2024-03-09T15:36:06.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.frLejz-09xZG80RbY6Vm6YG0DMML0nq_0 2024-03-09T15:36:06.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4076522922570804563', time=2024-03-09T17:24:04.819168+0000, chain.objs.size()=1 2024-03-09T15:36:06.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Xzzc2z38iOfK4IOKgm7p_s7f6ayP6BB_0 2024-03-09T15:36:06.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4748482888222171446', time=2024-03-09T17:24:05.226032+0000, chain.objs.size()=1 2024-03-09T15:36:06.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8KfAIdC-JeiHti-JfgcdkIrpv9svjwX_0 2024-03-09T15:36:06.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16369084364024975307', time=2024-03-09T17:24:05.322915+0000, chain.objs.size()=1 2024-03-09T15:36:06.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.NLGKcWdzQO4wX-JnMRgB_CST9UnZOKm_0 2024-03-09T15:36:06.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.217544888302879478', time=2024-03-09T17:24:06.234813+0000, chain.objs.size()=1 2024-03-09T15:36:06.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HhN1kt08jzthP8P9DT8SISY4ggNZIZW_0 2024-03-09T15:36:06.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17184124566751843227', time=2024-03-09T17:24:08.329847+0000, chain.objs.size()=1 2024-03-09T15:36:06.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.YP-yk7ECg3_Qdwj3ZjCuOxb1CZZS3y3_0 2024-03-09T15:36:06.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2123615766259273029', time=2024-03-09T17:24:08.499875+0000, chain.objs.size()=1 2024-03-09T15:36:06.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.QvTobk3tk-Rz48rgtsYeLM41_fuAdcc_0 2024-03-09T15:36:06.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14555140136660192559', time=2024-03-09T17:24:08.752799+0000, chain.objs.size()=1 2024-03-09T15:36:06.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ASI7wU3anch4PyemGg5PPP-D_M1stjN_0 2024-03-09T15:36:06.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6672286794120297141', time=2024-03-09T17:24:08.804954+0000, chain.objs.size()=1 2024-03-09T15:36:06.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.blSye5j7HfKSG3KwJryc9MudbIPPh-G_0 2024-03-09T15:36:06.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13802872357881037527', time=2024-03-09T17:24:09.313269+0000, chain.objs.size()=1 2024-03-09T15:36:06.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0QVSpdDeP59eXuHtOQDa-2CCn2t1e8Y_0 2024-03-09T15:36:06.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17246843567330820447', time=2024-03-09T17:24:12.665349+0000, chain.objs.size()=1 2024-03-09T15:36:06.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.R-iGEFsk3N11InfcHC_JoqYnugWg6-q_0 2024-03-09T15:36:06.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11176047117087486986', time=2024-03-09T17:24:12.910750+0000, chain.objs.size()=1 2024-03-09T15:36:06.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.hF1PzLyrRgfIAus5YAuJnSf8SUW3rCX_0 2024-03-09T15:36:06.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15586441267234013569', time=2024-03-09T17:24:14.040640+0000, chain.objs.size()=1 2024-03-09T15:36:06.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.400+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.EZ5zoCOf5XM2YrBlWCBNYf4R9OTmx1s_0 2024-03-09T15:36:06.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5423591712312935758', time=2024-03-09T17:24:17.649098+0000, chain.objs.size()=1 2024-03-09T15:36:06.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.GNl0v7k22eu2Z4FlGaHXsXbu0CLgB1B_0 2024-03-09T15:36:06.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6060523029268106298', time=2024-03-09T17:24:18.186106+0000, chain.objs.size()=1 2024-03-09T15:36:06.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.dBoGIgpnalZxNGVhPfMZUlxT5i-Q75c_0 2024-03-09T15:36:06.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8982072211509633051', time=2024-03-09T17:24:18.461725+0000, chain.objs.size()=1 2024-03-09T15:36:06.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.jkW3p0M590Hsj0j295QcfyfXP5uXtgV_0 2024-03-09T15:36:06.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10904051744905336268', time=2024-03-09T17:24:18.526699+0000, chain.objs.size()=1 2024-03-09T15:36:06.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.g_EY71xsITRTKXTrAJs7YauBx5BBKFe_0 2024-03-09T15:36:06.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15280388470615606607', time=2024-03-09T17:24:19.026053+0000, chain.objs.size()=1 2024-03-09T15:36:06.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9-8MCpjIcAe3tH30eZt0r0gRQsQld3e_0 2024-03-09T15:36:06.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.869784990805754157', time=2024-03-09T17:24:19.500644+0000, chain.objs.size()=1 2024-03-09T15:36:06.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.KzlR-IoJr6qGIXLfX5TU548YCKkcyhV_0 2024-03-09T15:36:06.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17238289722983409816', time=2024-03-09T17:24:21.027404+0000, chain.objs.size()=1 2024-03-09T15:36:06.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Sm1kXDPo-1g5cQaT2xVgu2jtnpayz-i_0 2024-03-09T15:36:06.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2899331973636119584', time=2024-03-09T17:24:21.906582+0000, chain.objs.size()=1 2024-03-09T15:36:06.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.VtMccAoUj4tfHFcdDEtyMnAQ3IxrRpR_0 2024-03-09T15:36:06.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5385135096690932417', time=2024-03-09T17:24:22.662938+0000, chain.objs.size()=1 2024-03-09T15:36:06.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.C56McVGT-Y4Drn6DxvAYPqejT4X3eZ9_0 2024-03-09T15:36:06.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13642830648974075444', time=2024-03-09T17:24:24.565887+0000, chain.objs.size()=1 2024-03-09T15:36:06.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.nrfz7nGgk2EBheHrv88-U7LUNdv50nP_0 2024-03-09T15:36:06.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2066429648125254266', time=2024-03-09T17:24:26.653958+0000, chain.objs.size()=1 2024-03-09T15:36:06.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.kQ2q2IflQKV11l87eZAwlfqGXkKdWMf_0 2024-03-09T15:36:06.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14554239631737687010', time=2024-03-09T17:24:27.162951+0000, chain.objs.size()=1 2024-03-09T15:36:06.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Sniru6dLbkxvNyxs5KQvc3AuQ5SLXH4_0 2024-03-09T15:36:06.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.653795618686100629', time=2024-03-09T17:24:27.530243+0000, chain.objs.size()=1 2024-03-09T15:36:06.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_._LvpY-vF2v37NPmoNxwzWSga6nWUY6a_0 2024-03-09T15:36:06.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2058811718835352313', time=2024-03-09T17:24:27.753201+0000, chain.objs.size()=1 2024-03-09T15:36:06.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.gUtFYo0XE_fAt5IMz_wq4L-7X-lByPb_0 2024-03-09T15:36:06.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.328790887901257864', time=2024-03-09T17:24:29.589191+0000, chain.objs.size()=1 2024-03-09T15:36:06.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.404+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.EzZBj4s-vi5VpyjhzPv0qAdc2DxMkDO_0 2024-03-09T15:36:06.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15207308720955480979', time=2024-03-09T17:25:16.339441+0000, chain.objs.size()=1 2024-03-09T15:36:06.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.271__shadow_.-_NUF0_Sfy1_yHKsC5QjFiXaf4ms4Bl_0 2024-03-09T15:36:06.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11412681596483340020', time=2024-03-09T17:25:16.341160+0000, chain.objs.size()=1 2024-03-09T15:36:06.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.271__shadow_.qDE64MlJmmd4aWhuoBBRcNuAz-KAK8L_0 2024-03-09T15:36:06.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10796660738402876637', time=2024-03-09T17:25:39.754327+0000, chain.objs.size()=1 2024-03-09T15:36:06.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.279__shadow_.SrEOuISU4Er3z6fek46sp21A0RcKyQm_0 2024-03-09T15:36:06.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7032816216141006518', time=2024-03-09T17:25:42.668030+0000, chain.objs.size()=1 2024-03-09T15:36:06.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.280__shadow_.AWh68gNqOcx52UuXiKUO-U8ZaGHVwTV_0 2024-03-09T15:36:06.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6790573084041695022', time=2024-03-09T17:25:51.321754+0000, chain.objs.size()=1 2024-03-09T15:36:06.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.283__shadow_._Qu0DaL43rK2PWssHazh0QDqpopitMt_0 2024-03-09T15:36:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14747196258332334412', time=2024-03-09T17:26:01.410486+0000, chain.objs.size()=1 2024-03-09T15:36:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.Pc9BsbDQeQrwd1m3PALzlxDfrIZb7-r_0 2024-03-09T15:36:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3085186592892211918', time=2024-03-09T17:26:01.414027+0000, chain.objs.size()=1 2024-03-09T15:36:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.mYVz-c7DFhOitYnzVitGDYJ6NzksEX3_0 2024-03-09T15:36:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16586432382371309657', time=2024-03-09T17:26:01.759592+0000, chain.objs.size()=1 2024-03-09T15:36:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.FWI5CiDI8a0coWJCYm7JqZEnFAP5Gxf_0 2024-03-09T15:36:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5078628541816774755', time=2024-03-09T17:26:01.761474+0000, chain.objs.size()=1 2024-03-09T15:36:06.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.wlAa119ZDAkPJk7W5sNZHCTBuFiHEvq_0 2024-03-09T15:36:06.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6599432665152781405', time=2024-03-09T17:26:08.085085+0000, chain.objs.size()=1 2024-03-09T15:36:06.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.IO4rB51bKf-l1x2R8q9aV-LALKTaurR_0 2024-03-09T15:36:06.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8998106594510464054', time=2024-03-09T17:26:13.975664+0000, chain.objs.size()=1 2024-03-09T15:36:06.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.288__shadow_.7_zZ0iwSUwtwd5Lk2yGLKybL12Jp21D_0 2024-03-09T15:36:06.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11840463164904530495', time=2024-03-09T17:26:28.909538+0000, chain.objs.size()=1 2024-03-09T15:36:06.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.293__shadow_.a_io24F6MeNiuXSmGtmE5sPV2xl2lJ__0 2024-03-09T15:36:06.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15946353570639512536', time=2024-03-09T17:26:31.801906+0000, chain.objs.size()=1 2024-03-09T15:36:06.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.408+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.294__shadow_.oS9Bm8ZOIZU8OfpgQI0g_4PRLT_vuxg_0 2024-03-09T15:36:06.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.412+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.412+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-03-09T15:36:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=39, truncated=0, next_marker='' 2024-03-09T15:36:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6819613915206933963', time=2024-03-09T17:11:40.699332+0000, chain.objs.size()=1 2024-03-09T15:36:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.135__shadow_.txIvo0nAYyd2_L9w-zfnPS0mdmX87dM_0 2024-03-09T15:36:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11496695082705532122', time=2024-03-09T17:20:37.345742+0000, chain.objs.size()=1 2024-03-09T15:36:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.223__shadow_.zp4qLHyA0A8P-5QnPP3ZHa7Aw0n-I2__0 2024-03-09T15:36:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5359726616512147456', time=2024-03-09T17:20:48.823975+0000, chain.objs.size()=1 2024-03-09T15:36:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.227__shadow_.EPF7rSadLPNTw4YNzdqjYVt6yVaeKC3_0 2024-03-09T15:36:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.99257638448779106', time=2024-03-09T17:21:11.451744+0000, chain.objs.size()=1 2024-03-09T15:36:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.234__shadow_.Ouhdb-huYH6mLBVhFPg6zncMnqr8vBI_0 2024-03-09T15:36:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4125754209396320668', time=2024-03-09T17:21:52.098870+0000, chain.objs.size()=1 2024-03-09T15:36:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.247__shadow_.h-NiIzrXqqd7a7WRIFqU81xaJrhYurp_0 2024-03-09T15:36:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18083217381011673262', time=2024-03-09T17:21:57.701023+0000, chain.objs.size()=1 2024-03-09T15:36:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.249__shadow_._JpDR_42gTQ_VoEW2HmCASkxJ2GPA0o_0 2024-03-09T15:36:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12870165771303570130', time=2024-03-09T17:22:09.506084+0000, chain.objs.size()=1 2024-03-09T15:36:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.253__shadow_.QhYQr4vxrNJReUQ0HXrKVKq2RgczjSA_0 2024-03-09T15:36:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10369506029658968186', time=2024-03-09T17:23:58.853189+0000, chain.objs.size()=1 2024-03-09T15:36:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HnT0zjKbz-X2ssqFKn14CFXGO0ssS3U_0 2024-03-09T15:36:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16554334975156122796', time=2024-03-09T17:23:58.856860+0000, chain.objs.size()=1 2024-03-09T15:36:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.XIg3d3mrzFGQBUgD-P8dCtcIPl0yP1L_0 2024-03-09T15:36:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5971233332002598752', time=2024-03-09T17:23:59.751077+0000, chain.objs.size()=1 2024-03-09T15:36:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.T6FWU2hWZVwyYsBEazYvsWdPEpG9o7y_0 2024-03-09T15:36:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12476321012955693154', time=2024-03-09T17:24:00.633832+0000, chain.objs.size()=1 2024-03-09T15:36:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.OpFgF1Feq02OqiQEhhRRZTZ63dYT9MP_0 2024-03-09T15:36:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2704148161436528716', time=2024-03-09T17:24:01.054634+0000, chain.objs.size()=1 2024-03-09T15:36:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.416+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_._-wKsaRPhkHku29z21pzvD7JRtu3kaQ_0 2024-03-09T15:36:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4307897284291753741', time=2024-03-09T17:24:03.996612+0000, chain.objs.size()=1 2024-03-09T15:36:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ZVpm9yM54VGPUEgN9ANIYt8DO6RTBPq_0 2024-03-09T15:36:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17874458885585258879', time=2024-03-09T17:24:04.766068+0000, chain.objs.size()=1 2024-03-09T15:36:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qU-b4sNZlLpRTWUZ8U1S_LLLMfC6w_P_0 2024-03-09T15:36:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12057727078543123553', time=2024-03-09T17:24:06.695019+0000, chain.objs.size()=1 2024-03-09T15:36:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Us3PoRTjHP-4X04KyHVMLuywK5qVxDx_0 2024-03-09T15:36:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.107409084161272985', time=2024-03-09T17:24:09.503519+0000, chain.objs.size()=1 2024-03-09T15:36:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ls8rymu_t32AU9sJ1AwtQbr1Baz_jEU_0 2024-03-09T15:36:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12523421420981564225', time=2024-03-09T17:24:10.044583+0000, chain.objs.size()=1 2024-03-09T15:36:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.q5KFXYc7VE3bgbddD-mWUyQD2MwjMPm_0 2024-03-09T15:36:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11066497829679635653', time=2024-03-09T17:24:10.046929+0000, chain.objs.size()=1 2024-03-09T15:36:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ilqcIuQTPx0ZSYnxG47qNHNwsVpYEsp_0 2024-03-09T15:36:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.465435437636326539', time=2024-03-09T17:24:12.455072+0000, chain.objs.size()=1 2024-03-09T15:36:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.TY-3ySiIJwUytRmbYrRZP3Mf31x-UDx_0 2024-03-09T15:36:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3896757444212043076', time=2024-03-09T17:24:12.665480+0000, chain.objs.size()=1 2024-03-09T15:36:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8dChVvdHwiaZ4hhHtxbqQsG47Gx4KWK_0 2024-03-09T15:36:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2577627100637560103', time=2024-03-09T17:24:13.159801+0000, chain.objs.size()=1 2024-03-09T15:36:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.CoOSg14rGqdwimWC5tRAZF4PL9g7ix2_0 2024-03-09T15:36:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8899240673029331672', time=2024-03-09T17:24:13.487881+0000, chain.objs.size()=1 2024-03-09T15:36:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.z0SzlXgYpe4QiOP23Lm2ScOtyr56A3n_0 2024-03-09T15:36:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6495459858744085335', time=2024-03-09T17:24:13.804250+0000, chain.objs.size()=1 2024-03-09T15:36:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.KAZLbTcU5iOTSLlVD6rV3IFFcDGJ1w9_0 2024-03-09T15:36:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11120963341767076987', time=2024-03-09T17:24:14.308938+0000, chain.objs.size()=1 2024-03-09T15:36:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.E1ZPjstYRpkFZxMcoXSWjdfvmxbcmv7_0 2024-03-09T15:36:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.554643420362080176', time=2024-03-09T17:24:17.199753+0000, chain.objs.size()=1 2024-03-09T15:36:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.njErarQRlFERhSxgd95S1CaayNyQ08i_0 2024-03-09T15:36:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15731129032953745829', time=2024-03-09T17:24:17.861869+0000, chain.objs.size()=1 2024-03-09T15:36:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.rRN_k0G16lOt5mj6Cr-iaAmc8FjsgoF_0 2024-03-09T15:36:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7543911973769343734', time=2024-03-09T17:24:19.473797+0000, chain.objs.size()=1 2024-03-09T15:36:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.qXw3JyRwRaU2DCTKN_OSpeU4X2xWJH7_0 2024-03-09T15:36:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.214513987768347626', time=2024-03-09T17:24:21.871951+0000, chain.objs.size()=1 2024-03-09T15:36:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.kkSom2BZy4JQGu5xhi4h-rPiatR_pzr_0 2024-03-09T15:36:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11852379166026365940', time=2024-03-09T17:24:23.162351+0000, chain.objs.size()=1 2024-03-09T15:36:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.420+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.R7-bf6BBRpQCDoqBg9msORtQlHRqzsp_0 2024-03-09T15:36:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1008999692705646275', time=2024-03-09T17:24:25.430043+0000, chain.objs.size()=1 2024-03-09T15:36:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0sRhJtTNjvQ8bkmDO_TQeBpWgh3OFiP_0 2024-03-09T15:36:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3405389757478912623', time=2024-03-09T17:24:25.916017+0000, chain.objs.size()=1 2024-03-09T15:36:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HewhADzkW565aXiQpqnRvFuT0fJEARQ_0 2024-03-09T15:36:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5379858509835054504', time=2024-03-09T17:24:26.084425+0000, chain.objs.size()=1 2024-03-09T15:36:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.kteVjBTHmUwYeh4rIDFaHXPUajPvUv-_0 2024-03-09T15:36:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8465472139634863637', time=2024-03-09T17:24:27.000790+0000, chain.objs.size()=1 2024-03-09T15:36:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.DAEsnLwc3YSUvUEwXfoWfxJUNqzBoKe_0 2024-03-09T15:36:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1052850715648397727', time=2024-03-09T17:24:27.725394+0000, chain.objs.size()=1 2024-03-09T15:36:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Cr4CgUXcXNybjEICqUk2SENOEq8Ej0S_0 2024-03-09T15:36:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14608381330821805480', time=2024-03-09T17:24:29.154903+0000, chain.objs.size()=1 2024-03-09T15:36:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Zjb8IeIrXdtZnKra529UBcpqn6CfOwY_0 2024-03-09T15:36:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12781719476923539669', time=2024-03-09T17:24:30.978059+0000, chain.objs.size()=1 2024-03-09T15:36:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.B0LkOqOCMgwglBPxCk4PE5qCD-UKLGf_0 2024-03-09T15:36:06.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.577003246620088334', time=2024-03-09T17:24:56.415805+0000, chain.objs.size()=1 2024-03-09T15:36:06.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.264__shadow_.LHdgOiK67ulhHYkhO4oe08xmB_G-Ay0_0 2024-03-09T15:36:06.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17820023103494072071', time=2024-03-09T17:25:36.643045+0000, chain.objs.size()=1 2024-03-09T15:36:06.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.278__shadow_.NR9KqRe6R6Fr61IUbO3Fsl0iVIrPhM6_0 2024-03-09T15:36:06.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1337770080567562995', time=2024-03-09T17:33:09.889322+0000, chain.objs.size()=1 2024-03-09T15:36:06.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.424+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.327__shadow_.IfeYpEp8BuIahf8FqqADSLFumQ-ddof_0 2024-03-09T15:36:06.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.428+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.428+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-03-09T15:36:06.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=40, truncated=0, next_marker='' 2024-03-09T15:36:06.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16530828972071252730', time=2024-03-09T17:11:42.678540+0000, chain.objs.size()=1 2024-03-09T15:36:06.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.138__shadow_.oEaFJusq8pAvgKn_Y4VQs344Ieh2Dmb_0 2024-03-09T15:36:06.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_5rHANYfiNI0rkQHpCcaL7VzGVnkPGdF', time=2024-03-09T17:13:13.246555+0000, chain.objs.size()=1 2024-03-09T15:36:06.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__shadow_.36MBPCqVxBANSwO6NoGvqsybUrecGQ4_0 2024-03-09T15:36:06.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10148729249811642421', time=2024-03-09T17:21:14.269338+0000, chain.objs.size()=1 2024-03-09T15:36:06.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.235__shadow_.ek0555yJMVE0l3RQy61B2mHPH3_LUgp_0 2024-03-09T15:36:06.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16341941725690264332', time=2024-03-09T17:21:49.148122+0000, chain.objs.size()=1 2024-03-09T15:36:06.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.246__shadow_.WhmDWKXvren5ATwFkYJSmxP9qFyA0gu_0 2024-03-09T15:36:06.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15326910372084412208', time=2024-03-09T17:21:57.718962+0000, chain.objs.size()=1 2024-03-09T15:36:06.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.249__shadow_.6SL0ow4F-Y-OzHip83X-cBx1vj_-zmJ_0 2024-03-09T15:36:06.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1902085095640385737', time=2024-03-09T17:24:00.211562+0000, chain.objs.size()=1 2024-03-09T15:36:06.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.EHvlRAvX04g49lr2S8l1vOrz61PSoHH_0 2024-03-09T15:36:06.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12343492303088284138', time=2024-03-09T17:24:00.213757+0000, chain.objs.size()=1 2024-03-09T15:36:06.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0aEHyBin__gv1IwWfTW5MCLc_VQSwpE_0 2024-03-09T15:36:06.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9222020758702283676', time=2024-03-09T17:24:00.265862+0000, chain.objs.size()=1 2024-03-09T15:36:06.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.46rmo8aI07lajkYJk7xFqd_j2pH1WKh_0 2024-03-09T15:36:06.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12587716066265513364', time=2024-03-09T17:24:00.626817+0000, chain.objs.size()=1 2024-03-09T15:36:06.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0L3u7vKwxVT1KCyQ1_ckLJh-rGUuO8A_0 2024-03-09T15:36:06.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12778693232169989921', time=2024-03-09T17:24:01.195893+0000, chain.objs.size()=1 2024-03-09T15:36:06.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.SSAdVzA4ho1zP1DBoJh5p4bKE_rlPgB_0 2024-03-09T15:36:06.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6958398784153622254', time=2024-03-09T17:24:03.890508+0000, chain.objs.size()=1 2024-03-09T15:36:06.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.rtY6P6wWWtuIw8hT_UOum3JTMF60rKE_0 2024-03-09T15:36:06.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16615563178773026127', time=2024-03-09T17:24:04.318231+0000, chain.objs.size()=1 2024-03-09T15:36:06.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.DMnlMESPo7cBsQcQZukmMYiOv2GOrVr_0 2024-03-09T15:36:06.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8588909052625568263', time=2024-03-09T17:24:04.479781+0000, chain.objs.size()=1 2024-03-09T15:36:06.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PXh4WvR0einZ_x_idPxAJTzaGjtGcgA_0 2024-03-09T15:36:06.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9163142382293694188', time=2024-03-09T17:24:05.830421+0000, chain.objs.size()=1 2024-03-09T15:36:06.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.f88UII86l4YLylyGFXgCx6OQR-C2I5K_0 2024-03-09T15:36:06.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15216650068751543493', time=2024-03-09T17:24:08.477017+0000, chain.objs.size()=1 2024-03-09T15:36:06.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.DOaw7rkMzqbdP-pu_iQ0XlOXKvJhOLn_0 2024-03-09T15:36:06.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16511922002269301841', time=2024-03-09T17:24:12.664260+0000, chain.objs.size()=1 2024-03-09T15:36:06.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.vLdbSHH6XsH36z7_2T78AOegtASCEm3_0 2024-03-09T15:36:06.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7358818888238925154', time=2024-03-09T17:24:13.729789+0000, chain.objs.size()=1 2024-03-09T15:36:06.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2V3HIzywOLxgOzPvPurf9gkaiam2AYt_0 2024-03-09T15:36:06.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10192648120883705955', time=2024-03-09T17:24:15.175199+0000, chain.objs.size()=1 2024-03-09T15:36:06.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.TNwxJHnTdPOw6XuUqsVI8LE2MrlzHzu_0 2024-03-09T15:36:06.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2638384215670057024', time=2024-03-09T17:24:17.509132+0000, chain.objs.size()=1 2024-03-09T15:36:06.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ACoRQL9-JTD3zP473ZAlvwsPErtk9zE_0 2024-03-09T15:36:06.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15653588214029803556', time=2024-03-09T17:24:17.689243+0000, chain.objs.size()=1 2024-03-09T15:36:06.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.HMTrdcbTQoXzNG-cdhVYx4MlSPySSK8_0 2024-03-09T15:36:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1981367359189257441', time=2024-03-09T17:24:18.229304+0000, chain.objs.size()=1 2024-03-09T15:36:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.22IhK_n3TDvQ6aOtHFgFCEpxQnQSQbi_0 2024-03-09T15:36:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2514399745898815564', time=2024-03-09T17:24:18.787985+0000, chain.objs.size()=1 2024-03-09T15:36:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.v6DOYOHwJkfuLTy0INLjhweoCXGXuMc_0 2024-03-09T15:36:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.853728449895265906', time=2024-03-09T17:24:19.112965+0000, chain.objs.size()=1 2024-03-09T15:36:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.432+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.dloGBMSHDcok9B9F9LBz3CZHeTO40E-_0 2024-03-09T15:36:06.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9413314852723647586', time=2024-03-09T17:24:19.608880+0000, chain.objs.size()=1 2024-03-09T15:36:06.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.wZYRqrlnJGhIrxSmjWfjf_d1SAPxVze_0 2024-03-09T15:36:06.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4629144990951377224', time=2024-03-09T17:24:21.799195+0000, chain.objs.size()=1 2024-03-09T15:36:06.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.X_yM1ZO-4pdBD92MMltS0AyPl5HL_4Q_0 2024-03-09T15:36:06.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7094718436397639300', time=2024-03-09T17:24:21.873149+0000, chain.objs.size()=1 2024-03-09T15:36:06.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.O6DVUw_Qp4uXDV3UMDJBpZHCPsZuj1S_0 2024-03-09T15:36:06.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8225150891591094409', time=2024-03-09T17:24:22.656103+0000, chain.objs.size()=1 2024-03-09T15:36:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.FmroWlIlpvQu35tbsEo8gDT8Wc035K5_0 2024-03-09T15:36:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17851629654383199463', time=2024-03-09T17:24:26.082927+0000, chain.objs.size()=1 2024-03-09T15:36:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.AeX9VlLdoZJydBbqCJr8p6e7ejjTQ_F_0 2024-03-09T15:36:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.95989342476653563', time=2024-03-09T17:24:27.159799+0000, chain.objs.size()=1 2024-03-09T15:36:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.frDJ6ZHfbzun4SWJx21FTtl6_iAfzZX_0 2024-03-09T15:36:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5086719677126546582', time=2024-03-09T17:24:39.913159+0000, chain.objs.size()=1 2024-03-09T15:36:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.258__shadow_.T3l9qckBdjRe4XvBj-UH4z4Qw5lDnCU_0 2024-03-09T15:36:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12819159491601031022', time=2024-03-09T17:25:07.567307+0000, chain.objs.size()=1 2024-03-09T15:36:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.268__shadow_.DyjNZrZMkUnKL6edNabvTMymQdD5y3H_0 2024-03-09T15:36:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.658646315399623229', time=2024-03-09T17:26:01.832237+0000, chain.objs.size()=1 2024-03-09T15:36:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.N0ItTKLO-UheEFIwspmRS-_swBOZ_Rk_0 2024-03-09T15:36:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1277383493115743047', time=2024-03-09T17:26:07.937170+0000, chain.objs.size()=1 2024-03-09T15:36:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.IijkGX-l8bXnYHX-1wq7MID7xjiOgfB_0 2024-03-09T15:36:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3577094605771853954', time=2024-03-09T17:26:13.963117+0000, chain.objs.size()=1 2024-03-09T15:36:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.288__shadow_.DTVoqND0pZU6Ie6bBtIGaOPt3VnCjkE_0 2024-03-09T15:36:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10054156626268919706', time=2024-03-09T17:26:28.878884+0000, chain.objs.size()=1 2024-03-09T15:36:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.293__shadow_.LV0lR4rw68K5uybRI5hMrGCtdPiwOBX_0 2024-03-09T15:36:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5527094954862563854', time=2024-03-09T17:26:37.622935+0000, chain.objs.size()=1 2024-03-09T15:36:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.296__shadow_.oiAHkVfLJMZC6jMHTDznlUjYYTGpdUB_0 2024-03-09T15:36:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15516957459837644953', time=2024-03-09T17:26:40.583654+0000, chain.objs.size()=1 2024-03-09T15:36:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.436+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.297__shadow_.0FPVDPEHLdO4xeqtR5fJcoVhOI9uSuR_0 2024-03-09T15:36:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.440+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13099257331452844048', time=2024-03-09T17:26:46.761370+0000, chain.objs.size()=1 2024-03-09T15:36:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.440+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.299__shadow_.4xf2_EcepjXFIHxCPqfsZGP9dGeSZKw_0 2024-03-09T15:36:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.440+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2719473042500128090', time=2024-03-09T17:34:39.731223+0000, chain.objs.size()=1 2024-03-09T15:36:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.440+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.338__shadow_.h1UfiiEN7MDPtHDUsvRczuospPk-5_A_0 2024-03-09T15:36:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.440+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13966342974319038636', time=2024-03-09T17:34:56.715998+0000, chain.objs.size()=1 2024-03-09T15:36:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.440+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.341__shadow_.GTegpBhXnL_jzFWTxM6Z9p5_uD4MrKf_0 2024-03-09T15:36:06.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.440+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-03-09T15:36:06.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=48, truncated=0, next_marker='' 2024-03-09T15:36:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12268059910846733978', time=2024-03-09T17:21:03.532412+0000, chain.objs.size()=1 2024-03-09T15:36:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.231__shadow_.WEr-3-M2a0SRUmQHXcjGyI9Z504joil_0 2024-03-09T15:36:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13278346171073839777', time=2024-03-09T17:21:03.573201+0000, chain.objs.size()=1 2024-03-09T15:36:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.231__shadow_.23F9XW_vCQ1i3UCgFaGcc71gbmrfVh4_0 2024-03-09T15:36:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12015493621661687033', time=2024-03-09T17:21:26.089294+0000, chain.objs.size()=1 2024-03-09T15:36:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.238__shadow_.81RUIwqqHkcTBaO9osYYWC0gzHDkTl6_0 2024-03-09T15:36:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7102917595275197933', time=2024-03-09T17:21:43.463949+0000, chain.objs.size()=1 2024-03-09T15:36:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.244__shadow_.-r81aRJbaLWij2Dqk9YIHAsIvB37nmL_0 2024-03-09T15:36:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11754246618097483929', time=2024-03-09T17:21:46.329375+0000, chain.objs.size()=1 2024-03-09T15:36:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.245__shadow_.lcdWxiP1klDo5ybyTXalI1GybI5TmvJ_0 2024-03-09T15:36:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6411265142811147304', time=2024-03-09T17:21:49.166152+0000, chain.objs.size()=1 2024-03-09T15:36:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.246__shadow_.gahJ9Ywdr00fLj57kVpNLodgbmUNu2P_0 2024-03-09T15:36:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16664007071672379840', time=2024-03-09T17:24:00.939659+0000, chain.objs.size()=1 2024-03-09T15:36:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.GqzSmNdcreGaiboiOy870s0lJIsR8WQ_0 2024-03-09T15:36:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9872721842120429158', time=2024-03-09T17:24:04.379478+0000, chain.objs.size()=1 2024-03-09T15:36:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ymmtn-uqkA0CmMdEWzLyQK7_LrqRfoU_0 2024-03-09T15:36:06.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10179320232703958912', time=2024-03-09T17:24:05.294152+0000, chain.objs.size()=1 2024-03-09T15:36:06.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.TirMfphXnArUyXzH0RgMoOYFwgJuqy1_0 2024-03-09T15:36:06.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12402190121953769486', time=2024-03-09T17:24:05.559636+0000, chain.objs.size()=1 2024-03-09T15:36:06.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3APf-OArk_TAOfuYIvuHNI-JPSYb2Wz_0 2024-03-09T15:36:06.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15457861539485666112', time=2024-03-09T17:24:05.706217+0000, chain.objs.size()=1 2024-03-09T15:36:06.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Lfpas3r4v670wjWitlpSF4Uf9zCdTy4_0 2024-03-09T15:36:06.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5926635367599751681', time=2024-03-09T17:24:06.210390+0000, chain.objs.size()=1 2024-03-09T15:36:06.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.444+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.lf2VGzeoDzcHHvQVUj5j96x_jPPGWNb_0 2024-03-09T15:36:06.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11976719240397636476', time=2024-03-09T17:24:10.431404+0000, chain.objs.size()=1 2024-03-09T15:36:06.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.gnOw59Rc4sxrwoT92gT0MM739bWPzh9_0 2024-03-09T15:36:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6442018659275265906', time=2024-03-09T17:24:10.442062+0000, chain.objs.size()=1 2024-03-09T15:36:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ooOjqW42hiOJ234no9DzwB3UXGN2k-3_0 2024-03-09T15:36:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7304079294963651530', time=2024-03-09T17:24:11.147107+0000, chain.objs.size()=1 2024-03-09T15:36:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.U7DiiNkMO2IK0UyoXZrds6GnKhshj1L_0 2024-03-09T15:36:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10740195720856019350', time=2024-03-09T17:24:12.152248+0000, chain.objs.size()=1 2024-03-09T15:36:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.We95js10R4PqysZVzrfmJhinegdVnTg_0 2024-03-09T15:36:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12310429978821987640', time=2024-03-09T17:24:12.242244+0000, chain.objs.size()=1 2024-03-09T15:36:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iD2XhBmlRPJrdqREgl8uTX7L_oCjZAd_0 2024-03-09T15:36:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2605012191773414778', time=2024-03-09T17:24:14.358225+0000, chain.objs.size()=1 2024-03-09T15:36:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.lMywiDOJKU1iA2rXwoa708xtYeTUaRU_0 2024-03-09T15:36:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.184991064226397909', time=2024-03-09T17:24:14.764715+0000, chain.objs.size()=1 2024-03-09T15:36:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.rMif0EqE-Zx2gHymw11lwjLd3pedTjx_0 2024-03-09T15:36:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3816482857211263743', time=2024-03-09T17:24:15.383620+0000, chain.objs.size()=1 2024-03-09T15:36:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.UThmWiKACm1tjbbKjXHELPiLUxWJHXI_0 2024-03-09T15:36:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15719192991357682988', time=2024-03-09T17:24:15.449977+0000, chain.objs.size()=1 2024-03-09T15:36:06.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Y_tRvrDCvUOMdvBpjtLojENXtUfbwPo_0 2024-03-09T15:36:06.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7825570436560443138', time=2024-03-09T17:24:16.296051+0000, chain.objs.size()=1 2024-03-09T15:36:06.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.z3e1vJaQFOnvysgy8vWNH--jX5vPX5N_0 2024-03-09T15:36:06.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17886100234284124730', time=2024-03-09T17:24:16.756633+0000, chain.objs.size()=1 2024-03-09T15:36:06.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.nMjjCA-lIH3CTDRvvIHi21LJ3JdywWu_0 2024-03-09T15:36:06.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2344508212628497827', time=2024-03-09T17:24:17.848095+0000, chain.objs.size()=1 2024-03-09T15:36:06.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.448+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.GRBdOBOq4LqJ7s5wVlo7ITnw7AUT8xV_0 2024-03-09T15:36:06.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1856713751899332552', time=2024-03-09T17:24:18.590532+0000, chain.objs.size()=1 2024-03-09T15:36:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.egrO6i3dwjMya63qtqKt9Tmvb6fl5OW_0 2024-03-09T15:36:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1221010160953226142', time=2024-03-09T17:24:19.668084+0000, chain.objs.size()=1 2024-03-09T15:36:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sXdNhpynEoDeG7LLYvXzs3ck1vxWjwP_0 2024-03-09T15:36:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12278314425227972929', time=2024-03-09T17:24:20.432077+0000, chain.objs.size()=1 2024-03-09T15:36:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7iUlYjxwiUGo4-mnOwJL1a9CzN-2tgD_0 2024-03-09T15:36:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12646067602830419303', time=2024-03-09T17:24:22.266220+0000, chain.objs.size()=1 2024-03-09T15:36:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.CndSw17oIl46lAs1_Yy-36e0SfoJ9fh_0 2024-03-09T15:36:06.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4961740644060441820', time=2024-03-09T17:24:22.529729+0000, chain.objs.size()=1 2024-03-09T15:36:06.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.nWHy_UxWR8dhh4I2zubFYv5XjY0iA0k_0 2024-03-09T15:36:06.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15761041470127894298', time=2024-03-09T17:24:23.244644+0000, chain.objs.size()=1 2024-03-09T15:36:06.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-35HgXfNxbyb_3FqicC3FeJR0RiHZnO_0 2024-03-09T15:36:06.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17312843078864483068', time=2024-03-09T17:24:23.734875+0000, chain.objs.size()=1 2024-03-09T15:36:06.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.M-3Bc5Judkw_tniZpnOVbFe4x8_XqOW_0 2024-03-09T15:36:06.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5903434526476770174', time=2024-03-09T17:24:24.638025+0000, chain.objs.size()=1 2024-03-09T15:36:06.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.maQS20NmnVQjueW3cWiDmYHqGyYLKis_0 2024-03-09T15:36:06.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13558431108742914373', time=2024-03-09T17:24:27.109387+0000, chain.objs.size()=1 2024-03-09T15:36:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6JRXmlWZe4fqBb6u9A_MALUnlAxFzQK_0 2024-03-09T15:36:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1504171283498901807', time=2024-03-09T17:24:27.238379+0000, chain.objs.size()=1 2024-03-09T15:36:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LGUWcrl5U49YGyT_g2KLk1D4Uu58JeU_0 2024-03-09T15:36:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9678657809672115116', time=2024-03-09T17:24:27.863113+0000, chain.objs.size()=1 2024-03-09T15:36:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.d3Bw2eSwzomcQNG9yIFqLOiC-cPCKgI_0 2024-03-09T15:36:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12122936249780035878', time=2024-03-09T17:24:28.803390+0000, chain.objs.size()=1 2024-03-09T15:36:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.FC9-aDTEOtLiJTkF-cE-H5r6C_xgeYw_0 2024-03-09T15:36:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12676769909998735667', time=2024-03-09T17:24:29.599906+0000, chain.objs.size()=1 2024-03-09T15:36:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3FEMSFgiAaOZmxX_sesiS2_2DR-HRGH_0 2024-03-09T15:36:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5902384707995251182', time=2024-03-09T17:24:29.611024+0000, chain.objs.size()=1 2024-03-09T15:36:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2Eg16Lfcg7B9ZToKzmQneZZ7qg3d0lq_0 2024-03-09T15:36:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18356899999202337403', time=2024-03-09T17:24:30.774049+0000, chain.objs.size()=1 2024-03-09T15:36:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.452+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Nk10bD3QaIhlv-1NFrNTGDYiE3kHZmq_0 2024-03-09T15:36:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11679508371017409189', time=2024-03-09T17:24:59.241162+0000, chain.objs.size()=1 2024-03-09T15:36:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.265__shadow_.zBXXEXInVoerxPHNprNRSutOeRg0LYe_0 2024-03-09T15:36:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9384156113046019874', time=2024-03-09T17:25:10.545838+0000, chain.objs.size()=1 2024-03-09T15:36:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.269__shadow_.UdmSyLqdYG0evQJAxE8bcNTMijcYbhH_0 2024-03-09T15:36:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18325984127408954289', time=2024-03-09T17:25:30.934247+0000, chain.objs.size()=1 2024-03-09T15:36:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.276__shadow_.5o6liTWGUsYKEgmw1DPh8_Y_GM5v-ve_0 2024-03-09T15:36:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14577283905045108034', time=2024-03-09T17:25:33.698568+0000, chain.objs.size()=1 2024-03-09T15:36:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.277__shadow_.13_s5l8ID_JSwJQoKytocsVSyS4GAvb_0 2024-03-09T15:36:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14636435913813935135', time=2024-03-09T17:26:01.401799+0000, chain.objs.size()=1 2024-03-09T15:36:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.rEZSgqO5DlQRmWHsXD5qOaWZXnVwhUD_0 2024-03-09T15:36:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1009537998054231585', time=2024-03-09T17:26:01.764480+0000, chain.objs.size()=1 2024-03-09T15:36:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.epJ11UwGxmZOFSSbYTRWZHBAMvDhOr8_0 2024-03-09T15:36:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8233118640114821862', time=2024-03-09T17:26:08.011975+0000, chain.objs.size()=1 2024-03-09T15:36:06.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.LzkPjrZLrtGq9mDIWhXTVjf3nl5NK9p_0 2024-03-09T15:36:06.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3229551228513748562', time=2024-03-09T17:26:26.014646+0000, chain.objs.size()=1 2024-03-09T15:36:06.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.292__shadow_.b5PoDr0vnnT3wiwCQjBtdF4BiZXatwJ_0 2024-03-09T15:36:06.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6568989284261600510', time=2024-03-09T17:26:50.956352+0000, chain.objs.size()=1 2024-03-09T15:36:06.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.300__shadow:NJNRsnaQl33GiDYn3PXkEnGC8CpDGBS_.YIqVEOXlxi0Ccd8MlUCXZiniEP1KdZq_0 2024-03-09T15:36:06.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.456+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.460+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-03-09T15:36:06.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.460+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.460+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=42, truncated=0, next_marker='' 2024-03-09T15:36:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11581043295528366040', time=2024-03-09T17:21:14.269274+0000, chain.objs.size()=1 2024-03-09T15:36:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.235__shadow_.q-6Rp49gGVLVEsyKU8xNjlCsWs3IzQt_0 2024-03-09T15:36:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1162872848296026024', time=2024-03-09T17:21:23.175392+0000, chain.objs.size()=1 2024-03-09T15:36:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.237__shadow_.RgukwnLIWk7Uk2ZYAAJ1aKBzx-vKLXs_0 2024-03-09T15:36:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16624156384656222989', time=2024-03-09T17:21:31.918500+0000, chain.objs.size()=1 2024-03-09T15:36:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.240__shadow_.l-xZXWkuC1-GgGckncgGDEglUjK0CDj_0 2024-03-09T15:36:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11716083531427415246', time=2024-03-09T17:21:43.465148+0000, chain.objs.size()=1 2024-03-09T15:36:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.244__shadow_.CHCqcIy3P3EkO5AFKqff-JUFN_Kuaal_0 2024-03-09T15:36:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8881232511624545268', time=2024-03-09T17:23:59.438801+0000, chain.objs.size()=1 2024-03-09T15:36:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.5WZxClRVlERT6j1ygFyiKvUFQqnEFSX_0 2024-03-09T15:36:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17305603625054434641', time=2024-03-09T17:24:01.865099+0000, chain.objs.size()=1 2024-03-09T15:36:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.F3q17yBQ7s9q_3qMpfIa2sM8EBO0GF4_0 2024-03-09T15:36:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15659224599554112060', time=2024-03-09T17:24:01.908964+0000, chain.objs.size()=1 2024-03-09T15:36:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.48M8yBwXCLqx8ikPA-IbIGrbQL8BTxc_0 2024-03-09T15:36:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2095014238524235645', time=2024-03-09T17:24:03.497378+0000, chain.objs.size()=1 2024-03-09T15:36:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.T_7eD7L4uJYxUE_5Y9wudUAsZxbMe6k_0 2024-03-09T15:36:06.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8008685899661078641', time=2024-03-09T17:24:04.318146+0000, chain.objs.size()=1 2024-03-09T15:36:06.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.rfmOfzamNK0hSjB4sCUfJBxDkDpcYXc_0 2024-03-09T15:36:06.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8083726615737473878', time=2024-03-09T17:24:04.703768+0000, chain.objs.size()=1 2024-03-09T15:36:06.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_._1-fkCHE7TpyI5RsnquVyMufZdb7Dbk_0 2024-03-09T15:36:06.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15756251801082245052', time=2024-03-09T17:24:04.827105+0000, chain.objs.size()=1 2024-03-09T15:36:06.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.JY-iDjeFyLIAuaE04n5YHxcv3uMDjCd_0 2024-03-09T15:36:06.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11359113997285357464', time=2024-03-09T17:24:05.230776+0000, chain.objs.size()=1 2024-03-09T15:36:06.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.WOPl0KFkFfwqLpSzMFGb_HpJcczMu_M_0 2024-03-09T15:36:06.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17672485982633137285', time=2024-03-09T17:24:06.223773+0000, chain.objs.size()=1 2024-03-09T15:36:06.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.kxXRfFfOopssLTvtd8L-Lj_eXNPrsCJ_0 2024-03-09T15:36:06.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11004387967604185495', time=2024-03-09T17:24:10.187585+0000, chain.objs.size()=1 2024-03-09T15:36:06.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PN8yTRRoWJPtD0T8MOUnQYRCSQFWtdr_0 2024-03-09T15:36:06.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12637600566893837097', time=2024-03-09T17:24:10.901528+0000, chain.objs.size()=1 2024-03-09T15:36:06.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ZsMx5XLpZr2wXHOjrNiXLtRbmIv0rzx_0 2024-03-09T15:36:06.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3987368979546450218', time=2024-03-09T17:24:11.029674+0000, chain.objs.size()=1 2024-03-09T15:36:06.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.FyXj5xcu4NFxiqNK7XnG5ThbTMWIYtZ_0 2024-03-09T15:36:06.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8733667589222729138', time=2024-03-09T17:24:14.964821+0000, chain.objs.size()=1 2024-03-09T15:36:06.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.SFo9nfcP--9XAPKGR4MfEIVPZvmMQEG_0 2024-03-09T15:36:06.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16953216936336864516', time=2024-03-09T17:24:15.453378+0000, chain.objs.size()=1 2024-03-09T15:36:06.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Yw9krdGz4ZIaCckapu7nzewFp9MsxCw_0 2024-03-09T15:36:06.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15836825846283709776', time=2024-03-09T17:24:16.810428+0000, chain.objs.size()=1 2024-03-09T15:36:06.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.TOS7AHuJfYC9OEhl32Lw2BEJTTCbMCH_0 2024-03-09T15:36:06.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17403639952078002728', time=2024-03-09T17:24:17.044770+0000, chain.objs.size()=1 2024-03-09T15:36:06.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.TRSfM_XDBt6cVOtp9mRIb7EuDYra3Na_0 2024-03-09T15:36:06.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17580737240581260416', time=2024-03-09T17:24:17.688989+0000, chain.objs.size()=1 2024-03-09T15:36:06.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.IiJYDxzRCdPc4gEbxZ72kRYaUFUzmK__0 2024-03-09T15:36:06.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7608152194949295688', time=2024-03-09T17:24:17.968442+0000, chain.objs.size()=1 2024-03-09T15:36:06.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.zcloJhLVov3kY5FoaAYUsQxfRALkwCD_0 2024-03-09T15:36:06.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12337992352562772436', time=2024-03-09T17:24:18.983952+0000, chain.objs.size()=1 2024-03-09T15:36:06.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.464+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Np4LVKG4EfucMl1RuUqFdPTDbgAmS9H_0 2024-03-09T15:36:06.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6642619706725420275', time=2024-03-09T17:24:19.209619+0000, chain.objs.size()=1 2024-03-09T15:36:06.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ALCaakRsU5XyXxb6PzbCQY2qZIo65VR_0 2024-03-09T15:36:06.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8148831523679875794', time=2024-03-09T17:24:19.669001+0000, chain.objs.size()=1 2024-03-09T15:36:06.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.XHsDem3LGFt4E_uqY-JCxTJRMbwyZqJ_0 2024-03-09T15:36:06.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2280581691872235560', time=2024-03-09T17:24:21.419468+0000, chain.objs.size()=1 2024-03-09T15:36:06.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.nlg7xY832lvAOPglVKivGNbXKSSEb2l_0 2024-03-09T15:36:06.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8321690331341303899', time=2024-03-09T17:24:23.479775+0000, chain.objs.size()=1 2024-03-09T15:36:06.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.MKN5Zg8k-IYYFmcCP1ayyY1mZ6zAz8w_0 2024-03-09T15:36:06.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9602732138723977697', time=2024-03-09T17:24:24.569829+0000, chain.objs.size()=1 2024-03-09T15:36:06.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sffpqvL9OH55TOW0VzwvafUF0cT4MEU_0 2024-03-09T15:36:06.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2991574539859681968', time=2024-03-09T17:24:25.513909+0000, chain.objs.size()=1 2024-03-09T15:36:06.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6a3caMfRplr9SHiTyj8NM2ZXaAfGFDB_0 2024-03-09T15:36:06.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1046922809085078120', time=2024-03-09T17:24:26.325335+0000, chain.objs.size()=1 2024-03-09T15:36:06.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2rTQKah8uh2eSksc128jAryATWvFavS_0 2024-03-09T15:36:06.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12554963991389678582', time=2024-03-09T17:24:29.671749+0000, chain.objs.size()=1 2024-03-09T15:36:06.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.enKMAovgd5pm7TpOUjj3SGixtlNjUPC_0 2024-03-09T15:36:06.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16327626253592143043', time=2024-03-09T17:24:30.228166+0000, chain.objs.size()=1 2024-03-09T15:36:06.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.7SVU2FKLGJRKshntZd5CJ2M86hubM_m_0 2024-03-09T15:36:06.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13309496485491676901', time=2024-03-09T17:24:30.741915+0000, chain.objs.size()=1 2024-03-09T15:36:06.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.i5OGWohGNM8wVaqa1If7-utyp0v1gX0_0 2024-03-09T15:36:06.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14391569124576030507', time=2024-03-09T17:25:19.291211+0000, chain.objs.size()=1 2024-03-09T15:36:06.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.272__shadow_.6vNzxr5ku8OG-cYaGLl4soqtfRq9Opc_0 2024-03-09T15:36:06.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17484751454365373827', time=2024-03-09T17:25:30.933252+0000, chain.objs.size()=1 2024-03-09T15:36:06.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.276__shadow_.jISKCDoSbmigolZMfUOQk_lMhiUxTK6_0 2024-03-09T15:36:06.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17534519017151530351', time=2024-03-09T17:26:01.784958+0000, chain.objs.size()=1 2024-03-09T15:36:06.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.NVFxXo5KdKJuOfFj_dVPPiRaHCKu4Xh_0 2024-03-09T15:36:06.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8641320922914690773', time=2024-03-09T17:26:11.106867+0000, chain.objs.size()=1 2024-03-09T15:36:06.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.287__shadow_.9l1-CcXzJV10o1nbVpxALUblIOH7s9g_0 2024-03-09T15:36:06.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7085259068419164459', time=2024-03-09T17:26:22.927142+0000, chain.objs.size()=1 2024-03-09T15:36:06.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.291__shadow_.4Gg63JL73jkOR0NzXXvrfUFmUpKEcIa_0 2024-03-09T15:36:06.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10233883312179498201', time=2024-03-09T17:26:22.928916+0000, chain.objs.size()=1 2024-03-09T15:36:06.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.291__shadow_.vksYDk-i7n9w7pvQSrLezivw_z8OMJN_0 2024-03-09T15:36:06.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13931022396558745121', time=2024-03-09T17:26:26.008012+0000, chain.objs.size()=1 2024-03-09T15:36:06.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.292__shadow_.-eFUKFZ2-qcrFrHO3eOuOZhP0zaAnCE_0 2024-03-09T15:36:06.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18335307086158637764', time=2024-03-09T17:26:50.969295+0000, chain.objs.size()=1 2024-03-09T15:36:06.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.300__shadow:2zaRFktO5bWSuCEEaJg4R1pPe2n.OCX_.OR1azi19KIByuTQDRwFHQcT5yFqlrH__0 2024-03-09T15:36:06.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16637393693414346139', time=2024-03-09T17:30:13.358904+0000, chain.objs.size()=1 2024-03-09T15:36:06.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.468+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.308__shadow_.ESsPy60uM4Goy0itWpqW6_y0v8JkHQK_0 2024-03-09T15:36:06.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.472+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.472+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-03-09T15:36:06.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=54, truncated=0, next_marker='' 2024-03-09T15:36:06.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_bwkH3HRclMwG2ddYeBEFByvMO5aKmB3', time=2024-03-09T17:13:01.673705+0000, chain.objs.size()=2 2024-03-09T15:36:06.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.b1YsfRujLOq39GXj_JGNB3XPN14E-Rn_0 2024-03-09T15:36:06.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.b1YsfRujLOq39GXj_JGNB3XPN14E-Rn_1 2024-03-09T15:36:06.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='__1mZAmn7p1APtplfE7tNKMuPgzrv59Q', time=2024-03-09T17:13:03.364339+0000, chain.objs.size()=2 2024-03-09T15:36:06.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.KH-e3rwBuIEGVCxVGQ3lbMpHSXID8mf_0 2024-03-09T15:36:06.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.KH-e3rwBuIEGVCxVGQ3lbMpHSXID8mf_1 2024-03-09T15:36:06.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_ZVXaMYw4HBH1qYaz7dQg9xpk-nqbSz_', time=2024-03-09T17:13:25.327652+0000, chain.objs.size()=2 2024-03-09T15:36:06.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__shadow_.mWTjUtm3FrJ_eAqaP8N3Wo3DPo9Z0bU_0 2024-03-09T15:36:06.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.154__shadow_.mWTjUtm3FrJ_eAqaP8N3Wo3DPo9Z0bU_1 2024-03-09T15:36:06.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_qnaIx9oaRbtd97RaQcOCXtlUhVc5v26', time=2024-03-09T17:13:51.588087+0000, chain.objs.size()=2 2024-03-09T15:36:06.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.156__shadow_.bKAvJlLvrpiCV4UUx2PdA6SEe9HkptR_0 2024-03-09T15:36:06.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:8c209108-e1cd-4d45-9608-6f491550b24c.4644.156__shadow_.bKAvJlLvrpiCV4UUx2PdA6SEe9HkptR_1 2024-03-09T15:36:06.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='2~VVnZ_C0XnyZ6dce9-PsKDQUlowJwfG0', time=2024-03-09T17:16:27.554345+0000, chain.objs.size()=1 2024-03-09T15:36:06.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.195__multipart_mymultipart.2~VVnZ_C0XnyZ6dce9-PsKDQUlowJwfG0.1 2024-03-09T15:36:06.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17097957826922803090', time=2024-03-09T17:20:51.684177+0000, chain.objs.size()=1 2024-03-09T15:36:06.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.228__shadow_.LxW1MA_3Gpr58HXuuEEimC03oVMpMJL_0 2024-03-09T15:36:06.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13132493549568322221', time=2024-03-09T17:20:59.142696+0000, chain.objs.size()=1 2024-03-09T15:36:06.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.230__shadow_.mVYKO-kFazbk_FYQkgrJuAbElz5iFgX_0 2024-03-09T15:36:06.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17752180244143950651', time=2024-03-09T17:22:00.536443+0000, chain.objs.size()=1 2024-03-09T15:36:06.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.476+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.250__shadow_.ytRZYmE-G_aDe4m5Nq2ju9s5zTEUe5p_0 2024-03-09T15:36:06.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.480+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16125903682620327206', time=2024-03-09T17:23:58.785507+0000, chain.objs.size()=1 2024-03-09T15:36:06.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.480+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.u-JtvEwxDd--imZiUcT_VgW8g7Nagna_0 2024-03-09T15:36:06.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.480+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16939982264709297650', time=2024-03-09T17:23:59.218143+0000, chain.objs.size()=1 2024-03-09T15:36:06.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.480+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.NJGYwH9U0-1pE-G1B6_YDhaPYPQY8bL_0 2024-03-09T15:36:06.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17319063262386973234', time=2024-03-09T17:23:59.266209+0000, chain.objs.size()=1 2024-03-09T15:36:06.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.LOINPDpklJkh_Pc299GLEXZCHXSm6R5_0 2024-03-09T15:36:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5784048473655817880', time=2024-03-09T17:24:00.400659+0000, chain.objs.size()=1 2024-03-09T15:36:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.c6NX_Pf8_oACrXixhbzvt4mIFzecbZE_0 2024-03-09T15:36:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11254350685222359411', time=2024-03-09T17:24:00.630334+0000, chain.objs.size()=1 2024-03-09T15:36:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sBzDRYZCHJUCy9Q83AZOF1Dg-ZiFLey_0 2024-03-09T15:36:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16756448197935343863', time=2024-03-09T17:24:01.015275+0000, chain.objs.size()=1 2024-03-09T15:36:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.MEldlXF-LXQ0c1dGh_dyWH2QfzHubhi_0 2024-03-09T15:36:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17141932513679140547', time=2024-03-09T17:24:01.854246+0000, chain.objs.size()=1 2024-03-09T15:36:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.vWCKg6D_aZ-JFPycjzO6snjvmBSFuB9_0 2024-03-09T15:36:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5853773715938132280', time=2024-03-09T17:24:03.460812+0000, chain.objs.size()=1 2024-03-09T15:36:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.YwnuOKvhHI3nzks1tHc1atWYqkvYDJB_0 2024-03-09T15:36:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8239797396589061548', time=2024-03-09T17:24:03.902251+0000, chain.objs.size()=1 2024-03-09T15:36:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.tT4aZxUl5gTVM2It2zXoMaa3jd1rTOL_0 2024-03-09T15:36:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9720939572428741418', time=2024-03-09T17:24:03.971694+0000, chain.objs.size()=1 2024-03-09T15:36:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.r2Gf3wWd52accqgEmbZU6xv2_EH3f9x_0 2024-03-09T15:36:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10530010449751052720', time=2024-03-09T17:24:04.352278+0000, chain.objs.size()=1 2024-03-09T15:36:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.OiesafLsb5UTylKsKRpqbMOzmMjzb7O_0 2024-03-09T15:36:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3299841690767842527', time=2024-03-09T17:24:04.525352+0000, chain.objs.size()=1 2024-03-09T15:36:06.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1qebHMg5tKQhoSg-zqb2AFyH-bQHT2M_0 2024-03-09T15:36:06.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10429246215179383810', time=2024-03-09T17:24:04.761990+0000, chain.objs.size()=1 2024-03-09T15:36:06.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.bH6ICRpFpdRqzyIRc4B2of4TtsHGf6J_0 2024-03-09T15:36:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9108009910253011961', time=2024-03-09T17:24:04.854388+0000, chain.objs.size()=1 2024-03-09T15:36:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3mWlcU9JsOCICGgXRxlbz6E4SyJPtkk_0 2024-03-09T15:36:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18086954615920234956', time=2024-03-09T17:24:05.927081+0000, chain.objs.size()=1 2024-03-09T15:36:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.T_oQ2Dj_W6I0Nx7LrHAbn26HLfWbdCa_0 2024-03-09T15:36:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7755908702309025307', time=2024-03-09T17:24:06.120677+0000, chain.objs.size()=1 2024-03-09T15:36:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.wfo8dxlKkTiRSiV0rThnNdb6Yc505LI_0 2024-03-09T15:36:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13601843451030612444', time=2024-03-09T17:24:07.884083+0000, chain.objs.size()=1 2024-03-09T15:36:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.nyMIyskSpuHIexI4-8MeR-NJ4RVsXU1_0 2024-03-09T15:36:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11800465930309960471', time=2024-03-09T17:24:08.892731+0000, chain.objs.size()=1 2024-03-09T15:36:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.SDMSmYve6VBt1V4lNBZERjPNLJxUq89_0 2024-03-09T15:36:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2532852141394843613', time=2024-03-09T17:24:11.019975+0000, chain.objs.size()=1 2024-03-09T15:36:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.SBPv-9bBZTlYOcZid7RqW-Tgz4mQtFc_0 2024-03-09T15:36:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16151544462218147695', time=2024-03-09T17:24:11.165691+0000, chain.objs.size()=1 2024-03-09T15:36:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ZM0bFQ4RGzXLooYCZ0yQFdC2nPtt4mP_0 2024-03-09T15:36:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1494347856211131235', time=2024-03-09T17:24:12.170949+0000, chain.objs.size()=1 2024-03-09T15:36:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.akGulyl8jl81dFkF3bcuma88jjkP78N_0 2024-03-09T15:36:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5223404733015159743', time=2024-03-09T17:24:14.306720+0000, chain.objs.size()=1 2024-03-09T15:36:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.lTFScyZcJOURKpkEvV969lWcM7ebbw9_0 2024-03-09T15:36:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3064998834492520528', time=2024-03-09T17:24:14.308680+0000, chain.objs.size()=1 2024-03-09T15:36:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.9zJOKzwditXir7SpZjGwlHl16YZ3BCx_0 2024-03-09T15:36:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3090926805768422066', time=2024-03-09T17:24:16.323923+0000, chain.objs.size()=1 2024-03-09T15:36:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1H9ZB5D_p-10hPIO6-x5_PRCaZSyXDi_0 2024-03-09T15:36:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2533338021962370287', time=2024-03-09T17:24:16.325421+0000, chain.objs.size()=1 2024-03-09T15:36:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6kZn5Z9SLk_69c9M86Ef-kVSFz81158_0 2024-03-09T15:36:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8835423034841054217', time=2024-03-09T17:24:17.109198+0000, chain.objs.size()=1 2024-03-09T15:36:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.484+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Lk_0WlmEYCgtg4Ue3SuS34XmI4P4RVr_0 2024-03-09T15:36:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13221355340188532281', time=2024-03-09T17:24:18.935333+0000, chain.objs.size()=1 2024-03-09T15:36:06.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.MCyPHofLr-YTiYhp6cgtBmORKfbR29n_0 2024-03-09T15:36:06.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.265716027169697661', time=2024-03-09T17:24:19.558018+0000, chain.objs.size()=1 2024-03-09T15:36:06.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.shZ3rJd7dqCyO5EevN2jJOcTOboK85I_0 2024-03-09T15:36:06.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5290377057959097682', time=2024-03-09T17:24:20.988649+0000, chain.objs.size()=1 2024-03-09T15:36:06.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.ANCElqTWm5oPDdlJd8Qc8pkmyqWgXWq_0 2024-03-09T15:36:06.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17303170517778536652', time=2024-03-09T17:24:21.413402+0000, chain.objs.size()=1 2024-03-09T15:36:06.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.xAn_D-C-G0E8RK0u_xHDzI9jYlLX0nX_0 2024-03-09T15:36:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1346719612610957947', time=2024-03-09T17:24:21.777253+0000, chain.objs.size()=1 2024-03-09T15:36:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.kj5JyAZ9fQjsgjxY2C3AzhnAt-ks2uV_0 2024-03-09T15:36:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8029269886926885314', time=2024-03-09T17:24:24.727548+0000, chain.objs.size()=1 2024-03-09T15:36:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1e9iIdRpQ2NjX6P_x77zGZ2ZT51DyRp_0 2024-03-09T15:36:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7768830307025552927', time=2024-03-09T17:24:26.277101+0000, chain.objs.size()=1 2024-03-09T15:36:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.gQ7U_MzWHc7vDLzEX88nGuOif0zDHxD_0 2024-03-09T15:36:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15563202591448067914', time=2024-03-09T17:24:26.508862+0000, chain.objs.size()=1 2024-03-09T15:36:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6YC-3dwieC-JRKOMys4jnHsUhlkDp6a_0 2024-03-09T15:36:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14881393282473463656', time=2024-03-09T17:24:27.105567+0000, chain.objs.size()=1 2024-03-09T15:36:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.feXFh9C050iyIzpa2WagoaxvuV9n-YQ_0 2024-03-09T15:36:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.774542735819243823', time=2024-03-09T17:24:30.030126+0000, chain.objs.size()=1 2024-03-09T15:36:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1tSD5IwO5_LW2zaNt4feYfwnTqnwwNf_0 2024-03-09T15:36:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9242742630529692927', time=2024-03-09T17:24:30.304194+0000, chain.objs.size()=1 2024-03-09T15:36:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.q7C_1eNmCKdE6b3PPnEK62SDSgnSPIl_0 2024-03-09T15:36:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7718159902836381184', time=2024-03-09T17:24:30.306470+0000, chain.objs.size()=1 2024-03-09T15:36:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8mW-pOWOEtrvxf8Gc7U9rG2r28-Dl7B_0 2024-03-09T15:36:06.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17781151071089422698', time=2024-03-09T17:24:31.411103+0000, chain.objs.size()=1 2024-03-09T15:36:06.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.zJg0l4eVqUanzOM70afUsT48rs-lGGj_0 2024-03-09T15:36:06.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1264479465690440038', time=2024-03-09T17:24:34.306137+0000, chain.objs.size()=1 2024-03-09T15:36:06.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.256__shadow_.F5Kqgyfk5aNT4HQwGNkWDY1zSG-JcDA_0 2024-03-09T15:36:06.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1156825126239915382', time=2024-03-09T17:24:34.317852+0000, chain.objs.size()=1 2024-03-09T15:36:06.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.256__shadow_.T3llXTJ-RMM2lDh-77G5pQ3jow55WJN_0 2024-03-09T15:36:06.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13346798561920071158', time=2024-03-09T17:25:22.133778+0000, chain.objs.size()=1 2024-03-09T15:36:06.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.273__shadow_.U7QFNeLJTp1hQ0Zn_GGxvv-1hD-g2du_0 2024-03-09T15:36:06.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15108527138259656072', time=2024-03-09T17:25:45.489225+0000, chain.objs.size()=1 2024-03-09T15:36:06.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.488+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.281__shadow_.JCfXtRifqP6OL9I45i5vbrSrdeYpCkA_0 2024-03-09T15:36:06.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.492+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5805235290068820701', time=2024-03-09T17:25:45.503562+0000, chain.objs.size()=1 2024-03-09T15:36:06.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.492+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.281__shadow_.gQjBjQmh2pVyHBwlnQZ3DL7GsLc8Nx8_0 2024-03-09T15:36:06.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.492+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16517050963501539175', time=2024-03-09T17:26:07.740023+0000, chain.objs.size()=1 2024-03-09T15:36:06.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.492+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.4yOMpmyky6MJK4A_CrrSerTYF2wPi9h_0 2024-03-09T15:36:06.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.492+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16731762877796637999', time=2024-03-09T17:26:07.742280+0000, chain.objs.size()=1 2024-03-09T15:36:06.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.492+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.1fNshnif7OFlCZxQXgG0eq8SA6zfHA__0 2024-03-09T15:36:06.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.492+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-03-09T15:36:06.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=40, truncated=0, next_marker='' 2024-03-09T15:36:06.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8318571086933136533', time=2024-03-09T17:20:40.182245+0000, chain.objs.size()=1 2024-03-09T15:36:06.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.224__shadow_.M90iOqIWH1R86DkRICuueyetODrsM58_0 2024-03-09T15:36:06.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9082436227159361810', time=2024-03-09T17:21:26.112541+0000, chain.objs.size()=1 2024-03-09T15:36:06.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.238__shadow_._oUezPTJPHlB7lJ2h5zC0DpGgYBOu8b_0 2024-03-09T15:36:06.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1167726285220109846', time=2024-03-09T17:21:54.836129+0000, chain.objs.size()=1 2024-03-09T15:36:06.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.248__shadow_.ZGhhixtA4VkpqFwyzlilvHIeNi778mf_0 2024-03-09T15:36:06.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9660280151486852373', time=2024-03-09T17:22:03.285784+0000, chain.objs.size()=1 2024-03-09T15:36:06.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.251__shadow_.rOK056UQ1ZUB-707gLiAnw0BbGPAyV0_0 2024-03-09T15:36:06.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3741830542361815798', time=2024-03-09T17:23:59.262872+0000, chain.objs.size()=1 2024-03-09T15:36:06.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.a5kMa5iQjEiljn9tDouJWCDArCyjV0u_0 2024-03-09T15:36:06.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13472155018190807476', time=2024-03-09T17:23:59.755605+0000, chain.objs.size()=1 2024-03-09T15:36:06.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.wIbQZDznWVZhnML9dZDREHJ1JL3qNKd_0 2024-03-09T15:36:06.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3663862673645273592', time=2024-03-09T17:24:00.211813+0000, chain.objs.size()=1 2024-03-09T15:36:06.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.GXZrsvCAk-sUiKCwgm9SSIoWiLgvBKb_0 2024-03-09T15:36:06.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2440102150879210059', time=2024-03-09T17:24:01.490086+0000, chain.objs.size()=1 2024-03-09T15:36:06.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.zXgPi-MAhxwGBHt39P1VUmXRdVTIaoK_0 2024-03-09T15:36:06.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1743820411145875195', time=2024-03-09T17:24:03.890440+0000, chain.objs.size()=1 2024-03-09T15:36:06.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.6NK9gOF6C1C8eOsCFr3My3aajOYVXIb_0 2024-03-09T15:36:06.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1471024504482306984', time=2024-03-09T17:24:03.892134+0000, chain.objs.size()=1 2024-03-09T15:36:06.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.3hF-DXHexxOrVJcjxPMRGeBWb3H7Ic0_0 2024-03-09T15:36:06.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12036093161125774332', time=2024-03-09T17:24:05.566965+0000, chain.objs.size()=1 2024-03-09T15:36:06.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.wr4R5MjZPGiwEuD94lizCKnCAktUZK7_0 2024-03-09T15:36:06.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18279921818954630189', time=2024-03-09T17:24:05.763619+0000, chain.objs.size()=1 2024-03-09T15:36:06.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.496+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.RCGEK1NWdrs3ezeJQH0Ms-hOtT_o5WB_0 2024-03-09T15:36:06.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.682741367121282542', time=2024-03-09T17:24:06.223712+0000, chain.objs.size()=1 2024-03-09T15:36:06.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.FpJ6646DLOOxbRed4AlWURH3Sp1Bh7v_0 2024-03-09T15:36:06.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4895564784084562939', time=2024-03-09T17:24:06.642548+0000, chain.objs.size()=1 2024-03-09T15:36:06.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.jMQ2ZEvIdUTuHNhQtP39sFLESYWQtRz_0 2024-03-09T15:36:06.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15117255546476372908', time=2024-03-09T17:24:09.458033+0000, chain.objs.size()=1 2024-03-09T15:36:06.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.i1znOZWLvVfDsAmwd9el3XzGTajElHG_0 2024-03-09T15:36:06.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17193296590579671244', time=2024-03-09T17:24:10.186130+0000, chain.objs.size()=1 2024-03-09T15:36:06.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.xdo9BxcfmwLb896hgLH4UQYqqTI464a_0 2024-03-09T15:36:06.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9698789434557465539', time=2024-03-09T17:24:10.593782+0000, chain.objs.size()=1 2024-03-09T15:36:06.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.5Kfe-1UfqD8oKGNvQemXaVXitHJcDkO_0 2024-03-09T15:36:06.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4360527518869455456', time=2024-03-09T17:24:15.200203+0000, chain.objs.size()=1 2024-03-09T15:36:06.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.H6_AQdnCMHguTZ1gydevDw-zCwYXzBe_0 2024-03-09T15:36:06.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10598828884197032016', time=2024-03-09T17:24:16.340845+0000, chain.objs.size()=1 2024-03-09T15:36:06.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.lI00zLpUfE-FrKIUPIeroR37z9r9kgQ_0 2024-03-09T15:36:06.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4122293662741124400', time=2024-03-09T17:24:16.810601+0000, chain.objs.size()=1 2024-03-09T15:36:06.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.aBlki7t1234584JOs1NXjVtPR5zqE8g_0 2024-03-09T15:36:06.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12320195412860041753', time=2024-03-09T17:24:17.104784+0000, chain.objs.size()=1 2024-03-09T15:36:06.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Hrro8NQzkGIPtEmni5LKg36REuLIslU_0 2024-03-09T15:36:06.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10504658154626619598', time=2024-03-09T17:24:17.291307+0000, chain.objs.size()=1 2024-03-09T15:36:06.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.UV5Jv8rgbvt9OkKya1eTJb3zfVOnatA_0 2024-03-09T15:36:06.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7071309504628245740', time=2024-03-09T17:24:17.688905+0000, chain.objs.size()=1 2024-03-09T15:36:06.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.5IUHW95h9K6xM2L4dOB9tiJBLy_tRqV_0 2024-03-09T15:36:06.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11244407367279281949', time=2024-03-09T17:24:21.461235+0000, chain.objs.size()=1 2024-03-09T15:36:06.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.500+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Duhqs4ZdnRXZBx99ECgt2SGo4ytWT4F_0 2024-03-09T15:36:06.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5771701009627951587', time=2024-03-09T17:24:21.873267+0000, chain.objs.size()=1 2024-03-09T15:36:06.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.DWNlWHpTnhg72lkGntz1bMx06jHfccy_0 2024-03-09T15:36:06.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7053193233755637604', time=2024-03-09T17:24:22.259832+0000, chain.objs.size()=1 2024-03-09T15:36:06.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.cbmO5xKYbN3iM98wvSqWs9IghZG2u2q_0 2024-03-09T15:36:06.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11283749863345705909', time=2024-03-09T17:24:22.261597+0000, chain.objs.size()=1 2024-03-09T15:36:06.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.sgV-Oqb1uaOwYnd4GqdvtquUb1gVkAP_0 2024-03-09T15:36:06.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11564475490594996655', time=2024-03-09T17:24:25.015596+0000, chain.objs.size()=1 2024-03-09T15:36:06.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.MswWgPIyX2NGu4nU1P_lX9O71Z70adp_0 2024-03-09T15:36:06.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3489592745583595610', time=2024-03-09T17:24:25.143799+0000, chain.objs.size()=1 2024-03-09T15:36:06.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.XVcoDg0c2VnQhS8oOzQcsFZsJn9F8Vy_0 2024-03-09T15:36:06.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12219323019428040692', time=2024-03-09T17:24:27.520699+0000, chain.objs.size()=1 2024-03-09T15:36:06.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2F65IsdDiZUp6e1Uos7xBoT1xh4WTFZ_0 2024-03-09T15:36:06.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9087071907537511432', time=2024-03-09T17:24:29.197369+0000, chain.objs.size()=1 2024-03-09T15:36:06.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.NCTXJ-ZjfzLokrv5MPouLnfbbjVEzIZ_0 2024-03-09T15:36:06.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15803846733364982928', time=2024-03-09T17:24:29.199469+0000, chain.objs.size()=1 2024-03-09T15:36:06.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-d8nqtVFFCZwgkMtUxyxkZn85unf_KW_0 2024-03-09T15:36:06.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6402424523400743506', time=2024-03-09T17:24:29.671901+0000, chain.objs.size()=1 2024-03-09T15:36:06.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.emrAJl7W87C5f-HnjPth3egoMrs4zHh_0 2024-03-09T15:36:06.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1798444836119354882', time=2024-03-09T17:25:33.701895+0000, chain.objs.size()=1 2024-03-09T15:36:06.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.277__shadow_.6lS21nOBo74YKfWsxDvqLuu2O1gUZ3A_0 2024-03-09T15:36:06.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.763944430237967749', time=2024-03-09T17:25:39.729998+0000, chain.objs.size()=1 2024-03-09T15:36:06.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.279__shadow_.DoLcHsWPFy-elnqjfGaMpIrknmEc43I_0 2024-03-09T15:36:06.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.10074878867466406872', time=2024-03-09T17:26:31.798021+0000, chain.objs.size()=1 2024-03-09T15:36:06.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.294__shadow_.X60zJ2F2EJFL9X5EyHSb3eO7nZmkH_h_0 2024-03-09T15:36:06.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12169663617649699966', time=2024-03-09T17:26:40.573346+0000, chain.objs.size()=1 2024-03-09T15:36:06.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.297__shadow_.nmwh4Tko_LvaDzJrwj25VhLV3ipIz7f_0 2024-03-09T15:36:06.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8133836165823303811', time=2024-03-09T17:30:41.185026+0000, chain.objs.size()=1 2024-03-09T15:36:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.310__shadow_.7mo46B2REZ2qNEycRDtAaG5Mpt8i8vO_0 2024-03-09T15:36:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15718739288936944066', time=2024-03-09T17:31:28.617994+0000, chain.objs.size()=1 2024-03-09T15:36:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.315__shadow_.yr34pypmnym_sYqJoZUoPTTROlZqO_e_0 2024-03-09T15:36:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4277174730489166493', time=2024-03-09T17:34:52.818238+0000, chain.objs.size()=1 2024-03-09T15:36:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.504+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.340__shadow_.E_eX56jRVOEnMB4L3jrnLpsYm0eBW2p_0 2024-03-09T15:36:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.508+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.508+0000 7f471c3a7980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-03-09T15:36:06.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-03-09T15:36:06.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-03-09T15:36:06.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=49, truncated=0, next_marker='' 2024-03-09T15:36:06.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='_IbeX5SM5no6bDW-0ArLumZ4EuTuTaYF', time=2024-03-09T17:12:42.472144+0000, chain.objs.size()=1 2024-03-09T15:36:06.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:8c209108-e1cd-4d45-9608-6f491550b24c.4644.153__shadow_.9NIk-JUlMDbx4uOq4m5nnH-0k_rIoP8_0 2024-03-09T15:36:06.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1141132075551119623', time=2024-03-09T17:13:51.254858+0000, chain.objs.size()=2 2024-03-09T15:36:06.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.155__shadow_.dTGpVXE5LuysoYWBo2GM59_pkT1ZPcY_0 2024-03-09T15:36:06.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.155__shadow_.dTGpVXE5LuysoYWBo2GM59_pkT1ZPcY_1 2024-03-09T15:36:06.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11122508996628222761', time=2024-03-09T17:21:37.755592+0000, chain.objs.size()=1 2024-03-09T15:36:06.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.242__shadow_.91vmlIx2HytSGFfwUIrhAEg3ZO4QnCc_0 2024-03-09T15:36:06.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7622847091722470729', time=2024-03-09T17:21:40.690069+0000, chain.objs.size()=1 2024-03-09T15:36:06.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.243__shadow_.csrnWIAhE72g4SnJNg1yjWFhT0fNpmj_0 2024-03-09T15:36:06.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.18060859420931541955', time=2024-03-09T17:22:05.994660+0000, chain.objs.size()=1 2024-03-09T15:36:06.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.252__shadow_._ZctvcnmS_5o2PV2cMiRAjs_naE7SZi_0 2024-03-09T15:36:06.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15538528210676669632', time=2024-03-09T17:23:58.908898+0000, chain.objs.size()=1 2024-03-09T15:36:06.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.EviG8H7eOWoDQiXTxLP0g4q-ezM528y_0 2024-03-09T15:36:06.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.199736783745293648', time=2024-03-09T17:24:01.377654+0000, chain.objs.size()=1 2024-03-09T15:36:06.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.xIMBO32wqxzE6Ag1dvhaskE5DhjERrw_0 2024-03-09T15:36:06.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13177368889424713290', time=2024-03-09T17:24:01.399880+0000, chain.objs.size()=1 2024-03-09T15:36:06.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1dYzbjhW5XRHt4w4i6iZDiPEPImgpke_0 2024-03-09T15:36:06.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2385495777794538290', time=2024-03-09T17:24:05.009212+0000, chain.objs.size()=1 2024-03-09T15:36:06.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8tH6J0efaGpW9GcaZIoSe2A_zYgtHAb_0 2024-03-09T15:36:06.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.946570705620362152', time=2024-03-09T17:24:05.664867+0000, chain.objs.size()=1 2024-03-09T15:36:06.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Y4GF57-BINttx72L9bGmXlJrhvtPsgA_0 2024-03-09T15:36:06.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.3488610596988867907', time=2024-03-09T17:24:05.943244+0000, chain.objs.size()=1 2024-03-09T15:36:06.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.512+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.QRRUP5gMJvCp5kRIdycxpplwJ6sYpQ0_0 2024-03-09T15:36:06.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5302337742122858698', time=2024-03-09T17:24:06.207512+0000, chain.objs.size()=1 2024-03-09T15:36:06.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.1S-muyEBJkx4P8UPYUq0QR5mefzIOvs_0 2024-03-09T15:36:06.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11178070917219510922', time=2024-03-09T17:24:06.250408+0000, chain.objs.size()=1 2024-03-09T15:36:06.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.TNRdOGiEBcPbR1i1PItuicjCU-3tVBQ_0 2024-03-09T15:36:06.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.139304634923862852', time=2024-03-09T17:24:06.736861+0000, chain.objs.size()=1 2024-03-09T15:36:06.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.roFUFRAdLc6A8cOo1X8O7XMk8cgPOTV_0 2024-03-09T15:36:06.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1903367960871064289', time=2024-03-09T17:24:10.442983+0000, chain.objs.size()=1 2024-03-09T15:36:06.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.cxZkHf6HY4CAEIgB0dT-2HN_kuEbCb6_0 2024-03-09T15:36:06.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12435178647366981833', time=2024-03-09T17:24:13.140099+0000, chain.objs.size()=1 2024-03-09T15:36:06.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.A9kIuyGDBwWadBNWdddvOULywk_xKUK_0 2024-03-09T15:36:06.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12659667029557462284', time=2024-03-09T17:24:13.142488+0000, chain.objs.size()=1 2024-03-09T15:36:06.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.-FIjOd1R2zXkyXNGV30L1NgiXLgDkDv_0 2024-03-09T15:36:06.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14391779753567961200', time=2024-03-09T17:24:13.792436+0000, chain.objs.size()=1 2024-03-09T15:36:06.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.PQa24zU6Kv9CqlEDd5Zh7Csb5yz59rd_0 2024-03-09T15:36:06.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11108843461247234628', time=2024-03-09T17:24:13.863554+0000, chain.objs.size()=1 2024-03-09T15:36:06.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.iwUePlgiVu65Gc4M2A2K30ls1z-FQ2k_0 2024-03-09T15:36:06.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5063861585665882493', time=2024-03-09T17:24:14.294037+0000, chain.objs.size()=1 2024-03-09T15:36:06.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.YxGWOPCRuMuwsjxcRPskx0Ycpidd7M1_0 2024-03-09T15:36:06.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14829371887896726036', time=2024-03-09T17:24:14.296145+0000, chain.objs.size()=1 2024-03-09T15:36:06.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.fLUid-ycLdk5GyuD5Siy3naeVOFLEXt_0 2024-03-09T15:36:06.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7133574448448909293', time=2024-03-09T17:24:14.717060+0000, chain.objs.size()=1 2024-03-09T15:36:06.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.bym3McoyRv9ImoJBw0qOZT9CmnHeWOK_0 2024-03-09T15:36:06.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.13864514595290328771', time=2024-03-09T17:24:16.323699+0000, chain.objs.size()=1 2024-03-09T15:36:06.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.516+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.0XsuVEThM7jehrKk1pxwoR66WLVJCtn_0 2024-03-09T15:36:06.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2592481296846855037', time=2024-03-09T17:24:16.834277+0000, chain.objs.size()=1 2024-03-09T15:36:06.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.kj76rNtIhuvxDghMyIa50cVZepkQnXq_0 2024-03-09T15:36:06.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.17035395656154424813', time=2024-03-09T17:24:17.251073+0000, chain.objs.size()=1 2024-03-09T15:36:06.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.npMNuJ9ZMuhyzBfJ2sIypTMZ90D6PoY_0 2024-03-09T15:36:06.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15275855919651711404', time=2024-03-09T17:24:18.170075+0000, chain.objs.size()=1 2024-03-09T15:36:06.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.jux7-2ldcnTcCyVVV4M5GVD_7-Lm545_0 2024-03-09T15:36:06.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15368412278641499760', time=2024-03-09T17:24:20.490401+0000, chain.objs.size()=1 2024-03-09T15:36:06.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.L1HOrKpVs9URi85yn_lpMJr_vYyvSCN_0 2024-03-09T15:36:06.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.9249273499385236768', time=2024-03-09T17:24:20.917422+0000, chain.objs.size()=1 2024-03-09T15:36:06.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.8G6XX0cK0xwG4mgAQxBvohC9f5l82RZ_0 2024-03-09T15:36:06.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5205302133130435405', time=2024-03-09T17:24:22.366742+0000, chain.objs.size()=1 2024-03-09T15:36:06.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.turMuNxLBeAuPRZjBOHAPIcL-29eFSb_0 2024-03-09T15:36:06.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.15055511734443172024', time=2024-03-09T17:24:22.558384+0000, chain.objs.size()=1 2024-03-09T15:36:06.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.eP5qbRjWz_z08Omm1r87xCcKtSleBQT_0 2024-03-09T15:36:06.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.16853137546102209083', time=2024-03-09T17:24:22.842387+0000, chain.objs.size()=1 2024-03-09T15:36:06.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Y4_2tkd81x8GvidbvyjgPVD69yMXzWO_0 2024-03-09T15:36:06.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.14918885973189566646', time=2024-03-09T17:24:25.135725+0000, chain.objs.size()=1 2024-03-09T15:36:06.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.2H86qoG68hzNfmTX0e0Yx1fvHZyWLsw_0 2024-03-09T15:36:06.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.8682541028019800694', time=2024-03-09T17:24:25.143571+0000, chain.objs.size()=1 2024-03-09T15:36:06.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.f6qNxPPlRJv-Acje0aY9Cs2vBrZQDgy_0 2024-03-09T15:36:06.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5282129756416383695', time=2024-03-09T17:24:25.512531+0000, chain.objs.size()=1 2024-03-09T15:36:06.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.E6ltnpz1zzK9R-tpy6m9q9avaip3E8v_0 2024-03-09T15:36:06.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.1465313688739797953', time=2024-03-09T17:24:27.109929+0000, chain.objs.size()=1 2024-03-09T15:36:06.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.SknEOkFQwTYEihico3u10E6bnQJ8YwK_0 2024-03-09T15:36:06.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4427490667543957726', time=2024-03-09T17:24:29.142050+0000, chain.objs.size()=1 2024-03-09T15:36:06.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.RCcMVHG5a9uEbN_LelOFSNsEhSb8LV4_0 2024-03-09T15:36:06.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.5295877431542198625', time=2024-03-09T17:24:30.326803+0000, chain.objs.size()=1 2024-03-09T15:36:06.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.gVXbhf-pe6OD-IMG-RfKky4VlFUWOIm_0 2024-03-09T15:36:06.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4075474274477032456', time=2024-03-09T17:24:30.414256+0000, chain.objs.size()=1 2024-03-09T15:36:06.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.255__shadow_.Gk-Sk4TqdH4TpW1DWD56QcsPExxFoGp_0 2024-03-09T15:36:06.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.12267014954306240578', time=2024-03-09T17:24:48.147496+0000, chain.objs.size()=1 2024-03-09T15:36:06.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.261__shadow_.zFe_6SJ0EN4-ptzjXfVtoVo95NbY-ex_0 2024-03-09T15:36:06.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6557017117412917303', time=2024-03-09T17:24:48.165803+0000, chain.objs.size()=1 2024-03-09T15:36:06.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.261__shadow_.Km9rfwOkf0Zg_-JzHyH1zbhIOr6U3Qu_0 2024-03-09T15:36:06.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2234633076386877380', time=2024-03-09T17:24:56.416662+0000, chain.objs.size()=1 2024-03-09T15:36:06.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.264__shadow_.owEAwGMcPjwJFgtHQq7MHjjEOtMEN-4_0 2024-03-09T15:36:06.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11897028608162571874', time=2024-03-09T17:25:10.549669+0000, chain.objs.size()=1 2024-03-09T15:36:06.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.269__shadow_.lMEAjP5s1eo418LT-Wt5erHZiMmDNni_0 2024-03-09T15:36:06.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.7869916316622614804', time=2024-03-09T17:25:13.445439+0000, chain.objs.size()=1 2024-03-09T15:36:06.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.520+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.270__shadow_.LWoRf9Pqk73nTK3F9yP_n9NbMvNC2aj_0 2024-03-09T15:36:06.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.6693322993441528354', time=2024-03-09T17:25:45.519364+0000, chain.objs.size()=1 2024-03-09T15:36:06.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.281__shadow_.Ar8M9ac5FLivcPqhTQm8Gp1-dfF17Pk_0 2024-03-09T15:36:06.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.11714508921817661673', time=2024-03-09T17:26:01.517803+0000, chain.objs.size()=1 2024-03-09T15:36:06.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.285__shadow_.-L_wOjaZ-IxC87TA_JHiSUI0ebeoyPh_0 2024-03-09T15:36:06.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.907666278171362468', time=2024-03-09T17:26:07.924198+0000, chain.objs.size()=1 2024-03-09T15:36:06.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.FRIod7ykeJ6PCEGE2F3iufD6-yWdiwo_0 2024-03-09T15:36:06.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.429023225265693216', time=2024-03-09T17:26:07.930060+0000, chain.objs.size()=1 2024-03-09T15:36:06.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.286__shadow_.fXbQ4EehcgTvPzLSoQBReSSthDLMJe6_0 2024-03-09T15:36:06.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.2853336972140976604', time=2024-03-09T17:26:13.948642+0000, chain.objs.size()=1 2024-03-09T15:36:06.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.288__shadow_.NjXK6T-fsYDqh_7bME5UZeclYWV_1mD_0 2024-03-09T15:36:06.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 20 garbage collection: RGWGC::process iterating over entry tag='8c209108-e1cd-4d45-9608-6f491550b24c.4644.4443297106256688691', time=2024-03-09T17:26:28.894112+0000, chain.objs.size()=1 2024-03-09T15:36:06.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:8c209108-e1cd-4d45-9608-6f491550b24c.4644.293__shadow_.8nv5igzpK6kAnWL_EAoQGC0dT37E4uC_0 2024-03-09T15:36:06.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.524+0000 7f471c3a7980 5 garbage collection: RGWGC::process removing entries, marker: 2024-03-09T15:36:06.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.528+0000 7f45cffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-03-09T15:36:06.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.532+0000 7f471c3a7980 20 remove_watcher() i=0 2024-03-09T15:36:06.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.532+0000 7f471c3a7980 2 removed watcher, disabling cache 2024-03-09T15:36:06.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.532+0000 7f471c3a7980 20 remove_watcher() i=1 2024-03-09T15:36:06.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.532+0000 7f471c3a7980 20 remove_watcher() i=2 2024-03-09T15:36:06.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.532+0000 7f471c3a7980 20 remove_watcher() i=3 2024-03-09T15:36:06.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.536+0000 7f471c3a7980 20 remove_watcher() i=4 2024-03-09T15:36:06.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.536+0000 7f471c3a7980 20 remove_watcher() i=5 2024-03-09T15:36:06.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.536+0000 7f471c3a7980 20 remove_watcher() i=6 2024-03-09T15:36:06.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-09T15:36:06.536+0000 7f471c3a7980 20 remove_watcher() i=7 2024-03-09T15:36:06.554 DEBUG:teuthology.run_tasks:Unwinding manager openssl_keys 2024-03-09T15:36:06.564 DEBUG:teuthology.orchestra.run.smithi001:> rm -f /home/ubuntu/cephtest/ca/root.crt /home/ubuntu/cephtest/ca/root.key 2024-03-09T15:36:06.601 DEBUG:teuthology.orchestra.run.smithi001:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2024-03-09T15:36:06.648 DEBUG:teuthology.orchestra.run.smithi001:> rm -f /home/ubuntu/cephtest/ca/rgw.client.0.crt /home/ubuntu/cephtest/ca/rgw.client.0.key 2024-03-09T15:36:06.697 DEBUG:teuthology.orchestra.run.smithi001:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2024-03-09T15:36:06.749 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm /usr/local/share/ca-certificates/root.crt 2024-03-09T15:36:06.809 DEBUG:teuthology.orchestra.run.smithi001:> sudo update-ca-certificates 2024-03-09T15:36:06.866 INFO:teuthology.orchestra.run.smithi001.stdout:Updating certificates in /etc/ssl/certs... 2024-03-09T15:36:07.303 INFO:teuthology.orchestra.run.smithi001.stdout:0 added, 0 removed; done. 2024-03-09T15:36:07.304 INFO:teuthology.orchestra.run.smithi001.stdout:Running hooks in /etc/ca-certificates/update.d... 2024-03-09T15:36:07.305 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:36:07.518 INFO:teuthology.orchestra.run.smithi001.stdout:done. 2024-03-09T15:36:07.518 INFO:teuthology.orchestra.run.smithi001.stdout:done. 2024-03-09T15:36:07.521 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-09T15:36:07.533 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-09T15:36:07.534 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T15:36:07.937 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:36:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-09T15:36:07.978 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":2291,"stamp":"2024-03-09T15:36:07.248269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376290644,"num_objects":1788,"num_object_clones":0,"num_object_copies":3576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1788,"num_whiteouts":0,"num_read":77680,"num_read_kb":1570816,"num_write":67709,"num_write_kb":1988128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12459,"ondisk_log_size":12459,"up":786,"acting":786,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":786,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2001752,"kb_used_data":905560,"kb_used_omap":12,"kb_used_meta":1096115,"kb_avail":752972968,"statfs":{"total":773094113280,"available":771044319232,"internally_reserved":0,"allocated":927293440,"data_stored":902610116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1122422320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-66869968,"num_objects":-307,"num_object_clones":0,"num_object_copies":-614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-307,"num_whiteouts":0,"num_read":462,"num_read_kb":498,"num_write":368,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007256"},"pg_stats":[{"pgid":"68.1a","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.178131+0000","last_change":"2024-03-09T15:09:50.085348+0000","last_active":"2024-03-09T15:32:21.178131+0000","last_peered":"2024-03-09T15:32:21.178131+0000","last_clean":"2024-03-09T15:32:21.178131+0000","last_became_active":"2024-03-09T15:09:50.084784+0000","last_became_peered":"2024-03-09T15:09:50.084784+0000","last_unstale":"2024-03-09T15:32:21.178131+0000","last_undegraded":"2024-03-09T15:32:21.178131+0000","last_fullsized":"2024-03-09T15:32:21.178131+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:18:17.114617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.1d","version":"258'7","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.167423+0000","last_change":"2024-03-09T14:46:26.826294+0000","last_active":"2024-03-09T15:31:31.167423+0000","last_peered":"2024-03-09T15:31:31.167423+0000","last_clean":"2024-03-09T15:31:31.167423+0000","last_became_active":"2024-03-09T14:46:26.825404+0000","last_became_peered":"2024-03-09T14:46:26.825404+0000","last_unstale":"2024-03-09T15:31:31.167423+0000","last_undegraded":"2024-03-09T15:31:31.167423+0000","last_fullsized":"2024-03-09T15:31:31.167423+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:28:03.324789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20498,"num_write":6,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1c","version":"261'1","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:20.773210+0000","last_change":"2024-03-09T14:46:18.769137+0000","last_active":"2024-03-09T15:31:20.773210+0000","last_peered":"2024-03-09T15:31:20.773210+0000","last_clean":"2024-03-09T15:31:20.773210+0000","last_became_active":"2024-03-09T14:46:18.768383+0000","last_became_peered":"2024-03-09T14:46:18.768383+0000","last_unstale":"2024-03-09T15:31:20.773210+0000","last_undegraded":"2024-03-09T15:31:20.773210+0000","last_fullsized":"2024-03-09T15:31:20.773210+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:38:13.467012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":3072,"num_write":1,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.1f","version":"258'45","reported_seq":167,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.010472+0000","last_change":"2024-03-09T14:39:13.282130+0000","last_active":"2024-03-09T15:33:26.010472+0000","last_peered":"2024-03-09T15:33:26.010472+0000","last_clean":"2024-03-09T15:33:26.010472+0000","last_became_active":"2024-03-09T14:39:13.281020+0000","last_became_peered":"2024-03-09T14:39:13.281020+0000","last_unstale":"2024-03-09T15:33:26.010472+0000","last_undegraded":"2024-03-09T15:33:26.010472+0000","last_fullsized":"2024-03-09T15:33:26.010472+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:54.492385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":134,"num_write":70,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1b","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.961650+0000","last_change":"2024-03-09T15:09:50.083262+0000","last_active":"2024-03-09T15:32:20.961650+0000","last_peered":"2024-03-09T15:32:20.961650+0000","last_clean":"2024-03-09T15:32:20.961650+0000","last_became_active":"2024-03-09T15:09:50.082690+0000","last_became_peered":"2024-03-09T15:09:50.082690+0000","last_unstale":"2024-03-09T15:32:20.961650+0000","last_undegraded":"2024-03-09T15:32:20.961650+0000","last_fullsized":"2024-03-09T15:32:20.961650+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:18:09.424746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1c","version":"258'5","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162661+0000","last_change":"2024-03-09T14:46:26.807670+0000","last_active":"2024-03-09T15:31:26.162661+0000","last_peered":"2024-03-09T15:31:26.162661+0000","last_clean":"2024-03-09T15:31:26.162661+0000","last_became_active":"2024-03-09T14:46:26.806865+0000","last_became_peered":"2024-03-09T14:46:26.806865+0000","last_unstale":"2024-03-09T15:31:26.162661+0000","last_undegraded":"2024-03-09T15:31:26.162661+0000","last_fullsized":"2024-03-09T15:31:26.162661+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:05:04.558231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.1d","version":"261'1","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:21.130434+0000","last_change":"2024-03-09T14:46:18.780095+0000","last_active":"2024-03-09T15:31:21.130434+0000","last_peered":"2024-03-09T15:31:21.130434+0000","last_clean":"2024-03-09T15:31:21.130434+0000","last_became_active":"2024-03-09T14:46:18.779760+0000","last_became_peered":"2024-03-09T14:46:18.779760+0000","last_unstale":"2024-03-09T15:31:21.130434+0000","last_undegraded":"2024-03-09T15:31:21.130434+0000","last_fullsized":"2024-03-09T15:31:21.130434+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:12.211222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":1,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1e","version":"0'0","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:30.975652+0000","last_change":"2024-03-09T14:39:13.260541+0000","last_active":"2024-03-09T15:33:30.975652+0000","last_peered":"2024-03-09T15:33:30.975652+0000","last_clean":"2024-03-09T15:33:30.975652+0000","last_became_active":"2024-03-09T14:39:13.259731+0000","last_became_peered":"2024-03-09T14:39:13.259731+0000","last_unstale":"2024-03-09T15:33:30.975652+0000","last_undegraded":"2024-03-09T15:33:30.975652+0000","last_fullsized":"2024-03-09T15:33:30.975652+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:09:28.555448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.18","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.764910+0000","last_change":"2024-03-09T15:09:50.081224+0000","last_active":"2024-03-09T15:31:35.764910+0000","last_peered":"2024-03-09T15:31:35.764910+0000","last_clean":"2024-03-09T15:31:35.764910+0000","last_became_active":"2024-03-09T15:09:50.080578+0000","last_became_peered":"2024-03-09T15:09:50.080578+0000","last_unstale":"2024-03-09T15:31:35.764910+0000","last_undegraded":"2024-03-09T15:31:35.764910+0000","last_fullsized":"2024-03-09T15:31:35.764910+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:54:04.030480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.1f","version":"261'4","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.995705+0000","last_change":"2024-03-09T14:46:26.896619+0000","last_active":"2024-03-09T15:32:20.995705+0000","last_peered":"2024-03-09T15:32:20.995705+0000","last_clean":"2024-03-09T15:32:20.995705+0000","last_became_active":"2024-03-09T14:46:26.895608+0000","last_became_peered":"2024-03-09T14:46:26.895608+0000","last_unstale":"2024-03-09T15:32:20.995705+0000","last_undegraded":"2024-03-09T15:32:20.995705+0000","last_fullsized":"2024-03-09T15:32:20.995705+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:32:06.502138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":12288,"num_write":4,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1e","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:35.964975+0000","last_change":"2024-03-09T14:46:18.782402+0000","last_active":"2024-03-09T15:32:35.964975+0000","last_peered":"2024-03-09T15:32:35.964975+0000","last_clean":"2024-03-09T15:32:35.964975+0000","last_became_active":"2024-03-09T14:46:18.781856+0000","last_became_peered":"2024-03-09T14:46:18.781856+0000","last_unstale":"2024-03-09T15:32:35.964975+0000","last_undegraded":"2024-03-09T15:32:35.964975+0000","last_fullsized":"2024-03-09T15:32:35.964975+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:29:15.829032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.1d","version":"258'35","reported_seq":142,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.987080+0000","last_change":"2024-03-09T14:39:13.265101+0000","last_active":"2024-03-09T15:31:45.987080+0000","last_peered":"2024-03-09T15:31:45.987080+0000","last_clean":"2024-03-09T15:31:45.987080+0000","last_became_active":"2024-03-09T14:39:13.258351+0000","last_became_peered":"2024-03-09T14:39:13.258351+0000","last_unstale":"2024-03-09T15:31:45.987080+0000","last_undegraded":"2024-03-09T15:31:45.987080+0000","last_fullsized":"2024-03-09T15:31:45.987080+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:21:02.690940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":152,"num_read_kb":109,"num_write":57,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.19","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.777876+0000","last_change":"2024-03-09T15:09:50.083658+0000","last_active":"2024-03-09T15:31:40.777876+0000","last_peered":"2024-03-09T15:31:40.777876+0000","last_clean":"2024-03-09T15:31:40.777876+0000","last_became_active":"2024-03-09T15:09:50.081680+0000","last_became_peered":"2024-03-09T15:09:50.081680+0000","last_unstale":"2024-03-09T15:31:40.777876+0000","last_undegraded":"2024-03-09T15:31:40.777876+0000","last_fullsized":"2024-03-09T15:31:40.777876+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:56:53.220664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.1e","version":"261'4","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:21.009093+0000","last_change":"2024-03-09T14:47:07.798858+0000","last_active":"2024-03-09T15:33:21.009093+0000","last_peered":"2024-03-09T15:33:21.009093+0000","last_clean":"2024-03-09T15:33:21.009093+0000","last_became_active":"2024-03-09T14:47:07.798247+0000","last_became_peered":"2024-03-09T14:47:07.798247+0000","last_unstale":"2024-03-09T15:33:21.009093+0000","last_undegraded":"2024-03-09T15:33:21.009093+0000","last_fullsized":"2024-03-09T15:33:21.009093+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:45:56.993617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":16384,"num_write":9,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1f","version":"258'5","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.173075+0000","last_change":"2024-03-09T14:46:18.772485+0000","last_active":"2024-03-09T15:32:11.173075+0000","last_peered":"2024-03-09T15:32:11.173075+0000","last_clean":"2024-03-09T15:32:11.173075+0000","last_became_active":"2024-03-09T14:46:18.771731+0000","last_became_peered":"2024-03-09T14:46:18.771731+0000","last_unstale":"2024-03-09T15:32:11.173075+0000","last_undegraded":"2024-03-09T15:32:11.173075+0000","last_fullsized":"2024-03-09T15:32:11.173075+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:50:44.589948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5136,"num_write":5,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1c","version":"261'13","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.779361+0000","last_change":"2024-03-09T14:39:13.266571+0000","last_active":"2024-03-09T15:31:45.779361+0000","last_peered":"2024-03-09T15:31:45.779361+0000","last_clean":"2024-03-09T15:31:45.779361+0000","last_became_active":"2024-03-09T14:39:13.260945+0000","last_became_peered":"2024-03-09T14:39:13.260945+0000","last_unstale":"2024-03-09T15:31:45.779361+0000","last_undegraded":"2024-03-09T15:31:45.779361+0000","last_fullsized":"2024-03-09T15:31:45.779361+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:58:40.520455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":27,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1e","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.808730+0000","last_change":"2024-03-09T15:09:50.088230+0000","last_active":"2024-03-09T15:33:55.808730+0000","last_peered":"2024-03-09T15:33:55.808730+0000","last_clean":"2024-03-09T15:33:55.808730+0000","last_became_active":"2024-03-09T15:09:50.086789+0000","last_became_peered":"2024-03-09T15:09:50.086789+0000","last_unstale":"2024-03-09T15:33:55.808730+0000","last_undegraded":"2024-03-09T15:33:55.808730+0000","last_fullsized":"2024-03-09T15:33:55.808730+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:54:27.433652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.19","version":"261'6","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.109749+0000","last_change":"2024-03-09T14:46:26.819845+0000","last_active":"2024-03-09T15:36:06.109749+0000","last_peered":"2024-03-09T15:36:06.109749+0000","last_clean":"2024-03-09T15:36:06.109749+0000","last_became_active":"2024-03-09T14:46:26.819085+0000","last_became_peered":"2024-03-09T14:46:26.819085+0000","last_unstale":"2024-03-09T15:36:06.109749+0000","last_undegraded":"2024-03-09T15:36:06.109749+0000","last_fullsized":"2024-03-09T15:36:06.109749+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:17.867538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":12288,"num_write":11,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.18","version":"261'12","reported_seq":78,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:41.027571+0000","last_change":"2024-03-09T14:46:18.776960+0000","last_active":"2024-03-09T15:34:41.027571+0000","last_peered":"2024-03-09T15:34:41.027571+0000","last_clean":"2024-03-09T15:34:41.027571+0000","last_became_active":"2024-03-09T14:46:18.776161+0000","last_became_peered":"2024-03-09T14:46:18.776161+0000","last_unstale":"2024-03-09T15:34:41.027571+0000","last_undegraded":"2024-03-09T15:34:41.027571+0000","last_fullsized":"2024-03-09T15:34:41.027571+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:22:11.563005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":36889,"num_write":16,"num_write_kb":16407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.1b","version":"258'10","reported_seq":79,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.779245+0000","last_change":"2024-03-09T14:39:13.269298+0000","last_active":"2024-03-09T15:31:45.779245+0000","last_peered":"2024-03-09T15:31:45.779245+0000","last_clean":"2024-03-09T15:31:45.779245+0000","last_became_active":"2024-03-09T14:39:13.268179+0000","last_became_peered":"2024-03-09T14:39:13.268179+0000","last_unstale":"2024-03-09T15:31:45.779245+0000","last_undegraded":"2024-03-09T15:31:45.779245+0000","last_fullsized":"2024-03-09T15:31:45.779245+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:12:05.852927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1f","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:45.951375+0000","last_change":"2024-03-09T15:09:50.064176+0000","last_active":"2024-03-09T15:32:45.951375+0000","last_peered":"2024-03-09T15:32:45.951375+0000","last_clean":"2024-03-09T15:32:45.951375+0000","last_became_active":"2024-03-09T15:09:50.063705+0000","last_became_peered":"2024-03-09T15:09:50.063705+0000","last_unstale":"2024-03-09T15:32:45.951375+0000","last_undegraded":"2024-03-09T15:32:45.951375+0000","last_fullsized":"2024-03-09T15:32:45.951375+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:45.866476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.18","version":"261'2","reported_seq":46,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.775288+0000","last_change":"2024-03-09T14:46:26.824830+0000","last_active":"2024-03-09T15:32:25.775288+0000","last_peered":"2024-03-09T15:32:25.775288+0000","last_clean":"2024-03-09T15:32:25.775288+0000","last_became_active":"2024-03-09T14:46:26.824522+0000","last_became_peered":"2024-03-09T14:46:26.824522+0000","last_unstale":"2024-03-09T15:32:25.775288+0000","last_undegraded":"2024-03-09T15:32:25.775288+0000","last_fullsized":"2024-03-09T15:32:25.775288+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:09:09.414799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":2,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.19","version":"261'7","reported_seq":64,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.175135+0000","last_change":"2024-03-09T14:46:18.770577+0000","last_active":"2024-03-09T15:32:21.175135+0000","last_peered":"2024-03-09T15:32:21.175135+0000","last_clean":"2024-03-09T15:32:21.175135+0000","last_became_active":"2024-03-09T14:46:18.769730+0000","last_became_peered":"2024-03-09T14:46:18.769730+0000","last_unstale":"2024-03-09T15:32:21.175135+0000","last_undegraded":"2024-03-09T15:32:21.175135+0000","last_fullsized":"2024-03-09T15:32:21.175135+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:07:45.259457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340032,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":19456,"num_write":17,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1a","version":"261'44","reported_seq":164,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.767114+0000","last_change":"2024-03-09T14:39:13.274060+0000","last_active":"2024-03-09T15:31:45.767114+0000","last_peered":"2024-03-09T15:31:45.767114+0000","last_clean":"2024-03-09T15:31:45.767114+0000","last_became_active":"2024-03-09T14:39:13.273298+0000","last_became_peered":"2024-03-09T14:39:13.273298+0000","last_unstale":"2024-03-09T15:31:45.767114+0000","last_undegraded":"2024-03-09T15:31:45.767114+0000","last_fullsized":"2024-03-09T15:31:45.767114+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:16:28.025129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":138,"num_write":72,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.1c","version":"261'1","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188241+0000","last_change":"2024-03-09T15:09:50.075720+0000","last_active":"2024-03-09T15:32:21.188241+0000","last_peered":"2024-03-09T15:32:21.188241+0000","last_clean":"2024-03-09T15:32:21.188241+0000","last_became_active":"2024-03-09T15:09:50.074952+0000","last_became_peered":"2024-03-09T15:09:50.074952+0000","last_unstale":"2024-03-09T15:32:21.188241+0000","last_undegraded":"2024-03-09T15:32:21.188241+0000","last_fullsized":"2024-03-09T15:32:21.188241+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:21:35.439413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1b","version":"261'2","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:30.763859+0000","last_change":"2024-03-09T14:46:26.805382+0000","last_active":"2024-03-09T15:31:30.763859+0000","last_peered":"2024-03-09T15:31:30.763859+0000","last_clean":"2024-03-09T15:31:30.763859+0000","last_became_active":"2024-03-09T14:46:26.804622+0000","last_became_peered":"2024-03-09T14:46:26.804622+0000","last_unstale":"2024-03-09T15:31:30.763859+0000","last_undegraded":"2024-03-09T15:31:30.763859+0000","last_fullsized":"2024-03-09T15:31:30.763859+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:23.446296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":5120,"num_write":2,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.1a","version":"261'4","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:21.161277+0000","last_change":"2024-03-09T14:46:18.785531+0000","last_active":"2024-03-09T15:31:21.161277+0000","last_peered":"2024-03-09T15:31:21.161277+0000","last_clean":"2024-03-09T15:31:21.161277+0000","last_became_active":"2024-03-09T14:46:18.784821+0000","last_became_peered":"2024-03-09T14:46:18.784821+0000","last_unstale":"2024-03-09T15:31:21.161277+0000","last_undegraded":"2024-03-09T15:31:21.161277+0000","last_fullsized":"2024-03-09T15:31:21.161277+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:10:34.919135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4096,"num_write":9,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.19","version":"258'28","reported_seq":120,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:40.999888+0000","last_change":"2024-03-09T14:39:13.258289+0000","last_active":"2024-03-09T15:32:40.999888+0000","last_peered":"2024-03-09T15:32:40.999888+0000","last_clean":"2024-03-09T15:32:40.999888+0000","last_became_active":"2024-03-09T14:39:13.257473+0000","last_became_peered":"2024-03-09T14:39:13.257473+0000","last_unstale":"2024-03-09T15:32:40.999888+0000","last_undegraded":"2024-03-09T15:32:40.999888+0000","last_fullsized":"2024-03-09T15:32:40.999888+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:12:15.900385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":78,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1d","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188298+0000","last_change":"2024-03-09T15:09:50.075294+0000","last_active":"2024-03-09T15:32:21.188298+0000","last_peered":"2024-03-09T15:32:21.188298+0000","last_clean":"2024-03-09T15:32:21.188298+0000","last_became_active":"2024-03-09T15:09:50.074875+0000","last_became_peered":"2024-03-09T15:09:50.074875+0000","last_unstale":"2024-03-09T15:32:21.188298+0000","last_undegraded":"2024-03-09T15:32:21.188298+0000","last_fullsized":"2024-03-09T15:32:21.188298+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:24:33.386479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1a","version":"261'9","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:30.775394+0000","last_change":"2024-03-09T14:46:26.813798+0000","last_active":"2024-03-09T15:31:30.775394+0000","last_peered":"2024-03-09T15:31:30.775394+0000","last_clean":"2024-03-09T15:31:30.775394+0000","last_became_active":"2024-03-09T14:46:26.812889+0000","last_became_peered":"2024-03-09T14:46:26.812889+0000","last_unstale":"2024-03-09T15:31:30.775394+0000","last_undegraded":"2024-03-09T15:31:30.775394+0000","last_fullsized":"2024-03-09T15:31:30.775394+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:43:59.844987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":12290,"num_write":13,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.1b","version":"261'1","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:20.980983+0000","last_change":"2024-03-09T14:46:18.774605+0000","last_active":"2024-03-09T15:31:20.980983+0000","last_peered":"2024-03-09T15:31:20.980983+0000","last_clean":"2024-03-09T15:31:20.980983+0000","last_became_active":"2024-03-09T14:46:18.756394+0000","last_became_peered":"2024-03-09T14:46:18.756394+0000","last_unstale":"2024-03-09T15:31:20.980983+0000","last_undegraded":"2024-03-09T15:31:20.980983+0000","last_fullsized":"2024-03-09T15:31:20.980983+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:49.690847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40145,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":40,"num_write":1,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.18","version":"261'8","reported_seq":77,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.144778+0000","last_change":"2024-03-09T14:39:13.258846+0000","last_active":"2024-03-09T15:32:26.144778+0000","last_peered":"2024-03-09T15:32:26.144778+0000","last_clean":"2024-03-09T15:32:26.144778+0000","last_became_active":"2024-03-09T14:39:13.257996+0000","last_became_peered":"2024-03-09T14:39:13.257996+0000","last_unstale":"2024-03-09T15:32:26.144778+0000","last_undegraded":"2024-03-09T15:32:26.144778+0000","last_fullsized":"2024-03-09T15:32:26.144778+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:23:43.508747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.12","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.957209+0000","last_change":"2024-03-09T15:09:50.070950+0000","last_active":"2024-03-09T15:33:10.957209+0000","last_peered":"2024-03-09T15:33:10.957209+0000","last_clean":"2024-03-09T15:33:10.957209+0000","last_became_active":"2024-03-09T15:09:50.070345+0000","last_became_peered":"2024-03-09T15:09:50.070345+0000","last_unstale":"2024-03-09T15:33:10.957209+0000","last_undegraded":"2024-03-09T15:33:10.957209+0000","last_fullsized":"2024-03-09T15:33:10.957209+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:18:24.068533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.15","version":"0'0","reported_seq":42,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.788416+0000","last_change":"2024-03-09T14:46:26.891781+0000","last_active":"2024-03-09T15:32:25.788416+0000","last_peered":"2024-03-09T15:32:25.788416+0000","last_clean":"2024-03-09T15:32:25.788416+0000","last_became_active":"2024-03-09T14:46:26.890992+0000","last_became_peered":"2024-03-09T14:46:26.890992+0000","last_unstale":"2024-03-09T15:32:25.788416+0000","last_undegraded":"2024-03-09T15:32:25.788416+0000","last_fullsized":"2024-03-09T15:32:25.788416+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:56:14.439157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.14","version":"0'0","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:06.187943+0000","last_change":"2024-03-09T14:46:18.767760+0000","last_active":"2024-03-09T15:33:06.187943+0000","last_peered":"2024-03-09T15:33:06.187943+0000","last_clean":"2024-03-09T15:33:06.187943+0000","last_became_active":"2024-03-09T14:46:18.766777+0000","last_became_peered":"2024-03-09T14:46:18.766777+0000","last_unstale":"2024-03-09T15:33:06.187943+0000","last_undegraded":"2024-03-09T15:33:06.187943+0000","last_fullsized":"2024-03-09T15:33:06.187943+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:30:54.129379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.17","version":"258'19","reported_seq":106,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.996874+0000","last_change":"2024-03-09T14:39:13.281484+0000","last_active":"2024-03-09T15:32:25.996874+0000","last_peered":"2024-03-09T15:32:25.996874+0000","last_clean":"2024-03-09T15:32:25.996874+0000","last_became_active":"2024-03-09T14:39:13.279919+0000","last_became_peered":"2024-03-09T14:39:13.279919+0000","last_unstale":"2024-03-09T15:32:25.996874+0000","last_undegraded":"2024-03-09T15:32:25.996874+0000","last_fullsized":"2024-03-09T15:32:25.996874+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:09:49.670708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":64,"num_write":32,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.13","version":"261'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.971686+0000","last_change":"2024-03-09T15:09:50.079986+0000","last_active":"2024-03-09T15:33:10.971686+0000","last_peered":"2024-03-09T15:33:10.971686+0000","last_clean":"2024-03-09T15:33:10.971686+0000","last_became_active":"2024-03-09T15:09:50.079604+0000","last_became_peered":"2024-03-09T15:09:50.079604+0000","last_unstale":"2024-03-09T15:33:10.971686+0000","last_undegraded":"2024-03-09T15:33:10.971686+0000","last_fullsized":"2024-03-09T15:33:10.971686+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:30:05.194786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.14","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.800266+0000","last_change":"2024-03-09T14:46:26.894898+0000","last_active":"2024-03-09T15:33:20.800266+0000","last_peered":"2024-03-09T15:33:20.800266+0000","last_clean":"2024-03-09T15:33:20.800266+0000","last_became_active":"2024-03-09T14:46:26.894088+0000","last_became_peered":"2024-03-09T14:46:26.894088+0000","last_unstale":"2024-03-09T15:33:20.800266+0000","last_undegraded":"2024-03-09T15:33:20.800266+0000","last_fullsized":"2024-03-09T15:33:20.800266+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:10:41.693365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.15","version":"0'0","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:16.187056+0000","last_change":"2024-03-09T14:46:18.753568+0000","last_active":"2024-03-09T15:32:16.187056+0000","last_peered":"2024-03-09T15:32:16.187056+0000","last_clean":"2024-03-09T15:32:16.187056+0000","last_became_active":"2024-03-09T14:46:18.752413+0000","last_became_peered":"2024-03-09T14:46:18.752413+0000","last_unstale":"2024-03-09T15:32:16.187056+0000","last_undegraded":"2024-03-09T15:32:16.187056+0000","last_fullsized":"2024-03-09T15:32:16.187056+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:09:14.935604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.16","version":"261'52","reported_seq":184,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.788486+0000","last_change":"2024-03-09T14:39:13.263228+0000","last_active":"2024-03-09T15:32:25.788486+0000","last_peered":"2024-03-09T15:32:25.788486+0000","last_clean":"2024-03-09T15:32:25.788486+0000","last_became_active":"2024-03-09T14:39:13.262413+0000","last_became_peered":"2024-03-09T14:39:13.262413+0000","last_unstale":"2024-03-09T15:32:25.788486+0000","last_undegraded":"2024-03-09T15:32:25.788486+0000","last_fullsized":"2024-03-09T15:32:25.788486+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:02:22.902848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":163,"num_write":78,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.10","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.199392+0000","last_change":"2024-03-09T15:09:50.075112+0000","last_active":"2024-03-09T15:33:11.199392+0000","last_peered":"2024-03-09T15:33:11.199392+0000","last_clean":"2024-03-09T15:33:11.199392+0000","last_became_active":"2024-03-09T15:09:50.074274+0000","last_became_peered":"2024-03-09T15:09:50.074274+0000","last_unstale":"2024-03-09T15:33:11.199392+0000","last_undegraded":"2024-03-09T15:33:11.199392+0000","last_fullsized":"2024-03-09T15:33:11.199392+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:34:26.130292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.17","version":"0'0","reported_seq":42,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.788513+0000","last_change":"2024-03-09T14:46:26.894297+0000","last_active":"2024-03-09T15:32:25.788513+0000","last_peered":"2024-03-09T15:32:25.788513+0000","last_clean":"2024-03-09T15:32:25.788513+0000","last_became_active":"2024-03-09T14:46:26.893514+0000","last_became_peered":"2024-03-09T14:46:26.893514+0000","last_unstale":"2024-03-09T15:32:25.788513+0000","last_undegraded":"2024-03-09T15:32:25.788513+0000","last_fullsized":"2024-03-09T15:32:25.788513+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:03:25.018128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.16","version":"0'0","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:15.960497+0000","last_change":"2024-03-09T14:46:18.773133+0000","last_active":"2024-03-09T15:32:15.960497+0000","last_peered":"2024-03-09T15:32:15.960497+0000","last_clean":"2024-03-09T15:32:15.960497+0000","last_became_active":"2024-03-09T14:46:18.769274+0000","last_became_peered":"2024-03-09T14:46:18.769274+0000","last_unstale":"2024-03-09T15:32:15.960497+0000","last_undegraded":"2024-03-09T15:32:15.960497+0000","last_fullsized":"2024-03-09T15:32:15.960497+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:01:15.489364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.15","version":"0'0","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.962879+0000","last_change":"2024-03-09T14:39:13.263907+0000","last_active":"2024-03-09T15:32:25.962879+0000","last_peered":"2024-03-09T15:32:25.962879+0000","last_clean":"2024-03-09T15:32:25.962879+0000","last_became_active":"2024-03-09T14:39:13.262066+0000","last_became_peered":"2024-03-09T14:39:13.262066+0000","last_unstale":"2024-03-09T15:32:25.962879+0000","last_undegraded":"2024-03-09T15:32:25.962879+0000","last_fullsized":"2024-03-09T15:32:25.962879+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:55:26.909612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.11","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.188977+0000","last_change":"2024-03-09T15:09:50.086149+0000","last_active":"2024-03-09T15:33:11.188977+0000","last_peered":"2024-03-09T15:33:11.188977+0000","last_clean":"2024-03-09T15:33:11.188977+0000","last_became_active":"2024-03-09T15:09:50.085346+0000","last_became_peered":"2024-03-09T15:09:50.085346+0000","last_unstale":"2024-03-09T15:33:11.188977+0000","last_undegraded":"2024-03-09T15:33:11.188977+0000","last_fullsized":"2024-03-09T15:33:11.188977+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:54:52.652612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.16","version":"0'0","reported_seq":42,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.175271+0000","last_change":"2024-03-09T14:46:26.818659+0000","last_active":"2024-03-09T15:32:21.175271+0000","last_peered":"2024-03-09T15:32:21.175271+0000","last_clean":"2024-03-09T15:32:21.175271+0000","last_became_active":"2024-03-09T14:46:26.817864+0000","last_became_peered":"2024-03-09T14:46:26.817864+0000","last_unstale":"2024-03-09T15:32:21.175271+0000","last_undegraded":"2024-03-09T15:32:21.175271+0000","last_fullsized":"2024-03-09T15:32:21.175271+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:10:59.964957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.17","version":"0'0","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:15.994554+0000","last_change":"2024-03-09T14:46:18.780789+0000","last_active":"2024-03-09T15:32:15.994554+0000","last_peered":"2024-03-09T15:32:15.994554+0000","last_clean":"2024-03-09T15:32:15.994554+0000","last_became_active":"2024-03-09T14:46:18.776430+0000","last_became_peered":"2024-03-09T14:46:18.776430+0000","last_unstale":"2024-03-09T15:32:15.994554+0000","last_undegraded":"2024-03-09T15:32:15.994554+0000","last_fullsized":"2024-03-09T15:32:15.994554+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:39:21.125955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.14","version":"258'67","reported_seq":270,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:06.198380+0000","last_change":"2024-03-09T14:39:13.253820+0000","last_active":"2024-03-09T15:33:06.198380+0000","last_peered":"2024-03-09T15:33:06.198380+0000","last_clean":"2024-03-09T15:33:06.198380+0000","last_became_active":"2024-03-09T14:39:13.253007+0000","last_became_peered":"2024-03-09T14:39:13.253007+0000","last_unstale":"2024-03-09T15:33:06.198380+0000","last_undegraded":"2024-03-09T15:33:06.198380+0000","last_fullsized":"2024-03-09T15:33:06.198380+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:44:10.310392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":411,"num_read_kb":279,"num_write":87,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.16","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776888+0000","last_change":"2024-03-09T15:09:50.087752+0000","last_active":"2024-03-09T15:31:35.776888+0000","last_peered":"2024-03-09T15:31:35.776888+0000","last_clean":"2024-03-09T15:31:35.776888+0000","last_became_active":"2024-03-09T15:09:50.086173+0000","last_became_peered":"2024-03-09T15:09:50.086173+0000","last_unstale":"2024-03-09T15:31:35.776888+0000","last_undegraded":"2024-03-09T15:31:35.776888+0000","last_fullsized":"2024-03-09T15:31:35.776888+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:43:21.719422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.11","version":"261'7","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:25.986303+0000","last_change":"2024-03-09T14:46:26.817795+0000","last_active":"2024-03-09T15:35:25.986303+0000","last_peered":"2024-03-09T15:35:25.986303+0000","last_clean":"2024-03-09T15:35:25.986303+0000","last_became_active":"2024-03-09T14:46:26.817014+0000","last_became_peered":"2024-03-09T14:46:26.817014+0000","last_unstale":"2024-03-09T15:35:25.986303+0000","last_undegraded":"2024-03-09T15:35:25.986303+0000","last_fullsized":"2024-03-09T15:35:25.986303+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:58:54.836958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":23554,"num_write":11,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.10","version":"261'2","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:01.183449+0000","last_change":"2024-03-09T14:46:18.771622+0000","last_active":"2024-03-09T15:33:01.183449+0000","last_peered":"2024-03-09T15:33:01.183449+0000","last_clean":"2024-03-09T15:33:01.183449+0000","last_became_active":"2024-03-09T14:46:18.770936+0000","last_became_peered":"2024-03-09T14:46:18.770936+0000","last_unstale":"2024-03-09T15:33:01.183449+0000","last_undegraded":"2024-03-09T15:33:01.183449+0000","last_fullsized":"2024-03-09T15:33:01.183449+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:04.807309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":2,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.13","version":"258'16","reported_seq":95,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.196014+0000","last_change":"2024-03-09T14:39:13.266972+0000","last_active":"2024-03-09T15:32:56.196014+0000","last_peered":"2024-03-09T15:32:56.196014+0000","last_clean":"2024-03-09T15:32:56.196014+0000","last_became_active":"2024-03-09T14:39:13.266228+0000","last_became_peered":"2024-03-09T14:39:13.266228+0000","last_unstale":"2024-03-09T15:32:56.196014+0000","last_undegraded":"2024-03-09T15:32:56.196014+0000","last_fullsized":"2024-03-09T15:32:56.196014+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:02:45.648908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":45,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.17","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.984716+0000","last_change":"2024-03-09T15:09:50.075655+0000","last_active":"2024-03-09T15:31:35.984716+0000","last_peered":"2024-03-09T15:31:35.984716+0000","last_clean":"2024-03-09T15:31:35.984716+0000","last_became_active":"2024-03-09T15:09:50.075187+0000","last_became_peered":"2024-03-09T15:09:50.075187+0000","last_unstale":"2024-03-09T15:31:35.984716+0000","last_undegraded":"2024-03-09T15:31:35.984716+0000","last_fullsized":"2024-03-09T15:31:35.984716+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:25:34.672578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.10","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:05.956045+0000","last_change":"2024-03-09T14:46:26.803323+0000","last_active":"2024-03-09T15:33:05.956045+0000","last_peered":"2024-03-09T15:33:05.956045+0000","last_clean":"2024-03-09T15:33:05.956045+0000","last_became_active":"2024-03-09T14:46:26.802476+0000","last_became_peered":"2024-03-09T14:46:26.802476+0000","last_unstale":"2024-03-09T15:33:05.956045+0000","last_undegraded":"2024-03-09T15:33:05.956045+0000","last_fullsized":"2024-03-09T15:33:05.956045+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:04:33.172429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.11","version":"0'0","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.795596+0000","last_change":"2024-03-09T14:46:18.761711+0000","last_active":"2024-03-09T15:33:00.795596+0000","last_peered":"2024-03-09T15:33:00.795596+0000","last_clean":"2024-03-09T15:33:00.795596+0000","last_became_active":"2024-03-09T14:46:18.760959+0000","last_became_peered":"2024-03-09T14:46:18.760959+0000","last_unstale":"2024-03-09T15:33:00.795596+0000","last_undegraded":"2024-03-09T15:33:00.795596+0000","last_fullsized":"2024-03-09T15:33:00.795596+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:08.964116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.12","version":"258'7","reported_seq":77,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.003193+0000","last_change":"2024-03-09T14:39:13.274560+0000","last_active":"2024-03-09T15:32:56.003193+0000","last_peered":"2024-03-09T15:32:56.003193+0000","last_clean":"2024-03-09T15:32:56.003193+0000","last_became_active":"2024-03-09T14:39:13.273740+0000","last_became_peered":"2024-03-09T14:39:13.273740+0000","last_unstale":"2024-03-09T15:32:56.003193+0000","last_undegraded":"2024-03-09T15:32:56.003193+0000","last_fullsized":"2024-03-09T15:32:56.003193+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:17.942276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":26,"num_write":13,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.14","version":"260'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.007791+0000","last_change":"2024-03-09T15:09:50.065240+0000","last_active":"2024-03-09T15:33:16.007791+0000","last_peered":"2024-03-09T15:33:16.007791+0000","last_clean":"2024-03-09T15:33:16.007791+0000","last_became_active":"2024-03-09T15:09:50.064428+0000","last_became_peered":"2024-03-09T15:09:50.064428+0000","last_unstale":"2024-03-09T15:33:16.007791+0000","last_undegraded":"2024-03-09T15:33:16.007791+0000","last_fullsized":"2024-03-09T15:33:16.007791+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:34:02.983911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.13","version":"261'2","reported_seq":47,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.798066+0000","last_change":"2024-03-09T14:46:26.813826+0000","last_active":"2024-03-09T15:33:10.798066+0000","last_peered":"2024-03-09T15:33:10.798066+0000","last_clean":"2024-03-09T15:33:10.798066+0000","last_became_active":"2024-03-09T14:46:26.813088+0000","last_became_peered":"2024-03-09T14:46:26.813088+0000","last_unstale":"2024-03-09T15:33:10.798066+0000","last_undegraded":"2024-03-09T15:33:10.798066+0000","last_fullsized":"2024-03-09T15:33:10.798066+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:37:31.388531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":2,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.12","version":"261'4","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.782261+0000","last_change":"2024-03-09T14:46:18.783155+0000","last_active":"2024-03-09T15:33:00.782261+0000","last_peered":"2024-03-09T15:33:00.782261+0000","last_clean":"2024-03-09T15:33:00.782261+0000","last_became_active":"2024-03-09T14:46:18.782158+0000","last_became_peered":"2024-03-09T14:46:18.782158+0000","last_unstale":"2024-03-09T15:33:00.782261+0000","last_undegraded":"2024-03-09T15:33:00.782261+0000","last_fullsized":"2024-03-09T15:33:00.782261+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:44:00.127768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":11264,"num_write":4,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.11","version":"258'36","reported_seq":148,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.182425+0000","last_change":"2024-03-09T14:39:13.260927+0000","last_active":"2024-03-09T15:32:56.182425+0000","last_peered":"2024-03-09T15:32:56.182425+0000","last_clean":"2024-03-09T15:32:56.182425+0000","last_became_active":"2024-03-09T14:39:13.260219+0000","last_became_peered":"2024-03-09T14:39:13.260219+0000","last_unstale":"2024-03-09T15:32:56.182425+0000","last_undegraded":"2024-03-09T15:32:56.182425+0000","last_fullsized":"2024-03-09T15:32:56.182425+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:27:50.114739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":118,"num_write":57,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.15","version":"261'1","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776826+0000","last_change":"2024-03-09T15:09:50.082323+0000","last_active":"2024-03-09T15:31:35.776826+0000","last_peered":"2024-03-09T15:31:35.776826+0000","last_clean":"2024-03-09T15:31:35.776826+0000","last_became_active":"2024-03-09T15:09:50.081688+0000","last_became_peered":"2024-03-09T15:09:50.081688+0000","last_unstale":"2024-03-09T15:31:35.776826+0000","last_undegraded":"2024-03-09T15:31:35.776826+0000","last_fullsized":"2024-03-09T15:31:35.776826+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:51:04.000384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.12","version":"261'1","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784367+0000","last_change":"2024-03-09T14:46:26.806904+0000","last_active":"2024-03-09T15:33:10.784367+0000","last_peered":"2024-03-09T15:33:10.784367+0000","last_clean":"2024-03-09T15:33:10.784367+0000","last_became_active":"2024-03-09T14:46:26.806155+0000","last_became_peered":"2024-03-09T14:46:26.806155+0000","last_unstale":"2024-03-09T15:33:10.784367+0000","last_undegraded":"2024-03-09T15:33:10.784367+0000","last_fullsized":"2024-03-09T15:33:10.784367+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:30:04.752758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":3072,"num_write":1,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.13","version":"261'2","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:21.217889+0000","last_change":"2024-03-09T14:46:18.783226+0000","last_active":"2024-03-09T15:35:21.217889+0000","last_peered":"2024-03-09T15:35:21.217889+0000","last_clean":"2024-03-09T15:35:21.217889+0000","last_became_active":"2024-03-09T14:46:18.782340+0000","last_became_peered":"2024-03-09T14:46:18.782340+0000","last_unstale":"2024-03-09T15:35:21.217889+0000","last_undegraded":"2024-03-09T15:35:21.217889+0000","last_fullsized":"2024-03-09T15:35:21.217889+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:42:41.572332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":50,"num_write":2,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.10","version":"258'67","reported_seq":256,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.794304+0000","last_change":"2024-03-09T14:39:13.245901+0000","last_active":"2024-03-09T15:32:55.794304+0000","last_peered":"2024-03-09T15:32:55.794304+0000","last_clean":"2024-03-09T15:32:55.794304+0000","last_became_active":"2024-03-09T14:39:13.245107+0000","last_became_peered":"2024-03-09T14:39:13.245107+0000","last_unstale":"2024-03-09T15:32:55.794304+0000","last_undegraded":"2024-03-09T15:32:55.794304+0000","last_fullsized":"2024-03-09T15:32:55.794304+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:25:10.518351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":382,"num_read_kb":270,"num_write":93,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.a","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.978852+0000","last_change":"2024-03-09T15:09:50.080044+0000","last_active":"2024-03-09T15:33:45.978852+0000","last_peered":"2024-03-09T15:33:45.978852+0000","last_clean":"2024-03-09T15:33:45.978852+0000","last_became_active":"2024-03-09T15:09:50.079554+0000","last_became_peered":"2024-03-09T15:09:50.079554+0000","last_unstale":"2024-03-09T15:33:45.978852+0000","last_undegraded":"2024-03-09T15:33:45.978852+0000","last_fullsized":"2024-03-09T15:33:45.978852+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:15:33.287535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.d","version":"261'4","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.806260+0000","last_change":"2024-03-09T14:46:26.813839+0000","last_active":"2024-03-09T15:33:45.806260+0000","last_peered":"2024-03-09T15:33:45.806260+0000","last_clean":"2024-03-09T15:33:45.806260+0000","last_became_active":"2024-03-09T14:46:26.812859+0000","last_became_peered":"2024-03-09T14:46:26.812859+0000","last_unstale":"2024-03-09T15:33:45.806260+0000","last_undegraded":"2024-03-09T15:33:45.806260+0000","last_fullsized":"2024-03-09T15:33:45.806260+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:10:31.220080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":8192,"num_write":9,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.c","version":"261'10","reported_seq":69,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.236677+0000","last_change":"2024-03-09T14:46:18.767068+0000","last_active":"2024-03-09T15:35:51.236677+0000","last_peered":"2024-03-09T15:35:51.236677+0000","last_clean":"2024-03-09T15:35:51.236677+0000","last_became_active":"2024-03-09T14:46:18.766370+0000","last_became_peered":"2024-03-09T14:46:18.766370+0000","last_unstale":"2024-03-09T15:35:51.236677+0000","last_undegraded":"2024-03-09T15:35:51.236677+0000","last_fullsized":"2024-03-09T15:35:51.236677+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:41:36.450438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":20480,"num_write":15,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.f","version":"258'18","reported_seq":99,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:30.802658+0000","last_change":"2024-03-09T14:39:13.271586+0000","last_active":"2024-03-09T15:33:30.802658+0000","last_peered":"2024-03-09T15:33:30.802658+0000","last_clean":"2024-03-09T15:33:30.802658+0000","last_became_active":"2024-03-09T14:39:13.268971+0000","last_became_peered":"2024-03-09T14:39:13.268971+0000","last_unstale":"2024-03-09T15:33:30.802658+0000","last_undegraded":"2024-03-09T15:33:30.802658+0000","last_fullsized":"2024-03-09T15:33:30.802658+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:14:41.597456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.b","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188151+0000","last_change":"2024-03-09T15:09:50.070451+0000","last_active":"2024-03-09T15:32:21.188151+0000","last_peered":"2024-03-09T15:32:21.188151+0000","last_clean":"2024-03-09T15:32:21.188151+0000","last_became_active":"2024-03-09T15:09:50.069508+0000","last_became_peered":"2024-03-09T15:09:50.069508+0000","last_unstale":"2024-03-09T15:32:21.188151+0000","last_undegraded":"2024-03-09T15:32:21.188151+0000","last_fullsized":"2024-03-09T15:32:21.188151+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:39:13.850719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.c","version":"261'2","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.806160+0000","last_change":"2024-03-09T14:46:26.813805+0000","last_active":"2024-03-09T15:33:45.806160+0000","last_peered":"2024-03-09T15:33:45.806160+0000","last_clean":"2024-03-09T15:33:45.806160+0000","last_became_active":"2024-03-09T14:46:26.813009+0000","last_became_peered":"2024-03-09T14:46:26.813009+0000","last_unstale":"2024-03-09T15:33:45.806160+0000","last_undegraded":"2024-03-09T15:33:45.806160+0000","last_fullsized":"2024-03-09T15:33:45.806160+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:34:36.995455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2048,"num_write":7,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.d","version":"261'2","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.976768+0000","last_change":"2024-03-09T14:46:18.771558+0000","last_active":"2024-03-09T15:33:35.976768+0000","last_peered":"2024-03-09T15:33:35.976768+0000","last_clean":"2024-03-09T15:33:35.976768+0000","last_became_active":"2024-03-09T14:46:18.770732+0000","last_became_peered":"2024-03-09T14:46:18.770732+0000","last_unstale":"2024-03-09T15:33:35.976768+0000","last_undegraded":"2024-03-09T15:33:35.976768+0000","last_fullsized":"2024-03-09T15:33:35.976768+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:08:20.215164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.e","version":"261'8","reported_seq":79,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.190234+0000","last_change":"2024-03-09T14:39:13.272393+0000","last_active":"2024-03-09T15:33:16.190234+0000","last_peered":"2024-03-09T15:33:16.190234+0000","last_clean":"2024-03-09T15:33:16.190234+0000","last_became_active":"2024-03-09T14:39:13.271672+0000","last_became_peered":"2024-03-09T14:39:13.271672+0000","last_unstale":"2024-03-09T15:33:16.190234+0000","last_undegraded":"2024-03-09T15:33:16.190234+0000","last_fullsized":"2024-03-09T15:33:16.190234+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:35:02.345192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.8","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.790792+0000","last_change":"2024-03-09T15:09:50.064645+0000","last_active":"2024-03-09T15:33:40.790792+0000","last_peered":"2024-03-09T15:33:40.790792+0000","last_clean":"2024-03-09T15:33:40.790792+0000","last_became_active":"2024-03-09T15:09:50.061662+0000","last_became_peered":"2024-03-09T15:09:50.061662+0000","last_unstale":"2024-03-09T15:33:40.790792+0000","last_undegraded":"2024-03-09T15:33:40.790792+0000","last_fullsized":"2024-03-09T15:33:40.790792+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:32:01.749280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.f","version":"261'5","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.966844+0000","last_change":"2024-03-09T14:46:26.900254+0000","last_active":"2024-03-09T15:33:50.966844+0000","last_peered":"2024-03-09T15:33:50.966844+0000","last_clean":"2024-03-09T15:33:50.966844+0000","last_became_active":"2024-03-09T14:46:26.899430+0000","last_became_peered":"2024-03-09T14:46:26.899430+0000","last_unstale":"2024-03-09T15:33:50.966844+0000","last_undegraded":"2024-03-09T15:33:50.966844+0000","last_fullsized":"2024-03-09T15:33:50.966844+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:58:16.207053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.e","version":"258'3","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.012622+0000","last_change":"2024-03-09T14:46:18.769516+0000","last_active":"2024-03-09T15:33:36.012622+0000","last_peered":"2024-03-09T15:33:36.012622+0000","last_clean":"2024-03-09T15:33:36.012622+0000","last_became_active":"2024-03-09T14:46:18.768694+0000","last_became_peered":"2024-03-09T14:46:18.768694+0000","last_unstale":"2024-03-09T15:33:36.012622+0000","last_undegraded":"2024-03-09T15:33:36.012622+0000","last_fullsized":"2024-03-09T15:33:36.012622+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:10:50.806709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.d","version":"261'43","reported_seq":159,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:15.785461+0000","last_change":"2024-03-09T14:39:13.266337+0000","last_active":"2024-03-09T15:33:15.785461+0000","last_peered":"2024-03-09T15:33:15.785461+0000","last_clean":"2024-03-09T15:33:15.785461+0000","last_became_active":"2024-03-09T14:39:13.257275+0000","last_became_peered":"2024-03-09T14:39:13.257275+0000","last_unstale":"2024-03-09T15:33:15.785461+0000","last_undegraded":"2024-03-09T15:33:15.785461+0000","last_fullsized":"2024-03-09T15:33:15.785461+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:45:48.503144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":175,"num_read_kb":123,"num_write":70,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.9","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:41.206696+0000","last_change":"2024-03-09T15:09:50.077371+0000","last_active":"2024-03-09T15:33:41.206696+0000","last_peered":"2024-03-09T15:33:41.206696+0000","last_clean":"2024-03-09T15:33:41.206696+0000","last_became_active":"2024-03-09T15:09:50.076702+0000","last_became_peered":"2024-03-09T15:09:50.076702+0000","last_unstale":"2024-03-09T15:33:41.206696+0000","last_undegraded":"2024-03-09T15:33:41.206696+0000","last_fullsized":"2024-03-09T15:33:41.206696+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:02:55.914615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.e","version":"261'4","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.806311+0000","last_change":"2024-03-09T14:46:26.818547+0000","last_active":"2024-03-09T15:33:45.806311+0000","last_peered":"2024-03-09T15:33:45.806311+0000","last_clean":"2024-03-09T15:33:45.806311+0000","last_became_active":"2024-03-09T14:46:26.814709+0000","last_became_peered":"2024-03-09T14:46:26.814709+0000","last_unstale":"2024-03-09T15:33:45.806311+0000","last_undegraded":"2024-03-09T15:33:45.806311+0000","last_fullsized":"2024-03-09T15:33:45.806311+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:11:42.597841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":9,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.f","version":"261'4","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.965610+0000","last_change":"2024-03-09T14:46:18.777803+0000","last_active":"2024-03-09T15:33:45.965610+0000","last_peered":"2024-03-09T15:33:45.965610+0000","last_clean":"2024-03-09T15:33:45.965610+0000","last_became_active":"2024-03-09T14:46:18.766000+0000","last_became_peered":"2024-03-09T14:46:18.766000+0000","last_unstale":"2024-03-09T15:33:45.965610+0000","last_undegraded":"2024-03-09T15:33:45.965610+0000","last_fullsized":"2024-03-09T15:33:45.965610+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:41:01.857202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4096,"num_write":9,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.c","version":"0'0","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:15.972703+0000","last_change":"2024-03-09T14:39:13.271296+0000","last_active":"2024-03-09T15:33:15.972703+0000","last_peered":"2024-03-09T15:33:15.972703+0000","last_clean":"2024-03-09T15:33:15.972703+0000","last_became_active":"2024-03-09T14:39:13.270527+0000","last_became_peered":"2024-03-09T14:39:13.270527+0000","last_unstale":"2024-03-09T15:33:15.972703+0000","last_undegraded":"2024-03-09T15:33:15.972703+0000","last_fullsized":"2024-03-09T15:33:15.972703+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:25:04.404490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.e","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.787362+0000","last_change":"2024-03-09T15:09:50.087358+0000","last_active":"2024-03-09T15:32:20.787362+0000","last_peered":"2024-03-09T15:32:20.787362+0000","last_clean":"2024-03-09T15:32:20.787362+0000","last_became_active":"2024-03-09T15:09:50.085548+0000","last_became_peered":"2024-03-09T15:09:50.085548+0000","last_unstale":"2024-03-09T15:32:20.787362+0000","last_undegraded":"2024-03-09T15:32:20.787362+0000","last_fullsized":"2024-03-09T15:32:20.787362+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:13.595041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.9","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:56.169630+0000","last_change":"2024-03-09T14:46:26.802054+0000","last_active":"2024-03-09T15:31:56.169630+0000","last_peered":"2024-03-09T15:31:56.169630+0000","last_clean":"2024-03-09T15:31:56.169630+0000","last_became_active":"2024-03-09T14:46:26.801235+0000","last_became_peered":"2024-03-09T14:46:26.801235+0000","last_unstale":"2024-03-09T15:31:56.169630+0000","last_undegraded":"2024-03-09T15:31:56.169630+0000","last_fullsized":"2024-03-09T15:31:56.169630+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:49:20.412101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.8","version":"0'0","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.988186+0000","last_change":"2024-03-09T14:46:18.775300+0000","last_active":"2024-03-09T15:31:50.988186+0000","last_peered":"2024-03-09T15:31:50.988186+0000","last_clean":"2024-03-09T15:31:50.988186+0000","last_became_active":"2024-03-09T14:46:18.774547+0000","last_became_peered":"2024-03-09T14:46:18.774547+0000","last_unstale":"2024-03-09T15:31:50.988186+0000","last_undegraded":"2024-03-09T15:31:50.988186+0000","last_fullsized":"2024-03-09T15:31:50.988186+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:18:57.376200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.b","version":"258'10","reported_seq":85,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.007679+0000","last_change":"2024-03-09T14:39:13.258020+0000","last_active":"2024-03-09T15:33:16.007679+0000","last_peered":"2024-03-09T15:33:16.007679+0000","last_clean":"2024-03-09T15:33:16.007679+0000","last_became_active":"2024-03-09T14:39:13.257238+0000","last_became_peered":"2024-03-09T14:39:13.257238+0000","last_unstale":"2024-03-09T15:33:16.007679+0000","last_undegraded":"2024-03-09T15:33:16.007679+0000","last_fullsized":"2024-03-09T15:33:16.007679+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:40:13.243772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.f","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.189467+0000","last_change":"2024-03-09T15:09:50.071187+0000","last_active":"2024-03-09T15:32:26.189467+0000","last_peered":"2024-03-09T15:32:26.189467+0000","last_clean":"2024-03-09T15:32:26.189467+0000","last_became_active":"2024-03-09T15:09:50.070526+0000","last_became_peered":"2024-03-09T15:09:50.070526+0000","last_unstale":"2024-03-09T15:32:26.189467+0000","last_undegraded":"2024-03-09T15:32:26.189467+0000","last_fullsized":"2024-03-09T15:32:26.189467+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:03:16.280366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.8","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:01.183115+0000","last_change":"2024-03-09T14:46:26.894478+0000","last_active":"2024-03-09T15:32:01.183115+0000","last_peered":"2024-03-09T15:32:01.183115+0000","last_clean":"2024-03-09T15:32:01.183115+0000","last_became_active":"2024-03-09T14:46:26.893908+0000","last_became_peered":"2024-03-09T14:46:26.893908+0000","last_unstale":"2024-03-09T15:32:01.183115+0000","last_undegraded":"2024-03-09T15:32:01.183115+0000","last_fullsized":"2024-03-09T15:32:01.183115+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:43:51.901456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.9","version":"261'2","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:51.136707+0000","last_change":"2024-03-09T14:46:18.777691+0000","last_active":"2024-03-09T15:31:51.136707+0000","last_peered":"2024-03-09T15:31:51.136707+0000","last_clean":"2024-03-09T15:31:51.136707+0000","last_became_active":"2024-03-09T14:46:18.761367+0000","last_became_peered":"2024-03-09T14:46:18.761367+0000","last_unstale":"2024-03-09T15:31:51.136707+0000","last_undegraded":"2024-03-09T15:31:51.136707+0000","last_fullsized":"2024-03-09T15:31:51.136707+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:44:32.343854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.a","version":"258'23","reported_seq":123,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205291+0000","last_change":"2024-03-09T14:39:13.253482+0000","last_active":"2024-03-09T15:33:36.205291+0000","last_peered":"2024-03-09T15:33:36.205291+0000","last_clean":"2024-03-09T15:33:36.205291+0000","last_became_active":"2024-03-09T14:39:13.252547+0000","last_became_peered":"2024-03-09T14:39:13.252547+0000","last_unstale":"2024-03-09T15:33:36.205291+0000","last_undegraded":"2024-03-09T15:33:36.205291+0000","last_fullsized":"2024-03-09T15:33:36.205291+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:32:42.396388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":86,"num_write":38,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.c","version":"261'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188203+0000","last_change":"2024-03-09T15:09:50.070468+0000","last_active":"2024-03-09T15:32:21.188203+0000","last_peered":"2024-03-09T15:32:21.188203+0000","last_clean":"2024-03-09T15:32:21.188203+0000","last_became_active":"2024-03-09T15:09:50.069324+0000","last_became_peered":"2024-03-09T15:09:50.069324+0000","last_unstale":"2024-03-09T15:32:21.188203+0000","last_undegraded":"2024-03-09T15:32:21.188203+0000","last_fullsized":"2024-03-09T15:32:21.188203+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:23:32.203630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.b","version":"258'3","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.964465+0000","last_change":"2024-03-09T14:46:26.805564+0000","last_active":"2024-03-09T15:33:40.964465+0000","last_peered":"2024-03-09T15:33:40.964465+0000","last_clean":"2024-03-09T15:33:40.964465+0000","last_became_active":"2024-03-09T14:46:26.804615+0000","last_became_peered":"2024-03-09T14:46:26.804615+0000","last_unstale":"2024-03-09T15:33:40.964465+0000","last_undegraded":"2024-03-09T15:33:40.964465+0000","last_fullsized":"2024-03-09T15:33:40.964465+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:54:32.765203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.a","version":"258'3","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:00.990762+0000","last_change":"2024-03-09T14:46:18.769098+0000","last_active":"2024-03-09T15:32:00.990762+0000","last_peered":"2024-03-09T15:32:00.990762+0000","last_clean":"2024-03-09T15:32:00.990762+0000","last_became_active":"2024-03-09T14:46:18.768431+0000","last_became_peered":"2024-03-09T14:46:18.768431+0000","last_unstale":"2024-03-09T15:32:00.990762+0000","last_undegraded":"2024-03-09T15:32:00.990762+0000","last_fullsized":"2024-03-09T15:32:00.990762+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:28:28.024126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.9","version":"261'43","reported_seq":157,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.801289+0000","last_change":"2024-03-09T14:39:13.260724+0000","last_active":"2024-03-09T15:33:25.801289+0000","last_peered":"2024-03-09T15:33:25.801289+0000","last_clean":"2024-03-09T15:33:25.801289+0000","last_became_active":"2024-03-09T14:39:13.259975+0000","last_became_peered":"2024-03-09T14:39:13.259975+0000","last_unstale":"2024-03-09T15:33:25.801289+0000","last_undegraded":"2024-03-09T15:33:25.801289+0000","last_fullsized":"2024-03-09T15:33:25.801289+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:26:32.899703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":169,"num_read_kb":119,"num_write":74,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.d","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.995632+0000","last_change":"2024-03-09T15:09:50.084646+0000","last_active":"2024-03-09T15:32:20.995632+0000","last_peered":"2024-03-09T15:32:20.995632+0000","last_clean":"2024-03-09T15:32:20.995632+0000","last_became_active":"2024-03-09T15:09:50.084074+0000","last_became_peered":"2024-03-09T15:09:50.084074+0000","last_unstale":"2024-03-09T15:32:20.995632+0000","last_undegraded":"2024-03-09T15:32:20.995632+0000","last_fullsized":"2024-03-09T15:32:20.995632+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:16:13.790167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.a","version":"258'3","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:06.184462+0000","last_change":"2024-03-09T14:46:26.826875+0000","last_active":"2024-03-09T15:32:06.184462+0000","last_peered":"2024-03-09T15:32:06.184462+0000","last_clean":"2024-03-09T15:32:06.184462+0000","last_became_active":"2024-03-09T14:46:26.825949+0000","last_became_peered":"2024-03-09T14:46:26.825949+0000","last_unstale":"2024-03-09T15:32:06.184462+0000","last_undegraded":"2024-03-09T15:32:06.184462+0000","last_fullsized":"2024-03-09T15:32:06.184462+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:24:09.628828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.b","version":"261'2","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191328+0000","last_change":"2024-03-09T14:46:18.785518+0000","last_active":"2024-03-09T15:33:36.191328+0000","last_peered":"2024-03-09T15:33:36.191328+0000","last_clean":"2024-03-09T15:33:36.191328+0000","last_became_active":"2024-03-09T14:46:18.784869+0000","last_became_peered":"2024-03-09T14:46:18.784869+0000","last_unstale":"2024-03-09T15:33:36.191328+0000","last_undegraded":"2024-03-09T15:33:36.191328+0000","last_fullsized":"2024-03-09T15:33:36.191328+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:03:42.729066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":2,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.8","version":"258'10","reported_seq":86,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.787542+0000","last_change":"2024-03-09T14:39:13.264924+0000","last_active":"2024-03-09T15:33:25.787542+0000","last_peered":"2024-03-09T15:33:25.787542+0000","last_clean":"2024-03-09T15:33:25.787542+0000","last_became_active":"2024-03-09T14:39:13.264006+0000","last_became_peered":"2024-03-09T14:39:13.264006+0000","last_unstale":"2024-03-09T15:33:25.787542+0000","last_undegraded":"2024-03-09T15:33:25.787542+0000","last_fullsized":"2024-03-09T15:33:25.787542+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:24:57.691583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.2","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.193658+0000","last_change":"2024-03-09T15:09:50.066069+0000","last_active":"2024-03-09T15:32:46.193658+0000","last_peered":"2024-03-09T15:32:46.193658+0000","last_clean":"2024-03-09T15:32:46.193658+0000","last_became_active":"2024-03-09T15:09:50.065487+0000","last_became_peered":"2024-03-09T15:09:50.065487+0000","last_unstale":"2024-03-09T15:32:46.193658+0000","last_undegraded":"2024-03-09T15:32:46.193658+0000","last_fullsized":"2024-03-09T15:32:46.193658+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:16:19.163190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.5","version":"258'2","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:30.997928+0000","last_change":"2024-03-09T14:46:26.813029+0000","last_active":"2024-03-09T15:32:30.997928+0000","last_peered":"2024-03-09T15:32:30.997928+0000","last_clean":"2024-03-09T15:32:30.997928+0000","last_became_active":"2024-03-09T14:46:26.812137+0000","last_became_peered":"2024-03-09T14:46:26.812137+0000","last_unstale":"2024-03-09T15:32:30.997928+0000","last_undegraded":"2024-03-09T15:32:30.997928+0000","last_fullsized":"2024-03-09T15:32:30.997928+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:36:47.037465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4096,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.4","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.185772+0000","last_change":"2024-03-09T14:46:18.752392+0000","last_active":"2024-03-09T15:32:11.185772+0000","last_peered":"2024-03-09T15:32:11.185772+0000","last_clean":"2024-03-09T15:32:11.185772+0000","last_became_active":"2024-03-09T14:46:18.751608+0000","last_became_peered":"2024-03-09T14:46:18.751608+0000","last_unstale":"2024-03-09T15:32:11.185772+0000","last_undegraded":"2024-03-09T15:32:11.185772+0000","last_fullsized":"2024-03-09T15:32:11.185772+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:32:15.618158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.7","version":"261'8","reported_seq":80,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.010433+0000","last_change":"2024-03-09T14:39:13.280605+0000","last_active":"2024-03-09T15:33:26.010433+0000","last_peered":"2024-03-09T15:33:26.010433+0000","last_clean":"2024-03-09T15:33:26.010433+0000","last_became_active":"2024-03-09T14:39:13.273363+0000","last_became_peered":"2024-03-09T14:39:13.273363+0000","last_unstale":"2024-03-09T15:33:26.010433+0000","last_undegraded":"2024-03-09T15:33:26.010433+0000","last_fullsized":"2024-03-09T15:33:26.010433+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:45:47.783708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.3","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.183450+0000","last_change":"2024-03-09T15:09:50.078106+0000","last_active":"2024-03-09T15:32:46.183450+0000","last_peered":"2024-03-09T15:32:46.183450+0000","last_clean":"2024-03-09T15:32:46.183450+0000","last_became_active":"2024-03-09T15:09:50.072488+0000","last_became_peered":"2024-03-09T15:09:50.072488+0000","last_unstale":"2024-03-09T15:32:46.183450+0000","last_undegraded":"2024-03-09T15:32:46.183450+0000","last_fullsized":"2024-03-09T15:32:46.183450+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:20:34.288932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.4","version":"258'2","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.995582+0000","last_change":"2024-03-09T14:46:26.895288+0000","last_active":"2024-03-09T15:32:20.995582+0000","last_peered":"2024-03-09T15:32:20.995582+0000","last_clean":"2024-03-09T15:32:20.995582+0000","last_became_active":"2024-03-09T14:46:26.894506+0000","last_became_peered":"2024-03-09T14:46:26.894506+0000","last_unstale":"2024-03-09T15:32:20.995582+0000","last_undegraded":"2024-03-09T15:32:20.995582+0000","last_fullsized":"2024-03-09T15:32:20.995582+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:23:02.279766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":23,"num_write":2,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.5","version":"258'5","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.962822+0000","last_change":"2024-03-09T14:46:18.771982+0000","last_active":"2024-03-09T15:32:25.962822+0000","last_peered":"2024-03-09T15:32:25.962822+0000","last_clean":"2024-03-09T15:32:25.962822+0000","last_became_active":"2024-03-09T14:46:18.770799+0000","last_became_peered":"2024-03-09T14:46:18.770799+0000","last_unstale":"2024-03-09T15:32:25.962822+0000","last_undegraded":"2024-03-09T15:32:25.962822+0000","last_fullsized":"2024-03-09T15:32:25.962822+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:15:47.749134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.6","version":"261'22","reported_seq":112,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.787743+0000","last_change":"2024-03-09T14:39:13.268293+0000","last_active":"2024-03-09T15:33:25.787743+0000","last_peered":"2024-03-09T15:33:25.787743+0000","last_clean":"2024-03-09T15:33:25.787743+0000","last_became_active":"2024-03-09T14:39:13.266246+0000","last_became_peered":"2024-03-09T14:39:13.266246+0000","last_unstale":"2024-03-09T15:33:25.787743+0000","last_undegraded":"2024-03-09T15:33:25.787743+0000","last_fullsized":"2024-03-09T15:33:25.787743+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:11:53.843138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":65,"num_write":41,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.0","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:00.968771+0000","last_change":"2024-03-09T15:09:50.064673+0000","last_active":"2024-03-09T15:34:00.968771+0000","last_peered":"2024-03-09T15:34:00.968771+0000","last_clean":"2024-03-09T15:34:00.968771+0000","last_became_active":"2024-03-09T15:09:50.063350+0000","last_became_peered":"2024-03-09T15:09:50.063350+0000","last_unstale":"2024-03-09T15:34:00.968771+0000","last_undegraded":"2024-03-09T15:34:00.968771+0000","last_fullsized":"2024-03-09T15:34:00.968771+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:55.461559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.7","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:01.183396+0000","last_change":"2024-03-09T14:46:26.897079+0000","last_active":"2024-03-09T15:32:01.183396+0000","last_peered":"2024-03-09T15:32:01.183396+0000","last_clean":"2024-03-09T15:32:01.183396+0000","last_became_active":"2024-03-09T14:46:26.896503+0000","last_became_peered":"2024-03-09T14:46:26.896503+0000","last_unstale":"2024-03-09T15:32:01.183396+0000","last_undegraded":"2024-03-09T15:32:01.183396+0000","last_fullsized":"2024-03-09T15:32:01.183396+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:57:37.168283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.6","version":"261'2","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:01.200543+0000","last_change":"2024-03-09T14:46:18.779289+0000","last_active":"2024-03-09T15:34:01.200543+0000","last_peered":"2024-03-09T15:34:01.200543+0000","last_clean":"2024-03-09T15:34:01.200543+0000","last_became_active":"2024-03-09T14:46:18.778572+0000","last_became_peered":"2024-03-09T14:46:18.778572+0000","last_unstale":"2024-03-09T15:34:01.200543+0000","last_undegraded":"2024-03-09T15:34:01.200543+0000","last_fullsized":"2024-03-09T15:34:01.200543+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:28:53.930043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.5","version":"258'18","reported_seq":107,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205525+0000","last_change":"2024-03-09T14:39:13.264712+0000","last_active":"2024-03-09T15:33:36.205525+0000","last_peered":"2024-03-09T15:33:36.205525+0000","last_clean":"2024-03-09T15:33:36.205525+0000","last_became_active":"2024-03-09T14:39:13.263833+0000","last_became_peered":"2024-03-09T14:39:13.263833+0000","last_unstale":"2024-03-09T15:33:36.205525+0000","last_undegraded":"2024-03-09T15:33:36.205525+0000","last_fullsized":"2024-03-09T15:33:36.205525+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:10:25.603434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":58,"num_write":29,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.1","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.180226+0000","last_change":"2024-03-09T15:09:50.069405+0000","last_active":"2024-03-09T15:32:46.180226+0000","last_peered":"2024-03-09T15:32:46.180226+0000","last_clean":"2024-03-09T15:32:46.180226+0000","last_became_active":"2024-03-09T15:09:50.067947+0000","last_became_peered":"2024-03-09T15:09:50.067947+0000","last_unstale":"2024-03-09T15:32:46.180226+0000","last_undegraded":"2024-03-09T15:32:46.180226+0000","last_fullsized":"2024-03-09T15:32:46.180226+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:29:17.390497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"67.6","version":"261'2","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:56.169833+0000","last_change":"2024-03-09T14:46:26.809411+0000","last_active":"2024-03-09T15:31:56.169833+0000","last_peered":"2024-03-09T15:31:56.169833+0000","last_clean":"2024-03-09T15:31:56.169833+0000","last_became_active":"2024-03-09T14:46:26.808384+0000","last_became_peered":"2024-03-09T14:46:26.808384+0000","last_unstale":"2024-03-09T15:31:56.169833+0000","last_undegraded":"2024-03-09T15:31:56.169833+0000","last_fullsized":"2024-03-09T15:31:56.169833+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:53:31.797035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90604,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":180,"num_write":2,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.7","version":"261'7","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:51.180691+0000","last_change":"2024-03-09T14:46:18.763286+0000","last_active":"2024-03-09T15:31:51.180691+0000","last_peered":"2024-03-09T15:31:51.180691+0000","last_clean":"2024-03-09T15:31:51.180691+0000","last_became_active":"2024-03-09T14:46:18.762472+0000","last_became_peered":"2024-03-09T14:46:18.762472+0000","last_unstale":"2024-03-09T15:31:51.180691+0000","last_undegraded":"2024-03-09T15:31:51.180691+0000","last_fullsized":"2024-03-09T15:31:51.180691+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:30.905827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":30722,"num_write":16,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.4","version":"258'10","reported_seq":87,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.960925+0000","last_change":"2024-03-09T14:39:13.270957+0000","last_active":"2024-03-09T15:33:25.960925+0000","last_peered":"2024-03-09T15:33:25.960925+0000","last_clean":"2024-03-09T15:33:25.960925+0000","last_became_active":"2024-03-09T14:39:13.270238+0000","last_became_peered":"2024-03-09T14:39:13.270238+0000","last_unstale":"2024-03-09T15:33:25.960925+0000","last_undegraded":"2024-03-09T15:33:25.960925+0000","last_fullsized":"2024-03-09T15:33:25.960925+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:25:19.656005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.6","version":"260'1","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.977825+0000","last_change":"2024-03-09T15:09:50.072623+0000","last_active":"2024-03-09T15:33:40.977825+0000","last_peered":"2024-03-09T15:33:40.977825+0000","last_clean":"2024-03-09T15:33:40.977825+0000","last_became_active":"2024-03-09T15:09:50.072180+0000","last_became_peered":"2024-03-09T15:09:50.072180+0000","last_unstale":"2024-03-09T15:33:40.977825+0000","last_undegraded":"2024-03-09T15:33:40.977825+0000","last_fullsized":"2024-03-09T15:33:40.977825+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:17:47.602511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1","version":"0'0","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:16.142380+0000","last_change":"2024-03-09T14:46:26.799783+0000","last_active":"2024-03-09T15:32:16.142380+0000","last_peered":"2024-03-09T15:32:16.142380+0000","last_clean":"2024-03-09T15:32:16.142380+0000","last_became_active":"2024-03-09T14:46:26.799019+0000","last_became_peered":"2024-03-09T14:46:26.799019+0000","last_unstale":"2024-03-09T15:32:16.142380+0000","last_undegraded":"2024-03-09T15:32:16.142380+0000","last_fullsized":"2024-03-09T15:32:16.142380+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:37:13.610839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.0","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.189237+0000","last_change":"2024-03-09T14:46:18.753025+0000","last_active":"2024-03-09T15:32:26.189237+0000","last_peered":"2024-03-09T15:32:26.189237+0000","last_clean":"2024-03-09T15:32:26.189237+0000","last_became_active":"2024-03-09T14:46:18.752290+0000","last_became_peered":"2024-03-09T14:46:18.752290+0000","last_unstale":"2024-03-09T15:32:26.189237+0000","last_undegraded":"2024-03-09T15:32:26.189237+0000","last_fullsized":"2024-03-09T15:32:26.189237+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:33:49.576573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.3","version":"261'61","reported_seq":215,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.192284+0000","last_change":"2024-03-09T14:39:13.259947+0000","last_active":"2024-03-09T15:33:26.192284+0000","last_peered":"2024-03-09T15:33:26.192284+0000","last_clean":"2024-03-09T15:33:26.192284+0000","last_became_active":"2024-03-09T14:39:13.259125+0000","last_became_peered":"2024-03-09T14:39:13.259125+0000","last_unstale":"2024-03-09T15:33:26.192284+0000","last_undegraded":"2024-03-09T15:33:26.192284+0000","last_fullsized":"2024-03-09T15:33:26.192284+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:52:15.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":281,"num_read_kb":205,"num_write":98,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.7","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:41.014114+0000","last_change":"2024-03-09T15:09:50.064879+0000","last_active":"2024-03-09T15:33:41.014114+0000","last_peered":"2024-03-09T15:33:41.014114+0000","last_clean":"2024-03-09T15:33:41.014114+0000","last_became_active":"2024-03-09T15:09:50.062429+0000","last_became_peered":"2024-03-09T15:09:50.062429+0000","last_unstale":"2024-03-09T15:33:41.014114+0000","last_undegraded":"2024-03-09T15:33:41.014114+0000","last_fullsized":"2024-03-09T15:33:41.014114+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:10:20.402757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.0","version":"261'5","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:36.207793+0000","last_change":"2024-03-09T14:46:26.816110+0000","last_active":"2024-03-09T15:34:36.207793+0000","last_peered":"2024-03-09T15:34:36.207793+0000","last_clean":"2024-03-09T15:34:36.207793+0000","last_became_active":"2024-03-09T14:46:26.815311+0000","last_became_peered":"2024-03-09T14:46:26.815311+0000","last_unstale":"2024-03-09T15:34:36.207793+0000","last_undegraded":"2024-03-09T15:34:36.207793+0000","last_fullsized":"2024-03-09T15:34:36.207793+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:26:21.047330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":16384,"num_write":10,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1","version":"258'5","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.185673+0000","last_change":"2024-03-09T14:46:18.758437+0000","last_active":"2024-03-09T15:32:11.185673+0000","last_peered":"2024-03-09T15:32:11.185673+0000","last_clean":"2024-03-09T15:32:11.185673+0000","last_became_active":"2024-03-09T14:46:18.757694+0000","last_became_peered":"2024-03-09T14:46:18.757694+0000","last_unstale":"2024-03-09T15:32:11.185673+0000","last_undegraded":"2024-03-09T15:32:11.185673+0000","last_fullsized":"2024-03-09T15:32:11.185673+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:47:19.096625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":4098,"num_write":9,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.2","version":"258'20","reported_seq":110,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.188905+0000","last_change":"2024-03-09T14:39:13.265013+0000","last_active":"2024-03-09T15:33:26.188905+0000","last_peered":"2024-03-09T15:33:26.188905+0000","last_clean":"2024-03-09T15:33:26.188905+0000","last_became_active":"2024-03-09T14:39:13.264203+0000","last_became_peered":"2024-03-09T14:39:13.264203+0000","last_unstale":"2024-03-09T15:33:26.188905+0000","last_undegraded":"2024-03-09T15:33:26.188905+0000","last_fullsized":"2024-03-09T15:33:26.188905+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:46:43.366482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":65,"num_write":31,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.4","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.001035+0000","last_change":"2024-03-09T15:09:50.070703+0000","last_active":"2024-03-09T15:32:46.001035+0000","last_peered":"2024-03-09T15:32:46.001035+0000","last_clean":"2024-03-09T15:32:46.001035+0000","last_became_active":"2024-03-09T15:09:50.066222+0000","last_became_peered":"2024-03-09T15:09:50.066222+0000","last_unstale":"2024-03-09T15:32:46.001035+0000","last_undegraded":"2024-03-09T15:32:46.001035+0000","last_fullsized":"2024-03-09T15:32:46.001035+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:54:19.245692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.3","version":"0'0","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.774297+0000","last_change":"2024-03-09T14:46:26.826931+0000","last_active":"2024-03-09T15:32:20.774297+0000","last_peered":"2024-03-09T15:32:20.774297+0000","last_clean":"2024-03-09T15:32:20.774297+0000","last_became_active":"2024-03-09T14:46:26.826105+0000","last_became_peered":"2024-03-09T14:46:26.826105+0000","last_unstale":"2024-03-09T15:32:20.774297+0000","last_undegraded":"2024-03-09T15:32:20.774297+0000","last_fullsized":"2024-03-09T15:32:20.774297+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:19:31.840994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.2","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.141234+0000","last_change":"2024-03-09T14:46:18.749838+0000","last_active":"2024-03-09T15:32:11.141234+0000","last_peered":"2024-03-09T15:32:11.141234+0000","last_clean":"2024-03-09T15:32:11.141234+0000","last_became_active":"2024-03-09T14:46:18.749091+0000","last_became_peered":"2024-03-09T14:46:18.749091+0000","last_unstale":"2024-03-09T15:32:11.141234+0000","last_undegraded":"2024-03-09T15:32:11.141234+0000","last_fullsized":"2024-03-09T15:32:11.141234+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:47:14.332817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1","version":"261'25","reported_seq":125,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.787625+0000","last_change":"2024-03-09T14:39:13.257789+0000","last_active":"2024-03-09T15:33:25.787625+0000","last_peered":"2024-03-09T15:33:25.787625+0000","last_clean":"2024-03-09T15:33:25.787625+0000","last_became_active":"2024-03-09T14:39:13.255589+0000","last_became_peered":"2024-03-09T14:39:13.255589+0000","last_unstale":"2024-03-09T15:33:25.787625+0000","last_undegraded":"2024-03-09T15:33:25.787625+0000","last_fullsized":"2024-03-09T15:33:25.787625+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:53:30.259823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":79,"num_write":35,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.5","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:51.194924+0000","last_change":"2024-03-09T15:09:50.083562+0000","last_active":"2024-03-09T15:32:51.194924+0000","last_peered":"2024-03-09T15:32:51.194924+0000","last_clean":"2024-03-09T15:32:51.194924+0000","last_became_active":"2024-03-09T15:09:50.083064+0000","last_became_peered":"2024-03-09T15:09:50.083064+0000","last_unstale":"2024-03-09T15:32:51.194924+0000","last_undegraded":"2024-03-09T15:32:51.194924+0000","last_fullsized":"2024-03-09T15:32:51.194924+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:25:31.798282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.2","version":"258'3","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:16.174077+0000","last_change":"2024-03-09T14:46:26.808768+0000","last_active":"2024-03-09T15:32:16.174077+0000","last_peered":"2024-03-09T15:32:16.174077+0000","last_clean":"2024-03-09T15:32:16.174077+0000","last_became_active":"2024-03-09T14:46:26.807746+0000","last_became_peered":"2024-03-09T14:46:26.807746+0000","last_unstale":"2024-03-09T15:32:16.174077+0000","last_undegraded":"2024-03-09T15:32:16.174077+0000","last_fullsized":"2024-03-09T15:32:16.174077+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:58:17.140698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1024,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.3","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:10.772267+0000","last_change":"2024-03-09T14:46:18.765863+0000","last_active":"2024-03-09T15:32:10.772267+0000","last_peered":"2024-03-09T15:32:10.772267+0000","last_clean":"2024-03-09T15:32:10.772267+0000","last_became_active":"2024-03-09T14:46:18.765077+0000","last_became_peered":"2024-03-09T14:46:18.765077+0000","last_unstale":"2024-03-09T15:32:10.772267+0000","last_undegraded":"2024-03-09T15:32:10.772267+0000","last_fullsized":"2024-03-09T15:32:10.772267+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:26:58.084480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.0","version":"250'25","reported_seq":130,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.202779+0000","last_change":"2024-03-09T14:39:13.269464+0000","last_active":"2024-03-09T15:33:26.202779+0000","last_peered":"2024-03-09T15:33:26.202779+0000","last_clean":"2024-03-09T15:33:26.202779+0000","last_became_active":"2024-03-09T14:39:13.268553+0000","last_became_peered":"2024-03-09T14:39:13.268553+0000","last_unstale":"2024-03-09T15:33:26.202779+0000","last_undegraded":"2024-03-09T15:33:26.202779+0000","last_fullsized":"2024-03-09T15:33:26.202779+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:01:17.783740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":97,"num_write":32,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"261'102","reported_seq":659,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.187916+0000","last_change":"2024-03-09T14:28:32.340306+0000","last_active":"2024-03-09T15:36:06.187916+0000","last_peered":"2024-03-09T15:36:06.187916+0000","last_clean":"2024-03-09T15:36:06.187916+0000","last_became_active":"2024-03-09T14:28:32.339785+0000","last_became_peered":"2024-03-09T14:28:32.339785+0000","last_unstale":"2024-03-09T15:36:06.187916+0000","last_undegraded":"2024-03-09T15:36:06.187916+0000","last_fullsized":"2024-03-09T15:36:06.187916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:39:28.870360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":925441,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":118,"num_read_kb":5377,"num_write":361,"num_write_kb":16955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"261'146","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194366+0000","last_change":"2024-03-09T14:28:34.304676+0000","last_active":"2024-03-09T15:33:36.194366+0000","last_peered":"2024-03-09T15:33:36.194366+0000","last_clean":"2024-03-09T15:33:36.194366+0000","last_became_active":"2024-03-09T14:28:34.303980+0000","last_became_peered":"2024-03-09T14:28:34.303980+0000","last_unstale":"2024-03-09T15:33:36.194366+0000","last_undegraded":"2024-03-09T15:33:36.194366+0000","last_fullsized":"2024-03-09T15:33:36.194366+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:12:53.886509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":546,"num_read_kb":417,"num_write":228,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"261'119","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.937233+0000","last_change":"2024-03-09T14:28:32.307225+0000","last_active":"2024-03-09T15:36:05.937233+0000","last_peered":"2024-03-09T15:36:05.937233+0000","last_clean":"2024-03-09T15:36:05.937233+0000","last_became_active":"2024-03-09T14:28:32.306840+0000","last_became_peered":"2024-03-09T14:28:32.306840+0000","last_unstale":"2024-03-09T15:36:05.937233+0000","last_undegraded":"2024-03-09T15:36:05.937233+0000","last_fullsized":"2024-03-09T15:36:05.937233+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":100,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:46:26.657945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3560140,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":140,"num_read_kb":3004,"num_write":507,"num_write_kb":19720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"261'89","reported_seq":655,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:39.894721+0000","last_change":"2024-03-09T14:28:34.324788+0000","last_active":"2024-03-09T15:34:39.894721+0000","last_peered":"2024-03-09T15:34:39.894721+0000","last_clean":"2024-03-09T15:34:39.894721+0000","last_became_active":"2024-03-09T14:28:34.324053+0000","last_became_peered":"2024-03-09T14:28:34.324053+0000","last_unstale":"2024-03-09T15:34:39.894721+0000","last_undegraded":"2024-03-09T15:34:39.894721+0000","last_fullsized":"2024-03-09T15:34:39.894721+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:16:16.817770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":372,"num_read_kb":300,"num_write":141,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3d","version":"261'135","reported_seq":695,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.959352+0000","last_change":"2024-03-09T14:28:32.292874+0000","last_active":"2024-03-09T15:36:05.959352+0000","last_peered":"2024-03-09T15:36:05.959352+0000","last_clean":"2024-03-09T15:36:05.959352+0000","last_became_active":"2024-03-09T14:28:32.286966+0000","last_became_peered":"2024-03-09T14:28:32.286966+0000","last_unstale":"2024-03-09T15:36:05.959352+0000","last_undegraded":"2024-03-09T15:36:05.959352+0000","last_fullsized":"2024-03-09T15:36:05.959352+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":100,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:34.280241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291735,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":170,"num_read_kb":14171,"num_write":587,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"261'157","reported_seq":718,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191267+0000","last_change":"2024-03-09T14:28:34.307655+0000","last_active":"2024-03-09T15:33:36.191267+0000","last_peered":"2024-03-09T15:33:36.191267+0000","last_clean":"2024-03-09T15:33:36.191267+0000","last_became_active":"2024-03-09T14:28:34.306764+0000","last_became_peered":"2024-03-09T14:28:34.306764+0000","last_unstale":"2024-03-09T15:33:36.191267+0000","last_undegraded":"2024-03-09T15:33:36.191267+0000","last_fullsized":"2024-03-09T15:33:36.191267+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:19:26.321622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":481,"num_write":261,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3c","version":"261'177","reported_seq":769,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.211122+0000","last_change":"2024-03-09T14:28:32.310328+0000","last_active":"2024-03-09T15:36:06.211122+0000","last_peered":"2024-03-09T15:36:06.211122+0000","last_clean":"2024-03-09T15:36:06.211122+0000","last_became_active":"2024-03-09T14:28:32.305615+0000","last_became_peered":"2024-03-09T14:28:32.305615+0000","last_unstale":"2024-03-09T15:36:06.211122+0000","last_undegraded":"2024-03-09T15:36:06.211122+0000","last_fullsized":"2024-03-09T15:36:06.211122+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":100,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:45:12.810858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10569896,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":243,"num_read_kb":28937,"num_write":784,"num_write_kb":45835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"261'156","reported_seq":732,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.012477+0000","last_change":"2024-03-09T14:28:34.316937+0000","last_active":"2024-03-09T15:33:36.012477+0000","last_peered":"2024-03-09T15:33:36.012477+0000","last_clean":"2024-03-09T15:33:36.012477+0000","last_became_active":"2024-03-09T14:28:34.316500+0000","last_became_peered":"2024-03-09T14:28:34.316500+0000","last_unstale":"2024-03-09T15:33:36.012477+0000","last_undegraded":"2024-03-09T15:33:36.012477+0000","last_fullsized":"2024-03-09T15:33:36.012477+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":100,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:11:35.701163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":622,"num_read_kb":488,"num_write":257,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3b","version":"261'135","reported_seq":797,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.227826+0000","last_change":"2024-03-09T14:28:32.317675+0000","last_active":"2024-03-09T15:36:06.227826+0000","last_peered":"2024-03-09T15:36:06.227826+0000","last_clean":"2024-03-09T15:36:06.227826+0000","last_became_active":"2024-03-09T14:28:32.314923+0000","last_became_peered":"2024-03-09T14:28:32.314923+0000","last_unstale":"2024-03-09T15:36:06.227826+0000","last_undegraded":"2024-03-09T15:36:06.227826+0000","last_fullsized":"2024-03-09T15:36:06.227826+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":100,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:29:14.419941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1918192,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":282,"num_read_kb":17282,"num_write":502,"num_write_kb":14747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3c","version":"261'124","reported_seq":706,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.781267+0000","last_change":"2024-03-09T14:28:34.315343+0000","last_active":"2024-03-09T15:32:55.781267+0000","last_peered":"2024-03-09T15:32:55.781267+0000","last_clean":"2024-03-09T15:32:55.781267+0000","last_became_active":"2024-03-09T14:28:34.314617+0000","last_became_peered":"2024-03-09T14:28:34.314617+0000","last_unstale":"2024-03-09T15:32:55.781267+0000","last_undegraded":"2024-03-09T15:32:55.781267+0000","last_fullsized":"2024-03-09T15:32:55.781267+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:05:14.469183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":493,"num_read_kb":395,"num_write":185,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3a","version":"261'128","reported_seq":661,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.870081+0000","last_change":"2024-03-09T14:29:06.433441+0000","last_active":"2024-03-09T15:36:05.870081+0000","last_peered":"2024-03-09T15:36:05.870081+0000","last_clean":"2024-03-09T15:36:05.870081+0000","last_became_active":"2024-03-09T14:29:06.433145+0000","last_became_peered":"2024-03-09T14:29:06.433145+0000","last_unstale":"2024-03-09T15:36:05.870081+0000","last_undegraded":"2024-03-09T15:36:05.870081+0000","last_fullsized":"2024-03-09T15:36:05.870081+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:01:58.493721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3195196,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":189,"num_read_kb":6443,"num_write":585,"num_write_kb":33049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"261'274","reported_seq":939,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.185860+0000","last_change":"2024-03-09T14:28:34.314501+0000","last_active":"2024-03-09T15:33:11.185860+0000","last_peered":"2024-03-09T15:33:11.185860+0000","last_clean":"2024-03-09T15:33:11.185860+0000","last_became_active":"2024-03-09T14:28:34.310343+0000","last_became_peered":"2024-03-09T14:28:34.310343+0000","last_unstale":"2024-03-09T15:33:11.185860+0000","last_undegraded":"2024-03-09T15:33:11.185860+0000","last_fullsized":"2024-03-09T15:33:11.185860+0000","mapping_epoch":22,"log_start":"247'200","ondisk_log_start":"247'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":200,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:36:50.027757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1205,"num_read_kb":1028,"num_write":418,"num_write_kb":253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"261'123","reported_seq":679,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.218061+0000","last_change":"2024-03-09T14:28:32.314711+0000","last_active":"2024-03-09T15:36:06.218061+0000","last_peered":"2024-03-09T15:36:06.218061+0000","last_clean":"2024-03-09T15:36:06.218061+0000","last_became_active":"2024-03-09T14:28:32.314258+0000","last_became_peered":"2024-03-09T14:28:32.314258+0000","last_unstale":"2024-03-09T15:36:06.218061+0000","last_undegraded":"2024-03-09T15:36:06.218061+0000","last_fullsized":"2024-03-09T15:36:06.218061+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":100,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:21:00.179690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262376,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":159,"num_read_kb":17581,"num_write":456,"num_write_kb":22929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3e","version":"261'44","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.189153+0000","last_change":"2024-03-09T14:28:34.323080+0000","last_active":"2024-03-09T15:33:11.189153+0000","last_peered":"2024-03-09T15:33:11.189153+0000","last_clean":"2024-03-09T15:33:11.189153+0000","last_became_active":"2024-03-09T14:28:34.322206+0000","last_became_peered":"2024-03-09T14:28:34.322206+0000","last_unstale":"2024-03-09T15:33:11.189153+0000","last_undegraded":"2024-03-09T15:33:11.189153+0000","last_fullsized":"2024-03-09T15:33:11.189153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:55:41.173766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":201,"num_read_kb":171,"num_write":60,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"261'128","reported_seq":674,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.803635+0000","last_change":"2024-03-09T14:28:32.308873+0000","last_active":"2024-03-09T15:33:35.803635+0000","last_peered":"2024-03-09T15:33:35.803635+0000","last_clean":"2024-03-09T15:33:35.803635+0000","last_became_active":"2024-03-09T14:28:32.308522+0000","last_became_peered":"2024-03-09T14:28:32.308522+0000","last_unstale":"2024-03-09T15:33:35.803635+0000","last_undegraded":"2024-03-09T15:33:35.803635+0000","last_fullsized":"2024-03-09T15:33:35.803635+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:10:42.961664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6817986,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":153,"num_read_kb":14130,"num_write":524,"num_write_kb":36844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"261'52","reported_seq":603,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.971792+0000","last_change":"2024-03-09T14:28:34.321740+0000","last_active":"2024-03-09T15:33:10.971792+0000","last_peered":"2024-03-09T15:33:10.971792+0000","last_clean":"2024-03-09T15:33:10.971792+0000","last_became_active":"2024-03-09T14:28:34.320646+0000","last_became_peered":"2024-03-09T14:28:34.320646+0000","last_unstale":"2024-03-09T15:33:10.971792+0000","last_undegraded":"2024-03-09T15:33:10.971792+0000","last_fullsized":"2024-03-09T15:33:10.971792+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:32:54.751077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":174,"num_write":79,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"261'157","reported_seq":741,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.897610+0000","last_change":"2024-03-09T14:28:34.307366+0000","last_active":"2024-03-09T15:31:36.897610+0000","last_peered":"2024-03-09T15:31:36.897610+0000","last_clean":"2024-03-09T15:31:36.897610+0000","last_became_active":"2024-03-09T14:28:34.306592+0000","last_became_peered":"2024-03-09T14:28:34.306592+0000","last_unstale":"2024-03-09T15:31:36.897610+0000","last_undegraded":"2024-03-09T15:31:36.897610+0000","last_fullsized":"2024-03-09T15:31:36.897610+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:38:03.684472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":599,"num_read_kb":473,"num_write":232,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"261'147","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.857715+0000","last_change":"2024-03-09T14:29:06.438746+0000","last_active":"2024-03-09T15:36:05.857715+0000","last_peered":"2024-03-09T15:36:05.857715+0000","last_clean":"2024-03-09T15:36:05.857715+0000","last_became_active":"2024-03-09T14:29:06.438523+0000","last_became_peered":"2024-03-09T14:29:06.438523+0000","last_unstale":"2024-03-09T15:36:05.857715+0000","last_undegraded":"2024-03-09T15:36:05.857715+0000","last_fullsized":"2024-03-09T15:36:05.857715+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":100,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:59.554428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728952,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":279,"num_read_kb":1034,"num_write":583,"num_write_kb":27895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.31","version":"261'88","reported_seq":642,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.176622+0000","last_change":"2024-03-09T14:28:34.322326+0000","last_active":"2024-03-09T15:31:36.176622+0000","last_peered":"2024-03-09T15:31:36.176622+0000","last_clean":"2024-03-09T15:31:36.176622+0000","last_became_active":"2024-03-09T14:28:34.321352+0000","last_became_peered":"2024-03-09T14:28:34.321352+0000","last_unstale":"2024-03-09T15:31:36.176622+0000","last_undegraded":"2024-03-09T15:31:36.176622+0000","last_fullsized":"2024-03-09T15:31:36.176622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:00:49.569838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":342,"num_read_kb":270,"num_write":126,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"261'105","reported_seq":646,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.762868+0000","last_change":"2024-03-09T14:28:32.337035+0000","last_active":"2024-03-09T15:31:25.762868+0000","last_peered":"2024-03-09T15:31:25.762868+0000","last_clean":"2024-03-09T15:31:25.762868+0000","last_became_active":"2024-03-09T14:28:32.334208+0000","last_became_peered":"2024-03-09T14:28:32.334208+0000","last_unstale":"2024-03-09T15:31:25.762868+0000","last_undegraded":"2024-03-09T15:31:25.762868+0000","last_fullsized":"2024-03-09T15:31:25.762868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:15:51.011829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262534,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":117,"num_read_kb":128,"num_write":382,"num_write_kb":22112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"261'62","reported_seq":608,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:46.170836+0000","last_change":"2024-03-09T14:28:34.299187+0000","last_active":"2024-03-09T15:31:46.170836+0000","last_peered":"2024-03-09T15:31:46.170836+0000","last_clean":"2024-03-09T15:31:46.170836+0000","last_became_active":"2024-03-09T14:28:34.298428+0000","last_became_peered":"2024-03-09T14:28:34.298428+0000","last_unstale":"2024-03-09T15:31:46.170836+0000","last_undegraded":"2024-03-09T15:31:46.170836+0000","last_fullsized":"2024-03-09T15:31:46.170836+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:07.549211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":202,"num_write":92,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"261'96","reported_seq":651,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:43.752932+0000","last_change":"2024-03-09T14:28:32.333004+0000","last_active":"2024-03-09T15:34:43.752932+0000","last_peered":"2024-03-09T15:34:43.752932+0000","last_clean":"2024-03-09T15:34:43.752932+0000","last_became_active":"2024-03-09T14:28:32.318963+0000","last_became_peered":"2024-03-09T14:28:32.318963+0000","last_unstale":"2024-03-09T15:34:43.752932+0000","last_undegraded":"2024-03-09T15:34:43.752932+0000","last_fullsized":"2024-03-09T15:34:43.752932+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:52:04.706747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208569,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":146,"num_read_kb":1700,"num_write":417,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.33","version":"261'102","reported_seq":663,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.950025+0000","last_change":"2024-03-09T14:28:34.320630+0000","last_active":"2024-03-09T15:31:25.950025+0000","last_peered":"2024-03-09T15:31:25.950025+0000","last_clean":"2024-03-09T15:31:25.950025+0000","last_became_active":"2024-03-09T14:28:34.320331+0000","last_became_peered":"2024-03-09T14:28:34.320331+0000","last_unstale":"2024-03-09T15:31:25.950025+0000","last_undegraded":"2024-03-09T15:31:25.950025+0000","last_fullsized":"2024-03-09T15:31:25.950025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:03:19.130527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":320,"num_write":144,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"261'114","reported_seq":666,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.967812+0000","last_change":"2024-03-09T14:28:32.338458+0000","last_active":"2024-03-09T15:36:05.967812+0000","last_peered":"2024-03-09T15:36:05.967812+0000","last_clean":"2024-03-09T15:36:05.967812+0000","last_became_active":"2024-03-09T14:28:32.337934+0000","last_became_peered":"2024-03-09T14:28:32.337934+0000","last_unstale":"2024-03-09T15:36:05.967812+0000","last_undegraded":"2024-03-09T15:36:05.967812+0000","last_fullsized":"2024-03-09T15:36:05.967812+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:15:21.515518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554899,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":147,"num_read_kb":12763,"num_write":449,"num_write_kb":21372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"261'69","reported_seq":605,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.950065+0000","last_change":"2024-03-09T14:28:34.311072+0000","last_active":"2024-03-09T15:31:25.950065+0000","last_peered":"2024-03-09T15:31:25.950065+0000","last_clean":"2024-03-09T15:31:25.950065+0000","last_became_active":"2024-03-09T14:28:34.310703+0000","last_became_peered":"2024-03-09T14:28:34.310703+0000","last_unstale":"2024-03-09T15:31:25.950065+0000","last_undegraded":"2024-03-09T15:31:25.950065+0000","last_fullsized":"2024-03-09T15:31:25.950065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:16:09.385282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":272,"num_read_kb":220,"num_write":109,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"261'133","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.964740+0000","last_change":"2024-03-09T14:28:32.337031+0000","last_active":"2024-03-09T15:31:41.964740+0000","last_peered":"2024-03-09T15:31:41.964740+0000","last_clean":"2024-03-09T15:31:41.964740+0000","last_became_active":"2024-03-09T14:28:32.336636+0000","last_became_peered":"2024-03-09T14:28:32.336636+0000","last_unstale":"2024-03-09T15:31:41.964740+0000","last_undegraded":"2024-03-09T15:31:41.964740+0000","last_fullsized":"2024-03-09T15:31:41.964740+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:44:26.611692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262536,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":207,"num_read_kb":12490,"num_write":554,"num_write_kb":28462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"261'87","reported_seq":637,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162736+0000","last_change":"2024-03-09T14:28:34.316215+0000","last_active":"2024-03-09T15:31:26.162736+0000","last_peered":"2024-03-09T15:31:26.162736+0000","last_clean":"2024-03-09T15:31:26.162736+0000","last_became_active":"2024-03-09T14:28:34.314665+0000","last_became_peered":"2024-03-09T14:28:34.314665+0000","last_unstale":"2024-03-09T15:31:26.162736+0000","last_undegraded":"2024-03-09T15:31:26.162736+0000","last_fullsized":"2024-03-09T15:31:26.162736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:15.743713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":335,"num_read_kb":265,"num_write":124,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"261'107","reported_seq":668,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.937008+0000","last_change":"2024-03-09T14:28:32.298846+0000","last_active":"2024-03-09T15:36:05.937008+0000","last_peered":"2024-03-09T15:36:05.937008+0000","last_clean":"2024-03-09T15:36:05.937008+0000","last_became_active":"2024-03-09T14:28:32.298428+0000","last_became_peered":"2024-03-09T14:28:32.298428+0000","last_unstale":"2024-03-09T15:36:05.937008+0000","last_undegraded":"2024-03-09T15:36:05.937008+0000","last_fullsized":"2024-03-09T15:36:05.937008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:18:37.829113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583035,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":178,"num_read_kb":31150,"num_write":451,"num_write_kb":37586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"261'76","reported_seq":581,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:07.654968+0000","last_change":"2024-03-09T14:28:42.404373+0000","last_active":"2024-03-09T15:35:07.654968+0000","last_peered":"2024-03-09T15:35:07.654968+0000","last_clean":"2024-03-09T15:35:07.654968+0000","last_became_active":"2024-03-09T14:28:42.403553+0000","last_became_peered":"2024-03-09T14:28:42.403553+0000","last_unstale":"2024-03-09T15:35:07.654968+0000","last_undegraded":"2024-03-09T15:35:07.654968+0000","last_fullsized":"2024-03-09T15:35:07.654968+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:16:41.740933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":84,"num_write":184,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"261'98","reported_seq":664,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.780525+0000","last_change":"2024-03-09T14:29:06.439713+0000","last_active":"2024-03-09T15:31:50.780525+0000","last_peered":"2024-03-09T15:31:50.780525+0000","last_clean":"2024-03-09T15:31:50.780525+0000","last_became_active":"2024-03-09T14:29:06.439359+0000","last_became_peered":"2024-03-09T14:29:06.439359+0000","last_unstale":"2024-03-09T15:31:50.780525+0000","last_undegraded":"2024-03-09T15:31:50.780525+0000","last_fullsized":"2024-03-09T15:31:50.780525+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:01:04.305529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":413,"num_read_kb":333,"num_write":149,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"261'154","reported_seq":775,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.211631+0000","last_change":"2024-03-09T14:28:32.330235+0000","last_active":"2024-03-09T15:36:06.211631+0000","last_peered":"2024-03-09T15:36:06.211631+0000","last_clean":"2024-03-09T15:36:06.211631+0000","last_became_active":"2024-03-09T14:28:32.329457+0000","last_became_peered":"2024-03-09T14:28:32.329457+0000","last_unstale":"2024-03-09T15:36:06.211631+0000","last_undegraded":"2024-03-09T15:36:06.211631+0000","last_fullsized":"2024-03-09T15:36:06.211631+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":100,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:30:37.270692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5813235,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":252,"num_read_kb":141328,"num_write":592,"num_write_kb":37022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.166374+0000","last_change":"2024-03-09T14:28:36.321332+0000","last_active":"2024-03-09T15:31:26.166374+0000","last_peered":"2024-03-09T15:31:26.166374+0000","last_clean":"2024-03-09T15:31:26.166374+0000","last_became_active":"2024-03-09T14:28:36.320603+0000","last_became_peered":"2024-03-09T14:28:36.320603+0000","last_unstale":"2024-03-09T15:31:26.166374+0000","last_undegraded":"2024-03-09T15:31:26.166374+0000","last_fullsized":"2024-03-09T15:31:26.166374+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:20:34.317764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"261'226","reported_seq":832,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.691237+0000","last_change":"2024-03-09T14:28:38.327507+0000","last_active":"2024-03-09T15:35:58.691237+0000","last_peered":"2024-03-09T15:35:58.691237+0000","last_clean":"2024-03-09T15:35:58.691237+0000","last_became_active":"2024-03-09T14:28:38.326826+0000","last_became_peered":"2024-03-09T14:28:38.326826+0000","last_unstale":"2024-03-09T15:35:58.691237+0000","last_undegraded":"2024-03-09T15:35:58.691237+0000","last_fullsized":"2024-03-09T15:35:58.691237+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":200,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:46:18.261500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24948,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":356,"num_read_kb":580,"num_write":246,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.175558+0000","last_change":"2024-03-09T14:28:40.359830+0000","last_active":"2024-03-09T15:31:31.175558+0000","last_peered":"2024-03-09T15:31:31.175558+0000","last_clean":"2024-03-09T15:31:31.175558+0000","last_became_active":"2024-03-09T14:28:40.358954+0000","last_became_peered":"2024-03-09T14:28:40.358954+0000","last_unstale":"2024-03-09T15:31:31.175558+0000","last_undegraded":"2024-03-09T15:31:31.175558+0000","last_fullsized":"2024-03-09T15:31:31.175558+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:34.169828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"261'770","reported_seq":2383,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:53.987578+0000","last_change":"2024-03-09T14:28:42.395382+0000","last_active":"2024-03-09T15:35:53.987578+0000","last_peered":"2024-03-09T15:35:53.987578+0000","last_clean":"2024-03-09T15:35:53.987578+0000","last_became_active":"2024-03-09T14:28:42.394691+0000","last_became_peered":"2024-03-09T14:28:42.394691+0000","last_unstale":"2024-03-09T15:35:53.987578+0000","last_undegraded":"2024-03-09T15:35:53.987578+0000","last_fullsized":"2024-03-09T15:35:53.987578+0000","mapping_epoch":30,"log_start":"261'700","ondisk_log_start":"261'700","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":700,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:48.486613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":612,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2611,"num_read_kb":2601,"num_write":1560,"num_write_kb":569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"261'40","reported_seq":578,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:28.764187+0000","last_change":"2024-03-09T14:28:34.326083+0000","last_active":"2024-03-09T15:31:28.764187+0000","last_peered":"2024-03-09T15:31:28.764187+0000","last_clean":"2024-03-09T15:31:28.764187+0000","last_became_active":"2024-03-09T14:28:34.324569+0000","last_became_peered":"2024-03-09T14:28:34.324569+0000","last_unstale":"2024-03-09T15:31:28.764187+0000","last_undegraded":"2024-03-09T15:31:28.764187+0000","last_fullsized":"2024-03-09T15:31:28.764187+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:18.214769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":168,"num_read_kb":136,"num_write":58,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"261'145","reported_seq":715,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.866230+0000","last_change":"2024-03-09T14:28:32.310039+0000","last_active":"2024-03-09T15:36:05.866230+0000","last_peered":"2024-03-09T15:36:05.866230+0000","last_clean":"2024-03-09T15:36:05.866230+0000","last_became_active":"2024-03-09T14:28:32.309536+0000","last_became_peered":"2024-03-09T14:28:32.309536+0000","last_unstale":"2024-03-09T15:36:05.866230+0000","last_undegraded":"2024-03-09T15:36:05.866230+0000","last_fullsized":"2024-03-09T15:36:05.866230+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:28:43.638731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5530768,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":194,"num_read_kb":29533,"num_write":628,"num_write_kb":30801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.780570+0000","last_change":"2024-03-09T14:28:36.320450+0000","last_active":"2024-03-09T15:31:50.780570+0000","last_peered":"2024-03-09T15:31:50.780570+0000","last_clean":"2024-03-09T15:31:50.780570+0000","last_became_active":"2024-03-09T14:28:36.320079+0000","last_became_peered":"2024-03-09T14:28:36.320079+0000","last_unstale":"2024-03-09T15:31:50.780570+0000","last_undegraded":"2024-03-09T15:31:50.780570+0000","last_fullsized":"2024-03-09T15:31:50.780570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:19:38.894047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"261'87","reported_seq":623,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:52.371528+0000","last_change":"2024-03-09T14:28:38.341864+0000","last_active":"2024-03-09T15:31:52.371528+0000","last_peered":"2024-03-09T15:31:52.371528+0000","last_clean":"2024-03-09T15:31:52.371528+0000","last_became_active":"2024-03-09T14:28:38.340896+0000","last_became_peered":"2024-03-09T14:28:38.340896+0000","last_unstale":"2024-03-09T15:31:52.371528+0000","last_undegraded":"2024-03-09T15:31:52.371528+0000","last_fullsized":"2024-03-09T15:31:52.371528+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:19:17.825977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":101,"num_write":68,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.163891+0000","last_change":"2024-03-09T14:28:40.371660+0000","last_active":"2024-03-09T15:31:31.163891+0000","last_peered":"2024-03-09T15:31:31.163891+0000","last_clean":"2024-03-09T15:31:31.163891+0000","last_became_active":"2024-03-09T14:28:40.370788+0000","last_became_peered":"2024-03-09T14:28:40.370788+0000","last_unstale":"2024-03-09T15:31:31.163891+0000","last_undegraded":"2024-03-09T15:31:31.163891+0000","last_fullsized":"2024-03-09T15:31:31.163891+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:12:43.937289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"261'59","reported_seq":561,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.285043+0000","last_change":"2024-03-09T14:28:42.411392+0000","last_active":"2024-03-09T15:33:50.285043+0000","last_peered":"2024-03-09T15:33:50.285043+0000","last_clean":"2024-03-09T15:33:50.285043+0000","last_became_active":"2024-03-09T14:28:42.408738+0000","last_became_peered":"2024-03-09T14:28:42.408738+0000","last_unstale":"2024-03-09T15:33:50.285043+0000","last_undegraded":"2024-03-09T15:33:50.285043+0000","last_fullsized":"2024-03-09T15:33:50.285043+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:05:13.992640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":681,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":76,"num_read_kb":75,"num_write":140,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"261'157","reported_seq":745,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:31.430581+0000","last_change":"2024-03-09T14:28:34.305220+0000","last_active":"2024-03-09T15:33:31.430581+0000","last_peered":"2024-03-09T15:33:31.430581+0000","last_clean":"2024-03-09T15:33:31.430581+0000","last_became_active":"2024-03-09T14:28:34.304892+0000","last_became_peered":"2024-03-09T14:28:34.304892+0000","last_unstale":"2024-03-09T15:33:31.430581+0000","last_undegraded":"2024-03-09T15:33:31.430581+0000","last_fullsized":"2024-03-09T15:33:31.430581+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:10:30.883636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":622,"num_read_kb":492,"num_write":239,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"261'148","reported_seq":848,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.211631+0000","last_change":"2024-03-09T14:28:32.319206+0000","last_active":"2024-03-09T15:36:06.211631+0000","last_peered":"2024-03-09T15:36:06.211631+0000","last_clean":"2024-03-09T15:36:06.211631+0000","last_became_active":"2024-03-09T14:28:32.318428+0000","last_became_peered":"2024-03-09T14:28:32.318428+0000","last_unstale":"2024-03-09T15:36:06.211631+0000","last_undegraded":"2024-03-09T15:36:06.211631+0000","last_fullsized":"2024-03-09T15:36:06.211631+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":100,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:59:15.432941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6019791,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":531,"num_read_kb":113871,"num_write":606,"num_write_kb":46622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.774459+0000","last_change":"2024-03-09T14:28:36.320826+0000","last_active":"2024-03-09T15:31:25.774459+0000","last_peered":"2024-03-09T15:31:25.774459+0000","last_clean":"2024-03-09T15:31:25.774459+0000","last_became_active":"2024-03-09T14:28:36.320095+0000","last_became_peered":"2024-03-09T14:28:36.320095+0000","last_unstale":"2024-03-09T15:31:25.774459+0000","last_undegraded":"2024-03-09T15:31:25.774459+0000","last_fullsized":"2024-03-09T15:31:25.774459+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:58:26.878574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"261'103","reported_seq":653,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:02.768040+0000","last_change":"2024-03-09T14:28:38.339553+0000","last_active":"2024-03-09T15:34:02.768040+0000","last_peered":"2024-03-09T15:34:02.768040+0000","last_clean":"2024-03-09T15:34:02.768040+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:34:02.768040+0000","last_undegraded":"2024-03-09T15:34:02.768040+0000","last_fullsized":"2024-03-09T15:34:02.768040+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:37:31.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"261'504","reported_seq":2290,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.019957+0000","last_change":"2024-03-09T14:28:40.373218+0000","last_active":"2024-03-09T15:36:06.019957+0000","last_peered":"2024-03-09T15:36:06.019957+0000","last_clean":"2024-03-09T15:36:06.019957+0000","last_became_active":"2024-03-09T14:28:40.371507+0000","last_became_peered":"2024-03-09T14:28:40.371507+0000","last_unstale":"2024-03-09T15:36:06.019957+0000","last_undegraded":"2024-03-09T15:36:06.019957+0000","last_fullsized":"2024-03-09T15:36:06.019957+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":400,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:20:18.095183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"261'136","reported_seq":941,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:16.160267+0000","last_change":"2024-03-09T14:28:42.399490+0000","last_active":"2024-03-09T15:31:16.160267+0000","last_peered":"2024-03-09T15:31:16.160267+0000","last_clean":"2024-03-09T15:31:16.160267+0000","last_became_active":"2024-03-09T14:28:42.398498+0000","last_became_peered":"2024-03-09T14:28:42.398498+0000","last_unstale":"2024-03-09T15:31:16.160267+0000","last_undegraded":"2024-03-09T15:31:16.160267+0000","last_fullsized":"2024-03-09T15:31:16.160267+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:10:05.573339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":498,"num_read_kb":498,"num_write":277,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"261'53","reported_seq":592,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.774411+0000","last_change":"2024-03-09T14:28:34.330792+0000","last_active":"2024-03-09T15:31:25.774411+0000","last_peered":"2024-03-09T15:31:25.774411+0000","last_clean":"2024-03-09T15:31:25.774411+0000","last_became_active":"2024-03-09T14:28:34.330347+0000","last_became_peered":"2024-03-09T14:28:34.330347+0000","last_unstale":"2024-03-09T15:31:25.774411+0000","last_undegraded":"2024-03-09T15:31:25.774411+0000","last_fullsized":"2024-03-09T15:31:25.774411+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:55:20.226431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":212,"num_read_kb":171,"num_write":77,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"261'136","reported_seq":708,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.764851+0000","last_change":"2024-03-09T14:28:32.332979+0000","last_active":"2024-03-09T15:31:35.764851+0000","last_peered":"2024-03-09T15:31:35.764851+0000","last_clean":"2024-03-09T15:31:35.764851+0000","last_became_active":"2024-03-09T14:28:32.319819+0000","last_became_peered":"2024-03-09T14:28:32.319819+0000","last_unstale":"2024-03-09T15:31:35.764851+0000","last_undegraded":"2024-03-09T15:31:35.764851+0000","last_fullsized":"2024-03-09T15:31:35.764851+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:01:46.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5244864,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":201,"num_read_kb":15567,"num_write":549,"num_write_kb":34195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.166450+0000","last_change":"2024-03-09T14:28:36.311535+0000","last_active":"2024-03-09T15:31:26.166450+0000","last_peered":"2024-03-09T15:31:26.166450+0000","last_clean":"2024-03-09T15:31:26.166450+0000","last_became_active":"2024-03-09T14:28:36.310231+0000","last_became_peered":"2024-03-09T14:28:36.310231+0000","last_unstale":"2024-03-09T15:31:26.166450+0000","last_undegraded":"2024-03-09T15:31:26.166450+0000","last_fullsized":"2024-03-09T15:31:26.166450+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:55:50.175612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"261'167","reported_seq":746,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.679604+0000","last_change":"2024-03-09T14:28:38.599078+0000","last_active":"2024-03-09T15:35:58.679604+0000","last_peered":"2024-03-09T15:35:58.679604+0000","last_clean":"2024-03-09T15:35:58.679604+0000","last_became_active":"2024-03-09T14:28:38.598308+0000","last_became_peered":"2024-03-09T14:28:38.598308+0000","last_unstale":"2024-03-09T15:35:58.679604+0000","last_undegraded":"2024-03-09T15:35:58.679604+0000","last_fullsized":"2024-03-09T15:35:58.679604+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":100,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:10:09.151032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23129,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":269,"num_read_kb":489,"num_write":186,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"261'512","reported_seq":2498,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.733534+0000","last_change":"2024-03-09T14:28:40.388710+0000","last_active":"2024-03-09T15:36:05.733534+0000","last_peered":"2024-03-09T15:36:05.733534+0000","last_clean":"2024-03-09T15:36:05.733534+0000","last_became_active":"2024-03-09T14:28:40.387990+0000","last_became_peered":"2024-03-09T14:28:40.387990+0000","last_unstale":"2024-03-09T15:36:05.733534+0000","last_undegraded":"2024-03-09T15:36:05.733534+0000","last_fullsized":"2024-03-09T15:36:05.733534+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":500,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:21:47.017030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"261'101","reported_seq":678,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194321+0000","last_change":"2024-03-09T14:28:34.323491+0000","last_active":"2024-03-09T15:33:36.194321+0000","last_peered":"2024-03-09T15:33:36.194321+0000","last_clean":"2024-03-09T15:33:36.194321+0000","last_became_active":"2024-03-09T14:28:34.322731+0000","last_became_peered":"2024-03-09T14:28:34.322731+0000","last_unstale":"2024-03-09T15:33:36.194321+0000","last_undegraded":"2024-03-09T15:33:36.194321+0000","last_fullsized":"2024-03-09T15:33:36.194321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:34:36.142132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":330,"num_write":141,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"261'114","reported_seq":662,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.217253+0000","last_change":"2024-03-09T14:28:32.329755+0000","last_active":"2024-03-09T15:36:06.217253+0000","last_peered":"2024-03-09T15:36:06.217253+0000","last_clean":"2024-03-09T15:36:06.217253+0000","last_became_active":"2024-03-09T14:28:32.329148+0000","last_became_peered":"2024-03-09T14:28:32.329148+0000","last_unstale":"2024-03-09T15:36:06.217253+0000","last_undegraded":"2024-03-09T15:36:06.217253+0000","last_fullsized":"2024-03-09T15:36:06.217253+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:07:39.356330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7949644,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":138,"num_read_kb":1766,"num_write":504,"num_write_kb":18289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205246+0000","last_change":"2024-03-09T14:28:36.314780+0000","last_active":"2024-03-09T15:33:36.205246+0000","last_peered":"2024-03-09T15:33:36.205246+0000","last_clean":"2024-03-09T15:33:36.205246+0000","last_became_active":"2024-03-09T14:28:36.313904+0000","last_became_peered":"2024-03-09T14:28:36.313904+0000","last_unstale":"2024-03-09T15:33:36.205246+0000","last_undegraded":"2024-03-09T15:33:36.205246+0000","last_fullsized":"2024-03-09T15:33:36.205246+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:57:41.431102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"261'265","reported_seq":896,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:01.226737+0000","last_change":"2024-03-09T14:28:38.598064+0000","last_active":"2024-03-09T15:36:01.226737+0000","last_peered":"2024-03-09T15:36:01.226737+0000","last_clean":"2024-03-09T15:36:01.226737+0000","last_became_active":"2024-03-09T14:28:38.597247+0000","last_became_peered":"2024-03-09T14:28:38.597247+0000","last_unstale":"2024-03-09T15:36:01.226737+0000","last_undegraded":"2024-03-09T15:36:01.226737+0000","last_fullsized":"2024-03-09T15:36:01.226737+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":200,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:35:30.643121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22355,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":412,"num_read_kb":625,"num_write":280,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":495,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.134883+0000","last_change":"2024-03-09T14:28:40.360962+0000","last_active":"2024-03-09T15:31:41.134883+0000","last_peered":"2024-03-09T15:31:41.134883+0000","last_clean":"2024-03-09T15:31:41.134883+0000","last_became_active":"2024-03-09T14:28:40.359906+0000","last_became_peered":"2024-03-09T14:28:40.359906+0000","last_unstale":"2024-03-09T15:31:41.134883+0000","last_undegraded":"2024-03-09T15:31:41.134883+0000","last_fullsized":"2024-03-09T15:31:41.134883+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:33:27.073677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"261'518","reported_seq":1356,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.729973+0000","last_change":"2024-03-09T14:28:42.403475+0000","last_active":"2024-03-09T15:35:58.729973+0000","last_peered":"2024-03-09T15:35:58.729973+0000","last_clean":"2024-03-09T15:35:58.729973+0000","last_became_active":"2024-03-09T14:28:42.402739+0000","last_became_peered":"2024-03-09T14:28:42.402739+0000","last_unstale":"2024-03-09T15:35:58.729973+0000","last_undegraded":"2024-03-09T15:35:58.729973+0000","last_fullsized":"2024-03-09T15:35:58.729973+0000","mapping_epoch":30,"log_start":"261'500","ondisk_log_start":"261'500","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":500,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:49:57.794992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1336,"num_read_kb":1311,"num_write":1083,"num_write_kb":376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"261'92","reported_seq":662,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191097+0000","last_change":"2024-03-09T14:28:34.302153+0000","last_active":"2024-03-09T15:33:36.191097+0000","last_peered":"2024-03-09T15:33:36.191097+0000","last_clean":"2024-03-09T15:33:36.191097+0000","last_became_active":"2024-03-09T14:28:34.300795+0000","last_became_peered":"2024-03-09T14:28:34.300795+0000","last_unstale":"2024-03-09T15:33:36.191097+0000","last_undegraded":"2024-03-09T15:33:36.191097+0000","last_fullsized":"2024-03-09T15:33:36.191097+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:27:31.948784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":361,"num_read_kb":288,"num_write":150,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"261'133","reported_seq":706,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.234898+0000","last_change":"2024-03-09T14:28:32.318853+0000","last_active":"2024-03-09T15:36:06.234898+0000","last_peered":"2024-03-09T15:36:06.234898+0000","last_clean":"2024-03-09T15:36:06.234898+0000","last_became_active":"2024-03-09T14:28:32.318111+0000","last_became_peered":"2024-03-09T14:28:32.318111+0000","last_unstale":"2024-03-09T15:36:06.234898+0000","last_undegraded":"2024-03-09T15:36:06.234898+0000","last_fullsized":"2024-03-09T15:36:06.234898+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:51.120639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5549179,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":206,"num_read_kb":6230,"num_write":588,"num_write_kb":19779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.976641+0000","last_change":"2024-03-09T14:28:36.301940+0000","last_active":"2024-03-09T15:33:35.976641+0000","last_peered":"2024-03-09T15:33:35.976641+0000","last_clean":"2024-03-09T15:33:35.976641+0000","last_became_active":"2024-03-09T14:28:36.301227+0000","last_became_peered":"2024-03-09T14:28:36.301227+0000","last_unstale":"2024-03-09T15:33:35.976641+0000","last_undegraded":"2024-03-09T15:33:35.976641+0000","last_fullsized":"2024-03-09T15:33:35.976641+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:55:41.843139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"261'170","reported_seq":742,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.654250+0000","last_change":"2024-03-09T14:28:38.347678+0000","last_active":"2024-03-09T15:35:58.654250+0000","last_peered":"2024-03-09T15:35:58.654250+0000","last_clean":"2024-03-09T15:35:58.654250+0000","last_became_active":"2024-03-09T14:28:38.346837+0000","last_became_peered":"2024-03-09T14:28:38.346837+0000","last_unstale":"2024-03-09T15:35:58.654250+0000","last_undegraded":"2024-03-09T15:35:58.654250+0000","last_fullsized":"2024-03-09T15:35:58.654250+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":100,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:06.776774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28902,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":267,"num_read_kb":489,"num_write":184,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.808565+0000","last_change":"2024-03-09T14:28:40.391285+0000","last_active":"2024-03-09T15:33:55.808565+0000","last_peered":"2024-03-09T15:33:55.808565+0000","last_clean":"2024-03-09T15:33:55.808565+0000","last_became_active":"2024-03-09T14:28:40.389175+0000","last_became_peered":"2024-03-09T14:28:40.389175+0000","last_unstale":"2024-03-09T15:33:55.808565+0000","last_undegraded":"2024-03-09T15:33:55.808565+0000","last_fullsized":"2024-03-09T15:33:55.808565+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:33:43.955165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"261'58","reported_seq":553,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:02.730159+0000","last_change":"2024-03-09T14:28:42.401990+0000","last_active":"2024-03-09T15:34:02.730159+0000","last_peered":"2024-03-09T15:34:02.730159+0000","last_clean":"2024-03-09T15:34:02.730159+0000","last_became_active":"2024-03-09T14:28:42.401214+0000","last_became_peered":"2024-03-09T14:28:42.401214+0000","last_unstale":"2024-03-09T15:34:02.730159+0000","last_undegraded":"2024-03-09T15:34:02.730159+0000","last_fullsized":"2024-03-09T15:34:02.730159+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:24:20.023880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":47,"num_write":140,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"261'78","reported_seq":634,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.793908+0000","last_change":"2024-03-09T14:28:34.317931+0000","last_active":"2024-03-09T15:33:55.793908+0000","last_peered":"2024-03-09T15:33:55.793908+0000","last_clean":"2024-03-09T15:33:55.793908+0000","last_became_active":"2024-03-09T14:28:34.316812+0000","last_became_peered":"2024-03-09T14:28:34.316812+0000","last_unstale":"2024-03-09T15:33:55.793908+0000","last_undegraded":"2024-03-09T15:33:55.793908+0000","last_fullsized":"2024-03-09T15:33:55.793908+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:23:12.743671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":320,"num_read_kb":256,"num_write":116,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"261'100","reported_seq":663,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.219206+0000","last_change":"2024-03-09T14:29:06.446958+0000","last_active":"2024-03-09T15:36:06.219206+0000","last_peered":"2024-03-09T15:36:06.219206+0000","last_clean":"2024-03-09T15:36:06.219206+0000","last_became_active":"2024-03-09T14:29:06.446333+0000","last_became_peered":"2024-03-09T14:29:06.446333+0000","last_unstale":"2024-03-09T15:36:06.219206+0000","last_undegraded":"2024-03-09T15:36:06.219206+0000","last_fullsized":"2024-03-09T15:36:06.219206+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:09:44.978410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":139,"num_read_kb":1025,"num_write":394,"num_write_kb":11167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.980194+0000","last_change":"2024-03-09T14:28:36.304298+0000","last_active":"2024-03-09T15:33:50.980194+0000","last_peered":"2024-03-09T15:33:50.980194+0000","last_clean":"2024-03-09T15:33:50.980194+0000","last_became_active":"2024-03-09T14:28:36.303695+0000","last_became_peered":"2024-03-09T14:28:36.303695+0000","last_unstale":"2024-03-09T15:33:50.980194+0000","last_undegraded":"2024-03-09T15:33:50.980194+0000","last_fullsized":"2024-03-09T15:33:50.980194+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:16:14.481520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"261'217","reported_seq":821,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.705946+0000","last_change":"2024-03-09T14:28:38.340119+0000","last_active":"2024-03-09T15:35:58.705946+0000","last_peered":"2024-03-09T15:35:58.705946+0000","last_clean":"2024-03-09T15:35:58.705946+0000","last_became_active":"2024-03-09T14:28:38.339300+0000","last_became_peered":"2024-03-09T14:28:38.339300+0000","last_unstale":"2024-03-09T15:35:58.705946+0000","last_undegraded":"2024-03-09T15:35:58.705946+0000","last_fullsized":"2024-03-09T15:35:58.705946+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":200,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:56:14.245930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24501,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":344,"num_read_kb":579,"num_write":240,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"261'500","reported_seq":2509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.736377+0000","last_change":"2024-03-09T14:28:40.368533+0000","last_active":"2024-03-09T15:36:05.736377+0000","last_peered":"2024-03-09T15:36:05.736377+0000","last_clean":"2024-03-09T15:36:05.736377+0000","last_became_active":"2024-03-09T14:28:40.367119+0000","last_became_peered":"2024-03-09T14:28:40.367119+0000","last_unstale":"2024-03-09T15:36:05.736377+0000","last_undegraded":"2024-03-09T15:36:05.736377+0000","last_fullsized":"2024-03-09T15:36:05.736377+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":400,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:01.659994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"261'81","reported_seq":575,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:31.861227+0000","last_change":"2024-03-09T14:28:42.397381+0000","last_active":"2024-03-09T15:34:31.861227+0000","last_peered":"2024-03-09T15:34:31.861227+0000","last_clean":"2024-03-09T15:34:31.861227+0000","last_became_active":"2024-03-09T14:28:42.396389+0000","last_became_peered":"2024-03-09T14:28:42.396389+0000","last_unstale":"2024-03-09T15:34:31.861227+0000","last_undegraded":"2024-03-09T15:34:31.861227+0000","last_fullsized":"2024-03-09T15:34:31.861227+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:00:18.807761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":65,"num_write":200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"261'44","reported_seq":582,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.016397+0000","last_change":"2024-03-09T14:28:34.298294+0000","last_active":"2024-03-09T15:33:51.016397+0000","last_peered":"2024-03-09T15:33:51.016397+0000","last_clean":"2024-03-09T15:33:51.016397+0000","last_became_active":"2024-03-09T14:28:34.297080+0000","last_became_peered":"2024-03-09T14:28:34.297080+0000","last_unstale":"2024-03-09T15:33:51.016397+0000","last_undegraded":"2024-03-09T15:33:51.016397+0000","last_fullsized":"2024-03-09T15:33:51.016397+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:32.179794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":189,"num_read_kb":155,"num_write":63,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"261'128","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.989852+0000","last_change":"2024-03-09T14:29:06.445970+0000","last_active":"2024-03-09T15:36:05.989852+0000","last_peered":"2024-03-09T15:36:05.989852+0000","last_clean":"2024-03-09T15:36:05.989852+0000","last_became_active":"2024-03-09T14:29:06.445573+0000","last_became_peered":"2024-03-09T14:29:06.445573+0000","last_unstale":"2024-03-09T15:36:05.989852+0000","last_undegraded":"2024-03-09T15:36:05.989852+0000","last_fullsized":"2024-03-09T15:36:05.989852+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:30:46.074402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":185,"num_read_kb":157,"num_write":439,"num_write_kb":32761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.980070+0000","last_change":"2024-03-09T14:28:36.323155+0000","last_active":"2024-03-09T15:33:50.980070+0000","last_peered":"2024-03-09T15:33:50.980070+0000","last_clean":"2024-03-09T15:33:50.980070+0000","last_became_active":"2024-03-09T14:28:36.322638+0000","last_became_peered":"2024-03-09T14:28:36.322638+0000","last_unstale":"2024-03-09T15:33:50.980070+0000","last_undegraded":"2024-03-09T15:33:50.980070+0000","last_fullsized":"2024-03-09T15:33:50.980070+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:45:20.380783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"261'312","reported_seq":947,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.765276+0000","last_change":"2024-03-09T14:28:38.329540+0000","last_active":"2024-03-09T15:36:05.765276+0000","last_peered":"2024-03-09T15:36:05.765276+0000","last_clean":"2024-03-09T15:36:05.765276+0000","last_became_active":"2024-03-09T14:28:38.328897+0000","last_became_peered":"2024-03-09T14:28:38.328897+0000","last_unstale":"2024-03-09T15:36:05.765276+0000","last_undegraded":"2024-03-09T15:36:05.765276+0000","last_fullsized":"2024-03-09T15:36:05.765276+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":300,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:11:28.889923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60622,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":525,"num_read_kb":1064,"num_write":380,"num_write_kb":275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.977668+0000","last_change":"2024-03-09T14:28:40.384571+0000","last_active":"2024-03-09T15:33:40.977668+0000","last_peered":"2024-03-09T15:33:40.977668+0000","last_clean":"2024-03-09T15:33:40.977668+0000","last_became_active":"2024-03-09T14:28:40.383758+0000","last_became_peered":"2024-03-09T14:28:40.383758+0000","last_unstale":"2024-03-09T15:33:40.977668+0000","last_undegraded":"2024-03-09T15:33:40.977668+0000","last_fullsized":"2024-03-09T15:33:40.977668+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:24:41.089698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"261'54","reported_seq":565,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.965658+0000","last_change":"2024-03-09T14:28:42.402684+0000","last_active":"2024-03-09T15:33:45.965658+0000","last_peered":"2024-03-09T15:33:45.965658+0000","last_clean":"2024-03-09T15:33:45.965658+0000","last_became_active":"2024-03-09T14:28:42.401880+0000","last_became_peered":"2024-03-09T14:28:42.401880+0000","last_unstale":"2024-03-09T15:33:45.965658+0000","last_undegraded":"2024-03-09T15:33:45.965658+0000","last_fullsized":"2024-03-09T15:33:45.965658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:29:59.644550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":69,"num_write":120,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"261'79","reported_seq":639,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195621+0000","last_change":"2024-03-09T14:28:34.316156+0000","last_active":"2024-03-09T15:33:51.195621+0000","last_peered":"2024-03-09T15:33:51.195621+0000","last_clean":"2024-03-09T15:33:51.195621+0000","last_became_active":"2024-03-09T14:28:34.315432+0000","last_became_peered":"2024-03-09T14:28:34.315432+0000","last_unstale":"2024-03-09T15:33:51.195621+0000","last_undegraded":"2024-03-09T15:33:51.195621+0000","last_fullsized":"2024-03-09T15:33:51.195621+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:08:27.256304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":315,"num_read_kb":251,"num_write":126,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"261'126","reported_seq":664,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.807364+0000","last_change":"2024-03-09T14:28:32.300326+0000","last_active":"2024-03-09T15:33:50.807364+0000","last_peered":"2024-03-09T15:33:50.807364+0000","last_clean":"2024-03-09T15:33:50.807364+0000","last_became_active":"2024-03-09T14:28:32.299897+0000","last_became_peered":"2024-03-09T14:28:32.299897+0000","last_unstale":"2024-03-09T15:33:50.807364+0000","last_undegraded":"2024-03-09T15:33:50.807364+0000","last_fullsized":"2024-03-09T15:33:50.807364+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":100,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:25:33.818560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5546754,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":129,"num_read_kb":225,"num_write":522,"num_write_kb":26448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.980312+0000","last_change":"2024-03-09T14:28:36.320815+0000","last_active":"2024-03-09T15:33:50.980312+0000","last_peered":"2024-03-09T15:33:50.980312+0000","last_clean":"2024-03-09T15:33:50.980312+0000","last_became_active":"2024-03-09T14:28:36.320029+0000","last_became_peered":"2024-03-09T14:28:36.320029+0000","last_unstale":"2024-03-09T15:33:50.980312+0000","last_undegraded":"2024-03-09T15:33:50.980312+0000","last_fullsized":"2024-03-09T15:33:50.980312+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:23:44.855859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"261'205","reported_seq":792,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.664412+0000","last_change":"2024-03-09T14:28:38.339497+0000","last_active":"2024-03-09T15:35:58.664412+0000","last_peered":"2024-03-09T15:35:58.664412+0000","last_clean":"2024-03-09T15:35:58.664412+0000","last_became_active":"2024-03-09T14:28:38.338658+0000","last_became_peered":"2024-03-09T14:28:38.338658+0000","last_unstale":"2024-03-09T15:35:58.664412+0000","last_undegraded":"2024-03-09T15:35:58.664412+0000","last_fullsized":"2024-03-09T15:35:58.664412+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":100,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:42:50.615803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25762,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":343,"num_read_kb":574,"num_write":288,"num_write_kb":187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:55.989515+0000","last_change":"2024-03-09T14:28:40.379699+0000","last_active":"2024-03-09T15:31:55.989515+0000","last_peered":"2024-03-09T15:31:55.989515+0000","last_clean":"2024-03-09T15:31:55.989515+0000","last_became_active":"2024-03-09T14:28:40.378973+0000","last_became_peered":"2024-03-09T14:28:40.378973+0000","last_unstale":"2024-03-09T15:31:55.989515+0000","last_undegraded":"2024-03-09T15:31:55.989515+0000","last_fullsized":"2024-03-09T15:31:55.989515+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:53.400714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"261'42","reported_seq":535,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:00.822318+0000","last_change":"2024-03-09T14:28:42.398431+0000","last_active":"2024-03-09T15:35:00.822318+0000","last_peered":"2024-03-09T15:35:00.822318+0000","last_clean":"2024-03-09T15:35:00.822318+0000","last_became_active":"2024-03-09T14:28:42.397533+0000","last_became_peered":"2024-03-09T14:28:42.397533+0000","last_unstale":"2024-03-09T15:35:00.822318+0000","last_undegraded":"2024-03-09T15:35:00.822318+0000","last_fullsized":"2024-03-09T15:35:00.822318+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:54:11.336171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":91,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"261'56","reported_seq":616,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195764+0000","last_change":"2024-03-09T14:28:34.315224+0000","last_active":"2024-03-09T15:33:51.195764+0000","last_peered":"2024-03-09T15:33:51.195764+0000","last_clean":"2024-03-09T15:33:51.195764+0000","last_became_active":"2024-03-09T14:28:34.313475+0000","last_became_peered":"2024-03-09T14:28:34.313475+0000","last_unstale":"2024-03-09T15:33:51.195764+0000","last_undegraded":"2024-03-09T15:33:51.195764+0000","last_fullsized":"2024-03-09T15:33:51.195764+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:37:57.784425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":198,"num_write":78,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"261'105","reported_seq":657,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:40.924777+0000","last_change":"2024-03-09T14:28:32.319640+0000","last_active":"2024-03-09T15:35:40.924777+0000","last_peered":"2024-03-09T15:35:40.924777+0000","last_clean":"2024-03-09T15:35:40.924777+0000","last_became_active":"2024-03-09T14:28:32.316406+0000","last_became_peered":"2024-03-09T14:28:32.316406+0000","last_unstale":"2024-03-09T15:35:40.924777+0000","last_undegraded":"2024-03-09T15:35:40.924777+0000","last_fullsized":"2024-03-09T15:35:40.924777+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:33:19.144976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7947601,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":138,"num_read_kb":15736,"num_write":442,"num_write_kb":13578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.966668+0000","last_change":"2024-03-09T14:28:36.319712+0000","last_active":"2024-03-09T15:33:50.966668+0000","last_peered":"2024-03-09T15:33:50.966668+0000","last_clean":"2024-03-09T15:33:50.966668+0000","last_became_active":"2024-03-09T14:28:36.319210+0000","last_became_peered":"2024-03-09T14:28:36.319210+0000","last_unstale":"2024-03-09T15:33:50.966668+0000","last_undegraded":"2024-03-09T15:33:50.966668+0000","last_fullsized":"2024-03-09T15:33:50.966668+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:26:56.771418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"261'217","reported_seq":821,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.969548+0000","last_change":"2024-03-09T14:28:38.351282+0000","last_active":"2024-03-09T15:36:05.969548+0000","last_peered":"2024-03-09T15:36:05.969548+0000","last_clean":"2024-03-09T15:36:05.969548+0000","last_became_active":"2024-03-09T14:28:38.350390+0000","last_became_peered":"2024-03-09T14:28:38.350390+0000","last_unstale":"2024-03-09T15:36:05.969548+0000","last_undegraded":"2024-03-09T15:36:05.969548+0000","last_fullsized":"2024-03-09T15:36:05.969548+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":200,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:30:58.004755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9540,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":338,"num_read_kb":580,"num_write":230,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"261'473","reported_seq":2357,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.722396+0000","last_change":"2024-03-09T14:28:40.374487+0000","last_active":"2024-03-09T15:36:05.722396+0000","last_peered":"2024-03-09T15:36:05.722396+0000","last_clean":"2024-03-09T15:36:05.722396+0000","last_became_active":"2024-03-09T14:28:40.373360+0000","last_became_peered":"2024-03-09T14:28:40.373360+0000","last_unstale":"2024-03-09T15:36:05.722396+0000","last_undegraded":"2024-03-09T15:36:05.722396+0000","last_fullsized":"2024-03-09T15:36:05.722396+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":400,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:24:48.737477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"261'104","reported_seq":650,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:52.932955+0000","last_change":"2024-03-09T14:28:42.409080+0000","last_active":"2024-03-09T15:34:52.932955+0000","last_peered":"2024-03-09T15:34:52.932955+0000","last_clean":"2024-03-09T15:34:52.932955+0000","last_became_active":"2024-03-09T14:28:42.408233+0000","last_became_peered":"2024-03-09T14:28:42.408233+0000","last_unstale":"2024-03-09T15:34:52.932955+0000","last_undegraded":"2024-03-09T15:34:52.932955+0000","last_fullsized":"2024-03-09T15:34:52.932955+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:59:27.171699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":274,"num_read_kb":227,"num_write":264,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"261'72","reported_seq":621,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195874+0000","last_change":"2024-03-09T14:28:34.310932+0000","last_active":"2024-03-09T15:33:51.195874+0000","last_peered":"2024-03-09T15:33:51.195874+0000","last_clean":"2024-03-09T15:33:51.195874+0000","last_became_active":"2024-03-09T14:28:34.310061+0000","last_became_peered":"2024-03-09T14:28:34.310061+0000","last_unstale":"2024-03-09T15:33:51.195874+0000","last_undegraded":"2024-03-09T15:33:51.195874+0000","last_fullsized":"2024-03-09T15:33:51.195874+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:56:23.692037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":294,"num_read_kb":233,"num_write":114,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"261'141","reported_seq":748,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.928584+0000","last_change":"2024-03-09T14:28:32.313964+0000","last_active":"2024-03-09T15:36:05.928584+0000","last_peered":"2024-03-09T15:36:05.928584+0000","last_clean":"2024-03-09T15:36:05.928584+0000","last_became_active":"2024-03-09T14:28:32.313566+0000","last_became_peered":"2024-03-09T14:28:32.313566+0000","last_unstale":"2024-03-09T15:36:05.928584+0000","last_undegraded":"2024-03-09T15:36:05.928584+0000","last_fullsized":"2024-03-09T15:36:05.928584+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":100,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:06:35.181059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":569877,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":220,"num_read_kb":84880,"num_write":542,"num_write_kb":19718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"32'7","reported_seq":1459,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.269380+0000","last_change":"2024-03-09T14:28:36.327203+0000","last_active":"2024-03-09T15:36:05.269380+0000","last_peered":"2024-03-09T15:36:05.269380+0000","last_clean":"2024-03-09T15:36:05.269380+0000","last_became_active":"2024-03-09T14:28:36.325820+0000","last_became_peered":"2024-03-09T14:28:36.325820+0000","last_unstale":"2024-03-09T15:36:05.269380+0000","last_undegraded":"2024-03-09T15:36:05.269380+0000","last_fullsized":"2024-03-09T15:36:05.269380+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:03:03.441773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1195,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1118,"num_read_kb":1421,"num_write":14,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"261'86","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:52.540471+0000","last_change":"2024-03-09T14:28:38.596269+0000","last_active":"2024-03-09T15:32:52.540471+0000","last_peered":"2024-03-09T15:32:52.540471+0000","last_clean":"2024-03-09T15:32:52.540471+0000","last_became_active":"2024-03-09T14:28:38.595478+0000","last_became_peered":"2024-03-09T14:28:38.595478+0000","last_unstale":"2024-03-09T15:32:52.540471+0000","last_undegraded":"2024-03-09T15:32:52.540471+0000","last_fullsized":"2024-03-09T15:32:52.540471+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:43:55.683017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":105,"num_read_kb":100,"num_write":70,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"261'488","reported_seq":2293,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.725131+0000","last_change":"2024-03-09T14:28:40.375535+0000","last_active":"2024-03-09T15:36:05.725131+0000","last_peered":"2024-03-09T15:36:05.725131+0000","last_clean":"2024-03-09T15:36:05.725131+0000","last_became_active":"2024-03-09T14:28:40.374408+0000","last_became_peered":"2024-03-09T14:28:40.374408+0000","last_unstale":"2024-03-09T15:36:05.725131+0000","last_undegraded":"2024-03-09T15:36:05.725131+0000","last_fullsized":"2024-03-09T15:36:05.725131+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":400,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:00:18.508302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"261'93","reported_seq":596,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:39.863116+0000","last_change":"2024-03-09T14:28:42.412396+0000","last_active":"2024-03-09T15:34:39.863116+0000","last_peered":"2024-03-09T15:34:39.863116+0000","last_clean":"2024-03-09T15:34:39.863116+0000","last_became_active":"2024-03-09T14:28:42.411523+0000","last_became_peered":"2024-03-09T14:28:42.411523+0000","last_unstale":"2024-03-09T15:34:39.863116+0000","last_undegraded":"2024-03-09T15:34:39.863116+0000","last_fullsized":"2024-03-09T15:34:39.863116+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:23:50.431623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":72,"num_write":221,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"261'42","reported_seq":595,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.134950+0000","last_change":"2024-03-09T14:28:34.317632+0000","last_active":"2024-03-09T15:31:41.134950+0000","last_peered":"2024-03-09T15:31:41.134950+0000","last_clean":"2024-03-09T15:31:41.134950+0000","last_became_active":"2024-03-09T14:28:34.317226+0000","last_became_peered":"2024-03-09T14:28:34.317226+0000","last_unstale":"2024-03-09T15:31:41.134950+0000","last_undegraded":"2024-03-09T15:31:41.134950+0000","last_fullsized":"2024-03-09T15:31:41.134950+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:48:25.758532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":188,"num_read_kb":158,"num_write":62,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"261'93","reported_seq":629,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.803584+0000","last_change":"2024-03-09T14:28:32.295693+0000","last_active":"2024-03-09T15:33:35.803584+0000","last_peered":"2024-03-09T15:33:35.803584+0000","last_clean":"2024-03-09T15:33:35.803584+0000","last_became_active":"2024-03-09T14:28:32.295179+0000","last_became_peered":"2024-03-09T14:28:32.295179+0000","last_unstale":"2024-03-09T15:33:35.803584+0000","last_undegraded":"2024-03-09T15:33:35.803584+0000","last_fullsized":"2024-03-09T15:33:35.803584+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:55:35.164395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262386,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":120,"num_read_kb":124,"num_write":367,"num_write_kb":5920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195837+0000","last_change":"2024-03-09T14:28:36.319885+0000","last_active":"2024-03-09T15:33:51.195837+0000","last_peered":"2024-03-09T15:33:51.195837+0000","last_clean":"2024-03-09T15:33:51.195837+0000","last_became_active":"2024-03-09T14:28:36.319102+0000","last_became_peered":"2024-03-09T14:28:36.319102+0000","last_unstale":"2024-03-09T15:33:51.195837+0000","last_undegraded":"2024-03-09T15:33:51.195837+0000","last_fullsized":"2024-03-09T15:33:51.195837+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:44:50.268293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"261'311","reported_seq":934,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.971810+0000","last_change":"2024-03-09T14:28:38.338131+0000","last_active":"2024-03-09T15:35:58.971810+0000","last_peered":"2024-03-09T15:35:58.971810+0000","last_clean":"2024-03-09T15:35:58.971810+0000","last_became_active":"2024-03-09T14:28:38.337254+0000","last_became_peered":"2024-03-09T14:28:38.337254+0000","last_unstale":"2024-03-09T15:35:58.971810+0000","last_undegraded":"2024-03-09T15:35:58.971810+0000","last_fullsized":"2024-03-09T15:35:58.971810+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":300,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:49.247631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29175,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":664,"num_read_kb":922,"num_write":784,"num_write_kb":628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.766105+0000","last_change":"2024-03-09T14:28:40.378550+0000","last_active":"2024-03-09T15:31:40.766105+0000","last_peered":"2024-03-09T15:31:40.766105+0000","last_clean":"2024-03-09T15:31:40.766105+0000","last_became_active":"2024-03-09T14:28:40.377714+0000","last_became_peered":"2024-03-09T14:28:40.377714+0000","last_unstale":"2024-03-09T15:31:40.766105+0000","last_undegraded":"2024-03-09T15:31:40.766105+0000","last_fullsized":"2024-03-09T15:31:40.766105+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:21:50.389100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"261'68","reported_seq":567,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.959943+0000","last_change":"2024-03-09T14:28:42.402550+0000","last_active":"2024-03-09T15:33:20.959943+0000","last_peered":"2024-03-09T15:33:20.959943+0000","last_clean":"2024-03-09T15:33:20.959943+0000","last_became_active":"2024-03-09T14:28:42.401775+0000","last_became_peered":"2024-03-09T14:28:42.401775+0000","last_unstale":"2024-03-09T15:33:20.959943+0000","last_undegraded":"2024-03-09T15:33:20.959943+0000","last_fullsized":"2024-03-09T15:33:20.959943+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:47:20.741043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":77,"num_write":153,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"261'174","reported_seq":825,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:30.788820+0000","last_change":"2024-03-09T14:28:34.327727+0000","last_active":"2024-03-09T15:33:30.788820+0000","last_peered":"2024-03-09T15:33:30.788820+0000","last_clean":"2024-03-09T15:33:30.788820+0000","last_became_active":"2024-03-09T14:28:34.326919+0000","last_became_peered":"2024-03-09T14:28:34.326919+0000","last_unstale":"2024-03-09T15:33:30.788820+0000","last_undegraded":"2024-03-09T15:33:30.788820+0000","last_fullsized":"2024-03-09T15:33:30.788820+0000","mapping_epoch":22,"log_start":"34'100","ondisk_log_start":"34'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:50:06.490347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":914,"num_read_kb":819,"num_write":300,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":681,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.189283+0000","last_change":"2024-03-09T14:28:12.265518+0000","last_active":"2024-03-09T15:32:26.189283+0000","last_peered":"2024-03-09T15:32:26.189283+0000","last_clean":"2024-03-09T15:32:26.189283+0000","last_became_active":"2024-03-09T14:28:12.262430+0000","last_became_peered":"2024-03-09T14:28:12.262430+0000","last_unstale":"2024-03-09T15:32:26.189283+0000","last_undegraded":"2024-03-09T15:32:26.189283+0000","last_fullsized":"2024-03-09T15:32:26.189283+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_clean_scrub_stamp":"2024-03-09T14:28:11.179640+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:14:17.888044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.185851+0000","last_change":"2024-03-09T14:28:19.233336+0000","last_active":"2024-03-09T15:32:56.185851+0000","last_peered":"2024-03-09T15:32:56.185851+0000","last_clean":"2024-03-09T15:32:56.185851+0000","last_became_active":"2024-03-09T14:28:19.232517+0000","last_became_peered":"2024-03-09T14:28:19.232517+0000","last_unstale":"2024-03-09T15:32:56.185851+0000","last_undegraded":"2024-03-09T15:32:56.185851+0000","last_fullsized":"2024-03-09T15:32:56.185851+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:53:39.430154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"261'111","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:39.045175+0000","last_change":"2024-03-09T14:28:32.300845+0000","last_active":"2024-03-09T15:34:39.045175+0000","last_peered":"2024-03-09T15:34:39.045175+0000","last_clean":"2024-03-09T15:34:39.045175+0000","last_became_active":"2024-03-09T14:28:32.300468+0000","last_became_peered":"2024-03-09T14:28:32.300468+0000","last_unstale":"2024-03-09T15:34:39.045175+0000","last_undegraded":"2024-03-09T15:34:39.045175+0000","last_fullsized":"2024-03-09T15:34:39.045175+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:58:14.894309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3439332,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":154,"num_read_kb":6450,"num_write":438,"num_write_kb":21101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.199546+0000","last_change":"2024-03-09T14:28:36.318953+0000","last_active":"2024-03-09T15:33:11.199546+0000","last_peered":"2024-03-09T15:33:11.199546+0000","last_clean":"2024-03-09T15:33:11.199546+0000","last_became_active":"2024-03-09T14:28:36.318172+0000","last_became_peered":"2024-03-09T14:28:36.318172+0000","last_unstale":"2024-03-09T15:33:11.199546+0000","last_undegraded":"2024-03-09T15:33:11.199546+0000","last_fullsized":"2024-03-09T15:33:11.199546+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:14:08.117277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"261'493","reported_seq":1200,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.910451+0000","last_change":"2024-03-09T14:28:38.349794+0000","last_active":"2024-03-09T15:36:05.910451+0000","last_peered":"2024-03-09T15:36:05.910451+0000","last_clean":"2024-03-09T15:36:05.910451+0000","last_became_active":"2024-03-09T14:28:38.348941+0000","last_became_peered":"2024-03-09T14:28:38.348941+0000","last_unstale":"2024-03-09T15:36:05.910451+0000","last_undegraded":"2024-03-09T15:36:05.910451+0000","last_fullsized":"2024-03-09T15:36:05.910451+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":400,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:10:34.907672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74232,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":969,"num_read_kb":1810,"num_write":926,"num_write_kb":765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:41.014076+0000","last_change":"2024-03-09T14:28:40.364392+0000","last_active":"2024-03-09T15:33:41.014076+0000","last_peered":"2024-03-09T15:33:41.014076+0000","last_clean":"2024-03-09T15:33:41.014076+0000","last_became_active":"2024-03-09T14:28:40.362781+0000","last_became_peered":"2024-03-09T14:28:40.362781+0000","last_unstale":"2024-03-09T15:33:41.014076+0000","last_undegraded":"2024-03-09T15:33:41.014076+0000","last_fullsized":"2024-03-09T15:33:41.014076+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:30:02.543875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"261'118","reported_seq":817,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:45.170047+0000","last_change":"2024-03-09T14:28:42.399378+0000","last_active":"2024-03-09T15:34:45.170047+0000","last_peered":"2024-03-09T15:34:45.170047+0000","last_clean":"2024-03-09T15:34:45.170047+0000","last_became_active":"2024-03-09T14:28:42.398626+0000","last_became_peered":"2024-03-09T14:28:42.398626+0000","last_unstale":"2024-03-09T15:34:45.170047+0000","last_undegraded":"2024-03-09T15:34:45.170047+0000","last_fullsized":"2024-03-09T15:34:45.170047+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:40:25.365667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":356,"num_read_kb":343,"num_write":215,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"261'90","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.012674+0000","last_change":"2024-03-09T14:28:34.297905+0000","last_active":"2024-03-09T15:33:36.012674+0000","last_peered":"2024-03-09T15:33:36.012674+0000","last_clean":"2024-03-09T15:33:36.012674+0000","last_became_active":"2024-03-09T14:28:34.297472+0000","last_became_peered":"2024-03-09T14:28:34.297472+0000","last_unstale":"2024-03-09T15:33:36.012674+0000","last_undegraded":"2024-03-09T15:33:36.012674+0000","last_fullsized":"2024-03-09T15:33:36.012674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:04:18.955082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":356,"num_read_kb":292,"num_write":141,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":525,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:31.190342+0000","last_change":"2024-03-09T14:28:19.242176+0000","last_active":"2024-03-09T15:32:31.190342+0000","last_peered":"2024-03-09T15:32:31.190342+0000","last_clean":"2024-03-09T15:32:31.190342+0000","last_became_active":"2024-03-09T14:28:19.241110+0000","last_became_peered":"2024-03-09T14:28:19.241110+0000","last_unstale":"2024-03-09T15:32:31.190342+0000","last_undegraded":"2024-03-09T15:32:31.190342+0000","last_fullsized":"2024-03-09T15:32:31.190342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:28:52.238193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"261'122","reported_seq":694,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.955245+0000","last_change":"2024-03-09T14:28:32.312953+0000","last_active":"2024-03-09T15:36:05.955245+0000","last_peered":"2024-03-09T15:36:05.955245+0000","last_clean":"2024-03-09T15:36:05.955245+0000","last_became_active":"2024-03-09T14:28:32.312536+0000","last_became_peered":"2024-03-09T14:28:32.312536+0000","last_unstale":"2024-03-09T15:36:05.955245+0000","last_undegraded":"2024-03-09T15:36:05.955245+0000","last_fullsized":"2024-03-09T15:36:05.955245+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:53:44.400275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3383,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":203,"num_read_kb":12489,"num_write":509,"num_write_kb":17953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194706+0000","last_change":"2024-03-09T14:28:36.307721+0000","last_active":"2024-03-09T15:33:36.194706+0000","last_peered":"2024-03-09T15:33:36.194706+0000","last_clean":"2024-03-09T15:33:36.194706+0000","last_became_active":"2024-03-09T14:28:36.306912+0000","last_became_peered":"2024-03-09T14:28:36.306912+0000","last_unstale":"2024-03-09T15:33:36.194706+0000","last_undegraded":"2024-03-09T15:33:36.194706+0000","last_fullsized":"2024-03-09T15:33:36.194706+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:19:47.054193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"261'837","reported_seq":2502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.790375+0000","last_change":"2024-03-09T14:28:38.329490+0000","last_active":"2024-03-09T15:36:05.790375+0000","last_peered":"2024-03-09T15:36:05.790375+0000","last_clean":"2024-03-09T15:36:05.790375+0000","last_became_active":"2024-03-09T14:28:38.328738+0000","last_became_peered":"2024-03-09T14:28:38.328738+0000","last_unstale":"2024-03-09T15:36:05.790375+0000","last_undegraded":"2024-03-09T15:36:05.790375+0000","last_fullsized":"2024-03-09T15:36:05.790375+0000","mapping_epoch":26,"log_start":"261'800","ondisk_log_start":"261'800","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":800,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:24:55.016529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77478,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1296,"num_read_kb":2021,"num_write":650,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.202990+0000","last_change":"2024-03-09T14:28:40.387247+0000","last_active":"2024-03-09T15:33:26.202990+0000","last_peered":"2024-03-09T15:33:26.202990+0000","last_clean":"2024-03-09T15:33:26.202990+0000","last_became_active":"2024-03-09T14:28:40.386451+0000","last_became_peered":"2024-03-09T14:28:40.386451+0000","last_unstale":"2024-03-09T15:33:26.202990+0000","last_undegraded":"2024-03-09T15:33:26.202990+0000","last_fullsized":"2024-03-09T15:33:26.202990+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:06:23.810341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"261'124","reported_seq":656,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.967916+0000","last_change":"2024-03-09T14:28:42.403919+0000","last_active":"2024-03-09T15:33:55.967916+0000","last_peered":"2024-03-09T15:33:55.967916+0000","last_clean":"2024-03-09T15:33:55.967916+0000","last_became_active":"2024-03-09T14:28:42.403013+0000","last_became_peered":"2024-03-09T14:28:42.403013+0000","last_unstale":"2024-03-09T15:33:55.967916+0000","last_undegraded":"2024-03-09T15:33:55.967916+0000","last_fullsized":"2024-03-09T15:33:55.967916+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:36:18.083589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1248,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":261,"num_read_kb":241,"num_write":300,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"261'48","reported_seq":601,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963559+0000","last_change":"2024-03-09T14:28:34.297150+0000","last_active":"2024-03-09T15:33:35.963559+0000","last_peered":"2024-03-09T15:33:35.963559+0000","last_clean":"2024-03-09T15:33:35.963559+0000","last_became_active":"2024-03-09T14:28:34.296733+0000","last_became_peered":"2024-03-09T14:28:34.296733+0000","last_unstale":"2024-03-09T15:33:35.963559+0000","last_undegraded":"2024-03-09T15:33:35.963559+0000","last_fullsized":"2024-03-09T15:33:35.963559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:29:16.007058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":200,"num_read_kb":166,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.968712+0000","last_change":"2024-03-09T14:28:19.437446+0000","last_active":"2024-03-09T15:32:55.968712+0000","last_peered":"2024-03-09T15:32:55.968712+0000","last_clean":"2024-03-09T15:32:55.968712+0000","last_became_active":"2024-03-09T14:28:19.436985+0000","last_became_peered":"2024-03-09T14:28:19.436985+0000","last_unstale":"2024-03-09T15:32:55.968712+0000","last_undegraded":"2024-03-09T15:32:55.968712+0000","last_fullsized":"2024-03-09T15:32:55.968712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:46:38.528697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"261'134","reported_seq":791,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.893599+0000","last_change":"2024-03-09T14:28:32.303463+0000","last_active":"2024-03-09T15:36:05.893599+0000","last_peered":"2024-03-09T15:36:05.893599+0000","last_clean":"2024-03-09T15:36:05.893599+0000","last_became_active":"2024-03-09T14:28:32.303176+0000","last_became_peered":"2024-03-09T14:28:32.303176+0000","last_unstale":"2024-03-09T15:36:05.893599+0000","last_undegraded":"2024-03-09T15:36:05.893599+0000","last_fullsized":"2024-03-09T15:36:05.893599+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:28:51.916628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8554935,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":468,"num_read_kb":3616,"num_write":574,"num_write_kb":29269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"25'1","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.976814+0000","last_change":"2024-03-09T14:28:36.306679+0000","last_active":"2024-03-09T15:33:35.976814+0000","last_peered":"2024-03-09T15:33:35.976814+0000","last_clean":"2024-03-09T15:33:35.976814+0000","last_became_active":"2024-03-09T14:28:36.305971+0000","last_became_peered":"2024-03-09T14:28:36.305971+0000","last_unstale":"2024-03-09T15:33:35.976814+0000","last_undegraded":"2024-03-09T15:33:35.976814+0000","last_fullsized":"2024-03-09T15:33:35.976814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:21:21.395131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"261'230","reported_seq":1598,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.934522+0000","last_change":"2024-03-09T14:28:38.598201+0000","last_active":"2024-03-09T15:35:58.934522+0000","last_peered":"2024-03-09T15:35:58.934522+0000","last_clean":"2024-03-09T15:35:58.934522+0000","last_became_active":"2024-03-09T14:28:38.597403+0000","last_became_peered":"2024-03-09T14:28:38.597403+0000","last_unstale":"2024-03-09T15:35:58.934522+0000","last_undegraded":"2024-03-09T15:35:58.934522+0000","last_fullsized":"2024-03-09T15:35:58.934522+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":200,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:44:05.046714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":226,"num_read_kb":216,"num_write":195,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.007838+0000","last_change":"2024-03-09T14:28:40.372884+0000","last_active":"2024-03-09T15:33:16.007838+0000","last_peered":"2024-03-09T15:33:16.007838+0000","last_clean":"2024-03-09T15:33:16.007838+0000","last_became_active":"2024-03-09T14:28:40.371383+0000","last_became_peered":"2024-03-09T14:28:40.371383+0000","last_unstale":"2024-03-09T15:33:16.007838+0000","last_undegraded":"2024-03-09T15:33:16.007838+0000","last_fullsized":"2024-03-09T15:33:16.007838+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:01:15.189556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"261'201","reported_seq":791,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:19.528609+0000","last_change":"2024-03-09T14:28:42.396620+0000","last_active":"2024-03-09T15:34:19.528609+0000","last_peered":"2024-03-09T15:34:19.528609+0000","last_clean":"2024-03-09T15:34:19.528609+0000","last_became_active":"2024-03-09T14:28:42.395781+0000","last_became_peered":"2024-03-09T14:28:42.395781+0000","last_unstale":"2024-03-09T15:34:19.528609+0000","last_undegraded":"2024-03-09T15:34:19.528609+0000","last_fullsized":"2024-03-09T15:34:19.528609+0000","mapping_epoch":30,"log_start":"247'100","ondisk_log_start":"247'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":100,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:38:14.404007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":435,"num_read_kb":456,"num_write":425,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"261'74","reported_seq":618,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:31.876523+0000","last_change":"2024-03-09T14:28:34.313660+0000","last_active":"2024-03-09T15:34:31.876523+0000","last_peered":"2024-03-09T15:34:31.876523+0000","last_clean":"2024-03-09T15:34:31.876523+0000","last_became_active":"2024-03-09T14:28:34.313302+0000","last_became_peered":"2024-03-09T14:28:34.313302+0000","last_unstale":"2024-03-09T15:34:31.876523+0000","last_undegraded":"2024-03-09T15:34:31.876523+0000","last_fullsized":"2024-03-09T15:34:31.876523+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:32:32.206071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":225,"num_write":104,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":522,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.185786+0000","last_change":"2024-03-09T14:28:19.238720+0000","last_active":"2024-03-09T15:32:56.185786+0000","last_peered":"2024-03-09T15:32:56.185786+0000","last_clean":"2024-03-09T15:32:56.185786+0000","last_became_active":"2024-03-09T14:28:19.237937+0000","last_became_peered":"2024-03-09T14:28:19.237937+0000","last_unstale":"2024-03-09T15:32:56.185786+0000","last_undegraded":"2024-03-09T15:32:56.185786+0000","last_fullsized":"2024-03-09T15:32:56.185786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:29:01.306246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"261'107","reported_seq":682,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.961490+0000","last_change":"2024-03-09T14:28:32.312937+0000","last_active":"2024-03-09T15:36:05.961490+0000","last_peered":"2024-03-09T15:36:05.961490+0000","last_clean":"2024-03-09T15:36:05.961490+0000","last_became_active":"2024-03-09T14:28:32.303963+0000","last_became_peered":"2024-03-09T14:28:32.303963+0000","last_unstale":"2024-03-09T15:36:05.961490+0000","last_undegraded":"2024-03-09T15:36:05.961490+0000","last_fullsized":"2024-03-09T15:36:05.961490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:54:34.862486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6966901,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":155,"num_read_kb":15115,"num_write":410,"num_write_kb":32085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.010297+0000","last_change":"2024-03-09T14:28:36.318128+0000","last_active":"2024-03-09T15:33:26.010297+0000","last_peered":"2024-03-09T15:33:26.010297+0000","last_clean":"2024-03-09T15:33:26.010297+0000","last_became_active":"2024-03-09T14:28:36.317586+0000","last_became_peered":"2024-03-09T14:28:36.317586+0000","last_unstale":"2024-03-09T15:33:26.010297+0000","last_undegraded":"2024-03-09T15:33:26.010297+0000","last_fullsized":"2024-03-09T15:33:26.010297+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:46:55.791256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"261'129","reported_seq":682,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:52.970721+0000","last_change":"2024-03-09T14:28:38.335628+0000","last_active":"2024-03-09T15:34:52.970721+0000","last_peered":"2024-03-09T15:34:52.970721+0000","last_clean":"2024-03-09T15:34:52.970721+0000","last_became_active":"2024-03-09T14:28:38.334463+0000","last_became_peered":"2024-03-09T14:28:38.334463+0000","last_unstale":"2024-03-09T15:34:52.970721+0000","last_undegraded":"2024-03-09T15:34:52.970721+0000","last_fullsized":"2024-03-09T15:34:52.970721+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:56:33.829971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":178,"num_read_kb":172,"num_write":132,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.790948+0000","last_change":"2024-03-09T14:28:40.371173+0000","last_active":"2024-03-09T15:33:40.790948+0000","last_peered":"2024-03-09T15:33:40.790948+0000","last_clean":"2024-03-09T15:33:40.790948+0000","last_became_active":"2024-03-09T14:28:40.370436+0000","last_became_peered":"2024-03-09T14:28:40.370436+0000","last_unstale":"2024-03-09T15:33:40.790948+0000","last_undegraded":"2024-03-09T15:33:40.790948+0000","last_fullsized":"2024-03-09T15:33:40.790948+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:23:11.745507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"261'47","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:48.549510+0000","last_change":"2024-03-09T14:28:42.398975+0000","last_active":"2024-03-09T15:33:48.549510+0000","last_peered":"2024-03-09T15:33:48.549510+0000","last_clean":"2024-03-09T15:33:48.549510+0000","last_became_active":"2024-03-09T14:28:42.397058+0000","last_became_peered":"2024-03-09T14:28:42.397058+0000","last_unstale":"2024-03-09T15:33:48.549510+0000","last_undegraded":"2024-03-09T15:33:48.549510+0000","last_fullsized":"2024-03-09T15:33:48.549510+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:02:27.312655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":100,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"261'43","reported_seq":593,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.957260+0000","last_change":"2024-03-09T14:28:34.297850+0000","last_active":"2024-03-09T15:33:10.957260+0000","last_peered":"2024-03-09T15:33:10.957260+0000","last_clean":"2024-03-09T15:33:10.957260+0000","last_became_active":"2024-03-09T14:28:34.296953+0000","last_became_peered":"2024-03-09T14:28:34.296953+0000","last_unstale":"2024-03-09T15:33:10.957260+0000","last_undegraded":"2024-03-09T15:33:10.957260+0000","last_fullsized":"2024-03-09T15:33:10.957260+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:37:50.390768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":162,"num_write":75,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"261'151","reported_seq":785,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.160202+0000","last_change":"2024-03-09T14:28:32.303697+0000","last_active":"2024-03-09T15:36:06.160202+0000","last_peered":"2024-03-09T15:36:06.160202+0000","last_clean":"2024-03-09T15:36:06.160202+0000","last_became_active":"2024-03-09T14:28:32.302970+0000","last_became_peered":"2024-03-09T14:28:32.302970+0000","last_unstale":"2024-03-09T15:36:06.160202+0000","last_undegraded":"2024-03-09T15:36:06.160202+0000","last_fullsized":"2024-03-09T15:36:06.160202+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":100,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:33:55.197166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655969,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":294,"num_read_kb":94521,"num_write":620,"num_write_kb":45805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":520,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.953596+0000","last_change":"2024-03-09T14:28:19.236670+0000","last_active":"2024-03-09T15:32:55.953596+0000","last_peered":"2024-03-09T15:32:55.953596+0000","last_clean":"2024-03-09T15:32:55.953596+0000","last_became_active":"2024-03-09T14:28:19.236240+0000","last_became_peered":"2024-03-09T14:28:19.236240+0000","last_unstale":"2024-03-09T15:32:55.953596+0000","last_undegraded":"2024-03-09T15:32:55.953596+0000","last_fullsized":"2024-03-09T15:32:55.953596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:00:09.869968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.006552+0000","last_change":"2024-03-09T14:28:36.315028+0000","last_active":"2024-03-09T15:33:11.006552+0000","last_peered":"2024-03-09T15:33:11.006552+0000","last_clean":"2024-03-09T15:33:11.006552+0000","last_became_active":"2024-03-09T14:28:36.314526+0000","last_became_peered":"2024-03-09T14:28:36.314526+0000","last_unstale":"2024-03-09T15:33:11.006552+0000","last_undegraded":"2024-03-09T15:33:11.006552+0000","last_fullsized":"2024-03-09T15:33:11.006552+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:24:49.871997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"261'381","reported_seq":1059,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.947780+0000","last_change":"2024-03-09T14:28:38.335922+0000","last_active":"2024-03-09T15:36:05.947780+0000","last_peered":"2024-03-09T15:36:05.947780+0000","last_clean":"2024-03-09T15:36:05.947780+0000","last_became_active":"2024-03-09T14:28:38.334976+0000","last_became_peered":"2024-03-09T14:28:38.334976+0000","last_unstale":"2024-03-09T15:36:05.947780+0000","last_undegraded":"2024-03-09T15:36:05.947780+0000","last_fullsized":"2024-03-09T15:36:05.947780+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":300,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:27:57.990498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34012,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":631,"num_read_kb":1165,"num_write":452,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.186765+0000","last_change":"2024-03-09T14:28:40.379714+0000","last_active":"2024-03-09T15:33:16.186765+0000","last_peered":"2024-03-09T15:33:16.186765+0000","last_clean":"2024-03-09T15:33:16.186765+0000","last_became_active":"2024-03-09T14:28:40.379108+0000","last_became_peered":"2024-03-09T14:28:40.379108+0000","last_unstale":"2024-03-09T15:33:16.186765+0000","last_undegraded":"2024-03-09T15:33:16.186765+0000","last_fullsized":"2024-03-09T15:33:16.186765+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:39:21.726823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"261'101","reported_seq":626,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:41.594325+0000","last_change":"2024-03-09T14:28:42.406107+0000","last_active":"2024-03-09T15:34:41.594325+0000","last_peered":"2024-03-09T15:34:41.594325+0000","last_clean":"2024-03-09T15:34:41.594325+0000","last_became_active":"2024-03-09T14:28:42.405277+0000","last_became_peered":"2024-03-09T14:28:42.405277+0000","last_unstale":"2024-03-09T15:34:41.594325+0000","last_undegraded":"2024-03-09T15:34:41.594325+0000","last_fullsized":"2024-03-09T15:34:41.594325+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:53:29.449584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":717,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185,"num_read_kb":160,"num_write":261,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"261'43","reported_seq":575,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.006582+0000","last_change":"2024-03-09T14:28:34.298018+0000","last_active":"2024-03-09T15:33:11.006582+0000","last_peered":"2024-03-09T15:33:11.006582+0000","last_clean":"2024-03-09T15:33:11.006582+0000","last_became_active":"2024-03-09T14:28:34.296660+0000","last_became_peered":"2024-03-09T14:28:34.296660+0000","last_unstale":"2024-03-09T15:33:11.006582+0000","last_undegraded":"2024-03-09T15:33:11.006582+0000","last_fullsized":"2024-03-09T15:33:11.006582+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:37:00.270467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":129,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"261'128","reported_seq":758,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.920215+0000","last_change":"2024-03-09T14:28:32.305873+0000","last_active":"2024-03-09T15:36:05.920215+0000","last_peered":"2024-03-09T15:36:05.920215+0000","last_clean":"2024-03-09T15:36:05.920215+0000","last_became_active":"2024-03-09T14:28:32.305318+0000","last_became_peered":"2024-03-09T14:28:32.305318+0000","last_unstale":"2024-03-09T15:36:05.920215+0000","last_undegraded":"2024-03-09T15:36:05.920215+0000","last_fullsized":"2024-03-09T15:36:05.920215+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:30:36.881076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3908854,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":337,"num_read_kb":10578,"num_write":436,"num_write_kb":20505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:06.198444+0000","last_change":"2024-03-09T14:28:19.241475+0000","last_active":"2024-03-09T15:33:06.198444+0000","last_peered":"2024-03-09T15:33:06.198444+0000","last_clean":"2024-03-09T15:33:06.198444+0000","last_became_active":"2024-03-09T14:28:19.240704+0000","last_became_peered":"2024-03-09T14:28:19.240704+0000","last_unstale":"2024-03-09T15:33:06.198444+0000","last_undegraded":"2024-03-09T15:33:06.198444+0000","last_fullsized":"2024-03-09T15:33:06.198444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:07:08.993017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784406+0000","last_change":"2024-03-09T14:28:36.315481+0000","last_active":"2024-03-09T15:33:10.784406+0000","last_peered":"2024-03-09T15:33:10.784406+0000","last_clean":"2024-03-09T15:33:10.784406+0000","last_became_active":"2024-03-09T14:28:36.314676+0000","last_became_peered":"2024-03-09T14:28:36.314676+0000","last_unstale":"2024-03-09T15:33:10.784406+0000","last_undegraded":"2024-03-09T15:33:10.784406+0000","last_fullsized":"2024-03-09T15:33:10.784406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:14:11.938307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"261'197","reported_seq":778,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:20.985108+0000","last_change":"2024-03-09T14:28:38.595995+0000","last_active":"2024-03-09T15:35:20.985108+0000","last_peered":"2024-03-09T15:35:20.985108+0000","last_clean":"2024-03-09T15:35:20.985108+0000","last_became_active":"2024-03-09T14:28:38.595028+0000","last_became_peered":"2024-03-09T14:28:38.595028+0000","last_unstale":"2024-03-09T15:35:20.985108+0000","last_undegraded":"2024-03-09T15:35:20.985108+0000","last_fullsized":"2024-03-09T15:35:20.985108+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":100,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:24:00.362746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":269,"num_read_kb":262,"num_write":230,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"261'500","reported_seq":2296,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.716276+0000","last_change":"2024-03-09T14:28:40.370167+0000","last_active":"2024-03-09T15:36:05.716276+0000","last_peered":"2024-03-09T15:36:05.716276+0000","last_clean":"2024-03-09T15:36:05.716276+0000","last_became_active":"2024-03-09T14:28:40.369165+0000","last_became_peered":"2024-03-09T14:28:40.369165+0000","last_unstale":"2024-03-09T15:36:05.716276+0000","last_undegraded":"2024-03-09T15:36:05.716276+0000","last_fullsized":"2024-03-09T15:36:05.716276+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":400,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:29:22.485672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"261'76","reported_seq":564,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:15.812925+0000","last_change":"2024-03-09T14:28:42.393153+0000","last_active":"2024-03-09T15:34:15.812925+0000","last_peered":"2024-03-09T15:34:15.812925+0000","last_clean":"2024-03-09T15:34:15.812925+0000","last_became_active":"2024-03-09T14:28:42.392402+0000","last_became_peered":"2024-03-09T14:28:42.392402+0000","last_unstale":"2024-03-09T15:34:15.812925+0000","last_undegraded":"2024-03-09T15:34:15.812925+0000","last_fullsized":"2024-03-09T15:34:15.812925+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:57:45.818538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":57,"num_write":172,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"261'48","reported_seq":589,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.973653+0000","last_change":"2024-03-09T14:29:06.440753+0000","last_active":"2024-03-09T15:33:20.973653+0000","last_peered":"2024-03-09T15:33:20.973653+0000","last_clean":"2024-03-09T15:33:20.973653+0000","last_became_active":"2024-03-09T14:29:06.440556+0000","last_became_peered":"2024-03-09T14:29:06.440556+0000","last_unstale":"2024-03-09T15:33:20.973653+0000","last_undegraded":"2024-03-09T15:33:20.973653+0000","last_fullsized":"2024-03-09T15:33:20.973653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:43:05.084646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":216,"num_read_kb":182,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"261'120","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.803883+0000","last_change":"2024-03-09T14:28:32.319314+0000","last_active":"2024-03-09T15:33:35.803883+0000","last_peered":"2024-03-09T15:33:35.803883+0000","last_clean":"2024-03-09T15:33:35.803883+0000","last_became_active":"2024-03-09T14:28:32.318971+0000","last_became_peered":"2024-03-09T14:28:32.318971+0000","last_unstale":"2024-03-09T15:33:35.803883+0000","last_undegraded":"2024-03-09T15:33:35.803883+0000","last_fullsized":"2024-03-09T15:33:35.803883+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:01:00.707624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243236,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":126,"num_read_kb":8647,"num_write":451,"num_write_kb":27350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.960027+0000","last_change":"2024-03-09T14:28:19.238362+0000","last_active":"2024-03-09T15:33:20.960027+0000","last_peered":"2024-03-09T15:33:20.960027+0000","last_clean":"2024-03-09T15:33:20.960027+0000","last_became_active":"2024-03-09T14:28:19.237750+0000","last_became_peered":"2024-03-09T14:28:19.237750+0000","last_unstale":"2024-03-09T15:33:20.960027+0000","last_undegraded":"2024-03-09T15:33:20.960027+0000","last_fullsized":"2024-03-09T15:33:20.960027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:19.699145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.185710+0000","last_change":"2024-03-09T14:28:36.328518+0000","last_active":"2024-03-09T15:33:11.185710+0000","last_peered":"2024-03-09T15:33:11.185710+0000","last_clean":"2024-03-09T15:33:11.185710+0000","last_became_active":"2024-03-09T14:28:36.327601+0000","last_became_peered":"2024-03-09T14:28:36.327601+0000","last_unstale":"2024-03-09T15:33:11.185710+0000","last_undegraded":"2024-03-09T15:33:11.185710+0000","last_fullsized":"2024-03-09T15:33:11.185710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:41:26.807068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"261'106","reported_seq":661,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.186823+0000","last_change":"2024-03-09T14:28:38.348479+0000","last_active":"2024-03-09T15:33:16.186823+0000","last_peered":"2024-03-09T15:33:16.186823+0000","last_clean":"2024-03-09T15:33:16.186823+0000","last_became_active":"2024-03-09T14:28:38.347750+0000","last_became_peered":"2024-03-09T14:28:38.347750+0000","last_unstale":"2024-03-09T15:33:16.186823+0000","last_undegraded":"2024-03-09T15:33:16.186823+0000","last_fullsized":"2024-03-09T15:33:16.186823+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:26:08.370238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"261'513","reported_seq":2485,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.727541+0000","last_change":"2024-03-09T14:28:40.387580+0000","last_active":"2024-03-09T15:36:05.727541+0000","last_peered":"2024-03-09T15:36:05.727541+0000","last_clean":"2024-03-09T15:36:05.727541+0000","last_became_active":"2024-03-09T14:28:40.386859+0000","last_became_peered":"2024-03-09T14:28:40.386859+0000","last_unstale":"2024-03-09T15:36:05.727541+0000","last_undegraded":"2024-03-09T15:36:05.727541+0000","last_fullsized":"2024-03-09T15:36:05.727541+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":500,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:17:24.125699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"261'106","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.987014+0000","last_change":"2024-03-09T14:28:42.406871+0000","last_active":"2024-03-09T15:31:45.987014+0000","last_peered":"2024-03-09T15:31:45.987014+0000","last_clean":"2024-03-09T15:31:45.987014+0000","last_became_active":"2024-03-09T14:28:42.406139+0000","last_became_peered":"2024-03-09T14:28:42.406139+0000","last_unstale":"2024-03-09T15:31:45.987014+0000","last_undegraded":"2024-03-09T15:31:45.987014+0000","last_fullsized":"2024-03-09T15:31:45.987014+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:24:57.719382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":65,"num_write":235,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"261'216","reported_seq":787,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:48.579968+0000","last_change":"2024-03-09T14:28:34.301025+0000","last_active":"2024-03-09T15:33:48.579968+0000","last_peered":"2024-03-09T15:33:48.579968+0000","last_clean":"2024-03-09T15:33:48.579968+0000","last_became_active":"2024-03-09T14:28:34.299988+0000","last_became_peered":"2024-03-09T14:28:34.299988+0000","last_unstale":"2024-03-09T15:33:48.579968+0000","last_undegraded":"2024-03-09T15:33:48.579968+0000","last_fullsized":"2024-03-09T15:33:48.579968+0000","mapping_epoch":22,"log_start":"258'200","ondisk_log_start":"258'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":200,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:05:09.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":843,"num_read_kb":645,"num_write":377,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"261'132","reported_seq":685,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.960146+0000","last_change":"2024-03-09T14:28:32.299582+0000","last_active":"2024-03-09T15:36:05.960146+0000","last_peered":"2024-03-09T15:36:05.960146+0000","last_clean":"2024-03-09T15:36:05.960146+0000","last_became_active":"2024-03-09T14:28:32.298977+0000","last_became_peered":"2024-03-09T14:28:32.298977+0000","last_unstale":"2024-03-09T15:36:05.960146+0000","last_undegraded":"2024-03-09T15:36:05.960146+0000","last_fullsized":"2024-03-09T15:36:05.960146+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:34:44.279787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243131,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":155,"num_read_kb":5781,"num_write":492,"num_write_kb":37567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.786634+0000","last_change":"2024-03-09T14:28:19.242561+0000","last_active":"2024-03-09T15:33:20.786634+0000","last_peered":"2024-03-09T15:33:20.786634+0000","last_clean":"2024-03-09T15:33:20.786634+0000","last_became_active":"2024-03-09T14:28:19.241700+0000","last_became_peered":"2024-03-09T14:28:19.241700+0000","last_unstale":"2024-03-09T15:33:20.786634+0000","last_undegraded":"2024-03-09T15:33:20.786634+0000","last_fullsized":"2024-03-09T15:33:20.786634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:15.462353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"32'3","reported_seq":822,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.255551+0000","last_change":"2024-03-09T14:28:36.317874+0000","last_active":"2024-03-09T15:36:05.255551+0000","last_peered":"2024-03-09T15:36:05.255551+0000","last_clean":"2024-03-09T15:36:05.255551+0000","last_became_active":"2024-03-09T14:28:36.317327+0000","last_became_peered":"2024-03-09T14:28:36.317327+0000","last_unstale":"2024-03-09T15:36:05.255551+0000","last_undegraded":"2024-03-09T15:36:05.255551+0000","last_fullsized":"2024-03-09T15:36:05.255551+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:07.321787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":484,"num_read_kb":482,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"261'465","reported_seq":1200,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.224674+0000","last_change":"2024-03-09T14:28:38.603808+0000","last_active":"2024-03-09T15:36:06.224674+0000","last_peered":"2024-03-09T15:36:06.224674+0000","last_clean":"2024-03-09T15:36:06.224674+0000","last_became_active":"2024-03-09T14:28:38.602908+0000","last_became_peered":"2024-03-09T14:28:38.602908+0000","last_unstale":"2024-03-09T15:36:06.224674+0000","last_undegraded":"2024-03-09T15:36:06.224674+0000","last_fullsized":"2024-03-09T15:36:06.224674+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":400,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:20:14.798348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58890,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":785,"num_read_kb":1500,"num_write":543,"num_write_kb":342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:15.799092+0000","last_change":"2024-03-09T14:28:40.376657+0000","last_active":"2024-03-09T15:33:15.799092+0000","last_peered":"2024-03-09T15:33:15.799092+0000","last_clean":"2024-03-09T15:33:15.799092+0000","last_became_active":"2024-03-09T14:28:40.375884+0000","last_became_peered":"2024-03-09T14:28:40.375884+0000","last_unstale":"2024-03-09T15:33:15.799092+0000","last_undegraded":"2024-03-09T15:33:15.799092+0000","last_fullsized":"2024-03-09T15:33:15.799092+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:28:12.389403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"261'54","reported_seq":551,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:33.602971+0000","last_change":"2024-03-09T14:28:42.406761+0000","last_active":"2024-03-09T15:34:33.602971+0000","last_peered":"2024-03-09T15:34:33.602971+0000","last_clean":"2024-03-09T15:34:33.602971+0000","last_became_active":"2024-03-09T14:28:42.405998+0000","last_became_peered":"2024-03-09T14:28:42.405998+0000","last_unstale":"2024-03-09T15:34:33.602971+0000","last_undegraded":"2024-03-09T15:34:33.602971+0000","last_fullsized":"2024-03-09T15:34:33.602971+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:02:27.576333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":52,"num_write":123,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1f","version":"261'146","reported_seq":723,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:09.596752+0000","last_change":"2024-03-09T14:28:42.410137+0000","last_active":"2024-03-09T15:34:09.596752+0000","last_peered":"2024-03-09T15:34:09.596752+0000","last_clean":"2024-03-09T15:34:09.596752+0000","last_became_active":"2024-03-09T14:28:42.408774+0000","last_became_peered":"2024-03-09T14:28:42.408774+0000","last_unstale":"2024-03-09T15:34:09.596752+0000","last_undegraded":"2024-03-09T15:34:09.596752+0000","last_fullsized":"2024-03-09T15:34:09.596752+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:50:49.883718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":256,"num_read_kb":233,"num_write":254,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"261'183","reported_seq":743,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:44.582989+0000","last_change":"2024-03-09T14:28:34.315742+0000","last_active":"2024-03-09T15:33:44.582989+0000","last_peered":"2024-03-09T15:33:44.582989+0000","last_clean":"2024-03-09T15:33:44.582989+0000","last_became_active":"2024-03-09T14:28:34.314893+0000","last_became_peered":"2024-03-09T14:28:34.314893+0000","last_unstale":"2024-03-09T15:33:44.582989+0000","last_undegraded":"2024-03-09T15:33:44.582989+0000","last_fullsized":"2024-03-09T15:33:44.582989+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":100,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:23:46.738791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":706,"num_read_kb":592,"num_write":291,"num_write_kb":155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"261'108","reported_seq":652,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.995245+0000","last_change":"2024-03-09T14:28:32.322792+0000","last_active":"2024-03-09T15:36:05.995245+0000","last_peered":"2024-03-09T15:36:05.995245+0000","last_clean":"2024-03-09T15:36:05.995245+0000","last_became_active":"2024-03-09T14:28:32.322364+0000","last_became_peered":"2024-03-09T14:28:32.322364+0000","last_unstale":"2024-03-09T15:36:05.995245+0000","last_undegraded":"2024-03-09T15:36:05.995245+0000","last_fullsized":"2024-03-09T15:36:05.995245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:58:42.664679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":128,"num_read_kb":128,"num_write":368,"num_write_kb":27813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.134035+0000","last_change":"2024-03-09T14:28:36.314599+0000","last_active":"2024-03-09T15:31:36.134035+0000","last_peered":"2024-03-09T15:31:36.134035+0000","last_clean":"2024-03-09T15:31:36.134035+0000","last_became_active":"2024-03-09T14:28:36.313984+0000","last_became_peered":"2024-03-09T14:28:36.313984+0000","last_unstale":"2024-03-09T15:31:36.134035+0000","last_undegraded":"2024-03-09T15:31:36.134035+0000","last_fullsized":"2024-03-09T15:31:36.134035+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:41:00.980683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"261'188","reported_seq":766,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.703452+0000","last_change":"2024-03-09T14:28:38.345412+0000","last_active":"2024-03-09T15:35:58.703452+0000","last_peered":"2024-03-09T15:35:58.703452+0000","last_clean":"2024-03-09T15:35:58.703452+0000","last_became_active":"2024-03-09T14:28:38.338262+0000","last_became_peered":"2024-03-09T14:28:38.338262+0000","last_unstale":"2024-03-09T15:35:58.703452+0000","last_undegraded":"2024-03-09T15:35:58.703452+0000","last_fullsized":"2024-03-09T15:35:58.703452+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":100,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:08:16.291678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29268,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":306,"num_read_kb":562,"num_write":216,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":489,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.777786+0000","last_change":"2024-03-09T14:28:40.390067+0000","last_active":"2024-03-09T15:31:40.777786+0000","last_peered":"2024-03-09T15:31:40.777786+0000","last_clean":"2024-03-09T15:31:40.777786+0000","last_became_active":"2024-03-09T14:28:40.389277+0000","last_became_peered":"2024-03-09T14:28:40.389277+0000","last_unstale":"2024-03-09T15:31:40.777786+0000","last_undegraded":"2024-03-09T15:31:40.777786+0000","last_fullsized":"2024-03-09T15:31:40.777786+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:02:25.553702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"261'54","reported_seq":549,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.795784+0000","last_change":"2024-03-09T14:28:42.411135+0000","last_active":"2024-03-09T15:33:00.795784+0000","last_peered":"2024-03-09T15:33:00.795784+0000","last_clean":"2024-03-09T15:33:00.795784+0000","last_became_active":"2024-03-09T14:28:42.409229+0000","last_became_peered":"2024-03-09T14:28:42.409229+0000","last_unstale":"2024-03-09T15:33:00.795784+0000","last_undegraded":"2024-03-09T15:33:00.795784+0000","last_fullsized":"2024-03-09T15:33:00.795784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:17:09.392755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":828,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":44,"num_write":116,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"261'78","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.134077+0000","last_change":"2024-03-09T14:28:34.320825+0000","last_active":"2024-03-09T15:31:36.134077+0000","last_peered":"2024-03-09T15:31:36.134077+0000","last_clean":"2024-03-09T15:31:36.134077+0000","last_became_active":"2024-03-09T14:28:34.320474+0000","last_became_peered":"2024-03-09T14:28:34.320474+0000","last_unstale":"2024-03-09T15:31:36.134077+0000","last_undegraded":"2024-03-09T15:31:36.134077+0000","last_fullsized":"2024-03-09T15:31:36.134077+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:15:48.878005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":231,"num_write":128,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"261'149","reported_seq":781,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.185355+0000","last_change":"2024-03-09T14:28:32.317900+0000","last_active":"2024-03-09T15:36:06.185355+0000","last_peered":"2024-03-09T15:36:06.185355+0000","last_clean":"2024-03-09T15:36:06.185355+0000","last_became_active":"2024-03-09T14:28:32.315355+0000","last_became_peered":"2024-03-09T14:28:32.315355+0000","last_unstale":"2024-03-09T15:36:06.185355+0000","last_undegraded":"2024-03-09T15:36:06.185355+0000","last_fullsized":"2024-03-09T15:36:06.185355+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:28:48.531373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655932,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":315,"num_read_kb":85989,"num_write":571,"num_write_kb":44811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776757+0000","last_change":"2024-03-09T14:28:36.311910+0000","last_active":"2024-03-09T15:31:35.776757+0000","last_peered":"2024-03-09T15:31:35.776757+0000","last_clean":"2024-03-09T15:31:35.776757+0000","last_became_active":"2024-03-09T14:28:36.311487+0000","last_became_peered":"2024-03-09T14:28:36.311487+0000","last_unstale":"2024-03-09T15:31:35.776757+0000","last_undegraded":"2024-03-09T15:31:35.776757+0000","last_fullsized":"2024-03-09T15:31:35.776757+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:04:59.933448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"261'39","reported_seq":550,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:31.436658+0000","last_change":"2024-03-09T14:28:38.333369+0000","last_active":"2024-03-09T15:33:31.436658+0000","last_peered":"2024-03-09T15:33:31.436658+0000","last_clean":"2024-03-09T15:33:31.436658+0000","last_became_active":"2024-03-09T14:28:38.332610+0000","last_became_peered":"2024-03-09T14:28:38.332610+0000","last_unstale":"2024-03-09T15:33:31.436658+0000","last_undegraded":"2024-03-09T15:33:31.436658+0000","last_fullsized":"2024-03-09T15:33:31.436658+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:56:36.283073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":41,"num_write":30,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":498,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.178031+0000","last_change":"2024-03-09T14:28:40.368708+0000","last_active":"2024-03-09T15:31:41.178031+0000","last_peered":"2024-03-09T15:31:41.178031+0000","last_clean":"2024-03-09T15:31:41.178031+0000","last_became_active":"2024-03-09T14:28:40.367795+0000","last_became_peered":"2024-03-09T14:28:40.367795+0000","last_unstale":"2024-03-09T15:31:41.178031+0000","last_undegraded":"2024-03-09T15:31:41.178031+0000","last_fullsized":"2024-03-09T15:31:41.178031+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:46:56.597658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"261'170","reported_seq":688,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:44.567149+0000","last_change":"2024-03-09T14:28:42.410358+0000","last_active":"2024-03-09T15:33:44.567149+0000","last_peered":"2024-03-09T15:33:44.567149+0000","last_clean":"2024-03-09T15:33:44.567149+0000","last_became_active":"2024-03-09T14:28:42.408913+0000","last_became_peered":"2024-03-09T14:28:42.408913+0000","last_unstale":"2024-03-09T15:33:44.567149+0000","last_undegraded":"2024-03-09T15:33:44.567149+0000","last_fullsized":"2024-03-09T15:33:44.567149+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":100,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:06:25.964093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":208,"num_write":351,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"261'54","reported_seq":593,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:52.364811+0000","last_change":"2024-03-09T14:28:34.331192+0000","last_active":"2024-03-09T15:31:52.364811+0000","last_peered":"2024-03-09T15:31:52.364811+0000","last_clean":"2024-03-09T15:31:52.364811+0000","last_became_active":"2024-03-09T14:28:34.330813+0000","last_became_peered":"2024-03-09T14:28:34.330813+0000","last_unstale":"2024-03-09T15:31:52.364811+0000","last_undegraded":"2024-03-09T15:31:52.364811+0000","last_fullsized":"2024-03-09T15:31:52.364811+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:02:15.115504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":220,"num_read_kb":178,"num_write":76,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"261'114","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.113284+0000","last_change":"2024-03-09T14:28:32.313709+0000","last_active":"2024-03-09T15:36:06.113284+0000","last_peered":"2024-03-09T15:36:06.113284+0000","last_clean":"2024-03-09T15:36:06.113284+0000","last_became_active":"2024-03-09T14:28:32.311843+0000","last_became_peered":"2024-03-09T14:28:32.311843+0000","last_unstale":"2024-03-09T15:36:06.113284+0000","last_undegraded":"2024-03-09T15:36:06.113284+0000","last_fullsized":"2024-03-09T15:36:06.113284+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:51:11.937911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1830596,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":144,"num_read_kb":5348,"num_write":462,"num_write_kb":35385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.176563+0000","last_change":"2024-03-09T14:28:36.324124+0000","last_active":"2024-03-09T15:31:36.176563+0000","last_peered":"2024-03-09T15:31:36.176563+0000","last_clean":"2024-03-09T15:31:36.176563+0000","last_became_active":"2024-03-09T14:28:36.323335+0000","last_became_peered":"2024-03-09T14:28:36.323335+0000","last_unstale":"2024-03-09T15:31:36.176563+0000","last_undegraded":"2024-03-09T15:31:36.176563+0000","last_fullsized":"2024-03-09T15:31:36.176563+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:09:55.020425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"261'219","reported_seq":812,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.676436+0000","last_change":"2024-03-09T14:28:38.345760+0000","last_active":"2024-03-09T15:35:58.676436+0000","last_peered":"2024-03-09T15:35:58.676436+0000","last_clean":"2024-03-09T15:35:58.676436+0000","last_became_active":"2024-03-09T14:28:38.344951+0000","last_became_peered":"2024-03-09T14:28:38.344951+0000","last_unstale":"2024-03-09T15:35:58.676436+0000","last_undegraded":"2024-03-09T15:35:58.676436+0000","last_fullsized":"2024-03-09T15:35:58.676436+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":200,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:52:01.179169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24078,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":344,"num_read_kb":584,"num_write":240,"num_write_kb":124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.766053+0000","last_change":"2024-03-09T14:28:40.374861+0000","last_active":"2024-03-09T15:31:40.766053+0000","last_peered":"2024-03-09T15:31:40.766053+0000","last_clean":"2024-03-09T15:31:40.766053+0000","last_became_active":"2024-03-09T14:28:40.370313+0000","last_became_peered":"2024-03-09T14:28:40.370313+0000","last_unstale":"2024-03-09T15:31:40.766053+0000","last_undegraded":"2024-03-09T15:31:40.766053+0000","last_fullsized":"2024-03-09T15:31:40.766053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:20:51.501674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"261'45","reported_seq":538,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:07.675530+0000","last_change":"2024-03-09T14:28:42.405094+0000","last_active":"2024-03-09T15:35:07.675530+0000","last_peered":"2024-03-09T15:35:07.675530+0000","last_clean":"2024-03-09T15:35:07.675530+0000","last_became_active":"2024-03-09T14:28:42.404360+0000","last_became_peered":"2024-03-09T14:28:42.404360+0000","last_unstale":"2024-03-09T15:35:07.675530+0000","last_undegraded":"2024-03-09T15:35:07.675530+0000","last_fullsized":"2024-03-09T15:35:07.675530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:33:45.977796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":43,"num_write":116,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"261'87","reported_seq":640,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776531+0000","last_change":"2024-03-09T14:28:34.313014+0000","last_active":"2024-03-09T15:31:35.776531+0000","last_peered":"2024-03-09T15:31:35.776531+0000","last_clean":"2024-03-09T15:31:35.776531+0000","last_became_active":"2024-03-09T14:28:34.305301+0000","last_became_peered":"2024-03-09T14:28:34.305301+0000","last_unstale":"2024-03-09T15:31:35.776531+0000","last_undegraded":"2024-03-09T15:31:35.776531+0000","last_fullsized":"2024-03-09T15:31:35.776531+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:46:37.987444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":317,"num_write":145,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"261'120","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.996239+0000","last_change":"2024-03-09T14:28:32.293199+0000","last_active":"2024-03-09T15:36:05.996239+0000","last_peered":"2024-03-09T15:36:05.996239+0000","last_clean":"2024-03-09T15:36:05.996239+0000","last_became_active":"2024-03-09T14:28:32.292393+0000","last_became_peered":"2024-03-09T14:28:32.292393+0000","last_unstale":"2024-03-09T15:36:05.996239+0000","last_undegraded":"2024-03-09T15:36:05.996239+0000","last_fullsized":"2024-03-09T15:36:05.996239+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:58:50.345089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7206306,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":165,"num_read_kb":14616,"num_write":445,"num_write_kb":23484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.764963+0000","last_change":"2024-03-09T14:28:36.329882+0000","last_active":"2024-03-09T15:31:35.764963+0000","last_peered":"2024-03-09T15:31:35.764963+0000","last_clean":"2024-03-09T15:31:35.764963+0000","last_became_active":"2024-03-09T14:28:36.328946+0000","last_became_peered":"2024-03-09T14:28:36.328946+0000","last_unstale":"2024-03-09T15:31:35.764963+0000","last_undegraded":"2024-03-09T15:31:35.764963+0000","last_fullsized":"2024-03-09T15:31:35.764963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:23:54.027658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"261'168","reported_seq":752,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.194863+0000","last_change":"2024-03-09T14:28:38.336043+0000","last_active":"2024-03-09T15:36:06.194863+0000","last_peered":"2024-03-09T15:36:06.194863+0000","last_clean":"2024-03-09T15:36:06.194863+0000","last_became_active":"2024-03-09T14:28:38.335107+0000","last_became_peered":"2024-03-09T14:28:38.335107+0000","last_unstale":"2024-03-09T15:36:06.194863+0000","last_undegraded":"2024-03-09T15:36:06.194863+0000","last_fullsized":"2024-03-09T15:36:06.194863+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":100,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:08:12.713928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9765,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":273,"num_read_kb":518,"num_write":186,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.985823+0000","last_change":"2024-03-09T14:28:40.364509+0000","last_active":"2024-03-09T15:31:40.985823+0000","last_peered":"2024-03-09T15:31:40.985823+0000","last_clean":"2024-03-09T15:31:40.985823+0000","last_became_active":"2024-03-09T14:28:40.363778+0000","last_became_peered":"2024-03-09T14:28:40.363778+0000","last_unstale":"2024-03-09T15:31:40.985823+0000","last_undegraded":"2024-03-09T15:31:40.985823+0000","last_fullsized":"2024-03-09T15:31:40.985823+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:49:40.990617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"261'159","reported_seq":733,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784514+0000","last_change":"2024-03-09T14:28:34.325746+0000","last_active":"2024-03-09T15:33:10.784514+0000","last_peered":"2024-03-09T15:33:10.784514+0000","last_clean":"2024-03-09T15:33:10.784514+0000","last_became_active":"2024-03-09T14:28:34.325031+0000","last_became_peered":"2024-03-09T14:28:34.325031+0000","last_unstale":"2024-03-09T15:33:10.784514+0000","last_undegraded":"2024-03-09T15:33:10.784514+0000","last_fullsized":"2024-03-09T15:33:10.784514+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:12:45.802989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":584,"num_read_kb":461,"num_write":254,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"261'97","reported_seq":636,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:52.210937+0000","last_change":"2024-03-09T14:28:32.296634+0000","last_active":"2024-03-09T15:31:52.210937+0000","last_peered":"2024-03-09T15:31:52.210937+0000","last_clean":"2024-03-09T15:31:52.210937+0000","last_became_active":"2024-03-09T14:28:32.296179+0000","last_became_peered":"2024-03-09T14:28:32.296179+0000","last_unstale":"2024-03-09T15:31:52.210937+0000","last_undegraded":"2024-03-09T15:31:52.210937+0000","last_fullsized":"2024-03-09T15:31:52.210937+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:07:03.555863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263400,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":122,"num_read_kb":11385,"num_write":357,"num_write_kb":17181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:51.002216+0000","last_change":"2024-03-09T14:28:36.320059+0000","last_active":"2024-03-09T15:32:51.002216+0000","last_peered":"2024-03-09T15:32:51.002216+0000","last_clean":"2024-03-09T15:32:51.002216+0000","last_became_active":"2024-03-09T14:28:36.319702+0000","last_became_peered":"2024-03-09T14:28:36.319702+0000","last_unstale":"2024-03-09T15:32:51.002216+0000","last_undegraded":"2024-03-09T15:32:51.002216+0000","last_fullsized":"2024-03-09T15:32:51.002216+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:38:28.003376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"261'309","reported_seq":944,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.877702+0000","last_change":"2024-03-09T14:28:38.333414+0000","last_active":"2024-03-09T15:36:05.877702+0000","last_peered":"2024-03-09T15:36:05.877702+0000","last_clean":"2024-03-09T15:36:05.877702+0000","last_became_active":"2024-03-09T14:28:38.332661+0000","last_became_peered":"2024-03-09T14:28:38.332661+0000","last_unstale":"2024-03-09T15:36:05.877702+0000","last_undegraded":"2024-03-09T15:36:05.877702+0000","last_fullsized":"2024-03-09T15:36:05.877702+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":200,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:09:54.404475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34580,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":556,"num_read_kb":1111,"num_write":442,"num_write_kb":360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:11.163194+0000","last_change":"2024-03-09T14:28:40.379385+0000","last_active":"2024-03-09T15:31:11.163194+0000","last_peered":"2024-03-09T15:31:11.163194+0000","last_clean":"2024-03-09T15:31:11.163194+0000","last_became_active":"2024-03-09T14:28:40.378829+0000","last_became_peered":"2024-03-09T14:28:40.378829+0000","last_unstale":"2024-03-09T15:31:11.163194+0000","last_undegraded":"2024-03-09T15:31:11.163194+0000","last_fullsized":"2024-03-09T15:31:11.163194+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:25:24.764687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"261'124","reported_seq":623,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.333094+0000","last_change":"2024-03-09T14:28:42.393487+0000","last_active":"2024-03-09T15:33:00.333094+0000","last_peered":"2024-03-09T15:33:00.333094+0000","last_clean":"2024-03-09T15:33:00.333094+0000","last_became_active":"2024-03-09T14:28:42.392509+0000","last_became_peered":"2024-03-09T14:28:42.392509+0000","last_unstale":"2024-03-09T15:33:00.333094+0000","last_undegraded":"2024-03-09T15:33:00.333094+0000","last_fullsized":"2024-03-09T15:33:00.333094+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:53:46.680420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":643,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":173,"num_write":275,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"261'132","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:51.002176+0000","last_change":"2024-03-09T14:28:34.330979+0000","last_active":"2024-03-09T15:32:51.002176+0000","last_peered":"2024-03-09T15:32:51.002176+0000","last_clean":"2024-03-09T15:32:51.002176+0000","last_became_active":"2024-03-09T14:28:34.330686+0000","last_became_peered":"2024-03-09T14:28:34.330686+0000","last_unstale":"2024-03-09T15:32:51.002176+0000","last_undegraded":"2024-03-09T15:32:51.002176+0000","last_fullsized":"2024-03-09T15:32:51.002176+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:07:36.006587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":527,"num_read_kb":451,"num_write":212,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"261'111","reported_seq":649,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.905029+0000","last_change":"2024-03-09T14:28:32.301602+0000","last_active":"2024-03-09T15:36:05.905029+0000","last_peered":"2024-03-09T15:36:05.905029+0000","last_clean":"2024-03-09T15:36:05.905029+0000","last_became_active":"2024-03-09T14:28:32.301087+0000","last_became_peered":"2024-03-09T14:28:32.301087+0000","last_unstale":"2024-03-09T15:36:05.905029+0000","last_undegraded":"2024-03-09T15:36:05.905029+0000","last_fullsized":"2024-03-09T15:36:05.905029+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:13:11.042530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":115,"num_read_kb":3393,"num_write":438,"num_write_kb":23270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784490+0000","last_change":"2024-03-09T14:28:36.329619+0000","last_active":"2024-03-09T15:33:10.784490+0000","last_peered":"2024-03-09T15:33:10.784490+0000","last_clean":"2024-03-09T15:33:10.784490+0000","last_became_active":"2024-03-09T14:28:36.328814+0000","last_became_peered":"2024-03-09T14:28:36.328814+0000","last_unstale":"2024-03-09T15:33:10.784490+0000","last_undegraded":"2024-03-09T15:33:10.784490+0000","last_fullsized":"2024-03-09T15:33:10.784490+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:24:43.021216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"261'252","reported_seq":855,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.221152+0000","last_change":"2024-03-09T14:28:38.332894+0000","last_active":"2024-03-09T15:36:06.221152+0000","last_peered":"2024-03-09T15:36:06.221152+0000","last_clean":"2024-03-09T15:36:06.221152+0000","last_became_active":"2024-03-09T14:28:38.331930+0000","last_became_peered":"2024-03-09T14:28:38.331930+0000","last_unstale":"2024-03-09T15:36:06.221152+0000","last_undegraded":"2024-03-09T15:36:06.221152+0000","last_fullsized":"2024-03-09T15:36:06.221152+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":200,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:36:58.809794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21830,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":424,"num_read_kb":798,"num_write":342,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:11.128169+0000","last_change":"2024-03-09T14:28:40.367660+0000","last_active":"2024-03-09T15:31:11.128169+0000","last_peered":"2024-03-09T15:31:11.128169+0000","last_clean":"2024-03-09T15:31:11.128169+0000","last_became_active":"2024-03-09T14:28:40.366723+0000","last_became_peered":"2024-03-09T14:28:40.366723+0000","last_unstale":"2024-03-09T15:31:11.128169+0000","last_undegraded":"2024-03-09T15:31:11.128169+0000","last_fullsized":"2024-03-09T15:31:11.128169+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:42:24.773777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"261'108","reported_seq":613,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:31.400225+0000","last_change":"2024-03-09T14:28:42.395993+0000","last_active":"2024-03-09T15:33:31.400225+0000","last_peered":"2024-03-09T15:33:31.400225+0000","last_clean":"2024-03-09T15:33:31.400225+0000","last_became_active":"2024-03-09T14:28:42.395375+0000","last_became_peered":"2024-03-09T14:28:42.395375+0000","last_unstale":"2024-03-09T15:33:31.400225+0000","last_undegraded":"2024-03-09T15:33:31.400225+0000","last_fullsized":"2024-03-09T15:33:31.400225+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:57:19.143770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1199,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":139,"num_read_kb":136,"num_write":341,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"261'99","reported_seq":686,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:45.201625+0000","last_change":"2024-03-09T14:28:34.312670+0000","last_active":"2024-03-09T15:34:45.201625+0000","last_peered":"2024-03-09T15:34:45.201625+0000","last_clean":"2024-03-09T15:34:45.201625+0000","last_became_active":"2024-03-09T14:28:34.312297+0000","last_became_peered":"2024-03-09T14:28:34.312297+0000","last_unstale":"2024-03-09T15:34:45.201625+0000","last_undegraded":"2024-03-09T15:34:45.201625+0000","last_fullsized":"2024-03-09T15:34:45.201625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:36:43.430324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":415,"num_read_kb":347,"num_write":146,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"261'101","reported_seq":669,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.985205+0000","last_change":"2024-03-09T14:28:32.314186+0000","last_active":"2024-03-09T15:36:05.985205+0000","last_peered":"2024-03-09T15:36:05.985205+0000","last_clean":"2024-03-09T15:36:05.985205+0000","last_became_active":"2024-03-09T14:28:32.312937+0000","last_became_peered":"2024-03-09T14:28:32.312937+0000","last_unstale":"2024-03-09T15:36:05.985205+0000","last_undegraded":"2024-03-09T15:36:05.985205+0000","last_fullsized":"2024-03-09T15:36:05.985205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:33:36.448497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262330,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":144,"num_read_kb":47424,"num_write":415,"num_write_kb":24173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191452+0000","last_change":"2024-03-09T14:28:36.316936+0000","last_active":"2024-03-09T15:33:36.191452+0000","last_peered":"2024-03-09T15:33:36.191452+0000","last_clean":"2024-03-09T15:33:36.191452+0000","last_became_active":"2024-03-09T14:28:36.316094+0000","last_became_peered":"2024-03-09T14:28:36.316094+0000","last_unstale":"2024-03-09T15:33:36.191452+0000","last_undegraded":"2024-03-09T15:33:36.191452+0000","last_fullsized":"2024-03-09T15:33:36.191452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:41:42.620548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"261'188","reported_seq":770,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.713172+0000","last_change":"2024-03-09T14:28:38.596294+0000","last_active":"2024-03-09T15:35:58.713172+0000","last_peered":"2024-03-09T15:35:58.713172+0000","last_clean":"2024-03-09T15:35:58.713172+0000","last_became_active":"2024-03-09T14:28:38.595522+0000","last_became_peered":"2024-03-09T14:28:38.595522+0000","last_unstale":"2024-03-09T15:35:58.713172+0000","last_undegraded":"2024-03-09T15:35:58.713172+0000","last_fullsized":"2024-03-09T15:35:58.713172+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":100,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:20:49.471413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25352,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":302,"num_read_kb":534,"num_write":212,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:50.952318+0000","last_change":"2024-03-09T14:28:40.377325+0000","last_active":"2024-03-09T15:32:50.952318+0000","last_peered":"2024-03-09T15:32:50.952318+0000","last_clean":"2024-03-09T15:32:50.952318+0000","last_became_active":"2024-03-09T14:28:40.376549+0000","last_became_peered":"2024-03-09T14:28:40.376549+0000","last_unstale":"2024-03-09T15:32:50.952318+0000","last_undegraded":"2024-03-09T15:32:50.952318+0000","last_fullsized":"2024-03-09T15:32:50.952318+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:59:06.393597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"261'42","reported_seq":537,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.728760+0000","last_change":"2024-03-09T14:28:42.395593+0000","last_active":"2024-03-09T15:35:58.728760+0000","last_peered":"2024-03-09T15:35:58.728760+0000","last_clean":"2024-03-09T15:35:58.728760+0000","last_became_active":"2024-03-09T14:28:42.394780+0000","last_became_peered":"2024-03-09T14:28:42.394780+0000","last_unstale":"2024-03-09T15:35:58.728760+0000","last_undegraded":"2024-03-09T15:35:58.728760+0000","last_fullsized":"2024-03-09T15:35:58.728760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:21:45.395383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":84,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"261'68","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:22.422461+0000","last_change":"2024-03-09T14:28:34.313519+0000","last_active":"2024-03-09T15:34:22.422461+0000","last_peered":"2024-03-09T15:34:22.422461+0000","last_clean":"2024-03-09T15:34:22.422461+0000","last_became_active":"2024-03-09T14:28:34.313125+0000","last_became_peered":"2024-03-09T14:28:34.313125+0000","last_unstale":"2024-03-09T15:34:22.422461+0000","last_undegraded":"2024-03-09T15:34:22.422461+0000","last_fullsized":"2024-03-09T15:34:22.422461+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:57:07.167382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"261'175","reported_seq":779,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.986932+0000","last_change":"2024-03-09T14:28:32.304248+0000","last_active":"2024-03-09T15:36:05.986932+0000","last_peered":"2024-03-09T15:36:05.986932+0000","last_clean":"2024-03-09T15:36:05.986932+0000","last_became_active":"2024-03-09T14:28:32.303835+0000","last_became_peered":"2024-03-09T14:28:32.303835+0000","last_unstale":"2024-03-09T15:36:05.986932+0000","last_undegraded":"2024-03-09T15:36:05.986932+0000","last_fullsized":"2024-03-09T15:36:05.986932+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":100,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:32:30.929760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":359,"num_read_kb":1431,"num_write":987,"num_write_kb":8832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963406+0000","last_change":"2024-03-09T14:28:36.322532+0000","last_active":"2024-03-09T15:33:35.963406+0000","last_peered":"2024-03-09T15:33:35.963406+0000","last_clean":"2024-03-09T15:33:35.963406+0000","last_became_active":"2024-03-09T14:28:36.322060+0000","last_became_peered":"2024-03-09T14:28:36.322060+0000","last_unstale":"2024-03-09T15:33:35.963406+0000","last_undegraded":"2024-03-09T15:33:35.963406+0000","last_fullsized":"2024-03-09T15:33:35.963406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:48:28.324190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"261'89","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.180393+0000","last_change":"2024-03-09T14:28:38.596598+0000","last_active":"2024-03-09T15:32:46.180393+0000","last_peered":"2024-03-09T15:32:46.180393+0000","last_clean":"2024-03-09T15:32:46.180393+0000","last_became_active":"2024-03-09T14:28:38.595749+0000","last_became_peered":"2024-03-09T14:28:38.595749+0000","last_unstale":"2024-03-09T15:32:46.180393+0000","last_undegraded":"2024-03-09T15:32:46.180393+0000","last_fullsized":"2024-03-09T15:32:46.180393+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:47:35.161583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":132,"num_read_kb":127,"num_write":124,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"261'488","reported_seq":2256,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.713372+0000","last_change":"2024-03-09T14:28:40.377689+0000","last_active":"2024-03-09T15:36:05.713372+0000","last_peered":"2024-03-09T15:36:05.713372+0000","last_clean":"2024-03-09T15:36:05.713372+0000","last_became_active":"2024-03-09T14:28:40.376946+0000","last_became_peered":"2024-03-09T14:28:40.376946+0000","last_unstale":"2024-03-09T15:36:05.713372+0000","last_undegraded":"2024-03-09T15:36:05.713372+0000","last_fullsized":"2024-03-09T15:36:05.713372+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":400,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:50:25.914007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.10","version":"261'45","reported_seq":549,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:48.564964+0000","last_change":"2024-03-09T14:28:42.399023+0000","last_active":"2024-03-09T15:33:48.564964+0000","last_peered":"2024-03-09T15:33:48.564964+0000","last_clean":"2024-03-09T15:33:48.564964+0000","last_became_active":"2024-03-09T14:28:42.397829+0000","last_became_peered":"2024-03-09T14:28:42.397829+0000","last_unstale":"2024-03-09T15:33:48.564964+0000","last_undegraded":"2024-03-09T15:33:48.564964+0000","last_fullsized":"2024-03-09T15:33:48.564964+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:36:00.626046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":97,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"261'144","reported_seq":675,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191372+0000","last_change":"2024-03-09T14:28:34.321426+0000","last_active":"2024-03-09T15:33:36.191372+0000","last_peered":"2024-03-09T15:33:36.191372+0000","last_clean":"2024-03-09T15:33:36.191372+0000","last_became_active":"2024-03-09T14:28:34.319891+0000","last_became_peered":"2024-03-09T14:28:34.319891+0000","last_unstale":"2024-03-09T15:33:36.191372+0000","last_undegraded":"2024-03-09T15:33:36.191372+0000","last_fullsized":"2024-03-09T15:33:36.191372+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:25:46.129222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":571,"num_read_kb":433,"num_write":263,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"261'145","reported_seq":791,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.201525+0000","last_change":"2024-03-09T14:28:32.323880+0000","last_active":"2024-03-09T15:36:06.201525+0000","last_peered":"2024-03-09T15:36:06.201525+0000","last_clean":"2024-03-09T15:36:06.201525+0000","last_became_active":"2024-03-09T14:28:32.323198+0000","last_became_peered":"2024-03-09T14:28:32.323198+0000","last_unstale":"2024-03-09T15:36:06.201525+0000","last_undegraded":"2024-03-09T15:36:06.201525+0000","last_fullsized":"2024-03-09T15:36:06.201525+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:10:52.793653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4801712,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":390,"num_read_kb":2095,"num_write":636,"num_write_kb":25158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205346+0000","last_change":"2024-03-09T14:28:36.329314+0000","last_active":"2024-03-09T15:33:36.205346+0000","last_peered":"2024-03-09T15:33:36.205346+0000","last_clean":"2024-03-09T15:33:36.205346+0000","last_became_active":"2024-03-09T14:28:36.328529+0000","last_became_peered":"2024-03-09T14:28:36.328529+0000","last_unstale":"2024-03-09T15:33:36.205346+0000","last_undegraded":"2024-03-09T15:33:36.205346+0000","last_fullsized":"2024-03-09T15:33:36.205346+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:42:17.375037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"261'224","reported_seq":826,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.002456+0000","last_change":"2024-03-09T14:28:38.335758+0000","last_active":"2024-03-09T15:36:06.002456+0000","last_peered":"2024-03-09T15:36:06.002456+0000","last_clean":"2024-03-09T15:36:06.002456+0000","last_became_active":"2024-03-09T14:28:38.334991+0000","last_became_peered":"2024-03-09T14:28:38.334991+0000","last_unstale":"2024-03-09T15:36:06.002456+0000","last_undegraded":"2024-03-09T15:36:06.002456+0000","last_fullsized":"2024-03-09T15:36:06.002456+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":200,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:19:26.980708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13314,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":373,"num_read_kb":687,"num_write":264,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:30.983178+0000","last_change":"2024-03-09T14:28:40.368865+0000","last_active":"2024-03-09T15:31:30.983178+0000","last_peered":"2024-03-09T15:31:30.983178+0000","last_clean":"2024-03-09T15:31:30.983178+0000","last_became_active":"2024-03-09T14:28:40.368039+0000","last_became_peered":"2024-03-09T14:28:40.368039+0000","last_unstale":"2024-03-09T15:31:30.983178+0000","last_undegraded":"2024-03-09T15:31:30.983178+0000","last_fullsized":"2024-03-09T15:31:30.983178+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:50:42.121684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"261'77","reported_seq":597,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:46.896952+0000","last_change":"2024-03-09T14:28:42.414457+0000","last_active":"2024-03-09T15:34:46.896952+0000","last_peered":"2024-03-09T15:34:46.896952+0000","last_clean":"2024-03-09T15:34:46.896952+0000","last_became_active":"2024-03-09T14:28:42.413840+0000","last_became_peered":"2024-03-09T14:28:42.413840+0000","last_unstale":"2024-03-09T15:34:46.896952+0000","last_undegraded":"2024-03-09T15:34:46.896952+0000","last_fullsized":"2024-03-09T15:34:46.896952+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:42:30.742160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":129,"num_write":187,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"261'36","reported_seq":573,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:52.977046+0000","last_change":"2024-03-09T14:28:34.318737+0000","last_active":"2024-03-09T15:35:52.977046+0000","last_peered":"2024-03-09T15:35:52.977046+0000","last_clean":"2024-03-09T15:35:52.977046+0000","last_became_active":"2024-03-09T14:28:34.314194+0000","last_became_peered":"2024-03-09T14:28:34.314194+0000","last_unstale":"2024-03-09T15:35:52.977046+0000","last_undegraded":"2024-03-09T15:35:52.977046+0000","last_fullsized":"2024-03-09T15:35:52.977046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:45:29.912431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":120,"num_write":54,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"261'128","reported_seq":781,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.160748+0000","last_change":"2024-03-09T14:28:32.309431+0000","last_active":"2024-03-09T15:36:06.160748+0000","last_peered":"2024-03-09T15:36:06.160748+0000","last_clean":"2024-03-09T15:36:06.160748+0000","last_became_active":"2024-03-09T14:28:32.307702+0000","last_became_peered":"2024-03-09T14:28:32.307702+0000","last_unstale":"2024-03-09T15:36:06.160748+0000","last_undegraded":"2024-03-09T15:36:06.160748+0000","last_fullsized":"2024-03-09T15:36:06.160748+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:51:59.883119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":254,"num_read_kb":1758,"num_write":464,"num_write_kb":23078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194464+0000","last_change":"2024-03-09T14:28:36.305006+0000","last_active":"2024-03-09T15:33:36.194464+0000","last_peered":"2024-03-09T15:33:36.194464+0000","last_clean":"2024-03-09T15:33:36.194464+0000","last_became_active":"2024-03-09T14:28:36.304241+0000","last_became_peered":"2024-03-09T14:28:36.304241+0000","last_unstale":"2024-03-09T15:33:36.194464+0000","last_undegraded":"2024-03-09T15:33:36.194464+0000","last_fullsized":"2024-03-09T15:33:36.194464+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:14:09.284541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"261'178","reported_seq":757,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.035481+0000","last_change":"2024-03-09T14:28:38.340000+0000","last_active":"2024-03-09T15:36:06.035481+0000","last_peered":"2024-03-09T15:36:06.035481+0000","last_clean":"2024-03-09T15:36:06.035481+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:36:06.035481+0000","last_undegraded":"2024-03-09T15:36:06.035481+0000","last_fullsized":"2024-03-09T15:36:06.035481+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":100,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:06:57.175032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":521,"num_write":190,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.768268+0000","last_change":"2024-03-09T14:28:40.375515+0000","last_active":"2024-03-09T15:31:50.768268+0000","last_peered":"2024-03-09T15:31:50.768268+0000","last_clean":"2024-03-09T15:31:50.768268+0000","last_became_active":"2024-03-09T14:28:40.371820+0000","last_became_peered":"2024-03-09T14:28:40.371820+0000","last_unstale":"2024-03-09T15:31:50.768268+0000","last_undegraded":"2024-03-09T15:31:50.768268+0000","last_fullsized":"2024-03-09T15:31:50.768268+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:15:53.172763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.16","version":"261'53","reported_seq":567,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.020972+0000","last_change":"2024-03-09T14:28:42.392999+0000","last_active":"2024-03-09T15:33:10.020972+0000","last_peered":"2024-03-09T15:33:10.020972+0000","last_clean":"2024-03-09T15:33:10.020972+0000","last_became_active":"2024-03-09T14:28:42.392326+0000","last_became_peered":"2024-03-09T14:28:42.392326+0000","last_unstale":"2024-03-09T15:33:10.020972+0000","last_undegraded":"2024-03-09T15:33:10.020972+0000","last_fullsized":"2024-03-09T15:33:10.020972+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:36.796428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":772,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":99,"num_write":117,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"261'70","reported_seq":632,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.777840+0000","last_change":"2024-03-09T14:28:34.324293+0000","last_active":"2024-03-09T15:31:40.777840+0000","last_peered":"2024-03-09T15:31:40.777840+0000","last_clean":"2024-03-09T15:31:40.777840+0000","last_became_active":"2024-03-09T14:28:34.323969+0000","last_became_peered":"2024-03-09T14:28:34.323969+0000","last_unstale":"2024-03-09T15:31:40.777840+0000","last_undegraded":"2024-03-09T15:31:40.777840+0000","last_fullsized":"2024-03-09T15:31:40.777840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:33:30.511134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":256,"num_write":98,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"261'149","reported_seq":740,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:50.793396+0000","last_change":"2024-03-09T14:28:32.338014+0000","last_active":"2024-03-09T15:32:50.793396+0000","last_peered":"2024-03-09T15:32:50.793396+0000","last_clean":"2024-03-09T15:32:50.793396+0000","last_became_active":"2024-03-09T14:28:32.337755+0000","last_became_peered":"2024-03-09T14:28:32.337755+0000","last_unstale":"2024-03-09T15:32:50.793396+0000","last_undegraded":"2024-03-09T15:32:50.793396+0000","last_fullsized":"2024-03-09T15:32:50.793396+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:19:21.576018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663899,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":272,"num_read_kb":3602,"num_write":640,"num_write_kb":11566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162410+0000","last_change":"2024-03-09T14:28:36.318326+0000","last_active":"2024-03-09T15:31:26.162410+0000","last_peered":"2024-03-09T15:31:26.162410+0000","last_clean":"2024-03-09T15:31:26.162410+0000","last_became_active":"2024-03-09T14:28:36.317569+0000","last_became_peered":"2024-03-09T14:28:36.317569+0000","last_unstale":"2024-03-09T15:31:26.162410+0000","last_undegraded":"2024-03-09T15:31:26.162410+0000","last_fullsized":"2024-03-09T15:31:26.162410+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:21:30.385778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"261'149","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.708300+0000","last_change":"2024-03-09T14:28:38.348095+0000","last_active":"2024-03-09T15:35:58.708300+0000","last_peered":"2024-03-09T15:35:58.708300+0000","last_clean":"2024-03-09T15:35:58.708300+0000","last_became_active":"2024-03-09T14:28:38.347370+0000","last_became_peered":"2024-03-09T14:28:38.347370+0000","last_unstale":"2024-03-09T15:35:58.708300+0000","last_undegraded":"2024-03-09T15:35:58.708300+0000","last_fullsized":"2024-03-09T15:35:58.708300+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:00:09.043640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32290,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":236,"num_read_kb":490,"num_write":168,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":495,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:11.159141+0000","last_change":"2024-03-09T14:28:40.377364+0000","last_active":"2024-03-09T15:31:11.159141+0000","last_peered":"2024-03-09T15:31:11.159141+0000","last_clean":"2024-03-09T15:31:11.159141+0000","last_became_active":"2024-03-09T14:28:40.376513+0000","last_became_peered":"2024-03-09T14:28:40.376513+0000","last_unstale":"2024-03-09T15:31:11.159141+0000","last_undegraded":"2024-03-09T15:31:11.159141+0000","last_fullsized":"2024-03-09T15:31:11.159141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:28.590369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"261'92","reported_seq":974,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:54.162220+0000","last_change":"2024-03-09T14:28:42.410566+0000","last_active":"2024-03-09T15:35:54.162220+0000","last_peered":"2024-03-09T15:35:54.162220+0000","last_clean":"2024-03-09T15:35:54.162220+0000","last_became_active":"2024-03-09T14:28:42.409717+0000","last_became_peered":"2024-03-09T14:28:42.409717+0000","last_unstale":"2024-03-09T15:35:54.162220+0000","last_undegraded":"2024-03-09T15:35:54.162220+0000","last_fullsized":"2024-03-09T15:35:54.162220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:07:09.353068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":500,"num_read_kb":493,"num_write":215,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"261'53","reported_seq":585,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162448+0000","last_change":"2024-03-09T14:28:34.314180+0000","last_active":"2024-03-09T15:31:26.162448+0000","last_peered":"2024-03-09T15:31:26.162448+0000","last_clean":"2024-03-09T15:31:26.162448+0000","last_became_active":"2024-03-09T14:28:34.313380+0000","last_became_peered":"2024-03-09T14:28:34.313380+0000","last_unstale":"2024-03-09T15:31:26.162448+0000","last_undegraded":"2024-03-09T15:31:26.162448+0000","last_fullsized":"2024-03-09T15:31:26.162448+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:29:27.979809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":197,"num_read_kb":156,"num_write":74,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"261'146","reported_seq":733,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.965677+0000","last_change":"2024-03-09T14:28:32.293553+0000","last_active":"2024-03-09T15:36:05.965677+0000","last_peered":"2024-03-09T15:36:05.965677+0000","last_clean":"2024-03-09T15:36:05.965677+0000","last_became_active":"2024-03-09T14:28:32.292862+0000","last_became_peered":"2024-03-09T14:28:32.292862+0000","last_unstale":"2024-03-09T15:36:05.965677+0000","last_undegraded":"2024-03-09T15:36:05.965677+0000","last_fullsized":"2024-03-09T15:36:05.965677+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:21:35.805317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2792030,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":249,"num_read_kb":35693,"num_write":653,"num_write_kb":24072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"25'1","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.132760+0000","last_change":"2024-03-09T14:28:36.314323+0000","last_active":"2024-03-09T15:31:31.132760+0000","last_peered":"2024-03-09T15:31:31.132760+0000","last_clean":"2024-03-09T15:31:31.132760+0000","last_became_active":"2024-03-09T14:28:36.313405+0000","last_became_peered":"2024-03-09T14:28:36.313405+0000","last_unstale":"2024-03-09T15:31:31.132760+0000","last_undegraded":"2024-03-09T15:31:31.132760+0000","last_fullsized":"2024-03-09T15:31:31.132760+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:46:02.220171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"261'207","reported_seq":799,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.716100+0000","last_change":"2024-03-09T14:28:38.333268+0000","last_active":"2024-03-09T15:35:58.716100+0000","last_peered":"2024-03-09T15:35:58.716100+0000","last_clean":"2024-03-09T15:35:58.716100+0000","last_became_active":"2024-03-09T14:28:38.330571+0000","last_became_peered":"2024-03-09T14:28:38.330571+0000","last_unstale":"2024-03-09T15:35:58.716100+0000","last_undegraded":"2024-03-09T15:35:58.716100+0000","last_fullsized":"2024-03-09T15:35:58.716100+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":100,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:13:06.110510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27588,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":340,"num_read_kb":612,"num_write":242,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":491,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:10.978895+0000","last_change":"2024-03-09T14:28:40.385742+0000","last_active":"2024-03-09T15:31:10.978895+0000","last_peered":"2024-03-09T15:31:10.978895+0000","last_clean":"2024-03-09T15:31:10.978895+0000","last_became_active":"2024-03-09T14:28:40.385080+0000","last_became_peered":"2024-03-09T14:28:40.385080+0000","last_unstale":"2024-03-09T15:31:10.978895+0000","last_undegraded":"2024-03-09T15:31:10.978895+0000","last_fullsized":"2024-03-09T15:31:10.978895+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:08:01.706715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"261'69","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:11.304332+0000","last_change":"2024-03-09T14:28:42.413260+0000","last_active":"2024-03-09T15:34:11.304332+0000","last_peered":"2024-03-09T15:34:11.304332+0000","last_clean":"2024-03-09T15:34:11.304332+0000","last_became_active":"2024-03-09T14:28:42.412616+0000","last_became_peered":"2024-03-09T14:28:42.412616+0000","last_unstale":"2024-03-09T15:34:11.304332+0000","last_undegraded":"2024-03-09T15:34:11.304332+0000","last_fullsized":"2024-03-09T15:34:11.304332+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:28:05.834021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":49,"num_write":163,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.27","version":"261'50","reported_seq":597,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963528+0000","last_change":"2024-03-09T14:28:34.307137+0000","last_active":"2024-03-09T15:33:35.963528+0000","last_peered":"2024-03-09T15:33:35.963528+0000","last_clean":"2024-03-09T15:33:35.963528+0000","last_became_active":"2024-03-09T14:28:34.306685+0000","last_became_peered":"2024-03-09T14:28:34.306685+0000","last_unstale":"2024-03-09T15:33:35.963528+0000","last_undegraded":"2024-03-09T15:33:35.963528+0000","last_fullsized":"2024-03-09T15:33:35.963528+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:19:14.669885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":204,"num_read_kb":164,"num_write":70,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"261'122","reported_seq":705,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:13.078377+0000","last_change":"2024-03-09T14:28:32.334098+0000","last_active":"2024-03-09T15:33:13.078377+0000","last_peered":"2024-03-09T15:33:13.078377+0000","last_clean":"2024-03-09T15:33:13.078377+0000","last_became_active":"2024-03-09T14:28:32.333008+0000","last_became_peered":"2024-03-09T14:28:32.333008+0000","last_unstale":"2024-03-09T15:33:13.078377+0000","last_undegraded":"2024-03-09T15:33:13.078377+0000","last_fullsized":"2024-03-09T15:33:13.078377+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:19:21.576018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1943763,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":219,"num_read_kb":10390,"num_write":536,"num_write_kb":33266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.26","version":"261'181","reported_seq":772,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963471+0000","last_change":"2024-03-09T14:28:34.314634+0000","last_active":"2024-03-09T15:33:35.963471+0000","last_peered":"2024-03-09T15:33:35.963471+0000","last_clean":"2024-03-09T15:33:35.963471+0000","last_became_active":"2024-03-09T14:28:34.314007+0000","last_became_peered":"2024-03-09T14:28:34.314007+0000","last_unstale":"2024-03-09T15:33:35.963471+0000","last_undegraded":"2024-03-09T15:33:35.963471+0000","last_fullsized":"2024-03-09T15:33:35.963471+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":100,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:50:26.398988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":729,"num_read_kb":586,"num_write":272,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"261'127","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.956327+0000","last_change":"2024-03-09T14:28:32.287770+0000","last_active":"2024-03-09T15:36:05.956327+0000","last_peered":"2024-03-09T15:36:05.956327+0000","last_clean":"2024-03-09T15:36:05.956327+0000","last_became_active":"2024-03-09T14:28:32.286609+0000","last_became_peered":"2024-03-09T14:28:32.286609+0000","last_unstale":"2024-03-09T15:36:05.956327+0000","last_undegraded":"2024-03-09T15:36:05.956327+0000","last_fullsized":"2024-03-09T15:36:05.956327+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":100,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:14.582228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":810412,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":129,"num_read_kb":1907,"num_write":474,"num_write_kb":32192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"261'93","reported_seq":657,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194646+0000","last_change":"2024-03-09T14:28:34.325388+0000","last_active":"2024-03-09T15:33:36.194646+0000","last_peered":"2024-03-09T15:33:36.194646+0000","last_clean":"2024-03-09T15:33:36.194646+0000","last_became_active":"2024-03-09T14:28:34.323541+0000","last_became_peered":"2024-03-09T14:28:34.323541+0000","last_unstale":"2024-03-09T15:33:36.194646+0000","last_undegraded":"2024-03-09T15:33:36.194646+0000","last_fullsized":"2024-03-09T15:33:36.194646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:35:47.676249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":384,"num_read_kb":306,"num_write":146,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"261'160","reported_seq":756,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.074470+0000","last_change":"2024-03-09T14:28:32.299059+0000","last_active":"2024-03-09T15:36:06.074470+0000","last_peered":"2024-03-09T15:36:06.074470+0000","last_clean":"2024-03-09T15:36:06.074470+0000","last_became_active":"2024-03-09T14:28:32.295699+0000","last_became_peered":"2024-03-09T14:28:32.295699+0000","last_unstale":"2024-03-09T15:36:06.074470+0000","last_undegraded":"2024-03-09T15:36:06.074470+0000","last_fullsized":"2024-03-09T15:36:06.074470+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":100,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:14:50.303305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":286,"num_read_kb":14189,"num_write":705,"num_write_kb":33521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"261'101","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963269+0000","last_change":"2024-03-09T14:28:34.322186+0000","last_active":"2024-03-09T15:33:35.963269+0000","last_peered":"2024-03-09T15:33:35.963269+0000","last_clean":"2024-03-09T15:33:35.963269+0000","last_became_active":"2024-03-09T14:28:34.321647+0000","last_became_peered":"2024-03-09T14:28:34.321647+0000","last_unstale":"2024-03-09T15:33:35.963269+0000","last_undegraded":"2024-03-09T15:33:35.963269+0000","last_fullsized":"2024-03-09T15:33:35.963269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:19:44.897314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":329,"num_write":161,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"261'119","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.887640+0000","last_change":"2024-03-09T14:28:32.298327+0000","last_active":"2024-03-09T15:36:05.887640+0000","last_peered":"2024-03-09T15:36:05.887640+0000","last_clean":"2024-03-09T15:36:05.887640+0000","last_became_active":"2024-03-09T14:28:32.297866+0000","last_became_peered":"2024-03-09T14:28:32.297866+0000","last_unstale":"2024-03-09T15:36:05.887640+0000","last_undegraded":"2024-03-09T15:36:05.887640+0000","last_fullsized":"2024-03-09T15:36:05.887640+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":100,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:49:38.048255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":154,"num_read_kb":3467,"num_write":486,"num_write_kb":23815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"261'52","reported_seq":601,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.801400+0000","last_change":"2024-03-09T14:28:34.330916+0000","last_active":"2024-03-09T15:33:25.801400+0000","last_peered":"2024-03-09T15:33:25.801400+0000","last_clean":"2024-03-09T15:33:25.801400+0000","last_became_active":"2024-03-09T14:28:34.330511+0000","last_became_peered":"2024-03-09T14:28:34.330511+0000","last_unstale":"2024-03-09T15:33:25.801400+0000","last_undegraded":"2024-03-09T15:33:25.801400+0000","last_fullsized":"2024-03-09T15:33:25.801400+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:29:16.007058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":225,"num_read_kb":185,"num_write":82,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"261'102","reported_seq":651,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:52.427722+0000","last_change":"2024-03-09T14:28:32.326489+0000","last_active":"2024-03-09T15:33:52.427722+0000","last_peered":"2024-03-09T15:33:52.427722+0000","last_clean":"2024-03-09T15:33:52.427722+0000","last_became_active":"2024-03-09T14:28:32.326148+0000","last_became_peered":"2024-03-09T14:28:32.326148+0000","last_unstale":"2024-03-09T15:33:52.427722+0000","last_undegraded":"2024-03-09T15:33:52.427722+0000","last_fullsized":"2024-03-09T15:33:52.427722+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:59:13.121365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":157,"num_read_kb":5614,"num_write":435,"num_write_kb":21169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"261'134","reported_seq":685,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.185748+0000","last_change":"2024-03-09T14:28:34.323012+0000","last_active":"2024-03-09T15:33:11.185748+0000","last_peered":"2024-03-09T15:33:11.185748+0000","last_clean":"2024-03-09T15:33:11.185748+0000","last_became_active":"2024-03-09T14:28:34.322225+0000","last_became_peered":"2024-03-09T14:28:34.322225+0000","last_unstale":"2024-03-09T15:33:11.185748+0000","last_undegraded":"2024-03-09T15:33:11.185748+0000","last_fullsized":"2024-03-09T15:33:11.185748+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:33:31.665609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":498,"num_read_kb":399,"num_write":194,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"261'145","reported_seq":711,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:45.990374+0000","last_change":"2024-03-09T14:28:32.300246+0000","last_active":"2024-03-09T15:35:45.990374+0000","last_peered":"2024-03-09T15:35:45.990374+0000","last_clean":"2024-03-09T15:35:45.990374+0000","last_became_active":"2024-03-09T14:28:32.299839+0000","last_became_peered":"2024-03-09T14:28:32.299839+0000","last_unstale":"2024-03-09T15:35:45.990374+0000","last_undegraded":"2024-03-09T15:35:45.990374+0000","last_fullsized":"2024-03-09T15:35:45.990374+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:05:01.790489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243228,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":189,"num_read_kb":18833,"num_write":580,"num_write_kb":39169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"261'74","reported_seq":608,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:17.871898+0000","last_change":"2024-03-09T14:28:34.315670+0000","last_active":"2024-03-09T15:34:17.871898+0000","last_peered":"2024-03-09T15:34:17.871898+0000","last_clean":"2024-03-09T15:34:17.871898+0000","last_became_active":"2024-03-09T14:28:34.315302+0000","last_became_peered":"2024-03-09T14:28:34.315302+0000","last_unstale":"2024-03-09T15:34:17.871898+0000","last_undegraded":"2024-03-09T15:34:17.871898+0000","last_fullsized":"2024-03-09T15:34:17.871898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:54:03.674581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":224,"num_write":116,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"261'122","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.789837+0000","last_change":"2024-03-09T14:28:32.336827+0000","last_active":"2024-03-09T15:33:35.789837+0000","last_peered":"2024-03-09T15:33:35.789837+0000","last_clean":"2024-03-09T15:33:35.789837+0000","last_became_active":"2024-03-09T14:28:32.334117+0000","last_became_peered":"2024-03-09T14:28:32.334117+0000","last_unstale":"2024-03-09T15:33:35.789837+0000","last_undegraded":"2024-03-09T15:33:35.789837+0000","last_fullsized":"2024-03-09T15:33:35.789837+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:58:15.793689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243207,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":155,"num_read_kb":192,"num_write":536,"num_write_kb":14566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"261'48","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784320+0000","last_change":"2024-03-09T14:28:34.323974+0000","last_active":"2024-03-09T15:33:10.784320+0000","last_peered":"2024-03-09T15:33:10.784320+0000","last_clean":"2024-03-09T15:33:10.784320+0000","last_became_active":"2024-03-09T14:28:34.323202+0000","last_became_peered":"2024-03-09T14:28:34.323202+0000","last_unstale":"2024-03-09T15:33:10.784320+0000","last_undegraded":"2024-03-09T15:33:10.784320+0000","last_fullsized":"2024-03-09T15:33:10.784320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:33:12.688593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":219,"num_read_kb":183,"num_write":70,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"261'117","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.803914+0000","last_change":"2024-03-09T14:28:32.300258+0000","last_active":"2024-03-09T15:33:35.803914+0000","last_peered":"2024-03-09T15:33:35.803914+0000","last_clean":"2024-03-09T15:33:35.803914+0000","last_became_active":"2024-03-09T14:28:32.299737+0000","last_became_peered":"2024-03-09T14:28:32.299737+0000","last_unstale":"2024-03-09T15:33:35.803914+0000","last_undegraded":"2024-03-09T15:33:35.803914+0000","last_fullsized":"2024-03-09T15:33:35.803914+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:26:30.693287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6457661,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":147,"num_read_kb":12613,"num_write":500,"num_write_kb":31618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"261'94","reported_seq":644,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.984747+0000","last_change":"2024-03-09T14:28:34.315866+0000","last_active":"2024-03-09T15:31:35.984747+0000","last_peered":"2024-03-09T15:31:35.984747+0000","last_clean":"2024-03-09T15:31:35.984747+0000","last_became_active":"2024-03-09T14:28:34.315451+0000","last_became_peered":"2024-03-09T14:28:34.315451+0000","last_unstale":"2024-03-09T15:31:35.984747+0000","last_undegraded":"2024-03-09T15:31:35.984747+0000","last_fullsized":"2024-03-09T15:31:35.984747+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:50:21.781962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":374,"num_read_kb":293,"num_write":145,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"261'131","reported_seq":697,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.181679+0000","last_change":"2024-03-09T14:28:32.305393+0000","last_active":"2024-03-09T15:36:06.181679+0000","last_peered":"2024-03-09T15:36:06.181679+0000","last_clean":"2024-03-09T15:36:06.181679+0000","last_became_active":"2024-03-09T14:28:32.304540+0000","last_became_peered":"2024-03-09T14:28:32.304540+0000","last_unstale":"2024-03-09T15:36:06.181679+0000","last_undegraded":"2024-03-09T15:36:06.181679+0000","last_fullsized":"2024-03-09T15:36:06.181679+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":100,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:12:43.181100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1656073,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":187,"num_read_kb":1806,"num_write":496,"num_write_kb":18066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2e","version":"261'39","reported_seq":599,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.165250+0000","last_change":"2024-03-09T14:28:34.314876+0000","last_active":"2024-03-09T15:31:36.165250+0000","last_peered":"2024-03-09T15:31:36.165250+0000","last_clean":"2024-03-09T15:31:36.165250+0000","last_became_active":"2024-03-09T14:28:34.312399+0000","last_became_peered":"2024-03-09T14:28:34.312399+0000","last_unstale":"2024-03-09T15:31:36.165250+0000","last_undegraded":"2024-03-09T15:31:36.165250+0000","last_fullsized":"2024-03-09T15:31:36.165250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:27:43.779494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":182,"num_read_kb":156,"num_write":57,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"261'124","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.807282+0000","last_change":"2024-03-09T14:28:32.314908+0000","last_active":"2024-03-09T15:33:50.807282+0000","last_peered":"2024-03-09T15:33:50.807282+0000","last_clean":"2024-03-09T15:33:50.807282+0000","last_became_active":"2024-03-09T14:28:32.314596+0000","last_became_peered":"2024-03-09T14:28:32.314596+0000","last_unstale":"2024-03-09T15:33:50.807282+0000","last_undegraded":"2024-03-09T15:33:50.807282+0000","last_fullsized":"2024-03-09T15:33:50.807282+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:22:39.392181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506330,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":127,"num_read_kb":12793,"num_write":440,"num_write_kb":35350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"261'64","reported_seq":625,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:56.824631+0000","last_change":"2024-03-09T14:28:34.325871+0000","last_active":"2024-03-09T15:34:56.824631+0000","last_peered":"2024-03-09T15:34:56.824631+0000","last_clean":"2024-03-09T15:34:56.824631+0000","last_became_active":"2024-03-09T14:28:34.325461+0000","last_became_peered":"2024-03-09T14:28:34.325461+0000","last_unstale":"2024-03-09T15:34:56.824631+0000","last_undegraded":"2024-03-09T15:34:56.824631+0000","last_fullsized":"2024-03-09T15:34:56.824631+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:02:18.977660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":264,"num_read_kb":214,"num_write":89,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"261'141","reported_seq":717,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.191880+0000","last_change":"2024-03-09T14:28:32.313085+0000","last_active":"2024-03-09T15:36:06.191880+0000","last_peered":"2024-03-09T15:36:06.191880+0000","last_clean":"2024-03-09T15:36:06.191880+0000","last_became_active":"2024-03-09T14:28:32.311750+0000","last_became_peered":"2024-03-09T14:28:32.311750+0000","last_unstale":"2024-03-09T15:36:06.191880+0000","last_undegraded":"2024-03-09T15:36:06.191880+0000","last_fullsized":"2024-03-09T15:36:06.191880+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":100,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:56:43.188934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":195,"num_read_kb":14896,"num_write":515,"num_write_kb":41774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"261'162","reported_seq":774,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195666+0000","last_change":"2024-03-09T14:28:34.313344+0000","last_active":"2024-03-09T15:33:51.195666+0000","last_peered":"2024-03-09T15:33:51.195666+0000","last_clean":"2024-03-09T15:33:51.195666+0000","last_became_active":"2024-03-09T14:28:34.312212+0000","last_became_peered":"2024-03-09T14:28:34.312212+0000","last_unstale":"2024-03-09T15:33:51.195666+0000","last_undegraded":"2024-03-09T15:33:51.195666+0000","last_fullsized":"2024-03-09T15:33:51.195666+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":100,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:46:46.876109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":664,"num_read_kb":532,"num_write":243,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2b","version":"261'133","reported_seq":705,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.985790+0000","last_change":"2024-03-09T14:28:32.309913+0000","last_active":"2024-03-09T15:36:05.985790+0000","last_peered":"2024-03-09T15:36:05.985790+0000","last_clean":"2024-03-09T15:36:05.985790+0000","last_became_active":"2024-03-09T14:28:32.309562+0000","last_became_peered":"2024-03-09T14:28:32.309562+0000","last_unstale":"2024-03-09T15:36:05.985790+0000","last_undegraded":"2024-03-09T15:36:05.985790+0000","last_fullsized":"2024-03-09T15:36:05.985790+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:47:28.794070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":227,"num_read_kb":5348,"num_write":554,"num_write_kb":20887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"261'117","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.208966+0000","last_change":"2024-03-09T14:28:34.322800+0000","last_active":"2024-03-09T15:33:51.208966+0000","last_peered":"2024-03-09T15:33:51.208966+0000","last_clean":"2024-03-09T15:33:51.208966+0000","last_became_active":"2024-03-09T14:28:34.321424+0000","last_became_peered":"2024-03-09T14:28:34.321424+0000","last_unstale":"2024-03-09T15:33:51.208966+0000","last_undegraded":"2024-03-09T15:33:51.208966+0000","last_fullsized":"2024-03-09T15:33:51.208966+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:39:19.495854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":513,"num_read_kb":409,"num_write":169,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"261'142","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:09.453343+0000","last_change":"2024-03-09T14:28:32.334610+0000","last_active":"2024-03-09T15:34:09.453343+0000","last_peered":"2024-03-09T15:34:09.453343+0000","last_clean":"2024-03-09T15:34:09.453343+0000","last_became_active":"2024-03-09T14:28:32.333838+0000","last_became_peered":"2024-03-09T14:28:32.333838+0000","last_unstale":"2024-03-09T15:34:09.453343+0000","last_undegraded":"2024-03-09T15:34:09.453343+0000","last_fullsized":"2024-03-09T15:34:09.453343+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":100,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:14:50.303305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":218,"num_read_kb":3315,"num_write":653,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2a","version":"261'71","reported_seq":635,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.016279+0000","last_change":"2024-03-09T14:28:34.297744+0000","last_active":"2024-03-09T15:33:51.016279+0000","last_peered":"2024-03-09T15:33:51.016279+0000","last_clean":"2024-03-09T15:33:51.016279+0000","last_became_active":"2024-03-09T14:28:34.297042+0000","last_became_peered":"2024-03-09T14:28:34.297042+0000","last_unstale":"2024-03-09T15:33:51.016279+0000","last_undegraded":"2024-03-09T15:33:51.016279+0000","last_fullsized":"2024-03-09T15:33:51.016279+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:17:51.890278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":242,"num_write":99,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"261'111","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.167856+0000","last_change":"2024-03-09T14:28:32.320093+0000","last_active":"2024-03-09T15:36:06.167856+0000","last_peered":"2024-03-09T15:36:06.167856+0000","last_clean":"2024-03-09T15:36:06.167856+0000","last_became_active":"2024-03-09T14:28:32.317797+0000","last_became_peered":"2024-03-09T14:28:32.317797+0000","last_unstale":"2024-03-09T15:36:06.167856+0000","last_undegraded":"2024-03-09T15:36:06.167856+0000","last_fullsized":"2024-03-09T15:36:06.167856+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:52:33.454224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312437,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":176,"num_read_kb":4139,"num_write":464,"num_write_kb":19145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"261'113","reported_seq":668,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.807317+0000","last_change":"2024-03-09T14:28:34.314621+0000","last_active":"2024-03-09T15:33:50.807317+0000","last_peered":"2024-03-09T15:33:50.807317+0000","last_clean":"2024-03-09T15:33:50.807317+0000","last_became_active":"2024-03-09T14:28:34.314073+0000","last_became_peered":"2024-03-09T14:28:34.314073+0000","last_unstale":"2024-03-09T15:33:50.807317+0000","last_undegraded":"2024-03-09T15:33:50.807317+0000","last_fullsized":"2024-03-09T15:33:50.807317+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":100,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:30:53.101187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":445,"num_read_kb":351,"num_write":167,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"261'117","reported_seq":693,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.959588+0000","last_change":"2024-03-09T14:28:32.334618+0000","last_active":"2024-03-09T15:36:05.959588+0000","last_peered":"2024-03-09T15:36:05.959588+0000","last_clean":"2024-03-09T15:36:05.959588+0000","last_became_active":"2024-03-09T14:28:32.334187+0000","last_became_peered":"2024-03-09T14:28:32.334187+0000","last_unstale":"2024-03-09T15:36:05.959588+0000","last_undegraded":"2024-03-09T15:36:05.959588+0000","last_fullsized":"2024-03-09T15:36:05.959588+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:12:37.415618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2181393,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":207,"num_read_kb":1842,"num_write":544,"num_write_kb":7884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"261'4028","reported_seq":4579,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:46.207813+0000","last_change":"2024-03-09T14:28:34.324195+0000","last_active":"2024-03-09T15:33:46.207813+0000","last_peered":"2024-03-09T15:33:46.207813+0000","last_clean":"2024-03-09T15:33:46.207813+0000","last_became_active":"2024-03-09T14:28:34.322421+0000","last_became_peered":"2024-03-09T14:28:34.322421+0000","last_unstale":"2024-03-09T15:33:46.207813+0000","last_undegraded":"2024-03-09T15:33:46.207813+0000","last_fullsized":"2024-03-09T15:33:46.207813+0000","mapping_epoch":22,"log_start":"261'4000","ondisk_log_start":"261'4000","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:55:11.911331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14144,"num_read_kb":11461,"num_write":6038,"num_write_kb":3015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"261'118","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.139262+0000","last_change":"2024-03-09T14:28:32.324991+0000","last_active":"2024-03-09T15:36:06.139262+0000","last_peered":"2024-03-09T15:36:06.139262+0000","last_clean":"2024-03-09T15:36:06.139262+0000","last_became_active":"2024-03-09T14:28:32.324166+0000","last_became_peered":"2024-03-09T14:28:32.324166+0000","last_unstale":"2024-03-09T15:36:06.139262+0000","last_undegraded":"2024-03-09T15:36:06.139262+0000","last_fullsized":"2024-03-09T15:36:06.139262+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:55:14.982039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272031,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":186,"num_read_kb":9428,"num_write":503,"num_write_kb":30713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"261'126","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.159876+0000","last_change":"2024-03-09T14:28:32.288697+0000","last_active":"2024-03-09T15:36:06.159876+0000","last_peered":"2024-03-09T15:36:06.159876+0000","last_clean":"2024-03-09T15:36:06.159876+0000","last_became_active":"2024-03-09T14:28:32.287937+0000","last_became_peered":"2024-03-09T14:28:32.287937+0000","last_unstale":"2024-03-09T15:36:06.159876+0000","last_undegraded":"2024-03-09T15:36:06.159876+0000","last_fullsized":"2024-03-09T15:36:06.159876+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":100,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:01:00.707624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243238,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":199,"num_read_kb":48289,"num_write":444,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"261'161","reported_seq":726,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.985994+0000","last_change":"2024-03-09T14:28:34.317008+0000","last_active":"2024-03-09T15:31:40.985994+0000","last_peered":"2024-03-09T15:31:40.985994+0000","last_clean":"2024-03-09T15:31:40.985994+0000","last_became_active":"2024-03-09T14:28:34.316689+0000","last_became_peered":"2024-03-09T14:28:34.316689+0000","last_unstale":"2024-03-09T15:31:40.985994+0000","last_undegraded":"2024-03-09T15:31:40.985994+0000","last_fullsized":"2024-03-09T15:31:40.985994+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:22:45.346641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":634,"num_read_kb":490,"num_write":268,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"261'144","reported_seq":722,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.889143+0000","last_change":"2024-03-09T14:28:32.313446+0000","last_active":"2024-03-09T15:36:05.889143+0000","last_peered":"2024-03-09T15:36:05.889143+0000","last_clean":"2024-03-09T15:36:05.889143+0000","last_became_active":"2024-03-09T14:28:32.313132+0000","last_became_peered":"2024-03-09T14:28:32.313132+0000","last_unstale":"2024-03-09T15:36:05.889143+0000","last_undegraded":"2024-03-09T15:36:05.889143+0000","last_fullsized":"2024-03-09T15:36:05.889143+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:30:18.199314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262382,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":242,"num_read_kb":5401,"num_write":622,"num_write_kb":32648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"261'41","reported_seq":586,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:02.721943+0000","last_change":"2024-03-09T14:28:34.304440+0000","last_active":"2024-03-09T15:34:02.721943+0000","last_peered":"2024-03-09T15:34:02.721943+0000","last_clean":"2024-03-09T15:34:02.721943+0000","last_became_active":"2024-03-09T14:28:34.303701+0000","last_became_peered":"2024-03-09T14:28:34.303701+0000","last_unstale":"2024-03-09T15:34:02.721943+0000","last_undegraded":"2024-03-09T15:34:02.721943+0000","last_fullsized":"2024-03-09T15:34:02.721943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:22:10.704885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":166,"num_read_kb":138,"num_write":64,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":65,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3686,"num_read_kb":2614,"num_write":1236,"num_write_kb":514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":13176,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":8518,"num_read_kb":8294,"num_write":8580,"num_write_kb":2546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":26352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2011,"ondisk_log_size":2011,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":578,"ondisk_log_size":578,"up":64,"acting":64,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":725340,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":11980,"num_read_kb":20116,"num_write":8932,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":743659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2014,"ondisk_log_size":2014,"up":64,"acting":64,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1719,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1602,"num_read_kb":1903,"num_write":22,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":38590,"num_read_kb":31236,"num_write":15314,"num_write_kb":7493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3941,"ondisk_log_size":3941,"up":128,"acting":128,"num_store_stats":8},{"poolid":68,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":210202860,"num_objects":1447,"num_object_clones":0,"num_object_copies":2894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1447,"num_whiteouts":0,"num_read":12930,"num_read_kb":1121153,"num_write":33168,"num_write_kb":1701153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440426496,"data_stored":430130434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2921,"ondisk_log_size":2921,"up":128,"acting":128,"num_store_stats":8},{"poolid":67,"num_pg":32,"stat_sum":{"num_bytes":81879532,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":124,"num_read_kb":192739,"num_write":140,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":160620544,"data_stored":160613336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":66,"num_pg":32,"stat_sum":{"num_bytes":82877649,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":122,"num_read_kb":192649,"num_write":140,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":170004480,"data_stored":170000061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575670,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334148,"kb_used_data":104444,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":94037692,"statfs":{"total":96636764160,"available":96294596608,"internally_reserved":0,"allocated":106950656,"data_stored":103672380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":235207114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575666,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292416,"kb_used_data":84920,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":94079424,"statfs":{"total":96636764160,"available":96337330176,"internally_reserved":0,"allocated":86958080,"data_stored":84225759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575667,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":174444,"kb_used_data":121060,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94197396,"statfs":{"total":96636764160,"available":96458133504,"internally_reserved":0,"allocated":123965440,"data_stored":120640043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575671,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":197704,"kb_used_data":140160,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94174136,"statfs":{"total":96636764160,"available":96434315264,"internally_reserved":0,"allocated":143523840,"data_stored":140316730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575666,"num_pgs":118,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":343308,"kb_used_data":135812,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":94028532,"statfs":{"total":96636764160,"available":96285216768,"internally_reserved":0,"allocated":139071488,"data_stored":135606966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":212466119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575668,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133712,"kb_used_data":80328,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94238128,"statfs":{"total":96636764160,"available":96499843072,"internally_reserved":0,"allocated":82255872,"data_stored":79602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575667,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":186168,"kb_used_data":100912,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94185672,"statfs":{"total":96636764160,"available":96446128128,"internally_reserved":0,"allocated":103333888,"data_stored":100615986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575669,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339852,"kb_used_data":137924,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":94031988,"statfs":{"total":96636764160,"available":96288755712,"internally_reserved":0,"allocated":141234176,"data_stored":137929952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73302016,"data_stored":71838616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41046016,"data_stored":40051454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":44478464,"data_stored":43472543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":64929792,"data_stored":63014498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":77041664,"data_stored":75526368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61636608,"data_stored":60503814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":33976320,"data_stored":32726952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":44015616,"data_stored":42996189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":92713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":120489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":29353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":164295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":29815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":115473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":83356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":108165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":3901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":2727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24158208,"data_stored":24157393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25219072,"data_stored":25216283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13631488,"data_stored":13631488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30449664,"data_stored":30448849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36700160,"data_stored":36700160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":25165824,"data_stored":25165824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":14680064,"data_stored":14680064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20017152,"data_stored":20013548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24117248,"data_stored":24117248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19922944,"data_stored":19922944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39845888,"data_stored":39845888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5337088,"data_stored":5333484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":22020096,"data_stored":22020096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24117248,"data_stored":24117248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T15:36:07.994 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T15:36:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:36:08.413 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-09T15:36:08.436 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":2291,"stamp":"2024-03-09T15:36:07.248269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376290644,"num_objects":1788,"num_object_clones":0,"num_object_copies":3576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1788,"num_whiteouts":0,"num_read":77680,"num_read_kb":1570816,"num_write":67709,"num_write_kb":1988128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12459,"ondisk_log_size":12459,"up":786,"acting":786,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":786,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2001752,"kb_used_data":905560,"kb_used_omap":12,"kb_used_meta":1096115,"kb_avail":752972968,"statfs":{"total":773094113280,"available":771044319232,"internally_reserved":0,"allocated":927293440,"data_stored":902610116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1122422320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-66869968,"num_objects":-307,"num_object_clones":0,"num_object_copies":-614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-307,"num_whiteouts":0,"num_read":462,"num_read_kb":498,"num_write":368,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007256"},"pg_stats":[{"pgid":"68.1a","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.178131+0000","last_change":"2024-03-09T15:09:50.085348+0000","last_active":"2024-03-09T15:32:21.178131+0000","last_peered":"2024-03-09T15:32:21.178131+0000","last_clean":"2024-03-09T15:32:21.178131+0000","last_became_active":"2024-03-09T15:09:50.084784+0000","last_became_peered":"2024-03-09T15:09:50.084784+0000","last_unstale":"2024-03-09T15:32:21.178131+0000","last_undegraded":"2024-03-09T15:32:21.178131+0000","last_fullsized":"2024-03-09T15:32:21.178131+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:18:17.114617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.1d","version":"258'7","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.167423+0000","last_change":"2024-03-09T14:46:26.826294+0000","last_active":"2024-03-09T15:31:31.167423+0000","last_peered":"2024-03-09T15:31:31.167423+0000","last_clean":"2024-03-09T15:31:31.167423+0000","last_became_active":"2024-03-09T14:46:26.825404+0000","last_became_peered":"2024-03-09T14:46:26.825404+0000","last_unstale":"2024-03-09T15:31:31.167423+0000","last_undegraded":"2024-03-09T15:31:31.167423+0000","last_fullsized":"2024-03-09T15:31:31.167423+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:28:03.324789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20498,"num_write":6,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1c","version":"261'1","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:20.773210+0000","last_change":"2024-03-09T14:46:18.769137+0000","last_active":"2024-03-09T15:31:20.773210+0000","last_peered":"2024-03-09T15:31:20.773210+0000","last_clean":"2024-03-09T15:31:20.773210+0000","last_became_active":"2024-03-09T14:46:18.768383+0000","last_became_peered":"2024-03-09T14:46:18.768383+0000","last_unstale":"2024-03-09T15:31:20.773210+0000","last_undegraded":"2024-03-09T15:31:20.773210+0000","last_fullsized":"2024-03-09T15:31:20.773210+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:38:13.467012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":3072,"num_write":1,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.1f","version":"258'45","reported_seq":167,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.010472+0000","last_change":"2024-03-09T14:39:13.282130+0000","last_active":"2024-03-09T15:33:26.010472+0000","last_peered":"2024-03-09T15:33:26.010472+0000","last_clean":"2024-03-09T15:33:26.010472+0000","last_became_active":"2024-03-09T14:39:13.281020+0000","last_became_peered":"2024-03-09T14:39:13.281020+0000","last_unstale":"2024-03-09T15:33:26.010472+0000","last_undegraded":"2024-03-09T15:33:26.010472+0000","last_fullsized":"2024-03-09T15:33:26.010472+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:54.492385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":134,"num_write":70,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1b","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.961650+0000","last_change":"2024-03-09T15:09:50.083262+0000","last_active":"2024-03-09T15:32:20.961650+0000","last_peered":"2024-03-09T15:32:20.961650+0000","last_clean":"2024-03-09T15:32:20.961650+0000","last_became_active":"2024-03-09T15:09:50.082690+0000","last_became_peered":"2024-03-09T15:09:50.082690+0000","last_unstale":"2024-03-09T15:32:20.961650+0000","last_undegraded":"2024-03-09T15:32:20.961650+0000","last_fullsized":"2024-03-09T15:32:20.961650+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:18:09.424746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1c","version":"258'5","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162661+0000","last_change":"2024-03-09T14:46:26.807670+0000","last_active":"2024-03-09T15:31:26.162661+0000","last_peered":"2024-03-09T15:31:26.162661+0000","last_clean":"2024-03-09T15:31:26.162661+0000","last_became_active":"2024-03-09T14:46:26.806865+0000","last_became_peered":"2024-03-09T14:46:26.806865+0000","last_unstale":"2024-03-09T15:31:26.162661+0000","last_undegraded":"2024-03-09T15:31:26.162661+0000","last_fullsized":"2024-03-09T15:31:26.162661+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:05:04.558231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.1d","version":"261'1","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:21.130434+0000","last_change":"2024-03-09T14:46:18.780095+0000","last_active":"2024-03-09T15:31:21.130434+0000","last_peered":"2024-03-09T15:31:21.130434+0000","last_clean":"2024-03-09T15:31:21.130434+0000","last_became_active":"2024-03-09T14:46:18.779760+0000","last_became_peered":"2024-03-09T14:46:18.779760+0000","last_unstale":"2024-03-09T15:31:21.130434+0000","last_undegraded":"2024-03-09T15:31:21.130434+0000","last_fullsized":"2024-03-09T15:31:21.130434+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:12.211222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":1,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1e","version":"0'0","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:30.975652+0000","last_change":"2024-03-09T14:39:13.260541+0000","last_active":"2024-03-09T15:33:30.975652+0000","last_peered":"2024-03-09T15:33:30.975652+0000","last_clean":"2024-03-09T15:33:30.975652+0000","last_became_active":"2024-03-09T14:39:13.259731+0000","last_became_peered":"2024-03-09T14:39:13.259731+0000","last_unstale":"2024-03-09T15:33:30.975652+0000","last_undegraded":"2024-03-09T15:33:30.975652+0000","last_fullsized":"2024-03-09T15:33:30.975652+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:09:28.555448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.18","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.764910+0000","last_change":"2024-03-09T15:09:50.081224+0000","last_active":"2024-03-09T15:31:35.764910+0000","last_peered":"2024-03-09T15:31:35.764910+0000","last_clean":"2024-03-09T15:31:35.764910+0000","last_became_active":"2024-03-09T15:09:50.080578+0000","last_became_peered":"2024-03-09T15:09:50.080578+0000","last_unstale":"2024-03-09T15:31:35.764910+0000","last_undegraded":"2024-03-09T15:31:35.764910+0000","last_fullsized":"2024-03-09T15:31:35.764910+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:54:04.030480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.1f","version":"261'4","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.995705+0000","last_change":"2024-03-09T14:46:26.896619+0000","last_active":"2024-03-09T15:32:20.995705+0000","last_peered":"2024-03-09T15:32:20.995705+0000","last_clean":"2024-03-09T15:32:20.995705+0000","last_became_active":"2024-03-09T14:46:26.895608+0000","last_became_peered":"2024-03-09T14:46:26.895608+0000","last_unstale":"2024-03-09T15:32:20.995705+0000","last_undegraded":"2024-03-09T15:32:20.995705+0000","last_fullsized":"2024-03-09T15:32:20.995705+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:32:06.502138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":12288,"num_write":4,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1e","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:35.964975+0000","last_change":"2024-03-09T14:46:18.782402+0000","last_active":"2024-03-09T15:32:35.964975+0000","last_peered":"2024-03-09T15:32:35.964975+0000","last_clean":"2024-03-09T15:32:35.964975+0000","last_became_active":"2024-03-09T14:46:18.781856+0000","last_became_peered":"2024-03-09T14:46:18.781856+0000","last_unstale":"2024-03-09T15:32:35.964975+0000","last_undegraded":"2024-03-09T15:32:35.964975+0000","last_fullsized":"2024-03-09T15:32:35.964975+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:29:15.829032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.1d","version":"258'35","reported_seq":142,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.987080+0000","last_change":"2024-03-09T14:39:13.265101+0000","last_active":"2024-03-09T15:31:45.987080+0000","last_peered":"2024-03-09T15:31:45.987080+0000","last_clean":"2024-03-09T15:31:45.987080+0000","last_became_active":"2024-03-09T14:39:13.258351+0000","last_became_peered":"2024-03-09T14:39:13.258351+0000","last_unstale":"2024-03-09T15:31:45.987080+0000","last_undegraded":"2024-03-09T15:31:45.987080+0000","last_fullsized":"2024-03-09T15:31:45.987080+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:21:02.690940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":152,"num_read_kb":109,"num_write":57,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.19","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.777876+0000","last_change":"2024-03-09T15:09:50.083658+0000","last_active":"2024-03-09T15:31:40.777876+0000","last_peered":"2024-03-09T15:31:40.777876+0000","last_clean":"2024-03-09T15:31:40.777876+0000","last_became_active":"2024-03-09T15:09:50.081680+0000","last_became_peered":"2024-03-09T15:09:50.081680+0000","last_unstale":"2024-03-09T15:31:40.777876+0000","last_undegraded":"2024-03-09T15:31:40.777876+0000","last_fullsized":"2024-03-09T15:31:40.777876+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:56:53.220664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.1e","version":"261'4","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:21.009093+0000","last_change":"2024-03-09T14:47:07.798858+0000","last_active":"2024-03-09T15:33:21.009093+0000","last_peered":"2024-03-09T15:33:21.009093+0000","last_clean":"2024-03-09T15:33:21.009093+0000","last_became_active":"2024-03-09T14:47:07.798247+0000","last_became_peered":"2024-03-09T14:47:07.798247+0000","last_unstale":"2024-03-09T15:33:21.009093+0000","last_undegraded":"2024-03-09T15:33:21.009093+0000","last_fullsized":"2024-03-09T15:33:21.009093+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:45:56.993617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":16384,"num_write":9,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1f","version":"258'5","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.173075+0000","last_change":"2024-03-09T14:46:18.772485+0000","last_active":"2024-03-09T15:32:11.173075+0000","last_peered":"2024-03-09T15:32:11.173075+0000","last_clean":"2024-03-09T15:32:11.173075+0000","last_became_active":"2024-03-09T14:46:18.771731+0000","last_became_peered":"2024-03-09T14:46:18.771731+0000","last_unstale":"2024-03-09T15:32:11.173075+0000","last_undegraded":"2024-03-09T15:32:11.173075+0000","last_fullsized":"2024-03-09T15:32:11.173075+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:50:44.589948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5136,"num_write":5,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1c","version":"261'13","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.779361+0000","last_change":"2024-03-09T14:39:13.266571+0000","last_active":"2024-03-09T15:31:45.779361+0000","last_peered":"2024-03-09T15:31:45.779361+0000","last_clean":"2024-03-09T15:31:45.779361+0000","last_became_active":"2024-03-09T14:39:13.260945+0000","last_became_peered":"2024-03-09T14:39:13.260945+0000","last_unstale":"2024-03-09T15:31:45.779361+0000","last_undegraded":"2024-03-09T15:31:45.779361+0000","last_fullsized":"2024-03-09T15:31:45.779361+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:58:40.520455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":27,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1e","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.808730+0000","last_change":"2024-03-09T15:09:50.088230+0000","last_active":"2024-03-09T15:33:55.808730+0000","last_peered":"2024-03-09T15:33:55.808730+0000","last_clean":"2024-03-09T15:33:55.808730+0000","last_became_active":"2024-03-09T15:09:50.086789+0000","last_became_peered":"2024-03-09T15:09:50.086789+0000","last_unstale":"2024-03-09T15:33:55.808730+0000","last_undegraded":"2024-03-09T15:33:55.808730+0000","last_fullsized":"2024-03-09T15:33:55.808730+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:54:27.433652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.19","version":"261'6","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.109749+0000","last_change":"2024-03-09T14:46:26.819845+0000","last_active":"2024-03-09T15:36:06.109749+0000","last_peered":"2024-03-09T15:36:06.109749+0000","last_clean":"2024-03-09T15:36:06.109749+0000","last_became_active":"2024-03-09T14:46:26.819085+0000","last_became_peered":"2024-03-09T14:46:26.819085+0000","last_unstale":"2024-03-09T15:36:06.109749+0000","last_undegraded":"2024-03-09T15:36:06.109749+0000","last_fullsized":"2024-03-09T15:36:06.109749+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:17.867538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":12288,"num_write":11,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.18","version":"261'12","reported_seq":78,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:41.027571+0000","last_change":"2024-03-09T14:46:18.776960+0000","last_active":"2024-03-09T15:34:41.027571+0000","last_peered":"2024-03-09T15:34:41.027571+0000","last_clean":"2024-03-09T15:34:41.027571+0000","last_became_active":"2024-03-09T14:46:18.776161+0000","last_became_peered":"2024-03-09T14:46:18.776161+0000","last_unstale":"2024-03-09T15:34:41.027571+0000","last_undegraded":"2024-03-09T15:34:41.027571+0000","last_fullsized":"2024-03-09T15:34:41.027571+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:22:11.563005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":36889,"num_write":16,"num_write_kb":16407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.1b","version":"258'10","reported_seq":79,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.779245+0000","last_change":"2024-03-09T14:39:13.269298+0000","last_active":"2024-03-09T15:31:45.779245+0000","last_peered":"2024-03-09T15:31:45.779245+0000","last_clean":"2024-03-09T15:31:45.779245+0000","last_became_active":"2024-03-09T14:39:13.268179+0000","last_became_peered":"2024-03-09T14:39:13.268179+0000","last_unstale":"2024-03-09T15:31:45.779245+0000","last_undegraded":"2024-03-09T15:31:45.779245+0000","last_fullsized":"2024-03-09T15:31:45.779245+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:12:05.852927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1f","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:45.951375+0000","last_change":"2024-03-09T15:09:50.064176+0000","last_active":"2024-03-09T15:32:45.951375+0000","last_peered":"2024-03-09T15:32:45.951375+0000","last_clean":"2024-03-09T15:32:45.951375+0000","last_became_active":"2024-03-09T15:09:50.063705+0000","last_became_peered":"2024-03-09T15:09:50.063705+0000","last_unstale":"2024-03-09T15:32:45.951375+0000","last_undegraded":"2024-03-09T15:32:45.951375+0000","last_fullsized":"2024-03-09T15:32:45.951375+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:45.866476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.18","version":"261'2","reported_seq":46,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.775288+0000","last_change":"2024-03-09T14:46:26.824830+0000","last_active":"2024-03-09T15:32:25.775288+0000","last_peered":"2024-03-09T15:32:25.775288+0000","last_clean":"2024-03-09T15:32:25.775288+0000","last_became_active":"2024-03-09T14:46:26.824522+0000","last_became_peered":"2024-03-09T14:46:26.824522+0000","last_unstale":"2024-03-09T15:32:25.775288+0000","last_undegraded":"2024-03-09T15:32:25.775288+0000","last_fullsized":"2024-03-09T15:32:25.775288+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:09:09.414799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":2,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.19","version":"261'7","reported_seq":64,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.175135+0000","last_change":"2024-03-09T14:46:18.770577+0000","last_active":"2024-03-09T15:32:21.175135+0000","last_peered":"2024-03-09T15:32:21.175135+0000","last_clean":"2024-03-09T15:32:21.175135+0000","last_became_active":"2024-03-09T14:46:18.769730+0000","last_became_peered":"2024-03-09T14:46:18.769730+0000","last_unstale":"2024-03-09T15:32:21.175135+0000","last_undegraded":"2024-03-09T15:32:21.175135+0000","last_fullsized":"2024-03-09T15:32:21.175135+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:07:45.259457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340032,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":19456,"num_write":17,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1a","version":"261'44","reported_seq":164,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.767114+0000","last_change":"2024-03-09T14:39:13.274060+0000","last_active":"2024-03-09T15:31:45.767114+0000","last_peered":"2024-03-09T15:31:45.767114+0000","last_clean":"2024-03-09T15:31:45.767114+0000","last_became_active":"2024-03-09T14:39:13.273298+0000","last_became_peered":"2024-03-09T14:39:13.273298+0000","last_unstale":"2024-03-09T15:31:45.767114+0000","last_undegraded":"2024-03-09T15:31:45.767114+0000","last_fullsized":"2024-03-09T15:31:45.767114+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:16:28.025129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":138,"num_write":72,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.1c","version":"261'1","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188241+0000","last_change":"2024-03-09T15:09:50.075720+0000","last_active":"2024-03-09T15:32:21.188241+0000","last_peered":"2024-03-09T15:32:21.188241+0000","last_clean":"2024-03-09T15:32:21.188241+0000","last_became_active":"2024-03-09T15:09:50.074952+0000","last_became_peered":"2024-03-09T15:09:50.074952+0000","last_unstale":"2024-03-09T15:32:21.188241+0000","last_undegraded":"2024-03-09T15:32:21.188241+0000","last_fullsized":"2024-03-09T15:32:21.188241+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:21:35.439413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1b","version":"261'2","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:30.763859+0000","last_change":"2024-03-09T14:46:26.805382+0000","last_active":"2024-03-09T15:31:30.763859+0000","last_peered":"2024-03-09T15:31:30.763859+0000","last_clean":"2024-03-09T15:31:30.763859+0000","last_became_active":"2024-03-09T14:46:26.804622+0000","last_became_peered":"2024-03-09T14:46:26.804622+0000","last_unstale":"2024-03-09T15:31:30.763859+0000","last_undegraded":"2024-03-09T15:31:30.763859+0000","last_fullsized":"2024-03-09T15:31:30.763859+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:23.446296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":5120,"num_write":2,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.1a","version":"261'4","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:21.161277+0000","last_change":"2024-03-09T14:46:18.785531+0000","last_active":"2024-03-09T15:31:21.161277+0000","last_peered":"2024-03-09T15:31:21.161277+0000","last_clean":"2024-03-09T15:31:21.161277+0000","last_became_active":"2024-03-09T14:46:18.784821+0000","last_became_peered":"2024-03-09T14:46:18.784821+0000","last_unstale":"2024-03-09T15:31:21.161277+0000","last_undegraded":"2024-03-09T15:31:21.161277+0000","last_fullsized":"2024-03-09T15:31:21.161277+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:10:34.919135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4096,"num_write":9,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.19","version":"258'28","reported_seq":120,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:40.999888+0000","last_change":"2024-03-09T14:39:13.258289+0000","last_active":"2024-03-09T15:32:40.999888+0000","last_peered":"2024-03-09T15:32:40.999888+0000","last_clean":"2024-03-09T15:32:40.999888+0000","last_became_active":"2024-03-09T14:39:13.257473+0000","last_became_peered":"2024-03-09T14:39:13.257473+0000","last_unstale":"2024-03-09T15:32:40.999888+0000","last_undegraded":"2024-03-09T15:32:40.999888+0000","last_fullsized":"2024-03-09T15:32:40.999888+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:12:15.900385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":78,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1d","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188298+0000","last_change":"2024-03-09T15:09:50.075294+0000","last_active":"2024-03-09T15:32:21.188298+0000","last_peered":"2024-03-09T15:32:21.188298+0000","last_clean":"2024-03-09T15:32:21.188298+0000","last_became_active":"2024-03-09T15:09:50.074875+0000","last_became_peered":"2024-03-09T15:09:50.074875+0000","last_unstale":"2024-03-09T15:32:21.188298+0000","last_undegraded":"2024-03-09T15:32:21.188298+0000","last_fullsized":"2024-03-09T15:32:21.188298+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:24:33.386479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1a","version":"261'9","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:30.775394+0000","last_change":"2024-03-09T14:46:26.813798+0000","last_active":"2024-03-09T15:31:30.775394+0000","last_peered":"2024-03-09T15:31:30.775394+0000","last_clean":"2024-03-09T15:31:30.775394+0000","last_became_active":"2024-03-09T14:46:26.812889+0000","last_became_peered":"2024-03-09T14:46:26.812889+0000","last_unstale":"2024-03-09T15:31:30.775394+0000","last_undegraded":"2024-03-09T15:31:30.775394+0000","last_fullsized":"2024-03-09T15:31:30.775394+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:43:59.844987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":12290,"num_write":13,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.1b","version":"261'1","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:20.980983+0000","last_change":"2024-03-09T14:46:18.774605+0000","last_active":"2024-03-09T15:31:20.980983+0000","last_peered":"2024-03-09T15:31:20.980983+0000","last_clean":"2024-03-09T15:31:20.980983+0000","last_became_active":"2024-03-09T14:46:18.756394+0000","last_became_peered":"2024-03-09T14:46:18.756394+0000","last_unstale":"2024-03-09T15:31:20.980983+0000","last_undegraded":"2024-03-09T15:31:20.980983+0000","last_fullsized":"2024-03-09T15:31:20.980983+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:49.690847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40145,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":40,"num_write":1,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.18","version":"261'8","reported_seq":77,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.144778+0000","last_change":"2024-03-09T14:39:13.258846+0000","last_active":"2024-03-09T15:32:26.144778+0000","last_peered":"2024-03-09T15:32:26.144778+0000","last_clean":"2024-03-09T15:32:26.144778+0000","last_became_active":"2024-03-09T14:39:13.257996+0000","last_became_peered":"2024-03-09T14:39:13.257996+0000","last_unstale":"2024-03-09T15:32:26.144778+0000","last_undegraded":"2024-03-09T15:32:26.144778+0000","last_fullsized":"2024-03-09T15:32:26.144778+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:23:43.508747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.12","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.957209+0000","last_change":"2024-03-09T15:09:50.070950+0000","last_active":"2024-03-09T15:33:10.957209+0000","last_peered":"2024-03-09T15:33:10.957209+0000","last_clean":"2024-03-09T15:33:10.957209+0000","last_became_active":"2024-03-09T15:09:50.070345+0000","last_became_peered":"2024-03-09T15:09:50.070345+0000","last_unstale":"2024-03-09T15:33:10.957209+0000","last_undegraded":"2024-03-09T15:33:10.957209+0000","last_fullsized":"2024-03-09T15:33:10.957209+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:18:24.068533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.15","version":"0'0","reported_seq":42,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.788416+0000","last_change":"2024-03-09T14:46:26.891781+0000","last_active":"2024-03-09T15:32:25.788416+0000","last_peered":"2024-03-09T15:32:25.788416+0000","last_clean":"2024-03-09T15:32:25.788416+0000","last_became_active":"2024-03-09T14:46:26.890992+0000","last_became_peered":"2024-03-09T14:46:26.890992+0000","last_unstale":"2024-03-09T15:32:25.788416+0000","last_undegraded":"2024-03-09T15:32:25.788416+0000","last_fullsized":"2024-03-09T15:32:25.788416+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:56:14.439157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.14","version":"0'0","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:06.187943+0000","last_change":"2024-03-09T14:46:18.767760+0000","last_active":"2024-03-09T15:33:06.187943+0000","last_peered":"2024-03-09T15:33:06.187943+0000","last_clean":"2024-03-09T15:33:06.187943+0000","last_became_active":"2024-03-09T14:46:18.766777+0000","last_became_peered":"2024-03-09T14:46:18.766777+0000","last_unstale":"2024-03-09T15:33:06.187943+0000","last_undegraded":"2024-03-09T15:33:06.187943+0000","last_fullsized":"2024-03-09T15:33:06.187943+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:30:54.129379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.17","version":"258'19","reported_seq":106,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.996874+0000","last_change":"2024-03-09T14:39:13.281484+0000","last_active":"2024-03-09T15:32:25.996874+0000","last_peered":"2024-03-09T15:32:25.996874+0000","last_clean":"2024-03-09T15:32:25.996874+0000","last_became_active":"2024-03-09T14:39:13.279919+0000","last_became_peered":"2024-03-09T14:39:13.279919+0000","last_unstale":"2024-03-09T15:32:25.996874+0000","last_undegraded":"2024-03-09T15:32:25.996874+0000","last_fullsized":"2024-03-09T15:32:25.996874+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:09:49.670708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":64,"num_write":32,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.13","version":"261'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.971686+0000","last_change":"2024-03-09T15:09:50.079986+0000","last_active":"2024-03-09T15:33:10.971686+0000","last_peered":"2024-03-09T15:33:10.971686+0000","last_clean":"2024-03-09T15:33:10.971686+0000","last_became_active":"2024-03-09T15:09:50.079604+0000","last_became_peered":"2024-03-09T15:09:50.079604+0000","last_unstale":"2024-03-09T15:33:10.971686+0000","last_undegraded":"2024-03-09T15:33:10.971686+0000","last_fullsized":"2024-03-09T15:33:10.971686+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:30:05.194786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.14","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.800266+0000","last_change":"2024-03-09T14:46:26.894898+0000","last_active":"2024-03-09T15:33:20.800266+0000","last_peered":"2024-03-09T15:33:20.800266+0000","last_clean":"2024-03-09T15:33:20.800266+0000","last_became_active":"2024-03-09T14:46:26.894088+0000","last_became_peered":"2024-03-09T14:46:26.894088+0000","last_unstale":"2024-03-09T15:33:20.800266+0000","last_undegraded":"2024-03-09T15:33:20.800266+0000","last_fullsized":"2024-03-09T15:33:20.800266+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:10:41.693365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.15","version":"0'0","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:16.187056+0000","last_change":"2024-03-09T14:46:18.753568+0000","last_active":"2024-03-09T15:32:16.187056+0000","last_peered":"2024-03-09T15:32:16.187056+0000","last_clean":"2024-03-09T15:32:16.187056+0000","last_became_active":"2024-03-09T14:46:18.752413+0000","last_became_peered":"2024-03-09T14:46:18.752413+0000","last_unstale":"2024-03-09T15:32:16.187056+0000","last_undegraded":"2024-03-09T15:32:16.187056+0000","last_fullsized":"2024-03-09T15:32:16.187056+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:09:14.935604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.16","version":"261'52","reported_seq":184,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.788486+0000","last_change":"2024-03-09T14:39:13.263228+0000","last_active":"2024-03-09T15:32:25.788486+0000","last_peered":"2024-03-09T15:32:25.788486+0000","last_clean":"2024-03-09T15:32:25.788486+0000","last_became_active":"2024-03-09T14:39:13.262413+0000","last_became_peered":"2024-03-09T14:39:13.262413+0000","last_unstale":"2024-03-09T15:32:25.788486+0000","last_undegraded":"2024-03-09T15:32:25.788486+0000","last_fullsized":"2024-03-09T15:32:25.788486+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:02:22.902848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":163,"num_write":78,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.10","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.199392+0000","last_change":"2024-03-09T15:09:50.075112+0000","last_active":"2024-03-09T15:33:11.199392+0000","last_peered":"2024-03-09T15:33:11.199392+0000","last_clean":"2024-03-09T15:33:11.199392+0000","last_became_active":"2024-03-09T15:09:50.074274+0000","last_became_peered":"2024-03-09T15:09:50.074274+0000","last_unstale":"2024-03-09T15:33:11.199392+0000","last_undegraded":"2024-03-09T15:33:11.199392+0000","last_fullsized":"2024-03-09T15:33:11.199392+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:34:26.130292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.17","version":"0'0","reported_seq":42,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.788513+0000","last_change":"2024-03-09T14:46:26.894297+0000","last_active":"2024-03-09T15:32:25.788513+0000","last_peered":"2024-03-09T15:32:25.788513+0000","last_clean":"2024-03-09T15:32:25.788513+0000","last_became_active":"2024-03-09T14:46:26.893514+0000","last_became_peered":"2024-03-09T14:46:26.893514+0000","last_unstale":"2024-03-09T15:32:25.788513+0000","last_undegraded":"2024-03-09T15:32:25.788513+0000","last_fullsized":"2024-03-09T15:32:25.788513+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:03:25.018128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.16","version":"0'0","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:15.960497+0000","last_change":"2024-03-09T14:46:18.773133+0000","last_active":"2024-03-09T15:32:15.960497+0000","last_peered":"2024-03-09T15:32:15.960497+0000","last_clean":"2024-03-09T15:32:15.960497+0000","last_became_active":"2024-03-09T14:46:18.769274+0000","last_became_peered":"2024-03-09T14:46:18.769274+0000","last_unstale":"2024-03-09T15:32:15.960497+0000","last_undegraded":"2024-03-09T15:32:15.960497+0000","last_fullsized":"2024-03-09T15:32:15.960497+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:01:15.489364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.15","version":"0'0","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.962879+0000","last_change":"2024-03-09T14:39:13.263907+0000","last_active":"2024-03-09T15:32:25.962879+0000","last_peered":"2024-03-09T15:32:25.962879+0000","last_clean":"2024-03-09T15:32:25.962879+0000","last_became_active":"2024-03-09T14:39:13.262066+0000","last_became_peered":"2024-03-09T14:39:13.262066+0000","last_unstale":"2024-03-09T15:32:25.962879+0000","last_undegraded":"2024-03-09T15:32:25.962879+0000","last_fullsized":"2024-03-09T15:32:25.962879+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:55:26.909612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.11","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.188977+0000","last_change":"2024-03-09T15:09:50.086149+0000","last_active":"2024-03-09T15:33:11.188977+0000","last_peered":"2024-03-09T15:33:11.188977+0000","last_clean":"2024-03-09T15:33:11.188977+0000","last_became_active":"2024-03-09T15:09:50.085346+0000","last_became_peered":"2024-03-09T15:09:50.085346+0000","last_unstale":"2024-03-09T15:33:11.188977+0000","last_undegraded":"2024-03-09T15:33:11.188977+0000","last_fullsized":"2024-03-09T15:33:11.188977+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:54:52.652612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.16","version":"0'0","reported_seq":42,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.175271+0000","last_change":"2024-03-09T14:46:26.818659+0000","last_active":"2024-03-09T15:32:21.175271+0000","last_peered":"2024-03-09T15:32:21.175271+0000","last_clean":"2024-03-09T15:32:21.175271+0000","last_became_active":"2024-03-09T14:46:26.817864+0000","last_became_peered":"2024-03-09T14:46:26.817864+0000","last_unstale":"2024-03-09T15:32:21.175271+0000","last_undegraded":"2024-03-09T15:32:21.175271+0000","last_fullsized":"2024-03-09T15:32:21.175271+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:10:59.964957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.17","version":"0'0","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:15.994554+0000","last_change":"2024-03-09T14:46:18.780789+0000","last_active":"2024-03-09T15:32:15.994554+0000","last_peered":"2024-03-09T15:32:15.994554+0000","last_clean":"2024-03-09T15:32:15.994554+0000","last_became_active":"2024-03-09T14:46:18.776430+0000","last_became_peered":"2024-03-09T14:46:18.776430+0000","last_unstale":"2024-03-09T15:32:15.994554+0000","last_undegraded":"2024-03-09T15:32:15.994554+0000","last_fullsized":"2024-03-09T15:32:15.994554+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:39:21.125955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.14","version":"258'67","reported_seq":270,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:06.198380+0000","last_change":"2024-03-09T14:39:13.253820+0000","last_active":"2024-03-09T15:33:06.198380+0000","last_peered":"2024-03-09T15:33:06.198380+0000","last_clean":"2024-03-09T15:33:06.198380+0000","last_became_active":"2024-03-09T14:39:13.253007+0000","last_became_peered":"2024-03-09T14:39:13.253007+0000","last_unstale":"2024-03-09T15:33:06.198380+0000","last_undegraded":"2024-03-09T15:33:06.198380+0000","last_fullsized":"2024-03-09T15:33:06.198380+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:44:10.310392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":411,"num_read_kb":279,"num_write":87,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.16","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776888+0000","last_change":"2024-03-09T15:09:50.087752+0000","last_active":"2024-03-09T15:31:35.776888+0000","last_peered":"2024-03-09T15:31:35.776888+0000","last_clean":"2024-03-09T15:31:35.776888+0000","last_became_active":"2024-03-09T15:09:50.086173+0000","last_became_peered":"2024-03-09T15:09:50.086173+0000","last_unstale":"2024-03-09T15:31:35.776888+0000","last_undegraded":"2024-03-09T15:31:35.776888+0000","last_fullsized":"2024-03-09T15:31:35.776888+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:43:21.719422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.11","version":"261'7","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:25.986303+0000","last_change":"2024-03-09T14:46:26.817795+0000","last_active":"2024-03-09T15:35:25.986303+0000","last_peered":"2024-03-09T15:35:25.986303+0000","last_clean":"2024-03-09T15:35:25.986303+0000","last_became_active":"2024-03-09T14:46:26.817014+0000","last_became_peered":"2024-03-09T14:46:26.817014+0000","last_unstale":"2024-03-09T15:35:25.986303+0000","last_undegraded":"2024-03-09T15:35:25.986303+0000","last_fullsized":"2024-03-09T15:35:25.986303+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:58:54.836958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":23554,"num_write":11,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.10","version":"261'2","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:01.183449+0000","last_change":"2024-03-09T14:46:18.771622+0000","last_active":"2024-03-09T15:33:01.183449+0000","last_peered":"2024-03-09T15:33:01.183449+0000","last_clean":"2024-03-09T15:33:01.183449+0000","last_became_active":"2024-03-09T14:46:18.770936+0000","last_became_peered":"2024-03-09T14:46:18.770936+0000","last_unstale":"2024-03-09T15:33:01.183449+0000","last_undegraded":"2024-03-09T15:33:01.183449+0000","last_fullsized":"2024-03-09T15:33:01.183449+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:04.807309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":2,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.13","version":"258'16","reported_seq":95,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.196014+0000","last_change":"2024-03-09T14:39:13.266972+0000","last_active":"2024-03-09T15:32:56.196014+0000","last_peered":"2024-03-09T15:32:56.196014+0000","last_clean":"2024-03-09T15:32:56.196014+0000","last_became_active":"2024-03-09T14:39:13.266228+0000","last_became_peered":"2024-03-09T14:39:13.266228+0000","last_unstale":"2024-03-09T15:32:56.196014+0000","last_undegraded":"2024-03-09T15:32:56.196014+0000","last_fullsized":"2024-03-09T15:32:56.196014+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:02:45.648908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":45,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.17","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.984716+0000","last_change":"2024-03-09T15:09:50.075655+0000","last_active":"2024-03-09T15:31:35.984716+0000","last_peered":"2024-03-09T15:31:35.984716+0000","last_clean":"2024-03-09T15:31:35.984716+0000","last_became_active":"2024-03-09T15:09:50.075187+0000","last_became_peered":"2024-03-09T15:09:50.075187+0000","last_unstale":"2024-03-09T15:31:35.984716+0000","last_undegraded":"2024-03-09T15:31:35.984716+0000","last_fullsized":"2024-03-09T15:31:35.984716+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:25:34.672578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.10","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:05.956045+0000","last_change":"2024-03-09T14:46:26.803323+0000","last_active":"2024-03-09T15:33:05.956045+0000","last_peered":"2024-03-09T15:33:05.956045+0000","last_clean":"2024-03-09T15:33:05.956045+0000","last_became_active":"2024-03-09T14:46:26.802476+0000","last_became_peered":"2024-03-09T14:46:26.802476+0000","last_unstale":"2024-03-09T15:33:05.956045+0000","last_undegraded":"2024-03-09T15:33:05.956045+0000","last_fullsized":"2024-03-09T15:33:05.956045+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:04:33.172429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.11","version":"0'0","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.795596+0000","last_change":"2024-03-09T14:46:18.761711+0000","last_active":"2024-03-09T15:33:00.795596+0000","last_peered":"2024-03-09T15:33:00.795596+0000","last_clean":"2024-03-09T15:33:00.795596+0000","last_became_active":"2024-03-09T14:46:18.760959+0000","last_became_peered":"2024-03-09T14:46:18.760959+0000","last_unstale":"2024-03-09T15:33:00.795596+0000","last_undegraded":"2024-03-09T15:33:00.795596+0000","last_fullsized":"2024-03-09T15:33:00.795596+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:08.964116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.12","version":"258'7","reported_seq":77,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.003193+0000","last_change":"2024-03-09T14:39:13.274560+0000","last_active":"2024-03-09T15:32:56.003193+0000","last_peered":"2024-03-09T15:32:56.003193+0000","last_clean":"2024-03-09T15:32:56.003193+0000","last_became_active":"2024-03-09T14:39:13.273740+0000","last_became_peered":"2024-03-09T14:39:13.273740+0000","last_unstale":"2024-03-09T15:32:56.003193+0000","last_undegraded":"2024-03-09T15:32:56.003193+0000","last_fullsized":"2024-03-09T15:32:56.003193+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:17.942276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":26,"num_write":13,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.14","version":"260'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.007791+0000","last_change":"2024-03-09T15:09:50.065240+0000","last_active":"2024-03-09T15:33:16.007791+0000","last_peered":"2024-03-09T15:33:16.007791+0000","last_clean":"2024-03-09T15:33:16.007791+0000","last_became_active":"2024-03-09T15:09:50.064428+0000","last_became_peered":"2024-03-09T15:09:50.064428+0000","last_unstale":"2024-03-09T15:33:16.007791+0000","last_undegraded":"2024-03-09T15:33:16.007791+0000","last_fullsized":"2024-03-09T15:33:16.007791+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:34:02.983911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.13","version":"261'2","reported_seq":47,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.798066+0000","last_change":"2024-03-09T14:46:26.813826+0000","last_active":"2024-03-09T15:33:10.798066+0000","last_peered":"2024-03-09T15:33:10.798066+0000","last_clean":"2024-03-09T15:33:10.798066+0000","last_became_active":"2024-03-09T14:46:26.813088+0000","last_became_peered":"2024-03-09T14:46:26.813088+0000","last_unstale":"2024-03-09T15:33:10.798066+0000","last_undegraded":"2024-03-09T15:33:10.798066+0000","last_fullsized":"2024-03-09T15:33:10.798066+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:37:31.388531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":2,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.12","version":"261'4","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.782261+0000","last_change":"2024-03-09T14:46:18.783155+0000","last_active":"2024-03-09T15:33:00.782261+0000","last_peered":"2024-03-09T15:33:00.782261+0000","last_clean":"2024-03-09T15:33:00.782261+0000","last_became_active":"2024-03-09T14:46:18.782158+0000","last_became_peered":"2024-03-09T14:46:18.782158+0000","last_unstale":"2024-03-09T15:33:00.782261+0000","last_undegraded":"2024-03-09T15:33:00.782261+0000","last_fullsized":"2024-03-09T15:33:00.782261+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:44:00.127768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":11264,"num_write":4,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.11","version":"258'36","reported_seq":148,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.182425+0000","last_change":"2024-03-09T14:39:13.260927+0000","last_active":"2024-03-09T15:32:56.182425+0000","last_peered":"2024-03-09T15:32:56.182425+0000","last_clean":"2024-03-09T15:32:56.182425+0000","last_became_active":"2024-03-09T14:39:13.260219+0000","last_became_peered":"2024-03-09T14:39:13.260219+0000","last_unstale":"2024-03-09T15:32:56.182425+0000","last_undegraded":"2024-03-09T15:32:56.182425+0000","last_fullsized":"2024-03-09T15:32:56.182425+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:27:50.114739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":118,"num_write":57,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.15","version":"261'1","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776826+0000","last_change":"2024-03-09T15:09:50.082323+0000","last_active":"2024-03-09T15:31:35.776826+0000","last_peered":"2024-03-09T15:31:35.776826+0000","last_clean":"2024-03-09T15:31:35.776826+0000","last_became_active":"2024-03-09T15:09:50.081688+0000","last_became_peered":"2024-03-09T15:09:50.081688+0000","last_unstale":"2024-03-09T15:31:35.776826+0000","last_undegraded":"2024-03-09T15:31:35.776826+0000","last_fullsized":"2024-03-09T15:31:35.776826+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:51:04.000384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.12","version":"261'1","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784367+0000","last_change":"2024-03-09T14:46:26.806904+0000","last_active":"2024-03-09T15:33:10.784367+0000","last_peered":"2024-03-09T15:33:10.784367+0000","last_clean":"2024-03-09T15:33:10.784367+0000","last_became_active":"2024-03-09T14:46:26.806155+0000","last_became_peered":"2024-03-09T14:46:26.806155+0000","last_unstale":"2024-03-09T15:33:10.784367+0000","last_undegraded":"2024-03-09T15:33:10.784367+0000","last_fullsized":"2024-03-09T15:33:10.784367+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:30:04.752758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":3072,"num_write":1,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.13","version":"261'2","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:21.217889+0000","last_change":"2024-03-09T14:46:18.783226+0000","last_active":"2024-03-09T15:35:21.217889+0000","last_peered":"2024-03-09T15:35:21.217889+0000","last_clean":"2024-03-09T15:35:21.217889+0000","last_became_active":"2024-03-09T14:46:18.782340+0000","last_became_peered":"2024-03-09T14:46:18.782340+0000","last_unstale":"2024-03-09T15:35:21.217889+0000","last_undegraded":"2024-03-09T15:35:21.217889+0000","last_fullsized":"2024-03-09T15:35:21.217889+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:42:41.572332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":50,"num_write":2,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.10","version":"258'67","reported_seq":256,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.794304+0000","last_change":"2024-03-09T14:39:13.245901+0000","last_active":"2024-03-09T15:32:55.794304+0000","last_peered":"2024-03-09T15:32:55.794304+0000","last_clean":"2024-03-09T15:32:55.794304+0000","last_became_active":"2024-03-09T14:39:13.245107+0000","last_became_peered":"2024-03-09T14:39:13.245107+0000","last_unstale":"2024-03-09T15:32:55.794304+0000","last_undegraded":"2024-03-09T15:32:55.794304+0000","last_fullsized":"2024-03-09T15:32:55.794304+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:25:10.518351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":382,"num_read_kb":270,"num_write":93,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.a","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.978852+0000","last_change":"2024-03-09T15:09:50.080044+0000","last_active":"2024-03-09T15:33:45.978852+0000","last_peered":"2024-03-09T15:33:45.978852+0000","last_clean":"2024-03-09T15:33:45.978852+0000","last_became_active":"2024-03-09T15:09:50.079554+0000","last_became_peered":"2024-03-09T15:09:50.079554+0000","last_unstale":"2024-03-09T15:33:45.978852+0000","last_undegraded":"2024-03-09T15:33:45.978852+0000","last_fullsized":"2024-03-09T15:33:45.978852+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:15:33.287535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.d","version":"261'4","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.806260+0000","last_change":"2024-03-09T14:46:26.813839+0000","last_active":"2024-03-09T15:33:45.806260+0000","last_peered":"2024-03-09T15:33:45.806260+0000","last_clean":"2024-03-09T15:33:45.806260+0000","last_became_active":"2024-03-09T14:46:26.812859+0000","last_became_peered":"2024-03-09T14:46:26.812859+0000","last_unstale":"2024-03-09T15:33:45.806260+0000","last_undegraded":"2024-03-09T15:33:45.806260+0000","last_fullsized":"2024-03-09T15:33:45.806260+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:10:31.220080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":8192,"num_write":9,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.c","version":"261'10","reported_seq":69,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.236677+0000","last_change":"2024-03-09T14:46:18.767068+0000","last_active":"2024-03-09T15:35:51.236677+0000","last_peered":"2024-03-09T15:35:51.236677+0000","last_clean":"2024-03-09T15:35:51.236677+0000","last_became_active":"2024-03-09T14:46:18.766370+0000","last_became_peered":"2024-03-09T14:46:18.766370+0000","last_unstale":"2024-03-09T15:35:51.236677+0000","last_undegraded":"2024-03-09T15:35:51.236677+0000","last_fullsized":"2024-03-09T15:35:51.236677+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:41:36.450438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":20480,"num_write":15,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.f","version":"258'18","reported_seq":99,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:30.802658+0000","last_change":"2024-03-09T14:39:13.271586+0000","last_active":"2024-03-09T15:33:30.802658+0000","last_peered":"2024-03-09T15:33:30.802658+0000","last_clean":"2024-03-09T15:33:30.802658+0000","last_became_active":"2024-03-09T14:39:13.268971+0000","last_became_peered":"2024-03-09T14:39:13.268971+0000","last_unstale":"2024-03-09T15:33:30.802658+0000","last_undegraded":"2024-03-09T15:33:30.802658+0000","last_fullsized":"2024-03-09T15:33:30.802658+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:14:41.597456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.b","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188151+0000","last_change":"2024-03-09T15:09:50.070451+0000","last_active":"2024-03-09T15:32:21.188151+0000","last_peered":"2024-03-09T15:32:21.188151+0000","last_clean":"2024-03-09T15:32:21.188151+0000","last_became_active":"2024-03-09T15:09:50.069508+0000","last_became_peered":"2024-03-09T15:09:50.069508+0000","last_unstale":"2024-03-09T15:32:21.188151+0000","last_undegraded":"2024-03-09T15:32:21.188151+0000","last_fullsized":"2024-03-09T15:32:21.188151+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:39:13.850719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.c","version":"261'2","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.806160+0000","last_change":"2024-03-09T14:46:26.813805+0000","last_active":"2024-03-09T15:33:45.806160+0000","last_peered":"2024-03-09T15:33:45.806160+0000","last_clean":"2024-03-09T15:33:45.806160+0000","last_became_active":"2024-03-09T14:46:26.813009+0000","last_became_peered":"2024-03-09T14:46:26.813009+0000","last_unstale":"2024-03-09T15:33:45.806160+0000","last_undegraded":"2024-03-09T15:33:45.806160+0000","last_fullsized":"2024-03-09T15:33:45.806160+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:34:36.995455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2048,"num_write":7,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.d","version":"261'2","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.976768+0000","last_change":"2024-03-09T14:46:18.771558+0000","last_active":"2024-03-09T15:33:35.976768+0000","last_peered":"2024-03-09T15:33:35.976768+0000","last_clean":"2024-03-09T15:33:35.976768+0000","last_became_active":"2024-03-09T14:46:18.770732+0000","last_became_peered":"2024-03-09T14:46:18.770732+0000","last_unstale":"2024-03-09T15:33:35.976768+0000","last_undegraded":"2024-03-09T15:33:35.976768+0000","last_fullsized":"2024-03-09T15:33:35.976768+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:08:20.215164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.e","version":"261'8","reported_seq":79,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.190234+0000","last_change":"2024-03-09T14:39:13.272393+0000","last_active":"2024-03-09T15:33:16.190234+0000","last_peered":"2024-03-09T15:33:16.190234+0000","last_clean":"2024-03-09T15:33:16.190234+0000","last_became_active":"2024-03-09T14:39:13.271672+0000","last_became_peered":"2024-03-09T14:39:13.271672+0000","last_unstale":"2024-03-09T15:33:16.190234+0000","last_undegraded":"2024-03-09T15:33:16.190234+0000","last_fullsized":"2024-03-09T15:33:16.190234+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:35:02.345192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.8","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.790792+0000","last_change":"2024-03-09T15:09:50.064645+0000","last_active":"2024-03-09T15:33:40.790792+0000","last_peered":"2024-03-09T15:33:40.790792+0000","last_clean":"2024-03-09T15:33:40.790792+0000","last_became_active":"2024-03-09T15:09:50.061662+0000","last_became_peered":"2024-03-09T15:09:50.061662+0000","last_unstale":"2024-03-09T15:33:40.790792+0000","last_undegraded":"2024-03-09T15:33:40.790792+0000","last_fullsized":"2024-03-09T15:33:40.790792+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:32:01.749280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.f","version":"261'5","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.966844+0000","last_change":"2024-03-09T14:46:26.900254+0000","last_active":"2024-03-09T15:33:50.966844+0000","last_peered":"2024-03-09T15:33:50.966844+0000","last_clean":"2024-03-09T15:33:50.966844+0000","last_became_active":"2024-03-09T14:46:26.899430+0000","last_became_peered":"2024-03-09T14:46:26.899430+0000","last_unstale":"2024-03-09T15:33:50.966844+0000","last_undegraded":"2024-03-09T15:33:50.966844+0000","last_fullsized":"2024-03-09T15:33:50.966844+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:58:16.207053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.e","version":"258'3","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.012622+0000","last_change":"2024-03-09T14:46:18.769516+0000","last_active":"2024-03-09T15:33:36.012622+0000","last_peered":"2024-03-09T15:33:36.012622+0000","last_clean":"2024-03-09T15:33:36.012622+0000","last_became_active":"2024-03-09T14:46:18.768694+0000","last_became_peered":"2024-03-09T14:46:18.768694+0000","last_unstale":"2024-03-09T15:33:36.012622+0000","last_undegraded":"2024-03-09T15:33:36.012622+0000","last_fullsized":"2024-03-09T15:33:36.012622+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:10:50.806709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.d","version":"261'43","reported_seq":159,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:15.785461+0000","last_change":"2024-03-09T14:39:13.266337+0000","last_active":"2024-03-09T15:33:15.785461+0000","last_peered":"2024-03-09T15:33:15.785461+0000","last_clean":"2024-03-09T15:33:15.785461+0000","last_became_active":"2024-03-09T14:39:13.257275+0000","last_became_peered":"2024-03-09T14:39:13.257275+0000","last_unstale":"2024-03-09T15:33:15.785461+0000","last_undegraded":"2024-03-09T15:33:15.785461+0000","last_fullsized":"2024-03-09T15:33:15.785461+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:45:48.503144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":175,"num_read_kb":123,"num_write":70,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.9","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:41.206696+0000","last_change":"2024-03-09T15:09:50.077371+0000","last_active":"2024-03-09T15:33:41.206696+0000","last_peered":"2024-03-09T15:33:41.206696+0000","last_clean":"2024-03-09T15:33:41.206696+0000","last_became_active":"2024-03-09T15:09:50.076702+0000","last_became_peered":"2024-03-09T15:09:50.076702+0000","last_unstale":"2024-03-09T15:33:41.206696+0000","last_undegraded":"2024-03-09T15:33:41.206696+0000","last_fullsized":"2024-03-09T15:33:41.206696+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:02:55.914615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.e","version":"261'4","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.806311+0000","last_change":"2024-03-09T14:46:26.818547+0000","last_active":"2024-03-09T15:33:45.806311+0000","last_peered":"2024-03-09T15:33:45.806311+0000","last_clean":"2024-03-09T15:33:45.806311+0000","last_became_active":"2024-03-09T14:46:26.814709+0000","last_became_peered":"2024-03-09T14:46:26.814709+0000","last_unstale":"2024-03-09T15:33:45.806311+0000","last_undegraded":"2024-03-09T15:33:45.806311+0000","last_fullsized":"2024-03-09T15:33:45.806311+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:11:42.597841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":9,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.f","version":"261'4","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.965610+0000","last_change":"2024-03-09T14:46:18.777803+0000","last_active":"2024-03-09T15:33:45.965610+0000","last_peered":"2024-03-09T15:33:45.965610+0000","last_clean":"2024-03-09T15:33:45.965610+0000","last_became_active":"2024-03-09T14:46:18.766000+0000","last_became_peered":"2024-03-09T14:46:18.766000+0000","last_unstale":"2024-03-09T15:33:45.965610+0000","last_undegraded":"2024-03-09T15:33:45.965610+0000","last_fullsized":"2024-03-09T15:33:45.965610+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:41:01.857202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4096,"num_write":9,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.c","version":"0'0","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:15.972703+0000","last_change":"2024-03-09T14:39:13.271296+0000","last_active":"2024-03-09T15:33:15.972703+0000","last_peered":"2024-03-09T15:33:15.972703+0000","last_clean":"2024-03-09T15:33:15.972703+0000","last_became_active":"2024-03-09T14:39:13.270527+0000","last_became_peered":"2024-03-09T14:39:13.270527+0000","last_unstale":"2024-03-09T15:33:15.972703+0000","last_undegraded":"2024-03-09T15:33:15.972703+0000","last_fullsized":"2024-03-09T15:33:15.972703+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:25:04.404490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.e","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.787362+0000","last_change":"2024-03-09T15:09:50.087358+0000","last_active":"2024-03-09T15:32:20.787362+0000","last_peered":"2024-03-09T15:32:20.787362+0000","last_clean":"2024-03-09T15:32:20.787362+0000","last_became_active":"2024-03-09T15:09:50.085548+0000","last_became_peered":"2024-03-09T15:09:50.085548+0000","last_unstale":"2024-03-09T15:32:20.787362+0000","last_undegraded":"2024-03-09T15:32:20.787362+0000","last_fullsized":"2024-03-09T15:32:20.787362+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:13.595041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.9","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:56.169630+0000","last_change":"2024-03-09T14:46:26.802054+0000","last_active":"2024-03-09T15:31:56.169630+0000","last_peered":"2024-03-09T15:31:56.169630+0000","last_clean":"2024-03-09T15:31:56.169630+0000","last_became_active":"2024-03-09T14:46:26.801235+0000","last_became_peered":"2024-03-09T14:46:26.801235+0000","last_unstale":"2024-03-09T15:31:56.169630+0000","last_undegraded":"2024-03-09T15:31:56.169630+0000","last_fullsized":"2024-03-09T15:31:56.169630+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:49:20.412101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.8","version":"0'0","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.988186+0000","last_change":"2024-03-09T14:46:18.775300+0000","last_active":"2024-03-09T15:31:50.988186+0000","last_peered":"2024-03-09T15:31:50.988186+0000","last_clean":"2024-03-09T15:31:50.988186+0000","last_became_active":"2024-03-09T14:46:18.774547+0000","last_became_peered":"2024-03-09T14:46:18.774547+0000","last_unstale":"2024-03-09T15:31:50.988186+0000","last_undegraded":"2024-03-09T15:31:50.988186+0000","last_fullsized":"2024-03-09T15:31:50.988186+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:18:57.376200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.b","version":"258'10","reported_seq":85,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.007679+0000","last_change":"2024-03-09T14:39:13.258020+0000","last_active":"2024-03-09T15:33:16.007679+0000","last_peered":"2024-03-09T15:33:16.007679+0000","last_clean":"2024-03-09T15:33:16.007679+0000","last_became_active":"2024-03-09T14:39:13.257238+0000","last_became_peered":"2024-03-09T14:39:13.257238+0000","last_unstale":"2024-03-09T15:33:16.007679+0000","last_undegraded":"2024-03-09T15:33:16.007679+0000","last_fullsized":"2024-03-09T15:33:16.007679+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:40:13.243772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.f","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.189467+0000","last_change":"2024-03-09T15:09:50.071187+0000","last_active":"2024-03-09T15:32:26.189467+0000","last_peered":"2024-03-09T15:32:26.189467+0000","last_clean":"2024-03-09T15:32:26.189467+0000","last_became_active":"2024-03-09T15:09:50.070526+0000","last_became_peered":"2024-03-09T15:09:50.070526+0000","last_unstale":"2024-03-09T15:32:26.189467+0000","last_undegraded":"2024-03-09T15:32:26.189467+0000","last_fullsized":"2024-03-09T15:32:26.189467+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:03:16.280366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.8","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:01.183115+0000","last_change":"2024-03-09T14:46:26.894478+0000","last_active":"2024-03-09T15:32:01.183115+0000","last_peered":"2024-03-09T15:32:01.183115+0000","last_clean":"2024-03-09T15:32:01.183115+0000","last_became_active":"2024-03-09T14:46:26.893908+0000","last_became_peered":"2024-03-09T14:46:26.893908+0000","last_unstale":"2024-03-09T15:32:01.183115+0000","last_undegraded":"2024-03-09T15:32:01.183115+0000","last_fullsized":"2024-03-09T15:32:01.183115+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:43:51.901456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.9","version":"261'2","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:51.136707+0000","last_change":"2024-03-09T14:46:18.777691+0000","last_active":"2024-03-09T15:31:51.136707+0000","last_peered":"2024-03-09T15:31:51.136707+0000","last_clean":"2024-03-09T15:31:51.136707+0000","last_became_active":"2024-03-09T14:46:18.761367+0000","last_became_peered":"2024-03-09T14:46:18.761367+0000","last_unstale":"2024-03-09T15:31:51.136707+0000","last_undegraded":"2024-03-09T15:31:51.136707+0000","last_fullsized":"2024-03-09T15:31:51.136707+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:44:32.343854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.a","version":"258'23","reported_seq":123,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205291+0000","last_change":"2024-03-09T14:39:13.253482+0000","last_active":"2024-03-09T15:33:36.205291+0000","last_peered":"2024-03-09T15:33:36.205291+0000","last_clean":"2024-03-09T15:33:36.205291+0000","last_became_active":"2024-03-09T14:39:13.252547+0000","last_became_peered":"2024-03-09T14:39:13.252547+0000","last_unstale":"2024-03-09T15:33:36.205291+0000","last_undegraded":"2024-03-09T15:33:36.205291+0000","last_fullsized":"2024-03-09T15:33:36.205291+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:32:42.396388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":86,"num_write":38,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.c","version":"261'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188203+0000","last_change":"2024-03-09T15:09:50.070468+0000","last_active":"2024-03-09T15:32:21.188203+0000","last_peered":"2024-03-09T15:32:21.188203+0000","last_clean":"2024-03-09T15:32:21.188203+0000","last_became_active":"2024-03-09T15:09:50.069324+0000","last_became_peered":"2024-03-09T15:09:50.069324+0000","last_unstale":"2024-03-09T15:32:21.188203+0000","last_undegraded":"2024-03-09T15:32:21.188203+0000","last_fullsized":"2024-03-09T15:32:21.188203+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:23:32.203630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.b","version":"258'3","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.964465+0000","last_change":"2024-03-09T14:46:26.805564+0000","last_active":"2024-03-09T15:33:40.964465+0000","last_peered":"2024-03-09T15:33:40.964465+0000","last_clean":"2024-03-09T15:33:40.964465+0000","last_became_active":"2024-03-09T14:46:26.804615+0000","last_became_peered":"2024-03-09T14:46:26.804615+0000","last_unstale":"2024-03-09T15:33:40.964465+0000","last_undegraded":"2024-03-09T15:33:40.964465+0000","last_fullsized":"2024-03-09T15:33:40.964465+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:54:32.765203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.a","version":"258'3","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:00.990762+0000","last_change":"2024-03-09T14:46:18.769098+0000","last_active":"2024-03-09T15:32:00.990762+0000","last_peered":"2024-03-09T15:32:00.990762+0000","last_clean":"2024-03-09T15:32:00.990762+0000","last_became_active":"2024-03-09T14:46:18.768431+0000","last_became_peered":"2024-03-09T14:46:18.768431+0000","last_unstale":"2024-03-09T15:32:00.990762+0000","last_undegraded":"2024-03-09T15:32:00.990762+0000","last_fullsized":"2024-03-09T15:32:00.990762+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:28:28.024126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.9","version":"261'43","reported_seq":157,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.801289+0000","last_change":"2024-03-09T14:39:13.260724+0000","last_active":"2024-03-09T15:33:25.801289+0000","last_peered":"2024-03-09T15:33:25.801289+0000","last_clean":"2024-03-09T15:33:25.801289+0000","last_became_active":"2024-03-09T14:39:13.259975+0000","last_became_peered":"2024-03-09T14:39:13.259975+0000","last_unstale":"2024-03-09T15:33:25.801289+0000","last_undegraded":"2024-03-09T15:33:25.801289+0000","last_fullsized":"2024-03-09T15:33:25.801289+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:26:32.899703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":169,"num_read_kb":119,"num_write":74,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.d","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.995632+0000","last_change":"2024-03-09T15:09:50.084646+0000","last_active":"2024-03-09T15:32:20.995632+0000","last_peered":"2024-03-09T15:32:20.995632+0000","last_clean":"2024-03-09T15:32:20.995632+0000","last_became_active":"2024-03-09T15:09:50.084074+0000","last_became_peered":"2024-03-09T15:09:50.084074+0000","last_unstale":"2024-03-09T15:32:20.995632+0000","last_undegraded":"2024-03-09T15:32:20.995632+0000","last_fullsized":"2024-03-09T15:32:20.995632+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:16:13.790167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.a","version":"258'3","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:06.184462+0000","last_change":"2024-03-09T14:46:26.826875+0000","last_active":"2024-03-09T15:32:06.184462+0000","last_peered":"2024-03-09T15:32:06.184462+0000","last_clean":"2024-03-09T15:32:06.184462+0000","last_became_active":"2024-03-09T14:46:26.825949+0000","last_became_peered":"2024-03-09T14:46:26.825949+0000","last_unstale":"2024-03-09T15:32:06.184462+0000","last_undegraded":"2024-03-09T15:32:06.184462+0000","last_fullsized":"2024-03-09T15:32:06.184462+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:24:09.628828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.b","version":"261'2","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191328+0000","last_change":"2024-03-09T14:46:18.785518+0000","last_active":"2024-03-09T15:33:36.191328+0000","last_peered":"2024-03-09T15:33:36.191328+0000","last_clean":"2024-03-09T15:33:36.191328+0000","last_became_active":"2024-03-09T14:46:18.784869+0000","last_became_peered":"2024-03-09T14:46:18.784869+0000","last_unstale":"2024-03-09T15:33:36.191328+0000","last_undegraded":"2024-03-09T15:33:36.191328+0000","last_fullsized":"2024-03-09T15:33:36.191328+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:03:42.729066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":2,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.8","version":"258'10","reported_seq":86,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.787542+0000","last_change":"2024-03-09T14:39:13.264924+0000","last_active":"2024-03-09T15:33:25.787542+0000","last_peered":"2024-03-09T15:33:25.787542+0000","last_clean":"2024-03-09T15:33:25.787542+0000","last_became_active":"2024-03-09T14:39:13.264006+0000","last_became_peered":"2024-03-09T14:39:13.264006+0000","last_unstale":"2024-03-09T15:33:25.787542+0000","last_undegraded":"2024-03-09T15:33:25.787542+0000","last_fullsized":"2024-03-09T15:33:25.787542+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:24:57.691583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.2","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.193658+0000","last_change":"2024-03-09T15:09:50.066069+0000","last_active":"2024-03-09T15:32:46.193658+0000","last_peered":"2024-03-09T15:32:46.193658+0000","last_clean":"2024-03-09T15:32:46.193658+0000","last_became_active":"2024-03-09T15:09:50.065487+0000","last_became_peered":"2024-03-09T15:09:50.065487+0000","last_unstale":"2024-03-09T15:32:46.193658+0000","last_undegraded":"2024-03-09T15:32:46.193658+0000","last_fullsized":"2024-03-09T15:32:46.193658+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:16:19.163190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.5","version":"258'2","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:30.997928+0000","last_change":"2024-03-09T14:46:26.813029+0000","last_active":"2024-03-09T15:32:30.997928+0000","last_peered":"2024-03-09T15:32:30.997928+0000","last_clean":"2024-03-09T15:32:30.997928+0000","last_became_active":"2024-03-09T14:46:26.812137+0000","last_became_peered":"2024-03-09T14:46:26.812137+0000","last_unstale":"2024-03-09T15:32:30.997928+0000","last_undegraded":"2024-03-09T15:32:30.997928+0000","last_fullsized":"2024-03-09T15:32:30.997928+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:36:47.037465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4096,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.4","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.185772+0000","last_change":"2024-03-09T14:46:18.752392+0000","last_active":"2024-03-09T15:32:11.185772+0000","last_peered":"2024-03-09T15:32:11.185772+0000","last_clean":"2024-03-09T15:32:11.185772+0000","last_became_active":"2024-03-09T14:46:18.751608+0000","last_became_peered":"2024-03-09T14:46:18.751608+0000","last_unstale":"2024-03-09T15:32:11.185772+0000","last_undegraded":"2024-03-09T15:32:11.185772+0000","last_fullsized":"2024-03-09T15:32:11.185772+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:32:15.618158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.7","version":"261'8","reported_seq":80,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.010433+0000","last_change":"2024-03-09T14:39:13.280605+0000","last_active":"2024-03-09T15:33:26.010433+0000","last_peered":"2024-03-09T15:33:26.010433+0000","last_clean":"2024-03-09T15:33:26.010433+0000","last_became_active":"2024-03-09T14:39:13.273363+0000","last_became_peered":"2024-03-09T14:39:13.273363+0000","last_unstale":"2024-03-09T15:33:26.010433+0000","last_undegraded":"2024-03-09T15:33:26.010433+0000","last_fullsized":"2024-03-09T15:33:26.010433+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:45:47.783708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.3","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.183450+0000","last_change":"2024-03-09T15:09:50.078106+0000","last_active":"2024-03-09T15:32:46.183450+0000","last_peered":"2024-03-09T15:32:46.183450+0000","last_clean":"2024-03-09T15:32:46.183450+0000","last_became_active":"2024-03-09T15:09:50.072488+0000","last_became_peered":"2024-03-09T15:09:50.072488+0000","last_unstale":"2024-03-09T15:32:46.183450+0000","last_undegraded":"2024-03-09T15:32:46.183450+0000","last_fullsized":"2024-03-09T15:32:46.183450+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:20:34.288932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.4","version":"258'2","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.995582+0000","last_change":"2024-03-09T14:46:26.895288+0000","last_active":"2024-03-09T15:32:20.995582+0000","last_peered":"2024-03-09T15:32:20.995582+0000","last_clean":"2024-03-09T15:32:20.995582+0000","last_became_active":"2024-03-09T14:46:26.894506+0000","last_became_peered":"2024-03-09T14:46:26.894506+0000","last_unstale":"2024-03-09T15:32:20.995582+0000","last_undegraded":"2024-03-09T15:32:20.995582+0000","last_fullsized":"2024-03-09T15:32:20.995582+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:23:02.279766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":23,"num_write":2,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.5","version":"258'5","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.962822+0000","last_change":"2024-03-09T14:46:18.771982+0000","last_active":"2024-03-09T15:32:25.962822+0000","last_peered":"2024-03-09T15:32:25.962822+0000","last_clean":"2024-03-09T15:32:25.962822+0000","last_became_active":"2024-03-09T14:46:18.770799+0000","last_became_peered":"2024-03-09T14:46:18.770799+0000","last_unstale":"2024-03-09T15:32:25.962822+0000","last_undegraded":"2024-03-09T15:32:25.962822+0000","last_fullsized":"2024-03-09T15:32:25.962822+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:15:47.749134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.6","version":"261'22","reported_seq":112,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.787743+0000","last_change":"2024-03-09T14:39:13.268293+0000","last_active":"2024-03-09T15:33:25.787743+0000","last_peered":"2024-03-09T15:33:25.787743+0000","last_clean":"2024-03-09T15:33:25.787743+0000","last_became_active":"2024-03-09T14:39:13.266246+0000","last_became_peered":"2024-03-09T14:39:13.266246+0000","last_unstale":"2024-03-09T15:33:25.787743+0000","last_undegraded":"2024-03-09T15:33:25.787743+0000","last_fullsized":"2024-03-09T15:33:25.787743+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:11:53.843138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":65,"num_write":41,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.0","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:00.968771+0000","last_change":"2024-03-09T15:09:50.064673+0000","last_active":"2024-03-09T15:34:00.968771+0000","last_peered":"2024-03-09T15:34:00.968771+0000","last_clean":"2024-03-09T15:34:00.968771+0000","last_became_active":"2024-03-09T15:09:50.063350+0000","last_became_peered":"2024-03-09T15:09:50.063350+0000","last_unstale":"2024-03-09T15:34:00.968771+0000","last_undegraded":"2024-03-09T15:34:00.968771+0000","last_fullsized":"2024-03-09T15:34:00.968771+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:55.461559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.7","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:01.183396+0000","last_change":"2024-03-09T14:46:26.897079+0000","last_active":"2024-03-09T15:32:01.183396+0000","last_peered":"2024-03-09T15:32:01.183396+0000","last_clean":"2024-03-09T15:32:01.183396+0000","last_became_active":"2024-03-09T14:46:26.896503+0000","last_became_peered":"2024-03-09T14:46:26.896503+0000","last_unstale":"2024-03-09T15:32:01.183396+0000","last_undegraded":"2024-03-09T15:32:01.183396+0000","last_fullsized":"2024-03-09T15:32:01.183396+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:57:37.168283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.6","version":"261'2","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:01.200543+0000","last_change":"2024-03-09T14:46:18.779289+0000","last_active":"2024-03-09T15:34:01.200543+0000","last_peered":"2024-03-09T15:34:01.200543+0000","last_clean":"2024-03-09T15:34:01.200543+0000","last_became_active":"2024-03-09T14:46:18.778572+0000","last_became_peered":"2024-03-09T14:46:18.778572+0000","last_unstale":"2024-03-09T15:34:01.200543+0000","last_undegraded":"2024-03-09T15:34:01.200543+0000","last_fullsized":"2024-03-09T15:34:01.200543+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:28:53.930043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.5","version":"258'18","reported_seq":107,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205525+0000","last_change":"2024-03-09T14:39:13.264712+0000","last_active":"2024-03-09T15:33:36.205525+0000","last_peered":"2024-03-09T15:33:36.205525+0000","last_clean":"2024-03-09T15:33:36.205525+0000","last_became_active":"2024-03-09T14:39:13.263833+0000","last_became_peered":"2024-03-09T14:39:13.263833+0000","last_unstale":"2024-03-09T15:33:36.205525+0000","last_undegraded":"2024-03-09T15:33:36.205525+0000","last_fullsized":"2024-03-09T15:33:36.205525+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:10:25.603434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":58,"num_write":29,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.1","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.180226+0000","last_change":"2024-03-09T15:09:50.069405+0000","last_active":"2024-03-09T15:32:46.180226+0000","last_peered":"2024-03-09T15:32:46.180226+0000","last_clean":"2024-03-09T15:32:46.180226+0000","last_became_active":"2024-03-09T15:09:50.067947+0000","last_became_peered":"2024-03-09T15:09:50.067947+0000","last_unstale":"2024-03-09T15:32:46.180226+0000","last_undegraded":"2024-03-09T15:32:46.180226+0000","last_fullsized":"2024-03-09T15:32:46.180226+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:29:17.390497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"67.6","version":"261'2","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:56.169833+0000","last_change":"2024-03-09T14:46:26.809411+0000","last_active":"2024-03-09T15:31:56.169833+0000","last_peered":"2024-03-09T15:31:56.169833+0000","last_clean":"2024-03-09T15:31:56.169833+0000","last_became_active":"2024-03-09T14:46:26.808384+0000","last_became_peered":"2024-03-09T14:46:26.808384+0000","last_unstale":"2024-03-09T15:31:56.169833+0000","last_undegraded":"2024-03-09T15:31:56.169833+0000","last_fullsized":"2024-03-09T15:31:56.169833+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:53:31.797035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90604,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":180,"num_write":2,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.7","version":"261'7","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:51.180691+0000","last_change":"2024-03-09T14:46:18.763286+0000","last_active":"2024-03-09T15:31:51.180691+0000","last_peered":"2024-03-09T15:31:51.180691+0000","last_clean":"2024-03-09T15:31:51.180691+0000","last_became_active":"2024-03-09T14:46:18.762472+0000","last_became_peered":"2024-03-09T14:46:18.762472+0000","last_unstale":"2024-03-09T15:31:51.180691+0000","last_undegraded":"2024-03-09T15:31:51.180691+0000","last_fullsized":"2024-03-09T15:31:51.180691+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:30.905827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":30722,"num_write":16,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.4","version":"258'10","reported_seq":87,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.960925+0000","last_change":"2024-03-09T14:39:13.270957+0000","last_active":"2024-03-09T15:33:25.960925+0000","last_peered":"2024-03-09T15:33:25.960925+0000","last_clean":"2024-03-09T15:33:25.960925+0000","last_became_active":"2024-03-09T14:39:13.270238+0000","last_became_peered":"2024-03-09T14:39:13.270238+0000","last_unstale":"2024-03-09T15:33:25.960925+0000","last_undegraded":"2024-03-09T15:33:25.960925+0000","last_fullsized":"2024-03-09T15:33:25.960925+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:25:19.656005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.6","version":"260'1","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.977825+0000","last_change":"2024-03-09T15:09:50.072623+0000","last_active":"2024-03-09T15:33:40.977825+0000","last_peered":"2024-03-09T15:33:40.977825+0000","last_clean":"2024-03-09T15:33:40.977825+0000","last_became_active":"2024-03-09T15:09:50.072180+0000","last_became_peered":"2024-03-09T15:09:50.072180+0000","last_unstale":"2024-03-09T15:33:40.977825+0000","last_undegraded":"2024-03-09T15:33:40.977825+0000","last_fullsized":"2024-03-09T15:33:40.977825+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:17:47.602511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1","version":"0'0","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:16.142380+0000","last_change":"2024-03-09T14:46:26.799783+0000","last_active":"2024-03-09T15:32:16.142380+0000","last_peered":"2024-03-09T15:32:16.142380+0000","last_clean":"2024-03-09T15:32:16.142380+0000","last_became_active":"2024-03-09T14:46:26.799019+0000","last_became_peered":"2024-03-09T14:46:26.799019+0000","last_unstale":"2024-03-09T15:32:16.142380+0000","last_undegraded":"2024-03-09T15:32:16.142380+0000","last_fullsized":"2024-03-09T15:32:16.142380+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:37:13.610839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.0","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.189237+0000","last_change":"2024-03-09T14:46:18.753025+0000","last_active":"2024-03-09T15:32:26.189237+0000","last_peered":"2024-03-09T15:32:26.189237+0000","last_clean":"2024-03-09T15:32:26.189237+0000","last_became_active":"2024-03-09T14:46:18.752290+0000","last_became_peered":"2024-03-09T14:46:18.752290+0000","last_unstale":"2024-03-09T15:32:26.189237+0000","last_undegraded":"2024-03-09T15:32:26.189237+0000","last_fullsized":"2024-03-09T15:32:26.189237+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:33:49.576573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.3","version":"261'61","reported_seq":215,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.192284+0000","last_change":"2024-03-09T14:39:13.259947+0000","last_active":"2024-03-09T15:33:26.192284+0000","last_peered":"2024-03-09T15:33:26.192284+0000","last_clean":"2024-03-09T15:33:26.192284+0000","last_became_active":"2024-03-09T14:39:13.259125+0000","last_became_peered":"2024-03-09T14:39:13.259125+0000","last_unstale":"2024-03-09T15:33:26.192284+0000","last_undegraded":"2024-03-09T15:33:26.192284+0000","last_fullsized":"2024-03-09T15:33:26.192284+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:52:15.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":281,"num_read_kb":205,"num_write":98,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.7","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:41.014114+0000","last_change":"2024-03-09T15:09:50.064879+0000","last_active":"2024-03-09T15:33:41.014114+0000","last_peered":"2024-03-09T15:33:41.014114+0000","last_clean":"2024-03-09T15:33:41.014114+0000","last_became_active":"2024-03-09T15:09:50.062429+0000","last_became_peered":"2024-03-09T15:09:50.062429+0000","last_unstale":"2024-03-09T15:33:41.014114+0000","last_undegraded":"2024-03-09T15:33:41.014114+0000","last_fullsized":"2024-03-09T15:33:41.014114+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:10:20.402757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.0","version":"261'5","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:36.207793+0000","last_change":"2024-03-09T14:46:26.816110+0000","last_active":"2024-03-09T15:34:36.207793+0000","last_peered":"2024-03-09T15:34:36.207793+0000","last_clean":"2024-03-09T15:34:36.207793+0000","last_became_active":"2024-03-09T14:46:26.815311+0000","last_became_peered":"2024-03-09T14:46:26.815311+0000","last_unstale":"2024-03-09T15:34:36.207793+0000","last_undegraded":"2024-03-09T15:34:36.207793+0000","last_fullsized":"2024-03-09T15:34:36.207793+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:26:21.047330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":16384,"num_write":10,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1","version":"258'5","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.185673+0000","last_change":"2024-03-09T14:46:18.758437+0000","last_active":"2024-03-09T15:32:11.185673+0000","last_peered":"2024-03-09T15:32:11.185673+0000","last_clean":"2024-03-09T15:32:11.185673+0000","last_became_active":"2024-03-09T14:46:18.757694+0000","last_became_peered":"2024-03-09T14:46:18.757694+0000","last_unstale":"2024-03-09T15:32:11.185673+0000","last_undegraded":"2024-03-09T15:32:11.185673+0000","last_fullsized":"2024-03-09T15:32:11.185673+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:47:19.096625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":4098,"num_write":9,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.2","version":"258'20","reported_seq":110,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.188905+0000","last_change":"2024-03-09T14:39:13.265013+0000","last_active":"2024-03-09T15:33:26.188905+0000","last_peered":"2024-03-09T15:33:26.188905+0000","last_clean":"2024-03-09T15:33:26.188905+0000","last_became_active":"2024-03-09T14:39:13.264203+0000","last_became_peered":"2024-03-09T14:39:13.264203+0000","last_unstale":"2024-03-09T15:33:26.188905+0000","last_undegraded":"2024-03-09T15:33:26.188905+0000","last_fullsized":"2024-03-09T15:33:26.188905+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:46:43.366482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":65,"num_write":31,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.4","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.001035+0000","last_change":"2024-03-09T15:09:50.070703+0000","last_active":"2024-03-09T15:32:46.001035+0000","last_peered":"2024-03-09T15:32:46.001035+0000","last_clean":"2024-03-09T15:32:46.001035+0000","last_became_active":"2024-03-09T15:09:50.066222+0000","last_became_peered":"2024-03-09T15:09:50.066222+0000","last_unstale":"2024-03-09T15:32:46.001035+0000","last_undegraded":"2024-03-09T15:32:46.001035+0000","last_fullsized":"2024-03-09T15:32:46.001035+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:54:19.245692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.3","version":"0'0","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.774297+0000","last_change":"2024-03-09T14:46:26.826931+0000","last_active":"2024-03-09T15:32:20.774297+0000","last_peered":"2024-03-09T15:32:20.774297+0000","last_clean":"2024-03-09T15:32:20.774297+0000","last_became_active":"2024-03-09T14:46:26.826105+0000","last_became_peered":"2024-03-09T14:46:26.826105+0000","last_unstale":"2024-03-09T15:32:20.774297+0000","last_undegraded":"2024-03-09T15:32:20.774297+0000","last_fullsized":"2024-03-09T15:32:20.774297+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:19:31.840994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.2","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.141234+0000","last_change":"2024-03-09T14:46:18.749838+0000","last_active":"2024-03-09T15:32:11.141234+0000","last_peered":"2024-03-09T15:32:11.141234+0000","last_clean":"2024-03-09T15:32:11.141234+0000","last_became_active":"2024-03-09T14:46:18.749091+0000","last_became_peered":"2024-03-09T14:46:18.749091+0000","last_unstale":"2024-03-09T15:32:11.141234+0000","last_undegraded":"2024-03-09T15:32:11.141234+0000","last_fullsized":"2024-03-09T15:32:11.141234+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:47:14.332817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1","version":"261'25","reported_seq":125,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.787625+0000","last_change":"2024-03-09T14:39:13.257789+0000","last_active":"2024-03-09T15:33:25.787625+0000","last_peered":"2024-03-09T15:33:25.787625+0000","last_clean":"2024-03-09T15:33:25.787625+0000","last_became_active":"2024-03-09T14:39:13.255589+0000","last_became_peered":"2024-03-09T14:39:13.255589+0000","last_unstale":"2024-03-09T15:33:25.787625+0000","last_undegraded":"2024-03-09T15:33:25.787625+0000","last_fullsized":"2024-03-09T15:33:25.787625+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:53:30.259823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":79,"num_write":35,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.5","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:51.194924+0000","last_change":"2024-03-09T15:09:50.083562+0000","last_active":"2024-03-09T15:32:51.194924+0000","last_peered":"2024-03-09T15:32:51.194924+0000","last_clean":"2024-03-09T15:32:51.194924+0000","last_became_active":"2024-03-09T15:09:50.083064+0000","last_became_peered":"2024-03-09T15:09:50.083064+0000","last_unstale":"2024-03-09T15:32:51.194924+0000","last_undegraded":"2024-03-09T15:32:51.194924+0000","last_fullsized":"2024-03-09T15:32:51.194924+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:25:31.798282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.2","version":"258'3","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:16.174077+0000","last_change":"2024-03-09T14:46:26.808768+0000","last_active":"2024-03-09T15:32:16.174077+0000","last_peered":"2024-03-09T15:32:16.174077+0000","last_clean":"2024-03-09T15:32:16.174077+0000","last_became_active":"2024-03-09T14:46:26.807746+0000","last_became_peered":"2024-03-09T14:46:26.807746+0000","last_unstale":"2024-03-09T15:32:16.174077+0000","last_undegraded":"2024-03-09T15:32:16.174077+0000","last_fullsized":"2024-03-09T15:32:16.174077+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:58:17.140698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1024,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.3","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:10.772267+0000","last_change":"2024-03-09T14:46:18.765863+0000","last_active":"2024-03-09T15:32:10.772267+0000","last_peered":"2024-03-09T15:32:10.772267+0000","last_clean":"2024-03-09T15:32:10.772267+0000","last_became_active":"2024-03-09T14:46:18.765077+0000","last_became_peered":"2024-03-09T14:46:18.765077+0000","last_unstale":"2024-03-09T15:32:10.772267+0000","last_undegraded":"2024-03-09T15:32:10.772267+0000","last_fullsized":"2024-03-09T15:32:10.772267+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:26:58.084480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.0","version":"250'25","reported_seq":130,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.202779+0000","last_change":"2024-03-09T14:39:13.269464+0000","last_active":"2024-03-09T15:33:26.202779+0000","last_peered":"2024-03-09T15:33:26.202779+0000","last_clean":"2024-03-09T15:33:26.202779+0000","last_became_active":"2024-03-09T14:39:13.268553+0000","last_became_peered":"2024-03-09T14:39:13.268553+0000","last_unstale":"2024-03-09T15:33:26.202779+0000","last_undegraded":"2024-03-09T15:33:26.202779+0000","last_fullsized":"2024-03-09T15:33:26.202779+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:01:17.783740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":97,"num_write":32,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"261'102","reported_seq":659,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.187916+0000","last_change":"2024-03-09T14:28:32.340306+0000","last_active":"2024-03-09T15:36:06.187916+0000","last_peered":"2024-03-09T15:36:06.187916+0000","last_clean":"2024-03-09T15:36:06.187916+0000","last_became_active":"2024-03-09T14:28:32.339785+0000","last_became_peered":"2024-03-09T14:28:32.339785+0000","last_unstale":"2024-03-09T15:36:06.187916+0000","last_undegraded":"2024-03-09T15:36:06.187916+0000","last_fullsized":"2024-03-09T15:36:06.187916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:39:28.870360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":925441,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":118,"num_read_kb":5377,"num_write":361,"num_write_kb":16955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"261'146","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194366+0000","last_change":"2024-03-09T14:28:34.304676+0000","last_active":"2024-03-09T15:33:36.194366+0000","last_peered":"2024-03-09T15:33:36.194366+0000","last_clean":"2024-03-09T15:33:36.194366+0000","last_became_active":"2024-03-09T14:28:34.303980+0000","last_became_peered":"2024-03-09T14:28:34.303980+0000","last_unstale":"2024-03-09T15:33:36.194366+0000","last_undegraded":"2024-03-09T15:33:36.194366+0000","last_fullsized":"2024-03-09T15:33:36.194366+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:12:53.886509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":546,"num_read_kb":417,"num_write":228,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"261'119","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.937233+0000","last_change":"2024-03-09T14:28:32.307225+0000","last_active":"2024-03-09T15:36:05.937233+0000","last_peered":"2024-03-09T15:36:05.937233+0000","last_clean":"2024-03-09T15:36:05.937233+0000","last_became_active":"2024-03-09T14:28:32.306840+0000","last_became_peered":"2024-03-09T14:28:32.306840+0000","last_unstale":"2024-03-09T15:36:05.937233+0000","last_undegraded":"2024-03-09T15:36:05.937233+0000","last_fullsized":"2024-03-09T15:36:05.937233+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":100,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:46:26.657945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3560140,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":140,"num_read_kb":3004,"num_write":507,"num_write_kb":19720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"261'89","reported_seq":655,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:39.894721+0000","last_change":"2024-03-09T14:28:34.324788+0000","last_active":"2024-03-09T15:34:39.894721+0000","last_peered":"2024-03-09T15:34:39.894721+0000","last_clean":"2024-03-09T15:34:39.894721+0000","last_became_active":"2024-03-09T14:28:34.324053+0000","last_became_peered":"2024-03-09T14:28:34.324053+0000","last_unstale":"2024-03-09T15:34:39.894721+0000","last_undegraded":"2024-03-09T15:34:39.894721+0000","last_fullsized":"2024-03-09T15:34:39.894721+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:16:16.817770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":372,"num_read_kb":300,"num_write":141,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3d","version":"261'135","reported_seq":695,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.959352+0000","last_change":"2024-03-09T14:28:32.292874+0000","last_active":"2024-03-09T15:36:05.959352+0000","last_peered":"2024-03-09T15:36:05.959352+0000","last_clean":"2024-03-09T15:36:05.959352+0000","last_became_active":"2024-03-09T14:28:32.286966+0000","last_became_peered":"2024-03-09T14:28:32.286966+0000","last_unstale":"2024-03-09T15:36:05.959352+0000","last_undegraded":"2024-03-09T15:36:05.959352+0000","last_fullsized":"2024-03-09T15:36:05.959352+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":100,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:34.280241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291735,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":170,"num_read_kb":14171,"num_write":587,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"261'157","reported_seq":718,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191267+0000","last_change":"2024-03-09T14:28:34.307655+0000","last_active":"2024-03-09T15:33:36.191267+0000","last_peered":"2024-03-09T15:33:36.191267+0000","last_clean":"2024-03-09T15:33:36.191267+0000","last_became_active":"2024-03-09T14:28:34.306764+0000","last_became_peered":"2024-03-09T14:28:34.306764+0000","last_unstale":"2024-03-09T15:33:36.191267+0000","last_undegraded":"2024-03-09T15:33:36.191267+0000","last_fullsized":"2024-03-09T15:33:36.191267+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:19:26.321622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":481,"num_write":261,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3c","version":"261'177","reported_seq":769,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.211122+0000","last_change":"2024-03-09T14:28:32.310328+0000","last_active":"2024-03-09T15:36:06.211122+0000","last_peered":"2024-03-09T15:36:06.211122+0000","last_clean":"2024-03-09T15:36:06.211122+0000","last_became_active":"2024-03-09T14:28:32.305615+0000","last_became_peered":"2024-03-09T14:28:32.305615+0000","last_unstale":"2024-03-09T15:36:06.211122+0000","last_undegraded":"2024-03-09T15:36:06.211122+0000","last_fullsized":"2024-03-09T15:36:06.211122+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":100,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:45:12.810858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10569896,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":243,"num_read_kb":28937,"num_write":784,"num_write_kb":45835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"261'156","reported_seq":732,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.012477+0000","last_change":"2024-03-09T14:28:34.316937+0000","last_active":"2024-03-09T15:33:36.012477+0000","last_peered":"2024-03-09T15:33:36.012477+0000","last_clean":"2024-03-09T15:33:36.012477+0000","last_became_active":"2024-03-09T14:28:34.316500+0000","last_became_peered":"2024-03-09T14:28:34.316500+0000","last_unstale":"2024-03-09T15:33:36.012477+0000","last_undegraded":"2024-03-09T15:33:36.012477+0000","last_fullsized":"2024-03-09T15:33:36.012477+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":100,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:11:35.701163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":622,"num_read_kb":488,"num_write":257,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3b","version":"261'135","reported_seq":797,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.227826+0000","last_change":"2024-03-09T14:28:32.317675+0000","last_active":"2024-03-09T15:36:06.227826+0000","last_peered":"2024-03-09T15:36:06.227826+0000","last_clean":"2024-03-09T15:36:06.227826+0000","last_became_active":"2024-03-09T14:28:32.314923+0000","last_became_peered":"2024-03-09T14:28:32.314923+0000","last_unstale":"2024-03-09T15:36:06.227826+0000","last_undegraded":"2024-03-09T15:36:06.227826+0000","last_fullsized":"2024-03-09T15:36:06.227826+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":100,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:29:14.419941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1918192,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":282,"num_read_kb":17282,"num_write":502,"num_write_kb":14747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3c","version":"261'124","reported_seq":706,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.781267+0000","last_change":"2024-03-09T14:28:34.315343+0000","last_active":"2024-03-09T15:32:55.781267+0000","last_peered":"2024-03-09T15:32:55.781267+0000","last_clean":"2024-03-09T15:32:55.781267+0000","last_became_active":"2024-03-09T14:28:34.314617+0000","last_became_peered":"2024-03-09T14:28:34.314617+0000","last_unstale":"2024-03-09T15:32:55.781267+0000","last_undegraded":"2024-03-09T15:32:55.781267+0000","last_fullsized":"2024-03-09T15:32:55.781267+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:05:14.469183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":493,"num_read_kb":395,"num_write":185,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3a","version":"261'128","reported_seq":661,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.870081+0000","last_change":"2024-03-09T14:29:06.433441+0000","last_active":"2024-03-09T15:36:05.870081+0000","last_peered":"2024-03-09T15:36:05.870081+0000","last_clean":"2024-03-09T15:36:05.870081+0000","last_became_active":"2024-03-09T14:29:06.433145+0000","last_became_peered":"2024-03-09T14:29:06.433145+0000","last_unstale":"2024-03-09T15:36:05.870081+0000","last_undegraded":"2024-03-09T15:36:05.870081+0000","last_fullsized":"2024-03-09T15:36:05.870081+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:01:58.493721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3195196,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":189,"num_read_kb":6443,"num_write":585,"num_write_kb":33049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"261'274","reported_seq":939,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.185860+0000","last_change":"2024-03-09T14:28:34.314501+0000","last_active":"2024-03-09T15:33:11.185860+0000","last_peered":"2024-03-09T15:33:11.185860+0000","last_clean":"2024-03-09T15:33:11.185860+0000","last_became_active":"2024-03-09T14:28:34.310343+0000","last_became_peered":"2024-03-09T14:28:34.310343+0000","last_unstale":"2024-03-09T15:33:11.185860+0000","last_undegraded":"2024-03-09T15:33:11.185860+0000","last_fullsized":"2024-03-09T15:33:11.185860+0000","mapping_epoch":22,"log_start":"247'200","ondisk_log_start":"247'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":200,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:36:50.027757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1205,"num_read_kb":1028,"num_write":418,"num_write_kb":253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"261'123","reported_seq":679,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.218061+0000","last_change":"2024-03-09T14:28:32.314711+0000","last_active":"2024-03-09T15:36:06.218061+0000","last_peered":"2024-03-09T15:36:06.218061+0000","last_clean":"2024-03-09T15:36:06.218061+0000","last_became_active":"2024-03-09T14:28:32.314258+0000","last_became_peered":"2024-03-09T14:28:32.314258+0000","last_unstale":"2024-03-09T15:36:06.218061+0000","last_undegraded":"2024-03-09T15:36:06.218061+0000","last_fullsized":"2024-03-09T15:36:06.218061+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":100,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:21:00.179690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262376,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":159,"num_read_kb":17581,"num_write":456,"num_write_kb":22929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3e","version":"261'44","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.189153+0000","last_change":"2024-03-09T14:28:34.323080+0000","last_active":"2024-03-09T15:33:11.189153+0000","last_peered":"2024-03-09T15:33:11.189153+0000","last_clean":"2024-03-09T15:33:11.189153+0000","last_became_active":"2024-03-09T14:28:34.322206+0000","last_became_peered":"2024-03-09T14:28:34.322206+0000","last_unstale":"2024-03-09T15:33:11.189153+0000","last_undegraded":"2024-03-09T15:33:11.189153+0000","last_fullsized":"2024-03-09T15:33:11.189153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:55:41.173766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":201,"num_read_kb":171,"num_write":60,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"261'128","reported_seq":674,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.803635+0000","last_change":"2024-03-09T14:28:32.308873+0000","last_active":"2024-03-09T15:33:35.803635+0000","last_peered":"2024-03-09T15:33:35.803635+0000","last_clean":"2024-03-09T15:33:35.803635+0000","last_became_active":"2024-03-09T14:28:32.308522+0000","last_became_peered":"2024-03-09T14:28:32.308522+0000","last_unstale":"2024-03-09T15:33:35.803635+0000","last_undegraded":"2024-03-09T15:33:35.803635+0000","last_fullsized":"2024-03-09T15:33:35.803635+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:10:42.961664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6817986,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":153,"num_read_kb":14130,"num_write":524,"num_write_kb":36844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"261'52","reported_seq":603,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.971792+0000","last_change":"2024-03-09T14:28:34.321740+0000","last_active":"2024-03-09T15:33:10.971792+0000","last_peered":"2024-03-09T15:33:10.971792+0000","last_clean":"2024-03-09T15:33:10.971792+0000","last_became_active":"2024-03-09T14:28:34.320646+0000","last_became_peered":"2024-03-09T14:28:34.320646+0000","last_unstale":"2024-03-09T15:33:10.971792+0000","last_undegraded":"2024-03-09T15:33:10.971792+0000","last_fullsized":"2024-03-09T15:33:10.971792+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:32:54.751077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":174,"num_write":79,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"261'157","reported_seq":741,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.897610+0000","last_change":"2024-03-09T14:28:34.307366+0000","last_active":"2024-03-09T15:31:36.897610+0000","last_peered":"2024-03-09T15:31:36.897610+0000","last_clean":"2024-03-09T15:31:36.897610+0000","last_became_active":"2024-03-09T14:28:34.306592+0000","last_became_peered":"2024-03-09T14:28:34.306592+0000","last_unstale":"2024-03-09T15:31:36.897610+0000","last_undegraded":"2024-03-09T15:31:36.897610+0000","last_fullsized":"2024-03-09T15:31:36.897610+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:38:03.684472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":599,"num_read_kb":473,"num_write":232,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"261'147","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.857715+0000","last_change":"2024-03-09T14:29:06.438746+0000","last_active":"2024-03-09T15:36:05.857715+0000","last_peered":"2024-03-09T15:36:05.857715+0000","last_clean":"2024-03-09T15:36:05.857715+0000","last_became_active":"2024-03-09T14:29:06.438523+0000","last_became_peered":"2024-03-09T14:29:06.438523+0000","last_unstale":"2024-03-09T15:36:05.857715+0000","last_undegraded":"2024-03-09T15:36:05.857715+0000","last_fullsized":"2024-03-09T15:36:05.857715+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":100,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:59.554428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728952,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":279,"num_read_kb":1034,"num_write":583,"num_write_kb":27895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.31","version":"261'88","reported_seq":642,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.176622+0000","last_change":"2024-03-09T14:28:34.322326+0000","last_active":"2024-03-09T15:31:36.176622+0000","last_peered":"2024-03-09T15:31:36.176622+0000","last_clean":"2024-03-09T15:31:36.176622+0000","last_became_active":"2024-03-09T14:28:34.321352+0000","last_became_peered":"2024-03-09T14:28:34.321352+0000","last_unstale":"2024-03-09T15:31:36.176622+0000","last_undegraded":"2024-03-09T15:31:36.176622+0000","last_fullsized":"2024-03-09T15:31:36.176622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:00:49.569838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":342,"num_read_kb":270,"num_write":126,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"261'105","reported_seq":646,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.762868+0000","last_change":"2024-03-09T14:28:32.337035+0000","last_active":"2024-03-09T15:31:25.762868+0000","last_peered":"2024-03-09T15:31:25.762868+0000","last_clean":"2024-03-09T15:31:25.762868+0000","last_became_active":"2024-03-09T14:28:32.334208+0000","last_became_peered":"2024-03-09T14:28:32.334208+0000","last_unstale":"2024-03-09T15:31:25.762868+0000","last_undegraded":"2024-03-09T15:31:25.762868+0000","last_fullsized":"2024-03-09T15:31:25.762868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:15:51.011829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262534,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":117,"num_read_kb":128,"num_write":382,"num_write_kb":22112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"261'62","reported_seq":608,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:46.170836+0000","last_change":"2024-03-09T14:28:34.299187+0000","last_active":"2024-03-09T15:31:46.170836+0000","last_peered":"2024-03-09T15:31:46.170836+0000","last_clean":"2024-03-09T15:31:46.170836+0000","last_became_active":"2024-03-09T14:28:34.298428+0000","last_became_peered":"2024-03-09T14:28:34.298428+0000","last_unstale":"2024-03-09T15:31:46.170836+0000","last_undegraded":"2024-03-09T15:31:46.170836+0000","last_fullsized":"2024-03-09T15:31:46.170836+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:07.549211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":202,"num_write":92,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"261'96","reported_seq":651,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:43.752932+0000","last_change":"2024-03-09T14:28:32.333004+0000","last_active":"2024-03-09T15:34:43.752932+0000","last_peered":"2024-03-09T15:34:43.752932+0000","last_clean":"2024-03-09T15:34:43.752932+0000","last_became_active":"2024-03-09T14:28:32.318963+0000","last_became_peered":"2024-03-09T14:28:32.318963+0000","last_unstale":"2024-03-09T15:34:43.752932+0000","last_undegraded":"2024-03-09T15:34:43.752932+0000","last_fullsized":"2024-03-09T15:34:43.752932+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:52:04.706747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208569,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":146,"num_read_kb":1700,"num_write":417,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.33","version":"261'102","reported_seq":663,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.950025+0000","last_change":"2024-03-09T14:28:34.320630+0000","last_active":"2024-03-09T15:31:25.950025+0000","last_peered":"2024-03-09T15:31:25.950025+0000","last_clean":"2024-03-09T15:31:25.950025+0000","last_became_active":"2024-03-09T14:28:34.320331+0000","last_became_peered":"2024-03-09T14:28:34.320331+0000","last_unstale":"2024-03-09T15:31:25.950025+0000","last_undegraded":"2024-03-09T15:31:25.950025+0000","last_fullsized":"2024-03-09T15:31:25.950025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:03:19.130527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":320,"num_write":144,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"261'114","reported_seq":666,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.967812+0000","last_change":"2024-03-09T14:28:32.338458+0000","last_active":"2024-03-09T15:36:05.967812+0000","last_peered":"2024-03-09T15:36:05.967812+0000","last_clean":"2024-03-09T15:36:05.967812+0000","last_became_active":"2024-03-09T14:28:32.337934+0000","last_became_peered":"2024-03-09T14:28:32.337934+0000","last_unstale":"2024-03-09T15:36:05.967812+0000","last_undegraded":"2024-03-09T15:36:05.967812+0000","last_fullsized":"2024-03-09T15:36:05.967812+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:15:21.515518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554899,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":147,"num_read_kb":12763,"num_write":449,"num_write_kb":21372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"261'69","reported_seq":605,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.950065+0000","last_change":"2024-03-09T14:28:34.311072+0000","last_active":"2024-03-09T15:31:25.950065+0000","last_peered":"2024-03-09T15:31:25.950065+0000","last_clean":"2024-03-09T15:31:25.950065+0000","last_became_active":"2024-03-09T14:28:34.310703+0000","last_became_peered":"2024-03-09T14:28:34.310703+0000","last_unstale":"2024-03-09T15:31:25.950065+0000","last_undegraded":"2024-03-09T15:31:25.950065+0000","last_fullsized":"2024-03-09T15:31:25.950065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:16:09.385282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":272,"num_read_kb":220,"num_write":109,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"261'133","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.964740+0000","last_change":"2024-03-09T14:28:32.337031+0000","last_active":"2024-03-09T15:31:41.964740+0000","last_peered":"2024-03-09T15:31:41.964740+0000","last_clean":"2024-03-09T15:31:41.964740+0000","last_became_active":"2024-03-09T14:28:32.336636+0000","last_became_peered":"2024-03-09T14:28:32.336636+0000","last_unstale":"2024-03-09T15:31:41.964740+0000","last_undegraded":"2024-03-09T15:31:41.964740+0000","last_fullsized":"2024-03-09T15:31:41.964740+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:44:26.611692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262536,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":207,"num_read_kb":12490,"num_write":554,"num_write_kb":28462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"261'87","reported_seq":637,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162736+0000","last_change":"2024-03-09T14:28:34.316215+0000","last_active":"2024-03-09T15:31:26.162736+0000","last_peered":"2024-03-09T15:31:26.162736+0000","last_clean":"2024-03-09T15:31:26.162736+0000","last_became_active":"2024-03-09T14:28:34.314665+0000","last_became_peered":"2024-03-09T14:28:34.314665+0000","last_unstale":"2024-03-09T15:31:26.162736+0000","last_undegraded":"2024-03-09T15:31:26.162736+0000","last_fullsized":"2024-03-09T15:31:26.162736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:15.743713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":335,"num_read_kb":265,"num_write":124,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"261'107","reported_seq":668,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.937008+0000","last_change":"2024-03-09T14:28:32.298846+0000","last_active":"2024-03-09T15:36:05.937008+0000","last_peered":"2024-03-09T15:36:05.937008+0000","last_clean":"2024-03-09T15:36:05.937008+0000","last_became_active":"2024-03-09T14:28:32.298428+0000","last_became_peered":"2024-03-09T14:28:32.298428+0000","last_unstale":"2024-03-09T15:36:05.937008+0000","last_undegraded":"2024-03-09T15:36:05.937008+0000","last_fullsized":"2024-03-09T15:36:05.937008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:18:37.829113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583035,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":178,"num_read_kb":31150,"num_write":451,"num_write_kb":37586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"261'76","reported_seq":581,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:07.654968+0000","last_change":"2024-03-09T14:28:42.404373+0000","last_active":"2024-03-09T15:35:07.654968+0000","last_peered":"2024-03-09T15:35:07.654968+0000","last_clean":"2024-03-09T15:35:07.654968+0000","last_became_active":"2024-03-09T14:28:42.403553+0000","last_became_peered":"2024-03-09T14:28:42.403553+0000","last_unstale":"2024-03-09T15:35:07.654968+0000","last_undegraded":"2024-03-09T15:35:07.654968+0000","last_fullsized":"2024-03-09T15:35:07.654968+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:16:41.740933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":84,"num_write":184,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"261'98","reported_seq":664,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.780525+0000","last_change":"2024-03-09T14:29:06.439713+0000","last_active":"2024-03-09T15:31:50.780525+0000","last_peered":"2024-03-09T15:31:50.780525+0000","last_clean":"2024-03-09T15:31:50.780525+0000","last_became_active":"2024-03-09T14:29:06.439359+0000","last_became_peered":"2024-03-09T14:29:06.439359+0000","last_unstale":"2024-03-09T15:31:50.780525+0000","last_undegraded":"2024-03-09T15:31:50.780525+0000","last_fullsized":"2024-03-09T15:31:50.780525+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:01:04.305529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":413,"num_read_kb":333,"num_write":149,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"261'154","reported_seq":775,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.211631+0000","last_change":"2024-03-09T14:28:32.330235+0000","last_active":"2024-03-09T15:36:06.211631+0000","last_peered":"2024-03-09T15:36:06.211631+0000","last_clean":"2024-03-09T15:36:06.211631+0000","last_became_active":"2024-03-09T14:28:32.329457+0000","last_became_peered":"2024-03-09T14:28:32.329457+0000","last_unstale":"2024-03-09T15:36:06.211631+0000","last_undegraded":"2024-03-09T15:36:06.211631+0000","last_fullsized":"2024-03-09T15:36:06.211631+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":100,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:30:37.270692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5813235,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":252,"num_read_kb":141328,"num_write":592,"num_write_kb":37022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.166374+0000","last_change":"2024-03-09T14:28:36.321332+0000","last_active":"2024-03-09T15:31:26.166374+0000","last_peered":"2024-03-09T15:31:26.166374+0000","last_clean":"2024-03-09T15:31:26.166374+0000","last_became_active":"2024-03-09T14:28:36.320603+0000","last_became_peered":"2024-03-09T14:28:36.320603+0000","last_unstale":"2024-03-09T15:31:26.166374+0000","last_undegraded":"2024-03-09T15:31:26.166374+0000","last_fullsized":"2024-03-09T15:31:26.166374+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:20:34.317764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"261'226","reported_seq":832,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.691237+0000","last_change":"2024-03-09T14:28:38.327507+0000","last_active":"2024-03-09T15:35:58.691237+0000","last_peered":"2024-03-09T15:35:58.691237+0000","last_clean":"2024-03-09T15:35:58.691237+0000","last_became_active":"2024-03-09T14:28:38.326826+0000","last_became_peered":"2024-03-09T14:28:38.326826+0000","last_unstale":"2024-03-09T15:35:58.691237+0000","last_undegraded":"2024-03-09T15:35:58.691237+0000","last_fullsized":"2024-03-09T15:35:58.691237+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":200,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:46:18.261500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24948,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":356,"num_read_kb":580,"num_write":246,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.175558+0000","last_change":"2024-03-09T14:28:40.359830+0000","last_active":"2024-03-09T15:31:31.175558+0000","last_peered":"2024-03-09T15:31:31.175558+0000","last_clean":"2024-03-09T15:31:31.175558+0000","last_became_active":"2024-03-09T14:28:40.358954+0000","last_became_peered":"2024-03-09T14:28:40.358954+0000","last_unstale":"2024-03-09T15:31:31.175558+0000","last_undegraded":"2024-03-09T15:31:31.175558+0000","last_fullsized":"2024-03-09T15:31:31.175558+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:34.169828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"261'770","reported_seq":2383,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:53.987578+0000","last_change":"2024-03-09T14:28:42.395382+0000","last_active":"2024-03-09T15:35:53.987578+0000","last_peered":"2024-03-09T15:35:53.987578+0000","last_clean":"2024-03-09T15:35:53.987578+0000","last_became_active":"2024-03-09T14:28:42.394691+0000","last_became_peered":"2024-03-09T14:28:42.394691+0000","last_unstale":"2024-03-09T15:35:53.987578+0000","last_undegraded":"2024-03-09T15:35:53.987578+0000","last_fullsized":"2024-03-09T15:35:53.987578+0000","mapping_epoch":30,"log_start":"261'700","ondisk_log_start":"261'700","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":700,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:48.486613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":612,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2611,"num_read_kb":2601,"num_write":1560,"num_write_kb":569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"261'40","reported_seq":578,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:28.764187+0000","last_change":"2024-03-09T14:28:34.326083+0000","last_active":"2024-03-09T15:31:28.764187+0000","last_peered":"2024-03-09T15:31:28.764187+0000","last_clean":"2024-03-09T15:31:28.764187+0000","last_became_active":"2024-03-09T14:28:34.324569+0000","last_became_peered":"2024-03-09T14:28:34.324569+0000","last_unstale":"2024-03-09T15:31:28.764187+0000","last_undegraded":"2024-03-09T15:31:28.764187+0000","last_fullsized":"2024-03-09T15:31:28.764187+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:18.214769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":168,"num_read_kb":136,"num_write":58,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"261'145","reported_seq":715,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.866230+0000","last_change":"2024-03-09T14:28:32.310039+0000","last_active":"2024-03-09T15:36:05.866230+0000","last_peered":"2024-03-09T15:36:05.866230+0000","last_clean":"2024-03-09T15:36:05.866230+0000","last_became_active":"2024-03-09T14:28:32.309536+0000","last_became_peered":"2024-03-09T14:28:32.309536+0000","last_unstale":"2024-03-09T15:36:05.866230+0000","last_undegraded":"2024-03-09T15:36:05.866230+0000","last_fullsized":"2024-03-09T15:36:05.866230+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:28:43.638731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5530768,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":194,"num_read_kb":29533,"num_write":628,"num_write_kb":30801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.780570+0000","last_change":"2024-03-09T14:28:36.320450+0000","last_active":"2024-03-09T15:31:50.780570+0000","last_peered":"2024-03-09T15:31:50.780570+0000","last_clean":"2024-03-09T15:31:50.780570+0000","last_became_active":"2024-03-09T14:28:36.320079+0000","last_became_peered":"2024-03-09T14:28:36.320079+0000","last_unstale":"2024-03-09T15:31:50.780570+0000","last_undegraded":"2024-03-09T15:31:50.780570+0000","last_fullsized":"2024-03-09T15:31:50.780570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:19:38.894047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"261'87","reported_seq":623,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:52.371528+0000","last_change":"2024-03-09T14:28:38.341864+0000","last_active":"2024-03-09T15:31:52.371528+0000","last_peered":"2024-03-09T15:31:52.371528+0000","last_clean":"2024-03-09T15:31:52.371528+0000","last_became_active":"2024-03-09T14:28:38.340896+0000","last_became_peered":"2024-03-09T14:28:38.340896+0000","last_unstale":"2024-03-09T15:31:52.371528+0000","last_undegraded":"2024-03-09T15:31:52.371528+0000","last_fullsized":"2024-03-09T15:31:52.371528+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:19:17.825977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":101,"num_write":68,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.163891+0000","last_change":"2024-03-09T14:28:40.371660+0000","last_active":"2024-03-09T15:31:31.163891+0000","last_peered":"2024-03-09T15:31:31.163891+0000","last_clean":"2024-03-09T15:31:31.163891+0000","last_became_active":"2024-03-09T14:28:40.370788+0000","last_became_peered":"2024-03-09T14:28:40.370788+0000","last_unstale":"2024-03-09T15:31:31.163891+0000","last_undegraded":"2024-03-09T15:31:31.163891+0000","last_fullsized":"2024-03-09T15:31:31.163891+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:12:43.937289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"261'59","reported_seq":561,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.285043+0000","last_change":"2024-03-09T14:28:42.411392+0000","last_active":"2024-03-09T15:33:50.285043+0000","last_peered":"2024-03-09T15:33:50.285043+0000","last_clean":"2024-03-09T15:33:50.285043+0000","last_became_active":"2024-03-09T14:28:42.408738+0000","last_became_peered":"2024-03-09T14:28:42.408738+0000","last_unstale":"2024-03-09T15:33:50.285043+0000","last_undegraded":"2024-03-09T15:33:50.285043+0000","last_fullsized":"2024-03-09T15:33:50.285043+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:05:13.992640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":681,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":76,"num_read_kb":75,"num_write":140,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"261'157","reported_seq":745,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:31.430581+0000","last_change":"2024-03-09T14:28:34.305220+0000","last_active":"2024-03-09T15:33:31.430581+0000","last_peered":"2024-03-09T15:33:31.430581+0000","last_clean":"2024-03-09T15:33:31.430581+0000","last_became_active":"2024-03-09T14:28:34.304892+0000","last_became_peered":"2024-03-09T14:28:34.304892+0000","last_unstale":"2024-03-09T15:33:31.430581+0000","last_undegraded":"2024-03-09T15:33:31.430581+0000","last_fullsized":"2024-03-09T15:33:31.430581+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:10:30.883636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":622,"num_read_kb":492,"num_write":239,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"261'148","reported_seq":848,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.211631+0000","last_change":"2024-03-09T14:28:32.319206+0000","last_active":"2024-03-09T15:36:06.211631+0000","last_peered":"2024-03-09T15:36:06.211631+0000","last_clean":"2024-03-09T15:36:06.211631+0000","last_became_active":"2024-03-09T14:28:32.318428+0000","last_became_peered":"2024-03-09T14:28:32.318428+0000","last_unstale":"2024-03-09T15:36:06.211631+0000","last_undegraded":"2024-03-09T15:36:06.211631+0000","last_fullsized":"2024-03-09T15:36:06.211631+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":100,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:59:15.432941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6019791,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":531,"num_read_kb":113871,"num_write":606,"num_write_kb":46622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.774459+0000","last_change":"2024-03-09T14:28:36.320826+0000","last_active":"2024-03-09T15:31:25.774459+0000","last_peered":"2024-03-09T15:31:25.774459+0000","last_clean":"2024-03-09T15:31:25.774459+0000","last_became_active":"2024-03-09T14:28:36.320095+0000","last_became_peered":"2024-03-09T14:28:36.320095+0000","last_unstale":"2024-03-09T15:31:25.774459+0000","last_undegraded":"2024-03-09T15:31:25.774459+0000","last_fullsized":"2024-03-09T15:31:25.774459+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:58:26.878574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"261'103","reported_seq":653,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:02.768040+0000","last_change":"2024-03-09T14:28:38.339553+0000","last_active":"2024-03-09T15:34:02.768040+0000","last_peered":"2024-03-09T15:34:02.768040+0000","last_clean":"2024-03-09T15:34:02.768040+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:34:02.768040+0000","last_undegraded":"2024-03-09T15:34:02.768040+0000","last_fullsized":"2024-03-09T15:34:02.768040+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:37:31.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"261'504","reported_seq":2290,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.019957+0000","last_change":"2024-03-09T14:28:40.373218+0000","last_active":"2024-03-09T15:36:06.019957+0000","last_peered":"2024-03-09T15:36:06.019957+0000","last_clean":"2024-03-09T15:36:06.019957+0000","last_became_active":"2024-03-09T14:28:40.371507+0000","last_became_peered":"2024-03-09T14:28:40.371507+0000","last_unstale":"2024-03-09T15:36:06.019957+0000","last_undegraded":"2024-03-09T15:36:06.019957+0000","last_fullsized":"2024-03-09T15:36:06.019957+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":400,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:20:18.095183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"261'136","reported_seq":941,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:16.160267+0000","last_change":"2024-03-09T14:28:42.399490+0000","last_active":"2024-03-09T15:31:16.160267+0000","last_peered":"2024-03-09T15:31:16.160267+0000","last_clean":"2024-03-09T15:31:16.160267+0000","last_became_active":"2024-03-09T14:28:42.398498+0000","last_became_peered":"2024-03-09T14:28:42.398498+0000","last_unstale":"2024-03-09T15:31:16.160267+0000","last_undegraded":"2024-03-09T15:31:16.160267+0000","last_fullsized":"2024-03-09T15:31:16.160267+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:10:05.573339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":498,"num_read_kb":498,"num_write":277,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"261'53","reported_seq":592,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.774411+0000","last_change":"2024-03-09T14:28:34.330792+0000","last_active":"2024-03-09T15:31:25.774411+0000","last_peered":"2024-03-09T15:31:25.774411+0000","last_clean":"2024-03-09T15:31:25.774411+0000","last_became_active":"2024-03-09T14:28:34.330347+0000","last_became_peered":"2024-03-09T14:28:34.330347+0000","last_unstale":"2024-03-09T15:31:25.774411+0000","last_undegraded":"2024-03-09T15:31:25.774411+0000","last_fullsized":"2024-03-09T15:31:25.774411+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:55:20.226431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":212,"num_read_kb":171,"num_write":77,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"261'136","reported_seq":708,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.764851+0000","last_change":"2024-03-09T14:28:32.332979+0000","last_active":"2024-03-09T15:31:35.764851+0000","last_peered":"2024-03-09T15:31:35.764851+0000","last_clean":"2024-03-09T15:31:35.764851+0000","last_became_active":"2024-03-09T14:28:32.319819+0000","last_became_peered":"2024-03-09T14:28:32.319819+0000","last_unstale":"2024-03-09T15:31:35.764851+0000","last_undegraded":"2024-03-09T15:31:35.764851+0000","last_fullsized":"2024-03-09T15:31:35.764851+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:01:46.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5244864,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":201,"num_read_kb":15567,"num_write":549,"num_write_kb":34195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.166450+0000","last_change":"2024-03-09T14:28:36.311535+0000","last_active":"2024-03-09T15:31:26.166450+0000","last_peered":"2024-03-09T15:31:26.166450+0000","last_clean":"2024-03-09T15:31:26.166450+0000","last_became_active":"2024-03-09T14:28:36.310231+0000","last_became_peered":"2024-03-09T14:28:36.310231+0000","last_unstale":"2024-03-09T15:31:26.166450+0000","last_undegraded":"2024-03-09T15:31:26.166450+0000","last_fullsized":"2024-03-09T15:31:26.166450+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:55:50.175612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"261'167","reported_seq":746,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.679604+0000","last_change":"2024-03-09T14:28:38.599078+0000","last_active":"2024-03-09T15:35:58.679604+0000","last_peered":"2024-03-09T15:35:58.679604+0000","last_clean":"2024-03-09T15:35:58.679604+0000","last_became_active":"2024-03-09T14:28:38.598308+0000","last_became_peered":"2024-03-09T14:28:38.598308+0000","last_unstale":"2024-03-09T15:35:58.679604+0000","last_undegraded":"2024-03-09T15:35:58.679604+0000","last_fullsized":"2024-03-09T15:35:58.679604+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":100,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:10:09.151032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23129,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":269,"num_read_kb":489,"num_write":186,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"261'512","reported_seq":2498,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.733534+0000","last_change":"2024-03-09T14:28:40.388710+0000","last_active":"2024-03-09T15:36:05.733534+0000","last_peered":"2024-03-09T15:36:05.733534+0000","last_clean":"2024-03-09T15:36:05.733534+0000","last_became_active":"2024-03-09T14:28:40.387990+0000","last_became_peered":"2024-03-09T14:28:40.387990+0000","last_unstale":"2024-03-09T15:36:05.733534+0000","last_undegraded":"2024-03-09T15:36:05.733534+0000","last_fullsized":"2024-03-09T15:36:05.733534+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":500,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:21:47.017030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"261'101","reported_seq":678,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194321+0000","last_change":"2024-03-09T14:28:34.323491+0000","last_active":"2024-03-09T15:33:36.194321+0000","last_peered":"2024-03-09T15:33:36.194321+0000","last_clean":"2024-03-09T15:33:36.194321+0000","last_became_active":"2024-03-09T14:28:34.322731+0000","last_became_peered":"2024-03-09T14:28:34.322731+0000","last_unstale":"2024-03-09T15:33:36.194321+0000","last_undegraded":"2024-03-09T15:33:36.194321+0000","last_fullsized":"2024-03-09T15:33:36.194321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:34:36.142132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":330,"num_write":141,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"261'114","reported_seq":662,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.217253+0000","last_change":"2024-03-09T14:28:32.329755+0000","last_active":"2024-03-09T15:36:06.217253+0000","last_peered":"2024-03-09T15:36:06.217253+0000","last_clean":"2024-03-09T15:36:06.217253+0000","last_became_active":"2024-03-09T14:28:32.329148+0000","last_became_peered":"2024-03-09T14:28:32.329148+0000","last_unstale":"2024-03-09T15:36:06.217253+0000","last_undegraded":"2024-03-09T15:36:06.217253+0000","last_fullsized":"2024-03-09T15:36:06.217253+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:07:39.356330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7949644,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":138,"num_read_kb":1766,"num_write":504,"num_write_kb":18289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205246+0000","last_change":"2024-03-09T14:28:36.314780+0000","last_active":"2024-03-09T15:33:36.205246+0000","last_peered":"2024-03-09T15:33:36.205246+0000","last_clean":"2024-03-09T15:33:36.205246+0000","last_became_active":"2024-03-09T14:28:36.313904+0000","last_became_peered":"2024-03-09T14:28:36.313904+0000","last_unstale":"2024-03-09T15:33:36.205246+0000","last_undegraded":"2024-03-09T15:33:36.205246+0000","last_fullsized":"2024-03-09T15:33:36.205246+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:57:41.431102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"261'265","reported_seq":896,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:01.226737+0000","last_change":"2024-03-09T14:28:38.598064+0000","last_active":"2024-03-09T15:36:01.226737+0000","last_peered":"2024-03-09T15:36:01.226737+0000","last_clean":"2024-03-09T15:36:01.226737+0000","last_became_active":"2024-03-09T14:28:38.597247+0000","last_became_peered":"2024-03-09T14:28:38.597247+0000","last_unstale":"2024-03-09T15:36:01.226737+0000","last_undegraded":"2024-03-09T15:36:01.226737+0000","last_fullsized":"2024-03-09T15:36:01.226737+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":200,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:35:30.643121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22355,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":412,"num_read_kb":625,"num_write":280,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":495,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.134883+0000","last_change":"2024-03-09T14:28:40.360962+0000","last_active":"2024-03-09T15:31:41.134883+0000","last_peered":"2024-03-09T15:31:41.134883+0000","last_clean":"2024-03-09T15:31:41.134883+0000","last_became_active":"2024-03-09T14:28:40.359906+0000","last_became_peered":"2024-03-09T14:28:40.359906+0000","last_unstale":"2024-03-09T15:31:41.134883+0000","last_undegraded":"2024-03-09T15:31:41.134883+0000","last_fullsized":"2024-03-09T15:31:41.134883+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:33:27.073677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"261'518","reported_seq":1356,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.729973+0000","last_change":"2024-03-09T14:28:42.403475+0000","last_active":"2024-03-09T15:35:58.729973+0000","last_peered":"2024-03-09T15:35:58.729973+0000","last_clean":"2024-03-09T15:35:58.729973+0000","last_became_active":"2024-03-09T14:28:42.402739+0000","last_became_peered":"2024-03-09T14:28:42.402739+0000","last_unstale":"2024-03-09T15:35:58.729973+0000","last_undegraded":"2024-03-09T15:35:58.729973+0000","last_fullsized":"2024-03-09T15:35:58.729973+0000","mapping_epoch":30,"log_start":"261'500","ondisk_log_start":"261'500","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":500,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:49:57.794992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1336,"num_read_kb":1311,"num_write":1083,"num_write_kb":376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"261'92","reported_seq":662,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191097+0000","last_change":"2024-03-09T14:28:34.302153+0000","last_active":"2024-03-09T15:33:36.191097+0000","last_peered":"2024-03-09T15:33:36.191097+0000","last_clean":"2024-03-09T15:33:36.191097+0000","last_became_active":"2024-03-09T14:28:34.300795+0000","last_became_peered":"2024-03-09T14:28:34.300795+0000","last_unstale":"2024-03-09T15:33:36.191097+0000","last_undegraded":"2024-03-09T15:33:36.191097+0000","last_fullsized":"2024-03-09T15:33:36.191097+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:27:31.948784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":361,"num_read_kb":288,"num_write":150,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"261'133","reported_seq":706,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.234898+0000","last_change":"2024-03-09T14:28:32.318853+0000","last_active":"2024-03-09T15:36:06.234898+0000","last_peered":"2024-03-09T15:36:06.234898+0000","last_clean":"2024-03-09T15:36:06.234898+0000","last_became_active":"2024-03-09T14:28:32.318111+0000","last_became_peered":"2024-03-09T14:28:32.318111+0000","last_unstale":"2024-03-09T15:36:06.234898+0000","last_undegraded":"2024-03-09T15:36:06.234898+0000","last_fullsized":"2024-03-09T15:36:06.234898+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:51.120639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5549179,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":206,"num_read_kb":6230,"num_write":588,"num_write_kb":19779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.976641+0000","last_change":"2024-03-09T14:28:36.301940+0000","last_active":"2024-03-09T15:33:35.976641+0000","last_peered":"2024-03-09T15:33:35.976641+0000","last_clean":"2024-03-09T15:33:35.976641+0000","last_became_active":"2024-03-09T14:28:36.301227+0000","last_became_peered":"2024-03-09T14:28:36.301227+0000","last_unstale":"2024-03-09T15:33:35.976641+0000","last_undegraded":"2024-03-09T15:33:35.976641+0000","last_fullsized":"2024-03-09T15:33:35.976641+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:55:41.843139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"261'170","reported_seq":742,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.654250+0000","last_change":"2024-03-09T14:28:38.347678+0000","last_active":"2024-03-09T15:35:58.654250+0000","last_peered":"2024-03-09T15:35:58.654250+0000","last_clean":"2024-03-09T15:35:58.654250+0000","last_became_active":"2024-03-09T14:28:38.346837+0000","last_became_peered":"2024-03-09T14:28:38.346837+0000","last_unstale":"2024-03-09T15:35:58.654250+0000","last_undegraded":"2024-03-09T15:35:58.654250+0000","last_fullsized":"2024-03-09T15:35:58.654250+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":100,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:06.776774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28902,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":267,"num_read_kb":489,"num_write":184,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.808565+0000","last_change":"2024-03-09T14:28:40.391285+0000","last_active":"2024-03-09T15:33:55.808565+0000","last_peered":"2024-03-09T15:33:55.808565+0000","last_clean":"2024-03-09T15:33:55.808565+0000","last_became_active":"2024-03-09T14:28:40.389175+0000","last_became_peered":"2024-03-09T14:28:40.389175+0000","last_unstale":"2024-03-09T15:33:55.808565+0000","last_undegraded":"2024-03-09T15:33:55.808565+0000","last_fullsized":"2024-03-09T15:33:55.808565+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:33:43.955165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"261'58","reported_seq":553,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:02.730159+0000","last_change":"2024-03-09T14:28:42.401990+0000","last_active":"2024-03-09T15:34:02.730159+0000","last_peered":"2024-03-09T15:34:02.730159+0000","last_clean":"2024-03-09T15:34:02.730159+0000","last_became_active":"2024-03-09T14:28:42.401214+0000","last_became_peered":"2024-03-09T14:28:42.401214+0000","last_unstale":"2024-03-09T15:34:02.730159+0000","last_undegraded":"2024-03-09T15:34:02.730159+0000","last_fullsized":"2024-03-09T15:34:02.730159+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:24:20.023880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":47,"num_write":140,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"261'78","reported_seq":634,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.793908+0000","last_change":"2024-03-09T14:28:34.317931+0000","last_active":"2024-03-09T15:33:55.793908+0000","last_peered":"2024-03-09T15:33:55.793908+0000","last_clean":"2024-03-09T15:33:55.793908+0000","last_became_active":"2024-03-09T14:28:34.316812+0000","last_became_peered":"2024-03-09T14:28:34.316812+0000","last_unstale":"2024-03-09T15:33:55.793908+0000","last_undegraded":"2024-03-09T15:33:55.793908+0000","last_fullsized":"2024-03-09T15:33:55.793908+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:23:12.743671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":320,"num_read_kb":256,"num_write":116,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"261'100","reported_seq":663,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.219206+0000","last_change":"2024-03-09T14:29:06.446958+0000","last_active":"2024-03-09T15:36:06.219206+0000","last_peered":"2024-03-09T15:36:06.219206+0000","last_clean":"2024-03-09T15:36:06.219206+0000","last_became_active":"2024-03-09T14:29:06.446333+0000","last_became_peered":"2024-03-09T14:29:06.446333+0000","last_unstale":"2024-03-09T15:36:06.219206+0000","last_undegraded":"2024-03-09T15:36:06.219206+0000","last_fullsized":"2024-03-09T15:36:06.219206+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:09:44.978410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":139,"num_read_kb":1025,"num_write":394,"num_write_kb":11167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.980194+0000","last_change":"2024-03-09T14:28:36.304298+0000","last_active":"2024-03-09T15:33:50.980194+0000","last_peered":"2024-03-09T15:33:50.980194+0000","last_clean":"2024-03-09T15:33:50.980194+0000","last_became_active":"2024-03-09T14:28:36.303695+0000","last_became_peered":"2024-03-09T14:28:36.303695+0000","last_unstale":"2024-03-09T15:33:50.980194+0000","last_undegraded":"2024-03-09T15:33:50.980194+0000","last_fullsized":"2024-03-09T15:33:50.980194+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:16:14.481520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"261'217","reported_seq":821,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.705946+0000","last_change":"2024-03-09T14:28:38.340119+0000","last_active":"2024-03-09T15:35:58.705946+0000","last_peered":"2024-03-09T15:35:58.705946+0000","last_clean":"2024-03-09T15:35:58.705946+0000","last_became_active":"2024-03-09T14:28:38.339300+0000","last_became_peered":"2024-03-09T14:28:38.339300+0000","last_unstale":"2024-03-09T15:35:58.705946+0000","last_undegraded":"2024-03-09T15:35:58.705946+0000","last_fullsized":"2024-03-09T15:35:58.705946+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":200,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:56:14.245930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24501,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":344,"num_read_kb":579,"num_write":240,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"261'500","reported_seq":2509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.736377+0000","last_change":"2024-03-09T14:28:40.368533+0000","last_active":"2024-03-09T15:36:05.736377+0000","last_peered":"2024-03-09T15:36:05.736377+0000","last_clean":"2024-03-09T15:36:05.736377+0000","last_became_active":"2024-03-09T14:28:40.367119+0000","last_became_peered":"2024-03-09T14:28:40.367119+0000","last_unstale":"2024-03-09T15:36:05.736377+0000","last_undegraded":"2024-03-09T15:36:05.736377+0000","last_fullsized":"2024-03-09T15:36:05.736377+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":400,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:01.659994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"261'81","reported_seq":575,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:31.861227+0000","last_change":"2024-03-09T14:28:42.397381+0000","last_active":"2024-03-09T15:34:31.861227+0000","last_peered":"2024-03-09T15:34:31.861227+0000","last_clean":"2024-03-09T15:34:31.861227+0000","last_became_active":"2024-03-09T14:28:42.396389+0000","last_became_peered":"2024-03-09T14:28:42.396389+0000","last_unstale":"2024-03-09T15:34:31.861227+0000","last_undegraded":"2024-03-09T15:34:31.861227+0000","last_fullsized":"2024-03-09T15:34:31.861227+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:00:18.807761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":65,"num_write":200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"261'44","reported_seq":582,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.016397+0000","last_change":"2024-03-09T14:28:34.298294+0000","last_active":"2024-03-09T15:33:51.016397+0000","last_peered":"2024-03-09T15:33:51.016397+0000","last_clean":"2024-03-09T15:33:51.016397+0000","last_became_active":"2024-03-09T14:28:34.297080+0000","last_became_peered":"2024-03-09T14:28:34.297080+0000","last_unstale":"2024-03-09T15:33:51.016397+0000","last_undegraded":"2024-03-09T15:33:51.016397+0000","last_fullsized":"2024-03-09T15:33:51.016397+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:32.179794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":189,"num_read_kb":155,"num_write":63,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"261'128","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.989852+0000","last_change":"2024-03-09T14:29:06.445970+0000","last_active":"2024-03-09T15:36:05.989852+0000","last_peered":"2024-03-09T15:36:05.989852+0000","last_clean":"2024-03-09T15:36:05.989852+0000","last_became_active":"2024-03-09T14:29:06.445573+0000","last_became_peered":"2024-03-09T14:29:06.445573+0000","last_unstale":"2024-03-09T15:36:05.989852+0000","last_undegraded":"2024-03-09T15:36:05.989852+0000","last_fullsized":"2024-03-09T15:36:05.989852+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:30:46.074402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":185,"num_read_kb":157,"num_write":439,"num_write_kb":32761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.980070+0000","last_change":"2024-03-09T14:28:36.323155+0000","last_active":"2024-03-09T15:33:50.980070+0000","last_peered":"2024-03-09T15:33:50.980070+0000","last_clean":"2024-03-09T15:33:50.980070+0000","last_became_active":"2024-03-09T14:28:36.322638+0000","last_became_peered":"2024-03-09T14:28:36.322638+0000","last_unstale":"2024-03-09T15:33:50.980070+0000","last_undegraded":"2024-03-09T15:33:50.980070+0000","last_fullsized":"2024-03-09T15:33:50.980070+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:45:20.380783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"261'312","reported_seq":947,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.765276+0000","last_change":"2024-03-09T14:28:38.329540+0000","last_active":"2024-03-09T15:36:05.765276+0000","last_peered":"2024-03-09T15:36:05.765276+0000","last_clean":"2024-03-09T15:36:05.765276+0000","last_became_active":"2024-03-09T14:28:38.328897+0000","last_became_peered":"2024-03-09T14:28:38.328897+0000","last_unstale":"2024-03-09T15:36:05.765276+0000","last_undegraded":"2024-03-09T15:36:05.765276+0000","last_fullsized":"2024-03-09T15:36:05.765276+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":300,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:11:28.889923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60622,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":525,"num_read_kb":1064,"num_write":380,"num_write_kb":275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.977668+0000","last_change":"2024-03-09T14:28:40.384571+0000","last_active":"2024-03-09T15:33:40.977668+0000","last_peered":"2024-03-09T15:33:40.977668+0000","last_clean":"2024-03-09T15:33:40.977668+0000","last_became_active":"2024-03-09T14:28:40.383758+0000","last_became_peered":"2024-03-09T14:28:40.383758+0000","last_unstale":"2024-03-09T15:33:40.977668+0000","last_undegraded":"2024-03-09T15:33:40.977668+0000","last_fullsized":"2024-03-09T15:33:40.977668+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:24:41.089698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"261'54","reported_seq":565,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.965658+0000","last_change":"2024-03-09T14:28:42.402684+0000","last_active":"2024-03-09T15:33:45.965658+0000","last_peered":"2024-03-09T15:33:45.965658+0000","last_clean":"2024-03-09T15:33:45.965658+0000","last_became_active":"2024-03-09T14:28:42.401880+0000","last_became_peered":"2024-03-09T14:28:42.401880+0000","last_unstale":"2024-03-09T15:33:45.965658+0000","last_undegraded":"2024-03-09T15:33:45.965658+0000","last_fullsized":"2024-03-09T15:33:45.965658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:29:59.644550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":69,"num_write":120,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"261'79","reported_seq":639,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195621+0000","last_change":"2024-03-09T14:28:34.316156+0000","last_active":"2024-03-09T15:33:51.195621+0000","last_peered":"2024-03-09T15:33:51.195621+0000","last_clean":"2024-03-09T15:33:51.195621+0000","last_became_active":"2024-03-09T14:28:34.315432+0000","last_became_peered":"2024-03-09T14:28:34.315432+0000","last_unstale":"2024-03-09T15:33:51.195621+0000","last_undegraded":"2024-03-09T15:33:51.195621+0000","last_fullsized":"2024-03-09T15:33:51.195621+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:08:27.256304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":315,"num_read_kb":251,"num_write":126,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"261'126","reported_seq":664,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.807364+0000","last_change":"2024-03-09T14:28:32.300326+0000","last_active":"2024-03-09T15:33:50.807364+0000","last_peered":"2024-03-09T15:33:50.807364+0000","last_clean":"2024-03-09T15:33:50.807364+0000","last_became_active":"2024-03-09T14:28:32.299897+0000","last_became_peered":"2024-03-09T14:28:32.299897+0000","last_unstale":"2024-03-09T15:33:50.807364+0000","last_undegraded":"2024-03-09T15:33:50.807364+0000","last_fullsized":"2024-03-09T15:33:50.807364+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":100,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:25:33.818560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5546754,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":129,"num_read_kb":225,"num_write":522,"num_write_kb":26448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.980312+0000","last_change":"2024-03-09T14:28:36.320815+0000","last_active":"2024-03-09T15:33:50.980312+0000","last_peered":"2024-03-09T15:33:50.980312+0000","last_clean":"2024-03-09T15:33:50.980312+0000","last_became_active":"2024-03-09T14:28:36.320029+0000","last_became_peered":"2024-03-09T14:28:36.320029+0000","last_unstale":"2024-03-09T15:33:50.980312+0000","last_undegraded":"2024-03-09T15:33:50.980312+0000","last_fullsized":"2024-03-09T15:33:50.980312+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:23:44.855859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"261'205","reported_seq":792,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.664412+0000","last_change":"2024-03-09T14:28:38.339497+0000","last_active":"2024-03-09T15:35:58.664412+0000","last_peered":"2024-03-09T15:35:58.664412+0000","last_clean":"2024-03-09T15:35:58.664412+0000","last_became_active":"2024-03-09T14:28:38.338658+0000","last_became_peered":"2024-03-09T14:28:38.338658+0000","last_unstale":"2024-03-09T15:35:58.664412+0000","last_undegraded":"2024-03-09T15:35:58.664412+0000","last_fullsized":"2024-03-09T15:35:58.664412+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":100,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:42:50.615803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25762,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":343,"num_read_kb":574,"num_write":288,"num_write_kb":187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:55.989515+0000","last_change":"2024-03-09T14:28:40.379699+0000","last_active":"2024-03-09T15:31:55.989515+0000","last_peered":"2024-03-09T15:31:55.989515+0000","last_clean":"2024-03-09T15:31:55.989515+0000","last_became_active":"2024-03-09T14:28:40.378973+0000","last_became_peered":"2024-03-09T14:28:40.378973+0000","last_unstale":"2024-03-09T15:31:55.989515+0000","last_undegraded":"2024-03-09T15:31:55.989515+0000","last_fullsized":"2024-03-09T15:31:55.989515+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:53.400714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"261'42","reported_seq":535,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:00.822318+0000","last_change":"2024-03-09T14:28:42.398431+0000","last_active":"2024-03-09T15:35:00.822318+0000","last_peered":"2024-03-09T15:35:00.822318+0000","last_clean":"2024-03-09T15:35:00.822318+0000","last_became_active":"2024-03-09T14:28:42.397533+0000","last_became_peered":"2024-03-09T14:28:42.397533+0000","last_unstale":"2024-03-09T15:35:00.822318+0000","last_undegraded":"2024-03-09T15:35:00.822318+0000","last_fullsized":"2024-03-09T15:35:00.822318+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:54:11.336171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":91,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"261'56","reported_seq":616,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195764+0000","last_change":"2024-03-09T14:28:34.315224+0000","last_active":"2024-03-09T15:33:51.195764+0000","last_peered":"2024-03-09T15:33:51.195764+0000","last_clean":"2024-03-09T15:33:51.195764+0000","last_became_active":"2024-03-09T14:28:34.313475+0000","last_became_peered":"2024-03-09T14:28:34.313475+0000","last_unstale":"2024-03-09T15:33:51.195764+0000","last_undegraded":"2024-03-09T15:33:51.195764+0000","last_fullsized":"2024-03-09T15:33:51.195764+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:37:57.784425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":198,"num_write":78,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"261'105","reported_seq":657,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:40.924777+0000","last_change":"2024-03-09T14:28:32.319640+0000","last_active":"2024-03-09T15:35:40.924777+0000","last_peered":"2024-03-09T15:35:40.924777+0000","last_clean":"2024-03-09T15:35:40.924777+0000","last_became_active":"2024-03-09T14:28:32.316406+0000","last_became_peered":"2024-03-09T14:28:32.316406+0000","last_unstale":"2024-03-09T15:35:40.924777+0000","last_undegraded":"2024-03-09T15:35:40.924777+0000","last_fullsized":"2024-03-09T15:35:40.924777+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:33:19.144976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7947601,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":138,"num_read_kb":15736,"num_write":442,"num_write_kb":13578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.966668+0000","last_change":"2024-03-09T14:28:36.319712+0000","last_active":"2024-03-09T15:33:50.966668+0000","last_peered":"2024-03-09T15:33:50.966668+0000","last_clean":"2024-03-09T15:33:50.966668+0000","last_became_active":"2024-03-09T14:28:36.319210+0000","last_became_peered":"2024-03-09T14:28:36.319210+0000","last_unstale":"2024-03-09T15:33:50.966668+0000","last_undegraded":"2024-03-09T15:33:50.966668+0000","last_fullsized":"2024-03-09T15:33:50.966668+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:26:56.771418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"261'217","reported_seq":821,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.969548+0000","last_change":"2024-03-09T14:28:38.351282+0000","last_active":"2024-03-09T15:36:05.969548+0000","last_peered":"2024-03-09T15:36:05.969548+0000","last_clean":"2024-03-09T15:36:05.969548+0000","last_became_active":"2024-03-09T14:28:38.350390+0000","last_became_peered":"2024-03-09T14:28:38.350390+0000","last_unstale":"2024-03-09T15:36:05.969548+0000","last_undegraded":"2024-03-09T15:36:05.969548+0000","last_fullsized":"2024-03-09T15:36:05.969548+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":200,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:30:58.004755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9540,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":338,"num_read_kb":580,"num_write":230,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"261'473","reported_seq":2357,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.722396+0000","last_change":"2024-03-09T14:28:40.374487+0000","last_active":"2024-03-09T15:36:05.722396+0000","last_peered":"2024-03-09T15:36:05.722396+0000","last_clean":"2024-03-09T15:36:05.722396+0000","last_became_active":"2024-03-09T14:28:40.373360+0000","last_became_peered":"2024-03-09T14:28:40.373360+0000","last_unstale":"2024-03-09T15:36:05.722396+0000","last_undegraded":"2024-03-09T15:36:05.722396+0000","last_fullsized":"2024-03-09T15:36:05.722396+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":400,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:24:48.737477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"261'104","reported_seq":650,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:52.932955+0000","last_change":"2024-03-09T14:28:42.409080+0000","last_active":"2024-03-09T15:34:52.932955+0000","last_peered":"2024-03-09T15:34:52.932955+0000","last_clean":"2024-03-09T15:34:52.932955+0000","last_became_active":"2024-03-09T14:28:42.408233+0000","last_became_peered":"2024-03-09T14:28:42.408233+0000","last_unstale":"2024-03-09T15:34:52.932955+0000","last_undegraded":"2024-03-09T15:34:52.932955+0000","last_fullsized":"2024-03-09T15:34:52.932955+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:59:27.171699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":274,"num_read_kb":227,"num_write":264,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"261'72","reported_seq":621,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195874+0000","last_change":"2024-03-09T14:28:34.310932+0000","last_active":"2024-03-09T15:33:51.195874+0000","last_peered":"2024-03-09T15:33:51.195874+0000","last_clean":"2024-03-09T15:33:51.195874+0000","last_became_active":"2024-03-09T14:28:34.310061+0000","last_became_peered":"2024-03-09T14:28:34.310061+0000","last_unstale":"2024-03-09T15:33:51.195874+0000","last_undegraded":"2024-03-09T15:33:51.195874+0000","last_fullsized":"2024-03-09T15:33:51.195874+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:56:23.692037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":294,"num_read_kb":233,"num_write":114,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"261'141","reported_seq":748,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.928584+0000","last_change":"2024-03-09T14:28:32.313964+0000","last_active":"2024-03-09T15:36:05.928584+0000","last_peered":"2024-03-09T15:36:05.928584+0000","last_clean":"2024-03-09T15:36:05.928584+0000","last_became_active":"2024-03-09T14:28:32.313566+0000","last_became_peered":"2024-03-09T14:28:32.313566+0000","last_unstale":"2024-03-09T15:36:05.928584+0000","last_undegraded":"2024-03-09T15:36:05.928584+0000","last_fullsized":"2024-03-09T15:36:05.928584+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":100,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:06:35.181059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":569877,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":220,"num_read_kb":84880,"num_write":542,"num_write_kb":19718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"32'7","reported_seq":1459,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.269380+0000","last_change":"2024-03-09T14:28:36.327203+0000","last_active":"2024-03-09T15:36:05.269380+0000","last_peered":"2024-03-09T15:36:05.269380+0000","last_clean":"2024-03-09T15:36:05.269380+0000","last_became_active":"2024-03-09T14:28:36.325820+0000","last_became_peered":"2024-03-09T14:28:36.325820+0000","last_unstale":"2024-03-09T15:36:05.269380+0000","last_undegraded":"2024-03-09T15:36:05.269380+0000","last_fullsized":"2024-03-09T15:36:05.269380+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:03:03.441773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1195,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1118,"num_read_kb":1421,"num_write":14,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"261'86","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:52.540471+0000","last_change":"2024-03-09T14:28:38.596269+0000","last_active":"2024-03-09T15:32:52.540471+0000","last_peered":"2024-03-09T15:32:52.540471+0000","last_clean":"2024-03-09T15:32:52.540471+0000","last_became_active":"2024-03-09T14:28:38.595478+0000","last_became_peered":"2024-03-09T14:28:38.595478+0000","last_unstale":"2024-03-09T15:32:52.540471+0000","last_undegraded":"2024-03-09T15:32:52.540471+0000","last_fullsized":"2024-03-09T15:32:52.540471+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:43:55.683017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":105,"num_read_kb":100,"num_write":70,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"261'488","reported_seq":2293,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.725131+0000","last_change":"2024-03-09T14:28:40.375535+0000","last_active":"2024-03-09T15:36:05.725131+0000","last_peered":"2024-03-09T15:36:05.725131+0000","last_clean":"2024-03-09T15:36:05.725131+0000","last_became_active":"2024-03-09T14:28:40.374408+0000","last_became_peered":"2024-03-09T14:28:40.374408+0000","last_unstale":"2024-03-09T15:36:05.725131+0000","last_undegraded":"2024-03-09T15:36:05.725131+0000","last_fullsized":"2024-03-09T15:36:05.725131+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":400,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:00:18.508302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"261'93","reported_seq":596,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:39.863116+0000","last_change":"2024-03-09T14:28:42.412396+0000","last_active":"2024-03-09T15:34:39.863116+0000","last_peered":"2024-03-09T15:34:39.863116+0000","last_clean":"2024-03-09T15:34:39.863116+0000","last_became_active":"2024-03-09T14:28:42.411523+0000","last_became_peered":"2024-03-09T14:28:42.411523+0000","last_unstale":"2024-03-09T15:34:39.863116+0000","last_undegraded":"2024-03-09T15:34:39.863116+0000","last_fullsized":"2024-03-09T15:34:39.863116+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:23:50.431623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":72,"num_write":221,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"261'42","reported_seq":595,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.134950+0000","last_change":"2024-03-09T14:28:34.317632+0000","last_active":"2024-03-09T15:31:41.134950+0000","last_peered":"2024-03-09T15:31:41.134950+0000","last_clean":"2024-03-09T15:31:41.134950+0000","last_became_active":"2024-03-09T14:28:34.317226+0000","last_became_peered":"2024-03-09T14:28:34.317226+0000","last_unstale":"2024-03-09T15:31:41.134950+0000","last_undegraded":"2024-03-09T15:31:41.134950+0000","last_fullsized":"2024-03-09T15:31:41.134950+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:48:25.758532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":188,"num_read_kb":158,"num_write":62,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"261'93","reported_seq":629,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.803584+0000","last_change":"2024-03-09T14:28:32.295693+0000","last_active":"2024-03-09T15:33:35.803584+0000","last_peered":"2024-03-09T15:33:35.803584+0000","last_clean":"2024-03-09T15:33:35.803584+0000","last_became_active":"2024-03-09T14:28:32.295179+0000","last_became_peered":"2024-03-09T14:28:32.295179+0000","last_unstale":"2024-03-09T15:33:35.803584+0000","last_undegraded":"2024-03-09T15:33:35.803584+0000","last_fullsized":"2024-03-09T15:33:35.803584+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:55:35.164395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262386,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":120,"num_read_kb":124,"num_write":367,"num_write_kb":5920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195837+0000","last_change":"2024-03-09T14:28:36.319885+0000","last_active":"2024-03-09T15:33:51.195837+0000","last_peered":"2024-03-09T15:33:51.195837+0000","last_clean":"2024-03-09T15:33:51.195837+0000","last_became_active":"2024-03-09T14:28:36.319102+0000","last_became_peered":"2024-03-09T14:28:36.319102+0000","last_unstale":"2024-03-09T15:33:51.195837+0000","last_undegraded":"2024-03-09T15:33:51.195837+0000","last_fullsized":"2024-03-09T15:33:51.195837+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:44:50.268293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"261'311","reported_seq":934,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.971810+0000","last_change":"2024-03-09T14:28:38.338131+0000","last_active":"2024-03-09T15:35:58.971810+0000","last_peered":"2024-03-09T15:35:58.971810+0000","last_clean":"2024-03-09T15:35:58.971810+0000","last_became_active":"2024-03-09T14:28:38.337254+0000","last_became_peered":"2024-03-09T14:28:38.337254+0000","last_unstale":"2024-03-09T15:35:58.971810+0000","last_undegraded":"2024-03-09T15:35:58.971810+0000","last_fullsized":"2024-03-09T15:35:58.971810+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":300,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:49.247631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29175,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":664,"num_read_kb":922,"num_write":784,"num_write_kb":628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.766105+0000","last_change":"2024-03-09T14:28:40.378550+0000","last_active":"2024-03-09T15:31:40.766105+0000","last_peered":"2024-03-09T15:31:40.766105+0000","last_clean":"2024-03-09T15:31:40.766105+0000","last_became_active":"2024-03-09T14:28:40.377714+0000","last_became_peered":"2024-03-09T14:28:40.377714+0000","last_unstale":"2024-03-09T15:31:40.766105+0000","last_undegraded":"2024-03-09T15:31:40.766105+0000","last_fullsized":"2024-03-09T15:31:40.766105+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:21:50.389100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"261'68","reported_seq":567,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.959943+0000","last_change":"2024-03-09T14:28:42.402550+0000","last_active":"2024-03-09T15:33:20.959943+0000","last_peered":"2024-03-09T15:33:20.959943+0000","last_clean":"2024-03-09T15:33:20.959943+0000","last_became_active":"2024-03-09T14:28:42.401775+0000","last_became_peered":"2024-03-09T14:28:42.401775+0000","last_unstale":"2024-03-09T15:33:20.959943+0000","last_undegraded":"2024-03-09T15:33:20.959943+0000","last_fullsized":"2024-03-09T15:33:20.959943+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:47:20.741043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":77,"num_write":153,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"261'174","reported_seq":825,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:30.788820+0000","last_change":"2024-03-09T14:28:34.327727+0000","last_active":"2024-03-09T15:33:30.788820+0000","last_peered":"2024-03-09T15:33:30.788820+0000","last_clean":"2024-03-09T15:33:30.788820+0000","last_became_active":"2024-03-09T14:28:34.326919+0000","last_became_peered":"2024-03-09T14:28:34.326919+0000","last_unstale":"2024-03-09T15:33:30.788820+0000","last_undegraded":"2024-03-09T15:33:30.788820+0000","last_fullsized":"2024-03-09T15:33:30.788820+0000","mapping_epoch":22,"log_start":"34'100","ondisk_log_start":"34'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:50:06.490347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":914,"num_read_kb":819,"num_write":300,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":681,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.189283+0000","last_change":"2024-03-09T14:28:12.265518+0000","last_active":"2024-03-09T15:32:26.189283+0000","last_peered":"2024-03-09T15:32:26.189283+0000","last_clean":"2024-03-09T15:32:26.189283+0000","last_became_active":"2024-03-09T14:28:12.262430+0000","last_became_peered":"2024-03-09T14:28:12.262430+0000","last_unstale":"2024-03-09T15:32:26.189283+0000","last_undegraded":"2024-03-09T15:32:26.189283+0000","last_fullsized":"2024-03-09T15:32:26.189283+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_clean_scrub_stamp":"2024-03-09T14:28:11.179640+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:14:17.888044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.185851+0000","last_change":"2024-03-09T14:28:19.233336+0000","last_active":"2024-03-09T15:32:56.185851+0000","last_peered":"2024-03-09T15:32:56.185851+0000","last_clean":"2024-03-09T15:32:56.185851+0000","last_became_active":"2024-03-09T14:28:19.232517+0000","last_became_peered":"2024-03-09T14:28:19.232517+0000","last_unstale":"2024-03-09T15:32:56.185851+0000","last_undegraded":"2024-03-09T15:32:56.185851+0000","last_fullsized":"2024-03-09T15:32:56.185851+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:53:39.430154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"261'111","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:39.045175+0000","last_change":"2024-03-09T14:28:32.300845+0000","last_active":"2024-03-09T15:34:39.045175+0000","last_peered":"2024-03-09T15:34:39.045175+0000","last_clean":"2024-03-09T15:34:39.045175+0000","last_became_active":"2024-03-09T14:28:32.300468+0000","last_became_peered":"2024-03-09T14:28:32.300468+0000","last_unstale":"2024-03-09T15:34:39.045175+0000","last_undegraded":"2024-03-09T15:34:39.045175+0000","last_fullsized":"2024-03-09T15:34:39.045175+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:58:14.894309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3439332,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":154,"num_read_kb":6450,"num_write":438,"num_write_kb":21101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.199546+0000","last_change":"2024-03-09T14:28:36.318953+0000","last_active":"2024-03-09T15:33:11.199546+0000","last_peered":"2024-03-09T15:33:11.199546+0000","last_clean":"2024-03-09T15:33:11.199546+0000","last_became_active":"2024-03-09T14:28:36.318172+0000","last_became_peered":"2024-03-09T14:28:36.318172+0000","last_unstale":"2024-03-09T15:33:11.199546+0000","last_undegraded":"2024-03-09T15:33:11.199546+0000","last_fullsized":"2024-03-09T15:33:11.199546+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:14:08.117277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"261'493","reported_seq":1200,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.910451+0000","last_change":"2024-03-09T14:28:38.349794+0000","last_active":"2024-03-09T15:36:05.910451+0000","last_peered":"2024-03-09T15:36:05.910451+0000","last_clean":"2024-03-09T15:36:05.910451+0000","last_became_active":"2024-03-09T14:28:38.348941+0000","last_became_peered":"2024-03-09T14:28:38.348941+0000","last_unstale":"2024-03-09T15:36:05.910451+0000","last_undegraded":"2024-03-09T15:36:05.910451+0000","last_fullsized":"2024-03-09T15:36:05.910451+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":400,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:10:34.907672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74232,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":969,"num_read_kb":1810,"num_write":926,"num_write_kb":765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:41.014076+0000","last_change":"2024-03-09T14:28:40.364392+0000","last_active":"2024-03-09T15:33:41.014076+0000","last_peered":"2024-03-09T15:33:41.014076+0000","last_clean":"2024-03-09T15:33:41.014076+0000","last_became_active":"2024-03-09T14:28:40.362781+0000","last_became_peered":"2024-03-09T14:28:40.362781+0000","last_unstale":"2024-03-09T15:33:41.014076+0000","last_undegraded":"2024-03-09T15:33:41.014076+0000","last_fullsized":"2024-03-09T15:33:41.014076+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:30:02.543875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"261'118","reported_seq":817,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:45.170047+0000","last_change":"2024-03-09T14:28:42.399378+0000","last_active":"2024-03-09T15:34:45.170047+0000","last_peered":"2024-03-09T15:34:45.170047+0000","last_clean":"2024-03-09T15:34:45.170047+0000","last_became_active":"2024-03-09T14:28:42.398626+0000","last_became_peered":"2024-03-09T14:28:42.398626+0000","last_unstale":"2024-03-09T15:34:45.170047+0000","last_undegraded":"2024-03-09T15:34:45.170047+0000","last_fullsized":"2024-03-09T15:34:45.170047+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:40:25.365667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":356,"num_read_kb":343,"num_write":215,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"261'90","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.012674+0000","last_change":"2024-03-09T14:28:34.297905+0000","last_active":"2024-03-09T15:33:36.012674+0000","last_peered":"2024-03-09T15:33:36.012674+0000","last_clean":"2024-03-09T15:33:36.012674+0000","last_became_active":"2024-03-09T14:28:34.297472+0000","last_became_peered":"2024-03-09T14:28:34.297472+0000","last_unstale":"2024-03-09T15:33:36.012674+0000","last_undegraded":"2024-03-09T15:33:36.012674+0000","last_fullsized":"2024-03-09T15:33:36.012674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:04:18.955082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":356,"num_read_kb":292,"num_write":141,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":525,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:31.190342+0000","last_change":"2024-03-09T14:28:19.242176+0000","last_active":"2024-03-09T15:32:31.190342+0000","last_peered":"2024-03-09T15:32:31.190342+0000","last_clean":"2024-03-09T15:32:31.190342+0000","last_became_active":"2024-03-09T14:28:19.241110+0000","last_became_peered":"2024-03-09T14:28:19.241110+0000","last_unstale":"2024-03-09T15:32:31.190342+0000","last_undegraded":"2024-03-09T15:32:31.190342+0000","last_fullsized":"2024-03-09T15:32:31.190342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:28:52.238193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"261'122","reported_seq":694,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.955245+0000","last_change":"2024-03-09T14:28:32.312953+0000","last_active":"2024-03-09T15:36:05.955245+0000","last_peered":"2024-03-09T15:36:05.955245+0000","last_clean":"2024-03-09T15:36:05.955245+0000","last_became_active":"2024-03-09T14:28:32.312536+0000","last_became_peered":"2024-03-09T14:28:32.312536+0000","last_unstale":"2024-03-09T15:36:05.955245+0000","last_undegraded":"2024-03-09T15:36:05.955245+0000","last_fullsized":"2024-03-09T15:36:05.955245+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:53:44.400275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3383,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":203,"num_read_kb":12489,"num_write":509,"num_write_kb":17953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194706+0000","last_change":"2024-03-09T14:28:36.307721+0000","last_active":"2024-03-09T15:33:36.194706+0000","last_peered":"2024-03-09T15:33:36.194706+0000","last_clean":"2024-03-09T15:33:36.194706+0000","last_became_active":"2024-03-09T14:28:36.306912+0000","last_became_peered":"2024-03-09T14:28:36.306912+0000","last_unstale":"2024-03-09T15:33:36.194706+0000","last_undegraded":"2024-03-09T15:33:36.194706+0000","last_fullsized":"2024-03-09T15:33:36.194706+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:19:47.054193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"261'837","reported_seq":2502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.790375+0000","last_change":"2024-03-09T14:28:38.329490+0000","last_active":"2024-03-09T15:36:05.790375+0000","last_peered":"2024-03-09T15:36:05.790375+0000","last_clean":"2024-03-09T15:36:05.790375+0000","last_became_active":"2024-03-09T14:28:38.328738+0000","last_became_peered":"2024-03-09T14:28:38.328738+0000","last_unstale":"2024-03-09T15:36:05.790375+0000","last_undegraded":"2024-03-09T15:36:05.790375+0000","last_fullsized":"2024-03-09T15:36:05.790375+0000","mapping_epoch":26,"log_start":"261'800","ondisk_log_start":"261'800","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":800,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:24:55.016529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77478,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1296,"num_read_kb":2021,"num_write":650,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.202990+0000","last_change":"2024-03-09T14:28:40.387247+0000","last_active":"2024-03-09T15:33:26.202990+0000","last_peered":"2024-03-09T15:33:26.202990+0000","last_clean":"2024-03-09T15:33:26.202990+0000","last_became_active":"2024-03-09T14:28:40.386451+0000","last_became_peered":"2024-03-09T14:28:40.386451+0000","last_unstale":"2024-03-09T15:33:26.202990+0000","last_undegraded":"2024-03-09T15:33:26.202990+0000","last_fullsized":"2024-03-09T15:33:26.202990+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:06:23.810341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"261'124","reported_seq":656,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.967916+0000","last_change":"2024-03-09T14:28:42.403919+0000","last_active":"2024-03-09T15:33:55.967916+0000","last_peered":"2024-03-09T15:33:55.967916+0000","last_clean":"2024-03-09T15:33:55.967916+0000","last_became_active":"2024-03-09T14:28:42.403013+0000","last_became_peered":"2024-03-09T14:28:42.403013+0000","last_unstale":"2024-03-09T15:33:55.967916+0000","last_undegraded":"2024-03-09T15:33:55.967916+0000","last_fullsized":"2024-03-09T15:33:55.967916+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:36:18.083589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1248,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":261,"num_read_kb":241,"num_write":300,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"261'48","reported_seq":601,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963559+0000","last_change":"2024-03-09T14:28:34.297150+0000","last_active":"2024-03-09T15:33:35.963559+0000","last_peered":"2024-03-09T15:33:35.963559+0000","last_clean":"2024-03-09T15:33:35.963559+0000","last_became_active":"2024-03-09T14:28:34.296733+0000","last_became_peered":"2024-03-09T14:28:34.296733+0000","last_unstale":"2024-03-09T15:33:35.963559+0000","last_undegraded":"2024-03-09T15:33:35.963559+0000","last_fullsized":"2024-03-09T15:33:35.963559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:29:16.007058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":200,"num_read_kb":166,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.968712+0000","last_change":"2024-03-09T14:28:19.437446+0000","last_active":"2024-03-09T15:32:55.968712+0000","last_peered":"2024-03-09T15:32:55.968712+0000","last_clean":"2024-03-09T15:32:55.968712+0000","last_became_active":"2024-03-09T14:28:19.436985+0000","last_became_peered":"2024-03-09T14:28:19.436985+0000","last_unstale":"2024-03-09T15:32:55.968712+0000","last_undegraded":"2024-03-09T15:32:55.968712+0000","last_fullsized":"2024-03-09T15:32:55.968712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:46:38.528697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"261'134","reported_seq":791,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.893599+0000","last_change":"2024-03-09T14:28:32.303463+0000","last_active":"2024-03-09T15:36:05.893599+0000","last_peered":"2024-03-09T15:36:05.893599+0000","last_clean":"2024-03-09T15:36:05.893599+0000","last_became_active":"2024-03-09T14:28:32.303176+0000","last_became_peered":"2024-03-09T14:28:32.303176+0000","last_unstale":"2024-03-09T15:36:05.893599+0000","last_undegraded":"2024-03-09T15:36:05.893599+0000","last_fullsized":"2024-03-09T15:36:05.893599+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:28:51.916628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8554935,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":468,"num_read_kb":3616,"num_write":574,"num_write_kb":29269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"25'1","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.976814+0000","last_change":"2024-03-09T14:28:36.306679+0000","last_active":"2024-03-09T15:33:35.976814+0000","last_peered":"2024-03-09T15:33:35.976814+0000","last_clean":"2024-03-09T15:33:35.976814+0000","last_became_active":"2024-03-09T14:28:36.305971+0000","last_became_peered":"2024-03-09T14:28:36.305971+0000","last_unstale":"2024-03-09T15:33:35.976814+0000","last_undegraded":"2024-03-09T15:33:35.976814+0000","last_fullsized":"2024-03-09T15:33:35.976814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:21:21.395131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"261'230","reported_seq":1598,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.934522+0000","last_change":"2024-03-09T14:28:38.598201+0000","last_active":"2024-03-09T15:35:58.934522+0000","last_peered":"2024-03-09T15:35:58.934522+0000","last_clean":"2024-03-09T15:35:58.934522+0000","last_became_active":"2024-03-09T14:28:38.597403+0000","last_became_peered":"2024-03-09T14:28:38.597403+0000","last_unstale":"2024-03-09T15:35:58.934522+0000","last_undegraded":"2024-03-09T15:35:58.934522+0000","last_fullsized":"2024-03-09T15:35:58.934522+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":200,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:44:05.046714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":226,"num_read_kb":216,"num_write":195,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.007838+0000","last_change":"2024-03-09T14:28:40.372884+0000","last_active":"2024-03-09T15:33:16.007838+0000","last_peered":"2024-03-09T15:33:16.007838+0000","last_clean":"2024-03-09T15:33:16.007838+0000","last_became_active":"2024-03-09T14:28:40.371383+0000","last_became_peered":"2024-03-09T14:28:40.371383+0000","last_unstale":"2024-03-09T15:33:16.007838+0000","last_undegraded":"2024-03-09T15:33:16.007838+0000","last_fullsized":"2024-03-09T15:33:16.007838+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:01:15.189556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"261'201","reported_seq":791,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:19.528609+0000","last_change":"2024-03-09T14:28:42.396620+0000","last_active":"2024-03-09T15:34:19.528609+0000","last_peered":"2024-03-09T15:34:19.528609+0000","last_clean":"2024-03-09T15:34:19.528609+0000","last_became_active":"2024-03-09T14:28:42.395781+0000","last_became_peered":"2024-03-09T14:28:42.395781+0000","last_unstale":"2024-03-09T15:34:19.528609+0000","last_undegraded":"2024-03-09T15:34:19.528609+0000","last_fullsized":"2024-03-09T15:34:19.528609+0000","mapping_epoch":30,"log_start":"247'100","ondisk_log_start":"247'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":100,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:38:14.404007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":435,"num_read_kb":456,"num_write":425,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"261'74","reported_seq":618,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:31.876523+0000","last_change":"2024-03-09T14:28:34.313660+0000","last_active":"2024-03-09T15:34:31.876523+0000","last_peered":"2024-03-09T15:34:31.876523+0000","last_clean":"2024-03-09T15:34:31.876523+0000","last_became_active":"2024-03-09T14:28:34.313302+0000","last_became_peered":"2024-03-09T14:28:34.313302+0000","last_unstale":"2024-03-09T15:34:31.876523+0000","last_undegraded":"2024-03-09T15:34:31.876523+0000","last_fullsized":"2024-03-09T15:34:31.876523+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:32:32.206071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":225,"num_write":104,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":522,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.185786+0000","last_change":"2024-03-09T14:28:19.238720+0000","last_active":"2024-03-09T15:32:56.185786+0000","last_peered":"2024-03-09T15:32:56.185786+0000","last_clean":"2024-03-09T15:32:56.185786+0000","last_became_active":"2024-03-09T14:28:19.237937+0000","last_became_peered":"2024-03-09T14:28:19.237937+0000","last_unstale":"2024-03-09T15:32:56.185786+0000","last_undegraded":"2024-03-09T15:32:56.185786+0000","last_fullsized":"2024-03-09T15:32:56.185786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:29:01.306246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"261'107","reported_seq":682,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.961490+0000","last_change":"2024-03-09T14:28:32.312937+0000","last_active":"2024-03-09T15:36:05.961490+0000","last_peered":"2024-03-09T15:36:05.961490+0000","last_clean":"2024-03-09T15:36:05.961490+0000","last_became_active":"2024-03-09T14:28:32.303963+0000","last_became_peered":"2024-03-09T14:28:32.303963+0000","last_unstale":"2024-03-09T15:36:05.961490+0000","last_undegraded":"2024-03-09T15:36:05.961490+0000","last_fullsized":"2024-03-09T15:36:05.961490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:54:34.862486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6966901,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":155,"num_read_kb":15115,"num_write":410,"num_write_kb":32085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.010297+0000","last_change":"2024-03-09T14:28:36.318128+0000","last_active":"2024-03-09T15:33:26.010297+0000","last_peered":"2024-03-09T15:33:26.010297+0000","last_clean":"2024-03-09T15:33:26.010297+0000","last_became_active":"2024-03-09T14:28:36.317586+0000","last_became_peered":"2024-03-09T14:28:36.317586+0000","last_unstale":"2024-03-09T15:33:26.010297+0000","last_undegraded":"2024-03-09T15:33:26.010297+0000","last_fullsized":"2024-03-09T15:33:26.010297+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:46:55.791256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"261'129","reported_seq":682,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:52.970721+0000","last_change":"2024-03-09T14:28:38.335628+0000","last_active":"2024-03-09T15:34:52.970721+0000","last_peered":"2024-03-09T15:34:52.970721+0000","last_clean":"2024-03-09T15:34:52.970721+0000","last_became_active":"2024-03-09T14:28:38.334463+0000","last_became_peered":"2024-03-09T14:28:38.334463+0000","last_unstale":"2024-03-09T15:34:52.970721+0000","last_undegraded":"2024-03-09T15:34:52.970721+0000","last_fullsized":"2024-03-09T15:34:52.970721+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:56:33.829971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":178,"num_read_kb":172,"num_write":132,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.790948+0000","last_change":"2024-03-09T14:28:40.371173+0000","last_active":"2024-03-09T15:33:40.790948+0000","last_peered":"2024-03-09T15:33:40.790948+0000","last_clean":"2024-03-09T15:33:40.790948+0000","last_became_active":"2024-03-09T14:28:40.370436+0000","last_became_peered":"2024-03-09T14:28:40.370436+0000","last_unstale":"2024-03-09T15:33:40.790948+0000","last_undegraded":"2024-03-09T15:33:40.790948+0000","last_fullsized":"2024-03-09T15:33:40.790948+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:23:11.745507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"261'47","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:48.549510+0000","last_change":"2024-03-09T14:28:42.398975+0000","last_active":"2024-03-09T15:33:48.549510+0000","last_peered":"2024-03-09T15:33:48.549510+0000","last_clean":"2024-03-09T15:33:48.549510+0000","last_became_active":"2024-03-09T14:28:42.397058+0000","last_became_peered":"2024-03-09T14:28:42.397058+0000","last_unstale":"2024-03-09T15:33:48.549510+0000","last_undegraded":"2024-03-09T15:33:48.549510+0000","last_fullsized":"2024-03-09T15:33:48.549510+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:02:27.312655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":100,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"261'43","reported_seq":593,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.957260+0000","last_change":"2024-03-09T14:28:34.297850+0000","last_active":"2024-03-09T15:33:10.957260+0000","last_peered":"2024-03-09T15:33:10.957260+0000","last_clean":"2024-03-09T15:33:10.957260+0000","last_became_active":"2024-03-09T14:28:34.296953+0000","last_became_peered":"2024-03-09T14:28:34.296953+0000","last_unstale":"2024-03-09T15:33:10.957260+0000","last_undegraded":"2024-03-09T15:33:10.957260+0000","last_fullsized":"2024-03-09T15:33:10.957260+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:37:50.390768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":162,"num_write":75,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"261'151","reported_seq":785,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.160202+0000","last_change":"2024-03-09T14:28:32.303697+0000","last_active":"2024-03-09T15:36:06.160202+0000","last_peered":"2024-03-09T15:36:06.160202+0000","last_clean":"2024-03-09T15:36:06.160202+0000","last_became_active":"2024-03-09T14:28:32.302970+0000","last_became_peered":"2024-03-09T14:28:32.302970+0000","last_unstale":"2024-03-09T15:36:06.160202+0000","last_undegraded":"2024-03-09T15:36:06.160202+0000","last_fullsized":"2024-03-09T15:36:06.160202+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":100,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:33:55.197166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655969,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":294,"num_read_kb":94521,"num_write":620,"num_write_kb":45805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":520,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.953596+0000","last_change":"2024-03-09T14:28:19.236670+0000","last_active":"2024-03-09T15:32:55.953596+0000","last_peered":"2024-03-09T15:32:55.953596+0000","last_clean":"2024-03-09T15:32:55.953596+0000","last_became_active":"2024-03-09T14:28:19.236240+0000","last_became_peered":"2024-03-09T14:28:19.236240+0000","last_unstale":"2024-03-09T15:32:55.953596+0000","last_undegraded":"2024-03-09T15:32:55.953596+0000","last_fullsized":"2024-03-09T15:32:55.953596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:00:09.869968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.006552+0000","last_change":"2024-03-09T14:28:36.315028+0000","last_active":"2024-03-09T15:33:11.006552+0000","last_peered":"2024-03-09T15:33:11.006552+0000","last_clean":"2024-03-09T15:33:11.006552+0000","last_became_active":"2024-03-09T14:28:36.314526+0000","last_became_peered":"2024-03-09T14:28:36.314526+0000","last_unstale":"2024-03-09T15:33:11.006552+0000","last_undegraded":"2024-03-09T15:33:11.006552+0000","last_fullsized":"2024-03-09T15:33:11.006552+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:24:49.871997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"261'381","reported_seq":1059,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.947780+0000","last_change":"2024-03-09T14:28:38.335922+0000","last_active":"2024-03-09T15:36:05.947780+0000","last_peered":"2024-03-09T15:36:05.947780+0000","last_clean":"2024-03-09T15:36:05.947780+0000","last_became_active":"2024-03-09T14:28:38.334976+0000","last_became_peered":"2024-03-09T14:28:38.334976+0000","last_unstale":"2024-03-09T15:36:05.947780+0000","last_undegraded":"2024-03-09T15:36:05.947780+0000","last_fullsized":"2024-03-09T15:36:05.947780+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":300,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:27:57.990498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34012,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":631,"num_read_kb":1165,"num_write":452,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.186765+0000","last_change":"2024-03-09T14:28:40.379714+0000","last_active":"2024-03-09T15:33:16.186765+0000","last_peered":"2024-03-09T15:33:16.186765+0000","last_clean":"2024-03-09T15:33:16.186765+0000","last_became_active":"2024-03-09T14:28:40.379108+0000","last_became_peered":"2024-03-09T14:28:40.379108+0000","last_unstale":"2024-03-09T15:33:16.186765+0000","last_undegraded":"2024-03-09T15:33:16.186765+0000","last_fullsized":"2024-03-09T15:33:16.186765+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:39:21.726823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"261'101","reported_seq":626,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:41.594325+0000","last_change":"2024-03-09T14:28:42.406107+0000","last_active":"2024-03-09T15:34:41.594325+0000","last_peered":"2024-03-09T15:34:41.594325+0000","last_clean":"2024-03-09T15:34:41.594325+0000","last_became_active":"2024-03-09T14:28:42.405277+0000","last_became_peered":"2024-03-09T14:28:42.405277+0000","last_unstale":"2024-03-09T15:34:41.594325+0000","last_undegraded":"2024-03-09T15:34:41.594325+0000","last_fullsized":"2024-03-09T15:34:41.594325+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:53:29.449584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":717,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185,"num_read_kb":160,"num_write":261,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"261'43","reported_seq":575,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.006582+0000","last_change":"2024-03-09T14:28:34.298018+0000","last_active":"2024-03-09T15:33:11.006582+0000","last_peered":"2024-03-09T15:33:11.006582+0000","last_clean":"2024-03-09T15:33:11.006582+0000","last_became_active":"2024-03-09T14:28:34.296660+0000","last_became_peered":"2024-03-09T14:28:34.296660+0000","last_unstale":"2024-03-09T15:33:11.006582+0000","last_undegraded":"2024-03-09T15:33:11.006582+0000","last_fullsized":"2024-03-09T15:33:11.006582+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:37:00.270467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":129,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"261'128","reported_seq":758,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.920215+0000","last_change":"2024-03-09T14:28:32.305873+0000","last_active":"2024-03-09T15:36:05.920215+0000","last_peered":"2024-03-09T15:36:05.920215+0000","last_clean":"2024-03-09T15:36:05.920215+0000","last_became_active":"2024-03-09T14:28:32.305318+0000","last_became_peered":"2024-03-09T14:28:32.305318+0000","last_unstale":"2024-03-09T15:36:05.920215+0000","last_undegraded":"2024-03-09T15:36:05.920215+0000","last_fullsized":"2024-03-09T15:36:05.920215+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:30:36.881076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3908854,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":337,"num_read_kb":10578,"num_write":436,"num_write_kb":20505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:06.198444+0000","last_change":"2024-03-09T14:28:19.241475+0000","last_active":"2024-03-09T15:33:06.198444+0000","last_peered":"2024-03-09T15:33:06.198444+0000","last_clean":"2024-03-09T15:33:06.198444+0000","last_became_active":"2024-03-09T14:28:19.240704+0000","last_became_peered":"2024-03-09T14:28:19.240704+0000","last_unstale":"2024-03-09T15:33:06.198444+0000","last_undegraded":"2024-03-09T15:33:06.198444+0000","last_fullsized":"2024-03-09T15:33:06.198444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:07:08.993017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784406+0000","last_change":"2024-03-09T14:28:36.315481+0000","last_active":"2024-03-09T15:33:10.784406+0000","last_peered":"2024-03-09T15:33:10.784406+0000","last_clean":"2024-03-09T15:33:10.784406+0000","last_became_active":"2024-03-09T14:28:36.314676+0000","last_became_peered":"2024-03-09T14:28:36.314676+0000","last_unstale":"2024-03-09T15:33:10.784406+0000","last_undegraded":"2024-03-09T15:33:10.784406+0000","last_fullsized":"2024-03-09T15:33:10.784406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:14:11.938307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"261'197","reported_seq":778,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:20.985108+0000","last_change":"2024-03-09T14:28:38.595995+0000","last_active":"2024-03-09T15:35:20.985108+0000","last_peered":"2024-03-09T15:35:20.985108+0000","last_clean":"2024-03-09T15:35:20.985108+0000","last_became_active":"2024-03-09T14:28:38.595028+0000","last_became_peered":"2024-03-09T14:28:38.595028+0000","last_unstale":"2024-03-09T15:35:20.985108+0000","last_undegraded":"2024-03-09T15:35:20.985108+0000","last_fullsized":"2024-03-09T15:35:20.985108+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":100,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:24:00.362746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":269,"num_read_kb":262,"num_write":230,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"261'500","reported_seq":2296,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.716276+0000","last_change":"2024-03-09T14:28:40.370167+0000","last_active":"2024-03-09T15:36:05.716276+0000","last_peered":"2024-03-09T15:36:05.716276+0000","last_clean":"2024-03-09T15:36:05.716276+0000","last_became_active":"2024-03-09T14:28:40.369165+0000","last_became_peered":"2024-03-09T14:28:40.369165+0000","last_unstale":"2024-03-09T15:36:05.716276+0000","last_undegraded":"2024-03-09T15:36:05.716276+0000","last_fullsized":"2024-03-09T15:36:05.716276+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":400,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:29:22.485672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"261'76","reported_seq":564,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:15.812925+0000","last_change":"2024-03-09T14:28:42.393153+0000","last_active":"2024-03-09T15:34:15.812925+0000","last_peered":"2024-03-09T15:34:15.812925+0000","last_clean":"2024-03-09T15:34:15.812925+0000","last_became_active":"2024-03-09T14:28:42.392402+0000","last_became_peered":"2024-03-09T14:28:42.392402+0000","last_unstale":"2024-03-09T15:34:15.812925+0000","last_undegraded":"2024-03-09T15:34:15.812925+0000","last_fullsized":"2024-03-09T15:34:15.812925+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:57:45.818538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":57,"num_write":172,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"261'48","reported_seq":589,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.973653+0000","last_change":"2024-03-09T14:29:06.440753+0000","last_active":"2024-03-09T15:33:20.973653+0000","last_peered":"2024-03-09T15:33:20.973653+0000","last_clean":"2024-03-09T15:33:20.973653+0000","last_became_active":"2024-03-09T14:29:06.440556+0000","last_became_peered":"2024-03-09T14:29:06.440556+0000","last_unstale":"2024-03-09T15:33:20.973653+0000","last_undegraded":"2024-03-09T15:33:20.973653+0000","last_fullsized":"2024-03-09T15:33:20.973653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:43:05.084646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":216,"num_read_kb":182,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"261'120","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.803883+0000","last_change":"2024-03-09T14:28:32.319314+0000","last_active":"2024-03-09T15:33:35.803883+0000","last_peered":"2024-03-09T15:33:35.803883+0000","last_clean":"2024-03-09T15:33:35.803883+0000","last_became_active":"2024-03-09T14:28:32.318971+0000","last_became_peered":"2024-03-09T14:28:32.318971+0000","last_unstale":"2024-03-09T15:33:35.803883+0000","last_undegraded":"2024-03-09T15:33:35.803883+0000","last_fullsized":"2024-03-09T15:33:35.803883+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:01:00.707624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243236,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":126,"num_read_kb":8647,"num_write":451,"num_write_kb":27350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.960027+0000","last_change":"2024-03-09T14:28:19.238362+0000","last_active":"2024-03-09T15:33:20.960027+0000","last_peered":"2024-03-09T15:33:20.960027+0000","last_clean":"2024-03-09T15:33:20.960027+0000","last_became_active":"2024-03-09T14:28:19.237750+0000","last_became_peered":"2024-03-09T14:28:19.237750+0000","last_unstale":"2024-03-09T15:33:20.960027+0000","last_undegraded":"2024-03-09T15:33:20.960027+0000","last_fullsized":"2024-03-09T15:33:20.960027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:19.699145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.185710+0000","last_change":"2024-03-09T14:28:36.328518+0000","last_active":"2024-03-09T15:33:11.185710+0000","last_peered":"2024-03-09T15:33:11.185710+0000","last_clean":"2024-03-09T15:33:11.185710+0000","last_became_active":"2024-03-09T14:28:36.327601+0000","last_became_peered":"2024-03-09T14:28:36.327601+0000","last_unstale":"2024-03-09T15:33:11.185710+0000","last_undegraded":"2024-03-09T15:33:11.185710+0000","last_fullsized":"2024-03-09T15:33:11.185710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:41:26.807068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"261'106","reported_seq":661,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.186823+0000","last_change":"2024-03-09T14:28:38.348479+0000","last_active":"2024-03-09T15:33:16.186823+0000","last_peered":"2024-03-09T15:33:16.186823+0000","last_clean":"2024-03-09T15:33:16.186823+0000","last_became_active":"2024-03-09T14:28:38.347750+0000","last_became_peered":"2024-03-09T14:28:38.347750+0000","last_unstale":"2024-03-09T15:33:16.186823+0000","last_undegraded":"2024-03-09T15:33:16.186823+0000","last_fullsized":"2024-03-09T15:33:16.186823+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:26:08.370238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"261'513","reported_seq":2485,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.727541+0000","last_change":"2024-03-09T14:28:40.387580+0000","last_active":"2024-03-09T15:36:05.727541+0000","last_peered":"2024-03-09T15:36:05.727541+0000","last_clean":"2024-03-09T15:36:05.727541+0000","last_became_active":"2024-03-09T14:28:40.386859+0000","last_became_peered":"2024-03-09T14:28:40.386859+0000","last_unstale":"2024-03-09T15:36:05.727541+0000","last_undegraded":"2024-03-09T15:36:05.727541+0000","last_fullsized":"2024-03-09T15:36:05.727541+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":500,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:17:24.125699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"261'106","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.987014+0000","last_change":"2024-03-09T14:28:42.406871+0000","last_active":"2024-03-09T15:31:45.987014+0000","last_peered":"2024-03-09T15:31:45.987014+0000","last_clean":"2024-03-09T15:31:45.987014+0000","last_became_active":"2024-03-09T14:28:42.406139+0000","last_became_peered":"2024-03-09T14:28:42.406139+0000","last_unstale":"2024-03-09T15:31:45.987014+0000","last_undegraded":"2024-03-09T15:31:45.987014+0000","last_fullsized":"2024-03-09T15:31:45.987014+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:24:57.719382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":65,"num_write":235,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"261'216","reported_seq":787,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:48.579968+0000","last_change":"2024-03-09T14:28:34.301025+0000","last_active":"2024-03-09T15:33:48.579968+0000","last_peered":"2024-03-09T15:33:48.579968+0000","last_clean":"2024-03-09T15:33:48.579968+0000","last_became_active":"2024-03-09T14:28:34.299988+0000","last_became_peered":"2024-03-09T14:28:34.299988+0000","last_unstale":"2024-03-09T15:33:48.579968+0000","last_undegraded":"2024-03-09T15:33:48.579968+0000","last_fullsized":"2024-03-09T15:33:48.579968+0000","mapping_epoch":22,"log_start":"258'200","ondisk_log_start":"258'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":200,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:05:09.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":843,"num_read_kb":645,"num_write":377,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"261'132","reported_seq":685,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.960146+0000","last_change":"2024-03-09T14:28:32.299582+0000","last_active":"2024-03-09T15:36:05.960146+0000","last_peered":"2024-03-09T15:36:05.960146+0000","last_clean":"2024-03-09T15:36:05.960146+0000","last_became_active":"2024-03-09T14:28:32.298977+0000","last_became_peered":"2024-03-09T14:28:32.298977+0000","last_unstale":"2024-03-09T15:36:05.960146+0000","last_undegraded":"2024-03-09T15:36:05.960146+0000","last_fullsized":"2024-03-09T15:36:05.960146+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:34:44.279787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243131,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":155,"num_read_kb":5781,"num_write":492,"num_write_kb":37567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.786634+0000","last_change":"2024-03-09T14:28:19.242561+0000","last_active":"2024-03-09T15:33:20.786634+0000","last_peered":"2024-03-09T15:33:20.786634+0000","last_clean":"2024-03-09T15:33:20.786634+0000","last_became_active":"2024-03-09T14:28:19.241700+0000","last_became_peered":"2024-03-09T14:28:19.241700+0000","last_unstale":"2024-03-09T15:33:20.786634+0000","last_undegraded":"2024-03-09T15:33:20.786634+0000","last_fullsized":"2024-03-09T15:33:20.786634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:15.462353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"32'3","reported_seq":822,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.255551+0000","last_change":"2024-03-09T14:28:36.317874+0000","last_active":"2024-03-09T15:36:05.255551+0000","last_peered":"2024-03-09T15:36:05.255551+0000","last_clean":"2024-03-09T15:36:05.255551+0000","last_became_active":"2024-03-09T14:28:36.317327+0000","last_became_peered":"2024-03-09T14:28:36.317327+0000","last_unstale":"2024-03-09T15:36:05.255551+0000","last_undegraded":"2024-03-09T15:36:05.255551+0000","last_fullsized":"2024-03-09T15:36:05.255551+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:07.321787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":484,"num_read_kb":482,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"261'465","reported_seq":1200,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.224674+0000","last_change":"2024-03-09T14:28:38.603808+0000","last_active":"2024-03-09T15:36:06.224674+0000","last_peered":"2024-03-09T15:36:06.224674+0000","last_clean":"2024-03-09T15:36:06.224674+0000","last_became_active":"2024-03-09T14:28:38.602908+0000","last_became_peered":"2024-03-09T14:28:38.602908+0000","last_unstale":"2024-03-09T15:36:06.224674+0000","last_undegraded":"2024-03-09T15:36:06.224674+0000","last_fullsized":"2024-03-09T15:36:06.224674+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":400,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:20:14.798348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58890,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":785,"num_read_kb":1500,"num_write":543,"num_write_kb":342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:15.799092+0000","last_change":"2024-03-09T14:28:40.376657+0000","last_active":"2024-03-09T15:33:15.799092+0000","last_peered":"2024-03-09T15:33:15.799092+0000","last_clean":"2024-03-09T15:33:15.799092+0000","last_became_active":"2024-03-09T14:28:40.375884+0000","last_became_peered":"2024-03-09T14:28:40.375884+0000","last_unstale":"2024-03-09T15:33:15.799092+0000","last_undegraded":"2024-03-09T15:33:15.799092+0000","last_fullsized":"2024-03-09T15:33:15.799092+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:28:12.389403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"261'54","reported_seq":551,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:33.602971+0000","last_change":"2024-03-09T14:28:42.406761+0000","last_active":"2024-03-09T15:34:33.602971+0000","last_peered":"2024-03-09T15:34:33.602971+0000","last_clean":"2024-03-09T15:34:33.602971+0000","last_became_active":"2024-03-09T14:28:42.405998+0000","last_became_peered":"2024-03-09T14:28:42.405998+0000","last_unstale":"2024-03-09T15:34:33.602971+0000","last_undegraded":"2024-03-09T15:34:33.602971+0000","last_fullsized":"2024-03-09T15:34:33.602971+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:02:27.576333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":52,"num_write":123,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1f","version":"261'146","reported_seq":723,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:09.596752+0000","last_change":"2024-03-09T14:28:42.410137+0000","last_active":"2024-03-09T15:34:09.596752+0000","last_peered":"2024-03-09T15:34:09.596752+0000","last_clean":"2024-03-09T15:34:09.596752+0000","last_became_active":"2024-03-09T14:28:42.408774+0000","last_became_peered":"2024-03-09T14:28:42.408774+0000","last_unstale":"2024-03-09T15:34:09.596752+0000","last_undegraded":"2024-03-09T15:34:09.596752+0000","last_fullsized":"2024-03-09T15:34:09.596752+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:50:49.883718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":256,"num_read_kb":233,"num_write":254,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"261'183","reported_seq":743,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:44.582989+0000","last_change":"2024-03-09T14:28:34.315742+0000","last_active":"2024-03-09T15:33:44.582989+0000","last_peered":"2024-03-09T15:33:44.582989+0000","last_clean":"2024-03-09T15:33:44.582989+0000","last_became_active":"2024-03-09T14:28:34.314893+0000","last_became_peered":"2024-03-09T14:28:34.314893+0000","last_unstale":"2024-03-09T15:33:44.582989+0000","last_undegraded":"2024-03-09T15:33:44.582989+0000","last_fullsized":"2024-03-09T15:33:44.582989+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":100,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:23:46.738791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":706,"num_read_kb":592,"num_write":291,"num_write_kb":155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"261'108","reported_seq":652,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.995245+0000","last_change":"2024-03-09T14:28:32.322792+0000","last_active":"2024-03-09T15:36:05.995245+0000","last_peered":"2024-03-09T15:36:05.995245+0000","last_clean":"2024-03-09T15:36:05.995245+0000","last_became_active":"2024-03-09T14:28:32.322364+0000","last_became_peered":"2024-03-09T14:28:32.322364+0000","last_unstale":"2024-03-09T15:36:05.995245+0000","last_undegraded":"2024-03-09T15:36:05.995245+0000","last_fullsized":"2024-03-09T15:36:05.995245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:58:42.664679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":128,"num_read_kb":128,"num_write":368,"num_write_kb":27813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.134035+0000","last_change":"2024-03-09T14:28:36.314599+0000","last_active":"2024-03-09T15:31:36.134035+0000","last_peered":"2024-03-09T15:31:36.134035+0000","last_clean":"2024-03-09T15:31:36.134035+0000","last_became_active":"2024-03-09T14:28:36.313984+0000","last_became_peered":"2024-03-09T14:28:36.313984+0000","last_unstale":"2024-03-09T15:31:36.134035+0000","last_undegraded":"2024-03-09T15:31:36.134035+0000","last_fullsized":"2024-03-09T15:31:36.134035+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:41:00.980683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"261'188","reported_seq":766,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.703452+0000","last_change":"2024-03-09T14:28:38.345412+0000","last_active":"2024-03-09T15:35:58.703452+0000","last_peered":"2024-03-09T15:35:58.703452+0000","last_clean":"2024-03-09T15:35:58.703452+0000","last_became_active":"2024-03-09T14:28:38.338262+0000","last_became_peered":"2024-03-09T14:28:38.338262+0000","last_unstale":"2024-03-09T15:35:58.703452+0000","last_undegraded":"2024-03-09T15:35:58.703452+0000","last_fullsized":"2024-03-09T15:35:58.703452+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":100,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:08:16.291678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29268,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":306,"num_read_kb":562,"num_write":216,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":489,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.777786+0000","last_change":"2024-03-09T14:28:40.390067+0000","last_active":"2024-03-09T15:31:40.777786+0000","last_peered":"2024-03-09T15:31:40.777786+0000","last_clean":"2024-03-09T15:31:40.777786+0000","last_became_active":"2024-03-09T14:28:40.389277+0000","last_became_peered":"2024-03-09T14:28:40.389277+0000","last_unstale":"2024-03-09T15:31:40.777786+0000","last_undegraded":"2024-03-09T15:31:40.777786+0000","last_fullsized":"2024-03-09T15:31:40.777786+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:02:25.553702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"261'54","reported_seq":549,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.795784+0000","last_change":"2024-03-09T14:28:42.411135+0000","last_active":"2024-03-09T15:33:00.795784+0000","last_peered":"2024-03-09T15:33:00.795784+0000","last_clean":"2024-03-09T15:33:00.795784+0000","last_became_active":"2024-03-09T14:28:42.409229+0000","last_became_peered":"2024-03-09T14:28:42.409229+0000","last_unstale":"2024-03-09T15:33:00.795784+0000","last_undegraded":"2024-03-09T15:33:00.795784+0000","last_fullsized":"2024-03-09T15:33:00.795784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:17:09.392755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":828,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":44,"num_write":116,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"261'78","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.134077+0000","last_change":"2024-03-09T14:28:34.320825+0000","last_active":"2024-03-09T15:31:36.134077+0000","last_peered":"2024-03-09T15:31:36.134077+0000","last_clean":"2024-03-09T15:31:36.134077+0000","last_became_active":"2024-03-09T14:28:34.320474+0000","last_became_peered":"2024-03-09T14:28:34.320474+0000","last_unstale":"2024-03-09T15:31:36.134077+0000","last_undegraded":"2024-03-09T15:31:36.134077+0000","last_fullsized":"2024-03-09T15:31:36.134077+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:15:48.878005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":231,"num_write":128,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"261'149","reported_seq":781,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.185355+0000","last_change":"2024-03-09T14:28:32.317900+0000","last_active":"2024-03-09T15:36:06.185355+0000","last_peered":"2024-03-09T15:36:06.185355+0000","last_clean":"2024-03-09T15:36:06.185355+0000","last_became_active":"2024-03-09T14:28:32.315355+0000","last_became_peered":"2024-03-09T14:28:32.315355+0000","last_unstale":"2024-03-09T15:36:06.185355+0000","last_undegraded":"2024-03-09T15:36:06.185355+0000","last_fullsized":"2024-03-09T15:36:06.185355+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:28:48.531373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655932,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":315,"num_read_kb":85989,"num_write":571,"num_write_kb":44811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776757+0000","last_change":"2024-03-09T14:28:36.311910+0000","last_active":"2024-03-09T15:31:35.776757+0000","last_peered":"2024-03-09T15:31:35.776757+0000","last_clean":"2024-03-09T15:31:35.776757+0000","last_became_active":"2024-03-09T14:28:36.311487+0000","last_became_peered":"2024-03-09T14:28:36.311487+0000","last_unstale":"2024-03-09T15:31:35.776757+0000","last_undegraded":"2024-03-09T15:31:35.776757+0000","last_fullsized":"2024-03-09T15:31:35.776757+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:04:59.933448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"261'39","reported_seq":550,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:31.436658+0000","last_change":"2024-03-09T14:28:38.333369+0000","last_active":"2024-03-09T15:33:31.436658+0000","last_peered":"2024-03-09T15:33:31.436658+0000","last_clean":"2024-03-09T15:33:31.436658+0000","last_became_active":"2024-03-09T14:28:38.332610+0000","last_became_peered":"2024-03-09T14:28:38.332610+0000","last_unstale":"2024-03-09T15:33:31.436658+0000","last_undegraded":"2024-03-09T15:33:31.436658+0000","last_fullsized":"2024-03-09T15:33:31.436658+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:56:36.283073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":41,"num_write":30,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":498,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.178031+0000","last_change":"2024-03-09T14:28:40.368708+0000","last_active":"2024-03-09T15:31:41.178031+0000","last_peered":"2024-03-09T15:31:41.178031+0000","last_clean":"2024-03-09T15:31:41.178031+0000","last_became_active":"2024-03-09T14:28:40.367795+0000","last_became_peered":"2024-03-09T14:28:40.367795+0000","last_unstale":"2024-03-09T15:31:41.178031+0000","last_undegraded":"2024-03-09T15:31:41.178031+0000","last_fullsized":"2024-03-09T15:31:41.178031+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:46:56.597658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"261'170","reported_seq":688,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:44.567149+0000","last_change":"2024-03-09T14:28:42.410358+0000","last_active":"2024-03-09T15:33:44.567149+0000","last_peered":"2024-03-09T15:33:44.567149+0000","last_clean":"2024-03-09T15:33:44.567149+0000","last_became_active":"2024-03-09T14:28:42.408913+0000","last_became_peered":"2024-03-09T14:28:42.408913+0000","last_unstale":"2024-03-09T15:33:44.567149+0000","last_undegraded":"2024-03-09T15:33:44.567149+0000","last_fullsized":"2024-03-09T15:33:44.567149+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":100,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:06:25.964093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":208,"num_write":351,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"261'54","reported_seq":593,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:52.364811+0000","last_change":"2024-03-09T14:28:34.331192+0000","last_active":"2024-03-09T15:31:52.364811+0000","last_peered":"2024-03-09T15:31:52.364811+0000","last_clean":"2024-03-09T15:31:52.364811+0000","last_became_active":"2024-03-09T14:28:34.330813+0000","last_became_peered":"2024-03-09T14:28:34.330813+0000","last_unstale":"2024-03-09T15:31:52.364811+0000","last_undegraded":"2024-03-09T15:31:52.364811+0000","last_fullsized":"2024-03-09T15:31:52.364811+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:02:15.115504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":220,"num_read_kb":178,"num_write":76,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"261'114","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.113284+0000","last_change":"2024-03-09T14:28:32.313709+0000","last_active":"2024-03-09T15:36:06.113284+0000","last_peered":"2024-03-09T15:36:06.113284+0000","last_clean":"2024-03-09T15:36:06.113284+0000","last_became_active":"2024-03-09T14:28:32.311843+0000","last_became_peered":"2024-03-09T14:28:32.311843+0000","last_unstale":"2024-03-09T15:36:06.113284+0000","last_undegraded":"2024-03-09T15:36:06.113284+0000","last_fullsized":"2024-03-09T15:36:06.113284+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:51:11.937911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1830596,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":144,"num_read_kb":5348,"num_write":462,"num_write_kb":35385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.176563+0000","last_change":"2024-03-09T14:28:36.324124+0000","last_active":"2024-03-09T15:31:36.176563+0000","last_peered":"2024-03-09T15:31:36.176563+0000","last_clean":"2024-03-09T15:31:36.176563+0000","last_became_active":"2024-03-09T14:28:36.323335+0000","last_became_peered":"2024-03-09T14:28:36.323335+0000","last_unstale":"2024-03-09T15:31:36.176563+0000","last_undegraded":"2024-03-09T15:31:36.176563+0000","last_fullsized":"2024-03-09T15:31:36.176563+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:09:55.020425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"261'219","reported_seq":812,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.676436+0000","last_change":"2024-03-09T14:28:38.345760+0000","last_active":"2024-03-09T15:35:58.676436+0000","last_peered":"2024-03-09T15:35:58.676436+0000","last_clean":"2024-03-09T15:35:58.676436+0000","last_became_active":"2024-03-09T14:28:38.344951+0000","last_became_peered":"2024-03-09T14:28:38.344951+0000","last_unstale":"2024-03-09T15:35:58.676436+0000","last_undegraded":"2024-03-09T15:35:58.676436+0000","last_fullsized":"2024-03-09T15:35:58.676436+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":200,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:52:01.179169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24078,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":344,"num_read_kb":584,"num_write":240,"num_write_kb":124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.766053+0000","last_change":"2024-03-09T14:28:40.374861+0000","last_active":"2024-03-09T15:31:40.766053+0000","last_peered":"2024-03-09T15:31:40.766053+0000","last_clean":"2024-03-09T15:31:40.766053+0000","last_became_active":"2024-03-09T14:28:40.370313+0000","last_became_peered":"2024-03-09T14:28:40.370313+0000","last_unstale":"2024-03-09T15:31:40.766053+0000","last_undegraded":"2024-03-09T15:31:40.766053+0000","last_fullsized":"2024-03-09T15:31:40.766053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:20:51.501674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"261'45","reported_seq":538,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:07.675530+0000","last_change":"2024-03-09T14:28:42.405094+0000","last_active":"2024-03-09T15:35:07.675530+0000","last_peered":"2024-03-09T15:35:07.675530+0000","last_clean":"2024-03-09T15:35:07.675530+0000","last_became_active":"2024-03-09T14:28:42.404360+0000","last_became_peered":"2024-03-09T14:28:42.404360+0000","last_unstale":"2024-03-09T15:35:07.675530+0000","last_undegraded":"2024-03-09T15:35:07.675530+0000","last_fullsized":"2024-03-09T15:35:07.675530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:33:45.977796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":43,"num_write":116,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"261'87","reported_seq":640,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776531+0000","last_change":"2024-03-09T14:28:34.313014+0000","last_active":"2024-03-09T15:31:35.776531+0000","last_peered":"2024-03-09T15:31:35.776531+0000","last_clean":"2024-03-09T15:31:35.776531+0000","last_became_active":"2024-03-09T14:28:34.305301+0000","last_became_peered":"2024-03-09T14:28:34.305301+0000","last_unstale":"2024-03-09T15:31:35.776531+0000","last_undegraded":"2024-03-09T15:31:35.776531+0000","last_fullsized":"2024-03-09T15:31:35.776531+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:46:37.987444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":317,"num_write":145,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"261'120","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.996239+0000","last_change":"2024-03-09T14:28:32.293199+0000","last_active":"2024-03-09T15:36:05.996239+0000","last_peered":"2024-03-09T15:36:05.996239+0000","last_clean":"2024-03-09T15:36:05.996239+0000","last_became_active":"2024-03-09T14:28:32.292393+0000","last_became_peered":"2024-03-09T14:28:32.292393+0000","last_unstale":"2024-03-09T15:36:05.996239+0000","last_undegraded":"2024-03-09T15:36:05.996239+0000","last_fullsized":"2024-03-09T15:36:05.996239+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:58:50.345089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7206306,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":165,"num_read_kb":14616,"num_write":445,"num_write_kb":23484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.764963+0000","last_change":"2024-03-09T14:28:36.329882+0000","last_active":"2024-03-09T15:31:35.764963+0000","last_peered":"2024-03-09T15:31:35.764963+0000","last_clean":"2024-03-09T15:31:35.764963+0000","last_became_active":"2024-03-09T14:28:36.328946+0000","last_became_peered":"2024-03-09T14:28:36.328946+0000","last_unstale":"2024-03-09T15:31:35.764963+0000","last_undegraded":"2024-03-09T15:31:35.764963+0000","last_fullsized":"2024-03-09T15:31:35.764963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:23:54.027658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"261'168","reported_seq":752,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.194863+0000","last_change":"2024-03-09T14:28:38.336043+0000","last_active":"2024-03-09T15:36:06.194863+0000","last_peered":"2024-03-09T15:36:06.194863+0000","last_clean":"2024-03-09T15:36:06.194863+0000","last_became_active":"2024-03-09T14:28:38.335107+0000","last_became_peered":"2024-03-09T14:28:38.335107+0000","last_unstale":"2024-03-09T15:36:06.194863+0000","last_undegraded":"2024-03-09T15:36:06.194863+0000","last_fullsized":"2024-03-09T15:36:06.194863+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":100,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:08:12.713928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9765,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":273,"num_read_kb":518,"num_write":186,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.985823+0000","last_change":"2024-03-09T14:28:40.364509+0000","last_active":"2024-03-09T15:31:40.985823+0000","last_peered":"2024-03-09T15:31:40.985823+0000","last_clean":"2024-03-09T15:31:40.985823+0000","last_became_active":"2024-03-09T14:28:40.363778+0000","last_became_peered":"2024-03-09T14:28:40.363778+0000","last_unstale":"2024-03-09T15:31:40.985823+0000","last_undegraded":"2024-03-09T15:31:40.985823+0000","last_fullsized":"2024-03-09T15:31:40.985823+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:49:40.990617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"261'159","reported_seq":733,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784514+0000","last_change":"2024-03-09T14:28:34.325746+0000","last_active":"2024-03-09T15:33:10.784514+0000","last_peered":"2024-03-09T15:33:10.784514+0000","last_clean":"2024-03-09T15:33:10.784514+0000","last_became_active":"2024-03-09T14:28:34.325031+0000","last_became_peered":"2024-03-09T14:28:34.325031+0000","last_unstale":"2024-03-09T15:33:10.784514+0000","last_undegraded":"2024-03-09T15:33:10.784514+0000","last_fullsized":"2024-03-09T15:33:10.784514+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:12:45.802989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":584,"num_read_kb":461,"num_write":254,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"261'97","reported_seq":636,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:52.210937+0000","last_change":"2024-03-09T14:28:32.296634+0000","last_active":"2024-03-09T15:31:52.210937+0000","last_peered":"2024-03-09T15:31:52.210937+0000","last_clean":"2024-03-09T15:31:52.210937+0000","last_became_active":"2024-03-09T14:28:32.296179+0000","last_became_peered":"2024-03-09T14:28:32.296179+0000","last_unstale":"2024-03-09T15:31:52.210937+0000","last_undegraded":"2024-03-09T15:31:52.210937+0000","last_fullsized":"2024-03-09T15:31:52.210937+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:07:03.555863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263400,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":122,"num_read_kb":11385,"num_write":357,"num_write_kb":17181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:51.002216+0000","last_change":"2024-03-09T14:28:36.320059+0000","last_active":"2024-03-09T15:32:51.002216+0000","last_peered":"2024-03-09T15:32:51.002216+0000","last_clean":"2024-03-09T15:32:51.002216+0000","last_became_active":"2024-03-09T14:28:36.319702+0000","last_became_peered":"2024-03-09T14:28:36.319702+0000","last_unstale":"2024-03-09T15:32:51.002216+0000","last_undegraded":"2024-03-09T15:32:51.002216+0000","last_fullsized":"2024-03-09T15:32:51.002216+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:38:28.003376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"261'309","reported_seq":944,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.877702+0000","last_change":"2024-03-09T14:28:38.333414+0000","last_active":"2024-03-09T15:36:05.877702+0000","last_peered":"2024-03-09T15:36:05.877702+0000","last_clean":"2024-03-09T15:36:05.877702+0000","last_became_active":"2024-03-09T14:28:38.332661+0000","last_became_peered":"2024-03-09T14:28:38.332661+0000","last_unstale":"2024-03-09T15:36:05.877702+0000","last_undegraded":"2024-03-09T15:36:05.877702+0000","last_fullsized":"2024-03-09T15:36:05.877702+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":200,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:09:54.404475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34580,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":556,"num_read_kb":1111,"num_write":442,"num_write_kb":360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:11.163194+0000","last_change":"2024-03-09T14:28:40.379385+0000","last_active":"2024-03-09T15:31:11.163194+0000","last_peered":"2024-03-09T15:31:11.163194+0000","last_clean":"2024-03-09T15:31:11.163194+0000","last_became_active":"2024-03-09T14:28:40.378829+0000","last_became_peered":"2024-03-09T14:28:40.378829+0000","last_unstale":"2024-03-09T15:31:11.163194+0000","last_undegraded":"2024-03-09T15:31:11.163194+0000","last_fullsized":"2024-03-09T15:31:11.163194+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:25:24.764687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"261'124","reported_seq":623,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.333094+0000","last_change":"2024-03-09T14:28:42.393487+0000","last_active":"2024-03-09T15:33:00.333094+0000","last_peered":"2024-03-09T15:33:00.333094+0000","last_clean":"2024-03-09T15:33:00.333094+0000","last_became_active":"2024-03-09T14:28:42.392509+0000","last_became_peered":"2024-03-09T14:28:42.392509+0000","last_unstale":"2024-03-09T15:33:00.333094+0000","last_undegraded":"2024-03-09T15:33:00.333094+0000","last_fullsized":"2024-03-09T15:33:00.333094+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:53:46.680420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":643,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":173,"num_write":275,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"261'132","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:51.002176+0000","last_change":"2024-03-09T14:28:34.330979+0000","last_active":"2024-03-09T15:32:51.002176+0000","last_peered":"2024-03-09T15:32:51.002176+0000","last_clean":"2024-03-09T15:32:51.002176+0000","last_became_active":"2024-03-09T14:28:34.330686+0000","last_became_peered":"2024-03-09T14:28:34.330686+0000","last_unstale":"2024-03-09T15:32:51.002176+0000","last_undegraded":"2024-03-09T15:32:51.002176+0000","last_fullsized":"2024-03-09T15:32:51.002176+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:07:36.006587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":527,"num_read_kb":451,"num_write":212,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"261'111","reported_seq":649,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.905029+0000","last_change":"2024-03-09T14:28:32.301602+0000","last_active":"2024-03-09T15:36:05.905029+0000","last_peered":"2024-03-09T15:36:05.905029+0000","last_clean":"2024-03-09T15:36:05.905029+0000","last_became_active":"2024-03-09T14:28:32.301087+0000","last_became_peered":"2024-03-09T14:28:32.301087+0000","last_unstale":"2024-03-09T15:36:05.905029+0000","last_undegraded":"2024-03-09T15:36:05.905029+0000","last_fullsized":"2024-03-09T15:36:05.905029+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:13:11.042530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":115,"num_read_kb":3393,"num_write":438,"num_write_kb":23270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784490+0000","last_change":"2024-03-09T14:28:36.329619+0000","last_active":"2024-03-09T15:33:10.784490+0000","last_peered":"2024-03-09T15:33:10.784490+0000","last_clean":"2024-03-09T15:33:10.784490+0000","last_became_active":"2024-03-09T14:28:36.328814+0000","last_became_peered":"2024-03-09T14:28:36.328814+0000","last_unstale":"2024-03-09T15:33:10.784490+0000","last_undegraded":"2024-03-09T15:33:10.784490+0000","last_fullsized":"2024-03-09T15:33:10.784490+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:24:43.021216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"261'252","reported_seq":855,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.221152+0000","last_change":"2024-03-09T14:28:38.332894+0000","last_active":"2024-03-09T15:36:06.221152+0000","last_peered":"2024-03-09T15:36:06.221152+0000","last_clean":"2024-03-09T15:36:06.221152+0000","last_became_active":"2024-03-09T14:28:38.331930+0000","last_became_peered":"2024-03-09T14:28:38.331930+0000","last_unstale":"2024-03-09T15:36:06.221152+0000","last_undegraded":"2024-03-09T15:36:06.221152+0000","last_fullsized":"2024-03-09T15:36:06.221152+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":200,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:36:58.809794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21830,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":424,"num_read_kb":798,"num_write":342,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:11.128169+0000","last_change":"2024-03-09T14:28:40.367660+0000","last_active":"2024-03-09T15:31:11.128169+0000","last_peered":"2024-03-09T15:31:11.128169+0000","last_clean":"2024-03-09T15:31:11.128169+0000","last_became_active":"2024-03-09T14:28:40.366723+0000","last_became_peered":"2024-03-09T14:28:40.366723+0000","last_unstale":"2024-03-09T15:31:11.128169+0000","last_undegraded":"2024-03-09T15:31:11.128169+0000","last_fullsized":"2024-03-09T15:31:11.128169+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:42:24.773777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"261'108","reported_seq":613,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:31.400225+0000","last_change":"2024-03-09T14:28:42.395993+0000","last_active":"2024-03-09T15:33:31.400225+0000","last_peered":"2024-03-09T15:33:31.400225+0000","last_clean":"2024-03-09T15:33:31.400225+0000","last_became_active":"2024-03-09T14:28:42.395375+0000","last_became_peered":"2024-03-09T14:28:42.395375+0000","last_unstale":"2024-03-09T15:33:31.400225+0000","last_undegraded":"2024-03-09T15:33:31.400225+0000","last_fullsized":"2024-03-09T15:33:31.400225+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:57:19.143770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1199,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":139,"num_read_kb":136,"num_write":341,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"261'99","reported_seq":686,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:45.201625+0000","last_change":"2024-03-09T14:28:34.312670+0000","last_active":"2024-03-09T15:34:45.201625+0000","last_peered":"2024-03-09T15:34:45.201625+0000","last_clean":"2024-03-09T15:34:45.201625+0000","last_became_active":"2024-03-09T14:28:34.312297+0000","last_became_peered":"2024-03-09T14:28:34.312297+0000","last_unstale":"2024-03-09T15:34:45.201625+0000","last_undegraded":"2024-03-09T15:34:45.201625+0000","last_fullsized":"2024-03-09T15:34:45.201625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:36:43.430324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":415,"num_read_kb":347,"num_write":146,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"261'101","reported_seq":669,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.985205+0000","last_change":"2024-03-09T14:28:32.314186+0000","last_active":"2024-03-09T15:36:05.985205+0000","last_peered":"2024-03-09T15:36:05.985205+0000","last_clean":"2024-03-09T15:36:05.985205+0000","last_became_active":"2024-03-09T14:28:32.312937+0000","last_became_peered":"2024-03-09T14:28:32.312937+0000","last_unstale":"2024-03-09T15:36:05.985205+0000","last_undegraded":"2024-03-09T15:36:05.985205+0000","last_fullsized":"2024-03-09T15:36:05.985205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:33:36.448497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262330,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":144,"num_read_kb":47424,"num_write":415,"num_write_kb":24173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191452+0000","last_change":"2024-03-09T14:28:36.316936+0000","last_active":"2024-03-09T15:33:36.191452+0000","last_peered":"2024-03-09T15:33:36.191452+0000","last_clean":"2024-03-09T15:33:36.191452+0000","last_became_active":"2024-03-09T14:28:36.316094+0000","last_became_peered":"2024-03-09T14:28:36.316094+0000","last_unstale":"2024-03-09T15:33:36.191452+0000","last_undegraded":"2024-03-09T15:33:36.191452+0000","last_fullsized":"2024-03-09T15:33:36.191452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:41:42.620548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"261'188","reported_seq":770,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.713172+0000","last_change":"2024-03-09T14:28:38.596294+0000","last_active":"2024-03-09T15:35:58.713172+0000","last_peered":"2024-03-09T15:35:58.713172+0000","last_clean":"2024-03-09T15:35:58.713172+0000","last_became_active":"2024-03-09T14:28:38.595522+0000","last_became_peered":"2024-03-09T14:28:38.595522+0000","last_unstale":"2024-03-09T15:35:58.713172+0000","last_undegraded":"2024-03-09T15:35:58.713172+0000","last_fullsized":"2024-03-09T15:35:58.713172+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":100,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:20:49.471413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25352,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":302,"num_read_kb":534,"num_write":212,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:50.952318+0000","last_change":"2024-03-09T14:28:40.377325+0000","last_active":"2024-03-09T15:32:50.952318+0000","last_peered":"2024-03-09T15:32:50.952318+0000","last_clean":"2024-03-09T15:32:50.952318+0000","last_became_active":"2024-03-09T14:28:40.376549+0000","last_became_peered":"2024-03-09T14:28:40.376549+0000","last_unstale":"2024-03-09T15:32:50.952318+0000","last_undegraded":"2024-03-09T15:32:50.952318+0000","last_fullsized":"2024-03-09T15:32:50.952318+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:59:06.393597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"261'42","reported_seq":537,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.728760+0000","last_change":"2024-03-09T14:28:42.395593+0000","last_active":"2024-03-09T15:35:58.728760+0000","last_peered":"2024-03-09T15:35:58.728760+0000","last_clean":"2024-03-09T15:35:58.728760+0000","last_became_active":"2024-03-09T14:28:42.394780+0000","last_became_peered":"2024-03-09T14:28:42.394780+0000","last_unstale":"2024-03-09T15:35:58.728760+0000","last_undegraded":"2024-03-09T15:35:58.728760+0000","last_fullsized":"2024-03-09T15:35:58.728760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:21:45.395383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":84,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"261'68","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:22.422461+0000","last_change":"2024-03-09T14:28:34.313519+0000","last_active":"2024-03-09T15:34:22.422461+0000","last_peered":"2024-03-09T15:34:22.422461+0000","last_clean":"2024-03-09T15:34:22.422461+0000","last_became_active":"2024-03-09T14:28:34.313125+0000","last_became_peered":"2024-03-09T14:28:34.313125+0000","last_unstale":"2024-03-09T15:34:22.422461+0000","last_undegraded":"2024-03-09T15:34:22.422461+0000","last_fullsized":"2024-03-09T15:34:22.422461+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:57:07.167382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"261'175","reported_seq":779,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.986932+0000","last_change":"2024-03-09T14:28:32.304248+0000","last_active":"2024-03-09T15:36:05.986932+0000","last_peered":"2024-03-09T15:36:05.986932+0000","last_clean":"2024-03-09T15:36:05.986932+0000","last_became_active":"2024-03-09T14:28:32.303835+0000","last_became_peered":"2024-03-09T14:28:32.303835+0000","last_unstale":"2024-03-09T15:36:05.986932+0000","last_undegraded":"2024-03-09T15:36:05.986932+0000","last_fullsized":"2024-03-09T15:36:05.986932+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":100,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:32:30.929760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":359,"num_read_kb":1431,"num_write":987,"num_write_kb":8832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963406+0000","last_change":"2024-03-09T14:28:36.322532+0000","last_active":"2024-03-09T15:33:35.963406+0000","last_peered":"2024-03-09T15:33:35.963406+0000","last_clean":"2024-03-09T15:33:35.963406+0000","last_became_active":"2024-03-09T14:28:36.322060+0000","last_became_peered":"2024-03-09T14:28:36.322060+0000","last_unstale":"2024-03-09T15:33:35.963406+0000","last_undegraded":"2024-03-09T15:33:35.963406+0000","last_fullsized":"2024-03-09T15:33:35.963406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:48:28.324190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"261'89","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.180393+0000","last_change":"2024-03-09T14:28:38.596598+0000","last_active":"2024-03-09T15:32:46.180393+0000","last_peered":"2024-03-09T15:32:46.180393+0000","last_clean":"2024-03-09T15:32:46.180393+0000","last_became_active":"2024-03-09T14:28:38.595749+0000","last_became_peered":"2024-03-09T14:28:38.595749+0000","last_unstale":"2024-03-09T15:32:46.180393+0000","last_undegraded":"2024-03-09T15:32:46.180393+0000","last_fullsized":"2024-03-09T15:32:46.180393+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:47:35.161583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":132,"num_read_kb":127,"num_write":124,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"261'488","reported_seq":2256,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.713372+0000","last_change":"2024-03-09T14:28:40.377689+0000","last_active":"2024-03-09T15:36:05.713372+0000","last_peered":"2024-03-09T15:36:05.713372+0000","last_clean":"2024-03-09T15:36:05.713372+0000","last_became_active":"2024-03-09T14:28:40.376946+0000","last_became_peered":"2024-03-09T14:28:40.376946+0000","last_unstale":"2024-03-09T15:36:05.713372+0000","last_undegraded":"2024-03-09T15:36:05.713372+0000","last_fullsized":"2024-03-09T15:36:05.713372+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":400,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:50:25.914007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.10","version":"261'45","reported_seq":549,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:48.564964+0000","last_change":"2024-03-09T14:28:42.399023+0000","last_active":"2024-03-09T15:33:48.564964+0000","last_peered":"2024-03-09T15:33:48.564964+0000","last_clean":"2024-03-09T15:33:48.564964+0000","last_became_active":"2024-03-09T14:28:42.397829+0000","last_became_peered":"2024-03-09T14:28:42.397829+0000","last_unstale":"2024-03-09T15:33:48.564964+0000","last_undegraded":"2024-03-09T15:33:48.564964+0000","last_fullsized":"2024-03-09T15:33:48.564964+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:36:00.626046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":97,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"261'144","reported_seq":675,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191372+0000","last_change":"2024-03-09T14:28:34.321426+0000","last_active":"2024-03-09T15:33:36.191372+0000","last_peered":"2024-03-09T15:33:36.191372+0000","last_clean":"2024-03-09T15:33:36.191372+0000","last_became_active":"2024-03-09T14:28:34.319891+0000","last_became_peered":"2024-03-09T14:28:34.319891+0000","last_unstale":"2024-03-09T15:33:36.191372+0000","last_undegraded":"2024-03-09T15:33:36.191372+0000","last_fullsized":"2024-03-09T15:33:36.191372+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:25:46.129222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":571,"num_read_kb":433,"num_write":263,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"261'145","reported_seq":791,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.201525+0000","last_change":"2024-03-09T14:28:32.323880+0000","last_active":"2024-03-09T15:36:06.201525+0000","last_peered":"2024-03-09T15:36:06.201525+0000","last_clean":"2024-03-09T15:36:06.201525+0000","last_became_active":"2024-03-09T14:28:32.323198+0000","last_became_peered":"2024-03-09T14:28:32.323198+0000","last_unstale":"2024-03-09T15:36:06.201525+0000","last_undegraded":"2024-03-09T15:36:06.201525+0000","last_fullsized":"2024-03-09T15:36:06.201525+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:10:52.793653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4801712,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":390,"num_read_kb":2095,"num_write":636,"num_write_kb":25158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205346+0000","last_change":"2024-03-09T14:28:36.329314+0000","last_active":"2024-03-09T15:33:36.205346+0000","last_peered":"2024-03-09T15:33:36.205346+0000","last_clean":"2024-03-09T15:33:36.205346+0000","last_became_active":"2024-03-09T14:28:36.328529+0000","last_became_peered":"2024-03-09T14:28:36.328529+0000","last_unstale":"2024-03-09T15:33:36.205346+0000","last_undegraded":"2024-03-09T15:33:36.205346+0000","last_fullsized":"2024-03-09T15:33:36.205346+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:42:17.375037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"261'224","reported_seq":826,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.002456+0000","last_change":"2024-03-09T14:28:38.335758+0000","last_active":"2024-03-09T15:36:06.002456+0000","last_peered":"2024-03-09T15:36:06.002456+0000","last_clean":"2024-03-09T15:36:06.002456+0000","last_became_active":"2024-03-09T14:28:38.334991+0000","last_became_peered":"2024-03-09T14:28:38.334991+0000","last_unstale":"2024-03-09T15:36:06.002456+0000","last_undegraded":"2024-03-09T15:36:06.002456+0000","last_fullsized":"2024-03-09T15:36:06.002456+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":200,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:19:26.980708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13314,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":373,"num_read_kb":687,"num_write":264,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:30.983178+0000","last_change":"2024-03-09T14:28:40.368865+0000","last_active":"2024-03-09T15:31:30.983178+0000","last_peered":"2024-03-09T15:31:30.983178+0000","last_clean":"2024-03-09T15:31:30.983178+0000","last_became_active":"2024-03-09T14:28:40.368039+0000","last_became_peered":"2024-03-09T14:28:40.368039+0000","last_unstale":"2024-03-09T15:31:30.983178+0000","last_undegraded":"2024-03-09T15:31:30.983178+0000","last_fullsized":"2024-03-09T15:31:30.983178+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:50:42.121684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"261'77","reported_seq":597,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:46.896952+0000","last_change":"2024-03-09T14:28:42.414457+0000","last_active":"2024-03-09T15:34:46.896952+0000","last_peered":"2024-03-09T15:34:46.896952+0000","last_clean":"2024-03-09T15:34:46.896952+0000","last_became_active":"2024-03-09T14:28:42.413840+0000","last_became_peered":"2024-03-09T14:28:42.413840+0000","last_unstale":"2024-03-09T15:34:46.896952+0000","last_undegraded":"2024-03-09T15:34:46.896952+0000","last_fullsized":"2024-03-09T15:34:46.896952+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:42:30.742160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":129,"num_write":187,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"261'36","reported_seq":573,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:52.977046+0000","last_change":"2024-03-09T14:28:34.318737+0000","last_active":"2024-03-09T15:35:52.977046+0000","last_peered":"2024-03-09T15:35:52.977046+0000","last_clean":"2024-03-09T15:35:52.977046+0000","last_became_active":"2024-03-09T14:28:34.314194+0000","last_became_peered":"2024-03-09T14:28:34.314194+0000","last_unstale":"2024-03-09T15:35:52.977046+0000","last_undegraded":"2024-03-09T15:35:52.977046+0000","last_fullsized":"2024-03-09T15:35:52.977046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:45:29.912431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":120,"num_write":54,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"261'128","reported_seq":781,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.160748+0000","last_change":"2024-03-09T14:28:32.309431+0000","last_active":"2024-03-09T15:36:06.160748+0000","last_peered":"2024-03-09T15:36:06.160748+0000","last_clean":"2024-03-09T15:36:06.160748+0000","last_became_active":"2024-03-09T14:28:32.307702+0000","last_became_peered":"2024-03-09T14:28:32.307702+0000","last_unstale":"2024-03-09T15:36:06.160748+0000","last_undegraded":"2024-03-09T15:36:06.160748+0000","last_fullsized":"2024-03-09T15:36:06.160748+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:51:59.883119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":254,"num_read_kb":1758,"num_write":464,"num_write_kb":23078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194464+0000","last_change":"2024-03-09T14:28:36.305006+0000","last_active":"2024-03-09T15:33:36.194464+0000","last_peered":"2024-03-09T15:33:36.194464+0000","last_clean":"2024-03-09T15:33:36.194464+0000","last_became_active":"2024-03-09T14:28:36.304241+0000","last_became_peered":"2024-03-09T14:28:36.304241+0000","last_unstale":"2024-03-09T15:33:36.194464+0000","last_undegraded":"2024-03-09T15:33:36.194464+0000","last_fullsized":"2024-03-09T15:33:36.194464+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:14:09.284541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"261'178","reported_seq":757,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.035481+0000","last_change":"2024-03-09T14:28:38.340000+0000","last_active":"2024-03-09T15:36:06.035481+0000","last_peered":"2024-03-09T15:36:06.035481+0000","last_clean":"2024-03-09T15:36:06.035481+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:36:06.035481+0000","last_undegraded":"2024-03-09T15:36:06.035481+0000","last_fullsized":"2024-03-09T15:36:06.035481+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":100,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:06:57.175032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":521,"num_write":190,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.768268+0000","last_change":"2024-03-09T14:28:40.375515+0000","last_active":"2024-03-09T15:31:50.768268+0000","last_peered":"2024-03-09T15:31:50.768268+0000","last_clean":"2024-03-09T15:31:50.768268+0000","last_became_active":"2024-03-09T14:28:40.371820+0000","last_became_peered":"2024-03-09T14:28:40.371820+0000","last_unstale":"2024-03-09T15:31:50.768268+0000","last_undegraded":"2024-03-09T15:31:50.768268+0000","last_fullsized":"2024-03-09T15:31:50.768268+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:15:53.172763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.16","version":"261'53","reported_seq":567,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.020972+0000","last_change":"2024-03-09T14:28:42.392999+0000","last_active":"2024-03-09T15:33:10.020972+0000","last_peered":"2024-03-09T15:33:10.020972+0000","last_clean":"2024-03-09T15:33:10.020972+0000","last_became_active":"2024-03-09T14:28:42.392326+0000","last_became_peered":"2024-03-09T14:28:42.392326+0000","last_unstale":"2024-03-09T15:33:10.020972+0000","last_undegraded":"2024-03-09T15:33:10.020972+0000","last_fullsized":"2024-03-09T15:33:10.020972+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:36.796428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":772,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":99,"num_write":117,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"261'70","reported_seq":632,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.777840+0000","last_change":"2024-03-09T14:28:34.324293+0000","last_active":"2024-03-09T15:31:40.777840+0000","last_peered":"2024-03-09T15:31:40.777840+0000","last_clean":"2024-03-09T15:31:40.777840+0000","last_became_active":"2024-03-09T14:28:34.323969+0000","last_became_peered":"2024-03-09T14:28:34.323969+0000","last_unstale":"2024-03-09T15:31:40.777840+0000","last_undegraded":"2024-03-09T15:31:40.777840+0000","last_fullsized":"2024-03-09T15:31:40.777840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:33:30.511134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":256,"num_write":98,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"261'149","reported_seq":740,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:50.793396+0000","last_change":"2024-03-09T14:28:32.338014+0000","last_active":"2024-03-09T15:32:50.793396+0000","last_peered":"2024-03-09T15:32:50.793396+0000","last_clean":"2024-03-09T15:32:50.793396+0000","last_became_active":"2024-03-09T14:28:32.337755+0000","last_became_peered":"2024-03-09T14:28:32.337755+0000","last_unstale":"2024-03-09T15:32:50.793396+0000","last_undegraded":"2024-03-09T15:32:50.793396+0000","last_fullsized":"2024-03-09T15:32:50.793396+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:19:21.576018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663899,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":272,"num_read_kb":3602,"num_write":640,"num_write_kb":11566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162410+0000","last_change":"2024-03-09T14:28:36.318326+0000","last_active":"2024-03-09T15:31:26.162410+0000","last_peered":"2024-03-09T15:31:26.162410+0000","last_clean":"2024-03-09T15:31:26.162410+0000","last_became_active":"2024-03-09T14:28:36.317569+0000","last_became_peered":"2024-03-09T14:28:36.317569+0000","last_unstale":"2024-03-09T15:31:26.162410+0000","last_undegraded":"2024-03-09T15:31:26.162410+0000","last_fullsized":"2024-03-09T15:31:26.162410+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:21:30.385778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"261'149","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.708300+0000","last_change":"2024-03-09T14:28:38.348095+0000","last_active":"2024-03-09T15:35:58.708300+0000","last_peered":"2024-03-09T15:35:58.708300+0000","last_clean":"2024-03-09T15:35:58.708300+0000","last_became_active":"2024-03-09T14:28:38.347370+0000","last_became_peered":"2024-03-09T14:28:38.347370+0000","last_unstale":"2024-03-09T15:35:58.708300+0000","last_undegraded":"2024-03-09T15:35:58.708300+0000","last_fullsized":"2024-03-09T15:35:58.708300+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:00:09.043640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32290,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":236,"num_read_kb":490,"num_write":168,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":495,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:11.159141+0000","last_change":"2024-03-09T14:28:40.377364+0000","last_active":"2024-03-09T15:31:11.159141+0000","last_peered":"2024-03-09T15:31:11.159141+0000","last_clean":"2024-03-09T15:31:11.159141+0000","last_became_active":"2024-03-09T14:28:40.376513+0000","last_became_peered":"2024-03-09T14:28:40.376513+0000","last_unstale":"2024-03-09T15:31:11.159141+0000","last_undegraded":"2024-03-09T15:31:11.159141+0000","last_fullsized":"2024-03-09T15:31:11.159141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:28.590369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"261'92","reported_seq":974,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:54.162220+0000","last_change":"2024-03-09T14:28:42.410566+0000","last_active":"2024-03-09T15:35:54.162220+0000","last_peered":"2024-03-09T15:35:54.162220+0000","last_clean":"2024-03-09T15:35:54.162220+0000","last_became_active":"2024-03-09T14:28:42.409717+0000","last_became_peered":"2024-03-09T14:28:42.409717+0000","last_unstale":"2024-03-09T15:35:54.162220+0000","last_undegraded":"2024-03-09T15:35:54.162220+0000","last_fullsized":"2024-03-09T15:35:54.162220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:07:09.353068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":500,"num_read_kb":493,"num_write":215,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"261'53","reported_seq":585,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162448+0000","last_change":"2024-03-09T14:28:34.314180+0000","last_active":"2024-03-09T15:31:26.162448+0000","last_peered":"2024-03-09T15:31:26.162448+0000","last_clean":"2024-03-09T15:31:26.162448+0000","last_became_active":"2024-03-09T14:28:34.313380+0000","last_became_peered":"2024-03-09T14:28:34.313380+0000","last_unstale":"2024-03-09T15:31:26.162448+0000","last_undegraded":"2024-03-09T15:31:26.162448+0000","last_fullsized":"2024-03-09T15:31:26.162448+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:29:27.979809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":197,"num_read_kb":156,"num_write":74,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"261'146","reported_seq":733,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.965677+0000","last_change":"2024-03-09T14:28:32.293553+0000","last_active":"2024-03-09T15:36:05.965677+0000","last_peered":"2024-03-09T15:36:05.965677+0000","last_clean":"2024-03-09T15:36:05.965677+0000","last_became_active":"2024-03-09T14:28:32.292862+0000","last_became_peered":"2024-03-09T14:28:32.292862+0000","last_unstale":"2024-03-09T15:36:05.965677+0000","last_undegraded":"2024-03-09T15:36:05.965677+0000","last_fullsized":"2024-03-09T15:36:05.965677+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:21:35.805317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2792030,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":249,"num_read_kb":35693,"num_write":653,"num_write_kb":24072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"25'1","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.132760+0000","last_change":"2024-03-09T14:28:36.314323+0000","last_active":"2024-03-09T15:31:31.132760+0000","last_peered":"2024-03-09T15:31:31.132760+0000","last_clean":"2024-03-09T15:31:31.132760+0000","last_became_active":"2024-03-09T14:28:36.313405+0000","last_became_peered":"2024-03-09T14:28:36.313405+0000","last_unstale":"2024-03-09T15:31:31.132760+0000","last_undegraded":"2024-03-09T15:31:31.132760+0000","last_fullsized":"2024-03-09T15:31:31.132760+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:46:02.220171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"261'207","reported_seq":799,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.716100+0000","last_change":"2024-03-09T14:28:38.333268+0000","last_active":"2024-03-09T15:35:58.716100+0000","last_peered":"2024-03-09T15:35:58.716100+0000","last_clean":"2024-03-09T15:35:58.716100+0000","last_became_active":"2024-03-09T14:28:38.330571+0000","last_became_peered":"2024-03-09T14:28:38.330571+0000","last_unstale":"2024-03-09T15:35:58.716100+0000","last_undegraded":"2024-03-09T15:35:58.716100+0000","last_fullsized":"2024-03-09T15:35:58.716100+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":100,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:13:06.110510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27588,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":340,"num_read_kb":612,"num_write":242,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":491,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:10.978895+0000","last_change":"2024-03-09T14:28:40.385742+0000","last_active":"2024-03-09T15:31:10.978895+0000","last_peered":"2024-03-09T15:31:10.978895+0000","last_clean":"2024-03-09T15:31:10.978895+0000","last_became_active":"2024-03-09T14:28:40.385080+0000","last_became_peered":"2024-03-09T14:28:40.385080+0000","last_unstale":"2024-03-09T15:31:10.978895+0000","last_undegraded":"2024-03-09T15:31:10.978895+0000","last_fullsized":"2024-03-09T15:31:10.978895+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:08:01.706715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"261'69","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:11.304332+0000","last_change":"2024-03-09T14:28:42.413260+0000","last_active":"2024-03-09T15:34:11.304332+0000","last_peered":"2024-03-09T15:34:11.304332+0000","last_clean":"2024-03-09T15:34:11.304332+0000","last_became_active":"2024-03-09T14:28:42.412616+0000","last_became_peered":"2024-03-09T14:28:42.412616+0000","last_unstale":"2024-03-09T15:34:11.304332+0000","last_undegraded":"2024-03-09T15:34:11.304332+0000","last_fullsized":"2024-03-09T15:34:11.304332+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:28:05.834021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":49,"num_write":163,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.27","version":"261'50","reported_seq":597,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963528+0000","last_change":"2024-03-09T14:28:34.307137+0000","last_active":"2024-03-09T15:33:35.963528+0000","last_peered":"2024-03-09T15:33:35.963528+0000","last_clean":"2024-03-09T15:33:35.963528+0000","last_became_active":"2024-03-09T14:28:34.306685+0000","last_became_peered":"2024-03-09T14:28:34.306685+0000","last_unstale":"2024-03-09T15:33:35.963528+0000","last_undegraded":"2024-03-09T15:33:35.963528+0000","last_fullsized":"2024-03-09T15:33:35.963528+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:19:14.669885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":204,"num_read_kb":164,"num_write":70,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"261'122","reported_seq":705,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:13.078377+0000","last_change":"2024-03-09T14:28:32.334098+0000","last_active":"2024-03-09T15:33:13.078377+0000","last_peered":"2024-03-09T15:33:13.078377+0000","last_clean":"2024-03-09T15:33:13.078377+0000","last_became_active":"2024-03-09T14:28:32.333008+0000","last_became_peered":"2024-03-09T14:28:32.333008+0000","last_unstale":"2024-03-09T15:33:13.078377+0000","last_undegraded":"2024-03-09T15:33:13.078377+0000","last_fullsized":"2024-03-09T15:33:13.078377+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:19:21.576018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1943763,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":219,"num_read_kb":10390,"num_write":536,"num_write_kb":33266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.26","version":"261'181","reported_seq":772,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963471+0000","last_change":"2024-03-09T14:28:34.314634+0000","last_active":"2024-03-09T15:33:35.963471+0000","last_peered":"2024-03-09T15:33:35.963471+0000","last_clean":"2024-03-09T15:33:35.963471+0000","last_became_active":"2024-03-09T14:28:34.314007+0000","last_became_peered":"2024-03-09T14:28:34.314007+0000","last_unstale":"2024-03-09T15:33:35.963471+0000","last_undegraded":"2024-03-09T15:33:35.963471+0000","last_fullsized":"2024-03-09T15:33:35.963471+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":100,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:50:26.398988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":729,"num_read_kb":586,"num_write":272,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"261'127","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.956327+0000","last_change":"2024-03-09T14:28:32.287770+0000","last_active":"2024-03-09T15:36:05.956327+0000","last_peered":"2024-03-09T15:36:05.956327+0000","last_clean":"2024-03-09T15:36:05.956327+0000","last_became_active":"2024-03-09T14:28:32.286609+0000","last_became_peered":"2024-03-09T14:28:32.286609+0000","last_unstale":"2024-03-09T15:36:05.956327+0000","last_undegraded":"2024-03-09T15:36:05.956327+0000","last_fullsized":"2024-03-09T15:36:05.956327+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":100,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:14.582228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":810412,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":129,"num_read_kb":1907,"num_write":474,"num_write_kb":32192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"261'93","reported_seq":657,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194646+0000","last_change":"2024-03-09T14:28:34.325388+0000","last_active":"2024-03-09T15:33:36.194646+0000","last_peered":"2024-03-09T15:33:36.194646+0000","last_clean":"2024-03-09T15:33:36.194646+0000","last_became_active":"2024-03-09T14:28:34.323541+0000","last_became_peered":"2024-03-09T14:28:34.323541+0000","last_unstale":"2024-03-09T15:33:36.194646+0000","last_undegraded":"2024-03-09T15:33:36.194646+0000","last_fullsized":"2024-03-09T15:33:36.194646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:35:47.676249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":384,"num_read_kb":306,"num_write":146,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"261'160","reported_seq":756,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.074470+0000","last_change":"2024-03-09T14:28:32.299059+0000","last_active":"2024-03-09T15:36:06.074470+0000","last_peered":"2024-03-09T15:36:06.074470+0000","last_clean":"2024-03-09T15:36:06.074470+0000","last_became_active":"2024-03-09T14:28:32.295699+0000","last_became_peered":"2024-03-09T14:28:32.295699+0000","last_unstale":"2024-03-09T15:36:06.074470+0000","last_undegraded":"2024-03-09T15:36:06.074470+0000","last_fullsized":"2024-03-09T15:36:06.074470+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":100,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:14:50.303305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":286,"num_read_kb":14189,"num_write":705,"num_write_kb":33521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"261'101","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963269+0000","last_change":"2024-03-09T14:28:34.322186+0000","last_active":"2024-03-09T15:33:35.963269+0000","last_peered":"2024-03-09T15:33:35.963269+0000","last_clean":"2024-03-09T15:33:35.963269+0000","last_became_active":"2024-03-09T14:28:34.321647+0000","last_became_peered":"2024-03-09T14:28:34.321647+0000","last_unstale":"2024-03-09T15:33:35.963269+0000","last_undegraded":"2024-03-09T15:33:35.963269+0000","last_fullsized":"2024-03-09T15:33:35.963269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:19:44.897314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":329,"num_write":161,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"261'119","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.887640+0000","last_change":"2024-03-09T14:28:32.298327+0000","last_active":"2024-03-09T15:36:05.887640+0000","last_peered":"2024-03-09T15:36:05.887640+0000","last_clean":"2024-03-09T15:36:05.887640+0000","last_became_active":"2024-03-09T14:28:32.297866+0000","last_became_peered":"2024-03-09T14:28:32.297866+0000","last_unstale":"2024-03-09T15:36:05.887640+0000","last_undegraded":"2024-03-09T15:36:05.887640+0000","last_fullsized":"2024-03-09T15:36:05.887640+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":100,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:49:38.048255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":154,"num_read_kb":3467,"num_write":486,"num_write_kb":23815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"261'52","reported_seq":601,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.801400+0000","last_change":"2024-03-09T14:28:34.330916+0000","last_active":"2024-03-09T15:33:25.801400+0000","last_peered":"2024-03-09T15:33:25.801400+0000","last_clean":"2024-03-09T15:33:25.801400+0000","last_became_active":"2024-03-09T14:28:34.330511+0000","last_became_peered":"2024-03-09T14:28:34.330511+0000","last_unstale":"2024-03-09T15:33:25.801400+0000","last_undegraded":"2024-03-09T15:33:25.801400+0000","last_fullsized":"2024-03-09T15:33:25.801400+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:29:16.007058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":225,"num_read_kb":185,"num_write":82,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"261'102","reported_seq":651,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:52.427722+0000","last_change":"2024-03-09T14:28:32.326489+0000","last_active":"2024-03-09T15:33:52.427722+0000","last_peered":"2024-03-09T15:33:52.427722+0000","last_clean":"2024-03-09T15:33:52.427722+0000","last_became_active":"2024-03-09T14:28:32.326148+0000","last_became_peered":"2024-03-09T14:28:32.326148+0000","last_unstale":"2024-03-09T15:33:52.427722+0000","last_undegraded":"2024-03-09T15:33:52.427722+0000","last_fullsized":"2024-03-09T15:33:52.427722+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:59:13.121365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":157,"num_read_kb":5614,"num_write":435,"num_write_kb":21169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"261'134","reported_seq":685,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.185748+0000","last_change":"2024-03-09T14:28:34.323012+0000","last_active":"2024-03-09T15:33:11.185748+0000","last_peered":"2024-03-09T15:33:11.185748+0000","last_clean":"2024-03-09T15:33:11.185748+0000","last_became_active":"2024-03-09T14:28:34.322225+0000","last_became_peered":"2024-03-09T14:28:34.322225+0000","last_unstale":"2024-03-09T15:33:11.185748+0000","last_undegraded":"2024-03-09T15:33:11.185748+0000","last_fullsized":"2024-03-09T15:33:11.185748+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:33:31.665609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":498,"num_read_kb":399,"num_write":194,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"261'145","reported_seq":711,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:45.990374+0000","last_change":"2024-03-09T14:28:32.300246+0000","last_active":"2024-03-09T15:35:45.990374+0000","last_peered":"2024-03-09T15:35:45.990374+0000","last_clean":"2024-03-09T15:35:45.990374+0000","last_became_active":"2024-03-09T14:28:32.299839+0000","last_became_peered":"2024-03-09T14:28:32.299839+0000","last_unstale":"2024-03-09T15:35:45.990374+0000","last_undegraded":"2024-03-09T15:35:45.990374+0000","last_fullsized":"2024-03-09T15:35:45.990374+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:05:01.790489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243228,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":189,"num_read_kb":18833,"num_write":580,"num_write_kb":39169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"261'74","reported_seq":608,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:17.871898+0000","last_change":"2024-03-09T14:28:34.315670+0000","last_active":"2024-03-09T15:34:17.871898+0000","last_peered":"2024-03-09T15:34:17.871898+0000","last_clean":"2024-03-09T15:34:17.871898+0000","last_became_active":"2024-03-09T14:28:34.315302+0000","last_became_peered":"2024-03-09T14:28:34.315302+0000","last_unstale":"2024-03-09T15:34:17.871898+0000","last_undegraded":"2024-03-09T15:34:17.871898+0000","last_fullsized":"2024-03-09T15:34:17.871898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:54:03.674581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":224,"num_write":116,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"261'122","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.789837+0000","last_change":"2024-03-09T14:28:32.336827+0000","last_active":"2024-03-09T15:33:35.789837+0000","last_peered":"2024-03-09T15:33:35.789837+0000","last_clean":"2024-03-09T15:33:35.789837+0000","last_became_active":"2024-03-09T14:28:32.334117+0000","last_became_peered":"2024-03-09T14:28:32.334117+0000","last_unstale":"2024-03-09T15:33:35.789837+0000","last_undegraded":"2024-03-09T15:33:35.789837+0000","last_fullsized":"2024-03-09T15:33:35.789837+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:58:15.793689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243207,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":155,"num_read_kb":192,"num_write":536,"num_write_kb":14566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"261'48","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784320+0000","last_change":"2024-03-09T14:28:34.323974+0000","last_active":"2024-03-09T15:33:10.784320+0000","last_peered":"2024-03-09T15:33:10.784320+0000","last_clean":"2024-03-09T15:33:10.784320+0000","last_became_active":"2024-03-09T14:28:34.323202+0000","last_became_peered":"2024-03-09T14:28:34.323202+0000","last_unstale":"2024-03-09T15:33:10.784320+0000","last_undegraded":"2024-03-09T15:33:10.784320+0000","last_fullsized":"2024-03-09T15:33:10.784320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:33:12.688593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":219,"num_read_kb":183,"num_write":70,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"261'117","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.803914+0000","last_change":"2024-03-09T14:28:32.300258+0000","last_active":"2024-03-09T15:33:35.803914+0000","last_peered":"2024-03-09T15:33:35.803914+0000","last_clean":"2024-03-09T15:33:35.803914+0000","last_became_active":"2024-03-09T14:28:32.299737+0000","last_became_peered":"2024-03-09T14:28:32.299737+0000","last_unstale":"2024-03-09T15:33:35.803914+0000","last_undegraded":"2024-03-09T15:33:35.803914+0000","last_fullsized":"2024-03-09T15:33:35.803914+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:26:30.693287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6457661,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":147,"num_read_kb":12613,"num_write":500,"num_write_kb":31618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"261'94","reported_seq":644,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.984747+0000","last_change":"2024-03-09T14:28:34.315866+0000","last_active":"2024-03-09T15:31:35.984747+0000","last_peered":"2024-03-09T15:31:35.984747+0000","last_clean":"2024-03-09T15:31:35.984747+0000","last_became_active":"2024-03-09T14:28:34.315451+0000","last_became_peered":"2024-03-09T14:28:34.315451+0000","last_unstale":"2024-03-09T15:31:35.984747+0000","last_undegraded":"2024-03-09T15:31:35.984747+0000","last_fullsized":"2024-03-09T15:31:35.984747+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:50:21.781962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":374,"num_read_kb":293,"num_write":145,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"261'131","reported_seq":697,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.181679+0000","last_change":"2024-03-09T14:28:32.305393+0000","last_active":"2024-03-09T15:36:06.181679+0000","last_peered":"2024-03-09T15:36:06.181679+0000","last_clean":"2024-03-09T15:36:06.181679+0000","last_became_active":"2024-03-09T14:28:32.304540+0000","last_became_peered":"2024-03-09T14:28:32.304540+0000","last_unstale":"2024-03-09T15:36:06.181679+0000","last_undegraded":"2024-03-09T15:36:06.181679+0000","last_fullsized":"2024-03-09T15:36:06.181679+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":100,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:12:43.181100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1656073,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":187,"num_read_kb":1806,"num_write":496,"num_write_kb":18066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2e","version":"261'39","reported_seq":599,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.165250+0000","last_change":"2024-03-09T14:28:34.314876+0000","last_active":"2024-03-09T15:31:36.165250+0000","last_peered":"2024-03-09T15:31:36.165250+0000","last_clean":"2024-03-09T15:31:36.165250+0000","last_became_active":"2024-03-09T14:28:34.312399+0000","last_became_peered":"2024-03-09T14:28:34.312399+0000","last_unstale":"2024-03-09T15:31:36.165250+0000","last_undegraded":"2024-03-09T15:31:36.165250+0000","last_fullsized":"2024-03-09T15:31:36.165250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:27:43.779494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":182,"num_read_kb":156,"num_write":57,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"261'124","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.807282+0000","last_change":"2024-03-09T14:28:32.314908+0000","last_active":"2024-03-09T15:33:50.807282+0000","last_peered":"2024-03-09T15:33:50.807282+0000","last_clean":"2024-03-09T15:33:50.807282+0000","last_became_active":"2024-03-09T14:28:32.314596+0000","last_became_peered":"2024-03-09T14:28:32.314596+0000","last_unstale":"2024-03-09T15:33:50.807282+0000","last_undegraded":"2024-03-09T15:33:50.807282+0000","last_fullsized":"2024-03-09T15:33:50.807282+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:22:39.392181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506330,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":127,"num_read_kb":12793,"num_write":440,"num_write_kb":35350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"261'64","reported_seq":625,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:56.824631+0000","last_change":"2024-03-09T14:28:34.325871+0000","last_active":"2024-03-09T15:34:56.824631+0000","last_peered":"2024-03-09T15:34:56.824631+0000","last_clean":"2024-03-09T15:34:56.824631+0000","last_became_active":"2024-03-09T14:28:34.325461+0000","last_became_peered":"2024-03-09T14:28:34.325461+0000","last_unstale":"2024-03-09T15:34:56.824631+0000","last_undegraded":"2024-03-09T15:34:56.824631+0000","last_fullsized":"2024-03-09T15:34:56.824631+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:02:18.977660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":264,"num_read_kb":214,"num_write":89,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"261'141","reported_seq":717,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.191880+0000","last_change":"2024-03-09T14:28:32.313085+0000","last_active":"2024-03-09T15:36:06.191880+0000","last_peered":"2024-03-09T15:36:06.191880+0000","last_clean":"2024-03-09T15:36:06.191880+0000","last_became_active":"2024-03-09T14:28:32.311750+0000","last_became_peered":"2024-03-09T14:28:32.311750+0000","last_unstale":"2024-03-09T15:36:06.191880+0000","last_undegraded":"2024-03-09T15:36:06.191880+0000","last_fullsized":"2024-03-09T15:36:06.191880+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":100,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:56:43.188934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":195,"num_read_kb":14896,"num_write":515,"num_write_kb":41774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"261'162","reported_seq":774,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195666+0000","last_change":"2024-03-09T14:28:34.313344+0000","last_active":"2024-03-09T15:33:51.195666+0000","last_peered":"2024-03-09T15:33:51.195666+0000","last_clean":"2024-03-09T15:33:51.195666+0000","last_became_active":"2024-03-09T14:28:34.312212+0000","last_became_peered":"2024-03-09T14:28:34.312212+0000","last_unstale":"2024-03-09T15:33:51.195666+0000","last_undegraded":"2024-03-09T15:33:51.195666+0000","last_fullsized":"2024-03-09T15:33:51.195666+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":100,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:46:46.876109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":664,"num_read_kb":532,"num_write":243,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2b","version":"261'133","reported_seq":705,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.985790+0000","last_change":"2024-03-09T14:28:32.309913+0000","last_active":"2024-03-09T15:36:05.985790+0000","last_peered":"2024-03-09T15:36:05.985790+0000","last_clean":"2024-03-09T15:36:05.985790+0000","last_became_active":"2024-03-09T14:28:32.309562+0000","last_became_peered":"2024-03-09T14:28:32.309562+0000","last_unstale":"2024-03-09T15:36:05.985790+0000","last_undegraded":"2024-03-09T15:36:05.985790+0000","last_fullsized":"2024-03-09T15:36:05.985790+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:47:28.794070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":227,"num_read_kb":5348,"num_write":554,"num_write_kb":20887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"261'117","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.208966+0000","last_change":"2024-03-09T14:28:34.322800+0000","last_active":"2024-03-09T15:33:51.208966+0000","last_peered":"2024-03-09T15:33:51.208966+0000","last_clean":"2024-03-09T15:33:51.208966+0000","last_became_active":"2024-03-09T14:28:34.321424+0000","last_became_peered":"2024-03-09T14:28:34.321424+0000","last_unstale":"2024-03-09T15:33:51.208966+0000","last_undegraded":"2024-03-09T15:33:51.208966+0000","last_fullsized":"2024-03-09T15:33:51.208966+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:39:19.495854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":513,"num_read_kb":409,"num_write":169,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"261'142","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:09.453343+0000","last_change":"2024-03-09T14:28:32.334610+0000","last_active":"2024-03-09T15:34:09.453343+0000","last_peered":"2024-03-09T15:34:09.453343+0000","last_clean":"2024-03-09T15:34:09.453343+0000","last_became_active":"2024-03-09T14:28:32.333838+0000","last_became_peered":"2024-03-09T14:28:32.333838+0000","last_unstale":"2024-03-09T15:34:09.453343+0000","last_undegraded":"2024-03-09T15:34:09.453343+0000","last_fullsized":"2024-03-09T15:34:09.453343+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":100,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:14:50.303305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":218,"num_read_kb":3315,"num_write":653,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2a","version":"261'71","reported_seq":635,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.016279+0000","last_change":"2024-03-09T14:28:34.297744+0000","last_active":"2024-03-09T15:33:51.016279+0000","last_peered":"2024-03-09T15:33:51.016279+0000","last_clean":"2024-03-09T15:33:51.016279+0000","last_became_active":"2024-03-09T14:28:34.297042+0000","last_became_peered":"2024-03-09T14:28:34.297042+0000","last_unstale":"2024-03-09T15:33:51.016279+0000","last_undegraded":"2024-03-09T15:33:51.016279+0000","last_fullsized":"2024-03-09T15:33:51.016279+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:17:51.890278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":242,"num_write":99,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"261'111","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.167856+0000","last_change":"2024-03-09T14:28:32.320093+0000","last_active":"2024-03-09T15:36:06.167856+0000","last_peered":"2024-03-09T15:36:06.167856+0000","last_clean":"2024-03-09T15:36:06.167856+0000","last_became_active":"2024-03-09T14:28:32.317797+0000","last_became_peered":"2024-03-09T14:28:32.317797+0000","last_unstale":"2024-03-09T15:36:06.167856+0000","last_undegraded":"2024-03-09T15:36:06.167856+0000","last_fullsized":"2024-03-09T15:36:06.167856+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:52:33.454224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312437,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":176,"num_read_kb":4139,"num_write":464,"num_write_kb":19145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"261'113","reported_seq":668,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.807317+0000","last_change":"2024-03-09T14:28:34.314621+0000","last_active":"2024-03-09T15:33:50.807317+0000","last_peered":"2024-03-09T15:33:50.807317+0000","last_clean":"2024-03-09T15:33:50.807317+0000","last_became_active":"2024-03-09T14:28:34.314073+0000","last_became_peered":"2024-03-09T14:28:34.314073+0000","last_unstale":"2024-03-09T15:33:50.807317+0000","last_undegraded":"2024-03-09T15:33:50.807317+0000","last_fullsized":"2024-03-09T15:33:50.807317+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":100,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:30:53.101187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":445,"num_read_kb":351,"num_write":167,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"261'117","reported_seq":693,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.959588+0000","last_change":"2024-03-09T14:28:32.334618+0000","last_active":"2024-03-09T15:36:05.959588+0000","last_peered":"2024-03-09T15:36:05.959588+0000","last_clean":"2024-03-09T15:36:05.959588+0000","last_became_active":"2024-03-09T14:28:32.334187+0000","last_became_peered":"2024-03-09T14:28:32.334187+0000","last_unstale":"2024-03-09T15:36:05.959588+0000","last_undegraded":"2024-03-09T15:36:05.959588+0000","last_fullsized":"2024-03-09T15:36:05.959588+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:12:37.415618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2181393,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":207,"num_read_kb":1842,"num_write":544,"num_write_kb":7884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"261'4028","reported_seq":4579,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:46.207813+0000","last_change":"2024-03-09T14:28:34.324195+0000","last_active":"2024-03-09T15:33:46.207813+0000","last_peered":"2024-03-09T15:33:46.207813+0000","last_clean":"2024-03-09T15:33:46.207813+0000","last_became_active":"2024-03-09T14:28:34.322421+0000","last_became_peered":"2024-03-09T14:28:34.322421+0000","last_unstale":"2024-03-09T15:33:46.207813+0000","last_undegraded":"2024-03-09T15:33:46.207813+0000","last_fullsized":"2024-03-09T15:33:46.207813+0000","mapping_epoch":22,"log_start":"261'4000","ondisk_log_start":"261'4000","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:55:11.911331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14144,"num_read_kb":11461,"num_write":6038,"num_write_kb":3015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"261'118","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.139262+0000","last_change":"2024-03-09T14:28:32.324991+0000","last_active":"2024-03-09T15:36:06.139262+0000","last_peered":"2024-03-09T15:36:06.139262+0000","last_clean":"2024-03-09T15:36:06.139262+0000","last_became_active":"2024-03-09T14:28:32.324166+0000","last_became_peered":"2024-03-09T14:28:32.324166+0000","last_unstale":"2024-03-09T15:36:06.139262+0000","last_undegraded":"2024-03-09T15:36:06.139262+0000","last_fullsized":"2024-03-09T15:36:06.139262+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:55:14.982039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272031,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":186,"num_read_kb":9428,"num_write":503,"num_write_kb":30713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"261'126","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.159876+0000","last_change":"2024-03-09T14:28:32.288697+0000","last_active":"2024-03-09T15:36:06.159876+0000","last_peered":"2024-03-09T15:36:06.159876+0000","last_clean":"2024-03-09T15:36:06.159876+0000","last_became_active":"2024-03-09T14:28:32.287937+0000","last_became_peered":"2024-03-09T14:28:32.287937+0000","last_unstale":"2024-03-09T15:36:06.159876+0000","last_undegraded":"2024-03-09T15:36:06.159876+0000","last_fullsized":"2024-03-09T15:36:06.159876+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":100,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:01:00.707624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243238,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":199,"num_read_kb":48289,"num_write":444,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"261'161","reported_seq":726,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.985994+0000","last_change":"2024-03-09T14:28:34.317008+0000","last_active":"2024-03-09T15:31:40.985994+0000","last_peered":"2024-03-09T15:31:40.985994+0000","last_clean":"2024-03-09T15:31:40.985994+0000","last_became_active":"2024-03-09T14:28:34.316689+0000","last_became_peered":"2024-03-09T14:28:34.316689+0000","last_unstale":"2024-03-09T15:31:40.985994+0000","last_undegraded":"2024-03-09T15:31:40.985994+0000","last_fullsized":"2024-03-09T15:31:40.985994+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:22:45.346641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":634,"num_read_kb":490,"num_write":268,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"261'144","reported_seq":722,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.889143+0000","last_change":"2024-03-09T14:28:32.313446+0000","last_active":"2024-03-09T15:36:05.889143+0000","last_peered":"2024-03-09T15:36:05.889143+0000","last_clean":"2024-03-09T15:36:05.889143+0000","last_became_active":"2024-03-09T14:28:32.313132+0000","last_became_peered":"2024-03-09T14:28:32.313132+0000","last_unstale":"2024-03-09T15:36:05.889143+0000","last_undegraded":"2024-03-09T15:36:05.889143+0000","last_fullsized":"2024-03-09T15:36:05.889143+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:30:18.199314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262382,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":242,"num_read_kb":5401,"num_write":622,"num_write_kb":32648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"261'41","reported_seq":586,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:02.721943+0000","last_change":"2024-03-09T14:28:34.304440+0000","last_active":"2024-03-09T15:34:02.721943+0000","last_peered":"2024-03-09T15:34:02.721943+0000","last_clean":"2024-03-09T15:34:02.721943+0000","last_became_active":"2024-03-09T14:28:34.303701+0000","last_became_peered":"2024-03-09T14:28:34.303701+0000","last_unstale":"2024-03-09T15:34:02.721943+0000","last_undegraded":"2024-03-09T15:34:02.721943+0000","last_fullsized":"2024-03-09T15:34:02.721943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:22:10.704885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":166,"num_read_kb":138,"num_write":64,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":65,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3686,"num_read_kb":2614,"num_write":1236,"num_write_kb":514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":13176,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":8518,"num_read_kb":8294,"num_write":8580,"num_write_kb":2546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":26352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2011,"ondisk_log_size":2011,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":578,"ondisk_log_size":578,"up":64,"acting":64,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":725340,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":11980,"num_read_kb":20116,"num_write":8932,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":743659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2014,"ondisk_log_size":2014,"up":64,"acting":64,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1719,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1602,"num_read_kb":1903,"num_write":22,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":38590,"num_read_kb":31236,"num_write":15314,"num_write_kb":7493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3941,"ondisk_log_size":3941,"up":128,"acting":128,"num_store_stats":8},{"poolid":68,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":210202860,"num_objects":1447,"num_object_clones":0,"num_object_copies":2894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1447,"num_whiteouts":0,"num_read":12930,"num_read_kb":1121153,"num_write":33168,"num_write_kb":1701153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440426496,"data_stored":430130434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2921,"ondisk_log_size":2921,"up":128,"acting":128,"num_store_stats":8},{"poolid":67,"num_pg":32,"stat_sum":{"num_bytes":81879532,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":124,"num_read_kb":192739,"num_write":140,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":160620544,"data_stored":160613336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":66,"num_pg":32,"stat_sum":{"num_bytes":82877649,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":122,"num_read_kb":192649,"num_write":140,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":170004480,"data_stored":170000061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575670,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334148,"kb_used_data":104444,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":94037692,"statfs":{"total":96636764160,"available":96294596608,"internally_reserved":0,"allocated":106950656,"data_stored":103672380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":235207114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575666,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292416,"kb_used_data":84920,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":94079424,"statfs":{"total":96636764160,"available":96337330176,"internally_reserved":0,"allocated":86958080,"data_stored":84225759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575667,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":174444,"kb_used_data":121060,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94197396,"statfs":{"total":96636764160,"available":96458133504,"internally_reserved":0,"allocated":123965440,"data_stored":120640043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575671,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":197704,"kb_used_data":140160,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94174136,"statfs":{"total":96636764160,"available":96434315264,"internally_reserved":0,"allocated":143523840,"data_stored":140316730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575666,"num_pgs":118,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":343308,"kb_used_data":135812,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":94028532,"statfs":{"total":96636764160,"available":96285216768,"internally_reserved":0,"allocated":139071488,"data_stored":135606966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":212466119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575668,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133712,"kb_used_data":80328,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94238128,"statfs":{"total":96636764160,"available":96499843072,"internally_reserved":0,"allocated":82255872,"data_stored":79602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575667,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":186168,"kb_used_data":100912,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94185672,"statfs":{"total":96636764160,"available":96446128128,"internally_reserved":0,"allocated":103333888,"data_stored":100615986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575669,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339852,"kb_used_data":137924,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":94031988,"statfs":{"total":96636764160,"available":96288755712,"internally_reserved":0,"allocated":141234176,"data_stored":137929952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73302016,"data_stored":71838616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41046016,"data_stored":40051454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":44478464,"data_stored":43472543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":64929792,"data_stored":63014498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":77041664,"data_stored":75526368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61636608,"data_stored":60503814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":33976320,"data_stored":32726952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":44015616,"data_stored":42996189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":92713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":120489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":29353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":164295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":29815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":115473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":83356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":108165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":3901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":2727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24158208,"data_stored":24157393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25219072,"data_stored":25216283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13631488,"data_stored":13631488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30449664,"data_stored":30448849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36700160,"data_stored":36700160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":25165824,"data_stored":25165824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":14680064,"data_stored":14680064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20017152,"data_stored":20013548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24117248,"data_stored":24117248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19922944,"data_stored":19922944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39845888,"data_stored":39845888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5337088,"data_stored":5333484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":22020096,"data_stored":22020096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24117248,"data_stored":24117248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T15:36:08.451 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-09T15:36:08.451 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T15:36:08.862 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:36:08.874 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-09T15:36:08.885 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":2291,"stamp":"2024-03-09T15:36:07.248269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376290644,"num_objects":1788,"num_object_clones":0,"num_object_copies":3576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1788,"num_whiteouts":0,"num_read":77680,"num_read_kb":1570816,"num_write":67709,"num_write_kb":1988128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12459,"ondisk_log_size":12459,"up":786,"acting":786,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":786,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2001752,"kb_used_data":905560,"kb_used_omap":12,"kb_used_meta":1096115,"kb_avail":752972968,"statfs":{"total":773094113280,"available":771044319232,"internally_reserved":0,"allocated":927293440,"data_stored":902610116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1122422320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-66869968,"num_objects":-307,"num_object_clones":0,"num_object_copies":-614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-307,"num_whiteouts":0,"num_read":462,"num_read_kb":498,"num_write":368,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007256"},"pg_stats":[{"pgid":"68.1a","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.178131+0000","last_change":"2024-03-09T15:09:50.085348+0000","last_active":"2024-03-09T15:32:21.178131+0000","last_peered":"2024-03-09T15:32:21.178131+0000","last_clean":"2024-03-09T15:32:21.178131+0000","last_became_active":"2024-03-09T15:09:50.084784+0000","last_became_peered":"2024-03-09T15:09:50.084784+0000","last_unstale":"2024-03-09T15:32:21.178131+0000","last_undegraded":"2024-03-09T15:32:21.178131+0000","last_fullsized":"2024-03-09T15:32:21.178131+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:18:17.114617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.1d","version":"258'7","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.167423+0000","last_change":"2024-03-09T14:46:26.826294+0000","last_active":"2024-03-09T15:31:31.167423+0000","last_peered":"2024-03-09T15:31:31.167423+0000","last_clean":"2024-03-09T15:31:31.167423+0000","last_became_active":"2024-03-09T14:46:26.825404+0000","last_became_peered":"2024-03-09T14:46:26.825404+0000","last_unstale":"2024-03-09T15:31:31.167423+0000","last_undegraded":"2024-03-09T15:31:31.167423+0000","last_fullsized":"2024-03-09T15:31:31.167423+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:28:03.324789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20498,"num_write":6,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1c","version":"261'1","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:20.773210+0000","last_change":"2024-03-09T14:46:18.769137+0000","last_active":"2024-03-09T15:31:20.773210+0000","last_peered":"2024-03-09T15:31:20.773210+0000","last_clean":"2024-03-09T15:31:20.773210+0000","last_became_active":"2024-03-09T14:46:18.768383+0000","last_became_peered":"2024-03-09T14:46:18.768383+0000","last_unstale":"2024-03-09T15:31:20.773210+0000","last_undegraded":"2024-03-09T15:31:20.773210+0000","last_fullsized":"2024-03-09T15:31:20.773210+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:38:13.467012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":3072,"num_write":1,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.1f","version":"258'45","reported_seq":167,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.010472+0000","last_change":"2024-03-09T14:39:13.282130+0000","last_active":"2024-03-09T15:33:26.010472+0000","last_peered":"2024-03-09T15:33:26.010472+0000","last_clean":"2024-03-09T15:33:26.010472+0000","last_became_active":"2024-03-09T14:39:13.281020+0000","last_became_peered":"2024-03-09T14:39:13.281020+0000","last_unstale":"2024-03-09T15:33:26.010472+0000","last_undegraded":"2024-03-09T15:33:26.010472+0000","last_fullsized":"2024-03-09T15:33:26.010472+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:54.492385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":134,"num_write":70,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1b","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.961650+0000","last_change":"2024-03-09T15:09:50.083262+0000","last_active":"2024-03-09T15:32:20.961650+0000","last_peered":"2024-03-09T15:32:20.961650+0000","last_clean":"2024-03-09T15:32:20.961650+0000","last_became_active":"2024-03-09T15:09:50.082690+0000","last_became_peered":"2024-03-09T15:09:50.082690+0000","last_unstale":"2024-03-09T15:32:20.961650+0000","last_undegraded":"2024-03-09T15:32:20.961650+0000","last_fullsized":"2024-03-09T15:32:20.961650+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:18:09.424746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1c","version":"258'5","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162661+0000","last_change":"2024-03-09T14:46:26.807670+0000","last_active":"2024-03-09T15:31:26.162661+0000","last_peered":"2024-03-09T15:31:26.162661+0000","last_clean":"2024-03-09T15:31:26.162661+0000","last_became_active":"2024-03-09T14:46:26.806865+0000","last_became_peered":"2024-03-09T14:46:26.806865+0000","last_unstale":"2024-03-09T15:31:26.162661+0000","last_undegraded":"2024-03-09T15:31:26.162661+0000","last_fullsized":"2024-03-09T15:31:26.162661+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:05:04.558231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.1d","version":"261'1","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:21.130434+0000","last_change":"2024-03-09T14:46:18.780095+0000","last_active":"2024-03-09T15:31:21.130434+0000","last_peered":"2024-03-09T15:31:21.130434+0000","last_clean":"2024-03-09T15:31:21.130434+0000","last_became_active":"2024-03-09T14:46:18.779760+0000","last_became_peered":"2024-03-09T14:46:18.779760+0000","last_unstale":"2024-03-09T15:31:21.130434+0000","last_undegraded":"2024-03-09T15:31:21.130434+0000","last_fullsized":"2024-03-09T15:31:21.130434+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:12.211222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":1,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1e","version":"0'0","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:30.975652+0000","last_change":"2024-03-09T14:39:13.260541+0000","last_active":"2024-03-09T15:33:30.975652+0000","last_peered":"2024-03-09T15:33:30.975652+0000","last_clean":"2024-03-09T15:33:30.975652+0000","last_became_active":"2024-03-09T14:39:13.259731+0000","last_became_peered":"2024-03-09T14:39:13.259731+0000","last_unstale":"2024-03-09T15:33:30.975652+0000","last_undegraded":"2024-03-09T15:33:30.975652+0000","last_fullsized":"2024-03-09T15:33:30.975652+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:09:28.555448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.18","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.764910+0000","last_change":"2024-03-09T15:09:50.081224+0000","last_active":"2024-03-09T15:31:35.764910+0000","last_peered":"2024-03-09T15:31:35.764910+0000","last_clean":"2024-03-09T15:31:35.764910+0000","last_became_active":"2024-03-09T15:09:50.080578+0000","last_became_peered":"2024-03-09T15:09:50.080578+0000","last_unstale":"2024-03-09T15:31:35.764910+0000","last_undegraded":"2024-03-09T15:31:35.764910+0000","last_fullsized":"2024-03-09T15:31:35.764910+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:54:04.030480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.1f","version":"261'4","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.995705+0000","last_change":"2024-03-09T14:46:26.896619+0000","last_active":"2024-03-09T15:32:20.995705+0000","last_peered":"2024-03-09T15:32:20.995705+0000","last_clean":"2024-03-09T15:32:20.995705+0000","last_became_active":"2024-03-09T14:46:26.895608+0000","last_became_peered":"2024-03-09T14:46:26.895608+0000","last_unstale":"2024-03-09T15:32:20.995705+0000","last_undegraded":"2024-03-09T15:32:20.995705+0000","last_fullsized":"2024-03-09T15:32:20.995705+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:32:06.502138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":12288,"num_write":4,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1e","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:35.964975+0000","last_change":"2024-03-09T14:46:18.782402+0000","last_active":"2024-03-09T15:32:35.964975+0000","last_peered":"2024-03-09T15:32:35.964975+0000","last_clean":"2024-03-09T15:32:35.964975+0000","last_became_active":"2024-03-09T14:46:18.781856+0000","last_became_peered":"2024-03-09T14:46:18.781856+0000","last_unstale":"2024-03-09T15:32:35.964975+0000","last_undegraded":"2024-03-09T15:32:35.964975+0000","last_fullsized":"2024-03-09T15:32:35.964975+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:29:15.829032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.1d","version":"258'35","reported_seq":142,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.987080+0000","last_change":"2024-03-09T14:39:13.265101+0000","last_active":"2024-03-09T15:31:45.987080+0000","last_peered":"2024-03-09T15:31:45.987080+0000","last_clean":"2024-03-09T15:31:45.987080+0000","last_became_active":"2024-03-09T14:39:13.258351+0000","last_became_peered":"2024-03-09T14:39:13.258351+0000","last_unstale":"2024-03-09T15:31:45.987080+0000","last_undegraded":"2024-03-09T15:31:45.987080+0000","last_fullsized":"2024-03-09T15:31:45.987080+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:21:02.690940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":152,"num_read_kb":109,"num_write":57,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.19","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.777876+0000","last_change":"2024-03-09T15:09:50.083658+0000","last_active":"2024-03-09T15:31:40.777876+0000","last_peered":"2024-03-09T15:31:40.777876+0000","last_clean":"2024-03-09T15:31:40.777876+0000","last_became_active":"2024-03-09T15:09:50.081680+0000","last_became_peered":"2024-03-09T15:09:50.081680+0000","last_unstale":"2024-03-09T15:31:40.777876+0000","last_undegraded":"2024-03-09T15:31:40.777876+0000","last_fullsized":"2024-03-09T15:31:40.777876+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:56:53.220664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.1e","version":"261'4","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:21.009093+0000","last_change":"2024-03-09T14:47:07.798858+0000","last_active":"2024-03-09T15:33:21.009093+0000","last_peered":"2024-03-09T15:33:21.009093+0000","last_clean":"2024-03-09T15:33:21.009093+0000","last_became_active":"2024-03-09T14:47:07.798247+0000","last_became_peered":"2024-03-09T14:47:07.798247+0000","last_unstale":"2024-03-09T15:33:21.009093+0000","last_undegraded":"2024-03-09T15:33:21.009093+0000","last_fullsized":"2024-03-09T15:33:21.009093+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:45:56.993617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":16384,"num_write":9,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1f","version":"258'5","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.173075+0000","last_change":"2024-03-09T14:46:18.772485+0000","last_active":"2024-03-09T15:32:11.173075+0000","last_peered":"2024-03-09T15:32:11.173075+0000","last_clean":"2024-03-09T15:32:11.173075+0000","last_became_active":"2024-03-09T14:46:18.771731+0000","last_became_peered":"2024-03-09T14:46:18.771731+0000","last_unstale":"2024-03-09T15:32:11.173075+0000","last_undegraded":"2024-03-09T15:32:11.173075+0000","last_fullsized":"2024-03-09T15:32:11.173075+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:50:44.589948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5136,"num_write":5,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1c","version":"261'13","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.779361+0000","last_change":"2024-03-09T14:39:13.266571+0000","last_active":"2024-03-09T15:31:45.779361+0000","last_peered":"2024-03-09T15:31:45.779361+0000","last_clean":"2024-03-09T15:31:45.779361+0000","last_became_active":"2024-03-09T14:39:13.260945+0000","last_became_peered":"2024-03-09T14:39:13.260945+0000","last_unstale":"2024-03-09T15:31:45.779361+0000","last_undegraded":"2024-03-09T15:31:45.779361+0000","last_fullsized":"2024-03-09T15:31:45.779361+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:58:40.520455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":27,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1e","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.808730+0000","last_change":"2024-03-09T15:09:50.088230+0000","last_active":"2024-03-09T15:33:55.808730+0000","last_peered":"2024-03-09T15:33:55.808730+0000","last_clean":"2024-03-09T15:33:55.808730+0000","last_became_active":"2024-03-09T15:09:50.086789+0000","last_became_peered":"2024-03-09T15:09:50.086789+0000","last_unstale":"2024-03-09T15:33:55.808730+0000","last_undegraded":"2024-03-09T15:33:55.808730+0000","last_fullsized":"2024-03-09T15:33:55.808730+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:54:27.433652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.19","version":"261'6","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.109749+0000","last_change":"2024-03-09T14:46:26.819845+0000","last_active":"2024-03-09T15:36:06.109749+0000","last_peered":"2024-03-09T15:36:06.109749+0000","last_clean":"2024-03-09T15:36:06.109749+0000","last_became_active":"2024-03-09T14:46:26.819085+0000","last_became_peered":"2024-03-09T14:46:26.819085+0000","last_unstale":"2024-03-09T15:36:06.109749+0000","last_undegraded":"2024-03-09T15:36:06.109749+0000","last_fullsized":"2024-03-09T15:36:06.109749+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:17.867538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":12288,"num_write":11,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.18","version":"261'12","reported_seq":78,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:41.027571+0000","last_change":"2024-03-09T14:46:18.776960+0000","last_active":"2024-03-09T15:34:41.027571+0000","last_peered":"2024-03-09T15:34:41.027571+0000","last_clean":"2024-03-09T15:34:41.027571+0000","last_became_active":"2024-03-09T14:46:18.776161+0000","last_became_peered":"2024-03-09T14:46:18.776161+0000","last_unstale":"2024-03-09T15:34:41.027571+0000","last_undegraded":"2024-03-09T15:34:41.027571+0000","last_fullsized":"2024-03-09T15:34:41.027571+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:22:11.563005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":36889,"num_write":16,"num_write_kb":16407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.1b","version":"258'10","reported_seq":79,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.779245+0000","last_change":"2024-03-09T14:39:13.269298+0000","last_active":"2024-03-09T15:31:45.779245+0000","last_peered":"2024-03-09T15:31:45.779245+0000","last_clean":"2024-03-09T15:31:45.779245+0000","last_became_active":"2024-03-09T14:39:13.268179+0000","last_became_peered":"2024-03-09T14:39:13.268179+0000","last_unstale":"2024-03-09T15:31:45.779245+0000","last_undegraded":"2024-03-09T15:31:45.779245+0000","last_fullsized":"2024-03-09T15:31:45.779245+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:12:05.852927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1f","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:45.951375+0000","last_change":"2024-03-09T15:09:50.064176+0000","last_active":"2024-03-09T15:32:45.951375+0000","last_peered":"2024-03-09T15:32:45.951375+0000","last_clean":"2024-03-09T15:32:45.951375+0000","last_became_active":"2024-03-09T15:09:50.063705+0000","last_became_peered":"2024-03-09T15:09:50.063705+0000","last_unstale":"2024-03-09T15:32:45.951375+0000","last_undegraded":"2024-03-09T15:32:45.951375+0000","last_fullsized":"2024-03-09T15:32:45.951375+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:45.866476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.18","version":"261'2","reported_seq":46,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.775288+0000","last_change":"2024-03-09T14:46:26.824830+0000","last_active":"2024-03-09T15:32:25.775288+0000","last_peered":"2024-03-09T15:32:25.775288+0000","last_clean":"2024-03-09T15:32:25.775288+0000","last_became_active":"2024-03-09T14:46:26.824522+0000","last_became_peered":"2024-03-09T14:46:26.824522+0000","last_unstale":"2024-03-09T15:32:25.775288+0000","last_undegraded":"2024-03-09T15:32:25.775288+0000","last_fullsized":"2024-03-09T15:32:25.775288+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:09:09.414799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":2,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.19","version":"261'7","reported_seq":64,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.175135+0000","last_change":"2024-03-09T14:46:18.770577+0000","last_active":"2024-03-09T15:32:21.175135+0000","last_peered":"2024-03-09T15:32:21.175135+0000","last_clean":"2024-03-09T15:32:21.175135+0000","last_became_active":"2024-03-09T14:46:18.769730+0000","last_became_peered":"2024-03-09T14:46:18.769730+0000","last_unstale":"2024-03-09T15:32:21.175135+0000","last_undegraded":"2024-03-09T15:32:21.175135+0000","last_fullsized":"2024-03-09T15:32:21.175135+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:07:45.259457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340032,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":19456,"num_write":17,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1a","version":"261'44","reported_seq":164,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.767114+0000","last_change":"2024-03-09T14:39:13.274060+0000","last_active":"2024-03-09T15:31:45.767114+0000","last_peered":"2024-03-09T15:31:45.767114+0000","last_clean":"2024-03-09T15:31:45.767114+0000","last_became_active":"2024-03-09T14:39:13.273298+0000","last_became_peered":"2024-03-09T14:39:13.273298+0000","last_unstale":"2024-03-09T15:31:45.767114+0000","last_undegraded":"2024-03-09T15:31:45.767114+0000","last_fullsized":"2024-03-09T15:31:45.767114+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:16:28.025129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":138,"num_write":72,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.1c","version":"261'1","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188241+0000","last_change":"2024-03-09T15:09:50.075720+0000","last_active":"2024-03-09T15:32:21.188241+0000","last_peered":"2024-03-09T15:32:21.188241+0000","last_clean":"2024-03-09T15:32:21.188241+0000","last_became_active":"2024-03-09T15:09:50.074952+0000","last_became_peered":"2024-03-09T15:09:50.074952+0000","last_unstale":"2024-03-09T15:32:21.188241+0000","last_undegraded":"2024-03-09T15:32:21.188241+0000","last_fullsized":"2024-03-09T15:32:21.188241+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:21:35.439413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1b","version":"261'2","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:30.763859+0000","last_change":"2024-03-09T14:46:26.805382+0000","last_active":"2024-03-09T15:31:30.763859+0000","last_peered":"2024-03-09T15:31:30.763859+0000","last_clean":"2024-03-09T15:31:30.763859+0000","last_became_active":"2024-03-09T14:46:26.804622+0000","last_became_peered":"2024-03-09T14:46:26.804622+0000","last_unstale":"2024-03-09T15:31:30.763859+0000","last_undegraded":"2024-03-09T15:31:30.763859+0000","last_fullsized":"2024-03-09T15:31:30.763859+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:23.446296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":5120,"num_write":2,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.1a","version":"261'4","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:21.161277+0000","last_change":"2024-03-09T14:46:18.785531+0000","last_active":"2024-03-09T15:31:21.161277+0000","last_peered":"2024-03-09T15:31:21.161277+0000","last_clean":"2024-03-09T15:31:21.161277+0000","last_became_active":"2024-03-09T14:46:18.784821+0000","last_became_peered":"2024-03-09T14:46:18.784821+0000","last_unstale":"2024-03-09T15:31:21.161277+0000","last_undegraded":"2024-03-09T15:31:21.161277+0000","last_fullsized":"2024-03-09T15:31:21.161277+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:10:34.919135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4096,"num_write":9,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.19","version":"258'28","reported_seq":120,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:40.999888+0000","last_change":"2024-03-09T14:39:13.258289+0000","last_active":"2024-03-09T15:32:40.999888+0000","last_peered":"2024-03-09T15:32:40.999888+0000","last_clean":"2024-03-09T15:32:40.999888+0000","last_became_active":"2024-03-09T14:39:13.257473+0000","last_became_peered":"2024-03-09T14:39:13.257473+0000","last_unstale":"2024-03-09T15:32:40.999888+0000","last_undegraded":"2024-03-09T15:32:40.999888+0000","last_fullsized":"2024-03-09T15:32:40.999888+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:12:15.900385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":78,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1d","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188298+0000","last_change":"2024-03-09T15:09:50.075294+0000","last_active":"2024-03-09T15:32:21.188298+0000","last_peered":"2024-03-09T15:32:21.188298+0000","last_clean":"2024-03-09T15:32:21.188298+0000","last_became_active":"2024-03-09T15:09:50.074875+0000","last_became_peered":"2024-03-09T15:09:50.074875+0000","last_unstale":"2024-03-09T15:32:21.188298+0000","last_undegraded":"2024-03-09T15:32:21.188298+0000","last_fullsized":"2024-03-09T15:32:21.188298+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:24:33.386479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1a","version":"261'9","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:30.775394+0000","last_change":"2024-03-09T14:46:26.813798+0000","last_active":"2024-03-09T15:31:30.775394+0000","last_peered":"2024-03-09T15:31:30.775394+0000","last_clean":"2024-03-09T15:31:30.775394+0000","last_became_active":"2024-03-09T14:46:26.812889+0000","last_became_peered":"2024-03-09T14:46:26.812889+0000","last_unstale":"2024-03-09T15:31:30.775394+0000","last_undegraded":"2024-03-09T15:31:30.775394+0000","last_fullsized":"2024-03-09T15:31:30.775394+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:43:59.844987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":12290,"num_write":13,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.1b","version":"261'1","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:20.980983+0000","last_change":"2024-03-09T14:46:18.774605+0000","last_active":"2024-03-09T15:31:20.980983+0000","last_peered":"2024-03-09T15:31:20.980983+0000","last_clean":"2024-03-09T15:31:20.980983+0000","last_became_active":"2024-03-09T14:46:18.756394+0000","last_became_peered":"2024-03-09T14:46:18.756394+0000","last_unstale":"2024-03-09T15:31:20.980983+0000","last_undegraded":"2024-03-09T15:31:20.980983+0000","last_fullsized":"2024-03-09T15:31:20.980983+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:49.690847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40145,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":40,"num_write":1,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.18","version":"261'8","reported_seq":77,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.144778+0000","last_change":"2024-03-09T14:39:13.258846+0000","last_active":"2024-03-09T15:32:26.144778+0000","last_peered":"2024-03-09T15:32:26.144778+0000","last_clean":"2024-03-09T15:32:26.144778+0000","last_became_active":"2024-03-09T14:39:13.257996+0000","last_became_peered":"2024-03-09T14:39:13.257996+0000","last_unstale":"2024-03-09T15:32:26.144778+0000","last_undegraded":"2024-03-09T15:32:26.144778+0000","last_fullsized":"2024-03-09T15:32:26.144778+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:23:43.508747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.12","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.957209+0000","last_change":"2024-03-09T15:09:50.070950+0000","last_active":"2024-03-09T15:33:10.957209+0000","last_peered":"2024-03-09T15:33:10.957209+0000","last_clean":"2024-03-09T15:33:10.957209+0000","last_became_active":"2024-03-09T15:09:50.070345+0000","last_became_peered":"2024-03-09T15:09:50.070345+0000","last_unstale":"2024-03-09T15:33:10.957209+0000","last_undegraded":"2024-03-09T15:33:10.957209+0000","last_fullsized":"2024-03-09T15:33:10.957209+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:18:24.068533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.15","version":"0'0","reported_seq":42,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.788416+0000","last_change":"2024-03-09T14:46:26.891781+0000","last_active":"2024-03-09T15:32:25.788416+0000","last_peered":"2024-03-09T15:32:25.788416+0000","last_clean":"2024-03-09T15:32:25.788416+0000","last_became_active":"2024-03-09T14:46:26.890992+0000","last_became_peered":"2024-03-09T14:46:26.890992+0000","last_unstale":"2024-03-09T15:32:25.788416+0000","last_undegraded":"2024-03-09T15:32:25.788416+0000","last_fullsized":"2024-03-09T15:32:25.788416+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:56:14.439157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.14","version":"0'0","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:06.187943+0000","last_change":"2024-03-09T14:46:18.767760+0000","last_active":"2024-03-09T15:33:06.187943+0000","last_peered":"2024-03-09T15:33:06.187943+0000","last_clean":"2024-03-09T15:33:06.187943+0000","last_became_active":"2024-03-09T14:46:18.766777+0000","last_became_peered":"2024-03-09T14:46:18.766777+0000","last_unstale":"2024-03-09T15:33:06.187943+0000","last_undegraded":"2024-03-09T15:33:06.187943+0000","last_fullsized":"2024-03-09T15:33:06.187943+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:30:54.129379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.17","version":"258'19","reported_seq":106,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.996874+0000","last_change":"2024-03-09T14:39:13.281484+0000","last_active":"2024-03-09T15:32:25.996874+0000","last_peered":"2024-03-09T15:32:25.996874+0000","last_clean":"2024-03-09T15:32:25.996874+0000","last_became_active":"2024-03-09T14:39:13.279919+0000","last_became_peered":"2024-03-09T14:39:13.279919+0000","last_unstale":"2024-03-09T15:32:25.996874+0000","last_undegraded":"2024-03-09T15:32:25.996874+0000","last_fullsized":"2024-03-09T15:32:25.996874+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:09:49.670708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":64,"num_write":32,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.13","version":"261'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.971686+0000","last_change":"2024-03-09T15:09:50.079986+0000","last_active":"2024-03-09T15:33:10.971686+0000","last_peered":"2024-03-09T15:33:10.971686+0000","last_clean":"2024-03-09T15:33:10.971686+0000","last_became_active":"2024-03-09T15:09:50.079604+0000","last_became_peered":"2024-03-09T15:09:50.079604+0000","last_unstale":"2024-03-09T15:33:10.971686+0000","last_undegraded":"2024-03-09T15:33:10.971686+0000","last_fullsized":"2024-03-09T15:33:10.971686+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:30:05.194786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.14","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.800266+0000","last_change":"2024-03-09T14:46:26.894898+0000","last_active":"2024-03-09T15:33:20.800266+0000","last_peered":"2024-03-09T15:33:20.800266+0000","last_clean":"2024-03-09T15:33:20.800266+0000","last_became_active":"2024-03-09T14:46:26.894088+0000","last_became_peered":"2024-03-09T14:46:26.894088+0000","last_unstale":"2024-03-09T15:33:20.800266+0000","last_undegraded":"2024-03-09T15:33:20.800266+0000","last_fullsized":"2024-03-09T15:33:20.800266+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:10:41.693365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.15","version":"0'0","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:16.187056+0000","last_change":"2024-03-09T14:46:18.753568+0000","last_active":"2024-03-09T15:32:16.187056+0000","last_peered":"2024-03-09T15:32:16.187056+0000","last_clean":"2024-03-09T15:32:16.187056+0000","last_became_active":"2024-03-09T14:46:18.752413+0000","last_became_peered":"2024-03-09T14:46:18.752413+0000","last_unstale":"2024-03-09T15:32:16.187056+0000","last_undegraded":"2024-03-09T15:32:16.187056+0000","last_fullsized":"2024-03-09T15:32:16.187056+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:09:14.935604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.16","version":"261'52","reported_seq":184,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.788486+0000","last_change":"2024-03-09T14:39:13.263228+0000","last_active":"2024-03-09T15:32:25.788486+0000","last_peered":"2024-03-09T15:32:25.788486+0000","last_clean":"2024-03-09T15:32:25.788486+0000","last_became_active":"2024-03-09T14:39:13.262413+0000","last_became_peered":"2024-03-09T14:39:13.262413+0000","last_unstale":"2024-03-09T15:32:25.788486+0000","last_undegraded":"2024-03-09T15:32:25.788486+0000","last_fullsized":"2024-03-09T15:32:25.788486+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:02:22.902848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":163,"num_write":78,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.10","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.199392+0000","last_change":"2024-03-09T15:09:50.075112+0000","last_active":"2024-03-09T15:33:11.199392+0000","last_peered":"2024-03-09T15:33:11.199392+0000","last_clean":"2024-03-09T15:33:11.199392+0000","last_became_active":"2024-03-09T15:09:50.074274+0000","last_became_peered":"2024-03-09T15:09:50.074274+0000","last_unstale":"2024-03-09T15:33:11.199392+0000","last_undegraded":"2024-03-09T15:33:11.199392+0000","last_fullsized":"2024-03-09T15:33:11.199392+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:34:26.130292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.17","version":"0'0","reported_seq":42,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.788513+0000","last_change":"2024-03-09T14:46:26.894297+0000","last_active":"2024-03-09T15:32:25.788513+0000","last_peered":"2024-03-09T15:32:25.788513+0000","last_clean":"2024-03-09T15:32:25.788513+0000","last_became_active":"2024-03-09T14:46:26.893514+0000","last_became_peered":"2024-03-09T14:46:26.893514+0000","last_unstale":"2024-03-09T15:32:25.788513+0000","last_undegraded":"2024-03-09T15:32:25.788513+0000","last_fullsized":"2024-03-09T15:32:25.788513+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:03:25.018128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.16","version":"0'0","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:15.960497+0000","last_change":"2024-03-09T14:46:18.773133+0000","last_active":"2024-03-09T15:32:15.960497+0000","last_peered":"2024-03-09T15:32:15.960497+0000","last_clean":"2024-03-09T15:32:15.960497+0000","last_became_active":"2024-03-09T14:46:18.769274+0000","last_became_peered":"2024-03-09T14:46:18.769274+0000","last_unstale":"2024-03-09T15:32:15.960497+0000","last_undegraded":"2024-03-09T15:32:15.960497+0000","last_fullsized":"2024-03-09T15:32:15.960497+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:01:15.489364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.15","version":"0'0","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.962879+0000","last_change":"2024-03-09T14:39:13.263907+0000","last_active":"2024-03-09T15:32:25.962879+0000","last_peered":"2024-03-09T15:32:25.962879+0000","last_clean":"2024-03-09T15:32:25.962879+0000","last_became_active":"2024-03-09T14:39:13.262066+0000","last_became_peered":"2024-03-09T14:39:13.262066+0000","last_unstale":"2024-03-09T15:32:25.962879+0000","last_undegraded":"2024-03-09T15:32:25.962879+0000","last_fullsized":"2024-03-09T15:32:25.962879+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:55:26.909612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.11","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.188977+0000","last_change":"2024-03-09T15:09:50.086149+0000","last_active":"2024-03-09T15:33:11.188977+0000","last_peered":"2024-03-09T15:33:11.188977+0000","last_clean":"2024-03-09T15:33:11.188977+0000","last_became_active":"2024-03-09T15:09:50.085346+0000","last_became_peered":"2024-03-09T15:09:50.085346+0000","last_unstale":"2024-03-09T15:33:11.188977+0000","last_undegraded":"2024-03-09T15:33:11.188977+0000","last_fullsized":"2024-03-09T15:33:11.188977+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:54:52.652612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.16","version":"0'0","reported_seq":42,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.175271+0000","last_change":"2024-03-09T14:46:26.818659+0000","last_active":"2024-03-09T15:32:21.175271+0000","last_peered":"2024-03-09T15:32:21.175271+0000","last_clean":"2024-03-09T15:32:21.175271+0000","last_became_active":"2024-03-09T14:46:26.817864+0000","last_became_peered":"2024-03-09T14:46:26.817864+0000","last_unstale":"2024-03-09T15:32:21.175271+0000","last_undegraded":"2024-03-09T15:32:21.175271+0000","last_fullsized":"2024-03-09T15:32:21.175271+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:10:59.964957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.17","version":"0'0","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:15.994554+0000","last_change":"2024-03-09T14:46:18.780789+0000","last_active":"2024-03-09T15:32:15.994554+0000","last_peered":"2024-03-09T15:32:15.994554+0000","last_clean":"2024-03-09T15:32:15.994554+0000","last_became_active":"2024-03-09T14:46:18.776430+0000","last_became_peered":"2024-03-09T14:46:18.776430+0000","last_unstale":"2024-03-09T15:32:15.994554+0000","last_undegraded":"2024-03-09T15:32:15.994554+0000","last_fullsized":"2024-03-09T15:32:15.994554+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:39:21.125955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.14","version":"258'67","reported_seq":270,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:06.198380+0000","last_change":"2024-03-09T14:39:13.253820+0000","last_active":"2024-03-09T15:33:06.198380+0000","last_peered":"2024-03-09T15:33:06.198380+0000","last_clean":"2024-03-09T15:33:06.198380+0000","last_became_active":"2024-03-09T14:39:13.253007+0000","last_became_peered":"2024-03-09T14:39:13.253007+0000","last_unstale":"2024-03-09T15:33:06.198380+0000","last_undegraded":"2024-03-09T15:33:06.198380+0000","last_fullsized":"2024-03-09T15:33:06.198380+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:44:10.310392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":411,"num_read_kb":279,"num_write":87,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.16","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776888+0000","last_change":"2024-03-09T15:09:50.087752+0000","last_active":"2024-03-09T15:31:35.776888+0000","last_peered":"2024-03-09T15:31:35.776888+0000","last_clean":"2024-03-09T15:31:35.776888+0000","last_became_active":"2024-03-09T15:09:50.086173+0000","last_became_peered":"2024-03-09T15:09:50.086173+0000","last_unstale":"2024-03-09T15:31:35.776888+0000","last_undegraded":"2024-03-09T15:31:35.776888+0000","last_fullsized":"2024-03-09T15:31:35.776888+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:43:21.719422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.11","version":"261'7","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:25.986303+0000","last_change":"2024-03-09T14:46:26.817795+0000","last_active":"2024-03-09T15:35:25.986303+0000","last_peered":"2024-03-09T15:35:25.986303+0000","last_clean":"2024-03-09T15:35:25.986303+0000","last_became_active":"2024-03-09T14:46:26.817014+0000","last_became_peered":"2024-03-09T14:46:26.817014+0000","last_unstale":"2024-03-09T15:35:25.986303+0000","last_undegraded":"2024-03-09T15:35:25.986303+0000","last_fullsized":"2024-03-09T15:35:25.986303+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:58:54.836958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":23554,"num_write":11,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.10","version":"261'2","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:01.183449+0000","last_change":"2024-03-09T14:46:18.771622+0000","last_active":"2024-03-09T15:33:01.183449+0000","last_peered":"2024-03-09T15:33:01.183449+0000","last_clean":"2024-03-09T15:33:01.183449+0000","last_became_active":"2024-03-09T14:46:18.770936+0000","last_became_peered":"2024-03-09T14:46:18.770936+0000","last_unstale":"2024-03-09T15:33:01.183449+0000","last_undegraded":"2024-03-09T15:33:01.183449+0000","last_fullsized":"2024-03-09T15:33:01.183449+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:04.807309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":2,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.13","version":"258'16","reported_seq":95,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.196014+0000","last_change":"2024-03-09T14:39:13.266972+0000","last_active":"2024-03-09T15:32:56.196014+0000","last_peered":"2024-03-09T15:32:56.196014+0000","last_clean":"2024-03-09T15:32:56.196014+0000","last_became_active":"2024-03-09T14:39:13.266228+0000","last_became_peered":"2024-03-09T14:39:13.266228+0000","last_unstale":"2024-03-09T15:32:56.196014+0000","last_undegraded":"2024-03-09T15:32:56.196014+0000","last_fullsized":"2024-03-09T15:32:56.196014+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:02:45.648908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":45,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.17","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.984716+0000","last_change":"2024-03-09T15:09:50.075655+0000","last_active":"2024-03-09T15:31:35.984716+0000","last_peered":"2024-03-09T15:31:35.984716+0000","last_clean":"2024-03-09T15:31:35.984716+0000","last_became_active":"2024-03-09T15:09:50.075187+0000","last_became_peered":"2024-03-09T15:09:50.075187+0000","last_unstale":"2024-03-09T15:31:35.984716+0000","last_undegraded":"2024-03-09T15:31:35.984716+0000","last_fullsized":"2024-03-09T15:31:35.984716+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:25:34.672578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.10","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:05.956045+0000","last_change":"2024-03-09T14:46:26.803323+0000","last_active":"2024-03-09T15:33:05.956045+0000","last_peered":"2024-03-09T15:33:05.956045+0000","last_clean":"2024-03-09T15:33:05.956045+0000","last_became_active":"2024-03-09T14:46:26.802476+0000","last_became_peered":"2024-03-09T14:46:26.802476+0000","last_unstale":"2024-03-09T15:33:05.956045+0000","last_undegraded":"2024-03-09T15:33:05.956045+0000","last_fullsized":"2024-03-09T15:33:05.956045+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:04:33.172429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.11","version":"0'0","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.795596+0000","last_change":"2024-03-09T14:46:18.761711+0000","last_active":"2024-03-09T15:33:00.795596+0000","last_peered":"2024-03-09T15:33:00.795596+0000","last_clean":"2024-03-09T15:33:00.795596+0000","last_became_active":"2024-03-09T14:46:18.760959+0000","last_became_peered":"2024-03-09T14:46:18.760959+0000","last_unstale":"2024-03-09T15:33:00.795596+0000","last_undegraded":"2024-03-09T15:33:00.795596+0000","last_fullsized":"2024-03-09T15:33:00.795596+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:08.964116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.12","version":"258'7","reported_seq":77,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.003193+0000","last_change":"2024-03-09T14:39:13.274560+0000","last_active":"2024-03-09T15:32:56.003193+0000","last_peered":"2024-03-09T15:32:56.003193+0000","last_clean":"2024-03-09T15:32:56.003193+0000","last_became_active":"2024-03-09T14:39:13.273740+0000","last_became_peered":"2024-03-09T14:39:13.273740+0000","last_unstale":"2024-03-09T15:32:56.003193+0000","last_undegraded":"2024-03-09T15:32:56.003193+0000","last_fullsized":"2024-03-09T15:32:56.003193+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:17.942276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":26,"num_write":13,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.14","version":"260'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.007791+0000","last_change":"2024-03-09T15:09:50.065240+0000","last_active":"2024-03-09T15:33:16.007791+0000","last_peered":"2024-03-09T15:33:16.007791+0000","last_clean":"2024-03-09T15:33:16.007791+0000","last_became_active":"2024-03-09T15:09:50.064428+0000","last_became_peered":"2024-03-09T15:09:50.064428+0000","last_unstale":"2024-03-09T15:33:16.007791+0000","last_undegraded":"2024-03-09T15:33:16.007791+0000","last_fullsized":"2024-03-09T15:33:16.007791+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:34:02.983911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.13","version":"261'2","reported_seq":47,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.798066+0000","last_change":"2024-03-09T14:46:26.813826+0000","last_active":"2024-03-09T15:33:10.798066+0000","last_peered":"2024-03-09T15:33:10.798066+0000","last_clean":"2024-03-09T15:33:10.798066+0000","last_became_active":"2024-03-09T14:46:26.813088+0000","last_became_peered":"2024-03-09T14:46:26.813088+0000","last_unstale":"2024-03-09T15:33:10.798066+0000","last_undegraded":"2024-03-09T15:33:10.798066+0000","last_fullsized":"2024-03-09T15:33:10.798066+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:37:31.388531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":2,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.12","version":"261'4","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.782261+0000","last_change":"2024-03-09T14:46:18.783155+0000","last_active":"2024-03-09T15:33:00.782261+0000","last_peered":"2024-03-09T15:33:00.782261+0000","last_clean":"2024-03-09T15:33:00.782261+0000","last_became_active":"2024-03-09T14:46:18.782158+0000","last_became_peered":"2024-03-09T14:46:18.782158+0000","last_unstale":"2024-03-09T15:33:00.782261+0000","last_undegraded":"2024-03-09T15:33:00.782261+0000","last_fullsized":"2024-03-09T15:33:00.782261+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:44:00.127768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":11264,"num_write":4,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.11","version":"258'36","reported_seq":148,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.182425+0000","last_change":"2024-03-09T14:39:13.260927+0000","last_active":"2024-03-09T15:32:56.182425+0000","last_peered":"2024-03-09T15:32:56.182425+0000","last_clean":"2024-03-09T15:32:56.182425+0000","last_became_active":"2024-03-09T14:39:13.260219+0000","last_became_peered":"2024-03-09T14:39:13.260219+0000","last_unstale":"2024-03-09T15:32:56.182425+0000","last_undegraded":"2024-03-09T15:32:56.182425+0000","last_fullsized":"2024-03-09T15:32:56.182425+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:27:50.114739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":118,"num_write":57,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.15","version":"261'1","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776826+0000","last_change":"2024-03-09T15:09:50.082323+0000","last_active":"2024-03-09T15:31:35.776826+0000","last_peered":"2024-03-09T15:31:35.776826+0000","last_clean":"2024-03-09T15:31:35.776826+0000","last_became_active":"2024-03-09T15:09:50.081688+0000","last_became_peered":"2024-03-09T15:09:50.081688+0000","last_unstale":"2024-03-09T15:31:35.776826+0000","last_undegraded":"2024-03-09T15:31:35.776826+0000","last_fullsized":"2024-03-09T15:31:35.776826+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:51:04.000384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.12","version":"261'1","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784367+0000","last_change":"2024-03-09T14:46:26.806904+0000","last_active":"2024-03-09T15:33:10.784367+0000","last_peered":"2024-03-09T15:33:10.784367+0000","last_clean":"2024-03-09T15:33:10.784367+0000","last_became_active":"2024-03-09T14:46:26.806155+0000","last_became_peered":"2024-03-09T14:46:26.806155+0000","last_unstale":"2024-03-09T15:33:10.784367+0000","last_undegraded":"2024-03-09T15:33:10.784367+0000","last_fullsized":"2024-03-09T15:33:10.784367+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:30:04.752758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":3072,"num_write":1,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.13","version":"261'2","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:21.217889+0000","last_change":"2024-03-09T14:46:18.783226+0000","last_active":"2024-03-09T15:35:21.217889+0000","last_peered":"2024-03-09T15:35:21.217889+0000","last_clean":"2024-03-09T15:35:21.217889+0000","last_became_active":"2024-03-09T14:46:18.782340+0000","last_became_peered":"2024-03-09T14:46:18.782340+0000","last_unstale":"2024-03-09T15:35:21.217889+0000","last_undegraded":"2024-03-09T15:35:21.217889+0000","last_fullsized":"2024-03-09T15:35:21.217889+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:42:41.572332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":50,"num_write":2,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.10","version":"258'67","reported_seq":256,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.794304+0000","last_change":"2024-03-09T14:39:13.245901+0000","last_active":"2024-03-09T15:32:55.794304+0000","last_peered":"2024-03-09T15:32:55.794304+0000","last_clean":"2024-03-09T15:32:55.794304+0000","last_became_active":"2024-03-09T14:39:13.245107+0000","last_became_peered":"2024-03-09T14:39:13.245107+0000","last_unstale":"2024-03-09T15:32:55.794304+0000","last_undegraded":"2024-03-09T15:32:55.794304+0000","last_fullsized":"2024-03-09T15:32:55.794304+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:25:10.518351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":382,"num_read_kb":270,"num_write":93,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.a","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.978852+0000","last_change":"2024-03-09T15:09:50.080044+0000","last_active":"2024-03-09T15:33:45.978852+0000","last_peered":"2024-03-09T15:33:45.978852+0000","last_clean":"2024-03-09T15:33:45.978852+0000","last_became_active":"2024-03-09T15:09:50.079554+0000","last_became_peered":"2024-03-09T15:09:50.079554+0000","last_unstale":"2024-03-09T15:33:45.978852+0000","last_undegraded":"2024-03-09T15:33:45.978852+0000","last_fullsized":"2024-03-09T15:33:45.978852+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:15:33.287535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.d","version":"261'4","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.806260+0000","last_change":"2024-03-09T14:46:26.813839+0000","last_active":"2024-03-09T15:33:45.806260+0000","last_peered":"2024-03-09T15:33:45.806260+0000","last_clean":"2024-03-09T15:33:45.806260+0000","last_became_active":"2024-03-09T14:46:26.812859+0000","last_became_peered":"2024-03-09T14:46:26.812859+0000","last_unstale":"2024-03-09T15:33:45.806260+0000","last_undegraded":"2024-03-09T15:33:45.806260+0000","last_fullsized":"2024-03-09T15:33:45.806260+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:10:31.220080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":8192,"num_write":9,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.c","version":"261'10","reported_seq":69,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.236677+0000","last_change":"2024-03-09T14:46:18.767068+0000","last_active":"2024-03-09T15:35:51.236677+0000","last_peered":"2024-03-09T15:35:51.236677+0000","last_clean":"2024-03-09T15:35:51.236677+0000","last_became_active":"2024-03-09T14:46:18.766370+0000","last_became_peered":"2024-03-09T14:46:18.766370+0000","last_unstale":"2024-03-09T15:35:51.236677+0000","last_undegraded":"2024-03-09T15:35:51.236677+0000","last_fullsized":"2024-03-09T15:35:51.236677+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:41:36.450438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":20480,"num_write":15,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.f","version":"258'18","reported_seq":99,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:30.802658+0000","last_change":"2024-03-09T14:39:13.271586+0000","last_active":"2024-03-09T15:33:30.802658+0000","last_peered":"2024-03-09T15:33:30.802658+0000","last_clean":"2024-03-09T15:33:30.802658+0000","last_became_active":"2024-03-09T14:39:13.268971+0000","last_became_peered":"2024-03-09T14:39:13.268971+0000","last_unstale":"2024-03-09T15:33:30.802658+0000","last_undegraded":"2024-03-09T15:33:30.802658+0000","last_fullsized":"2024-03-09T15:33:30.802658+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:14:41.597456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.b","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188151+0000","last_change":"2024-03-09T15:09:50.070451+0000","last_active":"2024-03-09T15:32:21.188151+0000","last_peered":"2024-03-09T15:32:21.188151+0000","last_clean":"2024-03-09T15:32:21.188151+0000","last_became_active":"2024-03-09T15:09:50.069508+0000","last_became_peered":"2024-03-09T15:09:50.069508+0000","last_unstale":"2024-03-09T15:32:21.188151+0000","last_undegraded":"2024-03-09T15:32:21.188151+0000","last_fullsized":"2024-03-09T15:32:21.188151+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:39:13.850719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.c","version":"261'2","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.806160+0000","last_change":"2024-03-09T14:46:26.813805+0000","last_active":"2024-03-09T15:33:45.806160+0000","last_peered":"2024-03-09T15:33:45.806160+0000","last_clean":"2024-03-09T15:33:45.806160+0000","last_became_active":"2024-03-09T14:46:26.813009+0000","last_became_peered":"2024-03-09T14:46:26.813009+0000","last_unstale":"2024-03-09T15:33:45.806160+0000","last_undegraded":"2024-03-09T15:33:45.806160+0000","last_fullsized":"2024-03-09T15:33:45.806160+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:34:36.995455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2048,"num_write":7,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.d","version":"261'2","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.976768+0000","last_change":"2024-03-09T14:46:18.771558+0000","last_active":"2024-03-09T15:33:35.976768+0000","last_peered":"2024-03-09T15:33:35.976768+0000","last_clean":"2024-03-09T15:33:35.976768+0000","last_became_active":"2024-03-09T14:46:18.770732+0000","last_became_peered":"2024-03-09T14:46:18.770732+0000","last_unstale":"2024-03-09T15:33:35.976768+0000","last_undegraded":"2024-03-09T15:33:35.976768+0000","last_fullsized":"2024-03-09T15:33:35.976768+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:08:20.215164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.e","version":"261'8","reported_seq":79,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.190234+0000","last_change":"2024-03-09T14:39:13.272393+0000","last_active":"2024-03-09T15:33:16.190234+0000","last_peered":"2024-03-09T15:33:16.190234+0000","last_clean":"2024-03-09T15:33:16.190234+0000","last_became_active":"2024-03-09T14:39:13.271672+0000","last_became_peered":"2024-03-09T14:39:13.271672+0000","last_unstale":"2024-03-09T15:33:16.190234+0000","last_undegraded":"2024-03-09T15:33:16.190234+0000","last_fullsized":"2024-03-09T15:33:16.190234+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:35:02.345192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.8","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.790792+0000","last_change":"2024-03-09T15:09:50.064645+0000","last_active":"2024-03-09T15:33:40.790792+0000","last_peered":"2024-03-09T15:33:40.790792+0000","last_clean":"2024-03-09T15:33:40.790792+0000","last_became_active":"2024-03-09T15:09:50.061662+0000","last_became_peered":"2024-03-09T15:09:50.061662+0000","last_unstale":"2024-03-09T15:33:40.790792+0000","last_undegraded":"2024-03-09T15:33:40.790792+0000","last_fullsized":"2024-03-09T15:33:40.790792+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:32:01.749280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.f","version":"261'5","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.966844+0000","last_change":"2024-03-09T14:46:26.900254+0000","last_active":"2024-03-09T15:33:50.966844+0000","last_peered":"2024-03-09T15:33:50.966844+0000","last_clean":"2024-03-09T15:33:50.966844+0000","last_became_active":"2024-03-09T14:46:26.899430+0000","last_became_peered":"2024-03-09T14:46:26.899430+0000","last_unstale":"2024-03-09T15:33:50.966844+0000","last_undegraded":"2024-03-09T15:33:50.966844+0000","last_fullsized":"2024-03-09T15:33:50.966844+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:58:16.207053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.e","version":"258'3","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.012622+0000","last_change":"2024-03-09T14:46:18.769516+0000","last_active":"2024-03-09T15:33:36.012622+0000","last_peered":"2024-03-09T15:33:36.012622+0000","last_clean":"2024-03-09T15:33:36.012622+0000","last_became_active":"2024-03-09T14:46:18.768694+0000","last_became_peered":"2024-03-09T14:46:18.768694+0000","last_unstale":"2024-03-09T15:33:36.012622+0000","last_undegraded":"2024-03-09T15:33:36.012622+0000","last_fullsized":"2024-03-09T15:33:36.012622+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:10:50.806709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.d","version":"261'43","reported_seq":159,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:15.785461+0000","last_change":"2024-03-09T14:39:13.266337+0000","last_active":"2024-03-09T15:33:15.785461+0000","last_peered":"2024-03-09T15:33:15.785461+0000","last_clean":"2024-03-09T15:33:15.785461+0000","last_became_active":"2024-03-09T14:39:13.257275+0000","last_became_peered":"2024-03-09T14:39:13.257275+0000","last_unstale":"2024-03-09T15:33:15.785461+0000","last_undegraded":"2024-03-09T15:33:15.785461+0000","last_fullsized":"2024-03-09T15:33:15.785461+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:45:48.503144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":175,"num_read_kb":123,"num_write":70,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.9","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:41.206696+0000","last_change":"2024-03-09T15:09:50.077371+0000","last_active":"2024-03-09T15:33:41.206696+0000","last_peered":"2024-03-09T15:33:41.206696+0000","last_clean":"2024-03-09T15:33:41.206696+0000","last_became_active":"2024-03-09T15:09:50.076702+0000","last_became_peered":"2024-03-09T15:09:50.076702+0000","last_unstale":"2024-03-09T15:33:41.206696+0000","last_undegraded":"2024-03-09T15:33:41.206696+0000","last_fullsized":"2024-03-09T15:33:41.206696+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:02:55.914615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.e","version":"261'4","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.806311+0000","last_change":"2024-03-09T14:46:26.818547+0000","last_active":"2024-03-09T15:33:45.806311+0000","last_peered":"2024-03-09T15:33:45.806311+0000","last_clean":"2024-03-09T15:33:45.806311+0000","last_became_active":"2024-03-09T14:46:26.814709+0000","last_became_peered":"2024-03-09T14:46:26.814709+0000","last_unstale":"2024-03-09T15:33:45.806311+0000","last_undegraded":"2024-03-09T15:33:45.806311+0000","last_fullsized":"2024-03-09T15:33:45.806311+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:11:42.597841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":9,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.f","version":"261'4","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.965610+0000","last_change":"2024-03-09T14:46:18.777803+0000","last_active":"2024-03-09T15:33:45.965610+0000","last_peered":"2024-03-09T15:33:45.965610+0000","last_clean":"2024-03-09T15:33:45.965610+0000","last_became_active":"2024-03-09T14:46:18.766000+0000","last_became_peered":"2024-03-09T14:46:18.766000+0000","last_unstale":"2024-03-09T15:33:45.965610+0000","last_undegraded":"2024-03-09T15:33:45.965610+0000","last_fullsized":"2024-03-09T15:33:45.965610+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:41:01.857202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4096,"num_write":9,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.c","version":"0'0","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:15.972703+0000","last_change":"2024-03-09T14:39:13.271296+0000","last_active":"2024-03-09T15:33:15.972703+0000","last_peered":"2024-03-09T15:33:15.972703+0000","last_clean":"2024-03-09T15:33:15.972703+0000","last_became_active":"2024-03-09T14:39:13.270527+0000","last_became_peered":"2024-03-09T14:39:13.270527+0000","last_unstale":"2024-03-09T15:33:15.972703+0000","last_undegraded":"2024-03-09T15:33:15.972703+0000","last_fullsized":"2024-03-09T15:33:15.972703+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:25:04.404490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.e","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.787362+0000","last_change":"2024-03-09T15:09:50.087358+0000","last_active":"2024-03-09T15:32:20.787362+0000","last_peered":"2024-03-09T15:32:20.787362+0000","last_clean":"2024-03-09T15:32:20.787362+0000","last_became_active":"2024-03-09T15:09:50.085548+0000","last_became_peered":"2024-03-09T15:09:50.085548+0000","last_unstale":"2024-03-09T15:32:20.787362+0000","last_undegraded":"2024-03-09T15:32:20.787362+0000","last_fullsized":"2024-03-09T15:32:20.787362+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:13.595041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.9","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:56.169630+0000","last_change":"2024-03-09T14:46:26.802054+0000","last_active":"2024-03-09T15:31:56.169630+0000","last_peered":"2024-03-09T15:31:56.169630+0000","last_clean":"2024-03-09T15:31:56.169630+0000","last_became_active":"2024-03-09T14:46:26.801235+0000","last_became_peered":"2024-03-09T14:46:26.801235+0000","last_unstale":"2024-03-09T15:31:56.169630+0000","last_undegraded":"2024-03-09T15:31:56.169630+0000","last_fullsized":"2024-03-09T15:31:56.169630+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:49:20.412101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.8","version":"0'0","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.988186+0000","last_change":"2024-03-09T14:46:18.775300+0000","last_active":"2024-03-09T15:31:50.988186+0000","last_peered":"2024-03-09T15:31:50.988186+0000","last_clean":"2024-03-09T15:31:50.988186+0000","last_became_active":"2024-03-09T14:46:18.774547+0000","last_became_peered":"2024-03-09T14:46:18.774547+0000","last_unstale":"2024-03-09T15:31:50.988186+0000","last_undegraded":"2024-03-09T15:31:50.988186+0000","last_fullsized":"2024-03-09T15:31:50.988186+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:18:57.376200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.b","version":"258'10","reported_seq":85,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.007679+0000","last_change":"2024-03-09T14:39:13.258020+0000","last_active":"2024-03-09T15:33:16.007679+0000","last_peered":"2024-03-09T15:33:16.007679+0000","last_clean":"2024-03-09T15:33:16.007679+0000","last_became_active":"2024-03-09T14:39:13.257238+0000","last_became_peered":"2024-03-09T14:39:13.257238+0000","last_unstale":"2024-03-09T15:33:16.007679+0000","last_undegraded":"2024-03-09T15:33:16.007679+0000","last_fullsized":"2024-03-09T15:33:16.007679+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:40:13.243772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.f","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.189467+0000","last_change":"2024-03-09T15:09:50.071187+0000","last_active":"2024-03-09T15:32:26.189467+0000","last_peered":"2024-03-09T15:32:26.189467+0000","last_clean":"2024-03-09T15:32:26.189467+0000","last_became_active":"2024-03-09T15:09:50.070526+0000","last_became_peered":"2024-03-09T15:09:50.070526+0000","last_unstale":"2024-03-09T15:32:26.189467+0000","last_undegraded":"2024-03-09T15:32:26.189467+0000","last_fullsized":"2024-03-09T15:32:26.189467+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:03:16.280366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.8","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:01.183115+0000","last_change":"2024-03-09T14:46:26.894478+0000","last_active":"2024-03-09T15:32:01.183115+0000","last_peered":"2024-03-09T15:32:01.183115+0000","last_clean":"2024-03-09T15:32:01.183115+0000","last_became_active":"2024-03-09T14:46:26.893908+0000","last_became_peered":"2024-03-09T14:46:26.893908+0000","last_unstale":"2024-03-09T15:32:01.183115+0000","last_undegraded":"2024-03-09T15:32:01.183115+0000","last_fullsized":"2024-03-09T15:32:01.183115+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:43:51.901456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.9","version":"261'2","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:51.136707+0000","last_change":"2024-03-09T14:46:18.777691+0000","last_active":"2024-03-09T15:31:51.136707+0000","last_peered":"2024-03-09T15:31:51.136707+0000","last_clean":"2024-03-09T15:31:51.136707+0000","last_became_active":"2024-03-09T14:46:18.761367+0000","last_became_peered":"2024-03-09T14:46:18.761367+0000","last_unstale":"2024-03-09T15:31:51.136707+0000","last_undegraded":"2024-03-09T15:31:51.136707+0000","last_fullsized":"2024-03-09T15:31:51.136707+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:44:32.343854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.a","version":"258'23","reported_seq":123,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205291+0000","last_change":"2024-03-09T14:39:13.253482+0000","last_active":"2024-03-09T15:33:36.205291+0000","last_peered":"2024-03-09T15:33:36.205291+0000","last_clean":"2024-03-09T15:33:36.205291+0000","last_became_active":"2024-03-09T14:39:13.252547+0000","last_became_peered":"2024-03-09T14:39:13.252547+0000","last_unstale":"2024-03-09T15:33:36.205291+0000","last_undegraded":"2024-03-09T15:33:36.205291+0000","last_fullsized":"2024-03-09T15:33:36.205291+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:32:42.396388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":86,"num_write":38,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.c","version":"261'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188203+0000","last_change":"2024-03-09T15:09:50.070468+0000","last_active":"2024-03-09T15:32:21.188203+0000","last_peered":"2024-03-09T15:32:21.188203+0000","last_clean":"2024-03-09T15:32:21.188203+0000","last_became_active":"2024-03-09T15:09:50.069324+0000","last_became_peered":"2024-03-09T15:09:50.069324+0000","last_unstale":"2024-03-09T15:32:21.188203+0000","last_undegraded":"2024-03-09T15:32:21.188203+0000","last_fullsized":"2024-03-09T15:32:21.188203+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:23:32.203630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.b","version":"258'3","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.964465+0000","last_change":"2024-03-09T14:46:26.805564+0000","last_active":"2024-03-09T15:33:40.964465+0000","last_peered":"2024-03-09T15:33:40.964465+0000","last_clean":"2024-03-09T15:33:40.964465+0000","last_became_active":"2024-03-09T14:46:26.804615+0000","last_became_peered":"2024-03-09T14:46:26.804615+0000","last_unstale":"2024-03-09T15:33:40.964465+0000","last_undegraded":"2024-03-09T15:33:40.964465+0000","last_fullsized":"2024-03-09T15:33:40.964465+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:54:32.765203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.a","version":"258'3","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:00.990762+0000","last_change":"2024-03-09T14:46:18.769098+0000","last_active":"2024-03-09T15:32:00.990762+0000","last_peered":"2024-03-09T15:32:00.990762+0000","last_clean":"2024-03-09T15:32:00.990762+0000","last_became_active":"2024-03-09T14:46:18.768431+0000","last_became_peered":"2024-03-09T14:46:18.768431+0000","last_unstale":"2024-03-09T15:32:00.990762+0000","last_undegraded":"2024-03-09T15:32:00.990762+0000","last_fullsized":"2024-03-09T15:32:00.990762+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:28:28.024126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.9","version":"261'43","reported_seq":157,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.801289+0000","last_change":"2024-03-09T14:39:13.260724+0000","last_active":"2024-03-09T15:33:25.801289+0000","last_peered":"2024-03-09T15:33:25.801289+0000","last_clean":"2024-03-09T15:33:25.801289+0000","last_became_active":"2024-03-09T14:39:13.259975+0000","last_became_peered":"2024-03-09T14:39:13.259975+0000","last_unstale":"2024-03-09T15:33:25.801289+0000","last_undegraded":"2024-03-09T15:33:25.801289+0000","last_fullsized":"2024-03-09T15:33:25.801289+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:26:32.899703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":169,"num_read_kb":119,"num_write":74,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.d","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.995632+0000","last_change":"2024-03-09T15:09:50.084646+0000","last_active":"2024-03-09T15:32:20.995632+0000","last_peered":"2024-03-09T15:32:20.995632+0000","last_clean":"2024-03-09T15:32:20.995632+0000","last_became_active":"2024-03-09T15:09:50.084074+0000","last_became_peered":"2024-03-09T15:09:50.084074+0000","last_unstale":"2024-03-09T15:32:20.995632+0000","last_undegraded":"2024-03-09T15:32:20.995632+0000","last_fullsized":"2024-03-09T15:32:20.995632+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:16:13.790167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.a","version":"258'3","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:06.184462+0000","last_change":"2024-03-09T14:46:26.826875+0000","last_active":"2024-03-09T15:32:06.184462+0000","last_peered":"2024-03-09T15:32:06.184462+0000","last_clean":"2024-03-09T15:32:06.184462+0000","last_became_active":"2024-03-09T14:46:26.825949+0000","last_became_peered":"2024-03-09T14:46:26.825949+0000","last_unstale":"2024-03-09T15:32:06.184462+0000","last_undegraded":"2024-03-09T15:32:06.184462+0000","last_fullsized":"2024-03-09T15:32:06.184462+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:24:09.628828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.b","version":"261'2","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191328+0000","last_change":"2024-03-09T14:46:18.785518+0000","last_active":"2024-03-09T15:33:36.191328+0000","last_peered":"2024-03-09T15:33:36.191328+0000","last_clean":"2024-03-09T15:33:36.191328+0000","last_became_active":"2024-03-09T14:46:18.784869+0000","last_became_peered":"2024-03-09T14:46:18.784869+0000","last_unstale":"2024-03-09T15:33:36.191328+0000","last_undegraded":"2024-03-09T15:33:36.191328+0000","last_fullsized":"2024-03-09T15:33:36.191328+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:03:42.729066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":6144,"num_write":2,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.8","version":"258'10","reported_seq":86,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.787542+0000","last_change":"2024-03-09T14:39:13.264924+0000","last_active":"2024-03-09T15:33:25.787542+0000","last_peered":"2024-03-09T15:33:25.787542+0000","last_clean":"2024-03-09T15:33:25.787542+0000","last_became_active":"2024-03-09T14:39:13.264006+0000","last_became_peered":"2024-03-09T14:39:13.264006+0000","last_unstale":"2024-03-09T15:33:25.787542+0000","last_undegraded":"2024-03-09T15:33:25.787542+0000","last_fullsized":"2024-03-09T15:33:25.787542+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:24:57.691583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.2","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.193658+0000","last_change":"2024-03-09T15:09:50.066069+0000","last_active":"2024-03-09T15:32:46.193658+0000","last_peered":"2024-03-09T15:32:46.193658+0000","last_clean":"2024-03-09T15:32:46.193658+0000","last_became_active":"2024-03-09T15:09:50.065487+0000","last_became_peered":"2024-03-09T15:09:50.065487+0000","last_unstale":"2024-03-09T15:32:46.193658+0000","last_undegraded":"2024-03-09T15:32:46.193658+0000","last_fullsized":"2024-03-09T15:32:46.193658+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:16:19.163190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.5","version":"258'2","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:30.997928+0000","last_change":"2024-03-09T14:46:26.813029+0000","last_active":"2024-03-09T15:32:30.997928+0000","last_peered":"2024-03-09T15:32:30.997928+0000","last_clean":"2024-03-09T15:32:30.997928+0000","last_became_active":"2024-03-09T14:46:26.812137+0000","last_became_peered":"2024-03-09T14:46:26.812137+0000","last_unstale":"2024-03-09T15:32:30.997928+0000","last_undegraded":"2024-03-09T15:32:30.997928+0000","last_fullsized":"2024-03-09T15:32:30.997928+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:36:47.037465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4096,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.4","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.185772+0000","last_change":"2024-03-09T14:46:18.752392+0000","last_active":"2024-03-09T15:32:11.185772+0000","last_peered":"2024-03-09T15:32:11.185772+0000","last_clean":"2024-03-09T15:32:11.185772+0000","last_became_active":"2024-03-09T14:46:18.751608+0000","last_became_peered":"2024-03-09T14:46:18.751608+0000","last_unstale":"2024-03-09T15:32:11.185772+0000","last_undegraded":"2024-03-09T15:32:11.185772+0000","last_fullsized":"2024-03-09T15:32:11.185772+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:32:15.618158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.7","version":"261'8","reported_seq":80,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.010433+0000","last_change":"2024-03-09T14:39:13.280605+0000","last_active":"2024-03-09T15:33:26.010433+0000","last_peered":"2024-03-09T15:33:26.010433+0000","last_clean":"2024-03-09T15:33:26.010433+0000","last_became_active":"2024-03-09T14:39:13.273363+0000","last_became_peered":"2024-03-09T14:39:13.273363+0000","last_unstale":"2024-03-09T15:33:26.010433+0000","last_undegraded":"2024-03-09T15:33:26.010433+0000","last_fullsized":"2024-03-09T15:33:26.010433+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:45:47.783708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.3","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.183450+0000","last_change":"2024-03-09T15:09:50.078106+0000","last_active":"2024-03-09T15:32:46.183450+0000","last_peered":"2024-03-09T15:32:46.183450+0000","last_clean":"2024-03-09T15:32:46.183450+0000","last_became_active":"2024-03-09T15:09:50.072488+0000","last_became_peered":"2024-03-09T15:09:50.072488+0000","last_unstale":"2024-03-09T15:32:46.183450+0000","last_undegraded":"2024-03-09T15:32:46.183450+0000","last_fullsized":"2024-03-09T15:32:46.183450+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:20:34.288932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.4","version":"258'2","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.995582+0000","last_change":"2024-03-09T14:46:26.895288+0000","last_active":"2024-03-09T15:32:20.995582+0000","last_peered":"2024-03-09T15:32:20.995582+0000","last_clean":"2024-03-09T15:32:20.995582+0000","last_became_active":"2024-03-09T14:46:26.894506+0000","last_became_peered":"2024-03-09T14:46:26.894506+0000","last_unstale":"2024-03-09T15:32:20.995582+0000","last_undegraded":"2024-03-09T15:32:20.995582+0000","last_fullsized":"2024-03-09T15:32:20.995582+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:23:02.279766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":23,"num_write":2,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.5","version":"258'5","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.962822+0000","last_change":"2024-03-09T14:46:18.771982+0000","last_active":"2024-03-09T15:32:25.962822+0000","last_peered":"2024-03-09T15:32:25.962822+0000","last_clean":"2024-03-09T15:32:25.962822+0000","last_became_active":"2024-03-09T14:46:18.770799+0000","last_became_peered":"2024-03-09T14:46:18.770799+0000","last_unstale":"2024-03-09T15:32:25.962822+0000","last_undegraded":"2024-03-09T15:32:25.962822+0000","last_fullsized":"2024-03-09T15:32:25.962822+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:15:47.749134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.6","version":"261'22","reported_seq":112,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.787743+0000","last_change":"2024-03-09T14:39:13.268293+0000","last_active":"2024-03-09T15:33:25.787743+0000","last_peered":"2024-03-09T15:33:25.787743+0000","last_clean":"2024-03-09T15:33:25.787743+0000","last_became_active":"2024-03-09T14:39:13.266246+0000","last_became_peered":"2024-03-09T14:39:13.266246+0000","last_unstale":"2024-03-09T15:33:25.787743+0000","last_undegraded":"2024-03-09T15:33:25.787743+0000","last_fullsized":"2024-03-09T15:33:25.787743+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:11:53.843138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":65,"num_write":41,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.0","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:00.968771+0000","last_change":"2024-03-09T15:09:50.064673+0000","last_active":"2024-03-09T15:34:00.968771+0000","last_peered":"2024-03-09T15:34:00.968771+0000","last_clean":"2024-03-09T15:34:00.968771+0000","last_became_active":"2024-03-09T15:09:50.063350+0000","last_became_peered":"2024-03-09T15:09:50.063350+0000","last_unstale":"2024-03-09T15:34:00.968771+0000","last_undegraded":"2024-03-09T15:34:00.968771+0000","last_fullsized":"2024-03-09T15:34:00.968771+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:55.461559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.7","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:01.183396+0000","last_change":"2024-03-09T14:46:26.897079+0000","last_active":"2024-03-09T15:32:01.183396+0000","last_peered":"2024-03-09T15:32:01.183396+0000","last_clean":"2024-03-09T15:32:01.183396+0000","last_became_active":"2024-03-09T14:46:26.896503+0000","last_became_peered":"2024-03-09T14:46:26.896503+0000","last_unstale":"2024-03-09T15:32:01.183396+0000","last_undegraded":"2024-03-09T15:32:01.183396+0000","last_fullsized":"2024-03-09T15:32:01.183396+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:57:37.168283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.6","version":"261'2","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:01.200543+0000","last_change":"2024-03-09T14:46:18.779289+0000","last_active":"2024-03-09T15:34:01.200543+0000","last_peered":"2024-03-09T15:34:01.200543+0000","last_clean":"2024-03-09T15:34:01.200543+0000","last_became_active":"2024-03-09T14:46:18.778572+0000","last_became_peered":"2024-03-09T14:46:18.778572+0000","last_unstale":"2024-03-09T15:34:01.200543+0000","last_undegraded":"2024-03-09T15:34:01.200543+0000","last_fullsized":"2024-03-09T15:34:01.200543+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:28:53.930043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.5","version":"258'18","reported_seq":107,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205525+0000","last_change":"2024-03-09T14:39:13.264712+0000","last_active":"2024-03-09T15:33:36.205525+0000","last_peered":"2024-03-09T15:33:36.205525+0000","last_clean":"2024-03-09T15:33:36.205525+0000","last_became_active":"2024-03-09T14:39:13.263833+0000","last_became_peered":"2024-03-09T14:39:13.263833+0000","last_unstale":"2024-03-09T15:33:36.205525+0000","last_undegraded":"2024-03-09T15:33:36.205525+0000","last_fullsized":"2024-03-09T15:33:36.205525+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:10:25.603434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":58,"num_write":29,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.1","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.180226+0000","last_change":"2024-03-09T15:09:50.069405+0000","last_active":"2024-03-09T15:32:46.180226+0000","last_peered":"2024-03-09T15:32:46.180226+0000","last_clean":"2024-03-09T15:32:46.180226+0000","last_became_active":"2024-03-09T15:09:50.067947+0000","last_became_peered":"2024-03-09T15:09:50.067947+0000","last_unstale":"2024-03-09T15:32:46.180226+0000","last_undegraded":"2024-03-09T15:32:46.180226+0000","last_fullsized":"2024-03-09T15:32:46.180226+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:29:17.390497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"67.6","version":"261'2","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:56.169833+0000","last_change":"2024-03-09T14:46:26.809411+0000","last_active":"2024-03-09T15:31:56.169833+0000","last_peered":"2024-03-09T15:31:56.169833+0000","last_clean":"2024-03-09T15:31:56.169833+0000","last_became_active":"2024-03-09T14:46:26.808384+0000","last_became_peered":"2024-03-09T14:46:26.808384+0000","last_unstale":"2024-03-09T15:31:56.169833+0000","last_undegraded":"2024-03-09T15:31:56.169833+0000","last_fullsized":"2024-03-09T15:31:56.169833+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:53:31.797035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90604,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":180,"num_write":2,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.7","version":"261'7","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:51.180691+0000","last_change":"2024-03-09T14:46:18.763286+0000","last_active":"2024-03-09T15:31:51.180691+0000","last_peered":"2024-03-09T15:31:51.180691+0000","last_clean":"2024-03-09T15:31:51.180691+0000","last_became_active":"2024-03-09T14:46:18.762472+0000","last_became_peered":"2024-03-09T14:46:18.762472+0000","last_unstale":"2024-03-09T15:31:51.180691+0000","last_undegraded":"2024-03-09T15:31:51.180691+0000","last_fullsized":"2024-03-09T15:31:51.180691+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:30.905827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":30722,"num_write":16,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.4","version":"258'10","reported_seq":87,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.960925+0000","last_change":"2024-03-09T14:39:13.270957+0000","last_active":"2024-03-09T15:33:25.960925+0000","last_peered":"2024-03-09T15:33:25.960925+0000","last_clean":"2024-03-09T15:33:25.960925+0000","last_became_active":"2024-03-09T14:39:13.270238+0000","last_became_peered":"2024-03-09T14:39:13.270238+0000","last_unstale":"2024-03-09T15:33:25.960925+0000","last_undegraded":"2024-03-09T15:33:25.960925+0000","last_fullsized":"2024-03-09T15:33:25.960925+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:25:19.656005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.6","version":"260'1","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.977825+0000","last_change":"2024-03-09T15:09:50.072623+0000","last_active":"2024-03-09T15:33:40.977825+0000","last_peered":"2024-03-09T15:33:40.977825+0000","last_clean":"2024-03-09T15:33:40.977825+0000","last_became_active":"2024-03-09T15:09:50.072180+0000","last_became_peered":"2024-03-09T15:09:50.072180+0000","last_unstale":"2024-03-09T15:33:40.977825+0000","last_undegraded":"2024-03-09T15:33:40.977825+0000","last_fullsized":"2024-03-09T15:33:40.977825+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:17:47.602511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1","version":"0'0","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:16.142380+0000","last_change":"2024-03-09T14:46:26.799783+0000","last_active":"2024-03-09T15:32:16.142380+0000","last_peered":"2024-03-09T15:32:16.142380+0000","last_clean":"2024-03-09T15:32:16.142380+0000","last_became_active":"2024-03-09T14:46:26.799019+0000","last_became_peered":"2024-03-09T14:46:26.799019+0000","last_unstale":"2024-03-09T15:32:16.142380+0000","last_undegraded":"2024-03-09T15:32:16.142380+0000","last_fullsized":"2024-03-09T15:32:16.142380+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:37:13.610839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.0","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.189237+0000","last_change":"2024-03-09T14:46:18.753025+0000","last_active":"2024-03-09T15:32:26.189237+0000","last_peered":"2024-03-09T15:32:26.189237+0000","last_clean":"2024-03-09T15:32:26.189237+0000","last_became_active":"2024-03-09T14:46:18.752290+0000","last_became_peered":"2024-03-09T14:46:18.752290+0000","last_unstale":"2024-03-09T15:32:26.189237+0000","last_undegraded":"2024-03-09T15:32:26.189237+0000","last_fullsized":"2024-03-09T15:32:26.189237+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:33:49.576573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.3","version":"261'61","reported_seq":215,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.192284+0000","last_change":"2024-03-09T14:39:13.259947+0000","last_active":"2024-03-09T15:33:26.192284+0000","last_peered":"2024-03-09T15:33:26.192284+0000","last_clean":"2024-03-09T15:33:26.192284+0000","last_became_active":"2024-03-09T14:39:13.259125+0000","last_became_peered":"2024-03-09T14:39:13.259125+0000","last_unstale":"2024-03-09T15:33:26.192284+0000","last_undegraded":"2024-03-09T15:33:26.192284+0000","last_fullsized":"2024-03-09T15:33:26.192284+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:52:15.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":281,"num_read_kb":205,"num_write":98,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.7","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:41.014114+0000","last_change":"2024-03-09T15:09:50.064879+0000","last_active":"2024-03-09T15:33:41.014114+0000","last_peered":"2024-03-09T15:33:41.014114+0000","last_clean":"2024-03-09T15:33:41.014114+0000","last_became_active":"2024-03-09T15:09:50.062429+0000","last_became_peered":"2024-03-09T15:09:50.062429+0000","last_unstale":"2024-03-09T15:33:41.014114+0000","last_undegraded":"2024-03-09T15:33:41.014114+0000","last_fullsized":"2024-03-09T15:33:41.014114+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:10:20.402757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.0","version":"261'5","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:36.207793+0000","last_change":"2024-03-09T14:46:26.816110+0000","last_active":"2024-03-09T15:34:36.207793+0000","last_peered":"2024-03-09T15:34:36.207793+0000","last_clean":"2024-03-09T15:34:36.207793+0000","last_became_active":"2024-03-09T14:46:26.815311+0000","last_became_peered":"2024-03-09T14:46:26.815311+0000","last_unstale":"2024-03-09T15:34:36.207793+0000","last_undegraded":"2024-03-09T15:34:36.207793+0000","last_fullsized":"2024-03-09T15:34:36.207793+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:26:21.047330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":16384,"num_write":10,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1","version":"258'5","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.185673+0000","last_change":"2024-03-09T14:46:18.758437+0000","last_active":"2024-03-09T15:32:11.185673+0000","last_peered":"2024-03-09T15:32:11.185673+0000","last_clean":"2024-03-09T15:32:11.185673+0000","last_became_active":"2024-03-09T14:46:18.757694+0000","last_became_peered":"2024-03-09T14:46:18.757694+0000","last_unstale":"2024-03-09T15:32:11.185673+0000","last_undegraded":"2024-03-09T15:32:11.185673+0000","last_fullsized":"2024-03-09T15:32:11.185673+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:47:19.096625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":4098,"num_write":9,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.2","version":"258'20","reported_seq":110,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.188905+0000","last_change":"2024-03-09T14:39:13.265013+0000","last_active":"2024-03-09T15:33:26.188905+0000","last_peered":"2024-03-09T15:33:26.188905+0000","last_clean":"2024-03-09T15:33:26.188905+0000","last_became_active":"2024-03-09T14:39:13.264203+0000","last_became_peered":"2024-03-09T14:39:13.264203+0000","last_unstale":"2024-03-09T15:33:26.188905+0000","last_undegraded":"2024-03-09T15:33:26.188905+0000","last_fullsized":"2024-03-09T15:33:26.188905+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:46:43.366482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":65,"num_write":31,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.4","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.001035+0000","last_change":"2024-03-09T15:09:50.070703+0000","last_active":"2024-03-09T15:32:46.001035+0000","last_peered":"2024-03-09T15:32:46.001035+0000","last_clean":"2024-03-09T15:32:46.001035+0000","last_became_active":"2024-03-09T15:09:50.066222+0000","last_became_peered":"2024-03-09T15:09:50.066222+0000","last_unstale":"2024-03-09T15:32:46.001035+0000","last_undegraded":"2024-03-09T15:32:46.001035+0000","last_fullsized":"2024-03-09T15:32:46.001035+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:54:19.245692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.3","version":"0'0","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.774297+0000","last_change":"2024-03-09T14:46:26.826931+0000","last_active":"2024-03-09T15:32:20.774297+0000","last_peered":"2024-03-09T15:32:20.774297+0000","last_clean":"2024-03-09T15:32:20.774297+0000","last_became_active":"2024-03-09T14:46:26.826105+0000","last_became_peered":"2024-03-09T14:46:26.826105+0000","last_unstale":"2024-03-09T15:32:20.774297+0000","last_undegraded":"2024-03-09T15:32:20.774297+0000","last_fullsized":"2024-03-09T15:32:20.774297+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:19:31.840994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.2","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.141234+0000","last_change":"2024-03-09T14:46:18.749838+0000","last_active":"2024-03-09T15:32:11.141234+0000","last_peered":"2024-03-09T15:32:11.141234+0000","last_clean":"2024-03-09T15:32:11.141234+0000","last_became_active":"2024-03-09T14:46:18.749091+0000","last_became_peered":"2024-03-09T14:46:18.749091+0000","last_unstale":"2024-03-09T15:32:11.141234+0000","last_undegraded":"2024-03-09T15:32:11.141234+0000","last_fullsized":"2024-03-09T15:32:11.141234+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:47:14.332817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1","version":"261'25","reported_seq":125,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.787625+0000","last_change":"2024-03-09T14:39:13.257789+0000","last_active":"2024-03-09T15:33:25.787625+0000","last_peered":"2024-03-09T15:33:25.787625+0000","last_clean":"2024-03-09T15:33:25.787625+0000","last_became_active":"2024-03-09T14:39:13.255589+0000","last_became_peered":"2024-03-09T14:39:13.255589+0000","last_unstale":"2024-03-09T15:33:25.787625+0000","last_undegraded":"2024-03-09T15:33:25.787625+0000","last_fullsized":"2024-03-09T15:33:25.787625+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:53:30.259823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":79,"num_write":35,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.5","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:51.194924+0000","last_change":"2024-03-09T15:09:50.083562+0000","last_active":"2024-03-09T15:32:51.194924+0000","last_peered":"2024-03-09T15:32:51.194924+0000","last_clean":"2024-03-09T15:32:51.194924+0000","last_became_active":"2024-03-09T15:09:50.083064+0000","last_became_peered":"2024-03-09T15:09:50.083064+0000","last_unstale":"2024-03-09T15:32:51.194924+0000","last_undegraded":"2024-03-09T15:32:51.194924+0000","last_fullsized":"2024-03-09T15:32:51.194924+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:25:31.798282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.2","version":"258'3","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:16.174077+0000","last_change":"2024-03-09T14:46:26.808768+0000","last_active":"2024-03-09T15:32:16.174077+0000","last_peered":"2024-03-09T15:32:16.174077+0000","last_clean":"2024-03-09T15:32:16.174077+0000","last_became_active":"2024-03-09T14:46:26.807746+0000","last_became_peered":"2024-03-09T14:46:26.807746+0000","last_unstale":"2024-03-09T15:32:16.174077+0000","last_undegraded":"2024-03-09T15:32:16.174077+0000","last_fullsized":"2024-03-09T15:32:16.174077+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:58:17.140698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1024,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.3","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:10.772267+0000","last_change":"2024-03-09T14:46:18.765863+0000","last_active":"2024-03-09T15:32:10.772267+0000","last_peered":"2024-03-09T15:32:10.772267+0000","last_clean":"2024-03-09T15:32:10.772267+0000","last_became_active":"2024-03-09T14:46:18.765077+0000","last_became_peered":"2024-03-09T14:46:18.765077+0000","last_unstale":"2024-03-09T15:32:10.772267+0000","last_undegraded":"2024-03-09T15:32:10.772267+0000","last_fullsized":"2024-03-09T15:32:10.772267+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:26:58.084480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.0","version":"250'25","reported_seq":130,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.202779+0000","last_change":"2024-03-09T14:39:13.269464+0000","last_active":"2024-03-09T15:33:26.202779+0000","last_peered":"2024-03-09T15:33:26.202779+0000","last_clean":"2024-03-09T15:33:26.202779+0000","last_became_active":"2024-03-09T14:39:13.268553+0000","last_became_peered":"2024-03-09T14:39:13.268553+0000","last_unstale":"2024-03-09T15:33:26.202779+0000","last_undegraded":"2024-03-09T15:33:26.202779+0000","last_fullsized":"2024-03-09T15:33:26.202779+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:01:17.783740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":97,"num_write":32,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"261'102","reported_seq":659,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.187916+0000","last_change":"2024-03-09T14:28:32.340306+0000","last_active":"2024-03-09T15:36:06.187916+0000","last_peered":"2024-03-09T15:36:06.187916+0000","last_clean":"2024-03-09T15:36:06.187916+0000","last_became_active":"2024-03-09T14:28:32.339785+0000","last_became_peered":"2024-03-09T14:28:32.339785+0000","last_unstale":"2024-03-09T15:36:06.187916+0000","last_undegraded":"2024-03-09T15:36:06.187916+0000","last_fullsized":"2024-03-09T15:36:06.187916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:39:28.870360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":925441,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":118,"num_read_kb":5377,"num_write":361,"num_write_kb":16955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"261'146","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194366+0000","last_change":"2024-03-09T14:28:34.304676+0000","last_active":"2024-03-09T15:33:36.194366+0000","last_peered":"2024-03-09T15:33:36.194366+0000","last_clean":"2024-03-09T15:33:36.194366+0000","last_became_active":"2024-03-09T14:28:34.303980+0000","last_became_peered":"2024-03-09T14:28:34.303980+0000","last_unstale":"2024-03-09T15:33:36.194366+0000","last_undegraded":"2024-03-09T15:33:36.194366+0000","last_fullsized":"2024-03-09T15:33:36.194366+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:12:53.886509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":546,"num_read_kb":417,"num_write":228,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"261'119","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.937233+0000","last_change":"2024-03-09T14:28:32.307225+0000","last_active":"2024-03-09T15:36:05.937233+0000","last_peered":"2024-03-09T15:36:05.937233+0000","last_clean":"2024-03-09T15:36:05.937233+0000","last_became_active":"2024-03-09T14:28:32.306840+0000","last_became_peered":"2024-03-09T14:28:32.306840+0000","last_unstale":"2024-03-09T15:36:05.937233+0000","last_undegraded":"2024-03-09T15:36:05.937233+0000","last_fullsized":"2024-03-09T15:36:05.937233+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":100,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:46:26.657945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3560140,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":140,"num_read_kb":3004,"num_write":507,"num_write_kb":19720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"261'89","reported_seq":655,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:39.894721+0000","last_change":"2024-03-09T14:28:34.324788+0000","last_active":"2024-03-09T15:34:39.894721+0000","last_peered":"2024-03-09T15:34:39.894721+0000","last_clean":"2024-03-09T15:34:39.894721+0000","last_became_active":"2024-03-09T14:28:34.324053+0000","last_became_peered":"2024-03-09T14:28:34.324053+0000","last_unstale":"2024-03-09T15:34:39.894721+0000","last_undegraded":"2024-03-09T15:34:39.894721+0000","last_fullsized":"2024-03-09T15:34:39.894721+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:16:16.817770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":372,"num_read_kb":300,"num_write":141,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3d","version":"261'135","reported_seq":695,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.959352+0000","last_change":"2024-03-09T14:28:32.292874+0000","last_active":"2024-03-09T15:36:05.959352+0000","last_peered":"2024-03-09T15:36:05.959352+0000","last_clean":"2024-03-09T15:36:05.959352+0000","last_became_active":"2024-03-09T14:28:32.286966+0000","last_became_peered":"2024-03-09T14:28:32.286966+0000","last_unstale":"2024-03-09T15:36:05.959352+0000","last_undegraded":"2024-03-09T15:36:05.959352+0000","last_fullsized":"2024-03-09T15:36:05.959352+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":100,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:34.280241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291735,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":170,"num_read_kb":14171,"num_write":587,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"261'157","reported_seq":718,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191267+0000","last_change":"2024-03-09T14:28:34.307655+0000","last_active":"2024-03-09T15:33:36.191267+0000","last_peered":"2024-03-09T15:33:36.191267+0000","last_clean":"2024-03-09T15:33:36.191267+0000","last_became_active":"2024-03-09T14:28:34.306764+0000","last_became_peered":"2024-03-09T14:28:34.306764+0000","last_unstale":"2024-03-09T15:33:36.191267+0000","last_undegraded":"2024-03-09T15:33:36.191267+0000","last_fullsized":"2024-03-09T15:33:36.191267+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:19:26.321622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":481,"num_write":261,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3c","version":"261'177","reported_seq":769,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.211122+0000","last_change":"2024-03-09T14:28:32.310328+0000","last_active":"2024-03-09T15:36:06.211122+0000","last_peered":"2024-03-09T15:36:06.211122+0000","last_clean":"2024-03-09T15:36:06.211122+0000","last_became_active":"2024-03-09T14:28:32.305615+0000","last_became_peered":"2024-03-09T14:28:32.305615+0000","last_unstale":"2024-03-09T15:36:06.211122+0000","last_undegraded":"2024-03-09T15:36:06.211122+0000","last_fullsized":"2024-03-09T15:36:06.211122+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":100,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:45:12.810858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10569896,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":243,"num_read_kb":28937,"num_write":784,"num_write_kb":45835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"261'156","reported_seq":732,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.012477+0000","last_change":"2024-03-09T14:28:34.316937+0000","last_active":"2024-03-09T15:33:36.012477+0000","last_peered":"2024-03-09T15:33:36.012477+0000","last_clean":"2024-03-09T15:33:36.012477+0000","last_became_active":"2024-03-09T14:28:34.316500+0000","last_became_peered":"2024-03-09T14:28:34.316500+0000","last_unstale":"2024-03-09T15:33:36.012477+0000","last_undegraded":"2024-03-09T15:33:36.012477+0000","last_fullsized":"2024-03-09T15:33:36.012477+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":100,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:11:35.701163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":622,"num_read_kb":488,"num_write":257,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3b","version":"261'135","reported_seq":797,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.227826+0000","last_change":"2024-03-09T14:28:32.317675+0000","last_active":"2024-03-09T15:36:06.227826+0000","last_peered":"2024-03-09T15:36:06.227826+0000","last_clean":"2024-03-09T15:36:06.227826+0000","last_became_active":"2024-03-09T14:28:32.314923+0000","last_became_peered":"2024-03-09T14:28:32.314923+0000","last_unstale":"2024-03-09T15:36:06.227826+0000","last_undegraded":"2024-03-09T15:36:06.227826+0000","last_fullsized":"2024-03-09T15:36:06.227826+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":100,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:29:14.419941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1918192,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":282,"num_read_kb":17282,"num_write":502,"num_write_kb":14747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3c","version":"261'124","reported_seq":706,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.781267+0000","last_change":"2024-03-09T14:28:34.315343+0000","last_active":"2024-03-09T15:32:55.781267+0000","last_peered":"2024-03-09T15:32:55.781267+0000","last_clean":"2024-03-09T15:32:55.781267+0000","last_became_active":"2024-03-09T14:28:34.314617+0000","last_became_peered":"2024-03-09T14:28:34.314617+0000","last_unstale":"2024-03-09T15:32:55.781267+0000","last_undegraded":"2024-03-09T15:32:55.781267+0000","last_fullsized":"2024-03-09T15:32:55.781267+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:05:14.469183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":493,"num_read_kb":395,"num_write":185,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3a","version":"261'128","reported_seq":661,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.870081+0000","last_change":"2024-03-09T14:29:06.433441+0000","last_active":"2024-03-09T15:36:05.870081+0000","last_peered":"2024-03-09T15:36:05.870081+0000","last_clean":"2024-03-09T15:36:05.870081+0000","last_became_active":"2024-03-09T14:29:06.433145+0000","last_became_peered":"2024-03-09T14:29:06.433145+0000","last_unstale":"2024-03-09T15:36:05.870081+0000","last_undegraded":"2024-03-09T15:36:05.870081+0000","last_fullsized":"2024-03-09T15:36:05.870081+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:01:58.493721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3195196,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":189,"num_read_kb":6443,"num_write":585,"num_write_kb":33049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"261'274","reported_seq":939,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.185860+0000","last_change":"2024-03-09T14:28:34.314501+0000","last_active":"2024-03-09T15:33:11.185860+0000","last_peered":"2024-03-09T15:33:11.185860+0000","last_clean":"2024-03-09T15:33:11.185860+0000","last_became_active":"2024-03-09T14:28:34.310343+0000","last_became_peered":"2024-03-09T14:28:34.310343+0000","last_unstale":"2024-03-09T15:33:11.185860+0000","last_undegraded":"2024-03-09T15:33:11.185860+0000","last_fullsized":"2024-03-09T15:33:11.185860+0000","mapping_epoch":22,"log_start":"247'200","ondisk_log_start":"247'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":200,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:36:50.027757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1205,"num_read_kb":1028,"num_write":418,"num_write_kb":253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"261'123","reported_seq":679,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.218061+0000","last_change":"2024-03-09T14:28:32.314711+0000","last_active":"2024-03-09T15:36:06.218061+0000","last_peered":"2024-03-09T15:36:06.218061+0000","last_clean":"2024-03-09T15:36:06.218061+0000","last_became_active":"2024-03-09T14:28:32.314258+0000","last_became_peered":"2024-03-09T14:28:32.314258+0000","last_unstale":"2024-03-09T15:36:06.218061+0000","last_undegraded":"2024-03-09T15:36:06.218061+0000","last_fullsized":"2024-03-09T15:36:06.218061+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":100,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:21:00.179690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262376,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":159,"num_read_kb":17581,"num_write":456,"num_write_kb":22929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3e","version":"261'44","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.189153+0000","last_change":"2024-03-09T14:28:34.323080+0000","last_active":"2024-03-09T15:33:11.189153+0000","last_peered":"2024-03-09T15:33:11.189153+0000","last_clean":"2024-03-09T15:33:11.189153+0000","last_became_active":"2024-03-09T14:28:34.322206+0000","last_became_peered":"2024-03-09T14:28:34.322206+0000","last_unstale":"2024-03-09T15:33:11.189153+0000","last_undegraded":"2024-03-09T15:33:11.189153+0000","last_fullsized":"2024-03-09T15:33:11.189153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:55:41.173766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":201,"num_read_kb":171,"num_write":60,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"261'128","reported_seq":674,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.803635+0000","last_change":"2024-03-09T14:28:32.308873+0000","last_active":"2024-03-09T15:33:35.803635+0000","last_peered":"2024-03-09T15:33:35.803635+0000","last_clean":"2024-03-09T15:33:35.803635+0000","last_became_active":"2024-03-09T14:28:32.308522+0000","last_became_peered":"2024-03-09T14:28:32.308522+0000","last_unstale":"2024-03-09T15:33:35.803635+0000","last_undegraded":"2024-03-09T15:33:35.803635+0000","last_fullsized":"2024-03-09T15:33:35.803635+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:10:42.961664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6817986,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":153,"num_read_kb":14130,"num_write":524,"num_write_kb":36844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"261'52","reported_seq":603,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.971792+0000","last_change":"2024-03-09T14:28:34.321740+0000","last_active":"2024-03-09T15:33:10.971792+0000","last_peered":"2024-03-09T15:33:10.971792+0000","last_clean":"2024-03-09T15:33:10.971792+0000","last_became_active":"2024-03-09T14:28:34.320646+0000","last_became_peered":"2024-03-09T14:28:34.320646+0000","last_unstale":"2024-03-09T15:33:10.971792+0000","last_undegraded":"2024-03-09T15:33:10.971792+0000","last_fullsized":"2024-03-09T15:33:10.971792+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:32:54.751077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":174,"num_write":79,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"261'157","reported_seq":741,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.897610+0000","last_change":"2024-03-09T14:28:34.307366+0000","last_active":"2024-03-09T15:31:36.897610+0000","last_peered":"2024-03-09T15:31:36.897610+0000","last_clean":"2024-03-09T15:31:36.897610+0000","last_became_active":"2024-03-09T14:28:34.306592+0000","last_became_peered":"2024-03-09T14:28:34.306592+0000","last_unstale":"2024-03-09T15:31:36.897610+0000","last_undegraded":"2024-03-09T15:31:36.897610+0000","last_fullsized":"2024-03-09T15:31:36.897610+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:38:03.684472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":599,"num_read_kb":473,"num_write":232,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"261'147","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.857715+0000","last_change":"2024-03-09T14:29:06.438746+0000","last_active":"2024-03-09T15:36:05.857715+0000","last_peered":"2024-03-09T15:36:05.857715+0000","last_clean":"2024-03-09T15:36:05.857715+0000","last_became_active":"2024-03-09T14:29:06.438523+0000","last_became_peered":"2024-03-09T14:29:06.438523+0000","last_unstale":"2024-03-09T15:36:05.857715+0000","last_undegraded":"2024-03-09T15:36:05.857715+0000","last_fullsized":"2024-03-09T15:36:05.857715+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":100,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:59.554428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728952,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":279,"num_read_kb":1034,"num_write":583,"num_write_kb":27895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.31","version":"261'88","reported_seq":642,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.176622+0000","last_change":"2024-03-09T14:28:34.322326+0000","last_active":"2024-03-09T15:31:36.176622+0000","last_peered":"2024-03-09T15:31:36.176622+0000","last_clean":"2024-03-09T15:31:36.176622+0000","last_became_active":"2024-03-09T14:28:34.321352+0000","last_became_peered":"2024-03-09T14:28:34.321352+0000","last_unstale":"2024-03-09T15:31:36.176622+0000","last_undegraded":"2024-03-09T15:31:36.176622+0000","last_fullsized":"2024-03-09T15:31:36.176622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:00:49.569838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":342,"num_read_kb":270,"num_write":126,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"261'105","reported_seq":646,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.762868+0000","last_change":"2024-03-09T14:28:32.337035+0000","last_active":"2024-03-09T15:31:25.762868+0000","last_peered":"2024-03-09T15:31:25.762868+0000","last_clean":"2024-03-09T15:31:25.762868+0000","last_became_active":"2024-03-09T14:28:32.334208+0000","last_became_peered":"2024-03-09T14:28:32.334208+0000","last_unstale":"2024-03-09T15:31:25.762868+0000","last_undegraded":"2024-03-09T15:31:25.762868+0000","last_fullsized":"2024-03-09T15:31:25.762868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:15:51.011829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262534,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":117,"num_read_kb":128,"num_write":382,"num_write_kb":22112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"261'62","reported_seq":608,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:46.170836+0000","last_change":"2024-03-09T14:28:34.299187+0000","last_active":"2024-03-09T15:31:46.170836+0000","last_peered":"2024-03-09T15:31:46.170836+0000","last_clean":"2024-03-09T15:31:46.170836+0000","last_became_active":"2024-03-09T14:28:34.298428+0000","last_became_peered":"2024-03-09T14:28:34.298428+0000","last_unstale":"2024-03-09T15:31:46.170836+0000","last_undegraded":"2024-03-09T15:31:46.170836+0000","last_fullsized":"2024-03-09T15:31:46.170836+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:07.549211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":202,"num_write":92,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"261'96","reported_seq":651,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:43.752932+0000","last_change":"2024-03-09T14:28:32.333004+0000","last_active":"2024-03-09T15:34:43.752932+0000","last_peered":"2024-03-09T15:34:43.752932+0000","last_clean":"2024-03-09T15:34:43.752932+0000","last_became_active":"2024-03-09T14:28:32.318963+0000","last_became_peered":"2024-03-09T14:28:32.318963+0000","last_unstale":"2024-03-09T15:34:43.752932+0000","last_undegraded":"2024-03-09T15:34:43.752932+0000","last_fullsized":"2024-03-09T15:34:43.752932+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:52:04.706747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208569,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":146,"num_read_kb":1700,"num_write":417,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.33","version":"261'102","reported_seq":663,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.950025+0000","last_change":"2024-03-09T14:28:34.320630+0000","last_active":"2024-03-09T15:31:25.950025+0000","last_peered":"2024-03-09T15:31:25.950025+0000","last_clean":"2024-03-09T15:31:25.950025+0000","last_became_active":"2024-03-09T14:28:34.320331+0000","last_became_peered":"2024-03-09T14:28:34.320331+0000","last_unstale":"2024-03-09T15:31:25.950025+0000","last_undegraded":"2024-03-09T15:31:25.950025+0000","last_fullsized":"2024-03-09T15:31:25.950025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:03:19.130527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":320,"num_write":144,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"261'114","reported_seq":666,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.967812+0000","last_change":"2024-03-09T14:28:32.338458+0000","last_active":"2024-03-09T15:36:05.967812+0000","last_peered":"2024-03-09T15:36:05.967812+0000","last_clean":"2024-03-09T15:36:05.967812+0000","last_became_active":"2024-03-09T14:28:32.337934+0000","last_became_peered":"2024-03-09T14:28:32.337934+0000","last_unstale":"2024-03-09T15:36:05.967812+0000","last_undegraded":"2024-03-09T15:36:05.967812+0000","last_fullsized":"2024-03-09T15:36:05.967812+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:15:21.515518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554899,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":147,"num_read_kb":12763,"num_write":449,"num_write_kb":21372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"261'69","reported_seq":605,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.950065+0000","last_change":"2024-03-09T14:28:34.311072+0000","last_active":"2024-03-09T15:31:25.950065+0000","last_peered":"2024-03-09T15:31:25.950065+0000","last_clean":"2024-03-09T15:31:25.950065+0000","last_became_active":"2024-03-09T14:28:34.310703+0000","last_became_peered":"2024-03-09T14:28:34.310703+0000","last_unstale":"2024-03-09T15:31:25.950065+0000","last_undegraded":"2024-03-09T15:31:25.950065+0000","last_fullsized":"2024-03-09T15:31:25.950065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:16:09.385282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":272,"num_read_kb":220,"num_write":109,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"261'133","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.964740+0000","last_change":"2024-03-09T14:28:32.337031+0000","last_active":"2024-03-09T15:31:41.964740+0000","last_peered":"2024-03-09T15:31:41.964740+0000","last_clean":"2024-03-09T15:31:41.964740+0000","last_became_active":"2024-03-09T14:28:32.336636+0000","last_became_peered":"2024-03-09T14:28:32.336636+0000","last_unstale":"2024-03-09T15:31:41.964740+0000","last_undegraded":"2024-03-09T15:31:41.964740+0000","last_fullsized":"2024-03-09T15:31:41.964740+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:44:26.611692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262536,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":207,"num_read_kb":12490,"num_write":554,"num_write_kb":28462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"261'87","reported_seq":637,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162736+0000","last_change":"2024-03-09T14:28:34.316215+0000","last_active":"2024-03-09T15:31:26.162736+0000","last_peered":"2024-03-09T15:31:26.162736+0000","last_clean":"2024-03-09T15:31:26.162736+0000","last_became_active":"2024-03-09T14:28:34.314665+0000","last_became_peered":"2024-03-09T14:28:34.314665+0000","last_unstale":"2024-03-09T15:31:26.162736+0000","last_undegraded":"2024-03-09T15:31:26.162736+0000","last_fullsized":"2024-03-09T15:31:26.162736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:15.743713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":335,"num_read_kb":265,"num_write":124,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"261'107","reported_seq":668,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.937008+0000","last_change":"2024-03-09T14:28:32.298846+0000","last_active":"2024-03-09T15:36:05.937008+0000","last_peered":"2024-03-09T15:36:05.937008+0000","last_clean":"2024-03-09T15:36:05.937008+0000","last_became_active":"2024-03-09T14:28:32.298428+0000","last_became_peered":"2024-03-09T14:28:32.298428+0000","last_unstale":"2024-03-09T15:36:05.937008+0000","last_undegraded":"2024-03-09T15:36:05.937008+0000","last_fullsized":"2024-03-09T15:36:05.937008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:18:37.829113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583035,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":178,"num_read_kb":31150,"num_write":451,"num_write_kb":37586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"261'76","reported_seq":581,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:07.654968+0000","last_change":"2024-03-09T14:28:42.404373+0000","last_active":"2024-03-09T15:35:07.654968+0000","last_peered":"2024-03-09T15:35:07.654968+0000","last_clean":"2024-03-09T15:35:07.654968+0000","last_became_active":"2024-03-09T14:28:42.403553+0000","last_became_peered":"2024-03-09T14:28:42.403553+0000","last_unstale":"2024-03-09T15:35:07.654968+0000","last_undegraded":"2024-03-09T15:35:07.654968+0000","last_fullsized":"2024-03-09T15:35:07.654968+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:16:41.740933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":84,"num_write":184,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"261'98","reported_seq":664,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.780525+0000","last_change":"2024-03-09T14:29:06.439713+0000","last_active":"2024-03-09T15:31:50.780525+0000","last_peered":"2024-03-09T15:31:50.780525+0000","last_clean":"2024-03-09T15:31:50.780525+0000","last_became_active":"2024-03-09T14:29:06.439359+0000","last_became_peered":"2024-03-09T14:29:06.439359+0000","last_unstale":"2024-03-09T15:31:50.780525+0000","last_undegraded":"2024-03-09T15:31:50.780525+0000","last_fullsized":"2024-03-09T15:31:50.780525+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:01:04.305529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":413,"num_read_kb":333,"num_write":149,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"261'154","reported_seq":775,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.211631+0000","last_change":"2024-03-09T14:28:32.330235+0000","last_active":"2024-03-09T15:36:06.211631+0000","last_peered":"2024-03-09T15:36:06.211631+0000","last_clean":"2024-03-09T15:36:06.211631+0000","last_became_active":"2024-03-09T14:28:32.329457+0000","last_became_peered":"2024-03-09T14:28:32.329457+0000","last_unstale":"2024-03-09T15:36:06.211631+0000","last_undegraded":"2024-03-09T15:36:06.211631+0000","last_fullsized":"2024-03-09T15:36:06.211631+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":100,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:30:37.270692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5813235,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":252,"num_read_kb":141328,"num_write":592,"num_write_kb":37022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.166374+0000","last_change":"2024-03-09T14:28:36.321332+0000","last_active":"2024-03-09T15:31:26.166374+0000","last_peered":"2024-03-09T15:31:26.166374+0000","last_clean":"2024-03-09T15:31:26.166374+0000","last_became_active":"2024-03-09T14:28:36.320603+0000","last_became_peered":"2024-03-09T14:28:36.320603+0000","last_unstale":"2024-03-09T15:31:26.166374+0000","last_undegraded":"2024-03-09T15:31:26.166374+0000","last_fullsized":"2024-03-09T15:31:26.166374+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:20:34.317764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"261'226","reported_seq":832,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.691237+0000","last_change":"2024-03-09T14:28:38.327507+0000","last_active":"2024-03-09T15:35:58.691237+0000","last_peered":"2024-03-09T15:35:58.691237+0000","last_clean":"2024-03-09T15:35:58.691237+0000","last_became_active":"2024-03-09T14:28:38.326826+0000","last_became_peered":"2024-03-09T14:28:38.326826+0000","last_unstale":"2024-03-09T15:35:58.691237+0000","last_undegraded":"2024-03-09T15:35:58.691237+0000","last_fullsized":"2024-03-09T15:35:58.691237+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":200,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:46:18.261500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24948,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":356,"num_read_kb":580,"num_write":246,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.175558+0000","last_change":"2024-03-09T14:28:40.359830+0000","last_active":"2024-03-09T15:31:31.175558+0000","last_peered":"2024-03-09T15:31:31.175558+0000","last_clean":"2024-03-09T15:31:31.175558+0000","last_became_active":"2024-03-09T14:28:40.358954+0000","last_became_peered":"2024-03-09T14:28:40.358954+0000","last_unstale":"2024-03-09T15:31:31.175558+0000","last_undegraded":"2024-03-09T15:31:31.175558+0000","last_fullsized":"2024-03-09T15:31:31.175558+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:34.169828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"261'770","reported_seq":2383,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:53.987578+0000","last_change":"2024-03-09T14:28:42.395382+0000","last_active":"2024-03-09T15:35:53.987578+0000","last_peered":"2024-03-09T15:35:53.987578+0000","last_clean":"2024-03-09T15:35:53.987578+0000","last_became_active":"2024-03-09T14:28:42.394691+0000","last_became_peered":"2024-03-09T14:28:42.394691+0000","last_unstale":"2024-03-09T15:35:53.987578+0000","last_undegraded":"2024-03-09T15:35:53.987578+0000","last_fullsized":"2024-03-09T15:35:53.987578+0000","mapping_epoch":30,"log_start":"261'700","ondisk_log_start":"261'700","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":700,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:48.486613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":612,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2611,"num_read_kb":2601,"num_write":1560,"num_write_kb":569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"261'40","reported_seq":578,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:28.764187+0000","last_change":"2024-03-09T14:28:34.326083+0000","last_active":"2024-03-09T15:31:28.764187+0000","last_peered":"2024-03-09T15:31:28.764187+0000","last_clean":"2024-03-09T15:31:28.764187+0000","last_became_active":"2024-03-09T14:28:34.324569+0000","last_became_peered":"2024-03-09T14:28:34.324569+0000","last_unstale":"2024-03-09T15:31:28.764187+0000","last_undegraded":"2024-03-09T15:31:28.764187+0000","last_fullsized":"2024-03-09T15:31:28.764187+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:18.214769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":168,"num_read_kb":136,"num_write":58,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"261'145","reported_seq":715,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.866230+0000","last_change":"2024-03-09T14:28:32.310039+0000","last_active":"2024-03-09T15:36:05.866230+0000","last_peered":"2024-03-09T15:36:05.866230+0000","last_clean":"2024-03-09T15:36:05.866230+0000","last_became_active":"2024-03-09T14:28:32.309536+0000","last_became_peered":"2024-03-09T14:28:32.309536+0000","last_unstale":"2024-03-09T15:36:05.866230+0000","last_undegraded":"2024-03-09T15:36:05.866230+0000","last_fullsized":"2024-03-09T15:36:05.866230+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:28:43.638731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5530768,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":194,"num_read_kb":29533,"num_write":628,"num_write_kb":30801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.780570+0000","last_change":"2024-03-09T14:28:36.320450+0000","last_active":"2024-03-09T15:31:50.780570+0000","last_peered":"2024-03-09T15:31:50.780570+0000","last_clean":"2024-03-09T15:31:50.780570+0000","last_became_active":"2024-03-09T14:28:36.320079+0000","last_became_peered":"2024-03-09T14:28:36.320079+0000","last_unstale":"2024-03-09T15:31:50.780570+0000","last_undegraded":"2024-03-09T15:31:50.780570+0000","last_fullsized":"2024-03-09T15:31:50.780570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:19:38.894047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"261'87","reported_seq":623,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:52.371528+0000","last_change":"2024-03-09T14:28:38.341864+0000","last_active":"2024-03-09T15:31:52.371528+0000","last_peered":"2024-03-09T15:31:52.371528+0000","last_clean":"2024-03-09T15:31:52.371528+0000","last_became_active":"2024-03-09T14:28:38.340896+0000","last_became_peered":"2024-03-09T14:28:38.340896+0000","last_unstale":"2024-03-09T15:31:52.371528+0000","last_undegraded":"2024-03-09T15:31:52.371528+0000","last_fullsized":"2024-03-09T15:31:52.371528+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:19:17.825977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":101,"num_write":68,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.163891+0000","last_change":"2024-03-09T14:28:40.371660+0000","last_active":"2024-03-09T15:31:31.163891+0000","last_peered":"2024-03-09T15:31:31.163891+0000","last_clean":"2024-03-09T15:31:31.163891+0000","last_became_active":"2024-03-09T14:28:40.370788+0000","last_became_peered":"2024-03-09T14:28:40.370788+0000","last_unstale":"2024-03-09T15:31:31.163891+0000","last_undegraded":"2024-03-09T15:31:31.163891+0000","last_fullsized":"2024-03-09T15:31:31.163891+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:12:43.937289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"261'59","reported_seq":561,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.285043+0000","last_change":"2024-03-09T14:28:42.411392+0000","last_active":"2024-03-09T15:33:50.285043+0000","last_peered":"2024-03-09T15:33:50.285043+0000","last_clean":"2024-03-09T15:33:50.285043+0000","last_became_active":"2024-03-09T14:28:42.408738+0000","last_became_peered":"2024-03-09T14:28:42.408738+0000","last_unstale":"2024-03-09T15:33:50.285043+0000","last_undegraded":"2024-03-09T15:33:50.285043+0000","last_fullsized":"2024-03-09T15:33:50.285043+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:05:13.992640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":681,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":76,"num_read_kb":75,"num_write":140,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"261'157","reported_seq":745,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:31.430581+0000","last_change":"2024-03-09T14:28:34.305220+0000","last_active":"2024-03-09T15:33:31.430581+0000","last_peered":"2024-03-09T15:33:31.430581+0000","last_clean":"2024-03-09T15:33:31.430581+0000","last_became_active":"2024-03-09T14:28:34.304892+0000","last_became_peered":"2024-03-09T14:28:34.304892+0000","last_unstale":"2024-03-09T15:33:31.430581+0000","last_undegraded":"2024-03-09T15:33:31.430581+0000","last_fullsized":"2024-03-09T15:33:31.430581+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:10:30.883636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":622,"num_read_kb":492,"num_write":239,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"261'148","reported_seq":848,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.211631+0000","last_change":"2024-03-09T14:28:32.319206+0000","last_active":"2024-03-09T15:36:06.211631+0000","last_peered":"2024-03-09T15:36:06.211631+0000","last_clean":"2024-03-09T15:36:06.211631+0000","last_became_active":"2024-03-09T14:28:32.318428+0000","last_became_peered":"2024-03-09T14:28:32.318428+0000","last_unstale":"2024-03-09T15:36:06.211631+0000","last_undegraded":"2024-03-09T15:36:06.211631+0000","last_fullsized":"2024-03-09T15:36:06.211631+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":100,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:59:15.432941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6019791,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":531,"num_read_kb":113871,"num_write":606,"num_write_kb":46622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.774459+0000","last_change":"2024-03-09T14:28:36.320826+0000","last_active":"2024-03-09T15:31:25.774459+0000","last_peered":"2024-03-09T15:31:25.774459+0000","last_clean":"2024-03-09T15:31:25.774459+0000","last_became_active":"2024-03-09T14:28:36.320095+0000","last_became_peered":"2024-03-09T14:28:36.320095+0000","last_unstale":"2024-03-09T15:31:25.774459+0000","last_undegraded":"2024-03-09T15:31:25.774459+0000","last_fullsized":"2024-03-09T15:31:25.774459+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:58:26.878574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"261'103","reported_seq":653,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:02.768040+0000","last_change":"2024-03-09T14:28:38.339553+0000","last_active":"2024-03-09T15:34:02.768040+0000","last_peered":"2024-03-09T15:34:02.768040+0000","last_clean":"2024-03-09T15:34:02.768040+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:34:02.768040+0000","last_undegraded":"2024-03-09T15:34:02.768040+0000","last_fullsized":"2024-03-09T15:34:02.768040+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:37:31.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"261'504","reported_seq":2290,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.019957+0000","last_change":"2024-03-09T14:28:40.373218+0000","last_active":"2024-03-09T15:36:06.019957+0000","last_peered":"2024-03-09T15:36:06.019957+0000","last_clean":"2024-03-09T15:36:06.019957+0000","last_became_active":"2024-03-09T14:28:40.371507+0000","last_became_peered":"2024-03-09T14:28:40.371507+0000","last_unstale":"2024-03-09T15:36:06.019957+0000","last_undegraded":"2024-03-09T15:36:06.019957+0000","last_fullsized":"2024-03-09T15:36:06.019957+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":400,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:20:18.095183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"261'136","reported_seq":941,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:16.160267+0000","last_change":"2024-03-09T14:28:42.399490+0000","last_active":"2024-03-09T15:31:16.160267+0000","last_peered":"2024-03-09T15:31:16.160267+0000","last_clean":"2024-03-09T15:31:16.160267+0000","last_became_active":"2024-03-09T14:28:42.398498+0000","last_became_peered":"2024-03-09T14:28:42.398498+0000","last_unstale":"2024-03-09T15:31:16.160267+0000","last_undegraded":"2024-03-09T15:31:16.160267+0000","last_fullsized":"2024-03-09T15:31:16.160267+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:10:05.573339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":498,"num_read_kb":498,"num_write":277,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"261'53","reported_seq":592,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.774411+0000","last_change":"2024-03-09T14:28:34.330792+0000","last_active":"2024-03-09T15:31:25.774411+0000","last_peered":"2024-03-09T15:31:25.774411+0000","last_clean":"2024-03-09T15:31:25.774411+0000","last_became_active":"2024-03-09T14:28:34.330347+0000","last_became_peered":"2024-03-09T14:28:34.330347+0000","last_unstale":"2024-03-09T15:31:25.774411+0000","last_undegraded":"2024-03-09T15:31:25.774411+0000","last_fullsized":"2024-03-09T15:31:25.774411+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:55:20.226431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":212,"num_read_kb":171,"num_write":77,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"261'136","reported_seq":708,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.764851+0000","last_change":"2024-03-09T14:28:32.332979+0000","last_active":"2024-03-09T15:31:35.764851+0000","last_peered":"2024-03-09T15:31:35.764851+0000","last_clean":"2024-03-09T15:31:35.764851+0000","last_became_active":"2024-03-09T14:28:32.319819+0000","last_became_peered":"2024-03-09T14:28:32.319819+0000","last_unstale":"2024-03-09T15:31:35.764851+0000","last_undegraded":"2024-03-09T15:31:35.764851+0000","last_fullsized":"2024-03-09T15:31:35.764851+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:01:46.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5244864,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":201,"num_read_kb":15567,"num_write":549,"num_write_kb":34195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.166450+0000","last_change":"2024-03-09T14:28:36.311535+0000","last_active":"2024-03-09T15:31:26.166450+0000","last_peered":"2024-03-09T15:31:26.166450+0000","last_clean":"2024-03-09T15:31:26.166450+0000","last_became_active":"2024-03-09T14:28:36.310231+0000","last_became_peered":"2024-03-09T14:28:36.310231+0000","last_unstale":"2024-03-09T15:31:26.166450+0000","last_undegraded":"2024-03-09T15:31:26.166450+0000","last_fullsized":"2024-03-09T15:31:26.166450+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:55:50.175612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"261'167","reported_seq":746,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.679604+0000","last_change":"2024-03-09T14:28:38.599078+0000","last_active":"2024-03-09T15:35:58.679604+0000","last_peered":"2024-03-09T15:35:58.679604+0000","last_clean":"2024-03-09T15:35:58.679604+0000","last_became_active":"2024-03-09T14:28:38.598308+0000","last_became_peered":"2024-03-09T14:28:38.598308+0000","last_unstale":"2024-03-09T15:35:58.679604+0000","last_undegraded":"2024-03-09T15:35:58.679604+0000","last_fullsized":"2024-03-09T15:35:58.679604+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":100,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:10:09.151032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23129,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":269,"num_read_kb":489,"num_write":186,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"261'512","reported_seq":2498,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.733534+0000","last_change":"2024-03-09T14:28:40.388710+0000","last_active":"2024-03-09T15:36:05.733534+0000","last_peered":"2024-03-09T15:36:05.733534+0000","last_clean":"2024-03-09T15:36:05.733534+0000","last_became_active":"2024-03-09T14:28:40.387990+0000","last_became_peered":"2024-03-09T14:28:40.387990+0000","last_unstale":"2024-03-09T15:36:05.733534+0000","last_undegraded":"2024-03-09T15:36:05.733534+0000","last_fullsized":"2024-03-09T15:36:05.733534+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":500,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:21:47.017030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"261'101","reported_seq":678,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194321+0000","last_change":"2024-03-09T14:28:34.323491+0000","last_active":"2024-03-09T15:33:36.194321+0000","last_peered":"2024-03-09T15:33:36.194321+0000","last_clean":"2024-03-09T15:33:36.194321+0000","last_became_active":"2024-03-09T14:28:34.322731+0000","last_became_peered":"2024-03-09T14:28:34.322731+0000","last_unstale":"2024-03-09T15:33:36.194321+0000","last_undegraded":"2024-03-09T15:33:36.194321+0000","last_fullsized":"2024-03-09T15:33:36.194321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:34:36.142132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":330,"num_write":141,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"261'114","reported_seq":662,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.217253+0000","last_change":"2024-03-09T14:28:32.329755+0000","last_active":"2024-03-09T15:36:06.217253+0000","last_peered":"2024-03-09T15:36:06.217253+0000","last_clean":"2024-03-09T15:36:06.217253+0000","last_became_active":"2024-03-09T14:28:32.329148+0000","last_became_peered":"2024-03-09T14:28:32.329148+0000","last_unstale":"2024-03-09T15:36:06.217253+0000","last_undegraded":"2024-03-09T15:36:06.217253+0000","last_fullsized":"2024-03-09T15:36:06.217253+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:07:39.356330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7949644,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":138,"num_read_kb":1766,"num_write":504,"num_write_kb":18289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205246+0000","last_change":"2024-03-09T14:28:36.314780+0000","last_active":"2024-03-09T15:33:36.205246+0000","last_peered":"2024-03-09T15:33:36.205246+0000","last_clean":"2024-03-09T15:33:36.205246+0000","last_became_active":"2024-03-09T14:28:36.313904+0000","last_became_peered":"2024-03-09T14:28:36.313904+0000","last_unstale":"2024-03-09T15:33:36.205246+0000","last_undegraded":"2024-03-09T15:33:36.205246+0000","last_fullsized":"2024-03-09T15:33:36.205246+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:57:41.431102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"261'265","reported_seq":896,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:01.226737+0000","last_change":"2024-03-09T14:28:38.598064+0000","last_active":"2024-03-09T15:36:01.226737+0000","last_peered":"2024-03-09T15:36:01.226737+0000","last_clean":"2024-03-09T15:36:01.226737+0000","last_became_active":"2024-03-09T14:28:38.597247+0000","last_became_peered":"2024-03-09T14:28:38.597247+0000","last_unstale":"2024-03-09T15:36:01.226737+0000","last_undegraded":"2024-03-09T15:36:01.226737+0000","last_fullsized":"2024-03-09T15:36:01.226737+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":200,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:35:30.643121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22355,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":412,"num_read_kb":625,"num_write":280,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":495,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.134883+0000","last_change":"2024-03-09T14:28:40.360962+0000","last_active":"2024-03-09T15:31:41.134883+0000","last_peered":"2024-03-09T15:31:41.134883+0000","last_clean":"2024-03-09T15:31:41.134883+0000","last_became_active":"2024-03-09T14:28:40.359906+0000","last_became_peered":"2024-03-09T14:28:40.359906+0000","last_unstale":"2024-03-09T15:31:41.134883+0000","last_undegraded":"2024-03-09T15:31:41.134883+0000","last_fullsized":"2024-03-09T15:31:41.134883+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:33:27.073677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"261'518","reported_seq":1356,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.729973+0000","last_change":"2024-03-09T14:28:42.403475+0000","last_active":"2024-03-09T15:35:58.729973+0000","last_peered":"2024-03-09T15:35:58.729973+0000","last_clean":"2024-03-09T15:35:58.729973+0000","last_became_active":"2024-03-09T14:28:42.402739+0000","last_became_peered":"2024-03-09T14:28:42.402739+0000","last_unstale":"2024-03-09T15:35:58.729973+0000","last_undegraded":"2024-03-09T15:35:58.729973+0000","last_fullsized":"2024-03-09T15:35:58.729973+0000","mapping_epoch":30,"log_start":"261'500","ondisk_log_start":"261'500","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":500,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:49:57.794992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1336,"num_read_kb":1311,"num_write":1083,"num_write_kb":376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"261'92","reported_seq":662,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191097+0000","last_change":"2024-03-09T14:28:34.302153+0000","last_active":"2024-03-09T15:33:36.191097+0000","last_peered":"2024-03-09T15:33:36.191097+0000","last_clean":"2024-03-09T15:33:36.191097+0000","last_became_active":"2024-03-09T14:28:34.300795+0000","last_became_peered":"2024-03-09T14:28:34.300795+0000","last_unstale":"2024-03-09T15:33:36.191097+0000","last_undegraded":"2024-03-09T15:33:36.191097+0000","last_fullsized":"2024-03-09T15:33:36.191097+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:27:31.948784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":361,"num_read_kb":288,"num_write":150,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"261'133","reported_seq":706,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.234898+0000","last_change":"2024-03-09T14:28:32.318853+0000","last_active":"2024-03-09T15:36:06.234898+0000","last_peered":"2024-03-09T15:36:06.234898+0000","last_clean":"2024-03-09T15:36:06.234898+0000","last_became_active":"2024-03-09T14:28:32.318111+0000","last_became_peered":"2024-03-09T14:28:32.318111+0000","last_unstale":"2024-03-09T15:36:06.234898+0000","last_undegraded":"2024-03-09T15:36:06.234898+0000","last_fullsized":"2024-03-09T15:36:06.234898+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:51.120639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5549179,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":206,"num_read_kb":6230,"num_write":588,"num_write_kb":19779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.976641+0000","last_change":"2024-03-09T14:28:36.301940+0000","last_active":"2024-03-09T15:33:35.976641+0000","last_peered":"2024-03-09T15:33:35.976641+0000","last_clean":"2024-03-09T15:33:35.976641+0000","last_became_active":"2024-03-09T14:28:36.301227+0000","last_became_peered":"2024-03-09T14:28:36.301227+0000","last_unstale":"2024-03-09T15:33:35.976641+0000","last_undegraded":"2024-03-09T15:33:35.976641+0000","last_fullsized":"2024-03-09T15:33:35.976641+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:55:41.843139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"261'170","reported_seq":742,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.654250+0000","last_change":"2024-03-09T14:28:38.347678+0000","last_active":"2024-03-09T15:35:58.654250+0000","last_peered":"2024-03-09T15:35:58.654250+0000","last_clean":"2024-03-09T15:35:58.654250+0000","last_became_active":"2024-03-09T14:28:38.346837+0000","last_became_peered":"2024-03-09T14:28:38.346837+0000","last_unstale":"2024-03-09T15:35:58.654250+0000","last_undegraded":"2024-03-09T15:35:58.654250+0000","last_fullsized":"2024-03-09T15:35:58.654250+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":100,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:06.776774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28902,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":267,"num_read_kb":489,"num_write":184,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.808565+0000","last_change":"2024-03-09T14:28:40.391285+0000","last_active":"2024-03-09T15:33:55.808565+0000","last_peered":"2024-03-09T15:33:55.808565+0000","last_clean":"2024-03-09T15:33:55.808565+0000","last_became_active":"2024-03-09T14:28:40.389175+0000","last_became_peered":"2024-03-09T14:28:40.389175+0000","last_unstale":"2024-03-09T15:33:55.808565+0000","last_undegraded":"2024-03-09T15:33:55.808565+0000","last_fullsized":"2024-03-09T15:33:55.808565+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:33:43.955165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"261'58","reported_seq":553,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:02.730159+0000","last_change":"2024-03-09T14:28:42.401990+0000","last_active":"2024-03-09T15:34:02.730159+0000","last_peered":"2024-03-09T15:34:02.730159+0000","last_clean":"2024-03-09T15:34:02.730159+0000","last_became_active":"2024-03-09T14:28:42.401214+0000","last_became_peered":"2024-03-09T14:28:42.401214+0000","last_unstale":"2024-03-09T15:34:02.730159+0000","last_undegraded":"2024-03-09T15:34:02.730159+0000","last_fullsized":"2024-03-09T15:34:02.730159+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:24:20.023880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":47,"num_write":140,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"261'78","reported_seq":634,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.793908+0000","last_change":"2024-03-09T14:28:34.317931+0000","last_active":"2024-03-09T15:33:55.793908+0000","last_peered":"2024-03-09T15:33:55.793908+0000","last_clean":"2024-03-09T15:33:55.793908+0000","last_became_active":"2024-03-09T14:28:34.316812+0000","last_became_peered":"2024-03-09T14:28:34.316812+0000","last_unstale":"2024-03-09T15:33:55.793908+0000","last_undegraded":"2024-03-09T15:33:55.793908+0000","last_fullsized":"2024-03-09T15:33:55.793908+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:23:12.743671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":320,"num_read_kb":256,"num_write":116,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"261'100","reported_seq":663,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.219206+0000","last_change":"2024-03-09T14:29:06.446958+0000","last_active":"2024-03-09T15:36:06.219206+0000","last_peered":"2024-03-09T15:36:06.219206+0000","last_clean":"2024-03-09T15:36:06.219206+0000","last_became_active":"2024-03-09T14:29:06.446333+0000","last_became_peered":"2024-03-09T14:29:06.446333+0000","last_unstale":"2024-03-09T15:36:06.219206+0000","last_undegraded":"2024-03-09T15:36:06.219206+0000","last_fullsized":"2024-03-09T15:36:06.219206+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:09:44.978410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":139,"num_read_kb":1025,"num_write":394,"num_write_kb":11167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.980194+0000","last_change":"2024-03-09T14:28:36.304298+0000","last_active":"2024-03-09T15:33:50.980194+0000","last_peered":"2024-03-09T15:33:50.980194+0000","last_clean":"2024-03-09T15:33:50.980194+0000","last_became_active":"2024-03-09T14:28:36.303695+0000","last_became_peered":"2024-03-09T14:28:36.303695+0000","last_unstale":"2024-03-09T15:33:50.980194+0000","last_undegraded":"2024-03-09T15:33:50.980194+0000","last_fullsized":"2024-03-09T15:33:50.980194+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:16:14.481520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"261'217","reported_seq":821,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.705946+0000","last_change":"2024-03-09T14:28:38.340119+0000","last_active":"2024-03-09T15:35:58.705946+0000","last_peered":"2024-03-09T15:35:58.705946+0000","last_clean":"2024-03-09T15:35:58.705946+0000","last_became_active":"2024-03-09T14:28:38.339300+0000","last_became_peered":"2024-03-09T14:28:38.339300+0000","last_unstale":"2024-03-09T15:35:58.705946+0000","last_undegraded":"2024-03-09T15:35:58.705946+0000","last_fullsized":"2024-03-09T15:35:58.705946+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":200,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:56:14.245930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24501,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":344,"num_read_kb":579,"num_write":240,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"261'500","reported_seq":2509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.736377+0000","last_change":"2024-03-09T14:28:40.368533+0000","last_active":"2024-03-09T15:36:05.736377+0000","last_peered":"2024-03-09T15:36:05.736377+0000","last_clean":"2024-03-09T15:36:05.736377+0000","last_became_active":"2024-03-09T14:28:40.367119+0000","last_became_peered":"2024-03-09T14:28:40.367119+0000","last_unstale":"2024-03-09T15:36:05.736377+0000","last_undegraded":"2024-03-09T15:36:05.736377+0000","last_fullsized":"2024-03-09T15:36:05.736377+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":400,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:01.659994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"261'81","reported_seq":575,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:31.861227+0000","last_change":"2024-03-09T14:28:42.397381+0000","last_active":"2024-03-09T15:34:31.861227+0000","last_peered":"2024-03-09T15:34:31.861227+0000","last_clean":"2024-03-09T15:34:31.861227+0000","last_became_active":"2024-03-09T14:28:42.396389+0000","last_became_peered":"2024-03-09T14:28:42.396389+0000","last_unstale":"2024-03-09T15:34:31.861227+0000","last_undegraded":"2024-03-09T15:34:31.861227+0000","last_fullsized":"2024-03-09T15:34:31.861227+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:00:18.807761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":65,"num_write":200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"261'44","reported_seq":582,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.016397+0000","last_change":"2024-03-09T14:28:34.298294+0000","last_active":"2024-03-09T15:33:51.016397+0000","last_peered":"2024-03-09T15:33:51.016397+0000","last_clean":"2024-03-09T15:33:51.016397+0000","last_became_active":"2024-03-09T14:28:34.297080+0000","last_became_peered":"2024-03-09T14:28:34.297080+0000","last_unstale":"2024-03-09T15:33:51.016397+0000","last_undegraded":"2024-03-09T15:33:51.016397+0000","last_fullsized":"2024-03-09T15:33:51.016397+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:32.179794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":189,"num_read_kb":155,"num_write":63,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"261'128","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.989852+0000","last_change":"2024-03-09T14:29:06.445970+0000","last_active":"2024-03-09T15:36:05.989852+0000","last_peered":"2024-03-09T15:36:05.989852+0000","last_clean":"2024-03-09T15:36:05.989852+0000","last_became_active":"2024-03-09T14:29:06.445573+0000","last_became_peered":"2024-03-09T14:29:06.445573+0000","last_unstale":"2024-03-09T15:36:05.989852+0000","last_undegraded":"2024-03-09T15:36:05.989852+0000","last_fullsized":"2024-03-09T15:36:05.989852+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:30:46.074402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":185,"num_read_kb":157,"num_write":439,"num_write_kb":32761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.980070+0000","last_change":"2024-03-09T14:28:36.323155+0000","last_active":"2024-03-09T15:33:50.980070+0000","last_peered":"2024-03-09T15:33:50.980070+0000","last_clean":"2024-03-09T15:33:50.980070+0000","last_became_active":"2024-03-09T14:28:36.322638+0000","last_became_peered":"2024-03-09T14:28:36.322638+0000","last_unstale":"2024-03-09T15:33:50.980070+0000","last_undegraded":"2024-03-09T15:33:50.980070+0000","last_fullsized":"2024-03-09T15:33:50.980070+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:45:20.380783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"261'312","reported_seq":947,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.765276+0000","last_change":"2024-03-09T14:28:38.329540+0000","last_active":"2024-03-09T15:36:05.765276+0000","last_peered":"2024-03-09T15:36:05.765276+0000","last_clean":"2024-03-09T15:36:05.765276+0000","last_became_active":"2024-03-09T14:28:38.328897+0000","last_became_peered":"2024-03-09T14:28:38.328897+0000","last_unstale":"2024-03-09T15:36:05.765276+0000","last_undegraded":"2024-03-09T15:36:05.765276+0000","last_fullsized":"2024-03-09T15:36:05.765276+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":300,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:11:28.889923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60622,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":525,"num_read_kb":1064,"num_write":380,"num_write_kb":275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.977668+0000","last_change":"2024-03-09T14:28:40.384571+0000","last_active":"2024-03-09T15:33:40.977668+0000","last_peered":"2024-03-09T15:33:40.977668+0000","last_clean":"2024-03-09T15:33:40.977668+0000","last_became_active":"2024-03-09T14:28:40.383758+0000","last_became_peered":"2024-03-09T14:28:40.383758+0000","last_unstale":"2024-03-09T15:33:40.977668+0000","last_undegraded":"2024-03-09T15:33:40.977668+0000","last_fullsized":"2024-03-09T15:33:40.977668+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:24:41.089698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"261'54","reported_seq":565,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.965658+0000","last_change":"2024-03-09T14:28:42.402684+0000","last_active":"2024-03-09T15:33:45.965658+0000","last_peered":"2024-03-09T15:33:45.965658+0000","last_clean":"2024-03-09T15:33:45.965658+0000","last_became_active":"2024-03-09T14:28:42.401880+0000","last_became_peered":"2024-03-09T14:28:42.401880+0000","last_unstale":"2024-03-09T15:33:45.965658+0000","last_undegraded":"2024-03-09T15:33:45.965658+0000","last_fullsized":"2024-03-09T15:33:45.965658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:29:59.644550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":69,"num_write":120,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"261'79","reported_seq":639,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195621+0000","last_change":"2024-03-09T14:28:34.316156+0000","last_active":"2024-03-09T15:33:51.195621+0000","last_peered":"2024-03-09T15:33:51.195621+0000","last_clean":"2024-03-09T15:33:51.195621+0000","last_became_active":"2024-03-09T14:28:34.315432+0000","last_became_peered":"2024-03-09T14:28:34.315432+0000","last_unstale":"2024-03-09T15:33:51.195621+0000","last_undegraded":"2024-03-09T15:33:51.195621+0000","last_fullsized":"2024-03-09T15:33:51.195621+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:08:27.256304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":315,"num_read_kb":251,"num_write":126,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"261'126","reported_seq":664,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.807364+0000","last_change":"2024-03-09T14:28:32.300326+0000","last_active":"2024-03-09T15:33:50.807364+0000","last_peered":"2024-03-09T15:33:50.807364+0000","last_clean":"2024-03-09T15:33:50.807364+0000","last_became_active":"2024-03-09T14:28:32.299897+0000","last_became_peered":"2024-03-09T14:28:32.299897+0000","last_unstale":"2024-03-09T15:33:50.807364+0000","last_undegraded":"2024-03-09T15:33:50.807364+0000","last_fullsized":"2024-03-09T15:33:50.807364+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":100,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:25:33.818560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5546754,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":129,"num_read_kb":225,"num_write":522,"num_write_kb":26448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.980312+0000","last_change":"2024-03-09T14:28:36.320815+0000","last_active":"2024-03-09T15:33:50.980312+0000","last_peered":"2024-03-09T15:33:50.980312+0000","last_clean":"2024-03-09T15:33:50.980312+0000","last_became_active":"2024-03-09T14:28:36.320029+0000","last_became_peered":"2024-03-09T14:28:36.320029+0000","last_unstale":"2024-03-09T15:33:50.980312+0000","last_undegraded":"2024-03-09T15:33:50.980312+0000","last_fullsized":"2024-03-09T15:33:50.980312+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:23:44.855859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"261'205","reported_seq":792,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.664412+0000","last_change":"2024-03-09T14:28:38.339497+0000","last_active":"2024-03-09T15:35:58.664412+0000","last_peered":"2024-03-09T15:35:58.664412+0000","last_clean":"2024-03-09T15:35:58.664412+0000","last_became_active":"2024-03-09T14:28:38.338658+0000","last_became_peered":"2024-03-09T14:28:38.338658+0000","last_unstale":"2024-03-09T15:35:58.664412+0000","last_undegraded":"2024-03-09T15:35:58.664412+0000","last_fullsized":"2024-03-09T15:35:58.664412+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":100,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:42:50.615803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25762,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":343,"num_read_kb":574,"num_write":288,"num_write_kb":187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:55.989515+0000","last_change":"2024-03-09T14:28:40.379699+0000","last_active":"2024-03-09T15:31:55.989515+0000","last_peered":"2024-03-09T15:31:55.989515+0000","last_clean":"2024-03-09T15:31:55.989515+0000","last_became_active":"2024-03-09T14:28:40.378973+0000","last_became_peered":"2024-03-09T14:28:40.378973+0000","last_unstale":"2024-03-09T15:31:55.989515+0000","last_undegraded":"2024-03-09T15:31:55.989515+0000","last_fullsized":"2024-03-09T15:31:55.989515+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:53.400714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"261'42","reported_seq":535,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:00.822318+0000","last_change":"2024-03-09T14:28:42.398431+0000","last_active":"2024-03-09T15:35:00.822318+0000","last_peered":"2024-03-09T15:35:00.822318+0000","last_clean":"2024-03-09T15:35:00.822318+0000","last_became_active":"2024-03-09T14:28:42.397533+0000","last_became_peered":"2024-03-09T14:28:42.397533+0000","last_unstale":"2024-03-09T15:35:00.822318+0000","last_undegraded":"2024-03-09T15:35:00.822318+0000","last_fullsized":"2024-03-09T15:35:00.822318+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:54:11.336171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":91,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"261'56","reported_seq":616,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195764+0000","last_change":"2024-03-09T14:28:34.315224+0000","last_active":"2024-03-09T15:33:51.195764+0000","last_peered":"2024-03-09T15:33:51.195764+0000","last_clean":"2024-03-09T15:33:51.195764+0000","last_became_active":"2024-03-09T14:28:34.313475+0000","last_became_peered":"2024-03-09T14:28:34.313475+0000","last_unstale":"2024-03-09T15:33:51.195764+0000","last_undegraded":"2024-03-09T15:33:51.195764+0000","last_fullsized":"2024-03-09T15:33:51.195764+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:37:57.784425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":198,"num_write":78,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"261'105","reported_seq":657,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:40.924777+0000","last_change":"2024-03-09T14:28:32.319640+0000","last_active":"2024-03-09T15:35:40.924777+0000","last_peered":"2024-03-09T15:35:40.924777+0000","last_clean":"2024-03-09T15:35:40.924777+0000","last_became_active":"2024-03-09T14:28:32.316406+0000","last_became_peered":"2024-03-09T14:28:32.316406+0000","last_unstale":"2024-03-09T15:35:40.924777+0000","last_undegraded":"2024-03-09T15:35:40.924777+0000","last_fullsized":"2024-03-09T15:35:40.924777+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:33:19.144976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7947601,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":138,"num_read_kb":15736,"num_write":442,"num_write_kb":13578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.966668+0000","last_change":"2024-03-09T14:28:36.319712+0000","last_active":"2024-03-09T15:33:50.966668+0000","last_peered":"2024-03-09T15:33:50.966668+0000","last_clean":"2024-03-09T15:33:50.966668+0000","last_became_active":"2024-03-09T14:28:36.319210+0000","last_became_peered":"2024-03-09T14:28:36.319210+0000","last_unstale":"2024-03-09T15:33:50.966668+0000","last_undegraded":"2024-03-09T15:33:50.966668+0000","last_fullsized":"2024-03-09T15:33:50.966668+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:26:56.771418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"261'217","reported_seq":821,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.969548+0000","last_change":"2024-03-09T14:28:38.351282+0000","last_active":"2024-03-09T15:36:05.969548+0000","last_peered":"2024-03-09T15:36:05.969548+0000","last_clean":"2024-03-09T15:36:05.969548+0000","last_became_active":"2024-03-09T14:28:38.350390+0000","last_became_peered":"2024-03-09T14:28:38.350390+0000","last_unstale":"2024-03-09T15:36:05.969548+0000","last_undegraded":"2024-03-09T15:36:05.969548+0000","last_fullsized":"2024-03-09T15:36:05.969548+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":200,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:30:58.004755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9540,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":338,"num_read_kb":580,"num_write":230,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"261'473","reported_seq":2357,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.722396+0000","last_change":"2024-03-09T14:28:40.374487+0000","last_active":"2024-03-09T15:36:05.722396+0000","last_peered":"2024-03-09T15:36:05.722396+0000","last_clean":"2024-03-09T15:36:05.722396+0000","last_became_active":"2024-03-09T14:28:40.373360+0000","last_became_peered":"2024-03-09T14:28:40.373360+0000","last_unstale":"2024-03-09T15:36:05.722396+0000","last_undegraded":"2024-03-09T15:36:05.722396+0000","last_fullsized":"2024-03-09T15:36:05.722396+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":400,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:24:48.737477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"261'104","reported_seq":650,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:52.932955+0000","last_change":"2024-03-09T14:28:42.409080+0000","last_active":"2024-03-09T15:34:52.932955+0000","last_peered":"2024-03-09T15:34:52.932955+0000","last_clean":"2024-03-09T15:34:52.932955+0000","last_became_active":"2024-03-09T14:28:42.408233+0000","last_became_peered":"2024-03-09T14:28:42.408233+0000","last_unstale":"2024-03-09T15:34:52.932955+0000","last_undegraded":"2024-03-09T15:34:52.932955+0000","last_fullsized":"2024-03-09T15:34:52.932955+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:59:27.171699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":274,"num_read_kb":227,"num_write":264,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"261'72","reported_seq":621,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195874+0000","last_change":"2024-03-09T14:28:34.310932+0000","last_active":"2024-03-09T15:33:51.195874+0000","last_peered":"2024-03-09T15:33:51.195874+0000","last_clean":"2024-03-09T15:33:51.195874+0000","last_became_active":"2024-03-09T14:28:34.310061+0000","last_became_peered":"2024-03-09T14:28:34.310061+0000","last_unstale":"2024-03-09T15:33:51.195874+0000","last_undegraded":"2024-03-09T15:33:51.195874+0000","last_fullsized":"2024-03-09T15:33:51.195874+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:56:23.692037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":294,"num_read_kb":233,"num_write":114,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"261'141","reported_seq":748,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.928584+0000","last_change":"2024-03-09T14:28:32.313964+0000","last_active":"2024-03-09T15:36:05.928584+0000","last_peered":"2024-03-09T15:36:05.928584+0000","last_clean":"2024-03-09T15:36:05.928584+0000","last_became_active":"2024-03-09T14:28:32.313566+0000","last_became_peered":"2024-03-09T14:28:32.313566+0000","last_unstale":"2024-03-09T15:36:05.928584+0000","last_undegraded":"2024-03-09T15:36:05.928584+0000","last_fullsized":"2024-03-09T15:36:05.928584+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":100,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:06:35.181059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":569877,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":220,"num_read_kb":84880,"num_write":542,"num_write_kb":19718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"32'7","reported_seq":1459,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.269380+0000","last_change":"2024-03-09T14:28:36.327203+0000","last_active":"2024-03-09T15:36:05.269380+0000","last_peered":"2024-03-09T15:36:05.269380+0000","last_clean":"2024-03-09T15:36:05.269380+0000","last_became_active":"2024-03-09T14:28:36.325820+0000","last_became_peered":"2024-03-09T14:28:36.325820+0000","last_unstale":"2024-03-09T15:36:05.269380+0000","last_undegraded":"2024-03-09T15:36:05.269380+0000","last_fullsized":"2024-03-09T15:36:05.269380+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:03:03.441773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1195,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1118,"num_read_kb":1421,"num_write":14,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"261'86","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:52.540471+0000","last_change":"2024-03-09T14:28:38.596269+0000","last_active":"2024-03-09T15:32:52.540471+0000","last_peered":"2024-03-09T15:32:52.540471+0000","last_clean":"2024-03-09T15:32:52.540471+0000","last_became_active":"2024-03-09T14:28:38.595478+0000","last_became_peered":"2024-03-09T14:28:38.595478+0000","last_unstale":"2024-03-09T15:32:52.540471+0000","last_undegraded":"2024-03-09T15:32:52.540471+0000","last_fullsized":"2024-03-09T15:32:52.540471+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:43:55.683017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":105,"num_read_kb":100,"num_write":70,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"261'488","reported_seq":2293,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.725131+0000","last_change":"2024-03-09T14:28:40.375535+0000","last_active":"2024-03-09T15:36:05.725131+0000","last_peered":"2024-03-09T15:36:05.725131+0000","last_clean":"2024-03-09T15:36:05.725131+0000","last_became_active":"2024-03-09T14:28:40.374408+0000","last_became_peered":"2024-03-09T14:28:40.374408+0000","last_unstale":"2024-03-09T15:36:05.725131+0000","last_undegraded":"2024-03-09T15:36:05.725131+0000","last_fullsized":"2024-03-09T15:36:05.725131+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":400,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:00:18.508302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"261'93","reported_seq":596,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:39.863116+0000","last_change":"2024-03-09T14:28:42.412396+0000","last_active":"2024-03-09T15:34:39.863116+0000","last_peered":"2024-03-09T15:34:39.863116+0000","last_clean":"2024-03-09T15:34:39.863116+0000","last_became_active":"2024-03-09T14:28:42.411523+0000","last_became_peered":"2024-03-09T14:28:42.411523+0000","last_unstale":"2024-03-09T15:34:39.863116+0000","last_undegraded":"2024-03-09T15:34:39.863116+0000","last_fullsized":"2024-03-09T15:34:39.863116+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:23:50.431623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":72,"num_write":221,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"261'42","reported_seq":595,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.134950+0000","last_change":"2024-03-09T14:28:34.317632+0000","last_active":"2024-03-09T15:31:41.134950+0000","last_peered":"2024-03-09T15:31:41.134950+0000","last_clean":"2024-03-09T15:31:41.134950+0000","last_became_active":"2024-03-09T14:28:34.317226+0000","last_became_peered":"2024-03-09T14:28:34.317226+0000","last_unstale":"2024-03-09T15:31:41.134950+0000","last_undegraded":"2024-03-09T15:31:41.134950+0000","last_fullsized":"2024-03-09T15:31:41.134950+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:48:25.758532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":188,"num_read_kb":158,"num_write":62,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"261'93","reported_seq":629,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.803584+0000","last_change":"2024-03-09T14:28:32.295693+0000","last_active":"2024-03-09T15:33:35.803584+0000","last_peered":"2024-03-09T15:33:35.803584+0000","last_clean":"2024-03-09T15:33:35.803584+0000","last_became_active":"2024-03-09T14:28:32.295179+0000","last_became_peered":"2024-03-09T14:28:32.295179+0000","last_unstale":"2024-03-09T15:33:35.803584+0000","last_undegraded":"2024-03-09T15:33:35.803584+0000","last_fullsized":"2024-03-09T15:33:35.803584+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:55:35.164395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262386,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":120,"num_read_kb":124,"num_write":367,"num_write_kb":5920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195837+0000","last_change":"2024-03-09T14:28:36.319885+0000","last_active":"2024-03-09T15:33:51.195837+0000","last_peered":"2024-03-09T15:33:51.195837+0000","last_clean":"2024-03-09T15:33:51.195837+0000","last_became_active":"2024-03-09T14:28:36.319102+0000","last_became_peered":"2024-03-09T14:28:36.319102+0000","last_unstale":"2024-03-09T15:33:51.195837+0000","last_undegraded":"2024-03-09T15:33:51.195837+0000","last_fullsized":"2024-03-09T15:33:51.195837+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:44:50.268293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"261'311","reported_seq":934,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.971810+0000","last_change":"2024-03-09T14:28:38.338131+0000","last_active":"2024-03-09T15:35:58.971810+0000","last_peered":"2024-03-09T15:35:58.971810+0000","last_clean":"2024-03-09T15:35:58.971810+0000","last_became_active":"2024-03-09T14:28:38.337254+0000","last_became_peered":"2024-03-09T14:28:38.337254+0000","last_unstale":"2024-03-09T15:35:58.971810+0000","last_undegraded":"2024-03-09T15:35:58.971810+0000","last_fullsized":"2024-03-09T15:35:58.971810+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":300,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:49.247631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29175,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":664,"num_read_kb":922,"num_write":784,"num_write_kb":628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.766105+0000","last_change":"2024-03-09T14:28:40.378550+0000","last_active":"2024-03-09T15:31:40.766105+0000","last_peered":"2024-03-09T15:31:40.766105+0000","last_clean":"2024-03-09T15:31:40.766105+0000","last_became_active":"2024-03-09T14:28:40.377714+0000","last_became_peered":"2024-03-09T14:28:40.377714+0000","last_unstale":"2024-03-09T15:31:40.766105+0000","last_undegraded":"2024-03-09T15:31:40.766105+0000","last_fullsized":"2024-03-09T15:31:40.766105+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:21:50.389100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"261'68","reported_seq":567,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.959943+0000","last_change":"2024-03-09T14:28:42.402550+0000","last_active":"2024-03-09T15:33:20.959943+0000","last_peered":"2024-03-09T15:33:20.959943+0000","last_clean":"2024-03-09T15:33:20.959943+0000","last_became_active":"2024-03-09T14:28:42.401775+0000","last_became_peered":"2024-03-09T14:28:42.401775+0000","last_unstale":"2024-03-09T15:33:20.959943+0000","last_undegraded":"2024-03-09T15:33:20.959943+0000","last_fullsized":"2024-03-09T15:33:20.959943+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:47:20.741043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":77,"num_write":153,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"261'174","reported_seq":825,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:30.788820+0000","last_change":"2024-03-09T14:28:34.327727+0000","last_active":"2024-03-09T15:33:30.788820+0000","last_peered":"2024-03-09T15:33:30.788820+0000","last_clean":"2024-03-09T15:33:30.788820+0000","last_became_active":"2024-03-09T14:28:34.326919+0000","last_became_peered":"2024-03-09T14:28:34.326919+0000","last_unstale":"2024-03-09T15:33:30.788820+0000","last_undegraded":"2024-03-09T15:33:30.788820+0000","last_fullsized":"2024-03-09T15:33:30.788820+0000","mapping_epoch":22,"log_start":"34'100","ondisk_log_start":"34'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:50:06.490347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":914,"num_read_kb":819,"num_write":300,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":681,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.189283+0000","last_change":"2024-03-09T14:28:12.265518+0000","last_active":"2024-03-09T15:32:26.189283+0000","last_peered":"2024-03-09T15:32:26.189283+0000","last_clean":"2024-03-09T15:32:26.189283+0000","last_became_active":"2024-03-09T14:28:12.262430+0000","last_became_peered":"2024-03-09T14:28:12.262430+0000","last_unstale":"2024-03-09T15:32:26.189283+0000","last_undegraded":"2024-03-09T15:32:26.189283+0000","last_fullsized":"2024-03-09T15:32:26.189283+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_clean_scrub_stamp":"2024-03-09T14:28:11.179640+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:14:17.888044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.185851+0000","last_change":"2024-03-09T14:28:19.233336+0000","last_active":"2024-03-09T15:32:56.185851+0000","last_peered":"2024-03-09T15:32:56.185851+0000","last_clean":"2024-03-09T15:32:56.185851+0000","last_became_active":"2024-03-09T14:28:19.232517+0000","last_became_peered":"2024-03-09T14:28:19.232517+0000","last_unstale":"2024-03-09T15:32:56.185851+0000","last_undegraded":"2024-03-09T15:32:56.185851+0000","last_fullsized":"2024-03-09T15:32:56.185851+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:53:39.430154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"261'111","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:39.045175+0000","last_change":"2024-03-09T14:28:32.300845+0000","last_active":"2024-03-09T15:34:39.045175+0000","last_peered":"2024-03-09T15:34:39.045175+0000","last_clean":"2024-03-09T15:34:39.045175+0000","last_became_active":"2024-03-09T14:28:32.300468+0000","last_became_peered":"2024-03-09T14:28:32.300468+0000","last_unstale":"2024-03-09T15:34:39.045175+0000","last_undegraded":"2024-03-09T15:34:39.045175+0000","last_fullsized":"2024-03-09T15:34:39.045175+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:58:14.894309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3439332,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":154,"num_read_kb":6450,"num_write":438,"num_write_kb":21101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.199546+0000","last_change":"2024-03-09T14:28:36.318953+0000","last_active":"2024-03-09T15:33:11.199546+0000","last_peered":"2024-03-09T15:33:11.199546+0000","last_clean":"2024-03-09T15:33:11.199546+0000","last_became_active":"2024-03-09T14:28:36.318172+0000","last_became_peered":"2024-03-09T14:28:36.318172+0000","last_unstale":"2024-03-09T15:33:11.199546+0000","last_undegraded":"2024-03-09T15:33:11.199546+0000","last_fullsized":"2024-03-09T15:33:11.199546+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:14:08.117277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"261'493","reported_seq":1200,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.910451+0000","last_change":"2024-03-09T14:28:38.349794+0000","last_active":"2024-03-09T15:36:05.910451+0000","last_peered":"2024-03-09T15:36:05.910451+0000","last_clean":"2024-03-09T15:36:05.910451+0000","last_became_active":"2024-03-09T14:28:38.348941+0000","last_became_peered":"2024-03-09T14:28:38.348941+0000","last_unstale":"2024-03-09T15:36:05.910451+0000","last_undegraded":"2024-03-09T15:36:05.910451+0000","last_fullsized":"2024-03-09T15:36:05.910451+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":400,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:10:34.907672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74232,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":969,"num_read_kb":1810,"num_write":926,"num_write_kb":765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:41.014076+0000","last_change":"2024-03-09T14:28:40.364392+0000","last_active":"2024-03-09T15:33:41.014076+0000","last_peered":"2024-03-09T15:33:41.014076+0000","last_clean":"2024-03-09T15:33:41.014076+0000","last_became_active":"2024-03-09T14:28:40.362781+0000","last_became_peered":"2024-03-09T14:28:40.362781+0000","last_unstale":"2024-03-09T15:33:41.014076+0000","last_undegraded":"2024-03-09T15:33:41.014076+0000","last_fullsized":"2024-03-09T15:33:41.014076+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:30:02.543875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"261'118","reported_seq":817,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:45.170047+0000","last_change":"2024-03-09T14:28:42.399378+0000","last_active":"2024-03-09T15:34:45.170047+0000","last_peered":"2024-03-09T15:34:45.170047+0000","last_clean":"2024-03-09T15:34:45.170047+0000","last_became_active":"2024-03-09T14:28:42.398626+0000","last_became_peered":"2024-03-09T14:28:42.398626+0000","last_unstale":"2024-03-09T15:34:45.170047+0000","last_undegraded":"2024-03-09T15:34:45.170047+0000","last_fullsized":"2024-03-09T15:34:45.170047+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:40:25.365667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":356,"num_read_kb":343,"num_write":215,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"261'90","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.012674+0000","last_change":"2024-03-09T14:28:34.297905+0000","last_active":"2024-03-09T15:33:36.012674+0000","last_peered":"2024-03-09T15:33:36.012674+0000","last_clean":"2024-03-09T15:33:36.012674+0000","last_became_active":"2024-03-09T14:28:34.297472+0000","last_became_peered":"2024-03-09T14:28:34.297472+0000","last_unstale":"2024-03-09T15:33:36.012674+0000","last_undegraded":"2024-03-09T15:33:36.012674+0000","last_fullsized":"2024-03-09T15:33:36.012674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:04:18.955082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":356,"num_read_kb":292,"num_write":141,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":525,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:31.190342+0000","last_change":"2024-03-09T14:28:19.242176+0000","last_active":"2024-03-09T15:32:31.190342+0000","last_peered":"2024-03-09T15:32:31.190342+0000","last_clean":"2024-03-09T15:32:31.190342+0000","last_became_active":"2024-03-09T14:28:19.241110+0000","last_became_peered":"2024-03-09T14:28:19.241110+0000","last_unstale":"2024-03-09T15:32:31.190342+0000","last_undegraded":"2024-03-09T15:32:31.190342+0000","last_fullsized":"2024-03-09T15:32:31.190342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:28:52.238193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"261'122","reported_seq":694,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.955245+0000","last_change":"2024-03-09T14:28:32.312953+0000","last_active":"2024-03-09T15:36:05.955245+0000","last_peered":"2024-03-09T15:36:05.955245+0000","last_clean":"2024-03-09T15:36:05.955245+0000","last_became_active":"2024-03-09T14:28:32.312536+0000","last_became_peered":"2024-03-09T14:28:32.312536+0000","last_unstale":"2024-03-09T15:36:05.955245+0000","last_undegraded":"2024-03-09T15:36:05.955245+0000","last_fullsized":"2024-03-09T15:36:05.955245+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:53:44.400275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3383,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":203,"num_read_kb":12489,"num_write":509,"num_write_kb":17953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194706+0000","last_change":"2024-03-09T14:28:36.307721+0000","last_active":"2024-03-09T15:33:36.194706+0000","last_peered":"2024-03-09T15:33:36.194706+0000","last_clean":"2024-03-09T15:33:36.194706+0000","last_became_active":"2024-03-09T14:28:36.306912+0000","last_became_peered":"2024-03-09T14:28:36.306912+0000","last_unstale":"2024-03-09T15:33:36.194706+0000","last_undegraded":"2024-03-09T15:33:36.194706+0000","last_fullsized":"2024-03-09T15:33:36.194706+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:19:47.054193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"261'837","reported_seq":2502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.790375+0000","last_change":"2024-03-09T14:28:38.329490+0000","last_active":"2024-03-09T15:36:05.790375+0000","last_peered":"2024-03-09T15:36:05.790375+0000","last_clean":"2024-03-09T15:36:05.790375+0000","last_became_active":"2024-03-09T14:28:38.328738+0000","last_became_peered":"2024-03-09T14:28:38.328738+0000","last_unstale":"2024-03-09T15:36:05.790375+0000","last_undegraded":"2024-03-09T15:36:05.790375+0000","last_fullsized":"2024-03-09T15:36:05.790375+0000","mapping_epoch":26,"log_start":"261'800","ondisk_log_start":"261'800","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":800,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:24:55.016529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77478,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1296,"num_read_kb":2021,"num_write":650,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.202990+0000","last_change":"2024-03-09T14:28:40.387247+0000","last_active":"2024-03-09T15:33:26.202990+0000","last_peered":"2024-03-09T15:33:26.202990+0000","last_clean":"2024-03-09T15:33:26.202990+0000","last_became_active":"2024-03-09T14:28:40.386451+0000","last_became_peered":"2024-03-09T14:28:40.386451+0000","last_unstale":"2024-03-09T15:33:26.202990+0000","last_undegraded":"2024-03-09T15:33:26.202990+0000","last_fullsized":"2024-03-09T15:33:26.202990+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:06:23.810341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"261'124","reported_seq":656,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.967916+0000","last_change":"2024-03-09T14:28:42.403919+0000","last_active":"2024-03-09T15:33:55.967916+0000","last_peered":"2024-03-09T15:33:55.967916+0000","last_clean":"2024-03-09T15:33:55.967916+0000","last_became_active":"2024-03-09T14:28:42.403013+0000","last_became_peered":"2024-03-09T14:28:42.403013+0000","last_unstale":"2024-03-09T15:33:55.967916+0000","last_undegraded":"2024-03-09T15:33:55.967916+0000","last_fullsized":"2024-03-09T15:33:55.967916+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:36:18.083589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1248,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":261,"num_read_kb":241,"num_write":300,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"261'48","reported_seq":601,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963559+0000","last_change":"2024-03-09T14:28:34.297150+0000","last_active":"2024-03-09T15:33:35.963559+0000","last_peered":"2024-03-09T15:33:35.963559+0000","last_clean":"2024-03-09T15:33:35.963559+0000","last_became_active":"2024-03-09T14:28:34.296733+0000","last_became_peered":"2024-03-09T14:28:34.296733+0000","last_unstale":"2024-03-09T15:33:35.963559+0000","last_undegraded":"2024-03-09T15:33:35.963559+0000","last_fullsized":"2024-03-09T15:33:35.963559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:29:16.007058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":200,"num_read_kb":166,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.968712+0000","last_change":"2024-03-09T14:28:19.437446+0000","last_active":"2024-03-09T15:32:55.968712+0000","last_peered":"2024-03-09T15:32:55.968712+0000","last_clean":"2024-03-09T15:32:55.968712+0000","last_became_active":"2024-03-09T14:28:19.436985+0000","last_became_peered":"2024-03-09T14:28:19.436985+0000","last_unstale":"2024-03-09T15:32:55.968712+0000","last_undegraded":"2024-03-09T15:32:55.968712+0000","last_fullsized":"2024-03-09T15:32:55.968712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:46:38.528697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"261'134","reported_seq":791,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.893599+0000","last_change":"2024-03-09T14:28:32.303463+0000","last_active":"2024-03-09T15:36:05.893599+0000","last_peered":"2024-03-09T15:36:05.893599+0000","last_clean":"2024-03-09T15:36:05.893599+0000","last_became_active":"2024-03-09T14:28:32.303176+0000","last_became_peered":"2024-03-09T14:28:32.303176+0000","last_unstale":"2024-03-09T15:36:05.893599+0000","last_undegraded":"2024-03-09T15:36:05.893599+0000","last_fullsized":"2024-03-09T15:36:05.893599+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:28:51.916628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8554935,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":468,"num_read_kb":3616,"num_write":574,"num_write_kb":29269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"25'1","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.976814+0000","last_change":"2024-03-09T14:28:36.306679+0000","last_active":"2024-03-09T15:33:35.976814+0000","last_peered":"2024-03-09T15:33:35.976814+0000","last_clean":"2024-03-09T15:33:35.976814+0000","last_became_active":"2024-03-09T14:28:36.305971+0000","last_became_peered":"2024-03-09T14:28:36.305971+0000","last_unstale":"2024-03-09T15:33:35.976814+0000","last_undegraded":"2024-03-09T15:33:35.976814+0000","last_fullsized":"2024-03-09T15:33:35.976814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:21:21.395131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"261'230","reported_seq":1598,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.934522+0000","last_change":"2024-03-09T14:28:38.598201+0000","last_active":"2024-03-09T15:35:58.934522+0000","last_peered":"2024-03-09T15:35:58.934522+0000","last_clean":"2024-03-09T15:35:58.934522+0000","last_became_active":"2024-03-09T14:28:38.597403+0000","last_became_peered":"2024-03-09T14:28:38.597403+0000","last_unstale":"2024-03-09T15:35:58.934522+0000","last_undegraded":"2024-03-09T15:35:58.934522+0000","last_fullsized":"2024-03-09T15:35:58.934522+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":200,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:44:05.046714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":226,"num_read_kb":216,"num_write":195,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.007838+0000","last_change":"2024-03-09T14:28:40.372884+0000","last_active":"2024-03-09T15:33:16.007838+0000","last_peered":"2024-03-09T15:33:16.007838+0000","last_clean":"2024-03-09T15:33:16.007838+0000","last_became_active":"2024-03-09T14:28:40.371383+0000","last_became_peered":"2024-03-09T14:28:40.371383+0000","last_unstale":"2024-03-09T15:33:16.007838+0000","last_undegraded":"2024-03-09T15:33:16.007838+0000","last_fullsized":"2024-03-09T15:33:16.007838+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:01:15.189556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"261'201","reported_seq":791,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:19.528609+0000","last_change":"2024-03-09T14:28:42.396620+0000","last_active":"2024-03-09T15:34:19.528609+0000","last_peered":"2024-03-09T15:34:19.528609+0000","last_clean":"2024-03-09T15:34:19.528609+0000","last_became_active":"2024-03-09T14:28:42.395781+0000","last_became_peered":"2024-03-09T14:28:42.395781+0000","last_unstale":"2024-03-09T15:34:19.528609+0000","last_undegraded":"2024-03-09T15:34:19.528609+0000","last_fullsized":"2024-03-09T15:34:19.528609+0000","mapping_epoch":30,"log_start":"247'100","ondisk_log_start":"247'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":100,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:38:14.404007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":435,"num_read_kb":456,"num_write":425,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"261'74","reported_seq":618,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:31.876523+0000","last_change":"2024-03-09T14:28:34.313660+0000","last_active":"2024-03-09T15:34:31.876523+0000","last_peered":"2024-03-09T15:34:31.876523+0000","last_clean":"2024-03-09T15:34:31.876523+0000","last_became_active":"2024-03-09T14:28:34.313302+0000","last_became_peered":"2024-03-09T14:28:34.313302+0000","last_unstale":"2024-03-09T15:34:31.876523+0000","last_undegraded":"2024-03-09T15:34:31.876523+0000","last_fullsized":"2024-03-09T15:34:31.876523+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:32:32.206071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":225,"num_write":104,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":522,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.185786+0000","last_change":"2024-03-09T14:28:19.238720+0000","last_active":"2024-03-09T15:32:56.185786+0000","last_peered":"2024-03-09T15:32:56.185786+0000","last_clean":"2024-03-09T15:32:56.185786+0000","last_became_active":"2024-03-09T14:28:19.237937+0000","last_became_peered":"2024-03-09T14:28:19.237937+0000","last_unstale":"2024-03-09T15:32:56.185786+0000","last_undegraded":"2024-03-09T15:32:56.185786+0000","last_fullsized":"2024-03-09T15:32:56.185786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:29:01.306246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"261'107","reported_seq":682,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.961490+0000","last_change":"2024-03-09T14:28:32.312937+0000","last_active":"2024-03-09T15:36:05.961490+0000","last_peered":"2024-03-09T15:36:05.961490+0000","last_clean":"2024-03-09T15:36:05.961490+0000","last_became_active":"2024-03-09T14:28:32.303963+0000","last_became_peered":"2024-03-09T14:28:32.303963+0000","last_unstale":"2024-03-09T15:36:05.961490+0000","last_undegraded":"2024-03-09T15:36:05.961490+0000","last_fullsized":"2024-03-09T15:36:05.961490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:54:34.862486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6966901,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":155,"num_read_kb":15115,"num_write":410,"num_write_kb":32085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.010297+0000","last_change":"2024-03-09T14:28:36.318128+0000","last_active":"2024-03-09T15:33:26.010297+0000","last_peered":"2024-03-09T15:33:26.010297+0000","last_clean":"2024-03-09T15:33:26.010297+0000","last_became_active":"2024-03-09T14:28:36.317586+0000","last_became_peered":"2024-03-09T14:28:36.317586+0000","last_unstale":"2024-03-09T15:33:26.010297+0000","last_undegraded":"2024-03-09T15:33:26.010297+0000","last_fullsized":"2024-03-09T15:33:26.010297+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:46:55.791256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"261'129","reported_seq":682,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:52.970721+0000","last_change":"2024-03-09T14:28:38.335628+0000","last_active":"2024-03-09T15:34:52.970721+0000","last_peered":"2024-03-09T15:34:52.970721+0000","last_clean":"2024-03-09T15:34:52.970721+0000","last_became_active":"2024-03-09T14:28:38.334463+0000","last_became_peered":"2024-03-09T14:28:38.334463+0000","last_unstale":"2024-03-09T15:34:52.970721+0000","last_undegraded":"2024-03-09T15:34:52.970721+0000","last_fullsized":"2024-03-09T15:34:52.970721+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:56:33.829971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":178,"num_read_kb":172,"num_write":132,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.790948+0000","last_change":"2024-03-09T14:28:40.371173+0000","last_active":"2024-03-09T15:33:40.790948+0000","last_peered":"2024-03-09T15:33:40.790948+0000","last_clean":"2024-03-09T15:33:40.790948+0000","last_became_active":"2024-03-09T14:28:40.370436+0000","last_became_peered":"2024-03-09T14:28:40.370436+0000","last_unstale":"2024-03-09T15:33:40.790948+0000","last_undegraded":"2024-03-09T15:33:40.790948+0000","last_fullsized":"2024-03-09T15:33:40.790948+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:23:11.745507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"261'47","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:48.549510+0000","last_change":"2024-03-09T14:28:42.398975+0000","last_active":"2024-03-09T15:33:48.549510+0000","last_peered":"2024-03-09T15:33:48.549510+0000","last_clean":"2024-03-09T15:33:48.549510+0000","last_became_active":"2024-03-09T14:28:42.397058+0000","last_became_peered":"2024-03-09T14:28:42.397058+0000","last_unstale":"2024-03-09T15:33:48.549510+0000","last_undegraded":"2024-03-09T15:33:48.549510+0000","last_fullsized":"2024-03-09T15:33:48.549510+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:02:27.312655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":100,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"261'43","reported_seq":593,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.957260+0000","last_change":"2024-03-09T14:28:34.297850+0000","last_active":"2024-03-09T15:33:10.957260+0000","last_peered":"2024-03-09T15:33:10.957260+0000","last_clean":"2024-03-09T15:33:10.957260+0000","last_became_active":"2024-03-09T14:28:34.296953+0000","last_became_peered":"2024-03-09T14:28:34.296953+0000","last_unstale":"2024-03-09T15:33:10.957260+0000","last_undegraded":"2024-03-09T15:33:10.957260+0000","last_fullsized":"2024-03-09T15:33:10.957260+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:37:50.390768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":162,"num_write":75,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"261'151","reported_seq":785,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.160202+0000","last_change":"2024-03-09T14:28:32.303697+0000","last_active":"2024-03-09T15:36:06.160202+0000","last_peered":"2024-03-09T15:36:06.160202+0000","last_clean":"2024-03-09T15:36:06.160202+0000","last_became_active":"2024-03-09T14:28:32.302970+0000","last_became_peered":"2024-03-09T14:28:32.302970+0000","last_unstale":"2024-03-09T15:36:06.160202+0000","last_undegraded":"2024-03-09T15:36:06.160202+0000","last_fullsized":"2024-03-09T15:36:06.160202+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":100,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:33:55.197166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655969,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":294,"num_read_kb":94521,"num_write":620,"num_write_kb":45805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":520,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.953596+0000","last_change":"2024-03-09T14:28:19.236670+0000","last_active":"2024-03-09T15:32:55.953596+0000","last_peered":"2024-03-09T15:32:55.953596+0000","last_clean":"2024-03-09T15:32:55.953596+0000","last_became_active":"2024-03-09T14:28:19.236240+0000","last_became_peered":"2024-03-09T14:28:19.236240+0000","last_unstale":"2024-03-09T15:32:55.953596+0000","last_undegraded":"2024-03-09T15:32:55.953596+0000","last_fullsized":"2024-03-09T15:32:55.953596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:00:09.869968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.006552+0000","last_change":"2024-03-09T14:28:36.315028+0000","last_active":"2024-03-09T15:33:11.006552+0000","last_peered":"2024-03-09T15:33:11.006552+0000","last_clean":"2024-03-09T15:33:11.006552+0000","last_became_active":"2024-03-09T14:28:36.314526+0000","last_became_peered":"2024-03-09T14:28:36.314526+0000","last_unstale":"2024-03-09T15:33:11.006552+0000","last_undegraded":"2024-03-09T15:33:11.006552+0000","last_fullsized":"2024-03-09T15:33:11.006552+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:24:49.871997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"261'381","reported_seq":1059,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.947780+0000","last_change":"2024-03-09T14:28:38.335922+0000","last_active":"2024-03-09T15:36:05.947780+0000","last_peered":"2024-03-09T15:36:05.947780+0000","last_clean":"2024-03-09T15:36:05.947780+0000","last_became_active":"2024-03-09T14:28:38.334976+0000","last_became_peered":"2024-03-09T14:28:38.334976+0000","last_unstale":"2024-03-09T15:36:05.947780+0000","last_undegraded":"2024-03-09T15:36:05.947780+0000","last_fullsized":"2024-03-09T15:36:05.947780+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":300,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:27:57.990498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34012,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":631,"num_read_kb":1165,"num_write":452,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.186765+0000","last_change":"2024-03-09T14:28:40.379714+0000","last_active":"2024-03-09T15:33:16.186765+0000","last_peered":"2024-03-09T15:33:16.186765+0000","last_clean":"2024-03-09T15:33:16.186765+0000","last_became_active":"2024-03-09T14:28:40.379108+0000","last_became_peered":"2024-03-09T14:28:40.379108+0000","last_unstale":"2024-03-09T15:33:16.186765+0000","last_undegraded":"2024-03-09T15:33:16.186765+0000","last_fullsized":"2024-03-09T15:33:16.186765+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:39:21.726823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"261'101","reported_seq":626,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:41.594325+0000","last_change":"2024-03-09T14:28:42.406107+0000","last_active":"2024-03-09T15:34:41.594325+0000","last_peered":"2024-03-09T15:34:41.594325+0000","last_clean":"2024-03-09T15:34:41.594325+0000","last_became_active":"2024-03-09T14:28:42.405277+0000","last_became_peered":"2024-03-09T14:28:42.405277+0000","last_unstale":"2024-03-09T15:34:41.594325+0000","last_undegraded":"2024-03-09T15:34:41.594325+0000","last_fullsized":"2024-03-09T15:34:41.594325+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:53:29.449584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":717,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185,"num_read_kb":160,"num_write":261,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"261'43","reported_seq":575,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.006582+0000","last_change":"2024-03-09T14:28:34.298018+0000","last_active":"2024-03-09T15:33:11.006582+0000","last_peered":"2024-03-09T15:33:11.006582+0000","last_clean":"2024-03-09T15:33:11.006582+0000","last_became_active":"2024-03-09T14:28:34.296660+0000","last_became_peered":"2024-03-09T14:28:34.296660+0000","last_unstale":"2024-03-09T15:33:11.006582+0000","last_undegraded":"2024-03-09T15:33:11.006582+0000","last_fullsized":"2024-03-09T15:33:11.006582+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:37:00.270467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":129,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"261'128","reported_seq":758,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.920215+0000","last_change":"2024-03-09T14:28:32.305873+0000","last_active":"2024-03-09T15:36:05.920215+0000","last_peered":"2024-03-09T15:36:05.920215+0000","last_clean":"2024-03-09T15:36:05.920215+0000","last_became_active":"2024-03-09T14:28:32.305318+0000","last_became_peered":"2024-03-09T14:28:32.305318+0000","last_unstale":"2024-03-09T15:36:05.920215+0000","last_undegraded":"2024-03-09T15:36:05.920215+0000","last_fullsized":"2024-03-09T15:36:05.920215+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:30:36.881076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3908854,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":337,"num_read_kb":10578,"num_write":436,"num_write_kb":20505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:06.198444+0000","last_change":"2024-03-09T14:28:19.241475+0000","last_active":"2024-03-09T15:33:06.198444+0000","last_peered":"2024-03-09T15:33:06.198444+0000","last_clean":"2024-03-09T15:33:06.198444+0000","last_became_active":"2024-03-09T14:28:19.240704+0000","last_became_peered":"2024-03-09T14:28:19.240704+0000","last_unstale":"2024-03-09T15:33:06.198444+0000","last_undegraded":"2024-03-09T15:33:06.198444+0000","last_fullsized":"2024-03-09T15:33:06.198444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:07:08.993017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784406+0000","last_change":"2024-03-09T14:28:36.315481+0000","last_active":"2024-03-09T15:33:10.784406+0000","last_peered":"2024-03-09T15:33:10.784406+0000","last_clean":"2024-03-09T15:33:10.784406+0000","last_became_active":"2024-03-09T14:28:36.314676+0000","last_became_peered":"2024-03-09T14:28:36.314676+0000","last_unstale":"2024-03-09T15:33:10.784406+0000","last_undegraded":"2024-03-09T15:33:10.784406+0000","last_fullsized":"2024-03-09T15:33:10.784406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:14:11.938307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"261'197","reported_seq":778,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:20.985108+0000","last_change":"2024-03-09T14:28:38.595995+0000","last_active":"2024-03-09T15:35:20.985108+0000","last_peered":"2024-03-09T15:35:20.985108+0000","last_clean":"2024-03-09T15:35:20.985108+0000","last_became_active":"2024-03-09T14:28:38.595028+0000","last_became_peered":"2024-03-09T14:28:38.595028+0000","last_unstale":"2024-03-09T15:35:20.985108+0000","last_undegraded":"2024-03-09T15:35:20.985108+0000","last_fullsized":"2024-03-09T15:35:20.985108+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":100,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:24:00.362746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":269,"num_read_kb":262,"num_write":230,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"261'500","reported_seq":2296,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.716276+0000","last_change":"2024-03-09T14:28:40.370167+0000","last_active":"2024-03-09T15:36:05.716276+0000","last_peered":"2024-03-09T15:36:05.716276+0000","last_clean":"2024-03-09T15:36:05.716276+0000","last_became_active":"2024-03-09T14:28:40.369165+0000","last_became_peered":"2024-03-09T14:28:40.369165+0000","last_unstale":"2024-03-09T15:36:05.716276+0000","last_undegraded":"2024-03-09T15:36:05.716276+0000","last_fullsized":"2024-03-09T15:36:05.716276+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":400,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:29:22.485672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"261'76","reported_seq":564,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:15.812925+0000","last_change":"2024-03-09T14:28:42.393153+0000","last_active":"2024-03-09T15:34:15.812925+0000","last_peered":"2024-03-09T15:34:15.812925+0000","last_clean":"2024-03-09T15:34:15.812925+0000","last_became_active":"2024-03-09T14:28:42.392402+0000","last_became_peered":"2024-03-09T14:28:42.392402+0000","last_unstale":"2024-03-09T15:34:15.812925+0000","last_undegraded":"2024-03-09T15:34:15.812925+0000","last_fullsized":"2024-03-09T15:34:15.812925+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:57:45.818538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":57,"num_write":172,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"261'48","reported_seq":589,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.973653+0000","last_change":"2024-03-09T14:29:06.440753+0000","last_active":"2024-03-09T15:33:20.973653+0000","last_peered":"2024-03-09T15:33:20.973653+0000","last_clean":"2024-03-09T15:33:20.973653+0000","last_became_active":"2024-03-09T14:29:06.440556+0000","last_became_peered":"2024-03-09T14:29:06.440556+0000","last_unstale":"2024-03-09T15:33:20.973653+0000","last_undegraded":"2024-03-09T15:33:20.973653+0000","last_fullsized":"2024-03-09T15:33:20.973653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:43:05.084646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":216,"num_read_kb":182,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"261'120","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.803883+0000","last_change":"2024-03-09T14:28:32.319314+0000","last_active":"2024-03-09T15:33:35.803883+0000","last_peered":"2024-03-09T15:33:35.803883+0000","last_clean":"2024-03-09T15:33:35.803883+0000","last_became_active":"2024-03-09T14:28:32.318971+0000","last_became_peered":"2024-03-09T14:28:32.318971+0000","last_unstale":"2024-03-09T15:33:35.803883+0000","last_undegraded":"2024-03-09T15:33:35.803883+0000","last_fullsized":"2024-03-09T15:33:35.803883+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:01:00.707624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243236,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":126,"num_read_kb":8647,"num_write":451,"num_write_kb":27350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.960027+0000","last_change":"2024-03-09T14:28:19.238362+0000","last_active":"2024-03-09T15:33:20.960027+0000","last_peered":"2024-03-09T15:33:20.960027+0000","last_clean":"2024-03-09T15:33:20.960027+0000","last_became_active":"2024-03-09T14:28:19.237750+0000","last_became_peered":"2024-03-09T14:28:19.237750+0000","last_unstale":"2024-03-09T15:33:20.960027+0000","last_undegraded":"2024-03-09T15:33:20.960027+0000","last_fullsized":"2024-03-09T15:33:20.960027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:19.699145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.185710+0000","last_change":"2024-03-09T14:28:36.328518+0000","last_active":"2024-03-09T15:33:11.185710+0000","last_peered":"2024-03-09T15:33:11.185710+0000","last_clean":"2024-03-09T15:33:11.185710+0000","last_became_active":"2024-03-09T14:28:36.327601+0000","last_became_peered":"2024-03-09T14:28:36.327601+0000","last_unstale":"2024-03-09T15:33:11.185710+0000","last_undegraded":"2024-03-09T15:33:11.185710+0000","last_fullsized":"2024-03-09T15:33:11.185710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:41:26.807068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"261'106","reported_seq":661,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.186823+0000","last_change":"2024-03-09T14:28:38.348479+0000","last_active":"2024-03-09T15:33:16.186823+0000","last_peered":"2024-03-09T15:33:16.186823+0000","last_clean":"2024-03-09T15:33:16.186823+0000","last_became_active":"2024-03-09T14:28:38.347750+0000","last_became_peered":"2024-03-09T14:28:38.347750+0000","last_unstale":"2024-03-09T15:33:16.186823+0000","last_undegraded":"2024-03-09T15:33:16.186823+0000","last_fullsized":"2024-03-09T15:33:16.186823+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:26:08.370238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"261'513","reported_seq":2485,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.727541+0000","last_change":"2024-03-09T14:28:40.387580+0000","last_active":"2024-03-09T15:36:05.727541+0000","last_peered":"2024-03-09T15:36:05.727541+0000","last_clean":"2024-03-09T15:36:05.727541+0000","last_became_active":"2024-03-09T14:28:40.386859+0000","last_became_peered":"2024-03-09T14:28:40.386859+0000","last_unstale":"2024-03-09T15:36:05.727541+0000","last_undegraded":"2024-03-09T15:36:05.727541+0000","last_fullsized":"2024-03-09T15:36:05.727541+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":500,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:17:24.125699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"261'106","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.987014+0000","last_change":"2024-03-09T14:28:42.406871+0000","last_active":"2024-03-09T15:31:45.987014+0000","last_peered":"2024-03-09T15:31:45.987014+0000","last_clean":"2024-03-09T15:31:45.987014+0000","last_became_active":"2024-03-09T14:28:42.406139+0000","last_became_peered":"2024-03-09T14:28:42.406139+0000","last_unstale":"2024-03-09T15:31:45.987014+0000","last_undegraded":"2024-03-09T15:31:45.987014+0000","last_fullsized":"2024-03-09T15:31:45.987014+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:24:57.719382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":65,"num_write":235,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"261'216","reported_seq":787,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:48.579968+0000","last_change":"2024-03-09T14:28:34.301025+0000","last_active":"2024-03-09T15:33:48.579968+0000","last_peered":"2024-03-09T15:33:48.579968+0000","last_clean":"2024-03-09T15:33:48.579968+0000","last_became_active":"2024-03-09T14:28:34.299988+0000","last_became_peered":"2024-03-09T14:28:34.299988+0000","last_unstale":"2024-03-09T15:33:48.579968+0000","last_undegraded":"2024-03-09T15:33:48.579968+0000","last_fullsized":"2024-03-09T15:33:48.579968+0000","mapping_epoch":22,"log_start":"258'200","ondisk_log_start":"258'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":200,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:05:09.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":843,"num_read_kb":645,"num_write":377,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"261'132","reported_seq":685,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.960146+0000","last_change":"2024-03-09T14:28:32.299582+0000","last_active":"2024-03-09T15:36:05.960146+0000","last_peered":"2024-03-09T15:36:05.960146+0000","last_clean":"2024-03-09T15:36:05.960146+0000","last_became_active":"2024-03-09T14:28:32.298977+0000","last_became_peered":"2024-03-09T14:28:32.298977+0000","last_unstale":"2024-03-09T15:36:05.960146+0000","last_undegraded":"2024-03-09T15:36:05.960146+0000","last_fullsized":"2024-03-09T15:36:05.960146+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:34:44.279787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243131,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":155,"num_read_kb":5781,"num_write":492,"num_write_kb":37567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.786634+0000","last_change":"2024-03-09T14:28:19.242561+0000","last_active":"2024-03-09T15:33:20.786634+0000","last_peered":"2024-03-09T15:33:20.786634+0000","last_clean":"2024-03-09T15:33:20.786634+0000","last_became_active":"2024-03-09T14:28:19.241700+0000","last_became_peered":"2024-03-09T14:28:19.241700+0000","last_unstale":"2024-03-09T15:33:20.786634+0000","last_undegraded":"2024-03-09T15:33:20.786634+0000","last_fullsized":"2024-03-09T15:33:20.786634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:15.462353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"32'3","reported_seq":822,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.255551+0000","last_change":"2024-03-09T14:28:36.317874+0000","last_active":"2024-03-09T15:36:05.255551+0000","last_peered":"2024-03-09T15:36:05.255551+0000","last_clean":"2024-03-09T15:36:05.255551+0000","last_became_active":"2024-03-09T14:28:36.317327+0000","last_became_peered":"2024-03-09T14:28:36.317327+0000","last_unstale":"2024-03-09T15:36:05.255551+0000","last_undegraded":"2024-03-09T15:36:05.255551+0000","last_fullsized":"2024-03-09T15:36:05.255551+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:07.321787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":484,"num_read_kb":482,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"261'465","reported_seq":1200,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.224674+0000","last_change":"2024-03-09T14:28:38.603808+0000","last_active":"2024-03-09T15:36:06.224674+0000","last_peered":"2024-03-09T15:36:06.224674+0000","last_clean":"2024-03-09T15:36:06.224674+0000","last_became_active":"2024-03-09T14:28:38.602908+0000","last_became_peered":"2024-03-09T14:28:38.602908+0000","last_unstale":"2024-03-09T15:36:06.224674+0000","last_undegraded":"2024-03-09T15:36:06.224674+0000","last_fullsized":"2024-03-09T15:36:06.224674+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":400,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:20:14.798348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58890,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":785,"num_read_kb":1500,"num_write":543,"num_write_kb":342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:15.799092+0000","last_change":"2024-03-09T14:28:40.376657+0000","last_active":"2024-03-09T15:33:15.799092+0000","last_peered":"2024-03-09T15:33:15.799092+0000","last_clean":"2024-03-09T15:33:15.799092+0000","last_became_active":"2024-03-09T14:28:40.375884+0000","last_became_peered":"2024-03-09T14:28:40.375884+0000","last_unstale":"2024-03-09T15:33:15.799092+0000","last_undegraded":"2024-03-09T15:33:15.799092+0000","last_fullsized":"2024-03-09T15:33:15.799092+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:28:12.389403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"261'54","reported_seq":551,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:33.602971+0000","last_change":"2024-03-09T14:28:42.406761+0000","last_active":"2024-03-09T15:34:33.602971+0000","last_peered":"2024-03-09T15:34:33.602971+0000","last_clean":"2024-03-09T15:34:33.602971+0000","last_became_active":"2024-03-09T14:28:42.405998+0000","last_became_peered":"2024-03-09T14:28:42.405998+0000","last_unstale":"2024-03-09T15:34:33.602971+0000","last_undegraded":"2024-03-09T15:34:33.602971+0000","last_fullsized":"2024-03-09T15:34:33.602971+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:02:27.576333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":52,"num_write":123,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1f","version":"261'146","reported_seq":723,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:09.596752+0000","last_change":"2024-03-09T14:28:42.410137+0000","last_active":"2024-03-09T15:34:09.596752+0000","last_peered":"2024-03-09T15:34:09.596752+0000","last_clean":"2024-03-09T15:34:09.596752+0000","last_became_active":"2024-03-09T14:28:42.408774+0000","last_became_peered":"2024-03-09T14:28:42.408774+0000","last_unstale":"2024-03-09T15:34:09.596752+0000","last_undegraded":"2024-03-09T15:34:09.596752+0000","last_fullsized":"2024-03-09T15:34:09.596752+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:50:49.883718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":256,"num_read_kb":233,"num_write":254,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"261'183","reported_seq":743,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:44.582989+0000","last_change":"2024-03-09T14:28:34.315742+0000","last_active":"2024-03-09T15:33:44.582989+0000","last_peered":"2024-03-09T15:33:44.582989+0000","last_clean":"2024-03-09T15:33:44.582989+0000","last_became_active":"2024-03-09T14:28:34.314893+0000","last_became_peered":"2024-03-09T14:28:34.314893+0000","last_unstale":"2024-03-09T15:33:44.582989+0000","last_undegraded":"2024-03-09T15:33:44.582989+0000","last_fullsized":"2024-03-09T15:33:44.582989+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":100,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:23:46.738791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":706,"num_read_kb":592,"num_write":291,"num_write_kb":155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"261'108","reported_seq":652,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.995245+0000","last_change":"2024-03-09T14:28:32.322792+0000","last_active":"2024-03-09T15:36:05.995245+0000","last_peered":"2024-03-09T15:36:05.995245+0000","last_clean":"2024-03-09T15:36:05.995245+0000","last_became_active":"2024-03-09T14:28:32.322364+0000","last_became_peered":"2024-03-09T14:28:32.322364+0000","last_unstale":"2024-03-09T15:36:05.995245+0000","last_undegraded":"2024-03-09T15:36:05.995245+0000","last_fullsized":"2024-03-09T15:36:05.995245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:58:42.664679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":128,"num_read_kb":128,"num_write":368,"num_write_kb":27813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.134035+0000","last_change":"2024-03-09T14:28:36.314599+0000","last_active":"2024-03-09T15:31:36.134035+0000","last_peered":"2024-03-09T15:31:36.134035+0000","last_clean":"2024-03-09T15:31:36.134035+0000","last_became_active":"2024-03-09T14:28:36.313984+0000","last_became_peered":"2024-03-09T14:28:36.313984+0000","last_unstale":"2024-03-09T15:31:36.134035+0000","last_undegraded":"2024-03-09T15:31:36.134035+0000","last_fullsized":"2024-03-09T15:31:36.134035+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:41:00.980683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"261'188","reported_seq":766,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.703452+0000","last_change":"2024-03-09T14:28:38.345412+0000","last_active":"2024-03-09T15:35:58.703452+0000","last_peered":"2024-03-09T15:35:58.703452+0000","last_clean":"2024-03-09T15:35:58.703452+0000","last_became_active":"2024-03-09T14:28:38.338262+0000","last_became_peered":"2024-03-09T14:28:38.338262+0000","last_unstale":"2024-03-09T15:35:58.703452+0000","last_undegraded":"2024-03-09T15:35:58.703452+0000","last_fullsized":"2024-03-09T15:35:58.703452+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":100,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:08:16.291678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29268,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":306,"num_read_kb":562,"num_write":216,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":489,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.777786+0000","last_change":"2024-03-09T14:28:40.390067+0000","last_active":"2024-03-09T15:31:40.777786+0000","last_peered":"2024-03-09T15:31:40.777786+0000","last_clean":"2024-03-09T15:31:40.777786+0000","last_became_active":"2024-03-09T14:28:40.389277+0000","last_became_peered":"2024-03-09T14:28:40.389277+0000","last_unstale":"2024-03-09T15:31:40.777786+0000","last_undegraded":"2024-03-09T15:31:40.777786+0000","last_fullsized":"2024-03-09T15:31:40.777786+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:02:25.553702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"261'54","reported_seq":549,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.795784+0000","last_change":"2024-03-09T14:28:42.411135+0000","last_active":"2024-03-09T15:33:00.795784+0000","last_peered":"2024-03-09T15:33:00.795784+0000","last_clean":"2024-03-09T15:33:00.795784+0000","last_became_active":"2024-03-09T14:28:42.409229+0000","last_became_peered":"2024-03-09T14:28:42.409229+0000","last_unstale":"2024-03-09T15:33:00.795784+0000","last_undegraded":"2024-03-09T15:33:00.795784+0000","last_fullsized":"2024-03-09T15:33:00.795784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:17:09.392755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":828,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":44,"num_write":116,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"261'78","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.134077+0000","last_change":"2024-03-09T14:28:34.320825+0000","last_active":"2024-03-09T15:31:36.134077+0000","last_peered":"2024-03-09T15:31:36.134077+0000","last_clean":"2024-03-09T15:31:36.134077+0000","last_became_active":"2024-03-09T14:28:34.320474+0000","last_became_peered":"2024-03-09T14:28:34.320474+0000","last_unstale":"2024-03-09T15:31:36.134077+0000","last_undegraded":"2024-03-09T15:31:36.134077+0000","last_fullsized":"2024-03-09T15:31:36.134077+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:15:48.878005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":231,"num_write":128,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"261'149","reported_seq":781,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.185355+0000","last_change":"2024-03-09T14:28:32.317900+0000","last_active":"2024-03-09T15:36:06.185355+0000","last_peered":"2024-03-09T15:36:06.185355+0000","last_clean":"2024-03-09T15:36:06.185355+0000","last_became_active":"2024-03-09T14:28:32.315355+0000","last_became_peered":"2024-03-09T14:28:32.315355+0000","last_unstale":"2024-03-09T15:36:06.185355+0000","last_undegraded":"2024-03-09T15:36:06.185355+0000","last_fullsized":"2024-03-09T15:36:06.185355+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:28:48.531373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655932,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":315,"num_read_kb":85989,"num_write":571,"num_write_kb":44811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776757+0000","last_change":"2024-03-09T14:28:36.311910+0000","last_active":"2024-03-09T15:31:35.776757+0000","last_peered":"2024-03-09T15:31:35.776757+0000","last_clean":"2024-03-09T15:31:35.776757+0000","last_became_active":"2024-03-09T14:28:36.311487+0000","last_became_peered":"2024-03-09T14:28:36.311487+0000","last_unstale":"2024-03-09T15:31:35.776757+0000","last_undegraded":"2024-03-09T15:31:35.776757+0000","last_fullsized":"2024-03-09T15:31:35.776757+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:04:59.933448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"261'39","reported_seq":550,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:31.436658+0000","last_change":"2024-03-09T14:28:38.333369+0000","last_active":"2024-03-09T15:33:31.436658+0000","last_peered":"2024-03-09T15:33:31.436658+0000","last_clean":"2024-03-09T15:33:31.436658+0000","last_became_active":"2024-03-09T14:28:38.332610+0000","last_became_peered":"2024-03-09T14:28:38.332610+0000","last_unstale":"2024-03-09T15:33:31.436658+0000","last_undegraded":"2024-03-09T15:33:31.436658+0000","last_fullsized":"2024-03-09T15:33:31.436658+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:56:36.283073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":41,"num_write":30,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":498,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.178031+0000","last_change":"2024-03-09T14:28:40.368708+0000","last_active":"2024-03-09T15:31:41.178031+0000","last_peered":"2024-03-09T15:31:41.178031+0000","last_clean":"2024-03-09T15:31:41.178031+0000","last_became_active":"2024-03-09T14:28:40.367795+0000","last_became_peered":"2024-03-09T14:28:40.367795+0000","last_unstale":"2024-03-09T15:31:41.178031+0000","last_undegraded":"2024-03-09T15:31:41.178031+0000","last_fullsized":"2024-03-09T15:31:41.178031+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:46:56.597658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"261'170","reported_seq":688,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:44.567149+0000","last_change":"2024-03-09T14:28:42.410358+0000","last_active":"2024-03-09T15:33:44.567149+0000","last_peered":"2024-03-09T15:33:44.567149+0000","last_clean":"2024-03-09T15:33:44.567149+0000","last_became_active":"2024-03-09T14:28:42.408913+0000","last_became_peered":"2024-03-09T14:28:42.408913+0000","last_unstale":"2024-03-09T15:33:44.567149+0000","last_undegraded":"2024-03-09T15:33:44.567149+0000","last_fullsized":"2024-03-09T15:33:44.567149+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":100,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:06:25.964093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":208,"num_write":351,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"261'54","reported_seq":593,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:52.364811+0000","last_change":"2024-03-09T14:28:34.331192+0000","last_active":"2024-03-09T15:31:52.364811+0000","last_peered":"2024-03-09T15:31:52.364811+0000","last_clean":"2024-03-09T15:31:52.364811+0000","last_became_active":"2024-03-09T14:28:34.330813+0000","last_became_peered":"2024-03-09T14:28:34.330813+0000","last_unstale":"2024-03-09T15:31:52.364811+0000","last_undegraded":"2024-03-09T15:31:52.364811+0000","last_fullsized":"2024-03-09T15:31:52.364811+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:02:15.115504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":220,"num_read_kb":178,"num_write":76,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"261'114","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.113284+0000","last_change":"2024-03-09T14:28:32.313709+0000","last_active":"2024-03-09T15:36:06.113284+0000","last_peered":"2024-03-09T15:36:06.113284+0000","last_clean":"2024-03-09T15:36:06.113284+0000","last_became_active":"2024-03-09T14:28:32.311843+0000","last_became_peered":"2024-03-09T14:28:32.311843+0000","last_unstale":"2024-03-09T15:36:06.113284+0000","last_undegraded":"2024-03-09T15:36:06.113284+0000","last_fullsized":"2024-03-09T15:36:06.113284+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:51:11.937911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1830596,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":144,"num_read_kb":5348,"num_write":462,"num_write_kb":35385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.176563+0000","last_change":"2024-03-09T14:28:36.324124+0000","last_active":"2024-03-09T15:31:36.176563+0000","last_peered":"2024-03-09T15:31:36.176563+0000","last_clean":"2024-03-09T15:31:36.176563+0000","last_became_active":"2024-03-09T14:28:36.323335+0000","last_became_peered":"2024-03-09T14:28:36.323335+0000","last_unstale":"2024-03-09T15:31:36.176563+0000","last_undegraded":"2024-03-09T15:31:36.176563+0000","last_fullsized":"2024-03-09T15:31:36.176563+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:09:55.020425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"261'219","reported_seq":812,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.676436+0000","last_change":"2024-03-09T14:28:38.345760+0000","last_active":"2024-03-09T15:35:58.676436+0000","last_peered":"2024-03-09T15:35:58.676436+0000","last_clean":"2024-03-09T15:35:58.676436+0000","last_became_active":"2024-03-09T14:28:38.344951+0000","last_became_peered":"2024-03-09T14:28:38.344951+0000","last_unstale":"2024-03-09T15:35:58.676436+0000","last_undegraded":"2024-03-09T15:35:58.676436+0000","last_fullsized":"2024-03-09T15:35:58.676436+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":200,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:52:01.179169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24078,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":344,"num_read_kb":584,"num_write":240,"num_write_kb":124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.766053+0000","last_change":"2024-03-09T14:28:40.374861+0000","last_active":"2024-03-09T15:31:40.766053+0000","last_peered":"2024-03-09T15:31:40.766053+0000","last_clean":"2024-03-09T15:31:40.766053+0000","last_became_active":"2024-03-09T14:28:40.370313+0000","last_became_peered":"2024-03-09T14:28:40.370313+0000","last_unstale":"2024-03-09T15:31:40.766053+0000","last_undegraded":"2024-03-09T15:31:40.766053+0000","last_fullsized":"2024-03-09T15:31:40.766053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:20:51.501674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"261'45","reported_seq":538,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:07.675530+0000","last_change":"2024-03-09T14:28:42.405094+0000","last_active":"2024-03-09T15:35:07.675530+0000","last_peered":"2024-03-09T15:35:07.675530+0000","last_clean":"2024-03-09T15:35:07.675530+0000","last_became_active":"2024-03-09T14:28:42.404360+0000","last_became_peered":"2024-03-09T14:28:42.404360+0000","last_unstale":"2024-03-09T15:35:07.675530+0000","last_undegraded":"2024-03-09T15:35:07.675530+0000","last_fullsized":"2024-03-09T15:35:07.675530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:33:45.977796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":43,"num_write":116,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"261'87","reported_seq":640,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776531+0000","last_change":"2024-03-09T14:28:34.313014+0000","last_active":"2024-03-09T15:31:35.776531+0000","last_peered":"2024-03-09T15:31:35.776531+0000","last_clean":"2024-03-09T15:31:35.776531+0000","last_became_active":"2024-03-09T14:28:34.305301+0000","last_became_peered":"2024-03-09T14:28:34.305301+0000","last_unstale":"2024-03-09T15:31:35.776531+0000","last_undegraded":"2024-03-09T15:31:35.776531+0000","last_fullsized":"2024-03-09T15:31:35.776531+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:46:37.987444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":317,"num_write":145,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"261'120","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.996239+0000","last_change":"2024-03-09T14:28:32.293199+0000","last_active":"2024-03-09T15:36:05.996239+0000","last_peered":"2024-03-09T15:36:05.996239+0000","last_clean":"2024-03-09T15:36:05.996239+0000","last_became_active":"2024-03-09T14:28:32.292393+0000","last_became_peered":"2024-03-09T14:28:32.292393+0000","last_unstale":"2024-03-09T15:36:05.996239+0000","last_undegraded":"2024-03-09T15:36:05.996239+0000","last_fullsized":"2024-03-09T15:36:05.996239+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:58:50.345089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7206306,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":165,"num_read_kb":14616,"num_write":445,"num_write_kb":23484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.764963+0000","last_change":"2024-03-09T14:28:36.329882+0000","last_active":"2024-03-09T15:31:35.764963+0000","last_peered":"2024-03-09T15:31:35.764963+0000","last_clean":"2024-03-09T15:31:35.764963+0000","last_became_active":"2024-03-09T14:28:36.328946+0000","last_became_peered":"2024-03-09T14:28:36.328946+0000","last_unstale":"2024-03-09T15:31:35.764963+0000","last_undegraded":"2024-03-09T15:31:35.764963+0000","last_fullsized":"2024-03-09T15:31:35.764963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:23:54.027658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"261'168","reported_seq":752,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.194863+0000","last_change":"2024-03-09T14:28:38.336043+0000","last_active":"2024-03-09T15:36:06.194863+0000","last_peered":"2024-03-09T15:36:06.194863+0000","last_clean":"2024-03-09T15:36:06.194863+0000","last_became_active":"2024-03-09T14:28:38.335107+0000","last_became_peered":"2024-03-09T14:28:38.335107+0000","last_unstale":"2024-03-09T15:36:06.194863+0000","last_undegraded":"2024-03-09T15:36:06.194863+0000","last_fullsized":"2024-03-09T15:36:06.194863+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":100,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:08:12.713928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9765,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":273,"num_read_kb":518,"num_write":186,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.985823+0000","last_change":"2024-03-09T14:28:40.364509+0000","last_active":"2024-03-09T15:31:40.985823+0000","last_peered":"2024-03-09T15:31:40.985823+0000","last_clean":"2024-03-09T15:31:40.985823+0000","last_became_active":"2024-03-09T14:28:40.363778+0000","last_became_peered":"2024-03-09T14:28:40.363778+0000","last_unstale":"2024-03-09T15:31:40.985823+0000","last_undegraded":"2024-03-09T15:31:40.985823+0000","last_fullsized":"2024-03-09T15:31:40.985823+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:49:40.990617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"261'159","reported_seq":733,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784514+0000","last_change":"2024-03-09T14:28:34.325746+0000","last_active":"2024-03-09T15:33:10.784514+0000","last_peered":"2024-03-09T15:33:10.784514+0000","last_clean":"2024-03-09T15:33:10.784514+0000","last_became_active":"2024-03-09T14:28:34.325031+0000","last_became_peered":"2024-03-09T14:28:34.325031+0000","last_unstale":"2024-03-09T15:33:10.784514+0000","last_undegraded":"2024-03-09T15:33:10.784514+0000","last_fullsized":"2024-03-09T15:33:10.784514+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:12:45.802989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":584,"num_read_kb":461,"num_write":254,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"261'97","reported_seq":636,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:52.210937+0000","last_change":"2024-03-09T14:28:32.296634+0000","last_active":"2024-03-09T15:31:52.210937+0000","last_peered":"2024-03-09T15:31:52.210937+0000","last_clean":"2024-03-09T15:31:52.210937+0000","last_became_active":"2024-03-09T14:28:32.296179+0000","last_became_peered":"2024-03-09T14:28:32.296179+0000","last_unstale":"2024-03-09T15:31:52.210937+0000","last_undegraded":"2024-03-09T15:31:52.210937+0000","last_fullsized":"2024-03-09T15:31:52.210937+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:07:03.555863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263400,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":122,"num_read_kb":11385,"num_write":357,"num_write_kb":17181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:51.002216+0000","last_change":"2024-03-09T14:28:36.320059+0000","last_active":"2024-03-09T15:32:51.002216+0000","last_peered":"2024-03-09T15:32:51.002216+0000","last_clean":"2024-03-09T15:32:51.002216+0000","last_became_active":"2024-03-09T14:28:36.319702+0000","last_became_peered":"2024-03-09T14:28:36.319702+0000","last_unstale":"2024-03-09T15:32:51.002216+0000","last_undegraded":"2024-03-09T15:32:51.002216+0000","last_fullsized":"2024-03-09T15:32:51.002216+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:38:28.003376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"261'309","reported_seq":944,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.877702+0000","last_change":"2024-03-09T14:28:38.333414+0000","last_active":"2024-03-09T15:36:05.877702+0000","last_peered":"2024-03-09T15:36:05.877702+0000","last_clean":"2024-03-09T15:36:05.877702+0000","last_became_active":"2024-03-09T14:28:38.332661+0000","last_became_peered":"2024-03-09T14:28:38.332661+0000","last_unstale":"2024-03-09T15:36:05.877702+0000","last_undegraded":"2024-03-09T15:36:05.877702+0000","last_fullsized":"2024-03-09T15:36:05.877702+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":200,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:09:54.404475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34580,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":556,"num_read_kb":1111,"num_write":442,"num_write_kb":360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:11.163194+0000","last_change":"2024-03-09T14:28:40.379385+0000","last_active":"2024-03-09T15:31:11.163194+0000","last_peered":"2024-03-09T15:31:11.163194+0000","last_clean":"2024-03-09T15:31:11.163194+0000","last_became_active":"2024-03-09T14:28:40.378829+0000","last_became_peered":"2024-03-09T14:28:40.378829+0000","last_unstale":"2024-03-09T15:31:11.163194+0000","last_undegraded":"2024-03-09T15:31:11.163194+0000","last_fullsized":"2024-03-09T15:31:11.163194+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:25:24.764687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"261'124","reported_seq":623,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.333094+0000","last_change":"2024-03-09T14:28:42.393487+0000","last_active":"2024-03-09T15:33:00.333094+0000","last_peered":"2024-03-09T15:33:00.333094+0000","last_clean":"2024-03-09T15:33:00.333094+0000","last_became_active":"2024-03-09T14:28:42.392509+0000","last_became_peered":"2024-03-09T14:28:42.392509+0000","last_unstale":"2024-03-09T15:33:00.333094+0000","last_undegraded":"2024-03-09T15:33:00.333094+0000","last_fullsized":"2024-03-09T15:33:00.333094+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:53:46.680420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":643,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":173,"num_write":275,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"261'132","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:51.002176+0000","last_change":"2024-03-09T14:28:34.330979+0000","last_active":"2024-03-09T15:32:51.002176+0000","last_peered":"2024-03-09T15:32:51.002176+0000","last_clean":"2024-03-09T15:32:51.002176+0000","last_became_active":"2024-03-09T14:28:34.330686+0000","last_became_peered":"2024-03-09T14:28:34.330686+0000","last_unstale":"2024-03-09T15:32:51.002176+0000","last_undegraded":"2024-03-09T15:32:51.002176+0000","last_fullsized":"2024-03-09T15:32:51.002176+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:07:36.006587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":527,"num_read_kb":451,"num_write":212,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"261'111","reported_seq":649,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.905029+0000","last_change":"2024-03-09T14:28:32.301602+0000","last_active":"2024-03-09T15:36:05.905029+0000","last_peered":"2024-03-09T15:36:05.905029+0000","last_clean":"2024-03-09T15:36:05.905029+0000","last_became_active":"2024-03-09T14:28:32.301087+0000","last_became_peered":"2024-03-09T14:28:32.301087+0000","last_unstale":"2024-03-09T15:36:05.905029+0000","last_undegraded":"2024-03-09T15:36:05.905029+0000","last_fullsized":"2024-03-09T15:36:05.905029+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:13:11.042530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":115,"num_read_kb":3393,"num_write":438,"num_write_kb":23270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784490+0000","last_change":"2024-03-09T14:28:36.329619+0000","last_active":"2024-03-09T15:33:10.784490+0000","last_peered":"2024-03-09T15:33:10.784490+0000","last_clean":"2024-03-09T15:33:10.784490+0000","last_became_active":"2024-03-09T14:28:36.328814+0000","last_became_peered":"2024-03-09T14:28:36.328814+0000","last_unstale":"2024-03-09T15:33:10.784490+0000","last_undegraded":"2024-03-09T15:33:10.784490+0000","last_fullsized":"2024-03-09T15:33:10.784490+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:24:43.021216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"261'252","reported_seq":855,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.221152+0000","last_change":"2024-03-09T14:28:38.332894+0000","last_active":"2024-03-09T15:36:06.221152+0000","last_peered":"2024-03-09T15:36:06.221152+0000","last_clean":"2024-03-09T15:36:06.221152+0000","last_became_active":"2024-03-09T14:28:38.331930+0000","last_became_peered":"2024-03-09T14:28:38.331930+0000","last_unstale":"2024-03-09T15:36:06.221152+0000","last_undegraded":"2024-03-09T15:36:06.221152+0000","last_fullsized":"2024-03-09T15:36:06.221152+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":200,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:36:58.809794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21830,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":424,"num_read_kb":798,"num_write":342,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:11.128169+0000","last_change":"2024-03-09T14:28:40.367660+0000","last_active":"2024-03-09T15:31:11.128169+0000","last_peered":"2024-03-09T15:31:11.128169+0000","last_clean":"2024-03-09T15:31:11.128169+0000","last_became_active":"2024-03-09T14:28:40.366723+0000","last_became_peered":"2024-03-09T14:28:40.366723+0000","last_unstale":"2024-03-09T15:31:11.128169+0000","last_undegraded":"2024-03-09T15:31:11.128169+0000","last_fullsized":"2024-03-09T15:31:11.128169+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:42:24.773777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"261'108","reported_seq":613,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:31.400225+0000","last_change":"2024-03-09T14:28:42.395993+0000","last_active":"2024-03-09T15:33:31.400225+0000","last_peered":"2024-03-09T15:33:31.400225+0000","last_clean":"2024-03-09T15:33:31.400225+0000","last_became_active":"2024-03-09T14:28:42.395375+0000","last_became_peered":"2024-03-09T14:28:42.395375+0000","last_unstale":"2024-03-09T15:33:31.400225+0000","last_undegraded":"2024-03-09T15:33:31.400225+0000","last_fullsized":"2024-03-09T15:33:31.400225+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:57:19.143770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1199,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":139,"num_read_kb":136,"num_write":341,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"261'99","reported_seq":686,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:45.201625+0000","last_change":"2024-03-09T14:28:34.312670+0000","last_active":"2024-03-09T15:34:45.201625+0000","last_peered":"2024-03-09T15:34:45.201625+0000","last_clean":"2024-03-09T15:34:45.201625+0000","last_became_active":"2024-03-09T14:28:34.312297+0000","last_became_peered":"2024-03-09T14:28:34.312297+0000","last_unstale":"2024-03-09T15:34:45.201625+0000","last_undegraded":"2024-03-09T15:34:45.201625+0000","last_fullsized":"2024-03-09T15:34:45.201625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:36:43.430324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":415,"num_read_kb":347,"num_write":146,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"261'101","reported_seq":669,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.985205+0000","last_change":"2024-03-09T14:28:32.314186+0000","last_active":"2024-03-09T15:36:05.985205+0000","last_peered":"2024-03-09T15:36:05.985205+0000","last_clean":"2024-03-09T15:36:05.985205+0000","last_became_active":"2024-03-09T14:28:32.312937+0000","last_became_peered":"2024-03-09T14:28:32.312937+0000","last_unstale":"2024-03-09T15:36:05.985205+0000","last_undegraded":"2024-03-09T15:36:05.985205+0000","last_fullsized":"2024-03-09T15:36:05.985205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:33:36.448497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262330,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":144,"num_read_kb":47424,"num_write":415,"num_write_kb":24173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191452+0000","last_change":"2024-03-09T14:28:36.316936+0000","last_active":"2024-03-09T15:33:36.191452+0000","last_peered":"2024-03-09T15:33:36.191452+0000","last_clean":"2024-03-09T15:33:36.191452+0000","last_became_active":"2024-03-09T14:28:36.316094+0000","last_became_peered":"2024-03-09T14:28:36.316094+0000","last_unstale":"2024-03-09T15:33:36.191452+0000","last_undegraded":"2024-03-09T15:33:36.191452+0000","last_fullsized":"2024-03-09T15:33:36.191452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:41:42.620548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"261'188","reported_seq":770,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.713172+0000","last_change":"2024-03-09T14:28:38.596294+0000","last_active":"2024-03-09T15:35:58.713172+0000","last_peered":"2024-03-09T15:35:58.713172+0000","last_clean":"2024-03-09T15:35:58.713172+0000","last_became_active":"2024-03-09T14:28:38.595522+0000","last_became_peered":"2024-03-09T14:28:38.595522+0000","last_unstale":"2024-03-09T15:35:58.713172+0000","last_undegraded":"2024-03-09T15:35:58.713172+0000","last_fullsized":"2024-03-09T15:35:58.713172+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":100,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:20:49.471413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25352,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":302,"num_read_kb":534,"num_write":212,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:50.952318+0000","last_change":"2024-03-09T14:28:40.377325+0000","last_active":"2024-03-09T15:32:50.952318+0000","last_peered":"2024-03-09T15:32:50.952318+0000","last_clean":"2024-03-09T15:32:50.952318+0000","last_became_active":"2024-03-09T14:28:40.376549+0000","last_became_peered":"2024-03-09T14:28:40.376549+0000","last_unstale":"2024-03-09T15:32:50.952318+0000","last_undegraded":"2024-03-09T15:32:50.952318+0000","last_fullsized":"2024-03-09T15:32:50.952318+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:59:06.393597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"261'42","reported_seq":537,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.728760+0000","last_change":"2024-03-09T14:28:42.395593+0000","last_active":"2024-03-09T15:35:58.728760+0000","last_peered":"2024-03-09T15:35:58.728760+0000","last_clean":"2024-03-09T15:35:58.728760+0000","last_became_active":"2024-03-09T14:28:42.394780+0000","last_became_peered":"2024-03-09T14:28:42.394780+0000","last_unstale":"2024-03-09T15:35:58.728760+0000","last_undegraded":"2024-03-09T15:35:58.728760+0000","last_fullsized":"2024-03-09T15:35:58.728760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:21:45.395383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":84,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"261'68","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:22.422461+0000","last_change":"2024-03-09T14:28:34.313519+0000","last_active":"2024-03-09T15:34:22.422461+0000","last_peered":"2024-03-09T15:34:22.422461+0000","last_clean":"2024-03-09T15:34:22.422461+0000","last_became_active":"2024-03-09T14:28:34.313125+0000","last_became_peered":"2024-03-09T14:28:34.313125+0000","last_unstale":"2024-03-09T15:34:22.422461+0000","last_undegraded":"2024-03-09T15:34:22.422461+0000","last_fullsized":"2024-03-09T15:34:22.422461+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:57:07.167382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"261'175","reported_seq":779,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.986932+0000","last_change":"2024-03-09T14:28:32.304248+0000","last_active":"2024-03-09T15:36:05.986932+0000","last_peered":"2024-03-09T15:36:05.986932+0000","last_clean":"2024-03-09T15:36:05.986932+0000","last_became_active":"2024-03-09T14:28:32.303835+0000","last_became_peered":"2024-03-09T14:28:32.303835+0000","last_unstale":"2024-03-09T15:36:05.986932+0000","last_undegraded":"2024-03-09T15:36:05.986932+0000","last_fullsized":"2024-03-09T15:36:05.986932+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":100,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:32:30.929760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":359,"num_read_kb":1431,"num_write":987,"num_write_kb":8832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963406+0000","last_change":"2024-03-09T14:28:36.322532+0000","last_active":"2024-03-09T15:33:35.963406+0000","last_peered":"2024-03-09T15:33:35.963406+0000","last_clean":"2024-03-09T15:33:35.963406+0000","last_became_active":"2024-03-09T14:28:36.322060+0000","last_became_peered":"2024-03-09T14:28:36.322060+0000","last_unstale":"2024-03-09T15:33:35.963406+0000","last_undegraded":"2024-03-09T15:33:35.963406+0000","last_fullsized":"2024-03-09T15:33:35.963406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:48:28.324190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"261'89","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.180393+0000","last_change":"2024-03-09T14:28:38.596598+0000","last_active":"2024-03-09T15:32:46.180393+0000","last_peered":"2024-03-09T15:32:46.180393+0000","last_clean":"2024-03-09T15:32:46.180393+0000","last_became_active":"2024-03-09T14:28:38.595749+0000","last_became_peered":"2024-03-09T14:28:38.595749+0000","last_unstale":"2024-03-09T15:32:46.180393+0000","last_undegraded":"2024-03-09T15:32:46.180393+0000","last_fullsized":"2024-03-09T15:32:46.180393+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:47:35.161583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":132,"num_read_kb":127,"num_write":124,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"261'488","reported_seq":2256,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.713372+0000","last_change":"2024-03-09T14:28:40.377689+0000","last_active":"2024-03-09T15:36:05.713372+0000","last_peered":"2024-03-09T15:36:05.713372+0000","last_clean":"2024-03-09T15:36:05.713372+0000","last_became_active":"2024-03-09T14:28:40.376946+0000","last_became_peered":"2024-03-09T14:28:40.376946+0000","last_unstale":"2024-03-09T15:36:05.713372+0000","last_undegraded":"2024-03-09T15:36:05.713372+0000","last_fullsized":"2024-03-09T15:36:05.713372+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":400,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:50:25.914007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.10","version":"261'45","reported_seq":549,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:48.564964+0000","last_change":"2024-03-09T14:28:42.399023+0000","last_active":"2024-03-09T15:33:48.564964+0000","last_peered":"2024-03-09T15:33:48.564964+0000","last_clean":"2024-03-09T15:33:48.564964+0000","last_became_active":"2024-03-09T14:28:42.397829+0000","last_became_peered":"2024-03-09T14:28:42.397829+0000","last_unstale":"2024-03-09T15:33:48.564964+0000","last_undegraded":"2024-03-09T15:33:48.564964+0000","last_fullsized":"2024-03-09T15:33:48.564964+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:36:00.626046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":97,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"261'144","reported_seq":675,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191372+0000","last_change":"2024-03-09T14:28:34.321426+0000","last_active":"2024-03-09T15:33:36.191372+0000","last_peered":"2024-03-09T15:33:36.191372+0000","last_clean":"2024-03-09T15:33:36.191372+0000","last_became_active":"2024-03-09T14:28:34.319891+0000","last_became_peered":"2024-03-09T14:28:34.319891+0000","last_unstale":"2024-03-09T15:33:36.191372+0000","last_undegraded":"2024-03-09T15:33:36.191372+0000","last_fullsized":"2024-03-09T15:33:36.191372+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:25:46.129222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":571,"num_read_kb":433,"num_write":263,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"261'145","reported_seq":791,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.201525+0000","last_change":"2024-03-09T14:28:32.323880+0000","last_active":"2024-03-09T15:36:06.201525+0000","last_peered":"2024-03-09T15:36:06.201525+0000","last_clean":"2024-03-09T15:36:06.201525+0000","last_became_active":"2024-03-09T14:28:32.323198+0000","last_became_peered":"2024-03-09T14:28:32.323198+0000","last_unstale":"2024-03-09T15:36:06.201525+0000","last_undegraded":"2024-03-09T15:36:06.201525+0000","last_fullsized":"2024-03-09T15:36:06.201525+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:10:52.793653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4801712,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":390,"num_read_kb":2095,"num_write":636,"num_write_kb":25158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205346+0000","last_change":"2024-03-09T14:28:36.329314+0000","last_active":"2024-03-09T15:33:36.205346+0000","last_peered":"2024-03-09T15:33:36.205346+0000","last_clean":"2024-03-09T15:33:36.205346+0000","last_became_active":"2024-03-09T14:28:36.328529+0000","last_became_peered":"2024-03-09T14:28:36.328529+0000","last_unstale":"2024-03-09T15:33:36.205346+0000","last_undegraded":"2024-03-09T15:33:36.205346+0000","last_fullsized":"2024-03-09T15:33:36.205346+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:42:17.375037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"261'224","reported_seq":826,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.002456+0000","last_change":"2024-03-09T14:28:38.335758+0000","last_active":"2024-03-09T15:36:06.002456+0000","last_peered":"2024-03-09T15:36:06.002456+0000","last_clean":"2024-03-09T15:36:06.002456+0000","last_became_active":"2024-03-09T14:28:38.334991+0000","last_became_peered":"2024-03-09T14:28:38.334991+0000","last_unstale":"2024-03-09T15:36:06.002456+0000","last_undegraded":"2024-03-09T15:36:06.002456+0000","last_fullsized":"2024-03-09T15:36:06.002456+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":200,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:19:26.980708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13314,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":373,"num_read_kb":687,"num_write":264,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:30.983178+0000","last_change":"2024-03-09T14:28:40.368865+0000","last_active":"2024-03-09T15:31:30.983178+0000","last_peered":"2024-03-09T15:31:30.983178+0000","last_clean":"2024-03-09T15:31:30.983178+0000","last_became_active":"2024-03-09T14:28:40.368039+0000","last_became_peered":"2024-03-09T14:28:40.368039+0000","last_unstale":"2024-03-09T15:31:30.983178+0000","last_undegraded":"2024-03-09T15:31:30.983178+0000","last_fullsized":"2024-03-09T15:31:30.983178+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:50:42.121684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"261'77","reported_seq":597,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:46.896952+0000","last_change":"2024-03-09T14:28:42.414457+0000","last_active":"2024-03-09T15:34:46.896952+0000","last_peered":"2024-03-09T15:34:46.896952+0000","last_clean":"2024-03-09T15:34:46.896952+0000","last_became_active":"2024-03-09T14:28:42.413840+0000","last_became_peered":"2024-03-09T14:28:42.413840+0000","last_unstale":"2024-03-09T15:34:46.896952+0000","last_undegraded":"2024-03-09T15:34:46.896952+0000","last_fullsized":"2024-03-09T15:34:46.896952+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:42:30.742160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":129,"num_write":187,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"261'36","reported_seq":573,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:52.977046+0000","last_change":"2024-03-09T14:28:34.318737+0000","last_active":"2024-03-09T15:35:52.977046+0000","last_peered":"2024-03-09T15:35:52.977046+0000","last_clean":"2024-03-09T15:35:52.977046+0000","last_became_active":"2024-03-09T14:28:34.314194+0000","last_became_peered":"2024-03-09T14:28:34.314194+0000","last_unstale":"2024-03-09T15:35:52.977046+0000","last_undegraded":"2024-03-09T15:35:52.977046+0000","last_fullsized":"2024-03-09T15:35:52.977046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:45:29.912431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":120,"num_write":54,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"261'128","reported_seq":781,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.160748+0000","last_change":"2024-03-09T14:28:32.309431+0000","last_active":"2024-03-09T15:36:06.160748+0000","last_peered":"2024-03-09T15:36:06.160748+0000","last_clean":"2024-03-09T15:36:06.160748+0000","last_became_active":"2024-03-09T14:28:32.307702+0000","last_became_peered":"2024-03-09T14:28:32.307702+0000","last_unstale":"2024-03-09T15:36:06.160748+0000","last_undegraded":"2024-03-09T15:36:06.160748+0000","last_fullsized":"2024-03-09T15:36:06.160748+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:51:59.883119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":254,"num_read_kb":1758,"num_write":464,"num_write_kb":23078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194464+0000","last_change":"2024-03-09T14:28:36.305006+0000","last_active":"2024-03-09T15:33:36.194464+0000","last_peered":"2024-03-09T15:33:36.194464+0000","last_clean":"2024-03-09T15:33:36.194464+0000","last_became_active":"2024-03-09T14:28:36.304241+0000","last_became_peered":"2024-03-09T14:28:36.304241+0000","last_unstale":"2024-03-09T15:33:36.194464+0000","last_undegraded":"2024-03-09T15:33:36.194464+0000","last_fullsized":"2024-03-09T15:33:36.194464+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:14:09.284541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"261'178","reported_seq":757,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.035481+0000","last_change":"2024-03-09T14:28:38.340000+0000","last_active":"2024-03-09T15:36:06.035481+0000","last_peered":"2024-03-09T15:36:06.035481+0000","last_clean":"2024-03-09T15:36:06.035481+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:36:06.035481+0000","last_undegraded":"2024-03-09T15:36:06.035481+0000","last_fullsized":"2024-03-09T15:36:06.035481+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":100,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:06:57.175032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":521,"num_write":190,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.768268+0000","last_change":"2024-03-09T14:28:40.375515+0000","last_active":"2024-03-09T15:31:50.768268+0000","last_peered":"2024-03-09T15:31:50.768268+0000","last_clean":"2024-03-09T15:31:50.768268+0000","last_became_active":"2024-03-09T14:28:40.371820+0000","last_became_peered":"2024-03-09T14:28:40.371820+0000","last_unstale":"2024-03-09T15:31:50.768268+0000","last_undegraded":"2024-03-09T15:31:50.768268+0000","last_fullsized":"2024-03-09T15:31:50.768268+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:15:53.172763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.16","version":"261'53","reported_seq":567,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.020972+0000","last_change":"2024-03-09T14:28:42.392999+0000","last_active":"2024-03-09T15:33:10.020972+0000","last_peered":"2024-03-09T15:33:10.020972+0000","last_clean":"2024-03-09T15:33:10.020972+0000","last_became_active":"2024-03-09T14:28:42.392326+0000","last_became_peered":"2024-03-09T14:28:42.392326+0000","last_unstale":"2024-03-09T15:33:10.020972+0000","last_undegraded":"2024-03-09T15:33:10.020972+0000","last_fullsized":"2024-03-09T15:33:10.020972+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:36.796428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":772,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":99,"num_write":117,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"261'70","reported_seq":632,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.777840+0000","last_change":"2024-03-09T14:28:34.324293+0000","last_active":"2024-03-09T15:31:40.777840+0000","last_peered":"2024-03-09T15:31:40.777840+0000","last_clean":"2024-03-09T15:31:40.777840+0000","last_became_active":"2024-03-09T14:28:34.323969+0000","last_became_peered":"2024-03-09T14:28:34.323969+0000","last_unstale":"2024-03-09T15:31:40.777840+0000","last_undegraded":"2024-03-09T15:31:40.777840+0000","last_fullsized":"2024-03-09T15:31:40.777840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:33:30.511134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":256,"num_write":98,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"261'149","reported_seq":740,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:50.793396+0000","last_change":"2024-03-09T14:28:32.338014+0000","last_active":"2024-03-09T15:32:50.793396+0000","last_peered":"2024-03-09T15:32:50.793396+0000","last_clean":"2024-03-09T15:32:50.793396+0000","last_became_active":"2024-03-09T14:28:32.337755+0000","last_became_peered":"2024-03-09T14:28:32.337755+0000","last_unstale":"2024-03-09T15:32:50.793396+0000","last_undegraded":"2024-03-09T15:32:50.793396+0000","last_fullsized":"2024-03-09T15:32:50.793396+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:19:21.576018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663899,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":272,"num_read_kb":3602,"num_write":640,"num_write_kb":11566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162410+0000","last_change":"2024-03-09T14:28:36.318326+0000","last_active":"2024-03-09T15:31:26.162410+0000","last_peered":"2024-03-09T15:31:26.162410+0000","last_clean":"2024-03-09T15:31:26.162410+0000","last_became_active":"2024-03-09T14:28:36.317569+0000","last_became_peered":"2024-03-09T14:28:36.317569+0000","last_unstale":"2024-03-09T15:31:26.162410+0000","last_undegraded":"2024-03-09T15:31:26.162410+0000","last_fullsized":"2024-03-09T15:31:26.162410+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:21:30.385778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"261'149","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.708300+0000","last_change":"2024-03-09T14:28:38.348095+0000","last_active":"2024-03-09T15:35:58.708300+0000","last_peered":"2024-03-09T15:35:58.708300+0000","last_clean":"2024-03-09T15:35:58.708300+0000","last_became_active":"2024-03-09T14:28:38.347370+0000","last_became_peered":"2024-03-09T14:28:38.347370+0000","last_unstale":"2024-03-09T15:35:58.708300+0000","last_undegraded":"2024-03-09T15:35:58.708300+0000","last_fullsized":"2024-03-09T15:35:58.708300+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:00:09.043640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32290,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":236,"num_read_kb":490,"num_write":168,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":495,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:11.159141+0000","last_change":"2024-03-09T14:28:40.377364+0000","last_active":"2024-03-09T15:31:11.159141+0000","last_peered":"2024-03-09T15:31:11.159141+0000","last_clean":"2024-03-09T15:31:11.159141+0000","last_became_active":"2024-03-09T14:28:40.376513+0000","last_became_peered":"2024-03-09T14:28:40.376513+0000","last_unstale":"2024-03-09T15:31:11.159141+0000","last_undegraded":"2024-03-09T15:31:11.159141+0000","last_fullsized":"2024-03-09T15:31:11.159141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:28.590369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"261'92","reported_seq":974,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:54.162220+0000","last_change":"2024-03-09T14:28:42.410566+0000","last_active":"2024-03-09T15:35:54.162220+0000","last_peered":"2024-03-09T15:35:54.162220+0000","last_clean":"2024-03-09T15:35:54.162220+0000","last_became_active":"2024-03-09T14:28:42.409717+0000","last_became_peered":"2024-03-09T14:28:42.409717+0000","last_unstale":"2024-03-09T15:35:54.162220+0000","last_undegraded":"2024-03-09T15:35:54.162220+0000","last_fullsized":"2024-03-09T15:35:54.162220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:07:09.353068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":500,"num_read_kb":493,"num_write":215,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"261'53","reported_seq":585,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162448+0000","last_change":"2024-03-09T14:28:34.314180+0000","last_active":"2024-03-09T15:31:26.162448+0000","last_peered":"2024-03-09T15:31:26.162448+0000","last_clean":"2024-03-09T15:31:26.162448+0000","last_became_active":"2024-03-09T14:28:34.313380+0000","last_became_peered":"2024-03-09T14:28:34.313380+0000","last_unstale":"2024-03-09T15:31:26.162448+0000","last_undegraded":"2024-03-09T15:31:26.162448+0000","last_fullsized":"2024-03-09T15:31:26.162448+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:29:27.979809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":197,"num_read_kb":156,"num_write":74,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"261'146","reported_seq":733,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.965677+0000","last_change":"2024-03-09T14:28:32.293553+0000","last_active":"2024-03-09T15:36:05.965677+0000","last_peered":"2024-03-09T15:36:05.965677+0000","last_clean":"2024-03-09T15:36:05.965677+0000","last_became_active":"2024-03-09T14:28:32.292862+0000","last_became_peered":"2024-03-09T14:28:32.292862+0000","last_unstale":"2024-03-09T15:36:05.965677+0000","last_undegraded":"2024-03-09T15:36:05.965677+0000","last_fullsized":"2024-03-09T15:36:05.965677+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:21:35.805317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2792030,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":249,"num_read_kb":35693,"num_write":653,"num_write_kb":24072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"25'1","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.132760+0000","last_change":"2024-03-09T14:28:36.314323+0000","last_active":"2024-03-09T15:31:31.132760+0000","last_peered":"2024-03-09T15:31:31.132760+0000","last_clean":"2024-03-09T15:31:31.132760+0000","last_became_active":"2024-03-09T14:28:36.313405+0000","last_became_peered":"2024-03-09T14:28:36.313405+0000","last_unstale":"2024-03-09T15:31:31.132760+0000","last_undegraded":"2024-03-09T15:31:31.132760+0000","last_fullsized":"2024-03-09T15:31:31.132760+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:46:02.220171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"261'207","reported_seq":799,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.716100+0000","last_change":"2024-03-09T14:28:38.333268+0000","last_active":"2024-03-09T15:35:58.716100+0000","last_peered":"2024-03-09T15:35:58.716100+0000","last_clean":"2024-03-09T15:35:58.716100+0000","last_became_active":"2024-03-09T14:28:38.330571+0000","last_became_peered":"2024-03-09T14:28:38.330571+0000","last_unstale":"2024-03-09T15:35:58.716100+0000","last_undegraded":"2024-03-09T15:35:58.716100+0000","last_fullsized":"2024-03-09T15:35:58.716100+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":100,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:13:06.110510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27588,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":340,"num_read_kb":612,"num_write":242,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":491,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:10.978895+0000","last_change":"2024-03-09T14:28:40.385742+0000","last_active":"2024-03-09T15:31:10.978895+0000","last_peered":"2024-03-09T15:31:10.978895+0000","last_clean":"2024-03-09T15:31:10.978895+0000","last_became_active":"2024-03-09T14:28:40.385080+0000","last_became_peered":"2024-03-09T14:28:40.385080+0000","last_unstale":"2024-03-09T15:31:10.978895+0000","last_undegraded":"2024-03-09T15:31:10.978895+0000","last_fullsized":"2024-03-09T15:31:10.978895+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:08:01.706715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"261'69","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:11.304332+0000","last_change":"2024-03-09T14:28:42.413260+0000","last_active":"2024-03-09T15:34:11.304332+0000","last_peered":"2024-03-09T15:34:11.304332+0000","last_clean":"2024-03-09T15:34:11.304332+0000","last_became_active":"2024-03-09T14:28:42.412616+0000","last_became_peered":"2024-03-09T14:28:42.412616+0000","last_unstale":"2024-03-09T15:34:11.304332+0000","last_undegraded":"2024-03-09T15:34:11.304332+0000","last_fullsized":"2024-03-09T15:34:11.304332+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:28:05.834021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":49,"num_write":163,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.27","version":"261'50","reported_seq":597,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963528+0000","last_change":"2024-03-09T14:28:34.307137+0000","last_active":"2024-03-09T15:33:35.963528+0000","last_peered":"2024-03-09T15:33:35.963528+0000","last_clean":"2024-03-09T15:33:35.963528+0000","last_became_active":"2024-03-09T14:28:34.306685+0000","last_became_peered":"2024-03-09T14:28:34.306685+0000","last_unstale":"2024-03-09T15:33:35.963528+0000","last_undegraded":"2024-03-09T15:33:35.963528+0000","last_fullsized":"2024-03-09T15:33:35.963528+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:19:14.669885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":204,"num_read_kb":164,"num_write":70,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"261'122","reported_seq":705,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:13.078377+0000","last_change":"2024-03-09T14:28:32.334098+0000","last_active":"2024-03-09T15:33:13.078377+0000","last_peered":"2024-03-09T15:33:13.078377+0000","last_clean":"2024-03-09T15:33:13.078377+0000","last_became_active":"2024-03-09T14:28:32.333008+0000","last_became_peered":"2024-03-09T14:28:32.333008+0000","last_unstale":"2024-03-09T15:33:13.078377+0000","last_undegraded":"2024-03-09T15:33:13.078377+0000","last_fullsized":"2024-03-09T15:33:13.078377+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:19:21.576018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1943763,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":219,"num_read_kb":10390,"num_write":536,"num_write_kb":33266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.26","version":"261'181","reported_seq":772,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963471+0000","last_change":"2024-03-09T14:28:34.314634+0000","last_active":"2024-03-09T15:33:35.963471+0000","last_peered":"2024-03-09T15:33:35.963471+0000","last_clean":"2024-03-09T15:33:35.963471+0000","last_became_active":"2024-03-09T14:28:34.314007+0000","last_became_peered":"2024-03-09T14:28:34.314007+0000","last_unstale":"2024-03-09T15:33:35.963471+0000","last_undegraded":"2024-03-09T15:33:35.963471+0000","last_fullsized":"2024-03-09T15:33:35.963471+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":100,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:50:26.398988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":729,"num_read_kb":586,"num_write":272,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"261'127","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.956327+0000","last_change":"2024-03-09T14:28:32.287770+0000","last_active":"2024-03-09T15:36:05.956327+0000","last_peered":"2024-03-09T15:36:05.956327+0000","last_clean":"2024-03-09T15:36:05.956327+0000","last_became_active":"2024-03-09T14:28:32.286609+0000","last_became_peered":"2024-03-09T14:28:32.286609+0000","last_unstale":"2024-03-09T15:36:05.956327+0000","last_undegraded":"2024-03-09T15:36:05.956327+0000","last_fullsized":"2024-03-09T15:36:05.956327+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":100,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:14.582228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":810412,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":129,"num_read_kb":1907,"num_write":474,"num_write_kb":32192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"261'93","reported_seq":657,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194646+0000","last_change":"2024-03-09T14:28:34.325388+0000","last_active":"2024-03-09T15:33:36.194646+0000","last_peered":"2024-03-09T15:33:36.194646+0000","last_clean":"2024-03-09T15:33:36.194646+0000","last_became_active":"2024-03-09T14:28:34.323541+0000","last_became_peered":"2024-03-09T14:28:34.323541+0000","last_unstale":"2024-03-09T15:33:36.194646+0000","last_undegraded":"2024-03-09T15:33:36.194646+0000","last_fullsized":"2024-03-09T15:33:36.194646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:35:47.676249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":384,"num_read_kb":306,"num_write":146,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"261'160","reported_seq":756,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.074470+0000","last_change":"2024-03-09T14:28:32.299059+0000","last_active":"2024-03-09T15:36:06.074470+0000","last_peered":"2024-03-09T15:36:06.074470+0000","last_clean":"2024-03-09T15:36:06.074470+0000","last_became_active":"2024-03-09T14:28:32.295699+0000","last_became_peered":"2024-03-09T14:28:32.295699+0000","last_unstale":"2024-03-09T15:36:06.074470+0000","last_undegraded":"2024-03-09T15:36:06.074470+0000","last_fullsized":"2024-03-09T15:36:06.074470+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":100,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:14:50.303305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":286,"num_read_kb":14189,"num_write":705,"num_write_kb":33521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"261'101","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963269+0000","last_change":"2024-03-09T14:28:34.322186+0000","last_active":"2024-03-09T15:33:35.963269+0000","last_peered":"2024-03-09T15:33:35.963269+0000","last_clean":"2024-03-09T15:33:35.963269+0000","last_became_active":"2024-03-09T14:28:34.321647+0000","last_became_peered":"2024-03-09T14:28:34.321647+0000","last_unstale":"2024-03-09T15:33:35.963269+0000","last_undegraded":"2024-03-09T15:33:35.963269+0000","last_fullsized":"2024-03-09T15:33:35.963269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:19:44.897314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":329,"num_write":161,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"261'119","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.887640+0000","last_change":"2024-03-09T14:28:32.298327+0000","last_active":"2024-03-09T15:36:05.887640+0000","last_peered":"2024-03-09T15:36:05.887640+0000","last_clean":"2024-03-09T15:36:05.887640+0000","last_became_active":"2024-03-09T14:28:32.297866+0000","last_became_peered":"2024-03-09T14:28:32.297866+0000","last_unstale":"2024-03-09T15:36:05.887640+0000","last_undegraded":"2024-03-09T15:36:05.887640+0000","last_fullsized":"2024-03-09T15:36:05.887640+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":100,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:49:38.048255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":154,"num_read_kb":3467,"num_write":486,"num_write_kb":23815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"261'52","reported_seq":601,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.801400+0000","last_change":"2024-03-09T14:28:34.330916+0000","last_active":"2024-03-09T15:33:25.801400+0000","last_peered":"2024-03-09T15:33:25.801400+0000","last_clean":"2024-03-09T15:33:25.801400+0000","last_became_active":"2024-03-09T14:28:34.330511+0000","last_became_peered":"2024-03-09T14:28:34.330511+0000","last_unstale":"2024-03-09T15:33:25.801400+0000","last_undegraded":"2024-03-09T15:33:25.801400+0000","last_fullsized":"2024-03-09T15:33:25.801400+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:29:16.007058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":225,"num_read_kb":185,"num_write":82,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"261'102","reported_seq":651,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:52.427722+0000","last_change":"2024-03-09T14:28:32.326489+0000","last_active":"2024-03-09T15:33:52.427722+0000","last_peered":"2024-03-09T15:33:52.427722+0000","last_clean":"2024-03-09T15:33:52.427722+0000","last_became_active":"2024-03-09T14:28:32.326148+0000","last_became_peered":"2024-03-09T14:28:32.326148+0000","last_unstale":"2024-03-09T15:33:52.427722+0000","last_undegraded":"2024-03-09T15:33:52.427722+0000","last_fullsized":"2024-03-09T15:33:52.427722+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:59:13.121365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":157,"num_read_kb":5614,"num_write":435,"num_write_kb":21169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"261'134","reported_seq":685,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.185748+0000","last_change":"2024-03-09T14:28:34.323012+0000","last_active":"2024-03-09T15:33:11.185748+0000","last_peered":"2024-03-09T15:33:11.185748+0000","last_clean":"2024-03-09T15:33:11.185748+0000","last_became_active":"2024-03-09T14:28:34.322225+0000","last_became_peered":"2024-03-09T14:28:34.322225+0000","last_unstale":"2024-03-09T15:33:11.185748+0000","last_undegraded":"2024-03-09T15:33:11.185748+0000","last_fullsized":"2024-03-09T15:33:11.185748+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:33:31.665609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":498,"num_read_kb":399,"num_write":194,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"261'145","reported_seq":711,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:45.990374+0000","last_change":"2024-03-09T14:28:32.300246+0000","last_active":"2024-03-09T15:35:45.990374+0000","last_peered":"2024-03-09T15:35:45.990374+0000","last_clean":"2024-03-09T15:35:45.990374+0000","last_became_active":"2024-03-09T14:28:32.299839+0000","last_became_peered":"2024-03-09T14:28:32.299839+0000","last_unstale":"2024-03-09T15:35:45.990374+0000","last_undegraded":"2024-03-09T15:35:45.990374+0000","last_fullsized":"2024-03-09T15:35:45.990374+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:05:01.790489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243228,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":189,"num_read_kb":18833,"num_write":580,"num_write_kb":39169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"261'74","reported_seq":608,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:17.871898+0000","last_change":"2024-03-09T14:28:34.315670+0000","last_active":"2024-03-09T15:34:17.871898+0000","last_peered":"2024-03-09T15:34:17.871898+0000","last_clean":"2024-03-09T15:34:17.871898+0000","last_became_active":"2024-03-09T14:28:34.315302+0000","last_became_peered":"2024-03-09T14:28:34.315302+0000","last_unstale":"2024-03-09T15:34:17.871898+0000","last_undegraded":"2024-03-09T15:34:17.871898+0000","last_fullsized":"2024-03-09T15:34:17.871898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:54:03.674581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":224,"num_write":116,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"261'122","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.789837+0000","last_change":"2024-03-09T14:28:32.336827+0000","last_active":"2024-03-09T15:33:35.789837+0000","last_peered":"2024-03-09T15:33:35.789837+0000","last_clean":"2024-03-09T15:33:35.789837+0000","last_became_active":"2024-03-09T14:28:32.334117+0000","last_became_peered":"2024-03-09T14:28:32.334117+0000","last_unstale":"2024-03-09T15:33:35.789837+0000","last_undegraded":"2024-03-09T15:33:35.789837+0000","last_fullsized":"2024-03-09T15:33:35.789837+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:58:15.793689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243207,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":155,"num_read_kb":192,"num_write":536,"num_write_kb":14566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"261'48","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784320+0000","last_change":"2024-03-09T14:28:34.323974+0000","last_active":"2024-03-09T15:33:10.784320+0000","last_peered":"2024-03-09T15:33:10.784320+0000","last_clean":"2024-03-09T15:33:10.784320+0000","last_became_active":"2024-03-09T14:28:34.323202+0000","last_became_peered":"2024-03-09T14:28:34.323202+0000","last_unstale":"2024-03-09T15:33:10.784320+0000","last_undegraded":"2024-03-09T15:33:10.784320+0000","last_fullsized":"2024-03-09T15:33:10.784320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:33:12.688593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":219,"num_read_kb":183,"num_write":70,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"261'117","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.803914+0000","last_change":"2024-03-09T14:28:32.300258+0000","last_active":"2024-03-09T15:33:35.803914+0000","last_peered":"2024-03-09T15:33:35.803914+0000","last_clean":"2024-03-09T15:33:35.803914+0000","last_became_active":"2024-03-09T14:28:32.299737+0000","last_became_peered":"2024-03-09T14:28:32.299737+0000","last_unstale":"2024-03-09T15:33:35.803914+0000","last_undegraded":"2024-03-09T15:33:35.803914+0000","last_fullsized":"2024-03-09T15:33:35.803914+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:26:30.693287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6457661,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":147,"num_read_kb":12613,"num_write":500,"num_write_kb":31618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"261'94","reported_seq":644,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.984747+0000","last_change":"2024-03-09T14:28:34.315866+0000","last_active":"2024-03-09T15:31:35.984747+0000","last_peered":"2024-03-09T15:31:35.984747+0000","last_clean":"2024-03-09T15:31:35.984747+0000","last_became_active":"2024-03-09T14:28:34.315451+0000","last_became_peered":"2024-03-09T14:28:34.315451+0000","last_unstale":"2024-03-09T15:31:35.984747+0000","last_undegraded":"2024-03-09T15:31:35.984747+0000","last_fullsized":"2024-03-09T15:31:35.984747+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:50:21.781962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":374,"num_read_kb":293,"num_write":145,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"261'131","reported_seq":697,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.181679+0000","last_change":"2024-03-09T14:28:32.305393+0000","last_active":"2024-03-09T15:36:06.181679+0000","last_peered":"2024-03-09T15:36:06.181679+0000","last_clean":"2024-03-09T15:36:06.181679+0000","last_became_active":"2024-03-09T14:28:32.304540+0000","last_became_peered":"2024-03-09T14:28:32.304540+0000","last_unstale":"2024-03-09T15:36:06.181679+0000","last_undegraded":"2024-03-09T15:36:06.181679+0000","last_fullsized":"2024-03-09T15:36:06.181679+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":100,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:12:43.181100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1656073,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":187,"num_read_kb":1806,"num_write":496,"num_write_kb":18066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2e","version":"261'39","reported_seq":599,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.165250+0000","last_change":"2024-03-09T14:28:34.314876+0000","last_active":"2024-03-09T15:31:36.165250+0000","last_peered":"2024-03-09T15:31:36.165250+0000","last_clean":"2024-03-09T15:31:36.165250+0000","last_became_active":"2024-03-09T14:28:34.312399+0000","last_became_peered":"2024-03-09T14:28:34.312399+0000","last_unstale":"2024-03-09T15:31:36.165250+0000","last_undegraded":"2024-03-09T15:31:36.165250+0000","last_fullsized":"2024-03-09T15:31:36.165250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:27:43.779494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":182,"num_read_kb":156,"num_write":57,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"261'124","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.807282+0000","last_change":"2024-03-09T14:28:32.314908+0000","last_active":"2024-03-09T15:33:50.807282+0000","last_peered":"2024-03-09T15:33:50.807282+0000","last_clean":"2024-03-09T15:33:50.807282+0000","last_became_active":"2024-03-09T14:28:32.314596+0000","last_became_peered":"2024-03-09T14:28:32.314596+0000","last_unstale":"2024-03-09T15:33:50.807282+0000","last_undegraded":"2024-03-09T15:33:50.807282+0000","last_fullsized":"2024-03-09T15:33:50.807282+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:22:39.392181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506330,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":127,"num_read_kb":12793,"num_write":440,"num_write_kb":35350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"261'64","reported_seq":625,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:56.824631+0000","last_change":"2024-03-09T14:28:34.325871+0000","last_active":"2024-03-09T15:34:56.824631+0000","last_peered":"2024-03-09T15:34:56.824631+0000","last_clean":"2024-03-09T15:34:56.824631+0000","last_became_active":"2024-03-09T14:28:34.325461+0000","last_became_peered":"2024-03-09T14:28:34.325461+0000","last_unstale":"2024-03-09T15:34:56.824631+0000","last_undegraded":"2024-03-09T15:34:56.824631+0000","last_fullsized":"2024-03-09T15:34:56.824631+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:02:18.977660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":264,"num_read_kb":214,"num_write":89,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"261'141","reported_seq":717,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.191880+0000","last_change":"2024-03-09T14:28:32.313085+0000","last_active":"2024-03-09T15:36:06.191880+0000","last_peered":"2024-03-09T15:36:06.191880+0000","last_clean":"2024-03-09T15:36:06.191880+0000","last_became_active":"2024-03-09T14:28:32.311750+0000","last_became_peered":"2024-03-09T14:28:32.311750+0000","last_unstale":"2024-03-09T15:36:06.191880+0000","last_undegraded":"2024-03-09T15:36:06.191880+0000","last_fullsized":"2024-03-09T15:36:06.191880+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":100,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:56:43.188934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":195,"num_read_kb":14896,"num_write":515,"num_write_kb":41774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"261'162","reported_seq":774,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195666+0000","last_change":"2024-03-09T14:28:34.313344+0000","last_active":"2024-03-09T15:33:51.195666+0000","last_peered":"2024-03-09T15:33:51.195666+0000","last_clean":"2024-03-09T15:33:51.195666+0000","last_became_active":"2024-03-09T14:28:34.312212+0000","last_became_peered":"2024-03-09T14:28:34.312212+0000","last_unstale":"2024-03-09T15:33:51.195666+0000","last_undegraded":"2024-03-09T15:33:51.195666+0000","last_fullsized":"2024-03-09T15:33:51.195666+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":100,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:46:46.876109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":664,"num_read_kb":532,"num_write":243,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2b","version":"261'133","reported_seq":705,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.985790+0000","last_change":"2024-03-09T14:28:32.309913+0000","last_active":"2024-03-09T15:36:05.985790+0000","last_peered":"2024-03-09T15:36:05.985790+0000","last_clean":"2024-03-09T15:36:05.985790+0000","last_became_active":"2024-03-09T14:28:32.309562+0000","last_became_peered":"2024-03-09T14:28:32.309562+0000","last_unstale":"2024-03-09T15:36:05.985790+0000","last_undegraded":"2024-03-09T15:36:05.985790+0000","last_fullsized":"2024-03-09T15:36:05.985790+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:47:28.794070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":227,"num_read_kb":5348,"num_write":554,"num_write_kb":20887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"261'117","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.208966+0000","last_change":"2024-03-09T14:28:34.322800+0000","last_active":"2024-03-09T15:33:51.208966+0000","last_peered":"2024-03-09T15:33:51.208966+0000","last_clean":"2024-03-09T15:33:51.208966+0000","last_became_active":"2024-03-09T14:28:34.321424+0000","last_became_peered":"2024-03-09T14:28:34.321424+0000","last_unstale":"2024-03-09T15:33:51.208966+0000","last_undegraded":"2024-03-09T15:33:51.208966+0000","last_fullsized":"2024-03-09T15:33:51.208966+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:39:19.495854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":513,"num_read_kb":409,"num_write":169,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"261'142","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:09.453343+0000","last_change":"2024-03-09T14:28:32.334610+0000","last_active":"2024-03-09T15:34:09.453343+0000","last_peered":"2024-03-09T15:34:09.453343+0000","last_clean":"2024-03-09T15:34:09.453343+0000","last_became_active":"2024-03-09T14:28:32.333838+0000","last_became_peered":"2024-03-09T14:28:32.333838+0000","last_unstale":"2024-03-09T15:34:09.453343+0000","last_undegraded":"2024-03-09T15:34:09.453343+0000","last_fullsized":"2024-03-09T15:34:09.453343+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":100,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:14:50.303305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":218,"num_read_kb":3315,"num_write":653,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2a","version":"261'71","reported_seq":635,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.016279+0000","last_change":"2024-03-09T14:28:34.297744+0000","last_active":"2024-03-09T15:33:51.016279+0000","last_peered":"2024-03-09T15:33:51.016279+0000","last_clean":"2024-03-09T15:33:51.016279+0000","last_became_active":"2024-03-09T14:28:34.297042+0000","last_became_peered":"2024-03-09T14:28:34.297042+0000","last_unstale":"2024-03-09T15:33:51.016279+0000","last_undegraded":"2024-03-09T15:33:51.016279+0000","last_fullsized":"2024-03-09T15:33:51.016279+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:17:51.890278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":242,"num_write":99,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"261'111","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.167856+0000","last_change":"2024-03-09T14:28:32.320093+0000","last_active":"2024-03-09T15:36:06.167856+0000","last_peered":"2024-03-09T15:36:06.167856+0000","last_clean":"2024-03-09T15:36:06.167856+0000","last_became_active":"2024-03-09T14:28:32.317797+0000","last_became_peered":"2024-03-09T14:28:32.317797+0000","last_unstale":"2024-03-09T15:36:06.167856+0000","last_undegraded":"2024-03-09T15:36:06.167856+0000","last_fullsized":"2024-03-09T15:36:06.167856+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:52:33.454224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312437,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":176,"num_read_kb":4139,"num_write":464,"num_write_kb":19145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"261'113","reported_seq":668,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.807317+0000","last_change":"2024-03-09T14:28:34.314621+0000","last_active":"2024-03-09T15:33:50.807317+0000","last_peered":"2024-03-09T15:33:50.807317+0000","last_clean":"2024-03-09T15:33:50.807317+0000","last_became_active":"2024-03-09T14:28:34.314073+0000","last_became_peered":"2024-03-09T14:28:34.314073+0000","last_unstale":"2024-03-09T15:33:50.807317+0000","last_undegraded":"2024-03-09T15:33:50.807317+0000","last_fullsized":"2024-03-09T15:33:50.807317+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":100,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:30:53.101187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":445,"num_read_kb":351,"num_write":167,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"261'117","reported_seq":693,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.959588+0000","last_change":"2024-03-09T14:28:32.334618+0000","last_active":"2024-03-09T15:36:05.959588+0000","last_peered":"2024-03-09T15:36:05.959588+0000","last_clean":"2024-03-09T15:36:05.959588+0000","last_became_active":"2024-03-09T14:28:32.334187+0000","last_became_peered":"2024-03-09T14:28:32.334187+0000","last_unstale":"2024-03-09T15:36:05.959588+0000","last_undegraded":"2024-03-09T15:36:05.959588+0000","last_fullsized":"2024-03-09T15:36:05.959588+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:12:37.415618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2181393,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":207,"num_read_kb":1842,"num_write":544,"num_write_kb":7884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"261'4028","reported_seq":4579,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:46.207813+0000","last_change":"2024-03-09T14:28:34.324195+0000","last_active":"2024-03-09T15:33:46.207813+0000","last_peered":"2024-03-09T15:33:46.207813+0000","last_clean":"2024-03-09T15:33:46.207813+0000","last_became_active":"2024-03-09T14:28:34.322421+0000","last_became_peered":"2024-03-09T14:28:34.322421+0000","last_unstale":"2024-03-09T15:33:46.207813+0000","last_undegraded":"2024-03-09T15:33:46.207813+0000","last_fullsized":"2024-03-09T15:33:46.207813+0000","mapping_epoch":22,"log_start":"261'4000","ondisk_log_start":"261'4000","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:55:11.911331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14144,"num_read_kb":11461,"num_write":6038,"num_write_kb":3015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"261'118","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.139262+0000","last_change":"2024-03-09T14:28:32.324991+0000","last_active":"2024-03-09T15:36:06.139262+0000","last_peered":"2024-03-09T15:36:06.139262+0000","last_clean":"2024-03-09T15:36:06.139262+0000","last_became_active":"2024-03-09T14:28:32.324166+0000","last_became_peered":"2024-03-09T14:28:32.324166+0000","last_unstale":"2024-03-09T15:36:06.139262+0000","last_undegraded":"2024-03-09T15:36:06.139262+0000","last_fullsized":"2024-03-09T15:36:06.139262+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:55:14.982039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272031,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":186,"num_read_kb":9428,"num_write":503,"num_write_kb":30713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"261'126","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.159876+0000","last_change":"2024-03-09T14:28:32.288697+0000","last_active":"2024-03-09T15:36:06.159876+0000","last_peered":"2024-03-09T15:36:06.159876+0000","last_clean":"2024-03-09T15:36:06.159876+0000","last_became_active":"2024-03-09T14:28:32.287937+0000","last_became_peered":"2024-03-09T14:28:32.287937+0000","last_unstale":"2024-03-09T15:36:06.159876+0000","last_undegraded":"2024-03-09T15:36:06.159876+0000","last_fullsized":"2024-03-09T15:36:06.159876+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":100,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:01:00.707624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243238,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":199,"num_read_kb":48289,"num_write":444,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"261'161","reported_seq":726,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.985994+0000","last_change":"2024-03-09T14:28:34.317008+0000","last_active":"2024-03-09T15:31:40.985994+0000","last_peered":"2024-03-09T15:31:40.985994+0000","last_clean":"2024-03-09T15:31:40.985994+0000","last_became_active":"2024-03-09T14:28:34.316689+0000","last_became_peered":"2024-03-09T14:28:34.316689+0000","last_unstale":"2024-03-09T15:31:40.985994+0000","last_undegraded":"2024-03-09T15:31:40.985994+0000","last_fullsized":"2024-03-09T15:31:40.985994+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:22:45.346641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":634,"num_read_kb":490,"num_write":268,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"261'144","reported_seq":722,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.889143+0000","last_change":"2024-03-09T14:28:32.313446+0000","last_active":"2024-03-09T15:36:05.889143+0000","last_peered":"2024-03-09T15:36:05.889143+0000","last_clean":"2024-03-09T15:36:05.889143+0000","last_became_active":"2024-03-09T14:28:32.313132+0000","last_became_peered":"2024-03-09T14:28:32.313132+0000","last_unstale":"2024-03-09T15:36:05.889143+0000","last_undegraded":"2024-03-09T15:36:05.889143+0000","last_fullsized":"2024-03-09T15:36:05.889143+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:30:18.199314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262382,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":242,"num_read_kb":5401,"num_write":622,"num_write_kb":32648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"261'41","reported_seq":586,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:02.721943+0000","last_change":"2024-03-09T14:28:34.304440+0000","last_active":"2024-03-09T15:34:02.721943+0000","last_peered":"2024-03-09T15:34:02.721943+0000","last_clean":"2024-03-09T15:34:02.721943+0000","last_became_active":"2024-03-09T14:28:34.303701+0000","last_became_peered":"2024-03-09T14:28:34.303701+0000","last_unstale":"2024-03-09T15:34:02.721943+0000","last_undegraded":"2024-03-09T15:34:02.721943+0000","last_fullsized":"2024-03-09T15:34:02.721943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:22:10.704885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":166,"num_read_kb":138,"num_write":64,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":65,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3686,"num_read_kb":2614,"num_write":1236,"num_write_kb":514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":13176,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":8518,"num_read_kb":8294,"num_write":8580,"num_write_kb":2546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":26352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2011,"ondisk_log_size":2011,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":578,"ondisk_log_size":578,"up":64,"acting":64,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":725340,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":11980,"num_read_kb":20116,"num_write":8932,"num_write_kb":5434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":743659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2014,"ondisk_log_size":2014,"up":64,"acting":64,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1719,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1602,"num_read_kb":1903,"num_write":22,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":38590,"num_read_kb":31236,"num_write":15314,"num_write_kb":7493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3941,"ondisk_log_size":3941,"up":128,"acting":128,"num_store_stats":8},{"poolid":68,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":210202860,"num_objects":1447,"num_object_clones":0,"num_object_copies":2894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1447,"num_whiteouts":0,"num_read":12930,"num_read_kb":1121153,"num_write":33168,"num_write_kb":1701153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440426496,"data_stored":430130434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2921,"ondisk_log_size":2921,"up":128,"acting":128,"num_store_stats":8},{"poolid":67,"num_pg":32,"stat_sum":{"num_bytes":81879532,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":124,"num_read_kb":192739,"num_write":140,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":160620544,"data_stored":160613336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":66,"num_pg":32,"stat_sum":{"num_bytes":82877649,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":122,"num_read_kb":192649,"num_write":140,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":170004480,"data_stored":170000061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575670,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334148,"kb_used_data":104444,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":94037692,"statfs":{"total":96636764160,"available":96294596608,"internally_reserved":0,"allocated":106950656,"data_stored":103672380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":235207114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575666,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292416,"kb_used_data":84920,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":94079424,"statfs":{"total":96636764160,"available":96337330176,"internally_reserved":0,"allocated":86958080,"data_stored":84225759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575667,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":174444,"kb_used_data":121060,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94197396,"statfs":{"total":96636764160,"available":96458133504,"internally_reserved":0,"allocated":123965440,"data_stored":120640043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575671,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":197704,"kb_used_data":140160,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94174136,"statfs":{"total":96636764160,"available":96434315264,"internally_reserved":0,"allocated":143523840,"data_stored":140316730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575666,"num_pgs":118,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":343308,"kb_used_data":135812,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":94028532,"statfs":{"total":96636764160,"available":96285216768,"internally_reserved":0,"allocated":139071488,"data_stored":135606966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":212466119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575668,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133712,"kb_used_data":80328,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94238128,"statfs":{"total":96636764160,"available":96499843072,"internally_reserved":0,"allocated":82255872,"data_stored":79602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575667,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":186168,"kb_used_data":100912,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94185672,"statfs":{"total":96636764160,"available":96446128128,"internally_reserved":0,"allocated":103333888,"data_stored":100615986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575669,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339852,"kb_used_data":137924,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":94031988,"statfs":{"total":96636764160,"available":96288755712,"internally_reserved":0,"allocated":141234176,"data_stored":137929952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73302016,"data_stored":71838616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41046016,"data_stored":40051454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":44478464,"data_stored":43472543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":64929792,"data_stored":63014498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":77041664,"data_stored":75526368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61636608,"data_stored":60503814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":33976320,"data_stored":32726952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":44015616,"data_stored":42996189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":92713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":120489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":29353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":164295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":29815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":115473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":83356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":108165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":3901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":2727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24158208,"data_stored":24157393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25219072,"data_stored":25216283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13631488,"data_stored":13631488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30449664,"data_stored":30448849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36700160,"data_stored":36700160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":25165824,"data_stored":25165824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":14680064,"data_stored":14680064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20017152,"data_stored":20013548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24117248,"data_stored":24117248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19922944,"data_stored":19922944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39845888,"data_stored":39845888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5337088,"data_stored":5333484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":22020096,"data_stored":22020096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24117248,"data_stored":24117248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T15:36:08.899 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T15:36:09.243 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:36:09.243 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":261,"fsid":"fe441b42-34a9-46d6-87e9-ac75455181ab","created":"2024-03-09T14:27:59.514149+0000","modified":"2024-03-09T15:09:51.034744+0000","last_up_change":"2024-03-09T14:28:10.177363+0000","last_in_change":"2024-03-09T14:28:01.753379+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":68,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-09T14:28:10.677796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-09T14:28:17.557699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2024-03-09T14:28:31.063651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2024-03-09T14:28:32.629693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":".rgw.root","create_time":"2024-03-09T14:28:34.403265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2024-03-09T14:28:36.631698+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2024-03-09T14:28:38.353176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2024-03-09T14:28:40.508881+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":65,"pool_name":"default.rgw.buckets.non-ec","create_time":"2024-03-09T14:39:11.300869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":66,"pool_name":"default.rgw.buckets.data.frozen","create_time":"2024-03-09T14:46:17.416610+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":67,"pool_name":"default.rgw.buckets.data.lukewarm","create_time":"2024-03-09T14:46:24.915050+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":68,"pool_name":"default.rgw.otp","create_time":"2024-03-09T15:09:48.710446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"261","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94cb6315-8a11-4b66-8842-2d1925ae9e28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":259,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6809","nonce":3097686011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6813","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6815","nonce":3097686011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6823","nonce":3097686011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":3097686011},{"type":"v1","addr":"172.21.15.1:6819","nonce":3097686011}]},"public_addr":"172.21.15.1:6809/3097686011","cluster_addr":"172.21.15.1:6815/3097686011","heartbeat_back_addr":"172.21.15.1:6823/3097686011","heartbeat_front_addr":"172.21.15.1:6819/3097686011","state":["exists","up"]},{"osd":1,"uuid":"c5007a43-9e9d-4bc4-bc3f-d8996bf2d160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":259,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6805","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6808","nonce":3585289272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6814","nonce":3585289272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6822","nonce":3585289272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":3585289272},{"type":"v1","addr":"172.21.15.1:6818","nonce":3585289272}]},"public_addr":"172.21.15.1:6808/3585289272","cluster_addr":"172.21.15.1:6814/3585289272","heartbeat_back_addr":"172.21.15.1:6822/3585289272","heartbeat_front_addr":"172.21.15.1:6818/3585289272","state":["exists","up"]},{"osd":2,"uuid":"8ef94bd2-a269-44a0-8870-eecdf5a35a1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":259,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6825","nonce":1666056338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6827","nonce":1666056338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6831","nonce":1666056338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1666056338},{"type":"v1","addr":"172.21.15.1:6829","nonce":1666056338}]},"public_addr":"172.21.15.1:6825/1666056338","cluster_addr":"172.21.15.1:6827/1666056338","heartbeat_back_addr":"172.21.15.1:6831/1666056338","heartbeat_front_addr":"172.21.15.1:6829/1666056338","state":["exists","up"]},{"osd":3,"uuid":"878834c0-0f19-4e43-bea6-4e8b4f972c67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":259,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6801","nonce":4091251076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6803","nonce":4091251076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6812","nonce":4091251076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":4091251076},{"type":"v1","addr":"172.21.15.1:6807","nonce":4091251076}]},"public_addr":"172.21.15.1:6801/4091251076","cluster_addr":"172.21.15.1:6803/4091251076","heartbeat_back_addr":"172.21.15.1:6812/4091251076","heartbeat_front_addr":"172.21.15.1:6807/4091251076","state":["exists","up"]},{"osd":4,"uuid":"787d3950-2e1f-4714-af31-12d811b3b8c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":259,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6817","nonce":859385596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6819","nonce":859385596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6823","nonce":859385596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":859385596},{"type":"v1","addr":"172.21.15.53:6821","nonce":859385596}]},"public_addr":"172.21.15.53:6817/859385596","cluster_addr":"172.21.15.53:6819/859385596","heartbeat_back_addr":"172.21.15.53:6823/859385596","heartbeat_front_addr":"172.21.15.53:6821/859385596","state":["exists","up"]},{"osd":5,"uuid":"8cb1f2fa-3eec-4750-a028-b8b3c07846d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":259,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6825","nonce":256097109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6827","nonce":256097109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6831","nonce":256097109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":256097109},{"type":"v1","addr":"172.21.15.53:6829","nonce":256097109}]},"public_addr":"172.21.15.53:6825/256097109","cluster_addr":"172.21.15.53:6827/256097109","heartbeat_back_addr":"172.21.15.53:6831/256097109","heartbeat_front_addr":"172.21.15.53:6829/256097109","state":["exists","up"]},{"osd":6,"uuid":"094803a5-1cb6-452c-abe9-890105c65516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":259,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6805","nonce":1545189291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6807","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6809","nonce":1545189291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6815","nonce":1545189291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":1545189291},{"type":"v1","addr":"172.21.15.53:6813","nonce":1545189291}]},"public_addr":"172.21.15.53:6805/1545189291","cluster_addr":"172.21.15.53:6809/1545189291","heartbeat_back_addr":"172.21.15.53:6815/1545189291","heartbeat_front_addr":"172.21.15.53:6813/1545189291","state":["exists","up"]},{"osd":7,"uuid":"6bbf6c2e-f1fe-4644-918d-53a25c557cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":259,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6801","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6802","nonce":315236776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6803","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6804","nonce":315236776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6812","nonce":315236776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":315236776},{"type":"v1","addr":"172.21.15.53:6808","nonce":315236776}]},"public_addr":"172.21.15.53:6802/315236776","cluster_addr":"172.21.15.53:6804/315236776","heartbeat_back_addr":"172.21.15.53:6812/315236776","heartbeat_front_addr":"172.21.15.53:6808/315236776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.295939+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.158214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.073987+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.239564+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.283109+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.389164+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.144938+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T14:28:07.031541+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":3,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"67.1e","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":26,"snaps":[{"begin":1,"length":2}]},{"pool":31,"snaps":[{"begin":1,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T15:36:10.262 INFO:tasks.ceph:Scrubbing osd.0 2024-03-09T15:36:10.263 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-09T15:36:10.427 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:36:10.427 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-09T15:36:10.427 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:36:10.443 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-09T15:36:10.736 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 0 to deep-scrub 2024-03-09T15:36:10.757 INFO:tasks.ceph:Scrubbing osd.1 2024-03-09T15:36:10.757 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-09T15:36:10.913 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:36:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-09T15:36:10.913 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:36:10.929 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-09T15:36:11.221 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 1 to deep-scrub 2024-03-09T15:36:11.241 INFO:tasks.ceph:Scrubbing osd.2 2024-03-09T15:36:11.242 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-09T15:36:11.397 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:36:11.397 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-09T15:36:11.397 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:36:11.415 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-09T15:36:11.707 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 2 to deep-scrub 2024-03-09T15:36:11.725 INFO:tasks.ceph:Scrubbing osd.3 2024-03-09T15:36:11.726 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-09T15:36:11.875 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:36:11.875 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-09T15:36:11.875 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:36:11.892 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-09T15:36:12.184 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 3 to deep-scrub 2024-03-09T15:36:12.204 INFO:tasks.ceph:Scrubbing osd.4 2024-03-09T15:36:12.204 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-09T15:36:12.351 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:36:12.351 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-09T15:36:12.351 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:36:12.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-09T15:36:12.647 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 4 to deep-scrub 2024-03-09T15:36:12.665 INFO:tasks.ceph:Scrubbing osd.5 2024-03-09T15:36:12.665 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-09T15:36:12.821 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:36:12.821 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-09T15:36:12.821 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:36:12.838 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-09T15:36:13.122 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 5 to deep-scrub 2024-03-09T15:36:13.140 INFO:tasks.ceph:Scrubbing osd.6 2024-03-09T15:36:13.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-09T15:36:13.299 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:36:13.299 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-09T15:36:13.300 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:36:13.316 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-09T15:36:13.611 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 6 to deep-scrub 2024-03-09T15:36:13.632 INFO:tasks.ceph:Scrubbing osd.7 2024-03-09T15:36:13.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-09T15:36:13.794 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-09T15:36:13.794 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-09T15:36:13.794 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-09T15:36:13.810 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-09T15:36:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 7 to deep-scrub 2024-03-09T15:36:14.120 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T15:36:14.484 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:36:14.499 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-09T15:36:14.513 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":2294,"stamp":"2024-03-09T15:36:13.253453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":169838241,"num_objects":629,"num_object_clones":0,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":0,"num_read":79040,"num_read_kb":1571790,"num_write":68959,"num_write_kb":1988151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12495,"ondisk_log_size":12495,"up":786,"acting":786,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":786,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1477184,"kb_used_data":360440,"kb_used_omap":12,"kb_used_meta":1115507,"kb_avail":753497536,"statfs":{"total":773094113280,"available":771581476864,"internally_reserved":0,"allocated":369090560,"data_stored":355965182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1142279728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-273322467,"num_objects":-1458,"num_object_clones":0,"num_object_copies":-2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1458,"num_whiteouts":0,"num_read":1759,"num_read_kb":1403,"num_write":1586,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008746"},"pg_stats":[{"pgid":"68.1a","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.178131+0000","last_change":"2024-03-09T15:09:50.085348+0000","last_active":"2024-03-09T15:32:21.178131+0000","last_peered":"2024-03-09T15:32:21.178131+0000","last_clean":"2024-03-09T15:32:21.178131+0000","last_became_active":"2024-03-09T15:09:50.084784+0000","last_became_peered":"2024-03-09T15:09:50.084784+0000","last_unstale":"2024-03-09T15:32:21.178131+0000","last_undegraded":"2024-03-09T15:32:21.178131+0000","last_fullsized":"2024-03-09T15:32:21.178131+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:18:17.114617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.1d","version":"258'7","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.167423+0000","last_change":"2024-03-09T14:46:26.826294+0000","last_active":"2024-03-09T15:31:31.167423+0000","last_peered":"2024-03-09T15:31:31.167423+0000","last_clean":"2024-03-09T15:31:31.167423+0000","last_became_active":"2024-03-09T14:46:26.825404+0000","last_became_peered":"2024-03-09T14:46:26.825404+0000","last_unstale":"2024-03-09T15:31:31.167423+0000","last_undegraded":"2024-03-09T15:31:31.167423+0000","last_fullsized":"2024-03-09T15:31:31.167423+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:28:03.324789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20498,"num_write":6,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1c","version":"261'2","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.807445+0000","last_change":"2024-03-09T14:46:18.769137+0000","last_active":"2024-03-09T15:33:50.807445+0000","last_peered":"2024-03-09T15:33:50.807445+0000","last_clean":"2024-03-09T15:33:50.807445+0000","last_became_active":"2024-03-09T14:46:18.768383+0000","last_became_peered":"2024-03-09T14:46:18.768383+0000","last_unstale":"2024-03-09T15:33:50.807445+0000","last_undegraded":"2024-03-09T15:33:50.807445+0000","last_fullsized":"2024-03-09T15:33:50.807445+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:38:13.467012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.1f","version":"258'45","reported_seq":168,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:31.040009+0000","last_change":"2024-03-09T14:39:13.282130+0000","last_active":"2024-03-09T15:35:31.040009+0000","last_peered":"2024-03-09T15:35:31.040009+0000","last_clean":"2024-03-09T15:35:31.040009+0000","last_became_active":"2024-03-09T14:39:13.281020+0000","last_became_peered":"2024-03-09T14:39:13.281020+0000","last_unstale":"2024-03-09T15:35:31.040009+0000","last_undegraded":"2024-03-09T15:35:31.040009+0000","last_fullsized":"2024-03-09T15:35:31.040009+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":134,"num_write":70,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1b","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.961650+0000","last_change":"2024-03-09T15:09:50.083262+0000","last_active":"2024-03-09T15:32:20.961650+0000","last_peered":"2024-03-09T15:32:20.961650+0000","last_clean":"2024-03-09T15:32:20.961650+0000","last_became_active":"2024-03-09T15:09:50.082690+0000","last_became_peered":"2024-03-09T15:09:50.082690+0000","last_unstale":"2024-03-09T15:32:20.961650+0000","last_undegraded":"2024-03-09T15:32:20.961650+0000","last_fullsized":"2024-03-09T15:32:20.961650+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:18:09.424746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1c","version":"258'5","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162661+0000","last_change":"2024-03-09T14:46:26.807670+0000","last_active":"2024-03-09T15:31:26.162661+0000","last_peered":"2024-03-09T15:31:26.162661+0000","last_clean":"2024-03-09T15:31:26.162661+0000","last_became_active":"2024-03-09T14:46:26.806865+0000","last_became_peered":"2024-03-09T14:46:26.806865+0000","last_unstale":"2024-03-09T15:31:26.162661+0000","last_undegraded":"2024-03-09T15:31:26.162661+0000","last_fullsized":"2024-03-09T15:31:26.162661+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:05:04.558231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.1d","version":"261'2","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.966745+0000","last_change":"2024-03-09T14:46:18.780095+0000","last_active":"2024-03-09T15:33:50.966745+0000","last_peered":"2024-03-09T15:33:50.966745+0000","last_clean":"2024-03-09T15:33:50.966745+0000","last_became_active":"2024-03-09T14:46:18.779760+0000","last_became_peered":"2024-03-09T14:46:18.779760+0000","last_unstale":"2024-03-09T15:33:50.966745+0000","last_undegraded":"2024-03-09T15:33:50.966745+0000","last_fullsized":"2024-03-09T15:33:50.966745+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:12.211222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1e","version":"0'0","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:30.975652+0000","last_change":"2024-03-09T14:39:13.260541+0000","last_active":"2024-03-09T15:33:30.975652+0000","last_peered":"2024-03-09T15:33:30.975652+0000","last_clean":"2024-03-09T15:33:30.975652+0000","last_became_active":"2024-03-09T14:39:13.259731+0000","last_became_peered":"2024-03-09T14:39:13.259731+0000","last_unstale":"2024-03-09T15:33:30.975652+0000","last_undegraded":"2024-03-09T15:33:30.975652+0000","last_fullsized":"2024-03-09T15:33:30.975652+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:09:28.555448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.18","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.764910+0000","last_change":"2024-03-09T15:09:50.081224+0000","last_active":"2024-03-09T15:31:35.764910+0000","last_peered":"2024-03-09T15:31:35.764910+0000","last_clean":"2024-03-09T15:31:35.764910+0000","last_became_active":"2024-03-09T15:09:50.080578+0000","last_became_peered":"2024-03-09T15:09:50.080578+0000","last_unstale":"2024-03-09T15:31:35.764910+0000","last_undegraded":"2024-03-09T15:31:35.764910+0000","last_fullsized":"2024-03-09T15:31:35.764910+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:54:04.030480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.1f","version":"261'5","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.156260+0000","last_change":"2024-03-09T14:46:26.896619+0000","last_active":"2024-03-09T15:36:06.156260+0000","last_peered":"2024-03-09T15:36:06.156260+0000","last_clean":"2024-03-09T15:36:06.156260+0000","last_became_active":"2024-03-09T14:46:26.895608+0000","last_became_peered":"2024-03-09T14:46:26.895608+0000","last_unstale":"2024-03-09T15:36:06.156260+0000","last_undegraded":"2024-03-09T15:36:06.156260+0000","last_fullsized":"2024-03-09T15:36:06.156260+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":12288,"num_write":5,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1e","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:35.964975+0000","last_change":"2024-03-09T14:46:18.782402+0000","last_active":"2024-03-09T15:32:35.964975+0000","last_peered":"2024-03-09T15:32:35.964975+0000","last_clean":"2024-03-09T15:32:35.964975+0000","last_became_active":"2024-03-09T14:46:18.781856+0000","last_became_peered":"2024-03-09T14:46:18.781856+0000","last_unstale":"2024-03-09T15:32:35.964975+0000","last_undegraded":"2024-03-09T15:32:35.964975+0000","last_fullsized":"2024-03-09T15:32:35.964975+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:29:15.829032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.1d","version":"258'35","reported_seq":143,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:16.022428+0000","last_change":"2024-03-09T14:39:13.265101+0000","last_active":"2024-03-09T15:34:16.022428+0000","last_peered":"2024-03-09T15:34:16.022428+0000","last_clean":"2024-03-09T15:34:16.022428+0000","last_became_active":"2024-03-09T14:39:13.258351+0000","last_became_peered":"2024-03-09T14:39:13.258351+0000","last_unstale":"2024-03-09T15:34:16.022428+0000","last_undegraded":"2024-03-09T15:34:16.022428+0000","last_fullsized":"2024-03-09T15:34:16.022428+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":152,"num_read_kb":109,"num_write":57,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.19","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.777876+0000","last_change":"2024-03-09T15:09:50.083658+0000","last_active":"2024-03-09T15:31:40.777876+0000","last_peered":"2024-03-09T15:31:40.777876+0000","last_clean":"2024-03-09T15:31:40.777876+0000","last_became_active":"2024-03-09T15:09:50.081680+0000","last_became_peered":"2024-03-09T15:09:50.081680+0000","last_unstale":"2024-03-09T15:31:40.777876+0000","last_undegraded":"2024-03-09T15:31:40.777876+0000","last_fullsized":"2024-03-09T15:31:40.777876+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:56:53.220664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.1e","version":"261'6","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.485897+0000","last_change":"2024-03-09T14:47:07.798858+0000","last_active":"2024-03-09T15:36:06.485897+0000","last_peered":"2024-03-09T15:36:06.485897+0000","last_clean":"2024-03-09T15:36:06.485897+0000","last_became_active":"2024-03-09T14:47:07.798247+0000","last_became_peered":"2024-03-09T14:47:07.798247+0000","last_unstale":"2024-03-09T15:36:06.485897+0000","last_undegraded":"2024-03-09T15:36:06.485897+0000","last_fullsized":"2024-03-09T15:36:06.485897+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":16384,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1f","version":"258'5","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.173075+0000","last_change":"2024-03-09T14:46:18.772485+0000","last_active":"2024-03-09T15:32:11.173075+0000","last_peered":"2024-03-09T15:32:11.173075+0000","last_clean":"2024-03-09T15:32:11.173075+0000","last_became_active":"2024-03-09T14:46:18.771731+0000","last_became_peered":"2024-03-09T14:46:18.771731+0000","last_unstale":"2024-03-09T15:32:11.173075+0000","last_undegraded":"2024-03-09T15:32:11.173075+0000","last_fullsized":"2024-03-09T15:32:11.173075+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:50:44.589948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5136,"num_write":5,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1c","version":"261'13","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.779361+0000","last_change":"2024-03-09T14:39:13.266571+0000","last_active":"2024-03-09T15:31:45.779361+0000","last_peered":"2024-03-09T15:31:45.779361+0000","last_clean":"2024-03-09T15:31:45.779361+0000","last_became_active":"2024-03-09T14:39:13.260945+0000","last_became_peered":"2024-03-09T14:39:13.260945+0000","last_unstale":"2024-03-09T15:31:45.779361+0000","last_undegraded":"2024-03-09T15:31:45.779361+0000","last_fullsized":"2024-03-09T15:31:45.779361+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:58:40.520455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":27,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1e","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.808730+0000","last_change":"2024-03-09T15:09:50.088230+0000","last_active":"2024-03-09T15:33:55.808730+0000","last_peered":"2024-03-09T15:33:55.808730+0000","last_clean":"2024-03-09T15:33:55.808730+0000","last_became_active":"2024-03-09T15:09:50.086789+0000","last_became_peered":"2024-03-09T15:09:50.086789+0000","last_unstale":"2024-03-09T15:33:55.808730+0000","last_undegraded":"2024-03-09T15:33:55.808730+0000","last_fullsized":"2024-03-09T15:33:55.808730+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:54:27.433652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.19","version":"261'6","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.109749+0000","last_change":"2024-03-09T14:46:26.819845+0000","last_active":"2024-03-09T15:36:06.109749+0000","last_peered":"2024-03-09T15:36:06.109749+0000","last_clean":"2024-03-09T15:36:06.109749+0000","last_became_active":"2024-03-09T14:46:26.819085+0000","last_became_peered":"2024-03-09T14:46:26.819085+0000","last_unstale":"2024-03-09T15:36:06.109749+0000","last_undegraded":"2024-03-09T15:36:06.109749+0000","last_fullsized":"2024-03-09T15:36:06.109749+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:17.867538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":12288,"num_write":11,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.18","version":"261'13","reported_seq":80,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.266742+0000","last_change":"2024-03-09T14:46:18.776960+0000","last_active":"2024-03-09T15:36:06.266742+0000","last_peered":"2024-03-09T15:36:06.266742+0000","last_clean":"2024-03-09T15:36:06.266742+0000","last_became_active":"2024-03-09T14:46:18.776161+0000","last_became_peered":"2024-03-09T14:46:18.776161+0000","last_unstale":"2024-03-09T15:36:06.266742+0000","last_undegraded":"2024-03-09T15:36:06.266742+0000","last_fullsized":"2024-03-09T15:36:06.266742+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":36889,"num_write":17,"num_write_kb":16407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.1b","version":"258'10","reported_seq":79,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.779245+0000","last_change":"2024-03-09T14:39:13.269298+0000","last_active":"2024-03-09T15:31:45.779245+0000","last_peered":"2024-03-09T15:31:45.779245+0000","last_clean":"2024-03-09T15:31:45.779245+0000","last_became_active":"2024-03-09T14:39:13.268179+0000","last_became_peered":"2024-03-09T14:39:13.268179+0000","last_unstale":"2024-03-09T15:31:45.779245+0000","last_undegraded":"2024-03-09T15:31:45.779245+0000","last_fullsized":"2024-03-09T15:31:45.779245+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:12:05.852927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1f","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:45.951375+0000","last_change":"2024-03-09T15:09:50.064176+0000","last_active":"2024-03-09T15:32:45.951375+0000","last_peered":"2024-03-09T15:32:45.951375+0000","last_clean":"2024-03-09T15:32:45.951375+0000","last_became_active":"2024-03-09T15:09:50.063705+0000","last_became_peered":"2024-03-09T15:09:50.063705+0000","last_unstale":"2024-03-09T15:32:45.951375+0000","last_undegraded":"2024-03-09T15:32:45.951375+0000","last_fullsized":"2024-03-09T15:32:45.951375+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:45.866476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.18","version":"261'3","reported_seq":47,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:50.805213+0000","last_change":"2024-03-09T14:46:26.824830+0000","last_active":"2024-03-09T15:34:50.805213+0000","last_peered":"2024-03-09T15:34:50.805213+0000","last_clean":"2024-03-09T15:34:50.805213+0000","last_became_active":"2024-03-09T14:46:26.824522+0000","last_became_peered":"2024-03-09T14:46:26.824522+0000","last_unstale":"2024-03-09T15:34:50.805213+0000","last_undegraded":"2024-03-09T15:34:50.805213+0000","last_fullsized":"2024-03-09T15:34:50.805213+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:09:09.414799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.19","version":"261'9","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.134448+0000","last_change":"2024-03-09T14:46:18.770577+0000","last_active":"2024-03-09T15:36:06.134448+0000","last_peered":"2024-03-09T15:36:06.134448+0000","last_clean":"2024-03-09T15:36:06.134448+0000","last_became_active":"2024-03-09T14:46:18.769730+0000","last_became_peered":"2024-03-09T14:46:18.769730+0000","last_unstale":"2024-03-09T15:36:06.134448+0000","last_undegraded":"2024-03-09T15:36:06.134448+0000","last_fullsized":"2024-03-09T15:36:06.134448+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:07:45.259457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":19456,"num_write":19,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1a","version":"261'44","reported_seq":164,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:45.767114+0000","last_change":"2024-03-09T14:39:13.274060+0000","last_active":"2024-03-09T15:31:45.767114+0000","last_peered":"2024-03-09T15:31:45.767114+0000","last_clean":"2024-03-09T15:31:45.767114+0000","last_became_active":"2024-03-09T14:39:13.273298+0000","last_became_peered":"2024-03-09T14:39:13.273298+0000","last_unstale":"2024-03-09T15:31:45.767114+0000","last_undegraded":"2024-03-09T15:31:45.767114+0000","last_fullsized":"2024-03-09T15:31:45.767114+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:16:28.025129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":138,"num_write":72,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.1c","version":"261'1","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188241+0000","last_change":"2024-03-09T15:09:50.075720+0000","last_active":"2024-03-09T15:32:21.188241+0000","last_peered":"2024-03-09T15:32:21.188241+0000","last_clean":"2024-03-09T15:32:21.188241+0000","last_became_active":"2024-03-09T15:09:50.074952+0000","last_became_peered":"2024-03-09T15:09:50.074952+0000","last_unstale":"2024-03-09T15:32:21.188241+0000","last_undegraded":"2024-03-09T15:32:21.188241+0000","last_fullsized":"2024-03-09T15:32:21.188241+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:21:35.439413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1b","version":"261'3","reported_seq":46,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:00.795005+0000","last_change":"2024-03-09T14:46:26.805382+0000","last_active":"2024-03-09T15:34:00.795005+0000","last_peered":"2024-03-09T15:34:00.795005+0000","last_clean":"2024-03-09T15:34:00.795005+0000","last_became_active":"2024-03-09T14:46:26.804622+0000","last_became_peered":"2024-03-09T14:46:26.804622+0000","last_unstale":"2024-03-09T15:34:00.795005+0000","last_undegraded":"2024-03-09T15:34:00.795005+0000","last_fullsized":"2024-03-09T15:34:00.795005+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:23.446296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.1a","version":"261'5","reported_seq":55,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195459+0000","last_change":"2024-03-09T14:46:18.785531+0000","last_active":"2024-03-09T15:33:51.195459+0000","last_peered":"2024-03-09T15:33:51.195459+0000","last_clean":"2024-03-09T15:33:51.195459+0000","last_became_active":"2024-03-09T14:46:18.784821+0000","last_became_peered":"2024-03-09T14:46:18.784821+0000","last_unstale":"2024-03-09T15:33:51.195459+0000","last_undegraded":"2024-03-09T15:33:51.195459+0000","last_fullsized":"2024-03-09T15:33:51.195459+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:10:34.919135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4096,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.19","version":"258'28","reported_seq":121,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:06.033339+0000","last_change":"2024-03-09T14:39:13.258289+0000","last_active":"2024-03-09T15:35:06.033339+0000","last_peered":"2024-03-09T15:35:06.033339+0000","last_clean":"2024-03-09T15:35:06.033339+0000","last_became_active":"2024-03-09T14:39:13.257473+0000","last_became_peered":"2024-03-09T14:39:13.257473+0000","last_unstale":"2024-03-09T15:35:06.033339+0000","last_undegraded":"2024-03-09T15:35:06.033339+0000","last_fullsized":"2024-03-09T15:35:06.033339+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":78,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1d","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188298+0000","last_change":"2024-03-09T15:09:50.075294+0000","last_active":"2024-03-09T15:32:21.188298+0000","last_peered":"2024-03-09T15:32:21.188298+0000","last_clean":"2024-03-09T15:32:21.188298+0000","last_became_active":"2024-03-09T15:09:50.074875+0000","last_became_peered":"2024-03-09T15:09:50.074875+0000","last_unstale":"2024-03-09T15:32:21.188298+0000","last_undegraded":"2024-03-09T15:32:21.188298+0000","last_fullsized":"2024-03-09T15:32:21.188298+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:24:33.386479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1a","version":"261'10","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:00.809350+0000","last_change":"2024-03-09T14:46:26.813798+0000","last_active":"2024-03-09T15:34:00.809350+0000","last_peered":"2024-03-09T15:34:00.809350+0000","last_clean":"2024-03-09T15:34:00.809350+0000","last_became_active":"2024-03-09T14:46:26.812889+0000","last_became_peered":"2024-03-09T14:46:26.812889+0000","last_unstale":"2024-03-09T15:34:00.809350+0000","last_undegraded":"2024-03-09T15:34:00.809350+0000","last_fullsized":"2024-03-09T15:34:00.809350+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:43:59.844987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":12290,"num_write":14,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.1b","version":"261'2","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.156218+0000","last_change":"2024-03-09T14:46:18.774605+0000","last_active":"2024-03-09T15:36:06.156218+0000","last_peered":"2024-03-09T15:36:06.156218+0000","last_clean":"2024-03-09T15:36:06.156218+0000","last_became_active":"2024-03-09T14:46:18.756394+0000","last_became_peered":"2024-03-09T14:46:18.756394+0000","last_unstale":"2024-03-09T15:36:06.156218+0000","last_undegraded":"2024-03-09T15:36:06.156218+0000","last_fullsized":"2024-03-09T15:36:06.156218+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":40,"num_write":2,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.18","version":"261'8","reported_seq":77,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.144778+0000","last_change":"2024-03-09T14:39:13.258846+0000","last_active":"2024-03-09T15:32:26.144778+0000","last_peered":"2024-03-09T15:32:26.144778+0000","last_clean":"2024-03-09T15:32:26.144778+0000","last_became_active":"2024-03-09T14:39:13.257996+0000","last_became_peered":"2024-03-09T14:39:13.257996+0000","last_unstale":"2024-03-09T15:32:26.144778+0000","last_undegraded":"2024-03-09T15:32:26.144778+0000","last_fullsized":"2024-03-09T15:32:26.144778+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:23:43.508747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.12","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.957209+0000","last_change":"2024-03-09T15:09:50.070950+0000","last_active":"2024-03-09T15:33:10.957209+0000","last_peered":"2024-03-09T15:33:10.957209+0000","last_clean":"2024-03-09T15:33:10.957209+0000","last_became_active":"2024-03-09T15:09:50.070345+0000","last_became_peered":"2024-03-09T15:09:50.070345+0000","last_unstale":"2024-03-09T15:33:10.957209+0000","last_undegraded":"2024-03-09T15:33:10.957209+0000","last_fullsized":"2024-03-09T15:33:10.957209+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:18:24.068533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.15","version":"0'0","reported_seq":42,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.788416+0000","last_change":"2024-03-09T14:46:26.891781+0000","last_active":"2024-03-09T15:32:25.788416+0000","last_peered":"2024-03-09T15:32:25.788416+0000","last_clean":"2024-03-09T15:32:25.788416+0000","last_became_active":"2024-03-09T14:46:26.890992+0000","last_became_peered":"2024-03-09T14:46:26.890992+0000","last_unstale":"2024-03-09T15:32:25.788416+0000","last_undegraded":"2024-03-09T15:32:25.788416+0000","last_fullsized":"2024-03-09T15:32:25.788416+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:56:14.439157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.14","version":"0'0","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:06.187943+0000","last_change":"2024-03-09T14:46:18.767760+0000","last_active":"2024-03-09T15:33:06.187943+0000","last_peered":"2024-03-09T15:33:06.187943+0000","last_clean":"2024-03-09T15:33:06.187943+0000","last_became_active":"2024-03-09T14:46:18.766777+0000","last_became_peered":"2024-03-09T14:46:18.766777+0000","last_unstale":"2024-03-09T15:33:06.187943+0000","last_undegraded":"2024-03-09T15:33:06.187943+0000","last_fullsized":"2024-03-09T15:33:06.187943+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:30:54.129379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.17","version":"258'19","reported_seq":107,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.030314+0000","last_change":"2024-03-09T14:39:13.281484+0000","last_active":"2024-03-09T15:34:51.030314+0000","last_peered":"2024-03-09T15:34:51.030314+0000","last_clean":"2024-03-09T15:34:51.030314+0000","last_became_active":"2024-03-09T14:39:13.279919+0000","last_became_peered":"2024-03-09T14:39:13.279919+0000","last_unstale":"2024-03-09T15:34:51.030314+0000","last_undegraded":"2024-03-09T15:34:51.030314+0000","last_fullsized":"2024-03-09T15:34:51.030314+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":64,"num_write":32,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.13","version":"261'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.971686+0000","last_change":"2024-03-09T15:09:50.079986+0000","last_active":"2024-03-09T15:33:10.971686+0000","last_peered":"2024-03-09T15:33:10.971686+0000","last_clean":"2024-03-09T15:33:10.971686+0000","last_became_active":"2024-03-09T15:09:50.079604+0000","last_became_peered":"2024-03-09T15:09:50.079604+0000","last_unstale":"2024-03-09T15:33:10.971686+0000","last_undegraded":"2024-03-09T15:33:10.971686+0000","last_fullsized":"2024-03-09T15:33:10.971686+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:30:05.194786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.14","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.800266+0000","last_change":"2024-03-09T14:46:26.894898+0000","last_active":"2024-03-09T15:33:20.800266+0000","last_peered":"2024-03-09T15:33:20.800266+0000","last_clean":"2024-03-09T15:33:20.800266+0000","last_became_active":"2024-03-09T14:46:26.894088+0000","last_became_peered":"2024-03-09T14:46:26.894088+0000","last_unstale":"2024-03-09T15:33:20.800266+0000","last_undegraded":"2024-03-09T15:33:20.800266+0000","last_fullsized":"2024-03-09T15:33:20.800266+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:10:41.693365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.15","version":"0'0","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:16.187056+0000","last_change":"2024-03-09T14:46:18.753568+0000","last_active":"2024-03-09T15:32:16.187056+0000","last_peered":"2024-03-09T15:32:16.187056+0000","last_clean":"2024-03-09T15:32:16.187056+0000","last_became_active":"2024-03-09T14:46:18.752413+0000","last_became_peered":"2024-03-09T14:46:18.752413+0000","last_unstale":"2024-03-09T15:32:16.187056+0000","last_undegraded":"2024-03-09T15:32:16.187056+0000","last_fullsized":"2024-03-09T15:32:16.187056+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:09:14.935604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.16","version":"261'52","reported_seq":184,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.788486+0000","last_change":"2024-03-09T14:39:13.263228+0000","last_active":"2024-03-09T15:32:25.788486+0000","last_peered":"2024-03-09T15:32:25.788486+0000","last_clean":"2024-03-09T15:32:25.788486+0000","last_became_active":"2024-03-09T14:39:13.262413+0000","last_became_peered":"2024-03-09T14:39:13.262413+0000","last_unstale":"2024-03-09T15:32:25.788486+0000","last_undegraded":"2024-03-09T15:32:25.788486+0000","last_fullsized":"2024-03-09T15:32:25.788486+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:02:22.902848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":163,"num_write":78,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.10","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.199392+0000","last_change":"2024-03-09T15:09:50.075112+0000","last_active":"2024-03-09T15:33:11.199392+0000","last_peered":"2024-03-09T15:33:11.199392+0000","last_clean":"2024-03-09T15:33:11.199392+0000","last_became_active":"2024-03-09T15:09:50.074274+0000","last_became_peered":"2024-03-09T15:09:50.074274+0000","last_unstale":"2024-03-09T15:33:11.199392+0000","last_undegraded":"2024-03-09T15:33:11.199392+0000","last_fullsized":"2024-03-09T15:33:11.199392+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:34:26.130292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.17","version":"0'0","reported_seq":42,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.788513+0000","last_change":"2024-03-09T14:46:26.894297+0000","last_active":"2024-03-09T15:32:25.788513+0000","last_peered":"2024-03-09T15:32:25.788513+0000","last_clean":"2024-03-09T15:32:25.788513+0000","last_became_active":"2024-03-09T14:46:26.893514+0000","last_became_peered":"2024-03-09T14:46:26.893514+0000","last_unstale":"2024-03-09T15:32:25.788513+0000","last_undegraded":"2024-03-09T15:32:25.788513+0000","last_fullsized":"2024-03-09T15:32:25.788513+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:03:25.018128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.16","version":"0'0","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:15.960497+0000","last_change":"2024-03-09T14:46:18.773133+0000","last_active":"2024-03-09T15:32:15.960497+0000","last_peered":"2024-03-09T15:32:15.960497+0000","last_clean":"2024-03-09T15:32:15.960497+0000","last_became_active":"2024-03-09T14:46:18.769274+0000","last_became_peered":"2024-03-09T14:46:18.769274+0000","last_unstale":"2024-03-09T15:32:15.960497+0000","last_undegraded":"2024-03-09T15:32:15.960497+0000","last_fullsized":"2024-03-09T15:32:15.960497+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:01:15.489364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.15","version":"0'0","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.962879+0000","last_change":"2024-03-09T14:39:13.263907+0000","last_active":"2024-03-09T15:32:25.962879+0000","last_peered":"2024-03-09T15:32:25.962879+0000","last_clean":"2024-03-09T15:32:25.962879+0000","last_became_active":"2024-03-09T14:39:13.262066+0000","last_became_peered":"2024-03-09T14:39:13.262066+0000","last_unstale":"2024-03-09T15:32:25.962879+0000","last_undegraded":"2024-03-09T15:32:25.962879+0000","last_fullsized":"2024-03-09T15:32:25.962879+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:55:26.909612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.11","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.188977+0000","last_change":"2024-03-09T15:09:50.086149+0000","last_active":"2024-03-09T15:33:11.188977+0000","last_peered":"2024-03-09T15:33:11.188977+0000","last_clean":"2024-03-09T15:33:11.188977+0000","last_became_active":"2024-03-09T15:09:50.085346+0000","last_became_peered":"2024-03-09T15:09:50.085346+0000","last_unstale":"2024-03-09T15:33:11.188977+0000","last_undegraded":"2024-03-09T15:33:11.188977+0000","last_fullsized":"2024-03-09T15:33:11.188977+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:54:52.652612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.16","version":"0'0","reported_seq":42,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.175271+0000","last_change":"2024-03-09T14:46:26.818659+0000","last_active":"2024-03-09T15:32:21.175271+0000","last_peered":"2024-03-09T15:32:21.175271+0000","last_clean":"2024-03-09T15:32:21.175271+0000","last_became_active":"2024-03-09T14:46:26.817864+0000","last_became_peered":"2024-03-09T14:46:26.817864+0000","last_unstale":"2024-03-09T15:32:21.175271+0000","last_undegraded":"2024-03-09T15:32:21.175271+0000","last_fullsized":"2024-03-09T15:32:21.175271+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:10:59.964957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.17","version":"0'0","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:41.027946+0000","last_change":"2024-03-09T14:46:18.780789+0000","last_active":"2024-03-09T15:34:41.027946+0000","last_peered":"2024-03-09T15:34:41.027946+0000","last_clean":"2024-03-09T15:34:41.027946+0000","last_became_active":"2024-03-09T14:46:18.776430+0000","last_became_peered":"2024-03-09T14:46:18.776430+0000","last_unstale":"2024-03-09T15:34:41.027946+0000","last_undegraded":"2024-03-09T15:34:41.027946+0000","last_fullsized":"2024-03-09T15:34:41.027946+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.14","version":"258'67","reported_seq":270,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:06.198380+0000","last_change":"2024-03-09T14:39:13.253820+0000","last_active":"2024-03-09T15:33:06.198380+0000","last_peered":"2024-03-09T15:33:06.198380+0000","last_clean":"2024-03-09T15:33:06.198380+0000","last_became_active":"2024-03-09T14:39:13.253007+0000","last_became_peered":"2024-03-09T14:39:13.253007+0000","last_unstale":"2024-03-09T15:33:06.198380+0000","last_undegraded":"2024-03-09T15:33:06.198380+0000","last_fullsized":"2024-03-09T15:33:06.198380+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:44:10.310392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":411,"num_read_kb":279,"num_write":87,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.16","version":"0'0","reported_seq":22,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776888+0000","last_change":"2024-03-09T15:09:50.087752+0000","last_active":"2024-03-09T15:31:35.776888+0000","last_peered":"2024-03-09T15:31:35.776888+0000","last_clean":"2024-03-09T15:31:35.776888+0000","last_became_active":"2024-03-09T15:09:50.086173+0000","last_became_peered":"2024-03-09T15:09:50.086173+0000","last_unstale":"2024-03-09T15:31:35.776888+0000","last_undegraded":"2024-03-09T15:31:35.776888+0000","last_fullsized":"2024-03-09T15:31:35.776888+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:43:21.719422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.11","version":"261'7","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:25.986303+0000","last_change":"2024-03-09T14:46:26.817795+0000","last_active":"2024-03-09T15:35:25.986303+0000","last_peered":"2024-03-09T15:35:25.986303+0000","last_clean":"2024-03-09T15:35:25.986303+0000","last_became_active":"2024-03-09T14:46:26.817014+0000","last_became_peered":"2024-03-09T14:46:26.817014+0000","last_unstale":"2024-03-09T15:35:25.986303+0000","last_undegraded":"2024-03-09T15:35:25.986303+0000","last_fullsized":"2024-03-09T15:35:25.986303+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:58:54.836958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":23554,"num_write":11,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.10","version":"261'3","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:21.012454+0000","last_change":"2024-03-09T14:46:18.771622+0000","last_active":"2024-03-09T15:35:21.012454+0000","last_peered":"2024-03-09T15:35:21.012454+0000","last_clean":"2024-03-09T15:35:21.012454+0000","last_became_active":"2024-03-09T14:46:18.770936+0000","last_became_peered":"2024-03-09T14:46:18.770936+0000","last_unstale":"2024-03-09T15:35:21.012454+0000","last_undegraded":"2024-03-09T15:35:21.012454+0000","last_fullsized":"2024-03-09T15:35:21.012454+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:04.807309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.13","version":"258'16","reported_seq":95,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.196014+0000","last_change":"2024-03-09T14:39:13.266972+0000","last_active":"2024-03-09T15:32:56.196014+0000","last_peered":"2024-03-09T15:32:56.196014+0000","last_clean":"2024-03-09T15:32:56.196014+0000","last_became_active":"2024-03-09T14:39:13.266228+0000","last_became_peered":"2024-03-09T14:39:13.266228+0000","last_unstale":"2024-03-09T15:32:56.196014+0000","last_undegraded":"2024-03-09T15:32:56.196014+0000","last_fullsized":"2024-03-09T15:32:56.196014+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:02:45.648908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":45,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.17","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:01.019023+0000","last_change":"2024-03-09T15:09:50.075655+0000","last_active":"2024-03-09T15:34:01.019023+0000","last_peered":"2024-03-09T15:34:01.019023+0000","last_clean":"2024-03-09T15:34:01.019023+0000","last_became_active":"2024-03-09T15:09:50.075187+0000","last_became_peered":"2024-03-09T15:09:50.075187+0000","last_unstale":"2024-03-09T15:34:01.019023+0000","last_undegraded":"2024-03-09T15:34:01.019023+0000","last_fullsized":"2024-03-09T15:34:01.019023+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.10","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:05.956045+0000","last_change":"2024-03-09T14:46:26.803323+0000","last_active":"2024-03-09T15:33:05.956045+0000","last_peered":"2024-03-09T15:33:05.956045+0000","last_clean":"2024-03-09T15:33:05.956045+0000","last_became_active":"2024-03-09T14:46:26.802476+0000","last_became_peered":"2024-03-09T14:46:26.802476+0000","last_unstale":"2024-03-09T15:33:05.956045+0000","last_undegraded":"2024-03-09T15:33:05.956045+0000","last_fullsized":"2024-03-09T15:33:05.956045+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:04:33.172429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.11","version":"0'0","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.795596+0000","last_change":"2024-03-09T14:46:18.761711+0000","last_active":"2024-03-09T15:33:00.795596+0000","last_peered":"2024-03-09T15:33:00.795596+0000","last_clean":"2024-03-09T15:33:00.795596+0000","last_became_active":"2024-03-09T14:46:18.760959+0000","last_became_peered":"2024-03-09T14:46:18.760959+0000","last_unstale":"2024-03-09T15:33:00.795596+0000","last_undegraded":"2024-03-09T15:33:00.795596+0000","last_fullsized":"2024-03-09T15:33:00.795596+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:08.964116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.12","version":"258'7","reported_seq":78,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:16.035873+0000","last_change":"2024-03-09T14:39:13.274560+0000","last_active":"2024-03-09T15:35:16.035873+0000","last_peered":"2024-03-09T15:35:16.035873+0000","last_clean":"2024-03-09T15:35:16.035873+0000","last_became_active":"2024-03-09T14:39:13.273740+0000","last_became_peered":"2024-03-09T14:39:13.273740+0000","last_unstale":"2024-03-09T15:35:16.035873+0000","last_undegraded":"2024-03-09T15:35:16.035873+0000","last_fullsized":"2024-03-09T15:35:16.035873+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":26,"num_write":13,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.14","version":"260'1","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:36.040914+0000","last_change":"2024-03-09T15:09:50.065240+0000","last_active":"2024-03-09T15:35:36.040914+0000","last_peered":"2024-03-09T15:35:36.040914+0000","last_clean":"2024-03-09T15:35:36.040914+0000","last_became_active":"2024-03-09T15:09:50.064428+0000","last_became_peered":"2024-03-09T15:09:50.064428+0000","last_unstale":"2024-03-09T15:35:36.040914+0000","last_undegraded":"2024-03-09T15:35:36.040914+0000","last_fullsized":"2024-03-09T15:35:36.040914+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.13","version":"261'3","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:30.828765+0000","last_change":"2024-03-09T14:46:26.813826+0000","last_active":"2024-03-09T15:35:30.828765+0000","last_peered":"2024-03-09T15:35:30.828765+0000","last_clean":"2024-03-09T15:35:30.828765+0000","last_became_active":"2024-03-09T14:46:26.813088+0000","last_became_peered":"2024-03-09T14:46:26.813088+0000","last_unstale":"2024-03-09T15:35:30.828765+0000","last_undegraded":"2024-03-09T15:35:30.828765+0000","last_fullsized":"2024-03-09T15:35:30.828765+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:37:31.388531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.12","version":"261'6","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.924112+0000","last_change":"2024-03-09T14:46:18.783155+0000","last_active":"2024-03-09T15:36:05.924112+0000","last_peered":"2024-03-09T15:36:05.924112+0000","last_clean":"2024-03-09T15:36:05.924112+0000","last_became_active":"2024-03-09T14:46:18.782158+0000","last_became_peered":"2024-03-09T14:46:18.782158+0000","last_unstale":"2024-03-09T15:36:05.924112+0000","last_undegraded":"2024-03-09T15:36:05.924112+0000","last_fullsized":"2024-03-09T15:36:05.924112+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:44:00.127768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":11264,"num_write":6,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.11","version":"258'36","reported_seq":148,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.182425+0000","last_change":"2024-03-09T14:39:13.260927+0000","last_active":"2024-03-09T15:32:56.182425+0000","last_peered":"2024-03-09T15:32:56.182425+0000","last_clean":"2024-03-09T15:32:56.182425+0000","last_became_active":"2024-03-09T14:39:13.260219+0000","last_became_peered":"2024-03-09T14:39:13.260219+0000","last_unstale":"2024-03-09T15:32:56.182425+0000","last_undegraded":"2024-03-09T15:32:56.182425+0000","last_fullsized":"2024-03-09T15:32:56.182425+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:27:50.114739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":118,"num_write":57,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.15","version":"261'1","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776826+0000","last_change":"2024-03-09T15:09:50.082323+0000","last_active":"2024-03-09T15:31:35.776826+0000","last_peered":"2024-03-09T15:31:35.776826+0000","last_clean":"2024-03-09T15:31:35.776826+0000","last_became_active":"2024-03-09T15:09:50.081688+0000","last_became_peered":"2024-03-09T15:09:50.081688+0000","last_unstale":"2024-03-09T15:31:35.776826+0000","last_undegraded":"2024-03-09T15:31:35.776826+0000","last_fullsized":"2024-03-09T15:31:35.776826+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:51:04.000384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.12","version":"261'2","reported_seq":46,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:30.813752+0000","last_change":"2024-03-09T14:46:26.806904+0000","last_active":"2024-03-09T15:35:30.813752+0000","last_peered":"2024-03-09T15:35:30.813752+0000","last_clean":"2024-03-09T15:35:30.813752+0000","last_became_active":"2024-03-09T14:46:26.806155+0000","last_became_peered":"2024-03-09T14:46:26.806155+0000","last_unstale":"2024-03-09T15:35:30.813752+0000","last_undegraded":"2024-03-09T15:35:30.813752+0000","last_fullsized":"2024-03-09T15:35:30.813752+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:30:04.752758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.13","version":"261'2","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:21.217889+0000","last_change":"2024-03-09T14:46:18.783226+0000","last_active":"2024-03-09T15:35:21.217889+0000","last_peered":"2024-03-09T15:35:21.217889+0000","last_clean":"2024-03-09T15:35:21.217889+0000","last_became_active":"2024-03-09T14:46:18.782340+0000","last_became_peered":"2024-03-09T14:46:18.782340+0000","last_unstale":"2024-03-09T15:35:21.217889+0000","last_undegraded":"2024-03-09T15:35:21.217889+0000","last_fullsized":"2024-03-09T15:35:21.217889+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:42:41.572332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":50,"num_write":2,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.10","version":"258'67","reported_seq":256,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.794304+0000","last_change":"2024-03-09T14:39:13.245901+0000","last_active":"2024-03-09T15:32:55.794304+0000","last_peered":"2024-03-09T15:32:55.794304+0000","last_clean":"2024-03-09T15:32:55.794304+0000","last_became_active":"2024-03-09T14:39:13.245107+0000","last_became_peered":"2024-03-09T14:39:13.245107+0000","last_unstale":"2024-03-09T15:32:55.794304+0000","last_undegraded":"2024-03-09T15:32:55.794304+0000","last_fullsized":"2024-03-09T15:32:55.794304+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:25:10.518351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":382,"num_read_kb":270,"num_write":93,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.a","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.978852+0000","last_change":"2024-03-09T15:09:50.080044+0000","last_active":"2024-03-09T15:33:45.978852+0000","last_peered":"2024-03-09T15:33:45.978852+0000","last_clean":"2024-03-09T15:33:45.978852+0000","last_became_active":"2024-03-09T15:09:50.079554+0000","last_became_peered":"2024-03-09T15:09:50.079554+0000","last_unstale":"2024-03-09T15:33:45.978852+0000","last_undegraded":"2024-03-09T15:33:45.978852+0000","last_fullsized":"2024-03-09T15:33:45.978852+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:15:33.287535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.d","version":"261'5","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:00.835088+0000","last_change":"2024-03-09T14:46:26.813839+0000","last_active":"2024-03-09T15:36:00.835088+0000","last_peered":"2024-03-09T15:36:00.835088+0000","last_clean":"2024-03-09T15:36:00.835088+0000","last_became_active":"2024-03-09T14:46:26.812859+0000","last_became_peered":"2024-03-09T14:46:26.812859+0000","last_unstale":"2024-03-09T15:36:00.835088+0000","last_undegraded":"2024-03-09T15:36:00.835088+0000","last_fullsized":"2024-03-09T15:36:00.835088+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:10:31.220080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":8192,"num_write":10,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.c","version":"261'11","reported_seq":70,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.184026+0000","last_change":"2024-03-09T14:46:18.767068+0000","last_active":"2024-03-09T15:36:06.184026+0000","last_peered":"2024-03-09T15:36:06.184026+0000","last_clean":"2024-03-09T15:36:06.184026+0000","last_became_active":"2024-03-09T14:46:18.766370+0000","last_became_peered":"2024-03-09T14:46:18.766370+0000","last_unstale":"2024-03-09T15:36:06.184026+0000","last_undegraded":"2024-03-09T15:36:06.184026+0000","last_fullsized":"2024-03-09T15:36:06.184026+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:41:36.450438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":20480,"num_write":16,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.f","version":"258'18","reported_seq":99,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:30.802658+0000","last_change":"2024-03-09T14:39:13.271586+0000","last_active":"2024-03-09T15:33:30.802658+0000","last_peered":"2024-03-09T15:33:30.802658+0000","last_clean":"2024-03-09T15:33:30.802658+0000","last_became_active":"2024-03-09T14:39:13.268971+0000","last_became_peered":"2024-03-09T14:39:13.268971+0000","last_unstale":"2024-03-09T15:33:30.802658+0000","last_undegraded":"2024-03-09T15:33:30.802658+0000","last_fullsized":"2024-03-09T15:33:30.802658+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:14:41.597456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.b","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188151+0000","last_change":"2024-03-09T15:09:50.070451+0000","last_active":"2024-03-09T15:32:21.188151+0000","last_peered":"2024-03-09T15:32:21.188151+0000","last_clean":"2024-03-09T15:32:21.188151+0000","last_became_active":"2024-03-09T15:09:50.069508+0000","last_became_peered":"2024-03-09T15:09:50.069508+0000","last_unstale":"2024-03-09T15:32:21.188151+0000","last_undegraded":"2024-03-09T15:32:21.188151+0000","last_fullsized":"2024-03-09T15:32:21.188151+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:39:13.850719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.c","version":"261'3","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:00.835020+0000","last_change":"2024-03-09T14:46:26.813805+0000","last_active":"2024-03-09T15:36:00.835020+0000","last_peered":"2024-03-09T15:36:00.835020+0000","last_clean":"2024-03-09T15:36:00.835020+0000","last_became_active":"2024-03-09T14:46:26.813009+0000","last_became_peered":"2024-03-09T14:46:26.813009+0000","last_unstale":"2024-03-09T15:36:00.835020+0000","last_undegraded":"2024-03-09T15:36:00.835020+0000","last_fullsized":"2024-03-09T15:36:00.835020+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:34:36.995455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2048,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.d","version":"261'3","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.004101+0000","last_change":"2024-03-09T14:46:18.771558+0000","last_active":"2024-03-09T15:35:51.004101+0000","last_peered":"2024-03-09T15:35:51.004101+0000","last_clean":"2024-03-09T15:35:51.004101+0000","last_became_active":"2024-03-09T14:46:18.770732+0000","last_became_peered":"2024-03-09T14:46:18.770732+0000","last_unstale":"2024-03-09T15:35:51.004101+0000","last_undegraded":"2024-03-09T15:35:51.004101+0000","last_fullsized":"2024-03-09T15:35:51.004101+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:08:20.215164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.e","version":"261'8","reported_seq":79,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.190234+0000","last_change":"2024-03-09T14:39:13.272393+0000","last_active":"2024-03-09T15:33:16.190234+0000","last_peered":"2024-03-09T15:33:16.190234+0000","last_clean":"2024-03-09T15:33:16.190234+0000","last_became_active":"2024-03-09T14:39:13.271672+0000","last_became_peered":"2024-03-09T14:39:13.271672+0000","last_unstale":"2024-03-09T15:33:16.190234+0000","last_undegraded":"2024-03-09T15:33:16.190234+0000","last_fullsized":"2024-03-09T15:33:16.190234+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:35:02.345192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.8","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.790792+0000","last_change":"2024-03-09T15:09:50.064645+0000","last_active":"2024-03-09T15:33:40.790792+0000","last_peered":"2024-03-09T15:33:40.790792+0000","last_clean":"2024-03-09T15:33:40.790792+0000","last_became_active":"2024-03-09T15:09:50.061662+0000","last_became_peered":"2024-03-09T15:09:50.061662+0000","last_unstale":"2024-03-09T15:33:40.790792+0000","last_undegraded":"2024-03-09T15:33:40.790792+0000","last_fullsized":"2024-03-09T15:33:40.790792+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:32:01.749280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.f","version":"261'6","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.994123+0000","last_change":"2024-03-09T14:46:26.900254+0000","last_active":"2024-03-09T15:36:05.994123+0000","last_peered":"2024-03-09T15:36:05.994123+0000","last_clean":"2024-03-09T15:36:05.994123+0000","last_became_active":"2024-03-09T14:46:26.899430+0000","last_became_peered":"2024-03-09T14:46:26.899430+0000","last_unstale":"2024-03-09T15:36:05.994123+0000","last_undegraded":"2024-03-09T15:36:05.994123+0000","last_fullsized":"2024-03-09T15:36:05.994123+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:58:16.207053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5120,"num_write":11,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.e","version":"258'3","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.043939+0000","last_change":"2024-03-09T14:46:18.769516+0000","last_active":"2024-03-09T15:35:51.043939+0000","last_peered":"2024-03-09T15:35:51.043939+0000","last_clean":"2024-03-09T15:35:51.043939+0000","last_became_active":"2024-03-09T14:46:18.768694+0000","last_became_peered":"2024-03-09T14:46:18.768694+0000","last_unstale":"2024-03-09T15:35:51.043939+0000","last_undegraded":"2024-03-09T15:35:51.043939+0000","last_fullsized":"2024-03-09T15:35:51.043939+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.d","version":"261'43","reported_seq":159,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:15.785461+0000","last_change":"2024-03-09T14:39:13.266337+0000","last_active":"2024-03-09T15:33:15.785461+0000","last_peered":"2024-03-09T15:33:15.785461+0000","last_clean":"2024-03-09T15:33:15.785461+0000","last_became_active":"2024-03-09T14:39:13.257275+0000","last_became_peered":"2024-03-09T14:39:13.257275+0000","last_unstale":"2024-03-09T15:33:15.785461+0000","last_undegraded":"2024-03-09T15:33:15.785461+0000","last_fullsized":"2024-03-09T15:33:15.785461+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:45:48.503144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":175,"num_read_kb":123,"num_write":70,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.9","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:41.206696+0000","last_change":"2024-03-09T15:09:50.077371+0000","last_active":"2024-03-09T15:33:41.206696+0000","last_peered":"2024-03-09T15:33:41.206696+0000","last_clean":"2024-03-09T15:33:41.206696+0000","last_became_active":"2024-03-09T15:09:50.076702+0000","last_became_peered":"2024-03-09T15:09:50.076702+0000","last_unstale":"2024-03-09T15:33:41.206696+0000","last_undegraded":"2024-03-09T15:33:41.206696+0000","last_fullsized":"2024-03-09T15:33:41.206696+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:02:55.914615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.e","version":"261'6","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.110086+0000","last_change":"2024-03-09T14:46:26.818547+0000","last_active":"2024-03-09T15:36:06.110086+0000","last_peered":"2024-03-09T15:36:06.110086+0000","last_clean":"2024-03-09T15:36:06.110086+0000","last_became_active":"2024-03-09T14:46:26.814709+0000","last_became_peered":"2024-03-09T14:46:26.814709+0000","last_unstale":"2024-03-09T15:36:06.110086+0000","last_undegraded":"2024-03-09T15:36:06.110086+0000","last_fullsized":"2024-03-09T15:36:06.110086+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:11:42.597841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.f","version":"261'5","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:00.993367+0000","last_change":"2024-03-09T14:46:18.777803+0000","last_active":"2024-03-09T15:36:00.993367+0000","last_peered":"2024-03-09T15:36:00.993367+0000","last_clean":"2024-03-09T15:36:00.993367+0000","last_became_active":"2024-03-09T14:46:18.766000+0000","last_became_peered":"2024-03-09T14:46:18.766000+0000","last_unstale":"2024-03-09T15:36:00.993367+0000","last_undegraded":"2024-03-09T15:36:00.993367+0000","last_fullsized":"2024-03-09T15:36:00.993367+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:41:01.857202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4096,"num_write":10,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.c","version":"0'0","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:15.972703+0000","last_change":"2024-03-09T14:39:13.271296+0000","last_active":"2024-03-09T15:33:15.972703+0000","last_peered":"2024-03-09T15:33:15.972703+0000","last_clean":"2024-03-09T15:33:15.972703+0000","last_became_active":"2024-03-09T14:39:13.270527+0000","last_became_peered":"2024-03-09T14:39:13.270527+0000","last_unstale":"2024-03-09T15:33:15.972703+0000","last_undegraded":"2024-03-09T15:33:15.972703+0000","last_fullsized":"2024-03-09T15:33:15.972703+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:25:04.404490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.e","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.787362+0000","last_change":"2024-03-09T15:09:50.087358+0000","last_active":"2024-03-09T15:32:20.787362+0000","last_peered":"2024-03-09T15:32:20.787362+0000","last_clean":"2024-03-09T15:32:20.787362+0000","last_became_active":"2024-03-09T15:09:50.085548+0000","last_became_peered":"2024-03-09T15:09:50.085548+0000","last_unstale":"2024-03-09T15:32:20.787362+0000","last_undegraded":"2024-03-09T15:32:20.787362+0000","last_fullsized":"2024-03-09T15:32:20.787362+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:13.595041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.9","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:56.169630+0000","last_change":"2024-03-09T14:46:26.802054+0000","last_active":"2024-03-09T15:31:56.169630+0000","last_peered":"2024-03-09T15:31:56.169630+0000","last_clean":"2024-03-09T15:31:56.169630+0000","last_became_active":"2024-03-09T14:46:26.801235+0000","last_became_peered":"2024-03-09T14:46:26.801235+0000","last_unstale":"2024-03-09T15:31:56.169630+0000","last_undegraded":"2024-03-09T15:31:56.169630+0000","last_fullsized":"2024-03-09T15:31:56.169630+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:49:20.412101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.8","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.740124+0000","last_change":"2024-03-09T14:46:18.775300+0000","last_active":"2024-03-09T15:36:10.740124+0000","last_peered":"2024-03-09T15:36:10.740124+0000","last_clean":"2024-03-09T15:36:10.740124+0000","last_became_active":"2024-03-09T14:46:18.774547+0000","last_became_peered":"2024-03-09T14:46:18.774547+0000","last_unstale":"2024-03-09T15:36:10.740124+0000","last_undegraded":"2024-03-09T15:36:10.740124+0000","last_fullsized":"2024-03-09T15:36:10.740124+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.b","version":"258'10","reported_seq":86,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:31.039794+0000","last_change":"2024-03-09T14:39:13.258020+0000","last_active":"2024-03-09T15:35:31.039794+0000","last_peered":"2024-03-09T15:35:31.039794+0000","last_clean":"2024-03-09T15:35:31.039794+0000","last_became_active":"2024-03-09T14:39:13.257238+0000","last_became_peered":"2024-03-09T14:39:13.257238+0000","last_unstale":"2024-03-09T15:35:31.039794+0000","last_undegraded":"2024-03-09T15:35:31.039794+0000","last_fullsized":"2024-03-09T15:35:31.039794+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.f","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.189467+0000","last_change":"2024-03-09T15:09:50.071187+0000","last_active":"2024-03-09T15:32:26.189467+0000","last_peered":"2024-03-09T15:32:26.189467+0000","last_clean":"2024-03-09T15:32:26.189467+0000","last_became_active":"2024-03-09T15:09:50.070526+0000","last_became_peered":"2024-03-09T15:09:50.070526+0000","last_unstale":"2024-03-09T15:32:26.189467+0000","last_undegraded":"2024-03-09T15:32:26.189467+0000","last_fullsized":"2024-03-09T15:32:26.189467+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:03:16.280366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.8","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:01.183115+0000","last_change":"2024-03-09T14:46:26.894478+0000","last_active":"2024-03-09T15:32:01.183115+0000","last_peered":"2024-03-09T15:32:01.183115+0000","last_clean":"2024-03-09T15:32:01.183115+0000","last_became_active":"2024-03-09T14:46:26.893908+0000","last_became_peered":"2024-03-09T14:46:26.893908+0000","last_unstale":"2024-03-09T15:32:01.183115+0000","last_undegraded":"2024-03-09T15:32:01.183115+0000","last_fullsized":"2024-03-09T15:32:01.183115+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:43:51.901456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.9","version":"261'3","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.369995+0000","last_change":"2024-03-09T14:46:18.777691+0000","last_active":"2024-03-09T15:36:06.369995+0000","last_peered":"2024-03-09T15:36:06.369995+0000","last_clean":"2024-03-09T15:36:06.369995+0000","last_became_active":"2024-03-09T14:46:18.761367+0000","last_became_peered":"2024-03-09T14:46:18.761367+0000","last_unstale":"2024-03-09T15:36:06.369995+0000","last_undegraded":"2024-03-09T15:36:06.369995+0000","last_fullsized":"2024-03-09T15:36:06.369995+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:44:32.343854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.a","version":"258'23","reported_seq":123,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205291+0000","last_change":"2024-03-09T14:39:13.253482+0000","last_active":"2024-03-09T15:33:36.205291+0000","last_peered":"2024-03-09T15:33:36.205291+0000","last_clean":"2024-03-09T15:33:36.205291+0000","last_became_active":"2024-03-09T14:39:13.252547+0000","last_became_peered":"2024-03-09T14:39:13.252547+0000","last_unstale":"2024-03-09T15:33:36.205291+0000","last_undegraded":"2024-03-09T15:33:36.205291+0000","last_fullsized":"2024-03-09T15:33:36.205291+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:32:42.396388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":86,"num_write":38,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.c","version":"261'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:21.188203+0000","last_change":"2024-03-09T15:09:50.070468+0000","last_active":"2024-03-09T15:32:21.188203+0000","last_peered":"2024-03-09T15:32:21.188203+0000","last_clean":"2024-03-09T15:32:21.188203+0000","last_became_active":"2024-03-09T15:09:50.069324+0000","last_became_peered":"2024-03-09T15:09:50.069324+0000","last_unstale":"2024-03-09T15:32:21.188203+0000","last_undegraded":"2024-03-09T15:32:21.188203+0000","last_fullsized":"2024-03-09T15:32:21.188203+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:23:32.203630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.b","version":"258'3","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.964465+0000","last_change":"2024-03-09T14:46:26.805564+0000","last_active":"2024-03-09T15:33:40.964465+0000","last_peered":"2024-03-09T15:33:40.964465+0000","last_clean":"2024-03-09T15:33:40.964465+0000","last_became_active":"2024-03-09T14:46:26.804615+0000","last_became_peered":"2024-03-09T14:46:26.804615+0000","last_unstale":"2024-03-09T15:33:40.964465+0000","last_undegraded":"2024-03-09T15:33:40.964465+0000","last_fullsized":"2024-03-09T15:33:40.964465+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:54:32.765203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.a","version":"258'3","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:11.021185+0000","last_change":"2024-03-09T14:46:18.769098+0000","last_active":"2024-03-09T15:34:11.021185+0000","last_peered":"2024-03-09T15:34:11.021185+0000","last_clean":"2024-03-09T15:34:11.021185+0000","last_became_active":"2024-03-09T14:46:18.768431+0000","last_became_peered":"2024-03-09T14:46:18.768431+0000","last_unstale":"2024-03-09T15:34:11.021185+0000","last_undegraded":"2024-03-09T15:34:11.021185+0000","last_fullsized":"2024-03-09T15:34:11.021185+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.9","version":"261'43","reported_seq":157,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.801289+0000","last_change":"2024-03-09T14:39:13.260724+0000","last_active":"2024-03-09T15:33:25.801289+0000","last_peered":"2024-03-09T15:33:25.801289+0000","last_clean":"2024-03-09T15:33:25.801289+0000","last_became_active":"2024-03-09T14:39:13.259975+0000","last_became_peered":"2024-03-09T14:39:13.259975+0000","last_unstale":"2024-03-09T15:33:25.801289+0000","last_undegraded":"2024-03-09T15:33:25.801289+0000","last_fullsized":"2024-03-09T15:33:25.801289+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:26:32.899703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":169,"num_read_kb":119,"num_write":74,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.d","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:36.026315+0000","last_change":"2024-03-09T15:09:50.084646+0000","last_active":"2024-03-09T15:34:36.026315+0000","last_peered":"2024-03-09T15:34:36.026315+0000","last_clean":"2024-03-09T15:34:36.026315+0000","last_became_active":"2024-03-09T15:09:50.084074+0000","last_became_peered":"2024-03-09T15:09:50.084074+0000","last_unstale":"2024-03-09T15:34:36.026315+0000","last_undegraded":"2024-03-09T15:34:36.026315+0000","last_fullsized":"2024-03-09T15:34:36.026315+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.a","version":"258'3","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:06.184462+0000","last_change":"2024-03-09T14:46:26.826875+0000","last_active":"2024-03-09T15:32:06.184462+0000","last_peered":"2024-03-09T15:32:06.184462+0000","last_clean":"2024-03-09T15:32:06.184462+0000","last_became_active":"2024-03-09T14:46:26.825949+0000","last_became_peered":"2024-03-09T14:46:26.825949+0000","last_unstale":"2024-03-09T15:32:06.184462+0000","last_undegraded":"2024-03-09T15:32:06.184462+0000","last_fullsized":"2024-03-09T15:32:06.184462+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:24:09.628828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.b","version":"261'3","reported_seq":55,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.019017+0000","last_change":"2024-03-09T14:46:18.785518+0000","last_active":"2024-03-09T15:35:51.019017+0000","last_peered":"2024-03-09T15:35:51.019017+0000","last_clean":"2024-03-09T15:35:51.019017+0000","last_became_active":"2024-03-09T14:46:18.784869+0000","last_became_peered":"2024-03-09T14:46:18.784869+0000","last_unstale":"2024-03-09T15:35:51.019017+0000","last_undegraded":"2024-03-09T15:35:51.019017+0000","last_fullsized":"2024-03-09T15:35:51.019017+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:03:42.729066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.8","version":"258'10","reported_seq":86,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.787542+0000","last_change":"2024-03-09T14:39:13.264924+0000","last_active":"2024-03-09T15:33:25.787542+0000","last_peered":"2024-03-09T15:33:25.787542+0000","last_clean":"2024-03-09T15:33:25.787542+0000","last_became_active":"2024-03-09T14:39:13.264006+0000","last_became_peered":"2024-03-09T14:39:13.264006+0000","last_unstale":"2024-03-09T15:33:25.787542+0000","last_undegraded":"2024-03-09T15:33:25.787542+0000","last_fullsized":"2024-03-09T15:33:25.787542+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:24:57.691583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.2","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.193658+0000","last_change":"2024-03-09T15:09:50.066069+0000","last_active":"2024-03-09T15:32:46.193658+0000","last_peered":"2024-03-09T15:32:46.193658+0000","last_clean":"2024-03-09T15:32:46.193658+0000","last_became_active":"2024-03-09T15:09:50.065487+0000","last_became_peered":"2024-03-09T15:09:50.065487+0000","last_unstale":"2024-03-09T15:32:46.193658+0000","last_undegraded":"2024-03-09T15:32:46.193658+0000","last_fullsized":"2024-03-09T15:32:46.193658+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:16:19.163190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.5","version":"258'2","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:36.026359+0000","last_change":"2024-03-09T14:46:26.813029+0000","last_active":"2024-03-09T15:34:36.026359+0000","last_peered":"2024-03-09T15:34:36.026359+0000","last_clean":"2024-03-09T15:34:36.026359+0000","last_became_active":"2024-03-09T14:46:26.812137+0000","last_became_peered":"2024-03-09T14:46:26.812137+0000","last_unstale":"2024-03-09T15:34:36.026359+0000","last_undegraded":"2024-03-09T15:34:36.026359+0000","last_fullsized":"2024-03-09T15:34:36.026359+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4096,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.4","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.185772+0000","last_change":"2024-03-09T14:46:18.752392+0000","last_active":"2024-03-09T15:32:11.185772+0000","last_peered":"2024-03-09T15:32:11.185772+0000","last_clean":"2024-03-09T15:32:11.185772+0000","last_became_active":"2024-03-09T14:46:18.751608+0000","last_became_peered":"2024-03-09T14:46:18.751608+0000","last_unstale":"2024-03-09T15:32:11.185772+0000","last_undegraded":"2024-03-09T15:32:11.185772+0000","last_fullsized":"2024-03-09T15:32:11.185772+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:32:15.618158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.7","version":"261'8","reported_seq":81,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:36.041062+0000","last_change":"2024-03-09T14:39:13.280605+0000","last_active":"2024-03-09T15:35:36.041062+0000","last_peered":"2024-03-09T15:35:36.041062+0000","last_clean":"2024-03-09T15:35:36.041062+0000","last_became_active":"2024-03-09T14:39:13.273363+0000","last_became_peered":"2024-03-09T14:39:13.273363+0000","last_unstale":"2024-03-09T15:35:36.041062+0000","last_undegraded":"2024-03-09T15:35:36.041062+0000","last_fullsized":"2024-03-09T15:35:36.041062+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.3","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.183450+0000","last_change":"2024-03-09T15:09:50.078106+0000","last_active":"2024-03-09T15:32:46.183450+0000","last_peered":"2024-03-09T15:32:46.183450+0000","last_clean":"2024-03-09T15:32:46.183450+0000","last_became_active":"2024-03-09T15:09:50.072488+0000","last_became_peered":"2024-03-09T15:09:50.072488+0000","last_unstale":"2024-03-09T15:32:46.183450+0000","last_undegraded":"2024-03-09T15:32:46.183450+0000","last_fullsized":"2024-03-09T15:32:46.183450+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:20:34.288932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.4","version":"258'2","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:26.024187+0000","last_change":"2024-03-09T14:46:26.895288+0000","last_active":"2024-03-09T15:34:26.024187+0000","last_peered":"2024-03-09T15:34:26.024187+0000","last_clean":"2024-03-09T15:34:26.024187+0000","last_became_active":"2024-03-09T14:46:26.894506+0000","last_became_peered":"2024-03-09T14:46:26.894506+0000","last_unstale":"2024-03-09T15:34:26.024187+0000","last_undegraded":"2024-03-09T15:34:26.024187+0000","last_fullsized":"2024-03-09T15:34:26.024187+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":23,"num_write":2,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.5","version":"258'5","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:25.962822+0000","last_change":"2024-03-09T14:46:18.771982+0000","last_active":"2024-03-09T15:32:25.962822+0000","last_peered":"2024-03-09T15:32:25.962822+0000","last_clean":"2024-03-09T15:32:25.962822+0000","last_became_active":"2024-03-09T14:46:18.770799+0000","last_became_peered":"2024-03-09T14:46:18.770799+0000","last_unstale":"2024-03-09T15:32:25.962822+0000","last_undegraded":"2024-03-09T15:32:25.962822+0000","last_fullsized":"2024-03-09T15:32:25.962822+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:15:47.749134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.6","version":"261'22","reported_seq":112,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.787743+0000","last_change":"2024-03-09T14:39:13.268293+0000","last_active":"2024-03-09T15:33:25.787743+0000","last_peered":"2024-03-09T15:33:25.787743+0000","last_clean":"2024-03-09T15:33:25.787743+0000","last_became_active":"2024-03-09T14:39:13.266246+0000","last_became_peered":"2024-03-09T14:39:13.266246+0000","last_unstale":"2024-03-09T15:33:25.787743+0000","last_undegraded":"2024-03-09T15:33:25.787743+0000","last_fullsized":"2024-03-09T15:33:25.787743+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:11:53.843138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":65,"num_write":41,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.0","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:00.968771+0000","last_change":"2024-03-09T15:09:50.064673+0000","last_active":"2024-03-09T15:34:00.968771+0000","last_peered":"2024-03-09T15:34:00.968771+0000","last_clean":"2024-03-09T15:34:00.968771+0000","last_became_active":"2024-03-09T15:09:50.063350+0000","last_became_peered":"2024-03-09T15:09:50.063350+0000","last_unstale":"2024-03-09T15:34:00.968771+0000","last_undegraded":"2024-03-09T15:34:00.968771+0000","last_fullsized":"2024-03-09T15:34:00.968771+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:55.461559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.7","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:01.183396+0000","last_change":"2024-03-09T14:46:26.897079+0000","last_active":"2024-03-09T15:32:01.183396+0000","last_peered":"2024-03-09T15:32:01.183396+0000","last_clean":"2024-03-09T15:32:01.183396+0000","last_became_active":"2024-03-09T14:46:26.896503+0000","last_became_peered":"2024-03-09T14:46:26.896503+0000","last_unstale":"2024-03-09T15:32:01.183396+0000","last_undegraded":"2024-03-09T15:32:01.183396+0000","last_fullsized":"2024-03-09T15:32:01.183396+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:57:37.168283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.6","version":"261'2","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.183625+0000","last_change":"2024-03-09T14:46:18.779289+0000","last_active":"2024-03-09T15:36:06.183625+0000","last_peered":"2024-03-09T15:36:06.183625+0000","last_clean":"2024-03-09T15:36:06.183625+0000","last_became_active":"2024-03-09T14:46:18.778572+0000","last_became_peered":"2024-03-09T14:46:18.778572+0000","last_unstale":"2024-03-09T15:36:06.183625+0000","last_undegraded":"2024-03-09T15:36:06.183625+0000","last_fullsized":"2024-03-09T15:36:06.183625+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:28:53.930043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.5","version":"258'18","reported_seq":107,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205525+0000","last_change":"2024-03-09T14:39:13.264712+0000","last_active":"2024-03-09T15:33:36.205525+0000","last_peered":"2024-03-09T15:33:36.205525+0000","last_clean":"2024-03-09T15:33:36.205525+0000","last_became_active":"2024-03-09T14:39:13.263833+0000","last_became_peered":"2024-03-09T14:39:13.263833+0000","last_unstale":"2024-03-09T15:33:36.205525+0000","last_undegraded":"2024-03-09T15:33:36.205525+0000","last_fullsized":"2024-03-09T15:33:36.205525+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:10:25.603434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":58,"num_write":29,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.1","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.180226+0000","last_change":"2024-03-09T15:09:50.069405+0000","last_active":"2024-03-09T15:32:46.180226+0000","last_peered":"2024-03-09T15:32:46.180226+0000","last_clean":"2024-03-09T15:32:46.180226+0000","last_became_active":"2024-03-09T15:09:50.067947+0000","last_became_peered":"2024-03-09T15:09:50.067947+0000","last_unstale":"2024-03-09T15:32:46.180226+0000","last_undegraded":"2024-03-09T15:32:46.180226+0000","last_fullsized":"2024-03-09T15:32:46.180226+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:29:17.390497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"67.6","version":"261'4","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.437789+0000","last_change":"2024-03-09T14:46:26.809411+0000","last_active":"2024-03-09T15:36:06.437789+0000","last_peered":"2024-03-09T15:36:06.437789+0000","last_clean":"2024-03-09T15:36:06.437789+0000","last_became_active":"2024-03-09T14:46:26.808384+0000","last_became_peered":"2024-03-09T14:46:26.808384+0000","last_unstale":"2024-03-09T15:36:06.437789+0000","last_undegraded":"2024-03-09T15:36:06.437789+0000","last_fullsized":"2024-03-09T15:36:06.437789+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:53:31.797035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":180,"num_write":4,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.7","version":"261'8","reported_seq":73,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.268626+0000","last_change":"2024-03-09T14:46:18.763286+0000","last_active":"2024-03-09T15:36:06.268626+0000","last_peered":"2024-03-09T15:36:06.268626+0000","last_clean":"2024-03-09T15:36:06.268626+0000","last_became_active":"2024-03-09T14:46:18.762472+0000","last_became_peered":"2024-03-09T14:46:18.762472+0000","last_unstale":"2024-03-09T15:36:06.268626+0000","last_undegraded":"2024-03-09T15:36:06.268626+0000","last_fullsized":"2024-03-09T15:36:06.268626+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:30.905827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":30722,"num_write":17,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.4","version":"258'10","reported_seq":87,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.960925+0000","last_change":"2024-03-09T14:39:13.270957+0000","last_active":"2024-03-09T15:33:25.960925+0000","last_peered":"2024-03-09T15:33:25.960925+0000","last_clean":"2024-03-09T15:33:25.960925+0000","last_became_active":"2024-03-09T14:39:13.270238+0000","last_became_peered":"2024-03-09T14:39:13.270238+0000","last_unstale":"2024-03-09T15:33:25.960925+0000","last_undegraded":"2024-03-09T15:33:25.960925+0000","last_fullsized":"2024-03-09T15:33:25.960925+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:25:19.656005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.6","version":"260'1","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.977825+0000","last_change":"2024-03-09T15:09:50.072623+0000","last_active":"2024-03-09T15:33:40.977825+0000","last_peered":"2024-03-09T15:33:40.977825+0000","last_clean":"2024-03-09T15:33:40.977825+0000","last_became_active":"2024-03-09T15:09:50.072180+0000","last_became_peered":"2024-03-09T15:09:50.072180+0000","last_unstale":"2024-03-09T15:33:40.977825+0000","last_undegraded":"2024-03-09T15:33:40.977825+0000","last_fullsized":"2024-03-09T15:33:40.977825+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:17:47.602511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1","version":"0'0","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:16.142380+0000","last_change":"2024-03-09T14:46:26.799783+0000","last_active":"2024-03-09T15:32:16.142380+0000","last_peered":"2024-03-09T15:32:16.142380+0000","last_clean":"2024-03-09T15:32:16.142380+0000","last_became_active":"2024-03-09T14:46:26.799019+0000","last_became_peered":"2024-03-09T14:46:26.799019+0000","last_unstale":"2024-03-09T15:32:16.142380+0000","last_undegraded":"2024-03-09T15:32:16.142380+0000","last_fullsized":"2024-03-09T15:32:16.142380+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:37:13.610839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.0","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.189237+0000","last_change":"2024-03-09T14:46:18.753025+0000","last_active":"2024-03-09T15:32:26.189237+0000","last_peered":"2024-03-09T15:32:26.189237+0000","last_clean":"2024-03-09T15:32:26.189237+0000","last_became_active":"2024-03-09T14:46:18.752290+0000","last_became_peered":"2024-03-09T14:46:18.752290+0000","last_unstale":"2024-03-09T15:32:26.189237+0000","last_undegraded":"2024-03-09T15:32:26.189237+0000","last_fullsized":"2024-03-09T15:32:26.189237+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:33:49.576573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.3","version":"261'61","reported_seq":215,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.192284+0000","last_change":"2024-03-09T14:39:13.259947+0000","last_active":"2024-03-09T15:33:26.192284+0000","last_peered":"2024-03-09T15:33:26.192284+0000","last_clean":"2024-03-09T15:33:26.192284+0000","last_became_active":"2024-03-09T14:39:13.259125+0000","last_became_peered":"2024-03-09T14:39:13.259125+0000","last_unstale":"2024-03-09T15:33:26.192284+0000","last_undegraded":"2024-03-09T15:33:26.192284+0000","last_fullsized":"2024-03-09T15:33:26.192284+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:52:15.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":281,"num_read_kb":205,"num_write":98,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.7","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.044007+0000","last_change":"2024-03-09T15:09:50.064879+0000","last_active":"2024-03-09T15:35:51.044007+0000","last_peered":"2024-03-09T15:35:51.044007+0000","last_clean":"2024-03-09T15:35:51.044007+0000","last_became_active":"2024-03-09T15:09:50.062429+0000","last_became_peered":"2024-03-09T15:09:50.062429+0000","last_unstale":"2024-03-09T15:35:51.044007+0000","last_undegraded":"2024-03-09T15:35:51.044007+0000","last_fullsized":"2024-03-09T15:35:51.044007+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.0","version":"261'6","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.015541+0000","last_change":"2024-03-09T14:46:26.816110+0000","last_active":"2024-03-09T15:36:06.015541+0000","last_peered":"2024-03-09T15:36:06.015541+0000","last_clean":"2024-03-09T15:36:06.015541+0000","last_became_active":"2024-03-09T14:46:26.815311+0000","last_became_peered":"2024-03-09T14:46:26.815311+0000","last_unstale":"2024-03-09T15:36:06.015541+0000","last_undegraded":"2024-03-09T15:36:06.015541+0000","last_fullsized":"2024-03-09T15:36:06.015541+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:26:21.047330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":16384,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1","version":"258'5","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.185673+0000","last_change":"2024-03-09T14:46:18.758437+0000","last_active":"2024-03-09T15:32:11.185673+0000","last_peered":"2024-03-09T15:32:11.185673+0000","last_clean":"2024-03-09T15:32:11.185673+0000","last_became_active":"2024-03-09T14:46:18.757694+0000","last_became_peered":"2024-03-09T14:46:18.757694+0000","last_unstale":"2024-03-09T15:32:11.185673+0000","last_undegraded":"2024-03-09T15:32:11.185673+0000","last_fullsized":"2024-03-09T15:32:11.185673+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:47:19.096625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":4098,"num_write":9,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.2","version":"258'20","reported_seq":110,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.188905+0000","last_change":"2024-03-09T14:39:13.265013+0000","last_active":"2024-03-09T15:33:26.188905+0000","last_peered":"2024-03-09T15:33:26.188905+0000","last_clean":"2024-03-09T15:33:26.188905+0000","last_became_active":"2024-03-09T14:39:13.264203+0000","last_became_peered":"2024-03-09T14:39:13.264203+0000","last_unstale":"2024-03-09T15:33:26.188905+0000","last_undegraded":"2024-03-09T15:33:26.188905+0000","last_fullsized":"2024-03-09T15:33:26.188905+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:46:43.366482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":65,"num_write":31,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.4","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.030150+0000","last_change":"2024-03-09T15:09:50.070703+0000","last_active":"2024-03-09T15:34:51.030150+0000","last_peered":"2024-03-09T15:34:51.030150+0000","last_clean":"2024-03-09T15:34:51.030150+0000","last_became_active":"2024-03-09T15:09:50.066222+0000","last_became_peered":"2024-03-09T15:09:50.066222+0000","last_unstale":"2024-03-09T15:34:51.030150+0000","last_undegraded":"2024-03-09T15:34:51.030150+0000","last_fullsized":"2024-03-09T15:34:51.030150+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.3","version":"0'0","reported_seq":45,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:20.774297+0000","last_change":"2024-03-09T14:46:26.826931+0000","last_active":"2024-03-09T15:32:20.774297+0000","last_peered":"2024-03-09T15:32:20.774297+0000","last_clean":"2024-03-09T15:32:20.774297+0000","last_became_active":"2024-03-09T14:46:26.826105+0000","last_became_peered":"2024-03-09T14:46:26.826105+0000","last_unstale":"2024-03-09T15:32:20.774297+0000","last_undegraded":"2024-03-09T15:32:20.774297+0000","last_fullsized":"2024-03-09T15:32:20.774297+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:19:31.840994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.2","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:11.141234+0000","last_change":"2024-03-09T14:46:18.749838+0000","last_active":"2024-03-09T15:32:11.141234+0000","last_peered":"2024-03-09T15:32:11.141234+0000","last_clean":"2024-03-09T15:32:11.141234+0000","last_became_active":"2024-03-09T14:46:18.749091+0000","last_became_peered":"2024-03-09T14:46:18.749091+0000","last_unstale":"2024-03-09T15:32:11.141234+0000","last_undegraded":"2024-03-09T15:32:11.141234+0000","last_fullsized":"2024-03-09T15:32:11.141234+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:47:14.332817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1","version":"261'25","reported_seq":125,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.787625+0000","last_change":"2024-03-09T14:39:13.257789+0000","last_active":"2024-03-09T15:33:25.787625+0000","last_peered":"2024-03-09T15:33:25.787625+0000","last_clean":"2024-03-09T15:33:25.787625+0000","last_became_active":"2024-03-09T14:39:13.255589+0000","last_became_peered":"2024-03-09T14:39:13.255589+0000","last_unstale":"2024-03-09T15:33:25.787625+0000","last_undegraded":"2024-03-09T15:33:25.787625+0000","last_fullsized":"2024-03-09T15:33:25.787625+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:53:30.259823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":79,"num_write":35,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.5","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:51.194924+0000","last_change":"2024-03-09T15:09:50.083562+0000","last_active":"2024-03-09T15:32:51.194924+0000","last_peered":"2024-03-09T15:32:51.194924+0000","last_clean":"2024-03-09T15:32:51.194924+0000","last_became_active":"2024-03-09T15:09:50.083064+0000","last_became_peered":"2024-03-09T15:09:50.083064+0000","last_unstale":"2024-03-09T15:32:51.194924+0000","last_undegraded":"2024-03-09T15:32:51.194924+0000","last_fullsized":"2024-03-09T15:32:51.194924+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:25:31.798282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.2","version":"258'3","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:16.174077+0000","last_change":"2024-03-09T14:46:26.808768+0000","last_active":"2024-03-09T15:32:16.174077+0000","last_peered":"2024-03-09T15:32:16.174077+0000","last_clean":"2024-03-09T15:32:16.174077+0000","last_became_active":"2024-03-09T14:46:26.807746+0000","last_became_peered":"2024-03-09T14:46:26.807746+0000","last_unstale":"2024-03-09T15:32:16.174077+0000","last_undegraded":"2024-03-09T15:32:16.174077+0000","last_fullsized":"2024-03-09T15:32:16.174077+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:58:17.140698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1024,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.3","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:10.772267+0000","last_change":"2024-03-09T14:46:18.765863+0000","last_active":"2024-03-09T15:32:10.772267+0000","last_peered":"2024-03-09T15:32:10.772267+0000","last_clean":"2024-03-09T15:32:10.772267+0000","last_became_active":"2024-03-09T14:46:18.765077+0000","last_became_peered":"2024-03-09T14:46:18.765077+0000","last_unstale":"2024-03-09T15:32:10.772267+0000","last_undegraded":"2024-03-09T15:32:10.772267+0000","last_fullsized":"2024-03-09T15:32:10.772267+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:26:58.084480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.0","version":"250'25","reported_seq":130,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.202779+0000","last_change":"2024-03-09T14:39:13.269464+0000","last_active":"2024-03-09T15:33:26.202779+0000","last_peered":"2024-03-09T15:33:26.202779+0000","last_clean":"2024-03-09T15:33:26.202779+0000","last_became_active":"2024-03-09T14:39:13.268553+0000","last_became_peered":"2024-03-09T14:39:13.268553+0000","last_unstale":"2024-03-09T15:33:26.202779+0000","last_undegraded":"2024-03-09T15:33:26.202779+0000","last_fullsized":"2024-03-09T15:33:26.202779+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:39:12.220263+0000","last_clean_scrub_stamp":"2024-03-09T14:39:12.220263+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:01:17.783740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":97,"num_write":32,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"261'114","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.426476+0000","last_change":"2024-03-09T14:28:32.340306+0000","last_active":"2024-03-09T15:36:06.426476+0000","last_peered":"2024-03-09T15:36:06.426476+0000","last_clean":"2024-03-09T15:36:06.426476+0000","last_became_active":"2024-03-09T14:28:32.339785+0000","last_became_peered":"2024-03-09T14:28:32.339785+0000","last_unstale":"2024-03-09T15:36:06.426476+0000","last_undegraded":"2024-03-09T15:36:06.426476+0000","last_fullsized":"2024-03-09T15:36:06.426476+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:39:28.870360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":130,"num_read_kb":5377,"num_write":373,"num_write_kb":16955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"261'146","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194366+0000","last_change":"2024-03-09T14:28:34.304676+0000","last_active":"2024-03-09T15:33:36.194366+0000","last_peered":"2024-03-09T15:33:36.194366+0000","last_clean":"2024-03-09T15:33:36.194366+0000","last_became_active":"2024-03-09T14:28:34.303980+0000","last_became_peered":"2024-03-09T14:28:34.303980+0000","last_unstale":"2024-03-09T15:33:36.194366+0000","last_undegraded":"2024-03-09T15:33:36.194366+0000","last_fullsized":"2024-03-09T15:33:36.194366+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:12:53.886509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":546,"num_read_kb":417,"num_write":228,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"261'135","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.469993+0000","last_change":"2024-03-09T14:28:32.307225+0000","last_active":"2024-03-09T15:36:06.469993+0000","last_peered":"2024-03-09T15:36:06.469993+0000","last_clean":"2024-03-09T15:36:06.469993+0000","last_became_active":"2024-03-09T14:28:32.306840+0000","last_became_peered":"2024-03-09T14:28:32.306840+0000","last_unstale":"2024-03-09T15:36:06.469993+0000","last_undegraded":"2024-03-09T15:36:06.469993+0000","last_fullsized":"2024-03-09T15:36:06.469993+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":100,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:46:26.657945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3389647,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":156,"num_read_kb":3004,"num_write":523,"num_write_kb":19720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"261'89","reported_seq":655,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:39.894721+0000","last_change":"2024-03-09T14:28:34.324788+0000","last_active":"2024-03-09T15:34:39.894721+0000","last_peered":"2024-03-09T15:34:39.894721+0000","last_clean":"2024-03-09T15:34:39.894721+0000","last_became_active":"2024-03-09T14:28:34.324053+0000","last_became_peered":"2024-03-09T14:28:34.324053+0000","last_unstale":"2024-03-09T15:34:39.894721+0000","last_undegraded":"2024-03-09T15:34:39.894721+0000","last_fullsized":"2024-03-09T15:34:39.894721+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:16:16.817770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":372,"num_read_kb":300,"num_write":141,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3d","version":"261'158","reported_seq":719,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.527086+0000","last_change":"2024-03-09T14:28:32.292874+0000","last_active":"2024-03-09T15:36:06.527086+0000","last_peered":"2024-03-09T15:36:06.527086+0000","last_clean":"2024-03-09T15:36:06.527086+0000","last_became_active":"2024-03-09T14:28:32.286966+0000","last_became_peered":"2024-03-09T14:28:32.286966+0000","last_unstale":"2024-03-09T15:36:06.527086+0000","last_undegraded":"2024-03-09T15:36:06.527086+0000","last_fullsized":"2024-03-09T15:36:06.527086+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":100,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":14171,"num_write":610,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"261'157","reported_seq":718,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191267+0000","last_change":"2024-03-09T14:28:34.307655+0000","last_active":"2024-03-09T15:33:36.191267+0000","last_peered":"2024-03-09T15:33:36.191267+0000","last_clean":"2024-03-09T15:33:36.191267+0000","last_became_active":"2024-03-09T14:28:34.306764+0000","last_became_peered":"2024-03-09T14:28:34.306764+0000","last_unstale":"2024-03-09T15:33:36.191267+0000","last_undegraded":"2024-03-09T15:33:36.191267+0000","last_fullsized":"2024-03-09T15:33:36.191267+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:19:26.321622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":481,"num_write":261,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3c","version":"261'190","reported_seq":782,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.459595+0000","last_change":"2024-03-09T14:28:32.310328+0000","last_active":"2024-03-09T15:36:06.459595+0000","last_peered":"2024-03-09T15:36:06.459595+0000","last_clean":"2024-03-09T15:36:06.459595+0000","last_became_active":"2024-03-09T14:28:32.305615+0000","last_became_peered":"2024-03-09T14:28:32.305615+0000","last_unstale":"2024-03-09T15:36:06.459595+0000","last_undegraded":"2024-03-09T15:36:06.459595+0000","last_fullsized":"2024-03-09T15:36:06.459595+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":100,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:45:12.810858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5326894,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":256,"num_read_kb":28937,"num_write":797,"num_write_kb":45835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"261'156","reported_seq":733,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.046701+0000","last_change":"2024-03-09T14:28:34.316937+0000","last_active":"2024-03-09T15:36:06.046701+0000","last_peered":"2024-03-09T15:36:06.046701+0000","last_clean":"2024-03-09T15:36:06.046701+0000","last_became_active":"2024-03-09T14:28:34.316500+0000","last_became_peered":"2024-03-09T14:28:34.316500+0000","last_unstale":"2024-03-09T15:36:06.046701+0000","last_undegraded":"2024-03-09T15:36:06.046701+0000","last_fullsized":"2024-03-09T15:36:06.046701+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":100,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":622,"num_read_kb":488,"num_write":257,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3b","version":"261'150","reported_seq":812,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.427470+0000","last_change":"2024-03-09T14:28:32.317675+0000","last_active":"2024-03-09T15:36:06.427470+0000","last_peered":"2024-03-09T15:36:06.427470+0000","last_clean":"2024-03-09T15:36:06.427470+0000","last_became_active":"2024-03-09T14:28:32.314923+0000","last_became_peered":"2024-03-09T14:28:32.314923+0000","last_unstale":"2024-03-09T15:36:06.427470+0000","last_undegraded":"2024-03-09T15:36:06.427470+0000","last_fullsized":"2024-03-09T15:36:06.427470+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:29:14.419941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1917997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":297,"num_read_kb":17282,"num_write":517,"num_write_kb":14747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3c","version":"261'124","reported_seq":706,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.781267+0000","last_change":"2024-03-09T14:28:34.315343+0000","last_active":"2024-03-09T15:32:55.781267+0000","last_peered":"2024-03-09T15:32:55.781267+0000","last_clean":"2024-03-09T15:32:55.781267+0000","last_became_active":"2024-03-09T14:28:34.314617+0000","last_became_peered":"2024-03-09T14:28:34.314617+0000","last_unstale":"2024-03-09T15:32:55.781267+0000","last_undegraded":"2024-03-09T15:32:55.781267+0000","last_fullsized":"2024-03-09T15:32:55.781267+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:05:14.469183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":493,"num_read_kb":395,"num_write":185,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3a","version":"261'147","reported_seq":680,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.506096+0000","last_change":"2024-03-09T14:29:06.433441+0000","last_active":"2024-03-09T15:36:06.506096+0000","last_peered":"2024-03-09T15:36:06.506096+0000","last_clean":"2024-03-09T15:36:06.506096+0000","last_became_active":"2024-03-09T14:29:06.433145+0000","last_became_peered":"2024-03-09T14:29:06.433145+0000","last_unstale":"2024-03-09T15:36:06.506096+0000","last_undegraded":"2024-03-09T15:36:06.506096+0000","last_fullsized":"2024-03-09T15:36:06.506096+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":100,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:01:58.493721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":6443,"num_write":604,"num_write_kb":33049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"261'274","reported_seq":939,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.185860+0000","last_change":"2024-03-09T14:28:34.314501+0000","last_active":"2024-03-09T15:33:11.185860+0000","last_peered":"2024-03-09T15:33:11.185860+0000","last_clean":"2024-03-09T15:33:11.185860+0000","last_became_active":"2024-03-09T14:28:34.310343+0000","last_became_peered":"2024-03-09T14:28:34.310343+0000","last_unstale":"2024-03-09T15:33:11.185860+0000","last_undegraded":"2024-03-09T15:33:11.185860+0000","last_fullsized":"2024-03-09T15:33:11.185860+0000","mapping_epoch":22,"log_start":"247'200","ondisk_log_start":"247'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":200,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:36:50.027757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1205,"num_read_kb":1028,"num_write":418,"num_write_kb":253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"261'133","reported_seq":689,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.470632+0000","last_change":"2024-03-09T14:28:32.314711+0000","last_active":"2024-03-09T15:36:06.470632+0000","last_peered":"2024-03-09T15:36:06.470632+0000","last_clean":"2024-03-09T15:36:06.470632+0000","last_became_active":"2024-03-09T14:28:32.314258+0000","last_became_peered":"2024-03-09T14:28:32.314258+0000","last_unstale":"2024-03-09T15:36:06.470632+0000","last_undegraded":"2024-03-09T15:36:06.470632+0000","last_fullsized":"2024-03-09T15:36:06.470632+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:21:00.179690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":169,"num_read_kb":17581,"num_write":466,"num_write_kb":22929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3e","version":"261'44","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.189153+0000","last_change":"2024-03-09T14:28:34.323080+0000","last_active":"2024-03-09T15:33:11.189153+0000","last_peered":"2024-03-09T15:33:11.189153+0000","last_clean":"2024-03-09T15:33:11.189153+0000","last_became_active":"2024-03-09T14:28:34.322206+0000","last_became_peered":"2024-03-09T14:28:34.322206+0000","last_unstale":"2024-03-09T15:33:11.189153+0000","last_undegraded":"2024-03-09T15:33:11.189153+0000","last_fullsized":"2024-03-09T15:33:11.189153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:55:41.173766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":201,"num_read_kb":171,"num_write":60,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"261'150","reported_seq":696,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.503852+0000","last_change":"2024-03-09T14:28:32.308873+0000","last_active":"2024-03-09T15:36:06.503852+0000","last_peered":"2024-03-09T15:36:06.503852+0000","last_clean":"2024-03-09T15:36:06.503852+0000","last_became_active":"2024-03-09T14:28:32.308522+0000","last_became_peered":"2024-03-09T14:28:32.308522+0000","last_unstale":"2024-03-09T15:36:06.503852+0000","last_undegraded":"2024-03-09T15:36:06.503852+0000","last_fullsized":"2024-03-09T15:36:06.503852+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:10:42.961664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525396,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":175,"num_read_kb":14130,"num_write":546,"num_write_kb":36844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"261'52","reported_seq":603,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.971792+0000","last_change":"2024-03-09T14:28:34.321740+0000","last_active":"2024-03-09T15:33:10.971792+0000","last_peered":"2024-03-09T15:33:10.971792+0000","last_clean":"2024-03-09T15:33:10.971792+0000","last_became_active":"2024-03-09T14:28:34.320646+0000","last_became_peered":"2024-03-09T14:28:34.320646+0000","last_unstale":"2024-03-09T15:33:10.971792+0000","last_undegraded":"2024-03-09T15:33:10.971792+0000","last_fullsized":"2024-03-09T15:33:10.971792+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:32:54.751077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":174,"num_write":79,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"261'157","reported_seq":741,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.897610+0000","last_change":"2024-03-09T14:28:34.307366+0000","last_active":"2024-03-09T15:31:36.897610+0000","last_peered":"2024-03-09T15:31:36.897610+0000","last_clean":"2024-03-09T15:31:36.897610+0000","last_became_active":"2024-03-09T14:28:34.306592+0000","last_became_peered":"2024-03-09T14:28:34.306592+0000","last_unstale":"2024-03-09T15:31:36.897610+0000","last_undegraded":"2024-03-09T15:31:36.897610+0000","last_fullsized":"2024-03-09T15:31:36.897610+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:38:03.684472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":599,"num_read_kb":473,"num_write":232,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"261'161","reported_seq":713,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.524468+0000","last_change":"2024-03-09T14:29:06.438746+0000","last_active":"2024-03-09T15:36:06.524468+0000","last_peered":"2024-03-09T15:36:06.524468+0000","last_clean":"2024-03-09T15:36:06.524468+0000","last_became_active":"2024-03-09T14:29:06.438523+0000","last_became_peered":"2024-03-09T14:29:06.438523+0000","last_unstale":"2024-03-09T15:36:06.524468+0000","last_undegraded":"2024-03-09T15:36:06.524468+0000","last_fullsized":"2024-03-09T15:36:06.524468+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:59.554428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728801,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":293,"num_read_kb":1034,"num_write":597,"num_write_kb":27895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.31","version":"261'88","reported_seq":642,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.176622+0000","last_change":"2024-03-09T14:28:34.322326+0000","last_active":"2024-03-09T15:31:36.176622+0000","last_peered":"2024-03-09T15:31:36.176622+0000","last_clean":"2024-03-09T15:31:36.176622+0000","last_became_active":"2024-03-09T14:28:34.321352+0000","last_became_peered":"2024-03-09T14:28:34.321352+0000","last_unstale":"2024-03-09T15:31:36.176622+0000","last_undegraded":"2024-03-09T15:31:36.176622+0000","last_fullsized":"2024-03-09T15:31:36.176622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:00:49.569838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":342,"num_read_kb":270,"num_write":126,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"261'131","reported_seq":672,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.455883+0000","last_change":"2024-03-09T14:28:32.337035+0000","last_active":"2024-03-09T15:36:06.455883+0000","last_peered":"2024-03-09T15:36:06.455883+0000","last_clean":"2024-03-09T15:36:06.455883+0000","last_became_active":"2024-03-09T14:28:32.334208+0000","last_became_peered":"2024-03-09T14:28:32.334208+0000","last_unstale":"2024-03-09T15:36:06.455883+0000","last_undegraded":"2024-03-09T15:36:06.455883+0000","last_fullsized":"2024-03-09T15:36:06.455883+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":100,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:15:51.011829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262185,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":143,"num_read_kb":128,"num_write":408,"num_write_kb":22112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"261'62","reported_seq":608,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:46.170836+0000","last_change":"2024-03-09T14:28:34.299187+0000","last_active":"2024-03-09T15:31:46.170836+0000","last_peered":"2024-03-09T15:31:46.170836+0000","last_clean":"2024-03-09T15:31:46.170836+0000","last_became_active":"2024-03-09T14:28:34.298428+0000","last_became_peered":"2024-03-09T14:28:34.298428+0000","last_unstale":"2024-03-09T15:31:46.170836+0000","last_undegraded":"2024-03-09T15:31:46.170836+0000","last_fullsized":"2024-03-09T15:31:46.170836+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:07.549211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":202,"num_write":92,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"261'114","reported_seq":669,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.422432+0000","last_change":"2024-03-09T14:28:32.333004+0000","last_active":"2024-03-09T15:36:06.422432+0000","last_peered":"2024-03-09T15:36:06.422432+0000","last_clean":"2024-03-09T15:36:06.422432+0000","last_became_active":"2024-03-09T14:28:32.318963+0000","last_became_peered":"2024-03-09T14:28:32.318963+0000","last_unstale":"2024-03-09T15:36:06.422432+0000","last_undegraded":"2024-03-09T15:36:06.422432+0000","last_fullsized":"2024-03-09T15:36:06.422432+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:52:04.706747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170436,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":164,"num_read_kb":1700,"num_write":435,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.33","version":"261'102","reported_seq":663,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.950025+0000","last_change":"2024-03-09T14:28:34.320630+0000","last_active":"2024-03-09T15:31:25.950025+0000","last_peered":"2024-03-09T15:31:25.950025+0000","last_clean":"2024-03-09T15:31:25.950025+0000","last_became_active":"2024-03-09T14:28:34.320331+0000","last_became_peered":"2024-03-09T14:28:34.320331+0000","last_unstale":"2024-03-09T15:31:25.950025+0000","last_undegraded":"2024-03-09T15:31:25.950025+0000","last_fullsized":"2024-03-09T15:31:25.950025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:03:19.130527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":320,"num_write":144,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"261'130","reported_seq":682,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.496295+0000","last_change":"2024-03-09T14:28:32.338458+0000","last_active":"2024-03-09T15:36:06.496295+0000","last_peered":"2024-03-09T15:36:06.496295+0000","last_clean":"2024-03-09T15:36:06.496295+0000","last_became_active":"2024-03-09T14:28:32.337934+0000","last_became_peered":"2024-03-09T14:28:32.337934+0000","last_unstale":"2024-03-09T15:36:06.496295+0000","last_undegraded":"2024-03-09T15:36:06.496295+0000","last_fullsized":"2024-03-09T15:36:06.496295+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:15:21.515518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263250,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":163,"num_read_kb":12763,"num_write":465,"num_write_kb":21372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"261'69","reported_seq":605,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.950065+0000","last_change":"2024-03-09T14:28:34.311072+0000","last_active":"2024-03-09T15:31:25.950065+0000","last_peered":"2024-03-09T15:31:25.950065+0000","last_clean":"2024-03-09T15:31:25.950065+0000","last_became_active":"2024-03-09T14:28:34.310703+0000","last_became_peered":"2024-03-09T14:28:34.310703+0000","last_unstale":"2024-03-09T15:31:25.950065+0000","last_undegraded":"2024-03-09T15:31:25.950065+0000","last_fullsized":"2024-03-09T15:31:25.950065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:16:09.385282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":272,"num_read_kb":220,"num_write":109,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"261'157","reported_seq":723,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.503630+0000","last_change":"2024-03-09T14:28:32.337031+0000","last_active":"2024-03-09T15:36:06.503630+0000","last_peered":"2024-03-09T15:36:06.503630+0000","last_clean":"2024-03-09T15:36:06.503630+0000","last_became_active":"2024-03-09T14:28:32.336636+0000","last_became_peered":"2024-03-09T14:28:32.336636+0000","last_unstale":"2024-03-09T15:36:06.503630+0000","last_undegraded":"2024-03-09T15:36:06.503630+0000","last_fullsized":"2024-03-09T15:36:06.503630+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:44:26.611692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262241,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":231,"num_read_kb":12490,"num_write":578,"num_write_kb":28462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"261'87","reported_seq":637,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162736+0000","last_change":"2024-03-09T14:28:34.316215+0000","last_active":"2024-03-09T15:31:26.162736+0000","last_peered":"2024-03-09T15:31:26.162736+0000","last_clean":"2024-03-09T15:31:26.162736+0000","last_became_active":"2024-03-09T14:28:34.314665+0000","last_became_peered":"2024-03-09T14:28:34.314665+0000","last_unstale":"2024-03-09T15:31:26.162736+0000","last_undegraded":"2024-03-09T15:31:26.162736+0000","last_fullsized":"2024-03-09T15:31:26.162736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:15.743713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":335,"num_read_kb":265,"num_write":124,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"261'120","reported_seq":682,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.520234+0000","last_change":"2024-03-09T14:28:32.298846+0000","last_active":"2024-03-09T15:36:06.520234+0000","last_peered":"2024-03-09T15:36:06.520234+0000","last_clean":"2024-03-09T15:36:06.520234+0000","last_became_active":"2024-03-09T14:28:32.298428+0000","last_became_peered":"2024-03-09T14:28:32.298428+0000","last_unstale":"2024-03-09T15:36:06.520234+0000","last_undegraded":"2024-03-09T15:36:06.520234+0000","last_fullsized":"2024-03-09T15:36:06.520234+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":191,"num_read_kb":31150,"num_write":464,"num_write_kb":37586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"261'76","reported_seq":581,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:07.654968+0000","last_change":"2024-03-09T14:28:42.404373+0000","last_active":"2024-03-09T15:35:07.654968+0000","last_peered":"2024-03-09T15:35:07.654968+0000","last_clean":"2024-03-09T15:35:07.654968+0000","last_became_active":"2024-03-09T14:28:42.403553+0000","last_became_peered":"2024-03-09T14:28:42.403553+0000","last_unstale":"2024-03-09T15:35:07.654968+0000","last_undegraded":"2024-03-09T15:35:07.654968+0000","last_fullsized":"2024-03-09T15:35:07.654968+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:16:41.740933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":84,"num_write":184,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"261'98","reported_seq":664,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.780525+0000","last_change":"2024-03-09T14:29:06.439713+0000","last_active":"2024-03-09T15:31:50.780525+0000","last_peered":"2024-03-09T15:31:50.780525+0000","last_clean":"2024-03-09T15:31:50.780525+0000","last_became_active":"2024-03-09T14:29:06.439359+0000","last_became_peered":"2024-03-09T14:29:06.439359+0000","last_unstale":"2024-03-09T15:31:50.780525+0000","last_undegraded":"2024-03-09T15:31:50.780525+0000","last_fullsized":"2024-03-09T15:31:50.780525+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:01:04.305529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":413,"num_read_kb":333,"num_write":149,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"261'165","reported_seq":786,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.496687+0000","last_change":"2024-03-09T14:28:32.330235+0000","last_active":"2024-03-09T15:36:06.496687+0000","last_peered":"2024-03-09T15:36:06.496687+0000","last_clean":"2024-03-09T15:36:06.496687+0000","last_became_active":"2024-03-09T14:28:32.329457+0000","last_became_peered":"2024-03-09T14:28:32.329457+0000","last_unstale":"2024-03-09T15:36:06.496687+0000","last_undegraded":"2024-03-09T15:36:06.496687+0000","last_fullsized":"2024-03-09T15:36:06.496687+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:30:37.270692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506100,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":263,"num_read_kb":141328,"num_write":603,"num_write_kb":37022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.166374+0000","last_change":"2024-03-09T14:28:36.321332+0000","last_active":"2024-03-09T15:31:26.166374+0000","last_peered":"2024-03-09T15:31:26.166374+0000","last_clean":"2024-03-09T15:31:26.166374+0000","last_became_active":"2024-03-09T14:28:36.320603+0000","last_became_peered":"2024-03-09T14:28:36.320603+0000","last_unstale":"2024-03-09T15:31:26.166374+0000","last_undegraded":"2024-03-09T15:31:26.166374+0000","last_fullsized":"2024-03-09T15:31:26.166374+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:20:34.317764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"261'229","reported_seq":840,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.363842+0000","last_change":"2024-03-09T14:28:38.327507+0000","last_active":"2024-03-09T15:36:06.363842+0000","last_peered":"2024-03-09T15:36:06.363842+0000","last_clean":"2024-03-09T15:36:06.363842+0000","last_became_active":"2024-03-09T14:28:38.326826+0000","last_became_peered":"2024-03-09T14:28:38.326826+0000","last_unstale":"2024-03-09T15:36:06.363842+0000","last_undegraded":"2024-03-09T15:36:06.363842+0000","last_fullsized":"2024-03-09T15:36:06.363842+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":200,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11101,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":365,"num_read_kb":621,"num_write":250,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.175558+0000","last_change":"2024-03-09T14:28:40.359830+0000","last_active":"2024-03-09T15:31:31.175558+0000","last_peered":"2024-03-09T15:31:31.175558+0000","last_clean":"2024-03-09T15:31:31.175558+0000","last_became_active":"2024-03-09T14:28:40.358954+0000","last_became_peered":"2024-03-09T14:28:40.358954+0000","last_unstale":"2024-03-09T15:31:31.175558+0000","last_undegraded":"2024-03-09T15:31:31.175558+0000","last_fullsized":"2024-03-09T15:31:31.175558+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:34.169828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"261'770","reported_seq":2383,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:53.987578+0000","last_change":"2024-03-09T14:28:42.395382+0000","last_active":"2024-03-09T15:35:53.987578+0000","last_peered":"2024-03-09T15:35:53.987578+0000","last_clean":"2024-03-09T15:35:53.987578+0000","last_became_active":"2024-03-09T14:28:42.394691+0000","last_became_peered":"2024-03-09T14:28:42.394691+0000","last_unstale":"2024-03-09T15:35:53.987578+0000","last_undegraded":"2024-03-09T15:35:53.987578+0000","last_fullsized":"2024-03-09T15:35:53.987578+0000","mapping_epoch":30,"log_start":"261'700","ondisk_log_start":"261'700","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":700,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:48.486613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":612,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2611,"num_read_kb":2601,"num_write":1560,"num_write_kb":569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"261'40","reported_seq":578,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:28.764187+0000","last_change":"2024-03-09T14:28:34.326083+0000","last_active":"2024-03-09T15:31:28.764187+0000","last_peered":"2024-03-09T15:31:28.764187+0000","last_clean":"2024-03-09T15:31:28.764187+0000","last_became_active":"2024-03-09T14:28:34.324569+0000","last_became_peered":"2024-03-09T14:28:34.324569+0000","last_unstale":"2024-03-09T15:31:28.764187+0000","last_undegraded":"2024-03-09T15:31:28.764187+0000","last_fullsized":"2024-03-09T15:31:28.764187+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:18.214769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":168,"num_read_kb":136,"num_write":58,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"261'169","reported_seq":740,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.528014+0000","last_change":"2024-03-09T14:28:32.310039+0000","last_active":"2024-03-09T15:36:06.528014+0000","last_peered":"2024-03-09T15:36:06.528014+0000","last_clean":"2024-03-09T15:36:06.528014+0000","last_became_active":"2024-03-09T14:28:32.309536+0000","last_became_peered":"2024-03-09T14:28:32.309536+0000","last_unstale":"2024-03-09T15:36:06.528014+0000","last_undegraded":"2024-03-09T15:36:06.528014+0000","last_fullsized":"2024-03-09T15:36:06.528014+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":100,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":29533,"num_write":652,"num_write_kb":30801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.780570+0000","last_change":"2024-03-09T14:28:36.320450+0000","last_active":"2024-03-09T15:31:50.780570+0000","last_peered":"2024-03-09T15:31:50.780570+0000","last_clean":"2024-03-09T15:31:50.780570+0000","last_became_active":"2024-03-09T14:28:36.320079+0000","last_became_peered":"2024-03-09T14:28:36.320079+0000","last_unstale":"2024-03-09T15:31:50.780570+0000","last_undegraded":"2024-03-09T15:31:50.780570+0000","last_fullsized":"2024-03-09T15:31:50.780570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:19:38.894047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"261'87","reported_seq":623,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:52.371528+0000","last_change":"2024-03-09T14:28:38.341864+0000","last_active":"2024-03-09T15:31:52.371528+0000","last_peered":"2024-03-09T15:31:52.371528+0000","last_clean":"2024-03-09T15:31:52.371528+0000","last_became_active":"2024-03-09T14:28:38.340896+0000","last_became_peered":"2024-03-09T14:28:38.340896+0000","last_unstale":"2024-03-09T15:31:52.371528+0000","last_undegraded":"2024-03-09T15:31:52.371528+0000","last_fullsized":"2024-03-09T15:31:52.371528+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:19:17.825977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":101,"num_write":68,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.163891+0000","last_change":"2024-03-09T14:28:40.371660+0000","last_active":"2024-03-09T15:31:31.163891+0000","last_peered":"2024-03-09T15:31:31.163891+0000","last_clean":"2024-03-09T15:31:31.163891+0000","last_became_active":"2024-03-09T14:28:40.370788+0000","last_became_peered":"2024-03-09T14:28:40.370788+0000","last_unstale":"2024-03-09T15:31:31.163891+0000","last_undegraded":"2024-03-09T15:31:31.163891+0000","last_fullsized":"2024-03-09T15:31:31.163891+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:12:43.937289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"261'59","reported_seq":562,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:52.547715+0000","last_change":"2024-03-09T14:28:42.411392+0000","last_active":"2024-03-09T15:33:52.547715+0000","last_peered":"2024-03-09T15:33:52.547715+0000","last_clean":"2024-03-09T15:33:52.547715+0000","last_became_active":"2024-03-09T14:28:42.408738+0000","last_became_peered":"2024-03-09T14:28:42.408738+0000","last_unstale":"2024-03-09T15:33:52.547715+0000","last_undegraded":"2024-03-09T15:33:52.547715+0000","last_fullsized":"2024-03-09T15:33:52.547715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":681,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":76,"num_read_kb":75,"num_write":140,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"261'157","reported_seq":746,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.016465+0000","last_change":"2024-03-09T14:28:34.305220+0000","last_active":"2024-03-09T15:33:51.016465+0000","last_peered":"2024-03-09T15:33:51.016465+0000","last_clean":"2024-03-09T15:33:51.016465+0000","last_became_active":"2024-03-09T14:28:34.304892+0000","last_became_peered":"2024-03-09T14:28:34.304892+0000","last_unstale":"2024-03-09T15:33:51.016465+0000","last_undegraded":"2024-03-09T15:33:51.016465+0000","last_fullsized":"2024-03-09T15:33:51.016465+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":622,"num_read_kb":492,"num_write":239,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"261'161","reported_seq":861,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.510886+0000","last_change":"2024-03-09T14:28:32.319206+0000","last_active":"2024-03-09T15:36:06.510886+0000","last_peered":"2024-03-09T15:36:06.510886+0000","last_clean":"2024-03-09T15:36:06.510886+0000","last_became_active":"2024-03-09T14:28:32.318428+0000","last_became_peered":"2024-03-09T14:28:32.318428+0000","last_unstale":"2024-03-09T15:36:06.510886+0000","last_undegraded":"2024-03-09T15:36:06.510886+0000","last_fullsized":"2024-03-09T15:36:06.510886+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:59:15.432941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505054,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":544,"num_read_kb":113871,"num_write":619,"num_write_kb":46622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.774459+0000","last_change":"2024-03-09T14:28:36.320826+0000","last_active":"2024-03-09T15:31:25.774459+0000","last_peered":"2024-03-09T15:31:25.774459+0000","last_clean":"2024-03-09T15:31:25.774459+0000","last_became_active":"2024-03-09T14:28:36.320095+0000","last_became_peered":"2024-03-09T14:28:36.320095+0000","last_unstale":"2024-03-09T15:31:25.774459+0000","last_undegraded":"2024-03-09T15:31:25.774459+0000","last_fullsized":"2024-03-09T15:31:25.774459+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:58:26.878574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"261'103","reported_seq":653,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:02.768040+0000","last_change":"2024-03-09T14:28:38.339553+0000","last_active":"2024-03-09T15:34:02.768040+0000","last_peered":"2024-03-09T15:34:02.768040+0000","last_clean":"2024-03-09T15:34:02.768040+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:34:02.768040+0000","last_undegraded":"2024-03-09T15:34:02.768040+0000","last_fullsized":"2024-03-09T15:34:02.768040+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:37:31.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"261'505","reported_seq":2291,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.123365+0000","last_change":"2024-03-09T14:28:40.373218+0000","last_active":"2024-03-09T15:36:06.123365+0000","last_peered":"2024-03-09T15:36:06.123365+0000","last_clean":"2024-03-09T15:36:06.123365+0000","last_became_active":"2024-03-09T14:28:40.371507+0000","last_became_peered":"2024-03-09T14:28:40.371507+0000","last_unstale":"2024-03-09T15:36:06.123365+0000","last_undegraded":"2024-03-09T15:36:06.123365+0000","last_fullsized":"2024-03-09T15:36:06.123365+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":400,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:20:18.095183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"261'136","reported_seq":941,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:16.160267+0000","last_change":"2024-03-09T14:28:42.399490+0000","last_active":"2024-03-09T15:31:16.160267+0000","last_peered":"2024-03-09T15:31:16.160267+0000","last_clean":"2024-03-09T15:31:16.160267+0000","last_became_active":"2024-03-09T14:28:42.398498+0000","last_became_peered":"2024-03-09T14:28:42.398498+0000","last_unstale":"2024-03-09T15:31:16.160267+0000","last_undegraded":"2024-03-09T15:31:16.160267+0000","last_fullsized":"2024-03-09T15:31:16.160267+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:10:05.573339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":498,"num_read_kb":498,"num_write":277,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"261'53","reported_seq":592,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:25.774411+0000","last_change":"2024-03-09T14:28:34.330792+0000","last_active":"2024-03-09T15:31:25.774411+0000","last_peered":"2024-03-09T15:31:25.774411+0000","last_clean":"2024-03-09T15:31:25.774411+0000","last_became_active":"2024-03-09T14:28:34.330347+0000","last_became_peered":"2024-03-09T14:28:34.330347+0000","last_unstale":"2024-03-09T15:31:25.774411+0000","last_undegraded":"2024-03-09T15:31:25.774411+0000","last_fullsized":"2024-03-09T15:31:25.774411+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:55:20.226431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":212,"num_read_kb":171,"num_write":77,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"261'166","reported_seq":738,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.488660+0000","last_change":"2024-03-09T14:28:32.332979+0000","last_active":"2024-03-09T15:36:06.488660+0000","last_peered":"2024-03-09T15:36:06.488660+0000","last_clean":"2024-03-09T15:36:06.488660+0000","last_became_active":"2024-03-09T14:28:32.319819+0000","last_became_peered":"2024-03-09T14:28:32.319819+0000","last_unstale":"2024-03-09T15:36:06.488660+0000","last_undegraded":"2024-03-09T15:36:06.488660+0000","last_fullsized":"2024-03-09T15:36:06.488660+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":100,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:01:46.540432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5244508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":231,"num_read_kb":15567,"num_write":579,"num_write_kb":34195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.166450+0000","last_change":"2024-03-09T14:28:36.311535+0000","last_active":"2024-03-09T15:31:26.166450+0000","last_peered":"2024-03-09T15:31:26.166450+0000","last_clean":"2024-03-09T15:31:26.166450+0000","last_became_active":"2024-03-09T14:28:36.310231+0000","last_became_peered":"2024-03-09T14:28:36.310231+0000","last_unstale":"2024-03-09T15:31:26.166450+0000","last_undegraded":"2024-03-09T15:31:26.166450+0000","last_fullsized":"2024-03-09T15:31:26.166450+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:55:50.175612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"261'170","reported_seq":753,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.278596+0000","last_change":"2024-03-09T14:28:38.599078+0000","last_active":"2024-03-09T15:36:06.278596+0000","last_peered":"2024-03-09T15:36:06.278596+0000","last_clean":"2024-03-09T15:36:06.278596+0000","last_became_active":"2024-03-09T14:28:38.598308+0000","last_became_peered":"2024-03-09T14:28:38.598308+0000","last_unstale":"2024-03-09T15:36:06.278596+0000","last_undegraded":"2024-03-09T15:36:06.278596+0000","last_fullsized":"2024-03-09T15:36:06.278596+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":100,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:10:09.151032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8724,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":532,"num_write":190,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"261'513","reported_seq":2499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.736212+0000","last_change":"2024-03-09T14:28:40.388710+0000","last_active":"2024-03-09T15:36:05.736212+0000","last_peered":"2024-03-09T15:36:05.736212+0000","last_clean":"2024-03-09T15:36:05.736212+0000","last_became_active":"2024-03-09T14:28:40.387990+0000","last_became_peered":"2024-03-09T14:28:40.387990+0000","last_unstale":"2024-03-09T15:36:05.736212+0000","last_undegraded":"2024-03-09T15:36:05.736212+0000","last_fullsized":"2024-03-09T15:36:05.736212+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":500,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:21:47.017030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"261'101","reported_seq":678,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194321+0000","last_change":"2024-03-09T14:28:34.323491+0000","last_active":"2024-03-09T15:33:36.194321+0000","last_peered":"2024-03-09T15:33:36.194321+0000","last_clean":"2024-03-09T15:33:36.194321+0000","last_became_active":"2024-03-09T14:28:34.322731+0000","last_became_peered":"2024-03-09T14:28:34.322731+0000","last_unstale":"2024-03-09T15:33:36.194321+0000","last_undegraded":"2024-03-09T15:33:36.194321+0000","last_fullsized":"2024-03-09T15:33:36.194321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:34:36.142132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":330,"num_write":141,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"261'123","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.505667+0000","last_change":"2024-03-09T14:28:32.329755+0000","last_active":"2024-03-09T15:36:06.505667+0000","last_peered":"2024-03-09T15:36:06.505667+0000","last_clean":"2024-03-09T15:36:06.505667+0000","last_became_active":"2024-03-09T14:28:32.329148+0000","last_became_peered":"2024-03-09T14:28:32.329148+0000","last_unstale":"2024-03-09T15:36:06.505667+0000","last_undegraded":"2024-03-09T15:36:06.505667+0000","last_fullsized":"2024-03-09T15:36:06.505667+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":100,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:07:39.356330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7947490,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":1766,"num_write":513,"num_write_kb":18289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205246+0000","last_change":"2024-03-09T14:28:36.314780+0000","last_active":"2024-03-09T15:33:36.205246+0000","last_peered":"2024-03-09T15:33:36.205246+0000","last_clean":"2024-03-09T15:33:36.205246+0000","last_became_active":"2024-03-09T14:28:36.313904+0000","last_became_peered":"2024-03-09T14:28:36.313904+0000","last_unstale":"2024-03-09T15:33:36.205246+0000","last_undegraded":"2024-03-09T15:33:36.205246+0000","last_fullsized":"2024-03-09T15:33:36.205246+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:57:41.431102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"261'268","reported_seq":903,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.431671+0000","last_change":"2024-03-09T14:28:38.598064+0000","last_active":"2024-03-09T15:36:06.431671+0000","last_peered":"2024-03-09T15:36:06.431671+0000","last_clean":"2024-03-09T15:36:06.431671+0000","last_became_active":"2024-03-09T14:28:38.597247+0000","last_became_peered":"2024-03-09T14:28:38.597247+0000","last_unstale":"2024-03-09T15:36:06.431671+0000","last_undegraded":"2024-03-09T15:36:06.431671+0000","last_fullsized":"2024-03-09T15:36:06.431671+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":200,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:35:30.643121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9398,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":421,"num_read_kb":664,"num_write":284,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.967946+0000","last_change":"2024-03-09T14:28:40.360962+0000","last_active":"2024-03-09T15:33:55.967946+0000","last_peered":"2024-03-09T15:33:55.967946+0000","last_clean":"2024-03-09T15:33:55.967946+0000","last_became_active":"2024-03-09T14:28:40.359906+0000","last_became_peered":"2024-03-09T14:28:40.359906+0000","last_unstale":"2024-03-09T15:33:55.967946+0000","last_undegraded":"2024-03-09T15:33:55.967946+0000","last_fullsized":"2024-03-09T15:33:55.967946+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:33:27.073677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"261'518","reported_seq":1356,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.729973+0000","last_change":"2024-03-09T14:28:42.403475+0000","last_active":"2024-03-09T15:35:58.729973+0000","last_peered":"2024-03-09T15:35:58.729973+0000","last_clean":"2024-03-09T15:35:58.729973+0000","last_became_active":"2024-03-09T14:28:42.402739+0000","last_became_peered":"2024-03-09T14:28:42.402739+0000","last_unstale":"2024-03-09T15:35:58.729973+0000","last_undegraded":"2024-03-09T15:35:58.729973+0000","last_fullsized":"2024-03-09T15:35:58.729973+0000","mapping_epoch":30,"log_start":"261'500","ondisk_log_start":"261'500","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":500,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:49:57.794992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1336,"num_read_kb":1311,"num_write":1083,"num_write_kb":376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"261'92","reported_seq":662,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191097+0000","last_change":"2024-03-09T14:28:34.302153+0000","last_active":"2024-03-09T15:33:36.191097+0000","last_peered":"2024-03-09T15:33:36.191097+0000","last_clean":"2024-03-09T15:33:36.191097+0000","last_became_active":"2024-03-09T14:28:34.300795+0000","last_became_peered":"2024-03-09T14:28:34.300795+0000","last_unstale":"2024-03-09T15:33:36.191097+0000","last_undegraded":"2024-03-09T15:33:36.191097+0000","last_fullsized":"2024-03-09T15:33:36.191097+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:27:31.948784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":361,"num_read_kb":288,"num_write":150,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"261'142","reported_seq":715,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.444592+0000","last_change":"2024-03-09T14:28:32.318853+0000","last_active":"2024-03-09T15:36:06.444592+0000","last_peered":"2024-03-09T15:36:06.444592+0000","last_clean":"2024-03-09T15:36:06.444592+0000","last_became_active":"2024-03-09T14:28:32.318111+0000","last_became_peered":"2024-03-09T14:28:32.318111+0000","last_unstale":"2024-03-09T15:36:06.444592+0000","last_undegraded":"2024-03-09T15:36:06.444592+0000","last_fullsized":"2024-03-09T15:36:06.444592+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":100,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:51.120639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242921,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":6230,"num_write":597,"num_write_kb":19779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.976641+0000","last_change":"2024-03-09T14:28:36.301940+0000","last_active":"2024-03-09T15:33:35.976641+0000","last_peered":"2024-03-09T15:33:35.976641+0000","last_clean":"2024-03-09T15:33:35.976641+0000","last_became_active":"2024-03-09T14:28:36.301227+0000","last_became_peered":"2024-03-09T14:28:36.301227+0000","last_unstale":"2024-03-09T15:33:35.976641+0000","last_undegraded":"2024-03-09T15:33:35.976641+0000","last_fullsized":"2024-03-09T15:33:35.976641+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:55:41.843139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"261'173","reported_seq":750,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.059201+0000","last_change":"2024-03-09T14:28:38.347678+0000","last_active":"2024-03-09T15:36:06.059201+0000","last_peered":"2024-03-09T15:36:06.059201+0000","last_clean":"2024-03-09T15:36:06.059201+0000","last_became_active":"2024-03-09T14:28:38.346837+0000","last_became_peered":"2024-03-09T14:28:38.346837+0000","last_unstale":"2024-03-09T15:36:06.059201+0000","last_undegraded":"2024-03-09T15:36:06.059201+0000","last_fullsized":"2024-03-09T15:36:06.059201+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":100,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:06.776774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16267,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":276,"num_read_kb":528,"num_write":188,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.808565+0000","last_change":"2024-03-09T14:28:40.391285+0000","last_active":"2024-03-09T15:33:55.808565+0000","last_peered":"2024-03-09T15:33:55.808565+0000","last_clean":"2024-03-09T15:33:55.808565+0000","last_became_active":"2024-03-09T14:28:40.389175+0000","last_became_peered":"2024-03-09T14:28:40.389175+0000","last_unstale":"2024-03-09T15:33:55.808565+0000","last_undegraded":"2024-03-09T15:33:55.808565+0000","last_fullsized":"2024-03-09T15:33:55.808565+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:33:43.955165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"261'58","reported_seq":553,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:02.730159+0000","last_change":"2024-03-09T14:28:42.401990+0000","last_active":"2024-03-09T15:34:02.730159+0000","last_peered":"2024-03-09T15:34:02.730159+0000","last_clean":"2024-03-09T15:34:02.730159+0000","last_became_active":"2024-03-09T14:28:42.401214+0000","last_became_peered":"2024-03-09T14:28:42.401214+0000","last_unstale":"2024-03-09T15:34:02.730159+0000","last_undegraded":"2024-03-09T15:34:02.730159+0000","last_fullsized":"2024-03-09T15:34:02.730159+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:24:20.023880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":47,"num_write":140,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"261'78","reported_seq":634,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.793908+0000","last_change":"2024-03-09T14:28:34.317931+0000","last_active":"2024-03-09T15:33:55.793908+0000","last_peered":"2024-03-09T15:33:55.793908+0000","last_clean":"2024-03-09T15:33:55.793908+0000","last_became_active":"2024-03-09T14:28:34.316812+0000","last_became_peered":"2024-03-09T14:28:34.316812+0000","last_unstale":"2024-03-09T15:33:55.793908+0000","last_undegraded":"2024-03-09T15:33:55.793908+0000","last_fullsized":"2024-03-09T15:33:55.793908+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:23:12.743671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":320,"num_read_kb":256,"num_write":116,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"261'108","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.472373+0000","last_change":"2024-03-09T14:29:06.446958+0000","last_active":"2024-03-09T15:36:06.472373+0000","last_peered":"2024-03-09T15:36:06.472373+0000","last_clean":"2024-03-09T15:36:06.472373+0000","last_became_active":"2024-03-09T14:29:06.446333+0000","last_became_peered":"2024-03-09T14:29:06.446333+0000","last_unstale":"2024-03-09T15:36:06.472373+0000","last_undegraded":"2024-03-09T15:36:06.472373+0000","last_fullsized":"2024-03-09T15:36:06.472373+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:09:44.978410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":147,"num_read_kb":1025,"num_write":402,"num_write_kb":11167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.980194+0000","last_change":"2024-03-09T14:28:36.304298+0000","last_active":"2024-03-09T15:33:50.980194+0000","last_peered":"2024-03-09T15:33:50.980194+0000","last_clean":"2024-03-09T15:33:50.980194+0000","last_became_active":"2024-03-09T14:28:36.303695+0000","last_became_peered":"2024-03-09T14:28:36.303695+0000","last_unstale":"2024-03-09T15:33:50.980194+0000","last_undegraded":"2024-03-09T15:33:50.980194+0000","last_fullsized":"2024-03-09T15:33:50.980194+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:16:14.481520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"261'220","reported_seq":828,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.462677+0000","last_change":"2024-03-09T14:28:38.340119+0000","last_active":"2024-03-09T15:36:06.462677+0000","last_peered":"2024-03-09T15:36:06.462677+0000","last_clean":"2024-03-09T15:36:06.462677+0000","last_became_active":"2024-03-09T14:28:38.339300+0000","last_became_peered":"2024-03-09T14:28:38.339300+0000","last_unstale":"2024-03-09T15:36:06.462677+0000","last_undegraded":"2024-03-09T15:36:06.462677+0000","last_fullsized":"2024-03-09T15:36:06.462677+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":200,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:56:14.245930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8477,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":353,"num_read_kb":624,"num_write":244,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"261'501","reported_seq":2511,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.541827+0000","last_change":"2024-03-09T14:28:40.368533+0000","last_active":"2024-03-09T15:36:06.541827+0000","last_peered":"2024-03-09T15:36:06.541827+0000","last_clean":"2024-03-09T15:36:06.541827+0000","last_became_active":"2024-03-09T14:28:40.367119+0000","last_became_peered":"2024-03-09T14:28:40.367119+0000","last_unstale":"2024-03-09T15:36:06.541827+0000","last_undegraded":"2024-03-09T15:36:06.541827+0000","last_fullsized":"2024-03-09T15:36:06.541827+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":400,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"261'81","reported_seq":576,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:56.045022+0000","last_change":"2024-03-09T14:28:42.397381+0000","last_active":"2024-03-09T15:35:56.045022+0000","last_peered":"2024-03-09T15:35:56.045022+0000","last_clean":"2024-03-09T15:35:56.045022+0000","last_became_active":"2024-03-09T14:28:42.396389+0000","last_became_peered":"2024-03-09T14:28:42.396389+0000","last_unstale":"2024-03-09T15:35:56.045022+0000","last_undegraded":"2024-03-09T15:35:56.045022+0000","last_fullsized":"2024-03-09T15:35:56.045022+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":65,"num_write":200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"261'44","reported_seq":583,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.046758+0000","last_change":"2024-03-09T14:28:34.298294+0000","last_active":"2024-03-09T15:36:06.046758+0000","last_peered":"2024-03-09T15:36:06.046758+0000","last_clean":"2024-03-09T15:36:06.046758+0000","last_became_active":"2024-03-09T14:28:34.297080+0000","last_became_peered":"2024-03-09T14:28:34.297080+0000","last_unstale":"2024-03-09T15:36:06.046758+0000","last_undegraded":"2024-03-09T15:36:06.046758+0000","last_fullsized":"2024-03-09T15:36:06.046758+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":189,"num_read_kb":155,"num_write":63,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"261'147","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.509373+0000","last_change":"2024-03-09T14:29:06.445970+0000","last_active":"2024-03-09T15:36:06.509373+0000","last_peered":"2024-03-09T15:36:06.509373+0000","last_clean":"2024-03-09T15:36:06.509373+0000","last_became_active":"2024-03-09T14:29:06.445573+0000","last_became_peered":"2024-03-09T14:29:06.445573+0000","last_unstale":"2024-03-09T15:36:06.509373+0000","last_undegraded":"2024-03-09T15:36:06.509373+0000","last_fullsized":"2024-03-09T15:36:06.509373+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":100,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:30:46.074402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":204,"num_read_kb":157,"num_write":458,"num_write_kb":32761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.980070+0000","last_change":"2024-03-09T14:28:36.323155+0000","last_active":"2024-03-09T15:33:50.980070+0000","last_peered":"2024-03-09T15:33:50.980070+0000","last_clean":"2024-03-09T15:33:50.980070+0000","last_became_active":"2024-03-09T14:28:36.322638+0000","last_became_peered":"2024-03-09T14:28:36.322638+0000","last_unstale":"2024-03-09T15:33:50.980070+0000","last_undegraded":"2024-03-09T15:33:50.980070+0000","last_fullsized":"2024-03-09T15:33:50.980070+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:45:20.380783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"261'318","reported_seq":962,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.499196+0000","last_change":"2024-03-09T14:28:38.329540+0000","last_active":"2024-03-09T15:36:06.499196+0000","last_peered":"2024-03-09T15:36:06.499196+0000","last_clean":"2024-03-09T15:36:06.499196+0000","last_became_active":"2024-03-09T14:28:38.328897+0000","last_became_peered":"2024-03-09T14:28:38.328897+0000","last_unstale":"2024-03-09T15:36:06.499196+0000","last_undegraded":"2024-03-09T15:36:06.499196+0000","last_fullsized":"2024-03-09T15:36:06.499196+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":300,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":543,"num_read_kb":1162,"num_write":388,"num_write_kb":277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.977668+0000","last_change":"2024-03-09T14:28:40.384571+0000","last_active":"2024-03-09T15:33:40.977668+0000","last_peered":"2024-03-09T15:33:40.977668+0000","last_clean":"2024-03-09T15:33:40.977668+0000","last_became_active":"2024-03-09T14:28:40.383758+0000","last_became_peered":"2024-03-09T14:28:40.383758+0000","last_unstale":"2024-03-09T15:33:40.977668+0000","last_undegraded":"2024-03-09T15:33:40.977668+0000","last_fullsized":"2024-03-09T15:33:40.977668+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:24:41.089698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"261'54","reported_seq":565,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:45.965658+0000","last_change":"2024-03-09T14:28:42.402684+0000","last_active":"2024-03-09T15:33:45.965658+0000","last_peered":"2024-03-09T15:33:45.965658+0000","last_clean":"2024-03-09T15:33:45.965658+0000","last_became_active":"2024-03-09T14:28:42.401880+0000","last_became_peered":"2024-03-09T14:28:42.401880+0000","last_unstale":"2024-03-09T15:33:45.965658+0000","last_undegraded":"2024-03-09T15:33:45.965658+0000","last_fullsized":"2024-03-09T15:33:45.965658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:29:59.644550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":69,"num_write":120,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"261'79","reported_seq":639,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195621+0000","last_change":"2024-03-09T14:28:34.316156+0000","last_active":"2024-03-09T15:33:51.195621+0000","last_peered":"2024-03-09T15:33:51.195621+0000","last_clean":"2024-03-09T15:33:51.195621+0000","last_became_active":"2024-03-09T14:28:34.315432+0000","last_became_peered":"2024-03-09T14:28:34.315432+0000","last_unstale":"2024-03-09T15:33:51.195621+0000","last_undegraded":"2024-03-09T15:33:51.195621+0000","last_fullsized":"2024-03-09T15:33:51.195621+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:08:27.256304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":315,"num_read_kb":251,"num_write":126,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"261'157","reported_seq":695,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.518916+0000","last_change":"2024-03-09T14:28:32.300326+0000","last_active":"2024-03-09T15:36:06.518916+0000","last_peered":"2024-03-09T15:36:06.518916+0000","last_clean":"2024-03-09T15:36:06.518916+0000","last_became_active":"2024-03-09T14:28:32.299897+0000","last_became_peered":"2024-03-09T14:28:32.299897+0000","last_unstale":"2024-03-09T15:36:06.518916+0000","last_undegraded":"2024-03-09T15:36:06.518916+0000","last_fullsized":"2024-03-09T15:36:06.518916+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:25:33.818560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506226,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":160,"num_read_kb":225,"num_write":553,"num_write_kb":26448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.980312+0000","last_change":"2024-03-09T14:28:36.320815+0000","last_active":"2024-03-09T15:33:50.980312+0000","last_peered":"2024-03-09T15:33:50.980312+0000","last_clean":"2024-03-09T15:33:50.980312+0000","last_became_active":"2024-03-09T14:28:36.320029+0000","last_became_peered":"2024-03-09T14:28:36.320029+0000","last_unstale":"2024-03-09T15:33:50.980312+0000","last_undegraded":"2024-03-09T15:33:50.980312+0000","last_fullsized":"2024-03-09T15:33:50.980312+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:23:44.855859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"261'208","reported_seq":800,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.148910+0000","last_change":"2024-03-09T14:28:38.339497+0000","last_active":"2024-03-09T15:36:06.148910+0000","last_peered":"2024-03-09T15:36:06.148910+0000","last_clean":"2024-03-09T15:36:06.148910+0000","last_became_active":"2024-03-09T14:28:38.338658+0000","last_became_peered":"2024-03-09T14:28:38.338658+0000","last_unstale":"2024-03-09T15:36:06.148910+0000","last_undegraded":"2024-03-09T15:36:06.148910+0000","last_fullsized":"2024-03-09T15:36:06.148910+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":100,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:42:50.615803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12662,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":352,"num_read_kb":613,"num_write":292,"num_write_kb":188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":495,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:11.021392+0000","last_change":"2024-03-09T14:28:40.379699+0000","last_active":"2024-03-09T15:34:11.021392+0000","last_peered":"2024-03-09T15:34:11.021392+0000","last_clean":"2024-03-09T15:34:11.021392+0000","last_became_active":"2024-03-09T14:28:40.378973+0000","last_became_peered":"2024-03-09T14:28:40.378973+0000","last_unstale":"2024-03-09T15:34:11.021392+0000","last_undegraded":"2024-03-09T15:34:11.021392+0000","last_fullsized":"2024-03-09T15:34:11.021392+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"261'42","reported_seq":535,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:00.822318+0000","last_change":"2024-03-09T14:28:42.398431+0000","last_active":"2024-03-09T15:35:00.822318+0000","last_peered":"2024-03-09T15:35:00.822318+0000","last_clean":"2024-03-09T15:35:00.822318+0000","last_became_active":"2024-03-09T14:28:42.397533+0000","last_became_peered":"2024-03-09T14:28:42.397533+0000","last_unstale":"2024-03-09T15:35:00.822318+0000","last_undegraded":"2024-03-09T15:35:00.822318+0000","last_fullsized":"2024-03-09T15:35:00.822318+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:54:11.336171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":91,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"261'56","reported_seq":616,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195764+0000","last_change":"2024-03-09T14:28:34.315224+0000","last_active":"2024-03-09T15:33:51.195764+0000","last_peered":"2024-03-09T15:33:51.195764+0000","last_clean":"2024-03-09T15:33:51.195764+0000","last_became_active":"2024-03-09T14:28:34.313475+0000","last_became_peered":"2024-03-09T14:28:34.313475+0000","last_unstale":"2024-03-09T15:33:51.195764+0000","last_undegraded":"2024-03-09T15:33:51.195764+0000","last_fullsized":"2024-03-09T15:33:51.195764+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:37:57.784425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":198,"num_write":78,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"261'129","reported_seq":681,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.475031+0000","last_change":"2024-03-09T14:28:32.319640+0000","last_active":"2024-03-09T15:36:06.475031+0000","last_peered":"2024-03-09T15:36:06.475031+0000","last_clean":"2024-03-09T15:36:06.475031+0000","last_became_active":"2024-03-09T14:28:32.316406+0000","last_became_peered":"2024-03-09T14:28:32.316406+0000","last_unstale":"2024-03-09T15:36:06.475031+0000","last_undegraded":"2024-03-09T15:36:06.475031+0000","last_fullsized":"2024-03-09T15:36:06.475031+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:33:19.144976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655835,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":162,"num_read_kb":15736,"num_write":466,"num_write_kb":13578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.966668+0000","last_change":"2024-03-09T14:28:36.319712+0000","last_active":"2024-03-09T15:33:50.966668+0000","last_peered":"2024-03-09T15:33:50.966668+0000","last_clean":"2024-03-09T15:33:50.966668+0000","last_became_active":"2024-03-09T14:28:36.319210+0000","last_became_peered":"2024-03-09T14:28:36.319210+0000","last_unstale":"2024-03-09T15:33:50.966668+0000","last_undegraded":"2024-03-09T15:33:50.966668+0000","last_fullsized":"2024-03-09T15:33:50.966668+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:26:56.771418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"261'217","reported_seq":821,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.969548+0000","last_change":"2024-03-09T14:28:38.351282+0000","last_active":"2024-03-09T15:36:05.969548+0000","last_peered":"2024-03-09T15:36:05.969548+0000","last_clean":"2024-03-09T15:36:05.969548+0000","last_became_active":"2024-03-09T14:28:38.350390+0000","last_became_peered":"2024-03-09T14:28:38.350390+0000","last_unstale":"2024-03-09T15:36:05.969548+0000","last_undegraded":"2024-03-09T15:36:05.969548+0000","last_fullsized":"2024-03-09T15:36:05.969548+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":200,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:30:58.004755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9540,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":338,"num_read_kb":580,"num_write":230,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"261'474","reported_seq":2359,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.537659+0000","last_change":"2024-03-09T14:28:40.374487+0000","last_active":"2024-03-09T15:36:06.537659+0000","last_peered":"2024-03-09T15:36:06.537659+0000","last_clean":"2024-03-09T15:36:06.537659+0000","last_became_active":"2024-03-09T14:28:40.373360+0000","last_became_peered":"2024-03-09T14:28:40.373360+0000","last_unstale":"2024-03-09T15:36:06.537659+0000","last_undegraded":"2024-03-09T15:36:06.537659+0000","last_fullsized":"2024-03-09T15:36:06.537659+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":400,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:24:48.737477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"261'104","reported_seq":650,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:52.932955+0000","last_change":"2024-03-09T14:28:42.409080+0000","last_active":"2024-03-09T15:34:52.932955+0000","last_peered":"2024-03-09T15:34:52.932955+0000","last_clean":"2024-03-09T15:34:52.932955+0000","last_became_active":"2024-03-09T14:28:42.408233+0000","last_became_peered":"2024-03-09T14:28:42.408233+0000","last_unstale":"2024-03-09T15:34:52.932955+0000","last_undegraded":"2024-03-09T15:34:52.932955+0000","last_fullsized":"2024-03-09T15:34:52.932955+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:59:27.171699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":274,"num_read_kb":227,"num_write":264,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"261'72","reported_seq":621,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195874+0000","last_change":"2024-03-09T14:28:34.310932+0000","last_active":"2024-03-09T15:33:51.195874+0000","last_peered":"2024-03-09T15:33:51.195874+0000","last_clean":"2024-03-09T15:33:51.195874+0000","last_became_active":"2024-03-09T14:28:34.310061+0000","last_became_peered":"2024-03-09T14:28:34.310061+0000","last_unstale":"2024-03-09T15:33:51.195874+0000","last_undegraded":"2024-03-09T15:33:51.195874+0000","last_fullsized":"2024-03-09T15:33:51.195874+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:56:23.692037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":294,"num_read_kb":233,"num_write":114,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"261'166","reported_seq":773,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.510416+0000","last_change":"2024-03-09T14:28:32.313964+0000","last_active":"2024-03-09T15:36:06.510416+0000","last_peered":"2024-03-09T15:36:06.510416+0000","last_clean":"2024-03-09T15:36:06.510416+0000","last_became_active":"2024-03-09T14:28:32.313566+0000","last_became_peered":"2024-03-09T14:28:32.313566+0000","last_unstale":"2024-03-09T15:36:06.510416+0000","last_undegraded":"2024-03-09T15:36:06.510416+0000","last_fullsized":"2024-03-09T15:36:06.510416+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":100,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:06:35.181059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":245,"num_read_kb":84880,"num_write":567,"num_write_kb":19718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"32'7","reported_seq":1459,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.269380+0000","last_change":"2024-03-09T14:28:36.327203+0000","last_active":"2024-03-09T15:36:05.269380+0000","last_peered":"2024-03-09T15:36:05.269380+0000","last_clean":"2024-03-09T15:36:05.269380+0000","last_became_active":"2024-03-09T14:28:36.325820+0000","last_became_peered":"2024-03-09T14:28:36.325820+0000","last_unstale":"2024-03-09T15:36:05.269380+0000","last_undegraded":"2024-03-09T15:36:05.269380+0000","last_fullsized":"2024-03-09T15:36:05.269380+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:03:03.441773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1195,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1118,"num_read_kb":1421,"num_write":14,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"261'86","reported_seq":616,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.808693+0000","last_change":"2024-03-09T14:28:38.596269+0000","last_active":"2024-03-09T15:33:55.808693+0000","last_peered":"2024-03-09T15:33:55.808693+0000","last_clean":"2024-03-09T15:33:55.808693+0000","last_became_active":"2024-03-09T14:28:38.595478+0000","last_became_peered":"2024-03-09T14:28:38.595478+0000","last_unstale":"2024-03-09T15:33:55.808693+0000","last_undegraded":"2024-03-09T15:33:55.808693+0000","last_fullsized":"2024-03-09T15:33:55.808693+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:43:55.683017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":105,"num_read_kb":100,"num_write":70,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"261'489","reported_seq":2295,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.538814+0000","last_change":"2024-03-09T14:28:40.375535+0000","last_active":"2024-03-09T15:36:06.538814+0000","last_peered":"2024-03-09T15:36:06.538814+0000","last_clean":"2024-03-09T15:36:06.538814+0000","last_became_active":"2024-03-09T14:28:40.374408+0000","last_became_peered":"2024-03-09T14:28:40.374408+0000","last_unstale":"2024-03-09T15:36:06.538814+0000","last_undegraded":"2024-03-09T15:36:06.538814+0000","last_fullsized":"2024-03-09T15:36:06.538814+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":400,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:00:18.508302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"261'93","reported_seq":596,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:39.863116+0000","last_change":"2024-03-09T14:28:42.412396+0000","last_active":"2024-03-09T15:34:39.863116+0000","last_peered":"2024-03-09T15:34:39.863116+0000","last_clean":"2024-03-09T15:34:39.863116+0000","last_became_active":"2024-03-09T14:28:42.411523+0000","last_became_peered":"2024-03-09T14:28:42.411523+0000","last_unstale":"2024-03-09T15:34:39.863116+0000","last_undegraded":"2024-03-09T15:34:39.863116+0000","last_fullsized":"2024-03-09T15:34:39.863116+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:23:50.431623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":72,"num_write":221,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"261'42","reported_seq":595,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.134950+0000","last_change":"2024-03-09T14:28:34.317632+0000","last_active":"2024-03-09T15:31:41.134950+0000","last_peered":"2024-03-09T15:31:41.134950+0000","last_clean":"2024-03-09T15:31:41.134950+0000","last_became_active":"2024-03-09T14:28:34.317226+0000","last_became_peered":"2024-03-09T14:28:34.317226+0000","last_unstale":"2024-03-09T15:31:41.134950+0000","last_undegraded":"2024-03-09T15:31:41.134950+0000","last_fullsized":"2024-03-09T15:31:41.134950+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:48:25.758532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":188,"num_read_kb":158,"num_write":62,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"261'116","reported_seq":652,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.524289+0000","last_change":"2024-03-09T14:28:32.295693+0000","last_active":"2024-03-09T15:36:06.524289+0000","last_peered":"2024-03-09T15:36:06.524289+0000","last_clean":"2024-03-09T15:36:06.524289+0000","last_became_active":"2024-03-09T14:28:32.295179+0000","last_became_peered":"2024-03-09T14:28:32.295179+0000","last_unstale":"2024-03-09T15:36:06.524289+0000","last_undegraded":"2024-03-09T15:36:06.524289+0000","last_fullsized":"2024-03-09T15:36:06.524289+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":100,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:55:35.164395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262159,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":143,"num_read_kb":124,"num_write":390,"num_write_kb":5920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195837+0000","last_change":"2024-03-09T14:28:36.319885+0000","last_active":"2024-03-09T15:33:51.195837+0000","last_peered":"2024-03-09T15:33:51.195837+0000","last_clean":"2024-03-09T15:33:51.195837+0000","last_became_active":"2024-03-09T14:28:36.319102+0000","last_became_peered":"2024-03-09T14:28:36.319102+0000","last_unstale":"2024-03-09T15:33:51.195837+0000","last_undegraded":"2024-03-09T15:33:51.195837+0000","last_fullsized":"2024-03-09T15:33:51.195837+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:44:50.268293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"261'314","reported_seq":942,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.382606+0000","last_change":"2024-03-09T14:28:38.338131+0000","last_active":"2024-03-09T15:36:06.382606+0000","last_peered":"2024-03-09T15:36:06.382606+0000","last_clean":"2024-03-09T15:36:06.382606+0000","last_became_active":"2024-03-09T14:28:38.337254+0000","last_became_peered":"2024-03-09T14:28:38.337254+0000","last_unstale":"2024-03-09T15:36:06.382606+0000","last_undegraded":"2024-03-09T15:36:06.382606+0000","last_fullsized":"2024-03-09T15:36:06.382606+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":300,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:49.247631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13936,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":673,"num_read_kb":965,"num_write":788,"num_write_kb":629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":495,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.793970+0000","last_change":"2024-03-09T14:28:40.378550+0000","last_active":"2024-03-09T15:33:55.793970+0000","last_peered":"2024-03-09T15:33:55.793970+0000","last_clean":"2024-03-09T15:33:55.793970+0000","last_became_active":"2024-03-09T14:28:40.377714+0000","last_became_peered":"2024-03-09T14:28:40.377714+0000","last_unstale":"2024-03-09T15:33:55.793970+0000","last_undegraded":"2024-03-09T15:33:55.793970+0000","last_fullsized":"2024-03-09T15:33:55.793970+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:21:50.389100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"261'68","reported_seq":567,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.959943+0000","last_change":"2024-03-09T14:28:42.402550+0000","last_active":"2024-03-09T15:33:20.959943+0000","last_peered":"2024-03-09T15:33:20.959943+0000","last_clean":"2024-03-09T15:33:20.959943+0000","last_became_active":"2024-03-09T14:28:42.401775+0000","last_became_peered":"2024-03-09T14:28:42.401775+0000","last_unstale":"2024-03-09T15:33:20.959943+0000","last_undegraded":"2024-03-09T15:33:20.959943+0000","last_fullsized":"2024-03-09T15:33:20.959943+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:47:20.741043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":77,"num_write":153,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"261'174","reported_seq":825,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:30.788820+0000","last_change":"2024-03-09T14:28:34.327727+0000","last_active":"2024-03-09T15:33:30.788820+0000","last_peered":"2024-03-09T15:33:30.788820+0000","last_clean":"2024-03-09T15:33:30.788820+0000","last_became_active":"2024-03-09T14:28:34.326919+0000","last_became_peered":"2024-03-09T14:28:34.326919+0000","last_unstale":"2024-03-09T15:33:30.788820+0000","last_undegraded":"2024-03-09T15:33:30.788820+0000","last_fullsized":"2024-03-09T15:33:30.788820+0000","mapping_epoch":22,"log_start":"34'100","ondisk_log_start":"34'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:50:06.490347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":914,"num_read_kb":819,"num_write":300,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":681,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:26.189283+0000","last_change":"2024-03-09T14:28:12.265518+0000","last_active":"2024-03-09T15:32:26.189283+0000","last_peered":"2024-03-09T15:32:26.189283+0000","last_clean":"2024-03-09T15:32:26.189283+0000","last_became_active":"2024-03-09T14:28:12.262430+0000","last_became_peered":"2024-03-09T14:28:12.262430+0000","last_unstale":"2024-03-09T15:32:26.189283+0000","last_undegraded":"2024-03-09T15:32:26.189283+0000","last_fullsized":"2024-03-09T15:32:26.189283+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_clean_scrub_stamp":"2024-03-09T14:28:11.179640+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:14:17.888044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.185851+0000","last_change":"2024-03-09T14:28:19.233336+0000","last_active":"2024-03-09T15:32:56.185851+0000","last_peered":"2024-03-09T15:32:56.185851+0000","last_clean":"2024-03-09T15:32:56.185851+0000","last_became_active":"2024-03-09T14:28:19.232517+0000","last_became_peered":"2024-03-09T14:28:19.232517+0000","last_unstale":"2024-03-09T15:32:56.185851+0000","last_undegraded":"2024-03-09T15:32:56.185851+0000","last_fullsized":"2024-03-09T15:32:56.185851+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:53:39.430154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"261'130","reported_seq":689,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.472755+0000","last_change":"2024-03-09T14:28:32.300845+0000","last_active":"2024-03-09T15:36:06.472755+0000","last_peered":"2024-03-09T15:36:06.472755+0000","last_clean":"2024-03-09T15:36:06.472755+0000","last_became_active":"2024-03-09T14:28:32.300468+0000","last_became_peered":"2024-03-09T14:28:32.300468+0000","last_unstale":"2024-03-09T15:36:06.472755+0000","last_undegraded":"2024-03-09T15:36:06.472755+0000","last_fullsized":"2024-03-09T15:36:06.472755+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:58:14.894309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311644,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":173,"num_read_kb":6450,"num_write":457,"num_write_kb":21101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.199546+0000","last_change":"2024-03-09T14:28:36.318953+0000","last_active":"2024-03-09T15:33:11.199546+0000","last_peered":"2024-03-09T15:33:11.199546+0000","last_clean":"2024-03-09T15:33:11.199546+0000","last_became_active":"2024-03-09T14:28:36.318172+0000","last_became_peered":"2024-03-09T14:28:36.318172+0000","last_unstale":"2024-03-09T15:33:11.199546+0000","last_undegraded":"2024-03-09T15:33:11.199546+0000","last_fullsized":"2024-03-09T15:33:11.199546+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:14:08.117277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"261'499","reported_seq":1214,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.416472+0000","last_change":"2024-03-09T14:28:38.349794+0000","last_active":"2024-03-09T15:36:06.416472+0000","last_peered":"2024-03-09T15:36:06.416472+0000","last_clean":"2024-03-09T15:36:06.416472+0000","last_became_active":"2024-03-09T14:28:38.348941+0000","last_became_peered":"2024-03-09T14:28:38.348941+0000","last_unstale":"2024-03-09T15:36:06.416472+0000","last_undegraded":"2024-03-09T15:36:06.416472+0000","last_fullsized":"2024-03-09T15:36:06.416472+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":400,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:10:34.907672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37400,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":987,"num_read_kb":1910,"num_write":934,"num_write_kb":767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.043973+0000","last_change":"2024-03-09T14:28:40.364392+0000","last_active":"2024-03-09T15:35:51.043973+0000","last_peered":"2024-03-09T15:35:51.043973+0000","last_clean":"2024-03-09T15:35:51.043973+0000","last_became_active":"2024-03-09T14:28:40.362781+0000","last_became_peered":"2024-03-09T14:28:40.362781+0000","last_unstale":"2024-03-09T15:35:51.043973+0000","last_undegraded":"2024-03-09T15:35:51.043973+0000","last_fullsized":"2024-03-09T15:35:51.043973+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"261'118","reported_seq":817,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:45.170047+0000","last_change":"2024-03-09T14:28:42.399378+0000","last_active":"2024-03-09T15:34:45.170047+0000","last_peered":"2024-03-09T15:34:45.170047+0000","last_clean":"2024-03-09T15:34:45.170047+0000","last_became_active":"2024-03-09T14:28:42.398626+0000","last_became_peered":"2024-03-09T14:28:42.398626+0000","last_unstale":"2024-03-09T15:34:45.170047+0000","last_undegraded":"2024-03-09T15:34:45.170047+0000","last_fullsized":"2024-03-09T15:34:45.170047+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:40:25.365667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":356,"num_read_kb":343,"num_write":215,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"261'90","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:46.042921+0000","last_change":"2024-03-09T14:28:34.297905+0000","last_active":"2024-03-09T15:35:46.042921+0000","last_peered":"2024-03-09T15:35:46.042921+0000","last_clean":"2024-03-09T15:35:46.042921+0000","last_became_active":"2024-03-09T14:28:34.297472+0000","last_became_peered":"2024-03-09T14:28:34.297472+0000","last_unstale":"2024-03-09T15:35:46.042921+0000","last_undegraded":"2024-03-09T15:35:46.042921+0000","last_fullsized":"2024-03-09T15:35:46.042921+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":356,"num_read_kb":292,"num_write":141,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":525,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:31.190342+0000","last_change":"2024-03-09T14:28:19.242176+0000","last_active":"2024-03-09T15:32:31.190342+0000","last_peered":"2024-03-09T15:32:31.190342+0000","last_clean":"2024-03-09T15:32:31.190342+0000","last_became_active":"2024-03-09T14:28:19.241110+0000","last_became_peered":"2024-03-09T14:28:19.241110+0000","last_unstale":"2024-03-09T15:32:31.190342+0000","last_undegraded":"2024-03-09T15:32:31.190342+0000","last_fullsized":"2024-03-09T15:32:31.190342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:28:52.238193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"261'138","reported_seq":711,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.525842+0000","last_change":"2024-03-09T14:28:32.312953+0000","last_active":"2024-03-09T15:36:06.525842+0000","last_peered":"2024-03-09T15:36:06.525842+0000","last_clean":"2024-03-09T15:36:06.525842+0000","last_became_active":"2024-03-09T14:28:32.312536+0000","last_became_peered":"2024-03-09T14:28:32.312536+0000","last_unstale":"2024-03-09T15:36:06.525842+0000","last_undegraded":"2024-03-09T15:36:06.525842+0000","last_fullsized":"2024-03-09T15:36:06.525842+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":100,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":219,"num_read_kb":12489,"num_write":525,"num_write_kb":17953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194706+0000","last_change":"2024-03-09T14:28:36.307721+0000","last_active":"2024-03-09T15:33:36.194706+0000","last_peered":"2024-03-09T15:33:36.194706+0000","last_clean":"2024-03-09T15:33:36.194706+0000","last_became_active":"2024-03-09T14:28:36.306912+0000","last_became_peered":"2024-03-09T14:28:36.306912+0000","last_unstale":"2024-03-09T15:33:36.194706+0000","last_undegraded":"2024-03-09T15:33:36.194706+0000","last_fullsized":"2024-03-09T15:33:36.194706+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:19:47.054193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"261'847","reported_seq":2524,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.399884+0000","last_change":"2024-03-09T14:28:38.329490+0000","last_active":"2024-03-09T15:36:06.399884+0000","last_peered":"2024-03-09T15:36:06.399884+0000","last_clean":"2024-03-09T15:36:06.399884+0000","last_became_active":"2024-03-09T14:28:38.328738+0000","last_became_peered":"2024-03-09T14:28:38.328738+0000","last_unstale":"2024-03-09T15:36:06.399884+0000","last_undegraded":"2024-03-09T15:36:06.399884+0000","last_fullsized":"2024-03-09T15:36:06.399884+0000","mapping_epoch":26,"log_start":"261'800","ondisk_log_start":"261'800","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":800,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:24:55.016529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34683,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1323,"num_read_kb":2148,"num_write":662,"num_write_kb":462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:26.202990+0000","last_change":"2024-03-09T14:28:40.387247+0000","last_active":"2024-03-09T15:33:26.202990+0000","last_peered":"2024-03-09T15:33:26.202990+0000","last_clean":"2024-03-09T15:33:26.202990+0000","last_became_active":"2024-03-09T14:28:40.386451+0000","last_became_peered":"2024-03-09T14:28:40.386451+0000","last_unstale":"2024-03-09T15:33:26.202990+0000","last_undegraded":"2024-03-09T15:33:26.202990+0000","last_fullsized":"2024-03-09T15:33:26.202990+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:06:23.810341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"261'124","reported_seq":656,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:55.967916+0000","last_change":"2024-03-09T14:28:42.403919+0000","last_active":"2024-03-09T15:33:55.967916+0000","last_peered":"2024-03-09T15:33:55.967916+0000","last_clean":"2024-03-09T15:33:55.967916+0000","last_became_active":"2024-03-09T14:28:42.403013+0000","last_became_peered":"2024-03-09T14:28:42.403013+0000","last_unstale":"2024-03-09T15:33:55.967916+0000","last_undegraded":"2024-03-09T15:33:55.967916+0000","last_fullsized":"2024-03-09T15:33:55.967916+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:36:18.083589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1248,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":261,"num_read_kb":241,"num_write":300,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"261'48","reported_seq":601,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963559+0000","last_change":"2024-03-09T14:28:34.297150+0000","last_active":"2024-03-09T15:33:35.963559+0000","last_peered":"2024-03-09T15:33:35.963559+0000","last_clean":"2024-03-09T15:33:35.963559+0000","last_became_active":"2024-03-09T14:28:34.296733+0000","last_became_peered":"2024-03-09T14:28:34.296733+0000","last_unstale":"2024-03-09T15:33:35.963559+0000","last_undegraded":"2024-03-09T15:33:35.963559+0000","last_fullsized":"2024-03-09T15:33:35.963559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:29:16.007058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":200,"num_read_kb":166,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.968712+0000","last_change":"2024-03-09T14:28:19.437446+0000","last_active":"2024-03-09T15:32:55.968712+0000","last_peered":"2024-03-09T15:32:55.968712+0000","last_clean":"2024-03-09T15:32:55.968712+0000","last_became_active":"2024-03-09T14:28:19.436985+0000","last_became_peered":"2024-03-09T14:28:19.436985+0000","last_unstale":"2024-03-09T15:32:55.968712+0000","last_undegraded":"2024-03-09T15:32:55.968712+0000","last_fullsized":"2024-03-09T15:32:55.968712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:46:38.528697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"261'152","reported_seq":809,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.487420+0000","last_change":"2024-03-09T14:28:32.303463+0000","last_active":"2024-03-09T15:36:06.487420+0000","last_peered":"2024-03-09T15:36:06.487420+0000","last_clean":"2024-03-09T15:36:06.487420+0000","last_became_active":"2024-03-09T14:28:32.303176+0000","last_became_peered":"2024-03-09T14:28:32.303176+0000","last_unstale":"2024-03-09T15:36:06.487420+0000","last_undegraded":"2024-03-09T15:36:06.487420+0000","last_fullsized":"2024-03-09T15:36:06.487420+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":100,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:28:51.916628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8554724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":486,"num_read_kb":3616,"num_write":592,"num_write_kb":29269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"25'1","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.976814+0000","last_change":"2024-03-09T14:28:36.306679+0000","last_active":"2024-03-09T15:33:35.976814+0000","last_peered":"2024-03-09T15:33:35.976814+0000","last_clean":"2024-03-09T15:33:35.976814+0000","last_became_active":"2024-03-09T14:28:36.305971+0000","last_became_peered":"2024-03-09T14:28:36.305971+0000","last_unstale":"2024-03-09T15:33:35.976814+0000","last_undegraded":"2024-03-09T15:33:35.976814+0000","last_fullsized":"2024-03-09T15:33:35.976814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:21:21.395131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"261'230","reported_seq":1598,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.934522+0000","last_change":"2024-03-09T14:28:38.598201+0000","last_active":"2024-03-09T15:35:58.934522+0000","last_peered":"2024-03-09T15:35:58.934522+0000","last_clean":"2024-03-09T15:35:58.934522+0000","last_became_active":"2024-03-09T14:28:38.597403+0000","last_became_peered":"2024-03-09T14:28:38.597403+0000","last_unstale":"2024-03-09T15:35:58.934522+0000","last_undegraded":"2024-03-09T15:35:58.934522+0000","last_fullsized":"2024-03-09T15:35:58.934522+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":200,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:44:05.046714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":226,"num_read_kb":216,"num_write":195,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":498,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:21.037287+0000","last_change":"2024-03-09T14:28:40.372884+0000","last_active":"2024-03-09T15:35:21.037287+0000","last_peered":"2024-03-09T15:35:21.037287+0000","last_clean":"2024-03-09T15:35:21.037287+0000","last_became_active":"2024-03-09T14:28:40.371383+0000","last_became_peered":"2024-03-09T14:28:40.371383+0000","last_unstale":"2024-03-09T15:35:21.037287+0000","last_undegraded":"2024-03-09T15:35:21.037287+0000","last_fullsized":"2024-03-09T15:35:21.037287+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"261'201","reported_seq":791,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:19.528609+0000","last_change":"2024-03-09T14:28:42.396620+0000","last_active":"2024-03-09T15:34:19.528609+0000","last_peered":"2024-03-09T15:34:19.528609+0000","last_clean":"2024-03-09T15:34:19.528609+0000","last_became_active":"2024-03-09T14:28:42.395781+0000","last_became_peered":"2024-03-09T14:28:42.395781+0000","last_unstale":"2024-03-09T15:34:19.528609+0000","last_undegraded":"2024-03-09T15:34:19.528609+0000","last_fullsized":"2024-03-09T15:34:19.528609+0000","mapping_epoch":30,"log_start":"247'100","ondisk_log_start":"247'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":100,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:38:14.404007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":435,"num_read_kb":456,"num_write":425,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"261'74","reported_seq":618,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:31.876523+0000","last_change":"2024-03-09T14:28:34.313660+0000","last_active":"2024-03-09T15:34:31.876523+0000","last_peered":"2024-03-09T15:34:31.876523+0000","last_clean":"2024-03-09T15:34:31.876523+0000","last_became_active":"2024-03-09T14:28:34.313302+0000","last_became_peered":"2024-03-09T14:28:34.313302+0000","last_unstale":"2024-03-09T15:34:31.876523+0000","last_undegraded":"2024-03-09T15:34:31.876523+0000","last_fullsized":"2024-03-09T15:34:31.876523+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:32:32.206071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":225,"num_write":104,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":522,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:56.185786+0000","last_change":"2024-03-09T14:28:19.238720+0000","last_active":"2024-03-09T15:32:56.185786+0000","last_peered":"2024-03-09T15:32:56.185786+0000","last_clean":"2024-03-09T15:32:56.185786+0000","last_became_active":"2024-03-09T14:28:19.237937+0000","last_became_peered":"2024-03-09T14:28:19.237937+0000","last_unstale":"2024-03-09T15:32:56.185786+0000","last_undegraded":"2024-03-09T15:32:56.185786+0000","last_fullsized":"2024-03-09T15:32:56.185786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:29:01.306246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"261'127","reported_seq":702,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.474766+0000","last_change":"2024-03-09T14:28:32.312937+0000","last_active":"2024-03-09T15:36:06.474766+0000","last_peered":"2024-03-09T15:36:06.474766+0000","last_clean":"2024-03-09T15:36:06.474766+0000","last_became_active":"2024-03-09T14:28:32.303963+0000","last_became_peered":"2024-03-09T14:28:32.303963+0000","last_unstale":"2024-03-09T15:36:06.474766+0000","last_undegraded":"2024-03-09T15:36:06.474766+0000","last_fullsized":"2024-03-09T15:36:06.474766+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":100,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:54:34.862486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6898850,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":175,"num_read_kb":15115,"num_write":430,"num_write_kb":32085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:31.039898+0000","last_change":"2024-03-09T14:28:36.318128+0000","last_active":"2024-03-09T15:35:31.039898+0000","last_peered":"2024-03-09T15:35:31.039898+0000","last_clean":"2024-03-09T15:35:31.039898+0000","last_became_active":"2024-03-09T14:28:36.317586+0000","last_became_peered":"2024-03-09T14:28:36.317586+0000","last_unstale":"2024-03-09T15:35:31.039898+0000","last_undegraded":"2024-03-09T15:35:31.039898+0000","last_fullsized":"2024-03-09T15:35:31.039898+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"261'129","reported_seq":682,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:52.970721+0000","last_change":"2024-03-09T14:28:38.335628+0000","last_active":"2024-03-09T15:34:52.970721+0000","last_peered":"2024-03-09T15:34:52.970721+0000","last_clean":"2024-03-09T15:34:52.970721+0000","last_became_active":"2024-03-09T14:28:38.334463+0000","last_became_peered":"2024-03-09T14:28:38.334463+0000","last_unstale":"2024-03-09T15:34:52.970721+0000","last_undegraded":"2024-03-09T15:34:52.970721+0000","last_fullsized":"2024-03-09T15:34:52.970721+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:56:33.829971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":178,"num_read_kb":172,"num_write":132,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.790948+0000","last_change":"2024-03-09T14:28:40.371173+0000","last_active":"2024-03-09T15:33:40.790948+0000","last_peered":"2024-03-09T15:33:40.790948+0000","last_clean":"2024-03-09T15:33:40.790948+0000","last_became_active":"2024-03-09T14:28:40.370436+0000","last_became_peered":"2024-03-09T14:28:40.370436+0000","last_unstale":"2024-03-09T15:33:40.790948+0000","last_undegraded":"2024-03-09T15:33:40.790948+0000","last_fullsized":"2024-03-09T15:33:40.790948+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:23:11.745507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"261'47","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:48.549510+0000","last_change":"2024-03-09T14:28:42.398975+0000","last_active":"2024-03-09T15:33:48.549510+0000","last_peered":"2024-03-09T15:33:48.549510+0000","last_clean":"2024-03-09T15:33:48.549510+0000","last_became_active":"2024-03-09T14:28:42.397058+0000","last_became_peered":"2024-03-09T14:28:42.397058+0000","last_unstale":"2024-03-09T15:33:48.549510+0000","last_undegraded":"2024-03-09T15:33:48.549510+0000","last_fullsized":"2024-03-09T15:33:48.549510+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:02:27.312655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":100,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"261'43","reported_seq":593,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.957260+0000","last_change":"2024-03-09T14:28:34.297850+0000","last_active":"2024-03-09T15:33:10.957260+0000","last_peered":"2024-03-09T15:33:10.957260+0000","last_clean":"2024-03-09T15:33:10.957260+0000","last_became_active":"2024-03-09T14:28:34.296953+0000","last_became_peered":"2024-03-09T14:28:34.296953+0000","last_unstale":"2024-03-09T15:33:10.957260+0000","last_undegraded":"2024-03-09T15:33:10.957260+0000","last_fullsized":"2024-03-09T15:33:10.957260+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:37:50.390768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":162,"num_write":75,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"261'163","reported_seq":797,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.489203+0000","last_change":"2024-03-09T14:28:32.303697+0000","last_active":"2024-03-09T15:36:06.489203+0000","last_peered":"2024-03-09T15:36:06.489203+0000","last_clean":"2024-03-09T15:36:06.489203+0000","last_became_active":"2024-03-09T14:28:32.302970+0000","last_became_peered":"2024-03-09T14:28:32.302970+0000","last_unstale":"2024-03-09T15:36:06.489203+0000","last_undegraded":"2024-03-09T15:36:06.489203+0000","last_fullsized":"2024-03-09T15:36:06.489203+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":100,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:33:55.197166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655822,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":94521,"num_write":632,"num_write_kb":45805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":520,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:55.953596+0000","last_change":"2024-03-09T14:28:19.236670+0000","last_active":"2024-03-09T15:32:55.953596+0000","last_peered":"2024-03-09T15:32:55.953596+0000","last_clean":"2024-03-09T15:32:55.953596+0000","last_became_active":"2024-03-09T14:28:19.236240+0000","last_became_peered":"2024-03-09T14:28:19.236240+0000","last_unstale":"2024-03-09T15:32:55.953596+0000","last_undegraded":"2024-03-09T15:32:55.953596+0000","last_fullsized":"2024-03-09T15:32:55.953596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:00:09.869968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:16.035977+0000","last_change":"2024-03-09T14:28:36.315028+0000","last_active":"2024-03-09T15:35:16.035977+0000","last_peered":"2024-03-09T15:35:16.035977+0000","last_clean":"2024-03-09T15:35:16.035977+0000","last_became_active":"2024-03-09T14:28:36.314526+0000","last_became_peered":"2024-03-09T14:28:36.314526+0000","last_unstale":"2024-03-09T15:35:16.035977+0000","last_undegraded":"2024-03-09T15:35:16.035977+0000","last_fullsized":"2024-03-09T15:35:16.035977+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"261'384","reported_seq":1067,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.295054+0000","last_change":"2024-03-09T14:28:38.335922+0000","last_active":"2024-03-09T15:36:06.295054+0000","last_peered":"2024-03-09T15:36:06.295054+0000","last_clean":"2024-03-09T15:36:06.295054+0000","last_became_active":"2024-03-09T14:28:38.334976+0000","last_became_peered":"2024-03-09T14:28:38.334976+0000","last_unstale":"2024-03-09T15:36:06.295054+0000","last_undegraded":"2024-03-09T15:36:06.295054+0000","last_fullsized":"2024-03-09T15:36:06.295054+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":300,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21042,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":640,"num_read_kb":1204,"num_write":456,"num_write_kb":268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.186765+0000","last_change":"2024-03-09T14:28:40.379714+0000","last_active":"2024-03-09T15:33:16.186765+0000","last_peered":"2024-03-09T15:33:16.186765+0000","last_clean":"2024-03-09T15:33:16.186765+0000","last_became_active":"2024-03-09T14:28:40.379108+0000","last_became_peered":"2024-03-09T14:28:40.379108+0000","last_unstale":"2024-03-09T15:33:16.186765+0000","last_undegraded":"2024-03-09T15:33:16.186765+0000","last_fullsized":"2024-03-09T15:33:16.186765+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:39:21.726823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"261'101","reported_seq":627,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:45.186435+0000","last_change":"2024-03-09T14:28:42.406107+0000","last_active":"2024-03-09T15:34:45.186435+0000","last_peered":"2024-03-09T15:34:45.186435+0000","last_clean":"2024-03-09T15:34:45.186435+0000","last_became_active":"2024-03-09T14:28:42.405277+0000","last_became_peered":"2024-03-09T14:28:42.405277+0000","last_unstale":"2024-03-09T15:34:45.186435+0000","last_undegraded":"2024-03-09T15:34:45.186435+0000","last_fullsized":"2024-03-09T15:34:45.186435+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":717,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185,"num_read_kb":160,"num_write":261,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"261'43","reported_seq":576,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:16.036023+0000","last_change":"2024-03-09T14:28:34.298018+0000","last_active":"2024-03-09T15:35:16.036023+0000","last_peered":"2024-03-09T15:35:16.036023+0000","last_clean":"2024-03-09T15:35:16.036023+0000","last_became_active":"2024-03-09T14:28:34.296660+0000","last_became_peered":"2024-03-09T14:28:34.296660+0000","last_unstale":"2024-03-09T15:35:16.036023+0000","last_undegraded":"2024-03-09T15:35:16.036023+0000","last_fullsized":"2024-03-09T15:35:16.036023+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":129,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"261'148","reported_seq":778,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.474422+0000","last_change":"2024-03-09T14:28:32.305873+0000","last_active":"2024-03-09T15:36:06.474422+0000","last_peered":"2024-03-09T15:36:06.474422+0000","last_clean":"2024-03-09T15:36:06.474422+0000","last_became_active":"2024-03-09T14:28:32.305318+0000","last_became_peered":"2024-03-09T14:28:32.305318+0000","last_unstale":"2024-03-09T15:36:06.474422+0000","last_undegraded":"2024-03-09T15:36:06.474422+0000","last_fullsized":"2024-03-09T15:36:06.474422+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":100,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:30:36.881076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262245,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":357,"num_read_kb":10578,"num_write":456,"num_write_kb":20505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:06.198444+0000","last_change":"2024-03-09T14:28:19.241475+0000","last_active":"2024-03-09T15:33:06.198444+0000","last_peered":"2024-03-09T15:33:06.198444+0000","last_clean":"2024-03-09T15:33:06.198444+0000","last_became_active":"2024-03-09T14:28:19.240704+0000","last_became_peered":"2024-03-09T14:28:19.240704+0000","last_unstale":"2024-03-09T15:33:06.198444+0000","last_undegraded":"2024-03-09T15:33:06.198444+0000","last_fullsized":"2024-03-09T15:33:06.198444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:07:08.993017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784406+0000","last_change":"2024-03-09T14:28:36.315481+0000","last_active":"2024-03-09T15:33:10.784406+0000","last_peered":"2024-03-09T15:33:10.784406+0000","last_clean":"2024-03-09T15:33:10.784406+0000","last_became_active":"2024-03-09T14:28:36.314676+0000","last_became_peered":"2024-03-09T14:28:36.314676+0000","last_unstale":"2024-03-09T15:33:10.784406+0000","last_undegraded":"2024-03-09T15:33:10.784406+0000","last_fullsized":"2024-03-09T15:33:10.784406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:14:11.938307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"261'197","reported_seq":778,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:20.985108+0000","last_change":"2024-03-09T14:28:38.595995+0000","last_active":"2024-03-09T15:35:20.985108+0000","last_peered":"2024-03-09T15:35:20.985108+0000","last_clean":"2024-03-09T15:35:20.985108+0000","last_became_active":"2024-03-09T14:28:38.595028+0000","last_became_peered":"2024-03-09T14:28:38.595028+0000","last_unstale":"2024-03-09T15:35:20.985108+0000","last_undegraded":"2024-03-09T15:35:20.985108+0000","last_fullsized":"2024-03-09T15:35:20.985108+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":100,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:24:00.362746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":269,"num_read_kb":262,"num_write":230,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"261'501","reported_seq":2297,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.719693+0000","last_change":"2024-03-09T14:28:40.370167+0000","last_active":"2024-03-09T15:36:05.719693+0000","last_peered":"2024-03-09T15:36:05.719693+0000","last_clean":"2024-03-09T15:36:05.719693+0000","last_became_active":"2024-03-09T14:28:40.369165+0000","last_became_peered":"2024-03-09T14:28:40.369165+0000","last_unstale":"2024-03-09T15:36:05.719693+0000","last_undegraded":"2024-03-09T15:36:05.719693+0000","last_fullsized":"2024-03-09T15:36:05.719693+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":400,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:29:22.485672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"261'76","reported_seq":564,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:15.812925+0000","last_change":"2024-03-09T14:28:42.393153+0000","last_active":"2024-03-09T15:34:15.812925+0000","last_peered":"2024-03-09T15:34:15.812925+0000","last_clean":"2024-03-09T15:34:15.812925+0000","last_became_active":"2024-03-09T14:28:42.392402+0000","last_became_peered":"2024-03-09T14:28:42.392402+0000","last_unstale":"2024-03-09T15:34:15.812925+0000","last_undegraded":"2024-03-09T15:34:15.812925+0000","last_fullsized":"2024-03-09T15:34:15.812925+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:57:45.818538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":57,"num_write":172,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"261'48","reported_seq":589,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.973653+0000","last_change":"2024-03-09T14:29:06.440753+0000","last_active":"2024-03-09T15:33:20.973653+0000","last_peered":"2024-03-09T15:33:20.973653+0000","last_clean":"2024-03-09T15:33:20.973653+0000","last_became_active":"2024-03-09T14:29:06.440556+0000","last_became_peered":"2024-03-09T14:29:06.440556+0000","last_unstale":"2024-03-09T15:33:20.973653+0000","last_undegraded":"2024-03-09T15:33:20.973653+0000","last_fullsized":"2024-03-09T15:33:20.973653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:43:05.084646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":216,"num_read_kb":182,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"261'145","reported_seq":690,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.491554+0000","last_change":"2024-03-09T14:28:32.319314+0000","last_active":"2024-03-09T15:36:06.491554+0000","last_peered":"2024-03-09T15:36:06.491554+0000","last_clean":"2024-03-09T15:36:06.491554+0000","last_became_active":"2024-03-09T14:28:32.318971+0000","last_became_peered":"2024-03-09T14:28:32.318971+0000","last_unstale":"2024-03-09T15:36:06.491554+0000","last_undegraded":"2024-03-09T15:36:06.491554+0000","last_fullsized":"2024-03-09T15:36:06.491554+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:01:00.707624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":151,"num_read_kb":8647,"num_write":476,"num_write_kb":27350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":519,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.960027+0000","last_change":"2024-03-09T14:28:19.238362+0000","last_active":"2024-03-09T15:33:20.960027+0000","last_peered":"2024-03-09T15:33:20.960027+0000","last_clean":"2024-03-09T15:33:20.960027+0000","last_became_active":"2024-03-09T14:28:19.237750+0000","last_became_peered":"2024-03-09T14:28:19.237750+0000","last_unstale":"2024-03-09T15:33:20.960027+0000","last_undegraded":"2024-03-09T15:33:20.960027+0000","last_fullsized":"2024-03-09T15:33:20.960027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:19.699145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.185710+0000","last_change":"2024-03-09T14:28:36.328518+0000","last_active":"2024-03-09T15:33:11.185710+0000","last_peered":"2024-03-09T15:33:11.185710+0000","last_clean":"2024-03-09T15:33:11.185710+0000","last_became_active":"2024-03-09T14:28:36.327601+0000","last_became_peered":"2024-03-09T14:28:36.327601+0000","last_unstale":"2024-03-09T15:33:11.185710+0000","last_undegraded":"2024-03-09T15:33:11.185710+0000","last_fullsized":"2024-03-09T15:33:11.185710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:41:26.807068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"261'106","reported_seq":661,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:16.186823+0000","last_change":"2024-03-09T14:28:38.348479+0000","last_active":"2024-03-09T15:33:16.186823+0000","last_peered":"2024-03-09T15:33:16.186823+0000","last_clean":"2024-03-09T15:33:16.186823+0000","last_became_active":"2024-03-09T14:28:38.347750+0000","last_became_peered":"2024-03-09T14:28:38.347750+0000","last_unstale":"2024-03-09T15:33:16.186823+0000","last_undegraded":"2024-03-09T15:33:16.186823+0000","last_fullsized":"2024-03-09T15:33:16.186823+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:26:08.370238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"261'514","reported_seq":2486,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.730035+0000","last_change":"2024-03-09T14:28:40.387580+0000","last_active":"2024-03-09T15:36:05.730035+0000","last_peered":"2024-03-09T15:36:05.730035+0000","last_clean":"2024-03-09T15:36:05.730035+0000","last_became_active":"2024-03-09T14:28:40.386859+0000","last_became_peered":"2024-03-09T14:28:40.386859+0000","last_unstale":"2024-03-09T15:36:05.730035+0000","last_undegraded":"2024-03-09T15:36:05.730035+0000","last_fullsized":"2024-03-09T15:36:05.730035+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":500,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:17:24.125699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"261'106","reported_seq":607,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:01.018924+0000","last_change":"2024-03-09T14:28:42.406871+0000","last_active":"2024-03-09T15:34:01.018924+0000","last_peered":"2024-03-09T15:34:01.018924+0000","last_clean":"2024-03-09T15:34:01.018924+0000","last_became_active":"2024-03-09T14:28:42.406139+0000","last_became_peered":"2024-03-09T14:28:42.406139+0000","last_unstale":"2024-03-09T15:34:01.018924+0000","last_undegraded":"2024-03-09T15:34:01.018924+0000","last_fullsized":"2024-03-09T15:34:01.018924+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":65,"num_write":235,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"261'216","reported_seq":787,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:48.579968+0000","last_change":"2024-03-09T14:28:34.301025+0000","last_active":"2024-03-09T15:33:48.579968+0000","last_peered":"2024-03-09T15:33:48.579968+0000","last_clean":"2024-03-09T15:33:48.579968+0000","last_became_active":"2024-03-09T14:28:34.299988+0000","last_became_peered":"2024-03-09T14:28:34.299988+0000","last_unstale":"2024-03-09T15:33:48.579968+0000","last_undegraded":"2024-03-09T15:33:48.579968+0000","last_fullsized":"2024-03-09T15:33:48.579968+0000","mapping_epoch":22,"log_start":"258'200","ondisk_log_start":"258'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":200,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:05:09.988637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":843,"num_read_kb":645,"num_write":377,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"261'150","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.454415+0000","last_change":"2024-03-09T14:28:32.299582+0000","last_active":"2024-03-09T15:36:06.454415+0000","last_peered":"2024-03-09T15:36:06.454415+0000","last_clean":"2024-03-09T15:36:06.454415+0000","last_became_active":"2024-03-09T14:28:32.298977+0000","last_became_peered":"2024-03-09T14:28:32.298977+0000","last_unstale":"2024-03-09T15:36:06.454415+0000","last_undegraded":"2024-03-09T15:36:06.454415+0000","last_fullsized":"2024-03-09T15:36:06.454415+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242905,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":173,"num_read_kb":5781,"num_write":510,"num_write_kb":37567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":518,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:20.786634+0000","last_change":"2024-03-09T14:28:19.242561+0000","last_active":"2024-03-09T15:33:20.786634+0000","last_peered":"2024-03-09T15:33:20.786634+0000","last_clean":"2024-03-09T15:33:20.786634+0000","last_became_active":"2024-03-09T14:28:19.241700+0000","last_became_peered":"2024-03-09T14:28:19.241700+0000","last_unstale":"2024-03-09T15:33:20.786634+0000","last_undegraded":"2024-03-09T15:33:20.786634+0000","last_fullsized":"2024-03-09T15:33:20.786634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:15.462353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"32'3","reported_seq":822,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.255551+0000","last_change":"2024-03-09T14:28:36.317874+0000","last_active":"2024-03-09T15:36:05.255551+0000","last_peered":"2024-03-09T15:36:05.255551+0000","last_clean":"2024-03-09T15:36:05.255551+0000","last_became_active":"2024-03-09T14:28:36.317327+0000","last_became_peered":"2024-03-09T14:28:36.317327+0000","last_unstale":"2024-03-09T15:36:05.255551+0000","last_undegraded":"2024-03-09T15:36:05.255551+0000","last_fullsized":"2024-03-09T15:36:05.255551+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:07.321787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":484,"num_read_kb":482,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"261'470","reported_seq":1209,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.346647+0000","last_change":"2024-03-09T14:28:38.603808+0000","last_active":"2024-03-09T15:36:06.346647+0000","last_peered":"2024-03-09T15:36:06.346647+0000","last_clean":"2024-03-09T15:36:06.346647+0000","last_became_active":"2024-03-09T14:28:38.602908+0000","last_became_peered":"2024-03-09T14:28:38.602908+0000","last_unstale":"2024-03-09T15:36:06.346647+0000","last_undegraded":"2024-03-09T15:36:06.346647+0000","last_fullsized":"2024-03-09T15:36:06.346647+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":400,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:20:14.798348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29305,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":797,"num_read_kb":1561,"num_write":550,"num_write_kb":344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:15.799092+0000","last_change":"2024-03-09T14:28:40.376657+0000","last_active":"2024-03-09T15:33:15.799092+0000","last_peered":"2024-03-09T15:33:15.799092+0000","last_clean":"2024-03-09T15:33:15.799092+0000","last_became_active":"2024-03-09T14:28:40.375884+0000","last_became_peered":"2024-03-09T14:28:40.375884+0000","last_unstale":"2024-03-09T15:33:15.799092+0000","last_undegraded":"2024-03-09T15:33:15.799092+0000","last_fullsized":"2024-03-09T15:33:15.799092+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:28:12.389403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"261'54","reported_seq":551,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:33.602971+0000","last_change":"2024-03-09T14:28:42.406761+0000","last_active":"2024-03-09T15:34:33.602971+0000","last_peered":"2024-03-09T15:34:33.602971+0000","last_clean":"2024-03-09T15:34:33.602971+0000","last_became_active":"2024-03-09T14:28:42.405998+0000","last_became_peered":"2024-03-09T14:28:42.405998+0000","last_unstale":"2024-03-09T15:34:33.602971+0000","last_undegraded":"2024-03-09T15:34:33.602971+0000","last_fullsized":"2024-03-09T15:34:33.602971+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:02:27.576333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":52,"num_write":123,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1f","version":"261'146","reported_seq":724,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:26.038835+0000","last_change":"2024-03-09T14:28:42.410137+0000","last_active":"2024-03-09T15:35:26.038835+0000","last_peered":"2024-03-09T15:35:26.038835+0000","last_clean":"2024-03-09T15:35:26.038835+0000","last_became_active":"2024-03-09T14:28:42.408774+0000","last_became_peered":"2024-03-09T14:28:42.408774+0000","last_unstale":"2024-03-09T15:35:26.038835+0000","last_undegraded":"2024-03-09T15:35:26.038835+0000","last_fullsized":"2024-03-09T15:35:26.038835+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":256,"num_read_kb":233,"num_write":254,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"261'183","reported_seq":743,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:44.582989+0000","last_change":"2024-03-09T14:28:34.315742+0000","last_active":"2024-03-09T15:33:44.582989+0000","last_peered":"2024-03-09T15:33:44.582989+0000","last_clean":"2024-03-09T15:33:44.582989+0000","last_became_active":"2024-03-09T14:28:34.314893+0000","last_became_peered":"2024-03-09T14:28:34.314893+0000","last_unstale":"2024-03-09T15:33:44.582989+0000","last_undegraded":"2024-03-09T15:33:44.582989+0000","last_fullsized":"2024-03-09T15:33:44.582989+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":100,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:23:46.738791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":706,"num_read_kb":592,"num_write":291,"num_write_kb":155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"261'129","reported_seq":673,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.494323+0000","last_change":"2024-03-09T14:28:32.322792+0000","last_active":"2024-03-09T15:36:06.494323+0000","last_peered":"2024-03-09T15:36:06.494323+0000","last_clean":"2024-03-09T15:36:06.494323+0000","last_became_active":"2024-03-09T14:28:32.322364+0000","last_became_peered":"2024-03-09T14:28:32.322364+0000","last_unstale":"2024-03-09T15:36:06.494323+0000","last_undegraded":"2024-03-09T15:36:06.494323+0000","last_fullsized":"2024-03-09T15:36:06.494323+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:58:42.664679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":128,"num_write":389,"num_write_kb":27813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.134035+0000","last_change":"2024-03-09T14:28:36.314599+0000","last_active":"2024-03-09T15:31:36.134035+0000","last_peered":"2024-03-09T15:31:36.134035+0000","last_clean":"2024-03-09T15:31:36.134035+0000","last_became_active":"2024-03-09T14:28:36.313984+0000","last_became_peered":"2024-03-09T14:28:36.313984+0000","last_unstale":"2024-03-09T15:31:36.134035+0000","last_undegraded":"2024-03-09T15:31:36.134035+0000","last_fullsized":"2024-03-09T15:31:36.134035+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:41:00.980683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"261'191","reported_seq":773,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.445876+0000","last_change":"2024-03-09T14:28:38.345412+0000","last_active":"2024-03-09T15:36:06.445876+0000","last_peered":"2024-03-09T15:36:06.445876+0000","last_clean":"2024-03-09T15:36:06.445876+0000","last_became_active":"2024-03-09T14:28:38.338262+0000","last_became_peered":"2024-03-09T14:28:38.338262+0000","last_unstale":"2024-03-09T15:36:06.445876+0000","last_undegraded":"2024-03-09T15:36:06.445876+0000","last_fullsized":"2024-03-09T15:36:06.445876+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":100,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:08:16.291678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15994,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":315,"num_read_kb":601,"num_write":220,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":489,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.777786+0000","last_change":"2024-03-09T14:28:40.390067+0000","last_active":"2024-03-09T15:31:40.777786+0000","last_peered":"2024-03-09T15:31:40.777786+0000","last_clean":"2024-03-09T15:31:40.777786+0000","last_became_active":"2024-03-09T14:28:40.389277+0000","last_became_peered":"2024-03-09T14:28:40.389277+0000","last_unstale":"2024-03-09T15:31:40.777786+0000","last_undegraded":"2024-03-09T15:31:40.777786+0000","last_fullsized":"2024-03-09T15:31:40.777786+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:02:25.553702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"261'54","reported_seq":549,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.795784+0000","last_change":"2024-03-09T14:28:42.411135+0000","last_active":"2024-03-09T15:33:00.795784+0000","last_peered":"2024-03-09T15:33:00.795784+0000","last_clean":"2024-03-09T15:33:00.795784+0000","last_became_active":"2024-03-09T14:28:42.409229+0000","last_became_peered":"2024-03-09T14:28:42.409229+0000","last_unstale":"2024-03-09T15:33:00.795784+0000","last_undegraded":"2024-03-09T15:33:00.795784+0000","last_fullsized":"2024-03-09T15:33:00.795784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:17:09.392755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":828,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":44,"num_write":116,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"261'78","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.134077+0000","last_change":"2024-03-09T14:28:34.320825+0000","last_active":"2024-03-09T15:31:36.134077+0000","last_peered":"2024-03-09T15:31:36.134077+0000","last_clean":"2024-03-09T15:31:36.134077+0000","last_became_active":"2024-03-09T14:28:34.320474+0000","last_became_peered":"2024-03-09T14:28:34.320474+0000","last_unstale":"2024-03-09T15:31:36.134077+0000","last_undegraded":"2024-03-09T15:31:36.134077+0000","last_fullsized":"2024-03-09T15:31:36.134077+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:15:48.878005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":231,"num_write":128,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"261'159","reported_seq":791,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.527046+0000","last_change":"2024-03-09T14:28:32.317900+0000","last_active":"2024-03-09T15:36:06.527046+0000","last_peered":"2024-03-09T15:36:06.527046+0000","last_clean":"2024-03-09T15:36:06.527046+0000","last_became_active":"2024-03-09T14:28:32.315355+0000","last_became_peered":"2024-03-09T14:28:32.315355+0000","last_unstale":"2024-03-09T15:36:06.527046+0000","last_undegraded":"2024-03-09T15:36:06.527046+0000","last_fullsized":"2024-03-09T15:36:06.527046+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:28:48.531373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":325,"num_read_kb":85989,"num_write":581,"num_write_kb":44811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776757+0000","last_change":"2024-03-09T14:28:36.311910+0000","last_active":"2024-03-09T15:31:35.776757+0000","last_peered":"2024-03-09T15:31:35.776757+0000","last_clean":"2024-03-09T15:31:35.776757+0000","last_became_active":"2024-03-09T14:28:36.311487+0000","last_became_peered":"2024-03-09T14:28:36.311487+0000","last_unstale":"2024-03-09T15:31:35.776757+0000","last_undegraded":"2024-03-09T15:31:35.776757+0000","last_fullsized":"2024-03-09T15:31:35.776757+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:04:59.933448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"261'39","reported_seq":551,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:01.018792+0000","last_change":"2024-03-09T14:28:38.333369+0000","last_active":"2024-03-09T15:34:01.018792+0000","last_peered":"2024-03-09T15:34:01.018792+0000","last_clean":"2024-03-09T15:34:01.018792+0000","last_became_active":"2024-03-09T14:28:38.332610+0000","last_became_peered":"2024-03-09T14:28:38.332610+0000","last_unstale":"2024-03-09T15:34:01.018792+0000","last_undegraded":"2024-03-09T15:34:01.018792+0000","last_fullsized":"2024-03-09T15:34:01.018792+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":41,"num_write":30,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":498,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:41.178031+0000","last_change":"2024-03-09T14:28:40.368708+0000","last_active":"2024-03-09T15:31:41.178031+0000","last_peered":"2024-03-09T15:31:41.178031+0000","last_clean":"2024-03-09T15:31:41.178031+0000","last_became_active":"2024-03-09T14:28:40.367795+0000","last_became_peered":"2024-03-09T14:28:40.367795+0000","last_unstale":"2024-03-09T15:31:41.178031+0000","last_undegraded":"2024-03-09T15:31:41.178031+0000","last_fullsized":"2024-03-09T15:31:41.178031+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:46:56.597658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"261'170","reported_seq":689,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:46.015240+0000","last_change":"2024-03-09T14:28:42.410358+0000","last_active":"2024-03-09T15:33:46.015240+0000","last_peered":"2024-03-09T15:33:46.015240+0000","last_clean":"2024-03-09T15:33:46.015240+0000","last_became_active":"2024-03-09T14:28:42.408913+0000","last_became_peered":"2024-03-09T14:28:42.408913+0000","last_unstale":"2024-03-09T15:33:46.015240+0000","last_undegraded":"2024-03-09T15:33:46.015240+0000","last_fullsized":"2024-03-09T15:33:46.015240+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":100,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":208,"num_write":351,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"261'54","reported_seq":593,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:52.364811+0000","last_change":"2024-03-09T14:28:34.331192+0000","last_active":"2024-03-09T15:31:52.364811+0000","last_peered":"2024-03-09T15:31:52.364811+0000","last_clean":"2024-03-09T15:31:52.364811+0000","last_became_active":"2024-03-09T14:28:34.330813+0000","last_became_peered":"2024-03-09T14:28:34.330813+0000","last_unstale":"2024-03-09T15:31:52.364811+0000","last_undegraded":"2024-03-09T15:31:52.364811+0000","last_fullsized":"2024-03-09T15:31:52.364811+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:02:15.115504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":220,"num_read_kb":178,"num_write":76,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"261'127","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.457570+0000","last_change":"2024-03-09T14:28:32.313709+0000","last_active":"2024-03-09T15:36:06.457570+0000","last_peered":"2024-03-09T15:36:06.457570+0000","last_clean":"2024-03-09T15:36:06.457570+0000","last_became_active":"2024-03-09T14:28:32.311843+0000","last_became_peered":"2024-03-09T14:28:32.311843+0000","last_unstale":"2024-03-09T15:36:06.457570+0000","last_undegraded":"2024-03-09T15:36:06.457570+0000","last_fullsized":"2024-03-09T15:36:06.457570+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":100,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:51:11.937911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655838,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":157,"num_read_kb":5348,"num_write":475,"num_write_kb":35385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.176563+0000","last_change":"2024-03-09T14:28:36.324124+0000","last_active":"2024-03-09T15:31:36.176563+0000","last_peered":"2024-03-09T15:31:36.176563+0000","last_clean":"2024-03-09T15:31:36.176563+0000","last_became_active":"2024-03-09T14:28:36.323335+0000","last_became_peered":"2024-03-09T14:28:36.323335+0000","last_unstale":"2024-03-09T15:31:36.176563+0000","last_undegraded":"2024-03-09T15:31:36.176563+0000","last_fullsized":"2024-03-09T15:31:36.176563+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:09:55.020425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"261'222","reported_seq":819,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.259063+0000","last_change":"2024-03-09T14:28:38.345760+0000","last_active":"2024-03-09T15:36:06.259063+0000","last_peered":"2024-03-09T15:36:06.259063+0000","last_clean":"2024-03-09T15:36:06.259063+0000","last_became_active":"2024-03-09T14:28:38.344951+0000","last_became_peered":"2024-03-09T14:28:38.344951+0000","last_unstale":"2024-03-09T15:36:06.259063+0000","last_undegraded":"2024-03-09T15:36:06.259063+0000","last_fullsized":"2024-03-09T15:36:06.259063+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":200,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:52:01.179169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7791,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":353,"num_read_kb":629,"num_write":244,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.766053+0000","last_change":"2024-03-09T14:28:40.374861+0000","last_active":"2024-03-09T15:31:40.766053+0000","last_peered":"2024-03-09T15:31:40.766053+0000","last_clean":"2024-03-09T15:31:40.766053+0000","last_became_active":"2024-03-09T14:28:40.370313+0000","last_became_peered":"2024-03-09T14:28:40.370313+0000","last_unstale":"2024-03-09T15:31:40.766053+0000","last_undegraded":"2024-03-09T15:31:40.766053+0000","last_fullsized":"2024-03-09T15:31:40.766053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:20:51.501674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"261'45","reported_seq":538,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:07.675530+0000","last_change":"2024-03-09T14:28:42.405094+0000","last_active":"2024-03-09T15:35:07.675530+0000","last_peered":"2024-03-09T15:35:07.675530+0000","last_clean":"2024-03-09T15:35:07.675530+0000","last_became_active":"2024-03-09T14:28:42.404360+0000","last_became_peered":"2024-03-09T14:28:42.404360+0000","last_unstale":"2024-03-09T15:35:07.675530+0000","last_undegraded":"2024-03-09T15:35:07.675530+0000","last_fullsized":"2024-03-09T15:35:07.675530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:33:45.977796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":43,"num_write":116,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"261'87","reported_seq":640,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.776531+0000","last_change":"2024-03-09T14:28:34.313014+0000","last_active":"2024-03-09T15:31:35.776531+0000","last_peered":"2024-03-09T15:31:35.776531+0000","last_clean":"2024-03-09T15:31:35.776531+0000","last_became_active":"2024-03-09T14:28:34.305301+0000","last_became_peered":"2024-03-09T14:28:34.305301+0000","last_unstale":"2024-03-09T15:31:35.776531+0000","last_undegraded":"2024-03-09T15:31:35.776531+0000","last_fullsized":"2024-03-09T15:31:35.776531+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:46:37.987444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":317,"num_write":145,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"261'143","reported_seq":700,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.526666+0000","last_change":"2024-03-09T14:28:32.293199+0000","last_active":"2024-03-09T15:36:06.526666+0000","last_peered":"2024-03-09T15:36:06.526666+0000","last_clean":"2024-03-09T15:36:06.526666+0000","last_became_active":"2024-03-09T14:28:32.292393+0000","last_became_peered":"2024-03-09T14:28:32.292393+0000","last_unstale":"2024-03-09T15:36:06.526666+0000","last_undegraded":"2024-03-09T15:36:06.526666+0000","last_fullsized":"2024-03-09T15:36:06.526666+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":100,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6900333,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":188,"num_read_kb":14616,"num_write":468,"num_write_kb":23484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:35.764963+0000","last_change":"2024-03-09T14:28:36.329882+0000","last_active":"2024-03-09T15:31:35.764963+0000","last_peered":"2024-03-09T15:31:35.764963+0000","last_clean":"2024-03-09T15:31:35.764963+0000","last_became_active":"2024-03-09T14:28:36.328946+0000","last_became_peered":"2024-03-09T14:28:36.328946+0000","last_unstale":"2024-03-09T15:31:35.764963+0000","last_undegraded":"2024-03-09T15:31:35.764963+0000","last_fullsized":"2024-03-09T15:31:35.764963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:23:54.027658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"261'168","reported_seq":752,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.194863+0000","last_change":"2024-03-09T14:28:38.336043+0000","last_active":"2024-03-09T15:36:06.194863+0000","last_peered":"2024-03-09T15:36:06.194863+0000","last_clean":"2024-03-09T15:36:06.194863+0000","last_became_active":"2024-03-09T14:28:38.335107+0000","last_became_peered":"2024-03-09T14:28:38.335107+0000","last_unstale":"2024-03-09T15:36:06.194863+0000","last_undegraded":"2024-03-09T15:36:06.194863+0000","last_fullsized":"2024-03-09T15:36:06.194863+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":100,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:08:12.713928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9765,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":273,"num_read_kb":518,"num_write":186,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:01.018870+0000","last_change":"2024-03-09T14:28:40.364509+0000","last_active":"2024-03-09T15:34:01.018870+0000","last_peered":"2024-03-09T15:34:01.018870+0000","last_clean":"2024-03-09T15:34:01.018870+0000","last_became_active":"2024-03-09T14:28:40.363778+0000","last_became_peered":"2024-03-09T14:28:40.363778+0000","last_unstale":"2024-03-09T15:34:01.018870+0000","last_undegraded":"2024-03-09T15:34:01.018870+0000","last_fullsized":"2024-03-09T15:34:01.018870+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"261'159","reported_seq":733,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784514+0000","last_change":"2024-03-09T14:28:34.325746+0000","last_active":"2024-03-09T15:33:10.784514+0000","last_peered":"2024-03-09T15:33:10.784514+0000","last_clean":"2024-03-09T15:33:10.784514+0000","last_became_active":"2024-03-09T14:28:34.325031+0000","last_became_peered":"2024-03-09T14:28:34.325031+0000","last_unstale":"2024-03-09T15:33:10.784514+0000","last_undegraded":"2024-03-09T15:33:10.784514+0000","last_fullsized":"2024-03-09T15:33:10.784514+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:12:45.802989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":584,"num_read_kb":461,"num_write":254,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"261'111","reported_seq":650,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.506125+0000","last_change":"2024-03-09T14:28:32.296634+0000","last_active":"2024-03-09T15:36:06.506125+0000","last_peered":"2024-03-09T15:36:06.506125+0000","last_clean":"2024-03-09T15:36:06.506125+0000","last_became_active":"2024-03-09T14:28:32.296179+0000","last_became_peered":"2024-03-09T14:28:32.296179+0000","last_unstale":"2024-03-09T15:36:06.506125+0000","last_undegraded":"2024-03-09T15:36:06.506125+0000","last_fullsized":"2024-03-09T15:36:06.506125+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:07:03.555863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263265,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":136,"num_read_kb":11385,"num_write":371,"num_write_kb":17181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.030256+0000","last_change":"2024-03-09T14:28:36.320059+0000","last_active":"2024-03-09T15:34:51.030256+0000","last_peered":"2024-03-09T15:34:51.030256+0000","last_clean":"2024-03-09T15:34:51.030256+0000","last_became_active":"2024-03-09T14:28:36.319702+0000","last_became_peered":"2024-03-09T14:28:36.319702+0000","last_unstale":"2024-03-09T15:34:51.030256+0000","last_undegraded":"2024-03-09T15:34:51.030256+0000","last_fullsized":"2024-03-09T15:34:51.030256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"261'312","reported_seq":952,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.175796+0000","last_change":"2024-03-09T14:28:38.333414+0000","last_active":"2024-03-09T15:36:06.175796+0000","last_peered":"2024-03-09T15:36:06.175796+0000","last_clean":"2024-03-09T15:36:06.175796+0000","last_became_active":"2024-03-09T14:28:38.332661+0000","last_became_peered":"2024-03-09T14:28:38.332661+0000","last_unstale":"2024-03-09T15:36:06.175796+0000","last_undegraded":"2024-03-09T15:36:06.175796+0000","last_fullsized":"2024-03-09T15:36:06.175796+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":300,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:09:54.404475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16427,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":565,"num_read_kb":1160,"num_write":446,"num_write_kb":361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":495,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:41.195688+0000","last_change":"2024-03-09T14:28:40.379385+0000","last_active":"2024-03-09T15:33:41.195688+0000","last_peered":"2024-03-09T15:33:41.195688+0000","last_clean":"2024-03-09T15:33:41.195688+0000","last_became_active":"2024-03-09T14:28:40.378829+0000","last_became_peered":"2024-03-09T14:28:40.378829+0000","last_unstale":"2024-03-09T15:33:41.195688+0000","last_undegraded":"2024-03-09T15:33:41.195688+0000","last_fullsized":"2024-03-09T15:33:41.195688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:25:24.764687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"261'124","reported_seq":623,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:00.333094+0000","last_change":"2024-03-09T14:28:42.393487+0000","last_active":"2024-03-09T15:33:00.333094+0000","last_peered":"2024-03-09T15:33:00.333094+0000","last_clean":"2024-03-09T15:33:00.333094+0000","last_became_active":"2024-03-09T14:28:42.392509+0000","last_became_peered":"2024-03-09T14:28:42.392509+0000","last_unstale":"2024-03-09T15:33:00.333094+0000","last_undegraded":"2024-03-09T15:33:00.333094+0000","last_fullsized":"2024-03-09T15:33:00.333094+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:53:46.680420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":643,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":173,"num_write":275,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"261'132","reported_seq":685,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.030230+0000","last_change":"2024-03-09T14:28:34.330979+0000","last_active":"2024-03-09T15:34:51.030230+0000","last_peered":"2024-03-09T15:34:51.030230+0000","last_clean":"2024-03-09T15:34:51.030230+0000","last_became_active":"2024-03-09T14:28:34.330686+0000","last_became_peered":"2024-03-09T14:28:34.330686+0000","last_unstale":"2024-03-09T15:34:51.030230+0000","last_undegraded":"2024-03-09T15:34:51.030230+0000","last_fullsized":"2024-03-09T15:34:51.030230+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":527,"num_read_kb":451,"num_write":212,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"261'133","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.444439+0000","last_change":"2024-03-09T14:28:32.301602+0000","last_active":"2024-03-09T15:36:06.444439+0000","last_peered":"2024-03-09T15:36:06.444439+0000","last_clean":"2024-03-09T15:36:06.444439+0000","last_became_active":"2024-03-09T14:28:32.301087+0000","last_became_peered":"2024-03-09T14:28:32.301087+0000","last_unstale":"2024-03-09T15:36:06.444439+0000","last_undegraded":"2024-03-09T15:36:06.444439+0000","last_fullsized":"2024-03-09T15:36:06.444439+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:13:11.042530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":137,"num_read_kb":3393,"num_write":460,"num_write_kb":23270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784490+0000","last_change":"2024-03-09T14:28:36.329619+0000","last_active":"2024-03-09T15:33:10.784490+0000","last_peered":"2024-03-09T15:33:10.784490+0000","last_clean":"2024-03-09T15:33:10.784490+0000","last_became_active":"2024-03-09T14:28:36.328814+0000","last_became_peered":"2024-03-09T14:28:36.328814+0000","last_unstale":"2024-03-09T15:33:10.784490+0000","last_undegraded":"2024-03-09T15:33:10.784490+0000","last_fullsized":"2024-03-09T15:33:10.784490+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:24:43.021216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"261'252","reported_seq":856,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.222241+0000","last_change":"2024-03-09T14:28:38.332894+0000","last_active":"2024-03-09T15:36:06.222241+0000","last_peered":"2024-03-09T15:36:06.222241+0000","last_clean":"2024-03-09T15:36:06.222241+0000","last_became_active":"2024-03-09T14:28:38.331930+0000","last_became_peered":"2024-03-09T14:28:38.331930+0000","last_unstale":"2024-03-09T15:36:06.222241+0000","last_undegraded":"2024-03-09T15:36:06.222241+0000","last_fullsized":"2024-03-09T15:36:06.222241+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":200,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:36:58.809794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21830,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":424,"num_read_kb":798,"num_write":342,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:40.964551+0000","last_change":"2024-03-09T14:28:40.367660+0000","last_active":"2024-03-09T15:33:40.964551+0000","last_peered":"2024-03-09T15:33:40.964551+0000","last_clean":"2024-03-09T15:33:40.964551+0000","last_became_active":"2024-03-09T14:28:40.366723+0000","last_became_peered":"2024-03-09T14:28:40.366723+0000","last_unstale":"2024-03-09T15:33:40.964551+0000","last_undegraded":"2024-03-09T15:33:40.964551+0000","last_fullsized":"2024-03-09T15:33:40.964551+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:42:24.773777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"261'108","reported_seq":613,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:31.400225+0000","last_change":"2024-03-09T14:28:42.395993+0000","last_active":"2024-03-09T15:33:31.400225+0000","last_peered":"2024-03-09T15:33:31.400225+0000","last_clean":"2024-03-09T15:33:31.400225+0000","last_became_active":"2024-03-09T14:28:42.395375+0000","last_became_peered":"2024-03-09T14:28:42.395375+0000","last_unstale":"2024-03-09T15:33:31.400225+0000","last_undegraded":"2024-03-09T15:33:31.400225+0000","last_fullsized":"2024-03-09T15:33:31.400225+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:57:19.143770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1199,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":139,"num_read_kb":136,"num_write":341,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"261'99","reported_seq":686,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:45.201625+0000","last_change":"2024-03-09T14:28:34.312670+0000","last_active":"2024-03-09T15:34:45.201625+0000","last_peered":"2024-03-09T15:34:45.201625+0000","last_clean":"2024-03-09T15:34:45.201625+0000","last_became_active":"2024-03-09T14:28:34.312297+0000","last_became_peered":"2024-03-09T14:28:34.312297+0000","last_unstale":"2024-03-09T15:34:45.201625+0000","last_undegraded":"2024-03-09T15:34:45.201625+0000","last_fullsized":"2024-03-09T15:34:45.201625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:36:43.430324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":415,"num_read_kb":347,"num_write":146,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"261'116","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.491659+0000","last_change":"2024-03-09T14:28:32.314186+0000","last_active":"2024-03-09T15:36:06.491659+0000","last_peered":"2024-03-09T15:36:06.491659+0000","last_clean":"2024-03-09T15:36:06.491659+0000","last_became_active":"2024-03-09T14:28:32.312937+0000","last_became_peered":"2024-03-09T14:28:32.312937+0000","last_unstale":"2024-03-09T15:36:06.491659+0000","last_undegraded":"2024-03-09T15:36:06.491659+0000","last_fullsized":"2024-03-09T15:36:06.491659+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":100,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:33:36.448497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262175,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":159,"num_read_kb":47424,"num_write":430,"num_write_kb":24173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191452+0000","last_change":"2024-03-09T14:28:36.316936+0000","last_active":"2024-03-09T15:33:36.191452+0000","last_peered":"2024-03-09T15:33:36.191452+0000","last_clean":"2024-03-09T15:33:36.191452+0000","last_became_active":"2024-03-09T14:28:36.316094+0000","last_became_peered":"2024-03-09T14:28:36.316094+0000","last_unstale":"2024-03-09T15:33:36.191452+0000","last_undegraded":"2024-03-09T15:33:36.191452+0000","last_fullsized":"2024-03-09T15:33:36.191452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:41:42.620548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"261'191","reported_seq":777,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.512158+0000","last_change":"2024-03-09T14:28:38.596294+0000","last_active":"2024-03-09T15:36:06.512158+0000","last_peered":"2024-03-09T15:36:06.512158+0000","last_clean":"2024-03-09T15:36:06.512158+0000","last_became_active":"2024-03-09T14:28:38.595522+0000","last_became_peered":"2024-03-09T14:28:38.595522+0000","last_unstale":"2024-03-09T15:36:06.512158+0000","last_undegraded":"2024-03-09T15:36:06.512158+0000","last_fullsized":"2024-03-09T15:36:06.512158+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":100,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:20:49.471413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12058,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":311,"num_read_kb":573,"num_write":216,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:50.952318+0000","last_change":"2024-03-09T14:28:40.377325+0000","last_active":"2024-03-09T15:32:50.952318+0000","last_peered":"2024-03-09T15:32:50.952318+0000","last_clean":"2024-03-09T15:32:50.952318+0000","last_became_active":"2024-03-09T14:28:40.376549+0000","last_became_peered":"2024-03-09T14:28:40.376549+0000","last_unstale":"2024-03-09T15:32:50.952318+0000","last_undegraded":"2024-03-09T15:32:50.952318+0000","last_fullsized":"2024-03-09T15:32:50.952318+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:59:06.393597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"261'42","reported_seq":537,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.728760+0000","last_change":"2024-03-09T14:28:42.395593+0000","last_active":"2024-03-09T15:35:58.728760+0000","last_peered":"2024-03-09T15:35:58.728760+0000","last_clean":"2024-03-09T15:35:58.728760+0000","last_became_active":"2024-03-09T14:28:42.394780+0000","last_became_peered":"2024-03-09T14:28:42.394780+0000","last_unstale":"2024-03-09T15:35:58.728760+0000","last_undegraded":"2024-03-09T15:35:58.728760+0000","last_fullsized":"2024-03-09T15:35:58.728760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:21:45.395383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":84,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"261'68","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:22.422461+0000","last_change":"2024-03-09T14:28:34.313519+0000","last_active":"2024-03-09T15:34:22.422461+0000","last_peered":"2024-03-09T15:34:22.422461+0000","last_clean":"2024-03-09T15:34:22.422461+0000","last_became_active":"2024-03-09T14:28:34.313125+0000","last_became_peered":"2024-03-09T14:28:34.313125+0000","last_unstale":"2024-03-09T15:34:22.422461+0000","last_undegraded":"2024-03-09T15:34:22.422461+0000","last_fullsized":"2024-03-09T15:34:22.422461+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:57:07.167382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"261'187","reported_seq":792,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.486344+0000","last_change":"2024-03-09T14:28:32.304248+0000","last_active":"2024-03-09T15:36:06.486344+0000","last_peered":"2024-03-09T15:36:06.486344+0000","last_clean":"2024-03-09T15:36:06.486344+0000","last_became_active":"2024-03-09T14:28:32.303835+0000","last_became_peered":"2024-03-09T14:28:32.303835+0000","last_unstale":"2024-03-09T15:36:06.486344+0000","last_undegraded":"2024-03-09T15:36:06.486344+0000","last_fullsized":"2024-03-09T15:36:06.486344+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":100,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":371,"num_read_kb":1431,"num_write":999,"num_write_kb":8832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963406+0000","last_change":"2024-03-09T14:28:36.322532+0000","last_active":"2024-03-09T15:33:35.963406+0000","last_peered":"2024-03-09T15:33:35.963406+0000","last_clean":"2024-03-09T15:33:35.963406+0000","last_became_active":"2024-03-09T14:28:36.322060+0000","last_became_peered":"2024-03-09T14:28:36.322060+0000","last_unstale":"2024-03-09T15:33:35.963406+0000","last_undegraded":"2024-03-09T15:33:35.963406+0000","last_fullsized":"2024-03-09T15:33:35.963406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:48:28.324190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"261'89","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:32:46.180393+0000","last_change":"2024-03-09T14:28:38.596598+0000","last_active":"2024-03-09T15:32:46.180393+0000","last_peered":"2024-03-09T15:32:46.180393+0000","last_clean":"2024-03-09T15:32:46.180393+0000","last_became_active":"2024-03-09T14:28:38.595749+0000","last_became_peered":"2024-03-09T14:28:38.595749+0000","last_unstale":"2024-03-09T15:32:46.180393+0000","last_undegraded":"2024-03-09T15:32:46.180393+0000","last_fullsized":"2024-03-09T15:32:46.180393+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:47:35.161583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":132,"num_read_kb":127,"num_write":124,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"261'489","reported_seq":2257,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.716779+0000","last_change":"2024-03-09T14:28:40.377689+0000","last_active":"2024-03-09T15:36:05.716779+0000","last_peered":"2024-03-09T15:36:05.716779+0000","last_clean":"2024-03-09T15:36:05.716779+0000","last_became_active":"2024-03-09T14:28:40.376946+0000","last_became_peered":"2024-03-09T14:28:40.376946+0000","last_unstale":"2024-03-09T15:36:05.716779+0000","last_undegraded":"2024-03-09T15:36:05.716779+0000","last_fullsized":"2024-03-09T15:36:05.716779+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":400,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:50:25.914007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.10","version":"261'45","reported_seq":549,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:48.564964+0000","last_change":"2024-03-09T14:28:42.399023+0000","last_active":"2024-03-09T15:33:48.564964+0000","last_peered":"2024-03-09T15:33:48.564964+0000","last_clean":"2024-03-09T15:33:48.564964+0000","last_became_active":"2024-03-09T14:28:42.397829+0000","last_became_peered":"2024-03-09T14:28:42.397829+0000","last_unstale":"2024-03-09T15:33:48.564964+0000","last_undegraded":"2024-03-09T15:33:48.564964+0000","last_fullsized":"2024-03-09T15:33:48.564964+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:36:00.626046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":97,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"261'144","reported_seq":675,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.191372+0000","last_change":"2024-03-09T14:28:34.321426+0000","last_active":"2024-03-09T15:33:36.191372+0000","last_peered":"2024-03-09T15:33:36.191372+0000","last_clean":"2024-03-09T15:33:36.191372+0000","last_became_active":"2024-03-09T14:28:34.319891+0000","last_became_peered":"2024-03-09T14:28:34.319891+0000","last_unstale":"2024-03-09T15:33:36.191372+0000","last_undegraded":"2024-03-09T15:33:36.191372+0000","last_fullsized":"2024-03-09T15:33:36.191372+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:25:46.129222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":571,"num_read_kb":433,"num_write":263,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"261'161","reported_seq":807,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.506189+0000","last_change":"2024-03-09T14:28:32.323880+0000","last_active":"2024-03-09T15:36:06.506189+0000","last_peered":"2024-03-09T15:36:06.506189+0000","last_clean":"2024-03-09T15:36:06.506189+0000","last_became_active":"2024-03-09T14:28:32.323198+0000","last_became_peered":"2024-03-09T14:28:32.323198+0000","last_unstale":"2024-03-09T15:36:06.506189+0000","last_undegraded":"2024-03-09T15:36:06.506189+0000","last_fullsized":"2024-03-09T15:36:06.506189+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:10:52.793653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4801550,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":406,"num_read_kb":2095,"num_write":652,"num_write_kb":25158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.205346+0000","last_change":"2024-03-09T14:28:36.329314+0000","last_active":"2024-03-09T15:33:36.205346+0000","last_peered":"2024-03-09T15:33:36.205346+0000","last_clean":"2024-03-09T15:33:36.205346+0000","last_became_active":"2024-03-09T14:28:36.328529+0000","last_became_peered":"2024-03-09T14:28:36.328529+0000","last_unstale":"2024-03-09T15:33:36.205346+0000","last_undegraded":"2024-03-09T15:33:36.205346+0000","last_fullsized":"2024-03-09T15:33:36.205346+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:42:17.375037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"261'224","reported_seq":826,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.002456+0000","last_change":"2024-03-09T14:28:38.335758+0000","last_active":"2024-03-09T15:36:06.002456+0000","last_peered":"2024-03-09T15:36:06.002456+0000","last_clean":"2024-03-09T15:36:06.002456+0000","last_became_active":"2024-03-09T14:28:38.334991+0000","last_became_peered":"2024-03-09T14:28:38.334991+0000","last_unstale":"2024-03-09T15:36:06.002456+0000","last_undegraded":"2024-03-09T15:36:06.002456+0000","last_fullsized":"2024-03-09T15:36:06.002456+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":200,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:19:26.980708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13314,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":373,"num_read_kb":687,"num_write":264,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:56.017473+0000","last_change":"2024-03-09T14:28:40.368865+0000","last_active":"2024-03-09T15:33:56.017473+0000","last_peered":"2024-03-09T15:33:56.017473+0000","last_clean":"2024-03-09T15:33:56.017473+0000","last_became_active":"2024-03-09T14:28:40.368039+0000","last_became_peered":"2024-03-09T14:28:40.368039+0000","last_unstale":"2024-03-09T15:33:56.017473+0000","last_undegraded":"2024-03-09T15:33:56.017473+0000","last_fullsized":"2024-03-09T15:33:56.017473+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"261'77","reported_seq":597,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:46.896952+0000","last_change":"2024-03-09T14:28:42.414457+0000","last_active":"2024-03-09T15:34:46.896952+0000","last_peered":"2024-03-09T15:34:46.896952+0000","last_clean":"2024-03-09T15:34:46.896952+0000","last_became_active":"2024-03-09T14:28:42.413840+0000","last_became_peered":"2024-03-09T14:28:42.413840+0000","last_unstale":"2024-03-09T15:34:46.896952+0000","last_undegraded":"2024-03-09T15:34:46.896952+0000","last_fullsized":"2024-03-09T15:34:46.896952+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:42:30.742160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":129,"num_write":187,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"261'36","reported_seq":573,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:52.977046+0000","last_change":"2024-03-09T14:28:34.318737+0000","last_active":"2024-03-09T15:35:52.977046+0000","last_peered":"2024-03-09T15:35:52.977046+0000","last_clean":"2024-03-09T15:35:52.977046+0000","last_became_active":"2024-03-09T14:28:34.314194+0000","last_became_peered":"2024-03-09T14:28:34.314194+0000","last_unstale":"2024-03-09T15:35:52.977046+0000","last_undegraded":"2024-03-09T15:35:52.977046+0000","last_fullsized":"2024-03-09T15:35:52.977046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:45:29.912431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":120,"num_write":54,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"261'147","reported_seq":800,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.524216+0000","last_change":"2024-03-09T14:28:32.309431+0000","last_active":"2024-03-09T15:36:06.524216+0000","last_peered":"2024-03-09T15:36:06.524216+0000","last_clean":"2024-03-09T15:36:06.524216+0000","last_became_active":"2024-03-09T14:28:32.307702+0000","last_became_peered":"2024-03-09T14:28:32.307702+0000","last_unstale":"2024-03-09T15:36:06.524216+0000","last_undegraded":"2024-03-09T15:36:06.524216+0000","last_fullsized":"2024-03-09T15:36:06.524216+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":100,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:51:59.883119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":273,"num_read_kb":1758,"num_write":483,"num_write_kb":23078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194464+0000","last_change":"2024-03-09T14:28:36.305006+0000","last_active":"2024-03-09T15:33:36.194464+0000","last_peered":"2024-03-09T15:33:36.194464+0000","last_clean":"2024-03-09T15:33:36.194464+0000","last_became_active":"2024-03-09T14:28:36.304241+0000","last_became_peered":"2024-03-09T14:28:36.304241+0000","last_unstale":"2024-03-09T15:33:36.194464+0000","last_undegraded":"2024-03-09T15:33:36.194464+0000","last_fullsized":"2024-03-09T15:33:36.194464+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:14:09.284541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"261'178","reported_seq":758,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.036773+0000","last_change":"2024-03-09T14:28:38.340000+0000","last_active":"2024-03-09T15:36:06.036773+0000","last_peered":"2024-03-09T15:36:06.036773+0000","last_clean":"2024-03-09T15:36:06.036773+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:36:06.036773+0000","last_undegraded":"2024-03-09T15:36:06.036773+0000","last_fullsized":"2024-03-09T15:36:06.036773+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":100,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":521,"num_write":190,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:50.768268+0000","last_change":"2024-03-09T14:28:40.375515+0000","last_active":"2024-03-09T15:31:50.768268+0000","last_peered":"2024-03-09T15:31:50.768268+0000","last_clean":"2024-03-09T15:31:50.768268+0000","last_became_active":"2024-03-09T14:28:40.371820+0000","last_became_peered":"2024-03-09T14:28:40.371820+0000","last_unstale":"2024-03-09T15:31:50.768268+0000","last_undegraded":"2024-03-09T15:31:50.768268+0000","last_fullsized":"2024-03-09T15:31:50.768268+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:15:53.172763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.16","version":"261'53","reported_seq":567,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.020972+0000","last_change":"2024-03-09T14:28:42.392999+0000","last_active":"2024-03-09T15:33:10.020972+0000","last_peered":"2024-03-09T15:33:10.020972+0000","last_clean":"2024-03-09T15:33:10.020972+0000","last_became_active":"2024-03-09T14:28:42.392326+0000","last_became_peered":"2024-03-09T14:28:42.392326+0000","last_unstale":"2024-03-09T15:33:10.020972+0000","last_undegraded":"2024-03-09T15:33:10.020972+0000","last_fullsized":"2024-03-09T15:33:10.020972+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:19:36.796428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":772,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":99,"num_write":117,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"261'70","reported_seq":632,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:40.777840+0000","last_change":"2024-03-09T14:28:34.324293+0000","last_active":"2024-03-09T15:31:40.777840+0000","last_peered":"2024-03-09T15:31:40.777840+0000","last_clean":"2024-03-09T15:31:40.777840+0000","last_became_active":"2024-03-09T14:28:34.323969+0000","last_became_peered":"2024-03-09T14:28:34.323969+0000","last_unstale":"2024-03-09T15:31:40.777840+0000","last_undegraded":"2024-03-09T15:31:40.777840+0000","last_fullsized":"2024-03-09T15:31:40.777840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:33:30.511134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":256,"num_write":98,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"261'176","reported_seq":767,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.508734+0000","last_change":"2024-03-09T14:28:32.338014+0000","last_active":"2024-03-09T15:36:06.508734+0000","last_peered":"2024-03-09T15:36:06.508734+0000","last_clean":"2024-03-09T15:36:06.508734+0000","last_became_active":"2024-03-09T14:28:32.337755+0000","last_became_peered":"2024-03-09T14:28:32.337755+0000","last_unstale":"2024-03-09T15:36:06.508734+0000","last_undegraded":"2024-03-09T15:36:06.508734+0000","last_fullsized":"2024-03-09T15:36:06.508734+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":100,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:19:21.576018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663535,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":299,"num_read_kb":3602,"num_write":667,"num_write_kb":11566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162410+0000","last_change":"2024-03-09T14:28:36.318326+0000","last_active":"2024-03-09T15:31:26.162410+0000","last_peered":"2024-03-09T15:31:26.162410+0000","last_clean":"2024-03-09T15:31:26.162410+0000","last_became_active":"2024-03-09T14:28:36.317569+0000","last_became_peered":"2024-03-09T14:28:36.317569+0000","last_unstale":"2024-03-09T15:31:26.162410+0000","last_undegraded":"2024-03-09T15:31:26.162410+0000","last_fullsized":"2024-03-09T15:31:26.162410+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:21:30.385778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"261'152","reported_seq":712,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.477480+0000","last_change":"2024-03-09T14:28:38.348095+0000","last_active":"2024-03-09T15:36:06.477480+0000","last_peered":"2024-03-09T15:36:06.477480+0000","last_clean":"2024-03-09T15:36:06.477480+0000","last_became_active":"2024-03-09T14:28:38.347370+0000","last_became_peered":"2024-03-09T14:28:38.347370+0000","last_unstale":"2024-03-09T15:36:06.477480+0000","last_undegraded":"2024-03-09T15:36:06.477480+0000","last_fullsized":"2024-03-09T15:36:06.477480+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":100,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:00:09.043640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18257,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":531,"num_write":172,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:41.192563+0000","last_change":"2024-03-09T14:28:40.377364+0000","last_active":"2024-03-09T15:33:41.192563+0000","last_peered":"2024-03-09T15:33:41.192563+0000","last_clean":"2024-03-09T15:33:41.192563+0000","last_became_active":"2024-03-09T14:28:40.376513+0000","last_became_peered":"2024-03-09T14:28:40.376513+0000","last_unstale":"2024-03-09T15:33:41.192563+0000","last_undegraded":"2024-03-09T15:33:41.192563+0000","last_fullsized":"2024-03-09T15:33:41.192563+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:28.590369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"261'92","reported_seq":974,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:54.162220+0000","last_change":"2024-03-09T14:28:42.410566+0000","last_active":"2024-03-09T15:35:54.162220+0000","last_peered":"2024-03-09T15:35:54.162220+0000","last_clean":"2024-03-09T15:35:54.162220+0000","last_became_active":"2024-03-09T14:28:42.409717+0000","last_became_peered":"2024-03-09T14:28:42.409717+0000","last_unstale":"2024-03-09T15:35:54.162220+0000","last_undegraded":"2024-03-09T15:35:54.162220+0000","last_fullsized":"2024-03-09T15:35:54.162220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:07:09.353068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":500,"num_read_kb":493,"num_write":215,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"261'53","reported_seq":585,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:26.162448+0000","last_change":"2024-03-09T14:28:34.314180+0000","last_active":"2024-03-09T15:31:26.162448+0000","last_peered":"2024-03-09T15:31:26.162448+0000","last_clean":"2024-03-09T15:31:26.162448+0000","last_became_active":"2024-03-09T14:28:34.313380+0000","last_became_peered":"2024-03-09T14:28:34.313380+0000","last_unstale":"2024-03-09T15:31:26.162448+0000","last_undegraded":"2024-03-09T15:31:26.162448+0000","last_fullsized":"2024-03-09T15:31:26.162448+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:29:27.979809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":197,"num_read_kb":156,"num_write":74,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"261'161","reported_seq":749,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.518065+0000","last_change":"2024-03-09T14:28:32.293553+0000","last_active":"2024-03-09T15:36:06.518065+0000","last_peered":"2024-03-09T15:36:06.518065+0000","last_clean":"2024-03-09T15:36:06.518065+0000","last_became_active":"2024-03-09T14:28:32.292862+0000","last_became_peered":"2024-03-09T14:28:32.292862+0000","last_unstale":"2024-03-09T15:36:06.518065+0000","last_undegraded":"2024-03-09T15:36:06.518065+0000","last_fullsized":"2024-03-09T15:36:06.518065+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":264,"num_read_kb":35693,"num_write":668,"num_write_kb":24072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"25'1","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:31.132760+0000","last_change":"2024-03-09T14:28:36.314323+0000","last_active":"2024-03-09T15:31:31.132760+0000","last_peered":"2024-03-09T15:31:31.132760+0000","last_clean":"2024-03-09T15:31:31.132760+0000","last_became_active":"2024-03-09T14:28:36.313405+0000","last_became_peered":"2024-03-09T14:28:36.313405+0000","last_unstale":"2024-03-09T15:31:31.132760+0000","last_undegraded":"2024-03-09T15:31:31.132760+0000","last_fullsized":"2024-03-09T15:31:31.132760+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:46:02.220171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"261'210","reported_seq":807,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.531121+0000","last_change":"2024-03-09T14:28:38.333268+0000","last_active":"2024-03-09T15:36:06.531121+0000","last_peered":"2024-03-09T15:36:06.531121+0000","last_clean":"2024-03-09T15:36:06.531121+0000","last_became_active":"2024-03-09T14:28:38.330571+0000","last_became_peered":"2024-03-09T14:28:38.330571+0000","last_unstale":"2024-03-09T15:36:06.531121+0000","last_undegraded":"2024-03-09T15:36:06.531121+0000","last_fullsized":"2024-03-09T15:36:06.531121+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":100,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:13:06.110510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11093,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":349,"num_read_kb":659,"num_write":246,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":493,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739939+0000","last_change":"2024-03-09T14:28:40.385742+0000","last_active":"2024-03-09T15:36:10.739939+0000","last_peered":"2024-03-09T15:36:10.739939+0000","last_clean":"2024-03-09T15:36:10.739939+0000","last_became_active":"2024-03-09T14:28:40.385080+0000","last_became_peered":"2024-03-09T14:28:40.385080+0000","last_unstale":"2024-03-09T15:36:10.739939+0000","last_undegraded":"2024-03-09T15:36:10.739939+0000","last_fullsized":"2024-03-09T15:36:10.739939+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"261'69","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:11.304332+0000","last_change":"2024-03-09T14:28:42.413260+0000","last_active":"2024-03-09T15:34:11.304332+0000","last_peered":"2024-03-09T15:34:11.304332+0000","last_clean":"2024-03-09T15:34:11.304332+0000","last_became_active":"2024-03-09T14:28:42.412616+0000","last_became_peered":"2024-03-09T14:28:42.412616+0000","last_unstale":"2024-03-09T15:34:11.304332+0000","last_undegraded":"2024-03-09T15:34:11.304332+0000","last_fullsized":"2024-03-09T15:34:11.304332+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:28:05.834021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":49,"num_write":163,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.27","version":"261'50","reported_seq":597,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963528+0000","last_change":"2024-03-09T14:28:34.307137+0000","last_active":"2024-03-09T15:33:35.963528+0000","last_peered":"2024-03-09T15:33:35.963528+0000","last_clean":"2024-03-09T15:33:35.963528+0000","last_became_active":"2024-03-09T14:28:34.306685+0000","last_became_peered":"2024-03-09T14:28:34.306685+0000","last_unstale":"2024-03-09T15:33:35.963528+0000","last_undegraded":"2024-03-09T15:33:35.963528+0000","last_fullsized":"2024-03-09T15:33:35.963528+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:19:14.669885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":204,"num_read_kb":164,"num_write":70,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"261'140","reported_seq":723,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.457964+0000","last_change":"2024-03-09T14:28:32.334098+0000","last_active":"2024-03-09T15:36:06.457964+0000","last_peered":"2024-03-09T15:36:06.457964+0000","last_clean":"2024-03-09T15:36:06.457964+0000","last_became_active":"2024-03-09T14:28:32.333008+0000","last_became_peered":"2024-03-09T14:28:32.333008+0000","last_unstale":"2024-03-09T15:36:06.457964+0000","last_undegraded":"2024-03-09T15:36:06.457964+0000","last_fullsized":"2024-03-09T15:36:06.457964+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":100,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:19:21.576018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655838,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":237,"num_read_kb":10390,"num_write":554,"num_write_kb":33266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.26","version":"261'181","reported_seq":772,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963471+0000","last_change":"2024-03-09T14:28:34.314634+0000","last_active":"2024-03-09T15:33:35.963471+0000","last_peered":"2024-03-09T15:33:35.963471+0000","last_clean":"2024-03-09T15:33:35.963471+0000","last_became_active":"2024-03-09T14:28:34.314007+0000","last_became_peered":"2024-03-09T14:28:34.314007+0000","last_unstale":"2024-03-09T15:33:35.963471+0000","last_undegraded":"2024-03-09T15:33:35.963471+0000","last_fullsized":"2024-03-09T15:33:35.963471+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":100,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:50:26.398988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":729,"num_read_kb":586,"num_write":272,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"261'145","reported_seq":694,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.522651+0000","last_change":"2024-03-09T14:28:32.287770+0000","last_active":"2024-03-09T15:36:06.522651+0000","last_peered":"2024-03-09T15:36:06.522651+0000","last_clean":"2024-03-09T15:36:06.522651+0000","last_became_active":"2024-03-09T14:28:32.286609+0000","last_became_peered":"2024-03-09T14:28:32.286609+0000","last_unstale":"2024-03-09T15:36:06.522651+0000","last_undegraded":"2024-03-09T15:36:06.522651+0000","last_fullsized":"2024-03-09T15:36:06.522651+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:14.582228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":810211,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":147,"num_read_kb":1907,"num_write":492,"num_write_kb":32192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"261'93","reported_seq":657,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:36.194646+0000","last_change":"2024-03-09T14:28:34.325388+0000","last_active":"2024-03-09T15:33:36.194646+0000","last_peered":"2024-03-09T15:33:36.194646+0000","last_clean":"2024-03-09T15:33:36.194646+0000","last_became_active":"2024-03-09T14:28:34.323541+0000","last_became_peered":"2024-03-09T14:28:34.323541+0000","last_unstale":"2024-03-09T15:33:36.194646+0000","last_undegraded":"2024-03-09T15:33:36.194646+0000","last_fullsized":"2024-03-09T15:33:36.194646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:35:47.676249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":384,"num_read_kb":306,"num_write":146,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"261'171","reported_seq":767,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.458201+0000","last_change":"2024-03-09T14:28:32.299059+0000","last_active":"2024-03-09T15:36:06.458201+0000","last_peered":"2024-03-09T15:36:06.458201+0000","last_clean":"2024-03-09T15:36:06.458201+0000","last_became_active":"2024-03-09T14:28:32.295699+0000","last_became_peered":"2024-03-09T14:28:32.295699+0000","last_unstale":"2024-03-09T15:36:06.458201+0000","last_undegraded":"2024-03-09T15:36:06.458201+0000","last_fullsized":"2024-03-09T15:36:06.458201+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":100,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:14:50.303305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":297,"num_read_kb":14189,"num_write":716,"num_write_kb":33521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"261'101","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:35.963269+0000","last_change":"2024-03-09T14:28:34.322186+0000","last_active":"2024-03-09T15:33:35.963269+0000","last_peered":"2024-03-09T15:33:35.963269+0000","last_clean":"2024-03-09T15:33:35.963269+0000","last_became_active":"2024-03-09T14:28:34.321647+0000","last_became_peered":"2024-03-09T14:28:34.321647+0000","last_unstale":"2024-03-09T15:33:35.963269+0000","last_undegraded":"2024-03-09T15:33:35.963269+0000","last_fullsized":"2024-03-09T15:33:35.963269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:19:44.897314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":329,"num_write":161,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"261'128","reported_seq":681,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.494943+0000","last_change":"2024-03-09T14:28:32.298327+0000","last_active":"2024-03-09T15:36:06.494943+0000","last_peered":"2024-03-09T15:36:06.494943+0000","last_clean":"2024-03-09T15:36:06.494943+0000","last_became_active":"2024-03-09T14:28:32.297866+0000","last_became_peered":"2024-03-09T14:28:32.297866+0000","last_unstale":"2024-03-09T15:36:06.494943+0000","last_undegraded":"2024-03-09T15:36:06.494943+0000","last_fullsized":"2024-03-09T15:36:06.494943+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":163,"num_read_kb":3467,"num_write":495,"num_write_kb":23815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"261'52","reported_seq":601,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:25.801400+0000","last_change":"2024-03-09T14:28:34.330916+0000","last_active":"2024-03-09T15:33:25.801400+0000","last_peered":"2024-03-09T15:33:25.801400+0000","last_clean":"2024-03-09T15:33:25.801400+0000","last_became_active":"2024-03-09T14:28:34.330511+0000","last_became_peered":"2024-03-09T14:28:34.330511+0000","last_unstale":"2024-03-09T15:33:25.801400+0000","last_undegraded":"2024-03-09T15:33:25.801400+0000","last_fullsized":"2024-03-09T15:33:25.801400+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T14:29:16.007058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":225,"num_read_kb":185,"num_write":82,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"261'118","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.460476+0000","last_change":"2024-03-09T14:28:32.326489+0000","last_active":"2024-03-09T15:36:06.460476+0000","last_peered":"2024-03-09T15:36:06.460476+0000","last_clean":"2024-03-09T15:36:06.460476+0000","last_became_active":"2024-03-09T14:28:32.326148+0000","last_became_peered":"2024-03-09T14:28:32.326148+0000","last_unstale":"2024-03-09T15:36:06.460476+0000","last_undegraded":"2024-03-09T15:36:06.460476+0000","last_fullsized":"2024-03-09T15:36:06.460476+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:59:13.121365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":173,"num_read_kb":5614,"num_write":451,"num_write_kb":21169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"261'134","reported_seq":685,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:11.185748+0000","last_change":"2024-03-09T14:28:34.323012+0000","last_active":"2024-03-09T15:33:11.185748+0000","last_peered":"2024-03-09T15:33:11.185748+0000","last_clean":"2024-03-09T15:33:11.185748+0000","last_became_active":"2024-03-09T14:28:34.322225+0000","last_became_peered":"2024-03-09T14:28:34.322225+0000","last_unstale":"2024-03-09T15:33:11.185748+0000","last_undegraded":"2024-03-09T15:33:11.185748+0000","last_fullsized":"2024-03-09T15:33:11.185748+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:33:31.665609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":498,"num_read_kb":399,"num_write":194,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"261'171","reported_seq":737,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.508869+0000","last_change":"2024-03-09T14:28:32.300246+0000","last_active":"2024-03-09T15:36:06.508869+0000","last_peered":"2024-03-09T15:36:06.508869+0000","last_clean":"2024-03-09T15:36:06.508869+0000","last_became_active":"2024-03-09T14:28:32.299839+0000","last_became_peered":"2024-03-09T14:28:32.299839+0000","last_unstale":"2024-03-09T15:36:06.508869+0000","last_undegraded":"2024-03-09T15:36:06.508869+0000","last_fullsized":"2024-03-09T15:36:06.508869+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":100,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:05:01.790489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242922,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":18833,"num_write":606,"num_write_kb":39169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"261'74","reported_seq":608,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:17.871898+0000","last_change":"2024-03-09T14:28:34.315670+0000","last_active":"2024-03-09T15:34:17.871898+0000","last_peered":"2024-03-09T15:34:17.871898+0000","last_clean":"2024-03-09T15:34:17.871898+0000","last_became_active":"2024-03-09T14:28:34.315302+0000","last_became_peered":"2024-03-09T14:28:34.315302+0000","last_unstale":"2024-03-09T15:34:17.871898+0000","last_undegraded":"2024-03-09T15:34:17.871898+0000","last_fullsized":"2024-03-09T15:34:17.871898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:54:03.674581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":224,"num_write":116,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"261'149","reported_seq":703,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.455757+0000","last_change":"2024-03-09T14:28:32.336827+0000","last_active":"2024-03-09T15:36:06.455757+0000","last_peered":"2024-03-09T15:36:06.455757+0000","last_clean":"2024-03-09T15:36:06.455757+0000","last_became_active":"2024-03-09T14:28:32.334117+0000","last_became_peered":"2024-03-09T14:28:32.334117+0000","last_unstale":"2024-03-09T15:36:06.455757+0000","last_undegraded":"2024-03-09T15:36:06.455757+0000","last_fullsized":"2024-03-09T15:36:06.455757+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:58:15.793689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":182,"num_read_kb":192,"num_write":563,"num_write_kb":14566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"261'48","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:10.784320+0000","last_change":"2024-03-09T14:28:34.323974+0000","last_active":"2024-03-09T15:33:10.784320+0000","last_peered":"2024-03-09T15:33:10.784320+0000","last_clean":"2024-03-09T15:33:10.784320+0000","last_became_active":"2024-03-09T14:28:34.323202+0000","last_became_peered":"2024-03-09T14:28:34.323202+0000","last_unstale":"2024-03-09T15:33:10.784320+0000","last_undegraded":"2024-03-09T15:33:10.784320+0000","last_fullsized":"2024-03-09T15:33:10.784320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:33:12.688593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":219,"num_read_kb":183,"num_write":70,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"261'138","reported_seq":686,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.472802+0000","last_change":"2024-03-09T14:28:32.300258+0000","last_active":"2024-03-09T15:36:06.472802+0000","last_peered":"2024-03-09T15:36:06.472802+0000","last_clean":"2024-03-09T15:36:06.472802+0000","last_became_active":"2024-03-09T14:28:32.299737+0000","last_became_peered":"2024-03-09T14:28:32.299737+0000","last_unstale":"2024-03-09T15:36:06.472802+0000","last_undegraded":"2024-03-09T15:36:06.472802+0000","last_fullsized":"2024-03-09T15:36:06.472802+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":100,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:26:30.693287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":168,"num_read_kb":12613,"num_write":521,"num_write_kb":31618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"261'94","reported_seq":645,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:56.017761+0000","last_change":"2024-03-09T14:28:34.315866+0000","last_active":"2024-03-09T15:33:56.017761+0000","last_peered":"2024-03-09T15:33:56.017761+0000","last_clean":"2024-03-09T15:33:56.017761+0000","last_became_active":"2024-03-09T14:28:34.315451+0000","last_became_peered":"2024-03-09T14:28:34.315451+0000","last_unstale":"2024-03-09T15:33:56.017761+0000","last_undegraded":"2024-03-09T15:33:56.017761+0000","last_fullsized":"2024-03-09T15:33:56.017761+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":374,"num_read_kb":293,"num_write":145,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"261'144","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.491057+0000","last_change":"2024-03-09T14:28:32.305393+0000","last_active":"2024-03-09T15:36:06.491057+0000","last_peered":"2024-03-09T15:36:06.491057+0000","last_clean":"2024-03-09T15:36:06.491057+0000","last_became_active":"2024-03-09T14:28:32.304540+0000","last_became_peered":"2024-03-09T14:28:32.304540+0000","last_unstale":"2024-03-09T15:36:06.491057+0000","last_undegraded":"2024-03-09T15:36:06.491057+0000","last_fullsized":"2024-03-09T15:36:06.491057+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:12:43.181100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655899,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":200,"num_read_kb":1806,"num_write":509,"num_write_kb":18066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2e","version":"261'39","reported_seq":599,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:31:36.165250+0000","last_change":"2024-03-09T14:28:34.314876+0000","last_active":"2024-03-09T15:31:36.165250+0000","last_peered":"2024-03-09T15:31:36.165250+0000","last_clean":"2024-03-09T15:31:36.165250+0000","last_became_active":"2024-03-09T14:28:34.312399+0000","last_became_peered":"2024-03-09T14:28:34.312399+0000","last_unstale":"2024-03-09T15:31:36.165250+0000","last_undegraded":"2024-03-09T15:31:36.165250+0000","last_fullsized":"2024-03-09T15:31:36.165250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:27:43.779494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":182,"num_read_kb":156,"num_write":57,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"261'145","reported_seq":688,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.495311+0000","last_change":"2024-03-09T14:28:32.314908+0000","last_active":"2024-03-09T15:36:06.495311+0000","last_peered":"2024-03-09T15:36:06.495311+0000","last_clean":"2024-03-09T15:36:06.495311+0000","last_became_active":"2024-03-09T14:28:32.314596+0000","last_became_peered":"2024-03-09T14:28:32.314596+0000","last_unstale":"2024-03-09T15:36:06.495311+0000","last_undegraded":"2024-03-09T15:36:06.495311+0000","last_fullsized":"2024-03-09T15:36:06.495311+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:22:39.392181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263241,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":148,"num_read_kb":12793,"num_write":461,"num_write_kb":35350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"261'64","reported_seq":625,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:56.824631+0000","last_change":"2024-03-09T14:28:34.325871+0000","last_active":"2024-03-09T15:34:56.824631+0000","last_peered":"2024-03-09T15:34:56.824631+0000","last_clean":"2024-03-09T15:34:56.824631+0000","last_became_active":"2024-03-09T14:28:34.325461+0000","last_became_peered":"2024-03-09T14:28:34.325461+0000","last_unstale":"2024-03-09T15:34:56.824631+0000","last_undegraded":"2024-03-09T15:34:56.824631+0000","last_fullsized":"2024-03-09T15:34:56.824631+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:02:18.977660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":264,"num_read_kb":214,"num_write":89,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"261'159","reported_seq":735,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.453227+0000","last_change":"2024-03-09T14:28:32.313085+0000","last_active":"2024-03-09T15:36:06.453227+0000","last_peered":"2024-03-09T15:36:06.453227+0000","last_clean":"2024-03-09T15:36:06.453227+0000","last_became_active":"2024-03-09T14:28:32.311750+0000","last_became_peered":"2024-03-09T14:28:32.311750+0000","last_unstale":"2024-03-09T15:36:06.453227+0000","last_undegraded":"2024-03-09T15:36:06.453227+0000","last_fullsized":"2024-03-09T15:36:06.453227+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:56:43.188934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":213,"num_read_kb":14896,"num_write":533,"num_write_kb":41774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"261'162","reported_seq":774,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.195666+0000","last_change":"2024-03-09T14:28:34.313344+0000","last_active":"2024-03-09T15:33:51.195666+0000","last_peered":"2024-03-09T15:33:51.195666+0000","last_clean":"2024-03-09T15:33:51.195666+0000","last_became_active":"2024-03-09T14:28:34.312212+0000","last_became_peered":"2024-03-09T14:28:34.312212+0000","last_unstale":"2024-03-09T15:33:51.195666+0000","last_undegraded":"2024-03-09T15:33:51.195666+0000","last_fullsized":"2024-03-09T15:33:51.195666+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":100,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:46:46.876109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":664,"num_read_kb":532,"num_write":243,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2b","version":"261'152","reported_seq":725,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.528226+0000","last_change":"2024-03-09T14:28:32.309913+0000","last_active":"2024-03-09T15:36:06.528226+0000","last_peered":"2024-03-09T15:36:06.528226+0000","last_clean":"2024-03-09T15:36:06.528226+0000","last_became_active":"2024-03-09T14:28:32.309562+0000","last_became_peered":"2024-03-09T14:28:32.309562+0000","last_unstale":"2024-03-09T15:36:06.528226+0000","last_undegraded":"2024-03-09T15:36:06.528226+0000","last_fullsized":"2024-03-09T15:36:06.528226+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":100,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":246,"num_read_kb":5348,"num_write":573,"num_write_kb":20887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"261'117","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:51.208966+0000","last_change":"2024-03-09T14:28:34.322800+0000","last_active":"2024-03-09T15:33:51.208966+0000","last_peered":"2024-03-09T15:33:51.208966+0000","last_clean":"2024-03-09T15:33:51.208966+0000","last_became_active":"2024-03-09T14:28:34.321424+0000","last_became_peered":"2024-03-09T14:28:34.321424+0000","last_unstale":"2024-03-09T15:33:51.208966+0000","last_undegraded":"2024-03-09T15:33:51.208966+0000","last_fullsized":"2024-03-09T15:33:51.208966+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:39:19.495854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":513,"num_read_kb":409,"num_write":169,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"261'162","reported_seq":730,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.454548+0000","last_change":"2024-03-09T14:28:32.334610+0000","last_active":"2024-03-09T15:36:06.454548+0000","last_peered":"2024-03-09T15:36:06.454548+0000","last_clean":"2024-03-09T15:36:06.454548+0000","last_became_active":"2024-03-09T14:28:32.333838+0000","last_became_peered":"2024-03-09T14:28:32.333838+0000","last_unstale":"2024-03-09T15:36:06.454548+0000","last_undegraded":"2024-03-09T15:36:06.454548+0000","last_fullsized":"2024-03-09T15:36:06.454548+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":100,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:14:50.303305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":238,"num_read_kb":3315,"num_write":673,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2a","version":"261'71","reported_seq":636,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.046616+0000","last_change":"2024-03-09T14:28:34.297744+0000","last_active":"2024-03-09T15:36:06.046616+0000","last_peered":"2024-03-09T15:36:06.046616+0000","last_clean":"2024-03-09T15:36:06.046616+0000","last_became_active":"2024-03-09T14:28:34.297042+0000","last_became_peered":"2024-03-09T14:28:34.297042+0000","last_unstale":"2024-03-09T15:36:06.046616+0000","last_undegraded":"2024-03-09T15:36:06.046616+0000","last_fullsized":"2024-03-09T15:36:06.046616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":242,"num_write":99,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"261'122","reported_seq":694,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.519947+0000","last_change":"2024-03-09T14:28:32.320093+0000","last_active":"2024-03-09T15:36:06.519947+0000","last_peered":"2024-03-09T15:36:06.519947+0000","last_clean":"2024-03-09T15:36:06.519947+0000","last_became_active":"2024-03-09T14:28:32.317797+0000","last_became_peered":"2024-03-09T14:28:32.317797+0000","last_unstale":"2024-03-09T15:36:06.519947+0000","last_undegraded":"2024-03-09T15:36:06.519947+0000","last_fullsized":"2024-03-09T15:36:06.519947+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:52:33.454224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263201,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":187,"num_read_kb":4139,"num_write":475,"num_write_kb":19145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"261'113","reported_seq":668,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:50.807317+0000","last_change":"2024-03-09T14:28:34.314621+0000","last_active":"2024-03-09T15:33:50.807317+0000","last_peered":"2024-03-09T15:33:50.807317+0000","last_clean":"2024-03-09T15:33:50.807317+0000","last_became_active":"2024-03-09T14:28:34.314073+0000","last_became_peered":"2024-03-09T14:28:34.314073+0000","last_unstale":"2024-03-09T15:33:50.807317+0000","last_undegraded":"2024-03-09T15:33:50.807317+0000","last_fullsized":"2024-03-09T15:33:50.807317+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":100,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:30:53.101187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":445,"num_read_kb":351,"num_write":167,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"261'134","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.410744+0000","last_change":"2024-03-09T14:28:32.334618+0000","last_active":"2024-03-09T15:36:06.410744+0000","last_peered":"2024-03-09T15:36:06.410744+0000","last_clean":"2024-03-09T15:36:06.410744+0000","last_became_active":"2024-03-09T14:28:32.334187+0000","last_became_peered":"2024-03-09T14:28:32.334187+0000","last_unstale":"2024-03-09T15:36:06.410744+0000","last_undegraded":"2024-03-09T15:36:06.410744+0000","last_fullsized":"2024-03-09T15:36:06.410744+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:12:37.415618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2181199,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":224,"num_read_kb":1842,"num_write":561,"num_write_kb":7884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"261'4028","reported_seq":4579,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:33:46.207813+0000","last_change":"2024-03-09T14:28:34.324195+0000","last_active":"2024-03-09T15:33:46.207813+0000","last_peered":"2024-03-09T15:33:46.207813+0000","last_clean":"2024-03-09T15:33:46.207813+0000","last_became_active":"2024-03-09T14:28:34.322421+0000","last_became_peered":"2024-03-09T14:28:34.322421+0000","last_unstale":"2024-03-09T15:33:46.207813+0000","last_undegraded":"2024-03-09T15:33:46.207813+0000","last_fullsized":"2024-03-09T15:33:46.207813+0000","mapping_epoch":22,"log_start":"261'4000","ondisk_log_start":"261'4000","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:55:11.911331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14144,"num_read_kb":11461,"num_write":6038,"num_write_kb":3015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"261'130","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.504221+0000","last_change":"2024-03-09T14:28:32.324991+0000","last_active":"2024-03-09T15:36:06.504221+0000","last_peered":"2024-03-09T15:36:06.504221+0000","last_clean":"2024-03-09T15:36:06.504221+0000","last_became_active":"2024-03-09T14:28:32.324166+0000","last_became_peered":"2024-03-09T14:28:32.324166+0000","last_unstale":"2024-03-09T15:36:06.504221+0000","last_undegraded":"2024-03-09T15:36:06.504221+0000","last_fullsized":"2024-03-09T15:36:06.504221+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:55:14.982039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269861,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":9428,"num_write":515,"num_write_kb":30713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"261'141","reported_seq":719,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.457951+0000","last_change":"2024-03-09T14:28:32.288697+0000","last_active":"2024-03-09T15:36:06.457951+0000","last_peered":"2024-03-09T15:36:06.457951+0000","last_clean":"2024-03-09T15:36:06.457951+0000","last_became_active":"2024-03-09T14:28:32.287937+0000","last_became_peered":"2024-03-09T14:28:32.287937+0000","last_unstale":"2024-03-09T15:36:06.457951+0000","last_undegraded":"2024-03-09T15:36:06.457951+0000","last_fullsized":"2024-03-09T15:36:06.457951+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":100,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:01:00.707624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243043,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":214,"num_read_kb":48289,"num_write":459,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"261'161","reported_seq":727,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:06.020137+0000","last_change":"2024-03-09T14:28:34.317008+0000","last_active":"2024-03-09T15:34:06.020137+0000","last_peered":"2024-03-09T15:34:06.020137+0000","last_clean":"2024-03-09T15:34:06.020137+0000","last_became_active":"2024-03-09T14:28:34.316689+0000","last_became_peered":"2024-03-09T14:28:34.316689+0000","last_unstale":"2024-03-09T15:34:06.020137+0000","last_undegraded":"2024-03-09T15:34:06.020137+0000","last_fullsized":"2024-03-09T15:34:06.020137+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":634,"num_read_kb":490,"num_write":268,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"261'157","reported_seq":735,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.428855+0000","last_change":"2024-03-09T14:28:32.313446+0000","last_active":"2024-03-09T15:36:06.428855+0000","last_peered":"2024-03-09T15:36:06.428855+0000","last_clean":"2024-03-09T15:36:06.428855+0000","last_became_active":"2024-03-09T14:28:32.313132+0000","last_became_peered":"2024-03-09T14:28:32.313132+0000","last_unstale":"2024-03-09T15:36:06.428855+0000","last_undegraded":"2024-03-09T15:36:06.428855+0000","last_fullsized":"2024-03-09T15:36:06.428855+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:30:18.199314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262227,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":255,"num_read_kb":5401,"num_write":635,"num_write_kb":32648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"261'41","reported_seq":586,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:02.721943+0000","last_change":"2024-03-09T14:28:34.304440+0000","last_active":"2024-03-09T15:34:02.721943+0000","last_peered":"2024-03-09T15:34:02.721943+0000","last_clean":"2024-03-09T15:34:02.721943+0000","last_became_active":"2024-03-09T14:28:34.303701+0000","last_became_peered":"2024-03-09T14:28:34.303701+0000","last_unstale":"2024-03-09T15:34:02.721943+0000","last_undegraded":"2024-03-09T15:34:02.721943+0000","last_fullsized":"2024-03-09T15:34:02.721943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:22:10.704885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":166,"num_read_kb":138,"num_write":64,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":65,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3686,"num_read_kb":2614,"num_write":1236,"num_write_kb":514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":13176,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":8518,"num_read_kb":8294,"num_write":8580,"num_write_kb":2546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":26352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2011,"ondisk_log_size":2011,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":64,"acting":64,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":377960,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":12181,"num_read_kb":21090,"num_write":9023,"num_write_kb":5457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1983,"ondisk_log_size":1983,"up":64,"acting":64,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1719,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1602,"num_read_kb":1903,"num_write":22,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":38590,"num_read_kb":31236,"num_write":15314,"num_write_kb":7493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3941,"ondisk_log_size":3941,"up":128,"acting":128,"num_store_stats":8},{"poolid":68,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":135300586,"num_objects":320,"num_object_clones":0,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":14057,"num_read_kb":1121153,"num_write":34295,"num_write_kb":1701153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":271892480,"data_stored":270601172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2948,"ondisk_log_size":2948,"up":128,"acting":128,"num_store_stats":8},{"poolid":67,"num_pg":32,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":140,"num_read_kb":192739,"num_write":156,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":66,"num_pg":32,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":138,"num_read_kb":192649,"num_write":156,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575671,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293540,"kb_used_data":45568,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":94078300,"statfs":{"total":96636764160,"available":96336179200,"internally_reserved":0,"allocated":46661632,"data_stored":45012346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":253688266},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575667,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":229904,"kb_used_data":22304,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":94141936,"statfs":{"total":96636764160,"available":96401342464,"internally_reserved":0,"allocated":22839296,"data_stored":21261512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575668,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132784,"kb_used_data":61164,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94239056,"statfs":{"total":96636764160,"available":96500793344,"internally_reserved":0,"allocated":62631936,"data_stored":60939198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575672,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100184,"kb_used_data":55084,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94271656,"statfs":{"total":96636764160,"available":96534175744,"internally_reserved":0,"allocated":56406016,"data_stored":54802077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575667,"num_pgs":118,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252988,"kb_used_data":41120,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":94118852,"statfs":{"total":96636764160,"available":96377704448,"internally_reserved":0,"allocated":42106880,"data_stored":40381748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":216725959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575669,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91444,"kb_used_data":38044,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94280396,"statfs":{"total":96636764160,"available":96543125504,"internally_reserved":0,"allocated":38957056,"data_stored":37358501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575668,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131540,"kb_used_data":41960,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94240300,"statfs":{"total":96636764160,"available":96502067200,"internally_reserved":0,"allocated":42967040,"data_stored":41342836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575670,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244800,"kb_used_data":55196,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":94127040,"statfs":{"total":96636764160,"available":96386088960,"internally_reserved":0,"allocated":56520704,"data_stored":54866964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40763392,"data_stored":40607042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23621632,"data_stored":23479247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30126080,"data_stored":29981692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31137792,"data_stored":30907070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":42364928,"data_stored":42183762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":48488448,"data_stored":48318049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":19247104,"data_stored":19128319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36143104,"data_stored":35995991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":3901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":2727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10485760,"data_stored":10485760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":1048576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15728640,"data_stored":15728640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":1048576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10485760,"data_stored":10485760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T15:36:14.533 INFO:tasks.ceph:pgid 68.1a last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.533 INFO:tasks.ceph:pgid 67.1d last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.533 INFO:tasks.ceph:pgid 66.1c last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.534 INFO:tasks.ceph:pgid 65.1f last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.534 INFO:tasks.ceph:pgid 68.1b last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.534 INFO:tasks.ceph:pgid 67.1c last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.534 INFO:tasks.ceph:pgid 66.1d last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.534 INFO:tasks.ceph:pgid 65.1e last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.535 INFO:tasks.ceph:pgid 68.18 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.535 INFO:tasks.ceph:pgid 67.1f last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.535 INFO:tasks.ceph:pgid 66.1e last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.535 INFO:tasks.ceph:pgid 65.1d last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.535 INFO:tasks.ceph:pgid 68.19 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.535 INFO:tasks.ceph:pgid 67.1e last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.535 INFO:tasks.ceph:pgid 66.1f last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.536 INFO:tasks.ceph:pgid 65.1c last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.536 INFO:tasks.ceph:pgid 68.1e last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.536 INFO:tasks.ceph:pgid 67.19 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.536 INFO:tasks.ceph:pgid 66.18 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.536 INFO:tasks.ceph:pgid 65.1b last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.536 INFO:tasks.ceph:pgid 68.1f last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.537 INFO:tasks.ceph:pgid 67.18 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.537 INFO:tasks.ceph:pgid 66.19 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.537 INFO:tasks.ceph:pgid 65.1a last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.537 INFO:tasks.ceph:pgid 68.1c last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.537 INFO:tasks.ceph:pgid 67.1b last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.537 INFO:tasks.ceph:pgid 66.1a last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.537 INFO:tasks.ceph:pgid 65.19 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.538 INFO:tasks.ceph:pgid 68.1d last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.538 INFO:tasks.ceph:pgid 67.1a last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.538 INFO:tasks.ceph:pgid 66.1b last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.538 INFO:tasks.ceph:pgid 65.18 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.538 INFO:tasks.ceph:pgid 68.12 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.539 INFO:tasks.ceph:pgid 67.15 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.539 INFO:tasks.ceph:pgid 66.14 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.539 INFO:tasks.ceph:pgid 65.17 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.539 INFO:tasks.ceph:pgid 68.13 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.539 INFO:tasks.ceph:pgid 67.14 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.539 INFO:tasks.ceph:pgid 66.15 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.540 INFO:tasks.ceph:pgid 65.16 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.540 INFO:tasks.ceph:pgid 68.10 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.540 INFO:tasks.ceph:pgid 67.17 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.540 INFO:tasks.ceph:pgid 66.16 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.540 INFO:tasks.ceph:pgid 65.15 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.541 INFO:tasks.ceph:pgid 68.11 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.541 INFO:tasks.ceph:pgid 67.16 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.541 INFO:tasks.ceph:pgid 66.17 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.541 INFO:tasks.ceph:pgid 65.14 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.541 INFO:tasks.ceph:pgid 68.16 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.542 INFO:tasks.ceph:pgid 67.11 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.542 INFO:tasks.ceph:pgid 66.10 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.542 INFO:tasks.ceph:pgid 65.13 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.542 INFO:tasks.ceph:pgid 68.17 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.542 INFO:tasks.ceph:pgid 67.10 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.543 INFO:tasks.ceph:pgid 66.11 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.543 INFO:tasks.ceph:pgid 65.12 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.543 INFO:tasks.ceph:pgid 68.14 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.543 INFO:tasks.ceph:pgid 67.13 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.543 INFO:tasks.ceph:pgid 66.12 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.544 INFO:tasks.ceph:pgid 65.11 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.544 INFO:tasks.ceph:pgid 68.15 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.544 INFO:tasks.ceph:pgid 67.12 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.544 INFO:tasks.ceph:pgid 66.13 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.544 INFO:tasks.ceph:pgid 65.10 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.544 INFO:tasks.ceph:pgid 68.a last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.545 INFO:tasks.ceph:pgid 67.d last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.545 INFO:tasks.ceph:pgid 66.c last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.545 INFO:tasks.ceph:pgid 65.f last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.545 INFO:tasks.ceph:pgid 68.b last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.546 INFO:tasks.ceph:pgid 67.c last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.546 INFO:tasks.ceph:pgid 66.d last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.546 INFO:tasks.ceph:pgid 65.e last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.546 INFO:tasks.ceph:pgid 68.8 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.547 INFO:tasks.ceph:pgid 67.f last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.547 INFO:tasks.ceph:pgid 66.e last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.547 INFO:tasks.ceph:pgid 65.d last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.547 INFO:tasks.ceph:pgid 68.9 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.547 INFO:tasks.ceph:pgid 67.e last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.548 INFO:tasks.ceph:pgid 66.f last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.548 INFO:tasks.ceph:pgid 65.c last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.548 INFO:tasks.ceph:pgid 68.e last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.548 INFO:tasks.ceph:pgid 67.9 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.548 INFO:tasks.ceph:pgid 66.8 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.548 INFO:tasks.ceph:pgid 65.b last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.549 INFO:tasks.ceph:pgid 68.f last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.549 INFO:tasks.ceph:pgid 67.8 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.549 INFO:tasks.ceph:pgid 66.9 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.549 INFO:tasks.ceph:pgid 65.a last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.549 INFO:tasks.ceph:pgid 68.c last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.550 INFO:tasks.ceph:pgid 67.b last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.550 INFO:tasks.ceph:pgid 66.a last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.550 INFO:tasks.ceph:pgid 65.9 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.550 INFO:tasks.ceph:pgid 68.d last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.550 INFO:tasks.ceph:pgid 67.a last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.551 INFO:tasks.ceph:pgid 66.b last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.551 INFO:tasks.ceph:pgid 65.8 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.551 INFO:tasks.ceph:pgid 68.2 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.551 INFO:tasks.ceph:pgid 67.5 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.551 INFO:tasks.ceph:pgid 66.4 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.552 INFO:tasks.ceph:pgid 65.7 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.552 INFO:tasks.ceph:pgid 68.3 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.552 INFO:tasks.ceph:pgid 67.4 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.552 INFO:tasks.ceph:pgid 66.5 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.552 INFO:tasks.ceph:pgid 65.6 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.553 INFO:tasks.ceph:pgid 68.0 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.553 INFO:tasks.ceph:pgid 67.7 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.553 INFO:tasks.ceph:pgid 66.6 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.553 INFO:tasks.ceph:pgid 65.5 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.553 INFO:tasks.ceph:pgid 68.1 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.553 INFO:tasks.ceph:pgid 67.6 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.554 INFO:tasks.ceph:pgid 66.7 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.554 INFO:tasks.ceph:pgid 65.4 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.554 INFO:tasks.ceph:pgid 68.6 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.554 INFO:tasks.ceph:pgid 67.1 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.554 INFO:tasks.ceph:pgid 66.0 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.555 INFO:tasks.ceph:pgid 65.3 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.555 INFO:tasks.ceph:pgid 68.7 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.555 INFO:tasks.ceph:pgid 67.0 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.555 INFO:tasks.ceph:pgid 66.1 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.555 INFO:tasks.ceph:pgid 65.2 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.556 INFO:tasks.ceph:pgid 68.4 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.556 INFO:tasks.ceph:pgid 67.3 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.556 INFO:tasks.ceph:pgid 66.2 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.556 INFO:tasks.ceph:pgid 65.1 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.556 INFO:tasks.ceph:pgid 68.5 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.557 INFO:tasks.ceph:pgid 67.2 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.557 INFO:tasks.ceph:pgid 66.3 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.557 INFO:tasks.ceph:pgid 65.0 last_scrub_stamp 2024-03-09T14:39:12.220263+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=39, tm_sec=12, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.557 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.557 INFO:tasks.ceph:pgid 4.38 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.557 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.558 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.558 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.558 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.558 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.558 INFO:tasks.ceph:pgid 4.3b last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.558 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.559 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.559 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.559 INFO:tasks.ceph:pgid 4.3d last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.559 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.559 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.559 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.560 INFO:tasks.ceph:pgid 4.3f last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.560 INFO:tasks.ceph:pgid 4.30 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.560 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.560 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.560 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.561 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.561 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.562 INFO:tasks.ceph:pgid 4.33 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.562 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.562 INFO:tasks.ceph:pgid 4.34 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.562 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.562 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.562 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.563 INFO:tasks.ceph:pgid 8.18 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.563 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.563 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.563 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.563 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.563 INFO:tasks.ceph:pgid 7.17 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.564 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.564 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.564 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.564 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.564 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.564 INFO:tasks.ceph:pgid 7.16 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.564 INFO:tasks.ceph:pgid 8.1a last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.565 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.565 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.565 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.565 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.565 INFO:tasks.ceph:pgid 7.15 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.565 INFO:tasks.ceph:pgid 8.1b last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.565 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.566 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.566 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.566 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.566 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.566 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.566 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.567 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.567 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.567 INFO:tasks.ceph:pgid 7.b last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.567 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.567 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.568 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.568 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.568 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.568 INFO:tasks.ceph:pgid 7.a last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.569 INFO:tasks.ceph:pgid 8.5 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.569 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.569 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.569 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.569 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.569 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.570 INFO:tasks.ceph:pgid 8.6 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.570 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.570 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.570 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.570 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.571 INFO:tasks.ceph:pgid 7.8 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.571 INFO:tasks.ceph:pgid 8.7 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.571 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.571 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.571 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.572 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.572 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.572 INFO:tasks.ceph:pgid 8.0 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.572 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.572 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.573 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.573 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.573 INFO:tasks.ceph:pgid 7.e last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.573 INFO:tasks.ceph:pgid 8.1 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.574 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.574 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.574 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.574 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.574 INFO:tasks.ceph:pgid 7.d last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.575 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.575 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.575 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.575 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.575 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.576 INFO:tasks.ceph:pgid 7.c last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.576 INFO:tasks.ceph:pgid 8.3 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.576 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.576 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-09T14:28:11.179640+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=11, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.577 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.577 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.577 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.577 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.577 INFO:tasks.ceph:pgid 7.6 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.578 INFO:tasks.ceph:pgid 8.9 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.578 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.578 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.578 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.579 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.579 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.579 INFO:tasks.ceph:pgid 7.5 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.579 INFO:tasks.ceph:pgid 8.a last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.579 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.580 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.580 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.580 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.580 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.580 INFO:tasks.ceph:pgid 7.4 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.580 INFO:tasks.ceph:pgid 8.b last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.581 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.581 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.581 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.581 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.582 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.582 INFO:tasks.ceph:pgid 7.7 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.582 INFO:tasks.ceph:pgid 8.8 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.582 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.582 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.583 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.583 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.583 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.583 INFO:tasks.ceph:pgid 7.1 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.583 INFO:tasks.ceph:pgid 8.e last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.583 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.584 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.584 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.584 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.584 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.584 INFO:tasks.ceph:pgid 7.0 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.585 INFO:tasks.ceph:pgid 8.f last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.585 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.585 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.585 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.585 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.585 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.586 INFO:tasks.ceph:pgid 7.3 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.586 INFO:tasks.ceph:pgid 8.c last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.586 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.586 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.586 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.586 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.587 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.587 INFO:tasks.ceph:pgid 7.2 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.587 INFO:tasks.ceph:pgid 8.d last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.587 INFO:tasks.ceph:pgid 8.1f last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.587 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.587 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.588 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.588 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.588 INFO:tasks.ceph:pgid 7.10 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.588 INFO:tasks.ceph:pgid 8.1e last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.588 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.588 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.588 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.589 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.589 INFO:tasks.ceph:pgid 7.11 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.589 INFO:tasks.ceph:pgid 8.1d last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.589 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.589 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.589 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.590 INFO:tasks.ceph:pgid 6.13 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.590 INFO:tasks.ceph:pgid 7.12 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.590 INFO:tasks.ceph:pgid 8.1c last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.590 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.590 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.590 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.591 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.591 INFO:tasks.ceph:pgid 7.13 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.591 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.591 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.591 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.591 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.591 INFO:tasks.ceph:pgid 7.1c last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.592 INFO:tasks.ceph:pgid 8.13 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.592 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.592 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.592 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.592 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.592 INFO:tasks.ceph:pgid 7.1d last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.593 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.593 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.593 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.593 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.593 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.593 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.594 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.594 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.594 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.594 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.594 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.594 INFO:tasks.ceph:pgid 7.1f last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.595 INFO:tasks.ceph:pgid 8.10 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.595 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.595 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.595 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.595 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.596 INFO:tasks.ceph:pgid 7.18 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.596 INFO:tasks.ceph:pgid 8.17 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.596 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.596 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.596 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.597 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.597 INFO:tasks.ceph:pgid 7.19 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.597 INFO:tasks.ceph:pgid 8.16 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.597 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.597 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.597 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.598 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.598 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.598 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.598 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.598 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.598 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.599 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.599 INFO:tasks.ceph:pgid 7.1b last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.599 INFO:tasks.ceph:pgid 8.14 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.599 INFO:tasks.ceph:pgid 4.27 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.599 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.599 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.600 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.600 INFO:tasks.ceph:pgid 4.25 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.600 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.600 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.600 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.600 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.601 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.601 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.601 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.601 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.601 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.601 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.601 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.602 INFO:tasks.ceph:pgid 4.2f last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.602 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.602 INFO:tasks.ceph:pgid 4.2e last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.602 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.602 INFO:tasks.ceph:pgid 4.2d last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.603 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.603 INFO:tasks.ceph:pgid 4.2c last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.603 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.603 INFO:tasks.ceph:pgid 4.2b last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.603 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.604 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.604 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.604 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.604 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.604 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.605 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.605 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.605 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.605 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.605 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:14.606 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-09T15:36:34.607 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T15:36:34.993 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:36:35.009 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-09T15:36:35.029 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":2304,"stamp":"2024-03-09T15:36:33.267627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":169838241,"num_objects":629,"num_object_clones":0,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":0,"num_read":79040,"num_read_kb":1571790,"num_write":68959,"num_write_kb":1988151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":266753,"num_omap_keys":667,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12251,"ondisk_log_size":12251,"up":786,"acting":786,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":786,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1476024,"kb_used_data":360440,"kb_used_omap":12,"kb_used_meta":1115507,"kb_avail":753498696,"statfs":{"total":773094113280,"available":771582664704,"internally_reserved":0,"allocated":369090560,"data_stored":355965182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1142279728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":263786,"num_omap_keys":657,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009630"},"pg_stats":[{"pgid":"68.1a","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.211347+0000","last_change":"2024-03-09T15:09:50.085348+0000","last_active":"2024-03-09T15:34:51.211347+0000","last_peered":"2024-03-09T15:34:51.211347+0000","last_clean":"2024-03-09T15:34:51.211347+0000","last_became_active":"2024-03-09T15:09:50.084784+0000","last_became_peered":"2024-03-09T15:09:50.084784+0000","last_unstale":"2024-03-09T15:34:51.211347+0000","last_undegraded":"2024-03-09T15:34:51.211347+0000","last_fullsized":"2024-03-09T15:34:51.211347+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.1d","version":"258'7","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.125221+0000","last_change":"2024-03-09T14:46:26.826294+0000","last_active":"2024-03-09T15:36:13.125221+0000","last_peered":"2024-03-09T15:36:13.125221+0000","last_clean":"2024-03-09T15:36:13.125221+0000","last_became_active":"2024-03-09T14:46:26.825404+0000","last_became_peered":"2024-03-09T14:46:26.825404+0000","last_unstale":"2024-03-09T15:36:13.125221+0000","last_undegraded":"2024-03-09T15:36:13.125221+0000","last_fullsized":"2024-03-09T15:36:13.125221+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20498,"num_write":6,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1c","version":"261'2","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.429218+0000","last_change":"2024-03-09T15:36:17.429218+0000","last_active":"2024-03-09T15:36:17.429218+0000","last_peered":"2024-03-09T15:36:17.429218+0000","last_clean":"2024-03-09T15:36:17.429218+0000","last_became_active":"2024-03-09T14:46:18.768383+0000","last_became_peered":"2024-03-09T14:46:18.768383+0000","last_unstale":"2024-03-09T15:36:17.429218+0000","last_undegraded":"2024-03-09T15:36:17.429218+0000","last_fullsized":"2024-03-09T15:36:17.429218+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:17.428960+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:17.428960+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.428960+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:03:37.237056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.1f","version":"258'45","reported_seq":176,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.184209+0000","last_change":"2024-03-09T15:36:12.184266+0000","last_active":"2024-03-09T15:36:12.184209+0000","last_peered":"2024-03-09T15:36:12.184209+0000","last_clean":"2024-03-09T15:36:12.184209+0000","last_became_active":"2024-03-09T14:39:13.281020+0000","last_became_peered":"2024-03-09T14:39:13.281020+0000","last_unstale":"2024-03-09T15:36:12.184209+0000","last_undegraded":"2024-03-09T15:36:12.184209+0000","last_fullsized":"2024-03-09T15:36:12.184209+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'45","last_scrub_stamp":"2024-03-09T15:36:12.183988+0000","last_deep_scrub":"258'45","last_deep_scrub_stamp":"2024-03-09T15:36:12.183988+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.183988+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:21:51.000354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":134,"num_write":70,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1b","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:50.992140+0000","last_change":"2024-03-09T15:09:50.083262+0000","last_active":"2024-03-09T15:34:50.992140+0000","last_peered":"2024-03-09T15:34:50.992140+0000","last_clean":"2024-03-09T15:34:50.992140+0000","last_became_active":"2024-03-09T15:09:50.082690+0000","last_became_peered":"2024-03-09T15:09:50.082690+0000","last_unstale":"2024-03-09T15:34:50.992140+0000","last_undegraded":"2024-03-09T15:34:50.992140+0000","last_fullsized":"2024-03-09T15:34:50.992140+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1c","version":"258'5","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.650521+0000","last_change":"2024-03-09T14:46:26.807670+0000","last_active":"2024-03-09T15:36:12.650521+0000","last_peered":"2024-03-09T15:36:12.650521+0000","last_clean":"2024-03-09T15:36:12.650521+0000","last_became_active":"2024-03-09T14:46:26.806865+0000","last_became_peered":"2024-03-09T14:46:26.806865+0000","last_unstale":"2024-03-09T15:36:12.650521+0000","last_undegraded":"2024-03-09T15:36:12.650521+0000","last_fullsized":"2024-03-09T15:36:12.650521+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.1d","version":"261'2","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.601517+0000","last_change":"2024-03-09T15:36:17.601517+0000","last_active":"2024-03-09T15:36:17.601517+0000","last_peered":"2024-03-09T15:36:17.601517+0000","last_clean":"2024-03-09T15:36:17.601517+0000","last_became_active":"2024-03-09T14:46:18.779760+0000","last_became_peered":"2024-03-09T14:46:18.779760+0000","last_unstale":"2024-03-09T15:36:17.601517+0000","last_undegraded":"2024-03-09T15:36:17.601517+0000","last_fullsized":"2024-03-09T15:36:17.601517+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:17.601237+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:17.601237+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.601237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:48:08.721381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1e","version":"0'0","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.210922+0000","last_change":"2024-03-09T15:36:22.210983+0000","last_active":"2024-03-09T15:36:22.210922+0000","last_peered":"2024-03-09T15:36:22.210922+0000","last_clean":"2024-03-09T15:36:22.210922+0000","last_became_active":"2024-03-09T14:39:13.259731+0000","last_became_peered":"2024-03-09T14:39:13.259731+0000","last_unstale":"2024-03-09T15:36:22.210922+0000","last_undegraded":"2024-03-09T15:36:22.210922+0000","last_fullsized":"2024-03-09T15:36:22.210922+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.210700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.210700+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.210700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:46:50.001547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.18","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.614119+0000","last_change":"2024-03-09T15:09:50.081224+0000","last_active":"2024-03-09T15:36:13.614119+0000","last_peered":"2024-03-09T15:36:13.614119+0000","last_clean":"2024-03-09T15:36:13.614119+0000","last_became_active":"2024-03-09T15:09:50.080578+0000","last_became_peered":"2024-03-09T15:09:50.080578+0000","last_unstale":"2024-03-09T15:36:13.614119+0000","last_undegraded":"2024-03-09T15:36:13.614119+0000","last_fullsized":"2024-03-09T15:36:13.614119+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.1f","version":"261'5","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.740388+0000","last_change":"2024-03-09T14:46:26.896619+0000","last_active":"2024-03-09T15:36:10.740388+0000","last_peered":"2024-03-09T15:36:10.740388+0000","last_clean":"2024-03-09T15:36:10.740388+0000","last_became_active":"2024-03-09T14:46:26.895608+0000","last_became_peered":"2024-03-09T14:46:26.895608+0000","last_unstale":"2024-03-09T15:36:10.740388+0000","last_undegraded":"2024-03-09T15:36:10.740388+0000","last_fullsized":"2024-03-09T15:36:10.740388+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":12288,"num_write":5,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1e","version":"0'0","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:35.988936+0000","last_change":"2024-03-09T14:46:18.782402+0000","last_active":"2024-03-09T15:34:35.988936+0000","last_peered":"2024-03-09T15:34:35.988936+0000","last_clean":"2024-03-09T15:34:35.988936+0000","last_became_active":"2024-03-09T14:46:18.781856+0000","last_became_peered":"2024-03-09T14:46:18.781856+0000","last_unstale":"2024-03-09T15:34:35.988936+0000","last_undegraded":"2024-03-09T15:34:35.988936+0000","last_fullsized":"2024-03-09T15:34:35.988936+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.1d","version":"258'35","reported_seq":151,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.273850+0000","last_change":"2024-03-09T15:36:17.273912+0000","last_active":"2024-03-09T15:36:17.273850+0000","last_peered":"2024-03-09T15:36:17.273850+0000","last_clean":"2024-03-09T15:36:17.273850+0000","last_became_active":"2024-03-09T14:39:13.258351+0000","last_became_peered":"2024-03-09T14:39:13.258351+0000","last_unstale":"2024-03-09T15:36:17.273850+0000","last_undegraded":"2024-03-09T15:36:17.273850+0000","last_fullsized":"2024-03-09T15:36:17.273850+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'35","last_scrub_stamp":"2024-03-09T15:36:17.273654+0000","last_deep_scrub":"258'35","last_deep_scrub_stamp":"2024-03-09T15:36:17.273654+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.273654+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:22:59.569323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":152,"num_read_kb":109,"num_write":57,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.19","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.187003+0000","last_change":"2024-03-09T15:09:50.083658+0000","last_active":"2024-03-09T15:36:12.187003+0000","last_peered":"2024-03-09T15:36:12.187003+0000","last_clean":"2024-03-09T15:36:12.187003+0000","last_became_active":"2024-03-09T15:09:50.081680+0000","last_became_peered":"2024-03-09T15:09:50.081680+0000","last_unstale":"2024-03-09T15:36:12.187003+0000","last_undegraded":"2024-03-09T15:36:12.187003+0000","last_fullsized":"2024-03-09T15:36:12.187003+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.1e","version":"261'6","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.485897+0000","last_change":"2024-03-09T14:47:07.798858+0000","last_active":"2024-03-09T15:36:06.485897+0000","last_peered":"2024-03-09T15:36:06.485897+0000","last_clean":"2024-03-09T15:36:06.485897+0000","last_became_active":"2024-03-09T14:47:07.798247+0000","last_became_peered":"2024-03-09T14:47:07.798247+0000","last_unstale":"2024-03-09T15:36:06.485897+0000","last_undegraded":"2024-03-09T15:36:06.485897+0000","last_fullsized":"2024-03-09T15:36:06.485897+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":16384,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1f","version":"258'5","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.180362+0000","last_change":"2024-03-09T15:36:25.180427+0000","last_active":"2024-03-09T15:36:25.180362+0000","last_peered":"2024-03-09T15:36:25.180362+0000","last_clean":"2024-03-09T15:36:25.180362+0000","last_became_active":"2024-03-09T14:46:18.771731+0000","last_became_peered":"2024-03-09T14:46:18.771731+0000","last_unstale":"2024-03-09T15:36:25.180362+0000","last_undegraded":"2024-03-09T15:36:25.180362+0000","last_fullsized":"2024-03-09T15:36:25.180362+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'5","last_scrub_stamp":"2024-03-09T15:36:25.180164+0000","last_deep_scrub":"258'5","last_deep_scrub_stamp":"2024-03-09T15:36:25.180164+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.180164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:55:32.192979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5136,"num_write":5,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1c","version":"261'13","reported_seq":98,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.507499+0000","last_change":"2024-03-09T15:36:15.507564+0000","last_active":"2024-03-09T15:36:15.507499+0000","last_peered":"2024-03-09T15:36:15.507499+0000","last_clean":"2024-03-09T15:36:15.507499+0000","last_became_active":"2024-03-09T14:39:13.260945+0000","last_became_peered":"2024-03-09T14:39:13.260945+0000","last_unstale":"2024-03-09T15:36:15.507499+0000","last_undegraded":"2024-03-09T15:36:15.507499+0000","last_fullsized":"2024-03-09T15:36:15.507499+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'13","last_scrub_stamp":"2024-03-09T15:36:15.507291+0000","last_deep_scrub":"261'13","last_deep_scrub_stamp":"2024-03-09T15:36:15.507291+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.507291+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:29:48.463973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":27,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1e","version":"0'0","reported_seq":26,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:55.834070+0000","last_change":"2024-03-09T15:09:50.088230+0000","last_active":"2024-03-09T15:35:55.834070+0000","last_peered":"2024-03-09T15:35:55.834070+0000","last_clean":"2024-03-09T15:35:55.834070+0000","last_became_active":"2024-03-09T15:09:50.086789+0000","last_became_peered":"2024-03-09T15:09:50.086789+0000","last_unstale":"2024-03-09T15:35:55.834070+0000","last_undegraded":"2024-03-09T15:35:55.834070+0000","last_fullsized":"2024-03-09T15:35:55.834070+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.19","version":"261'6","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.157492+0000","last_change":"2024-03-09T14:46:26.819845+0000","last_active":"2024-03-09T15:36:06.157492+0000","last_peered":"2024-03-09T15:36:06.157492+0000","last_clean":"2024-03-09T15:36:06.157492+0000","last_became_active":"2024-03-09T14:46:26.819085+0000","last_became_peered":"2024-03-09T14:46:26.819085+0000","last_unstale":"2024-03-09T15:36:06.157492+0000","last_undegraded":"2024-03-09T15:36:06.157492+0000","last_fullsized":"2024-03-09T15:36:06.157492+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":12288,"num_write":11,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.18","version":"261'14","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.270716+0000","last_change":"2024-03-09T15:36:27.270917+0000","last_active":"2024-03-09T15:36:27.270716+0000","last_peered":"2024-03-09T15:36:27.270716+0000","last_clean":"2024-03-09T15:36:27.270716+0000","last_became_active":"2024-03-09T14:46:18.776161+0000","last_became_peered":"2024-03-09T14:46:18.776161+0000","last_unstale":"2024-03-09T15:36:27.270716+0000","last_undegraded":"2024-03-09T15:36:27.270716+0000","last_fullsized":"2024-03-09T15:36:27.270716+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'14","last_scrub_stamp":"2024-03-09T15:36:27.270455+0000","last_deep_scrub":"261'14","last_deep_scrub_stamp":"2024-03-09T15:36:27.270455+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.270455+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:23:13.389149+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":36889,"num_write":17,"num_write_kb":16407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.1b","version":"258'10","reported_seq":88,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.553035+0000","last_change":"2024-03-09T15:36:14.553151+0000","last_active":"2024-03-09T15:36:14.553035+0000","last_peered":"2024-03-09T15:36:14.553035+0000","last_clean":"2024-03-09T15:36:14.553035+0000","last_became_active":"2024-03-09T14:39:13.268179+0000","last_became_peered":"2024-03-09T14:39:13.268179+0000","last_unstale":"2024-03-09T15:36:14.553035+0000","last_undegraded":"2024-03-09T15:36:14.553035+0000","last_fullsized":"2024-03-09T15:36:14.553035+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:14.552885+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:14.552885+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.552885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:39.435007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1f","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:50.978735+0000","last_change":"2024-03-09T15:09:50.064176+0000","last_active":"2024-03-09T15:34:50.978735+0000","last_peered":"2024-03-09T15:34:50.978735+0000","last_clean":"2024-03-09T15:34:50.978735+0000","last_became_active":"2024-03-09T15:09:50.063705+0000","last_became_peered":"2024-03-09T15:09:50.063705+0000","last_unstale":"2024-03-09T15:34:50.978735+0000","last_undegraded":"2024-03-09T15:34:50.978735+0000","last_fullsized":"2024-03-09T15:34:50.978735+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.18","version":"261'3","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.136139+0000","last_change":"2024-03-09T14:46:26.824830+0000","last_active":"2024-03-09T15:36:06.136139+0000","last_peered":"2024-03-09T15:36:06.136139+0000","last_clean":"2024-03-09T15:36:06.136139+0000","last_became_active":"2024-03-09T14:46:26.824522+0000","last_became_peered":"2024-03-09T14:46:26.824522+0000","last_unstale":"2024-03-09T15:36:06.136139+0000","last_undegraded":"2024-03-09T15:36:06.136139+0000","last_fullsized":"2024-03-09T15:36:06.136139+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.19","version":"261'9","reported_seq":75,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.166246+0000","last_change":"2024-03-09T15:36:17.166321+0000","last_active":"2024-03-09T15:36:17.166246+0000","last_peered":"2024-03-09T15:36:17.166246+0000","last_clean":"2024-03-09T15:36:17.166246+0000","last_became_active":"2024-03-09T14:46:18.769730+0000","last_became_peered":"2024-03-09T14:46:18.769730+0000","last_unstale":"2024-03-09T15:36:17.166246+0000","last_undegraded":"2024-03-09T15:36:17.166246+0000","last_fullsized":"2024-03-09T15:36:17.166246+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-03-09T15:36:17.165994+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-03-09T15:36:17.165994+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.165994+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:56:10.015397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":19456,"num_write":19,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1a","version":"261'44","reported_seq":173,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.008377+0000","last_change":"2024-03-09T15:36:14.008442+0000","last_active":"2024-03-09T15:36:14.008377+0000","last_peered":"2024-03-09T15:36:14.008377+0000","last_clean":"2024-03-09T15:36:14.008377+0000","last_became_active":"2024-03-09T14:39:13.273298+0000","last_became_peered":"2024-03-09T14:39:13.273298+0000","last_unstale":"2024-03-09T15:36:14.008377+0000","last_undegraded":"2024-03-09T15:36:14.008377+0000","last_fullsized":"2024-03-09T15:36:14.008377+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:14.008154+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:14.008154+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.008154+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:19:56.477799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":138,"num_write":72,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.1c","version":"261'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.222862+0000","last_change":"2024-03-09T15:09:50.075720+0000","last_active":"2024-03-09T15:34:51.222862+0000","last_peered":"2024-03-09T15:34:51.222862+0000","last_clean":"2024-03-09T15:34:51.222862+0000","last_became_active":"2024-03-09T15:09:50.074952+0000","last_became_peered":"2024-03-09T15:09:50.074952+0000","last_unstale":"2024-03-09T15:34:51.222862+0000","last_undegraded":"2024-03-09T15:34:51.222862+0000","last_fullsized":"2024-03-09T15:34:51.222862+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1b","version":"261'3","reported_seq":48,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.614077+0000","last_change":"2024-03-09T14:46:26.805382+0000","last_active":"2024-03-09T15:36:13.614077+0000","last_peered":"2024-03-09T15:36:13.614077+0000","last_clean":"2024-03-09T15:36:13.614077+0000","last_became_active":"2024-03-09T14:46:26.804622+0000","last_became_peered":"2024-03-09T14:46:26.804622+0000","last_unstale":"2024-03-09T15:36:13.614077+0000","last_undegraded":"2024-03-09T15:36:13.614077+0000","last_fullsized":"2024-03-09T15:36:13.614077+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.1a","version":"261'5","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.650388+0000","last_change":"2024-03-09T14:46:18.785531+0000","last_active":"2024-03-09T15:36:12.650388+0000","last_peered":"2024-03-09T15:36:12.650388+0000","last_clean":"2024-03-09T15:36:12.650388+0000","last_became_active":"2024-03-09T14:46:18.784821+0000","last_became_peered":"2024-03-09T14:46:18.784821+0000","last_unstale":"2024-03-09T15:36:12.650388+0000","last_undegraded":"2024-03-09T15:36:12.650388+0000","last_fullsized":"2024-03-09T15:36:12.650388+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4096,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.19","version":"258'28","reported_seq":129,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.263878+0000","last_change":"2024-03-09T15:36:18.263945+0000","last_active":"2024-03-09T15:36:18.263878+0000","last_peered":"2024-03-09T15:36:18.263878+0000","last_clean":"2024-03-09T15:36:18.263878+0000","last_became_active":"2024-03-09T14:39:13.257473+0000","last_became_peered":"2024-03-09T14:39:13.257473+0000","last_unstale":"2024-03-09T15:36:18.263878+0000","last_undegraded":"2024-03-09T15:36:18.263878+0000","last_fullsized":"2024-03-09T15:36:18.263878+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'28","last_scrub_stamp":"2024-03-09T15:36:18.263646+0000","last_deep_scrub":"258'28","last_deep_scrub_stamp":"2024-03-09T15:36:18.263646+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.263646+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:59:14.805307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":78,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1d","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.222939+0000","last_change":"2024-03-09T15:09:50.075294+0000","last_active":"2024-03-09T15:34:51.222939+0000","last_peered":"2024-03-09T15:34:51.222939+0000","last_clean":"2024-03-09T15:34:51.222939+0000","last_became_active":"2024-03-09T15:09:50.074875+0000","last_became_peered":"2024-03-09T15:09:50.074875+0000","last_unstale":"2024-03-09T15:34:51.222939+0000","last_undegraded":"2024-03-09T15:34:51.222939+0000","last_fullsized":"2024-03-09T15:34:51.222939+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1a","version":"261'11","reported_seq":72,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.456175+0000","last_change":"2024-03-09T15:36:21.456175+0000","last_active":"2024-03-09T15:36:21.456175+0000","last_peered":"2024-03-09T15:36:21.456175+0000","last_clean":"2024-03-09T15:36:21.456175+0000","last_became_active":"2024-03-09T14:46:26.812889+0000","last_became_peered":"2024-03-09T14:46:26.812889+0000","last_unstale":"2024-03-09T15:36:21.456175+0000","last_undegraded":"2024-03-09T15:36:21.456175+0000","last_fullsized":"2024-03-09T15:36:21.456175+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'11","last_scrub_stamp":"2024-03-09T15:36:21.455916+0000","last_deep_scrub":"261'11","last_deep_scrub_stamp":"2024-03-09T15:36:21.455916+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.455916+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:07:00.143287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":12290,"num_write":14,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.1b","version":"261'2","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.253249+0000","last_change":"2024-03-09T15:36:19.253249+0000","last_active":"2024-03-09T15:36:19.253249+0000","last_peered":"2024-03-09T15:36:19.253249+0000","last_clean":"2024-03-09T15:36:19.253249+0000","last_became_active":"2024-03-09T14:46:18.756394+0000","last_became_peered":"2024-03-09T14:46:18.756394+0000","last_unstale":"2024-03-09T15:36:19.253249+0000","last_undegraded":"2024-03-09T15:36:19.253249+0000","last_fullsized":"2024-03-09T15:36:19.253249+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:19.252989+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:19.252989+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.252989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:28:28.982640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":40,"num_write":2,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.18","version":"261'8","reported_seq":86,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.579241+0000","last_change":"2024-03-09T15:36:15.579306+0000","last_active":"2024-03-09T15:36:15.579241+0000","last_peered":"2024-03-09T15:36:15.579241+0000","last_clean":"2024-03-09T15:36:15.579241+0000","last_became_active":"2024-03-09T14:39:13.257996+0000","last_became_peered":"2024-03-09T14:39:13.257996+0000","last_unstale":"2024-03-09T15:36:15.579241+0000","last_undegraded":"2024-03-09T15:36:15.579241+0000","last_fullsized":"2024-03-09T15:36:15.579241+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:15.579034+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:15.579034+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.579034+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:42:49.650613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.12","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:30.987310+0000","last_change":"2024-03-09T15:09:50.070950+0000","last_active":"2024-03-09T15:35:30.987310+0000","last_peered":"2024-03-09T15:35:30.987310+0000","last_clean":"2024-03-09T15:35:30.987310+0000","last_became_active":"2024-03-09T15:09:50.070345+0000","last_became_peered":"2024-03-09T15:09:50.070345+0000","last_unstale":"2024-03-09T15:35:30.987310+0000","last_undegraded":"2024-03-09T15:35:30.987310+0000","last_fullsized":"2024-03-09T15:35:30.987310+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.15","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:50.820582+0000","last_change":"2024-03-09T14:46:26.891781+0000","last_active":"2024-03-09T15:34:50.820582+0000","last_peered":"2024-03-09T15:34:50.820582+0000","last_clean":"2024-03-09T15:34:50.820582+0000","last_became_active":"2024-03-09T14:46:26.890992+0000","last_became_peered":"2024-03-09T14:46:26.890992+0000","last_unstale":"2024-03-09T15:34:50.820582+0000","last_undegraded":"2024-03-09T15:34:50.820582+0000","last_fullsized":"2024-03-09T15:34:50.820582+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.14","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.701752+0000","last_change":"2024-03-09T15:36:22.701809+0000","last_active":"2024-03-09T15:36:22.701752+0000","last_peered":"2024-03-09T15:36:22.701752+0000","last_clean":"2024-03-09T15:36:22.701752+0000","last_became_active":"2024-03-09T14:46:18.766777+0000","last_became_peered":"2024-03-09T14:46:18.766777+0000","last_unstale":"2024-03-09T15:36:22.701752+0000","last_undegraded":"2024-03-09T15:36:22.701752+0000","last_fullsized":"2024-03-09T15:36:22.701752+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.701528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.701528+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.701528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:03:22.022639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.17","version":"258'19","reported_seq":115,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.216612+0000","last_change":"2024-03-09T15:36:13.216647+0000","last_active":"2024-03-09T15:36:13.216612+0000","last_peered":"2024-03-09T15:36:13.216612+0000","last_clean":"2024-03-09T15:36:13.216612+0000","last_became_active":"2024-03-09T14:39:13.279919+0000","last_became_peered":"2024-03-09T14:39:13.279919+0000","last_unstale":"2024-03-09T15:36:13.216612+0000","last_undegraded":"2024-03-09T15:36:13.216612+0000","last_fullsized":"2024-03-09T15:36:13.216612+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'19","last_scrub_stamp":"2024-03-09T15:36:13.216485+0000","last_deep_scrub":"258'19","last_deep_scrub_stamp":"2024-03-09T15:36:13.216485+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.216485+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:22:26.193181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":64,"num_write":32,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.13","version":"261'1","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:31.000329+0000","last_change":"2024-03-09T15:09:50.079986+0000","last_active":"2024-03-09T15:35:31.000329+0000","last_peered":"2024-03-09T15:35:31.000329+0000","last_clean":"2024-03-09T15:35:31.000329+0000","last_became_active":"2024-03-09T15:09:50.079604+0000","last_became_peered":"2024-03-09T15:09:50.079604+0000","last_unstale":"2024-03-09T15:35:31.000329+0000","last_undegraded":"2024-03-09T15:35:31.000329+0000","last_fullsized":"2024-03-09T15:35:31.000329+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.14","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:40.830834+0000","last_change":"2024-03-09T14:46:26.894898+0000","last_active":"2024-03-09T15:35:40.830834+0000","last_peered":"2024-03-09T15:35:40.830834+0000","last_clean":"2024-03-09T15:35:40.830834+0000","last_became_active":"2024-03-09T14:46:26.894088+0000","last_became_peered":"2024-03-09T14:46:26.894088+0000","last_unstale":"2024-03-09T15:35:40.830834+0000","last_undegraded":"2024-03-09T15:35:40.830834+0000","last_fullsized":"2024-03-09T15:35:40.830834+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.15","version":"0'0","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.232191+0000","last_change":"2024-03-09T15:36:16.232245+0000","last_active":"2024-03-09T15:36:16.232191+0000","last_peered":"2024-03-09T15:36:16.232191+0000","last_clean":"2024-03-09T15:36:16.232191+0000","last_became_active":"2024-03-09T14:46:18.752413+0000","last_became_peered":"2024-03-09T14:46:18.752413+0000","last_unstale":"2024-03-09T15:36:16.232191+0000","last_undegraded":"2024-03-09T15:36:16.232191+0000","last_fullsized":"2024-03-09T15:36:16.232191+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:16.232005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:16.232005+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.232005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:59:24.622129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.16","version":"261'52","reported_seq":193,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.459135+0000","last_change":"2024-03-09T15:36:16.459199+0000","last_active":"2024-03-09T15:36:16.459135+0000","last_peered":"2024-03-09T15:36:16.459135+0000","last_clean":"2024-03-09T15:36:16.459135+0000","last_became_active":"2024-03-09T14:39:13.262413+0000","last_became_peered":"2024-03-09T14:39:13.262413+0000","last_unstale":"2024-03-09T15:36:16.459135+0000","last_undegraded":"2024-03-09T15:36:16.459135+0000","last_fullsized":"2024-03-09T15:36:16.459135+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'52","last_scrub_stamp":"2024-03-09T15:36:16.458932+0000","last_deep_scrub":"261'52","last_deep_scrub_stamp":"2024-03-09T15:36:16.458932+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.458932+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:28:33.068618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":163,"num_write":78,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.10","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:31.231826+0000","last_change":"2024-03-09T15:09:50.075112+0000","last_active":"2024-03-09T15:35:31.231826+0000","last_peered":"2024-03-09T15:35:31.231826+0000","last_clean":"2024-03-09T15:35:31.231826+0000","last_became_active":"2024-03-09T15:09:50.074274+0000","last_became_peered":"2024-03-09T15:09:50.074274+0000","last_unstale":"2024-03-09T15:35:31.231826+0000","last_undegraded":"2024-03-09T15:35:31.231826+0000","last_fullsized":"2024-03-09T15:35:31.231826+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.17","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:50.820667+0000","last_change":"2024-03-09T14:46:26.894297+0000","last_active":"2024-03-09T15:34:50.820667+0000","last_peered":"2024-03-09T15:34:50.820667+0000","last_clean":"2024-03-09T15:34:50.820667+0000","last_became_active":"2024-03-09T14:46:26.893514+0000","last_became_peered":"2024-03-09T14:46:26.893514+0000","last_unstale":"2024-03-09T15:34:50.820667+0000","last_undegraded":"2024-03-09T15:34:50.820667+0000","last_fullsized":"2024-03-09T15:34:50.820667+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.16","version":"0'0","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:40.990103+0000","last_change":"2024-03-09T14:46:18.773133+0000","last_active":"2024-03-09T15:34:40.990103+0000","last_peered":"2024-03-09T15:34:40.990103+0000","last_clean":"2024-03-09T15:34:40.990103+0000","last_became_active":"2024-03-09T14:46:18.769274+0000","last_became_peered":"2024-03-09T14:46:18.769274+0000","last_unstale":"2024-03-09T15:34:40.990103+0000","last_undegraded":"2024-03-09T15:34:40.990103+0000","last_fullsized":"2024-03-09T15:34:40.990103+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.15","version":"0'0","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.220144+0000","last_change":"2024-03-09T15:36:24.220202+0000","last_active":"2024-03-09T15:36:24.220144+0000","last_peered":"2024-03-09T15:36:24.220144+0000","last_clean":"2024-03-09T15:36:24.220144+0000","last_became_active":"2024-03-09T14:39:13.262066+0000","last_became_peered":"2024-03-09T14:39:13.262066+0000","last_unstale":"2024-03-09T15:36:24.220144+0000","last_undegraded":"2024-03-09T15:36:24.220144+0000","last_fullsized":"2024-03-09T15:36:24.220144+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:24.219947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:24.219947+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.219947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:56:44.705423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.11","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:31.220191+0000","last_change":"2024-03-09T15:09:50.086149+0000","last_active":"2024-03-09T15:35:31.220191+0000","last_peered":"2024-03-09T15:35:31.220191+0000","last_clean":"2024-03-09T15:35:31.220191+0000","last_became_active":"2024-03-09T15:09:50.085346+0000","last_became_peered":"2024-03-09T15:09:50.085346+0000","last_unstale":"2024-03-09T15:35:31.220191+0000","last_undegraded":"2024-03-09T15:35:31.220191+0000","last_fullsized":"2024-03-09T15:35:31.220191+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.16","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:46.005141+0000","last_change":"2024-03-09T14:46:26.818659+0000","last_active":"2024-03-09T15:34:46.005141+0000","last_peered":"2024-03-09T15:34:46.005141+0000","last_clean":"2024-03-09T15:34:46.005141+0000","last_became_active":"2024-03-09T14:46:26.817864+0000","last_became_peered":"2024-03-09T14:46:26.817864+0000","last_unstale":"2024-03-09T15:34:46.005141+0000","last_undegraded":"2024-03-09T15:34:46.005141+0000","last_fullsized":"2024-03-09T15:34:46.005141+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.17","version":"0'0","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:41.027946+0000","last_change":"2024-03-09T14:46:18.780789+0000","last_active":"2024-03-09T15:34:41.027946+0000","last_peered":"2024-03-09T15:34:41.027946+0000","last_clean":"2024-03-09T15:34:41.027946+0000","last_became_active":"2024-03-09T14:46:18.776430+0000","last_became_peered":"2024-03-09T14:46:18.776430+0000","last_unstale":"2024-03-09T15:34:41.027946+0000","last_undegraded":"2024-03-09T15:34:41.027946+0000","last_fullsized":"2024-03-09T15:34:41.027946+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.14","version":"261'68","reported_seq":280,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.178145+0000","last_change":"2024-03-09T15:36:24.178221+0000","last_active":"2024-03-09T15:36:24.178145+0000","last_peered":"2024-03-09T15:36:24.178145+0000","last_clean":"2024-03-09T15:36:24.178145+0000","last_became_active":"2024-03-09T14:39:13.253007+0000","last_became_peered":"2024-03-09T14:39:13.253007+0000","last_unstale":"2024-03-09T15:36:24.178145+0000","last_undegraded":"2024-03-09T15:36:24.178145+0000","last_fullsized":"2024-03-09T15:36:24.178145+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'68","last_scrub_stamp":"2024-03-09T15:36:24.177894+0000","last_deep_scrub":"261'68","last_deep_scrub_stamp":"2024-03-09T15:36:24.177894+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.177894+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:09:13.568231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":411,"num_read_kb":279,"num_write":87,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4473,"num_omap_keys":9,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.16","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.187173+0000","last_change":"2024-03-09T15:09:50.087752+0000","last_active":"2024-03-09T15:36:12.187173+0000","last_peered":"2024-03-09T15:36:12.187173+0000","last_clean":"2024-03-09T15:36:12.187173+0000","last_became_active":"2024-03-09T15:09:50.086173+0000","last_became_peered":"2024-03-09T15:09:50.086173+0000","last_unstale":"2024-03-09T15:36:12.187173+0000","last_undegraded":"2024-03-09T15:36:12.187173+0000","last_fullsized":"2024-03-09T15:36:12.187173+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.11","version":"261'7","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.858797+0000","last_change":"2024-03-09T14:46:26.817795+0000","last_active":"2024-03-09T15:36:05.858797+0000","last_peered":"2024-03-09T15:36:05.858797+0000","last_clean":"2024-03-09T15:36:05.858797+0000","last_became_active":"2024-03-09T14:46:26.817014+0000","last_became_peered":"2024-03-09T14:46:26.817014+0000","last_unstale":"2024-03-09T15:36:05.858797+0000","last_undegraded":"2024-03-09T15:36:05.858797+0000","last_fullsized":"2024-03-09T15:36:05.858797+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":23554,"num_write":11,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.10","version":"261'3","reported_seq":63,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.121627+0000","last_change":"2024-03-09T15:36:21.121690+0000","last_active":"2024-03-09T15:36:21.121627+0000","last_peered":"2024-03-09T15:36:21.121627+0000","last_clean":"2024-03-09T15:36:21.121627+0000","last_became_active":"2024-03-09T14:46:18.770936+0000","last_became_peered":"2024-03-09T14:46:18.770936+0000","last_unstale":"2024-03-09T15:36:21.121627+0000","last_undegraded":"2024-03-09T15:36:21.121627+0000","last_fullsized":"2024-03-09T15:36:21.121627+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:21.121408+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:21.121408+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.121408+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:44:45.507890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.13","version":"258'16","reported_seq":104,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.204733+0000","last_change":"2024-03-09T15:36:22.204801+0000","last_active":"2024-03-09T15:36:22.204733+0000","last_peered":"2024-03-09T15:36:22.204733+0000","last_clean":"2024-03-09T15:36:22.204733+0000","last_became_active":"2024-03-09T14:39:13.266228+0000","last_became_peered":"2024-03-09T14:39:13.266228+0000","last_unstale":"2024-03-09T15:36:22.204733+0000","last_undegraded":"2024-03-09T15:36:22.204733+0000","last_fullsized":"2024-03-09T15:36:22.204733+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'16","last_scrub_stamp":"2024-03-09T15:36:22.204502+0000","last_deep_scrub":"258'16","last_deep_scrub_stamp":"2024-03-09T15:36:22.204502+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.204502+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:33:31.523340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":45,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.17","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.740548+0000","last_change":"2024-03-09T15:09:50.075655+0000","last_active":"2024-03-09T15:36:10.740548+0000","last_peered":"2024-03-09T15:36:10.740548+0000","last_clean":"2024-03-09T15:36:10.740548+0000","last_became_active":"2024-03-09T15:09:50.075187+0000","last_became_peered":"2024-03-09T15:09:50.075187+0000","last_unstale":"2024-03-09T15:36:10.740548+0000","last_undegraded":"2024-03-09T15:36:10.740548+0000","last_fullsized":"2024-03-09T15:36:10.740548+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.10","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:25.986200+0000","last_change":"2024-03-09T14:46:26.803323+0000","last_active":"2024-03-09T15:35:25.986200+0000","last_peered":"2024-03-09T15:35:25.986200+0000","last_clean":"2024-03-09T15:35:25.986200+0000","last_became_active":"2024-03-09T14:46:26.802476+0000","last_became_peered":"2024-03-09T14:46:26.802476+0000","last_unstale":"2024-03-09T15:35:25.986200+0000","last_undegraded":"2024-03-09T15:35:25.986200+0000","last_fullsized":"2024-03-09T15:35:25.986200+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.11","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.548131+0000","last_change":"2024-03-09T15:36:13.548167+0000","last_active":"2024-03-09T15:36:13.548131+0000","last_peered":"2024-03-09T15:36:13.548131+0000","last_clean":"2024-03-09T15:36:13.548131+0000","last_became_active":"2024-03-09T14:46:18.760959+0000","last_became_peered":"2024-03-09T14:46:18.760959+0000","last_unstale":"2024-03-09T15:36:13.548131+0000","last_undegraded":"2024-03-09T15:36:13.548131+0000","last_fullsized":"2024-03-09T15:36:13.548131+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:13.548016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:13.548016+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.548016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:46:22.640051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.12","version":"258'7","reported_seq":86,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.252904+0000","last_change":"2024-03-09T15:36:16.252969+0000","last_active":"2024-03-09T15:36:16.252904+0000","last_peered":"2024-03-09T15:36:16.252904+0000","last_clean":"2024-03-09T15:36:16.252904+0000","last_became_active":"2024-03-09T14:39:13.273740+0000","last_became_peered":"2024-03-09T14:39:13.273740+0000","last_unstale":"2024-03-09T15:36:16.252904+0000","last_undegraded":"2024-03-09T15:36:16.252904+0000","last_fullsized":"2024-03-09T15:36:16.252904+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'7","last_scrub_stamp":"2024-03-09T15:36:16.252708+0000","last_deep_scrub":"258'7","last_deep_scrub_stamp":"2024-03-09T15:36:16.252708+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.252708+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:42:28.881579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":26,"num_write":13,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.14","version":"260'1","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:36.040914+0000","last_change":"2024-03-09T15:09:50.065240+0000","last_active":"2024-03-09T15:35:36.040914+0000","last_peered":"2024-03-09T15:35:36.040914+0000","last_clean":"2024-03-09T15:35:36.040914+0000","last_became_active":"2024-03-09T15:09:50.064428+0000","last_became_peered":"2024-03-09T15:09:50.064428+0000","last_unstale":"2024-03-09T15:35:36.040914+0000","last_undegraded":"2024-03-09T15:35:36.040914+0000","last_fullsized":"2024-03-09T15:35:36.040914+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.13","version":"261'3","reported_seq":49,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.858476+0000","last_change":"2024-03-09T14:46:26.813826+0000","last_active":"2024-03-09T15:36:05.858476+0000","last_peered":"2024-03-09T15:36:05.858476+0000","last_clean":"2024-03-09T15:36:05.858476+0000","last_became_active":"2024-03-09T14:46:26.813088+0000","last_became_peered":"2024-03-09T14:46:26.813088+0000","last_unstale":"2024-03-09T15:36:05.858476+0000","last_undegraded":"2024-03-09T15:36:05.858476+0000","last_fullsized":"2024-03-09T15:36:05.858476+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.12","version":"261'6","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.182877+0000","last_change":"2024-03-09T14:46:18.783155+0000","last_active":"2024-03-09T15:36:06.182877+0000","last_peered":"2024-03-09T15:36:06.182877+0000","last_clean":"2024-03-09T15:36:06.182877+0000","last_became_active":"2024-03-09T14:46:18.782158+0000","last_became_peered":"2024-03-09T14:46:18.782158+0000","last_unstale":"2024-03-09T15:36:06.182877+0000","last_undegraded":"2024-03-09T15:36:06.182877+0000","last_fullsized":"2024-03-09T15:36:06.182877+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":11264,"num_write":6,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.11","version":"258'36","reported_seq":157,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.155312+0000","last_change":"2024-03-09T15:36:14.155374+0000","last_active":"2024-03-09T15:36:14.155312+0000","last_peered":"2024-03-09T15:36:14.155312+0000","last_clean":"2024-03-09T15:36:14.155312+0000","last_became_active":"2024-03-09T14:39:13.260219+0000","last_became_peered":"2024-03-09T14:39:13.260219+0000","last_unstale":"2024-03-09T15:36:14.155312+0000","last_undegraded":"2024-03-09T15:36:14.155312+0000","last_fullsized":"2024-03-09T15:36:14.155312+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'36","last_scrub_stamp":"2024-03-09T15:36:14.155108+0000","last_deep_scrub":"258'36","last_deep_scrub_stamp":"2024-03-09T15:36:14.155108+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.155108+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:26:18.598712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":118,"num_write":57,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.15","version":"261'1","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.187041+0000","last_change":"2024-03-09T15:09:50.082323+0000","last_active":"2024-03-09T15:36:12.187041+0000","last_peered":"2024-03-09T15:36:12.187041+0000","last_clean":"2024-03-09T15:36:12.187041+0000","last_became_active":"2024-03-09T15:09:50.081688+0000","last_became_peered":"2024-03-09T15:09:50.081688+0000","last_unstale":"2024-03-09T15:36:12.187041+0000","last_undegraded":"2024-03-09T15:36:12.187041+0000","last_fullsized":"2024-03-09T15:36:12.187041+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.12","version":"261'2","reported_seq":47,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.353091+0000","last_change":"2024-03-09T14:46:26.806904+0000","last_active":"2024-03-09T15:36:06.353091+0000","last_peered":"2024-03-09T15:36:06.353091+0000","last_clean":"2024-03-09T15:36:06.353091+0000","last_became_active":"2024-03-09T14:46:26.806155+0000","last_became_peered":"2024-03-09T14:46:26.806155+0000","last_unstale":"2024-03-09T15:36:06.353091+0000","last_undegraded":"2024-03-09T15:36:06.353091+0000","last_fullsized":"2024-03-09T15:36:06.353091+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.13","version":"261'2","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.155249+0000","last_change":"2024-03-09T14:46:18.783226+0000","last_active":"2024-03-09T15:36:06.155249+0000","last_peered":"2024-03-09T15:36:06.155249+0000","last_clean":"2024-03-09T15:36:06.155249+0000","last_became_active":"2024-03-09T14:46:18.782340+0000","last_became_peered":"2024-03-09T14:46:18.782340+0000","last_unstale":"2024-03-09T15:36:06.155249+0000","last_undegraded":"2024-03-09T15:36:06.155249+0000","last_fullsized":"2024-03-09T15:36:06.155249+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":50,"num_write":2,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.10","version":"261'68","reported_seq":266,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.437896+0000","last_change":"2024-03-09T15:36:19.437984+0000","last_active":"2024-03-09T15:36:19.437896+0000","last_peered":"2024-03-09T15:36:19.437896+0000","last_clean":"2024-03-09T15:36:19.437896+0000","last_became_active":"2024-03-09T14:39:13.245107+0000","last_became_peered":"2024-03-09T14:39:13.245107+0000","last_unstale":"2024-03-09T15:36:19.437896+0000","last_undegraded":"2024-03-09T15:36:19.437896+0000","last_fullsized":"2024-03-09T15:36:19.437896+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'68","last_scrub_stamp":"2024-03-09T15:36:19.437698+0000","last_deep_scrub":"261'68","last_deep_scrub_stamp":"2024-03-09T15:36:19.437698+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.437698+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:55:08.019711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":382,"num_read_kb":270,"num_write":93,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1976,"num_omap_keys":4,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.a","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:56.005177+0000","last_change":"2024-03-09T15:09:50.080044+0000","last_active":"2024-03-09T15:35:56.005177+0000","last_peered":"2024-03-09T15:35:56.005177+0000","last_clean":"2024-03-09T15:35:56.005177+0000","last_became_active":"2024-03-09T15:09:50.079554+0000","last_became_peered":"2024-03-09T15:09:50.079554+0000","last_unstale":"2024-03-09T15:35:56.005177+0000","last_undegraded":"2024-03-09T15:35:56.005177+0000","last_fullsized":"2024-03-09T15:35:56.005177+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.d","version":"261'5","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.470652+0000","last_change":"2024-03-09T15:36:25.470799+0000","last_active":"2024-03-09T15:36:25.470652+0000","last_peered":"2024-03-09T15:36:25.470652+0000","last_clean":"2024-03-09T15:36:25.470652+0000","last_became_active":"2024-03-09T14:46:26.812859+0000","last_became_peered":"2024-03-09T14:46:26.812859+0000","last_unstale":"2024-03-09T15:36:25.470652+0000","last_undegraded":"2024-03-09T15:36:25.470652+0000","last_fullsized":"2024-03-09T15:36:25.470652+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'5","last_scrub_stamp":"2024-03-09T15:36:25.470415+0000","last_deep_scrub":"261'5","last_deep_scrub_stamp":"2024-03-09T15:36:25.470415+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.470415+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:20:04.740693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":8192,"num_write":10,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.c","version":"261'11","reported_seq":79,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.263874+0000","last_change":"2024-03-09T15:36:17.264004+0000","last_active":"2024-03-09T15:36:17.263874+0000","last_peered":"2024-03-09T15:36:17.263874+0000","last_clean":"2024-03-09T15:36:17.263874+0000","last_became_active":"2024-03-09T14:46:18.766370+0000","last_became_peered":"2024-03-09T14:46:18.766370+0000","last_unstale":"2024-03-09T15:36:17.263874+0000","last_undegraded":"2024-03-09T15:36:17.263874+0000","last_fullsized":"2024-03-09T15:36:17.263874+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'11","last_scrub_stamp":"2024-03-09T15:36:17.263641+0000","last_deep_scrub":"261'11","last_deep_scrub_stamp":"2024-03-09T15:36:17.263641+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.263641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:28:25.038359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":20480,"num_write":16,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.f","version":"258'18","reported_seq":108,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.541046+0000","last_change":"2024-03-09T15:36:12.541106+0000","last_active":"2024-03-09T15:36:12.541046+0000","last_peered":"2024-03-09T15:36:12.541046+0000","last_clean":"2024-03-09T15:36:12.541046+0000","last_became_active":"2024-03-09T14:39:13.268971+0000","last_became_peered":"2024-03-09T14:39:13.268971+0000","last_unstale":"2024-03-09T15:36:12.541046+0000","last_undegraded":"2024-03-09T15:36:12.541046+0000","last_fullsized":"2024-03-09T15:36:12.541046+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'18","last_scrub_stamp":"2024-03-09T15:36:12.540848+0000","last_deep_scrub":"258'18","last_deep_scrub_stamp":"2024-03-09T15:36:12.540848+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.540848+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:51.392572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.b","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:36.219456+0000","last_change":"2024-03-09T15:09:50.070451+0000","last_active":"2024-03-09T15:34:36.219456+0000","last_peered":"2024-03-09T15:34:36.219456+0000","last_clean":"2024-03-09T15:34:36.219456+0000","last_became_active":"2024-03-09T15:09:50.069508+0000","last_became_peered":"2024-03-09T15:09:50.069508+0000","last_unstale":"2024-03-09T15:34:36.219456+0000","last_undegraded":"2024-03-09T15:34:36.219456+0000","last_fullsized":"2024-03-09T15:34:36.219456+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.c","version":"261'3","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.457377+0000","last_change":"2024-03-09T15:36:29.457444+0000","last_active":"2024-03-09T15:36:29.457377+0000","last_peered":"2024-03-09T15:36:29.457377+0000","last_clean":"2024-03-09T15:36:29.457377+0000","last_became_active":"2024-03-09T14:46:26.813009+0000","last_became_peered":"2024-03-09T14:46:26.813009+0000","last_unstale":"2024-03-09T15:36:29.457377+0000","last_undegraded":"2024-03-09T15:36:29.457377+0000","last_fullsized":"2024-03-09T15:36:29.457377+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:29.457166+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:29.457166+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.457166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:35:35.350489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2048,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.d","version":"261'3","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.370225+0000","last_change":"2024-03-09T14:46:18.771558+0000","last_active":"2024-03-09T15:36:06.370225+0000","last_peered":"2024-03-09T15:36:06.370225+0000","last_clean":"2024-03-09T15:36:06.370225+0000","last_became_active":"2024-03-09T14:46:18.770732+0000","last_became_peered":"2024-03-09T14:46:18.770732+0000","last_unstale":"2024-03-09T15:36:06.370225+0000","last_undegraded":"2024-03-09T15:36:06.370225+0000","last_fullsized":"2024-03-09T15:36:06.370225+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.e","version":"261'8","reported_seq":88,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.655370+0000","last_change":"2024-03-09T15:36:18.655429+0000","last_active":"2024-03-09T15:36:18.655370+0000","last_peered":"2024-03-09T15:36:18.655370+0000","last_clean":"2024-03-09T15:36:18.655370+0000","last_became_active":"2024-03-09T14:39:13.271672+0000","last_became_peered":"2024-03-09T14:39:13.271672+0000","last_unstale":"2024-03-09T15:36:18.655370+0000","last_undegraded":"2024-03-09T15:36:18.655370+0000","last_fullsized":"2024-03-09T15:36:18.655370+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:18.655167+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:18.655167+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.655167+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:25:51.301220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.8","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:50.817817+0000","last_change":"2024-03-09T15:09:50.064645+0000","last_active":"2024-03-09T15:35:50.817817+0000","last_peered":"2024-03-09T15:35:50.817817+0000","last_clean":"2024-03-09T15:35:50.817817+0000","last_became_active":"2024-03-09T15:09:50.061662+0000","last_became_peered":"2024-03-09T15:09:50.061662+0000","last_unstale":"2024-03-09T15:35:50.817817+0000","last_undegraded":"2024-03-09T15:35:50.817817+0000","last_fullsized":"2024-03-09T15:35:50.817817+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.f","version":"261'6","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.109316+0000","last_change":"2024-03-09T14:46:26.900254+0000","last_active":"2024-03-09T15:36:06.109316+0000","last_peered":"2024-03-09T15:36:06.109316+0000","last_clean":"2024-03-09T15:36:06.109316+0000","last_became_active":"2024-03-09T14:46:26.899430+0000","last_became_peered":"2024-03-09T14:46:26.899430+0000","last_unstale":"2024-03-09T15:36:06.109316+0000","last_undegraded":"2024-03-09T15:36:06.109316+0000","last_fullsized":"2024-03-09T15:36:06.109316+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5120,"num_write":11,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.e","version":"258'3","reported_seq":65,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.271195+0000","last_change":"2024-03-09T15:36:15.271259+0000","last_active":"2024-03-09T15:36:15.271195+0000","last_peered":"2024-03-09T15:36:15.271195+0000","last_clean":"2024-03-09T15:36:15.271195+0000","last_became_active":"2024-03-09T14:46:18.768694+0000","last_became_peered":"2024-03-09T14:46:18.768694+0000","last_unstale":"2024-03-09T15:36:15.271195+0000","last_undegraded":"2024-03-09T15:36:15.271195+0000","last_fullsized":"2024-03-09T15:36:15.271195+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:15.270937+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:15.270937+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.270937+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:49:15.823574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.d","version":"261'43","reported_seq":168,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.940181+0000","last_change":"2024-03-09T15:36:15.940345+0000","last_active":"2024-03-09T15:36:15.940181+0000","last_peered":"2024-03-09T15:36:15.940181+0000","last_clean":"2024-03-09T15:36:15.940181+0000","last_became_active":"2024-03-09T14:39:13.257275+0000","last_became_peered":"2024-03-09T14:39:13.257275+0000","last_unstale":"2024-03-09T15:36:15.940181+0000","last_undegraded":"2024-03-09T15:36:15.940181+0000","last_fullsized":"2024-03-09T15:36:15.940181+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:15.939952+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:15.939952+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.939952+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:40:15.055734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":175,"num_read_kb":123,"num_write":70,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.9","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.236580+0000","last_change":"2024-03-09T15:09:50.077371+0000","last_active":"2024-03-09T15:35:51.236580+0000","last_peered":"2024-03-09T15:35:51.236580+0000","last_clean":"2024-03-09T15:35:51.236580+0000","last_became_active":"2024-03-09T15:09:50.076702+0000","last_became_peered":"2024-03-09T15:09:50.076702+0000","last_unstale":"2024-03-09T15:35:51.236580+0000","last_undegraded":"2024-03-09T15:35:51.236580+0000","last_fullsized":"2024-03-09T15:35:51.236580+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.e","version":"261'6","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.353658+0000","last_change":"2024-03-09T14:46:26.818547+0000","last_active":"2024-03-09T15:36:06.353658+0000","last_peered":"2024-03-09T15:36:06.353658+0000","last_clean":"2024-03-09T15:36:06.353658+0000","last_became_active":"2024-03-09T14:46:26.814709+0000","last_became_peered":"2024-03-09T14:46:26.814709+0000","last_unstale":"2024-03-09T15:36:06.353658+0000","last_undegraded":"2024-03-09T15:36:06.353658+0000","last_fullsized":"2024-03-09T15:36:06.353658+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.f","version":"261'5","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.512325+0000","last_change":"2024-03-09T15:36:13.512372+0000","last_active":"2024-03-09T15:36:13.512325+0000","last_peered":"2024-03-09T15:36:13.512325+0000","last_clean":"2024-03-09T15:36:13.512325+0000","last_became_active":"2024-03-09T14:46:18.766000+0000","last_became_peered":"2024-03-09T14:46:18.766000+0000","last_unstale":"2024-03-09T15:36:13.512325+0000","last_undegraded":"2024-03-09T15:36:13.512325+0000","last_fullsized":"2024-03-09T15:36:13.512325+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'5","last_scrub_stamp":"2024-03-09T15:36:13.512169+0000","last_deep_scrub":"261'5","last_deep_scrub_stamp":"2024-03-09T15:36:13.512169+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.512169+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:07:18.735860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4096,"num_write":10,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.c","version":"0'0","reported_seq":68,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.206724+0000","last_change":"2024-03-09T15:36:26.206778+0000","last_active":"2024-03-09T15:36:26.206724+0000","last_peered":"2024-03-09T15:36:26.206724+0000","last_clean":"2024-03-09T15:36:26.206724+0000","last_became_active":"2024-03-09T14:39:13.270527+0000","last_became_peered":"2024-03-09T14:39:13.270527+0000","last_unstale":"2024-03-09T15:36:26.206724+0000","last_undegraded":"2024-03-09T15:36:26.206724+0000","last_fullsized":"2024-03-09T15:36:26.206724+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:26.206522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:26.206522+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.206522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:18:22.897656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.e","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:35.817150+0000","last_change":"2024-03-09T15:09:50.087358+0000","last_active":"2024-03-09T15:34:35.817150+0000","last_peered":"2024-03-09T15:34:35.817150+0000","last_clean":"2024-03-09T15:34:35.817150+0000","last_became_active":"2024-03-09T15:09:50.085548+0000","last_became_peered":"2024-03-09T15:09:50.085548+0000","last_unstale":"2024-03-09T15:34:35.817150+0000","last_undegraded":"2024-03-09T15:34:35.817150+0000","last_fullsized":"2024-03-09T15:34:35.817150+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.9","version":"0'0","reported_seq":46,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.650389+0000","last_change":"2024-03-09T14:46:26.802054+0000","last_active":"2024-03-09T15:36:12.650389+0000","last_peered":"2024-03-09T15:36:12.650389+0000","last_clean":"2024-03-09T15:36:12.650389+0000","last_became_active":"2024-03-09T14:46:26.801235+0000","last_became_peered":"2024-03-09T14:46:26.801235+0000","last_unstale":"2024-03-09T15:36:12.650389+0000","last_undegraded":"2024-03-09T15:36:12.650389+0000","last_fullsized":"2024-03-09T15:36:12.650389+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.8","version":"0'0","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.316818+0000","last_change":"2024-03-09T15:36:23.316871+0000","last_active":"2024-03-09T15:36:23.316818+0000","last_peered":"2024-03-09T15:36:23.316818+0000","last_clean":"2024-03-09T15:36:23.316818+0000","last_became_active":"2024-03-09T14:46:18.774547+0000","last_became_peered":"2024-03-09T14:46:18.774547+0000","last_unstale":"2024-03-09T15:36:23.316818+0000","last_undegraded":"2024-03-09T15:36:23.316818+0000","last_fullsized":"2024-03-09T15:36:23.316818+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:23.316611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:23.316611+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.316611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:09:52.305120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.b","version":"258'10","reported_seq":94,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.250282+0000","last_change":"2024-03-09T15:36:20.250347+0000","last_active":"2024-03-09T15:36:20.250282+0000","last_peered":"2024-03-09T15:36:20.250282+0000","last_clean":"2024-03-09T15:36:20.250282+0000","last_became_active":"2024-03-09T14:39:13.257238+0000","last_became_peered":"2024-03-09T14:39:13.257238+0000","last_unstale":"2024-03-09T15:36:20.250282+0000","last_undegraded":"2024-03-09T15:36:20.250282+0000","last_fullsized":"2024-03-09T15:36:20.250282+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:20.250091+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:20.250091+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.250091+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:55:12.169918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.f","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:41.220718+0000","last_change":"2024-03-09T15:09:50.071187+0000","last_active":"2024-03-09T15:34:41.220718+0000","last_peered":"2024-03-09T15:34:41.220718+0000","last_clean":"2024-03-09T15:34:41.220718+0000","last_became_active":"2024-03-09T15:09:50.070526+0000","last_became_peered":"2024-03-09T15:09:50.070526+0000","last_unstale":"2024-03-09T15:34:41.220718+0000","last_undegraded":"2024-03-09T15:34:41.220718+0000","last_fullsized":"2024-03-09T15:34:41.220718+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.8","version":"0'0","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.159602+0000","last_change":"2024-03-09T15:36:25.159656+0000","last_active":"2024-03-09T15:36:25.159602+0000","last_peered":"2024-03-09T15:36:25.159602+0000","last_clean":"2024-03-09T15:36:25.159602+0000","last_became_active":"2024-03-09T14:46:26.893908+0000","last_became_peered":"2024-03-09T14:46:26.893908+0000","last_unstale":"2024-03-09T15:36:25.159602+0000","last_undegraded":"2024-03-09T15:36:25.159602+0000","last_fullsized":"2024-03-09T15:36:25.159602+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.159401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.159401+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.159401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:51:03.092803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.9","version":"261'3","reported_seq":63,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.495189+0000","last_change":"2024-03-09T15:36:11.495211+0000","last_active":"2024-03-09T15:36:11.495189+0000","last_peered":"2024-03-09T15:36:11.495189+0000","last_clean":"2024-03-09T15:36:11.495189+0000","last_became_active":"2024-03-09T14:46:18.761367+0000","last_became_peered":"2024-03-09T14:46:18.761367+0000","last_unstale":"2024-03-09T15:36:11.495189+0000","last_undegraded":"2024-03-09T15:36:11.495189+0000","last_fullsized":"2024-03-09T15:36:11.495189+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:11.495111+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:11.495111+0000","last_clean_scrub_stamp":"2024-03-09T15:36:11.495111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:55:49.995490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.a","version":"258'23","reported_seq":132,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.129434+0000","last_change":"2024-03-09T15:36:26.129498+0000","last_active":"2024-03-09T15:36:26.129434+0000","last_peered":"2024-03-09T15:36:26.129434+0000","last_clean":"2024-03-09T15:36:26.129434+0000","last_became_active":"2024-03-09T14:39:13.252547+0000","last_became_peered":"2024-03-09T14:39:13.252547+0000","last_unstale":"2024-03-09T15:36:26.129434+0000","last_undegraded":"2024-03-09T15:36:26.129434+0000","last_fullsized":"2024-03-09T15:36:26.129434+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'23","last_scrub_stamp":"2024-03-09T15:36:26.129236+0000","last_deep_scrub":"258'23","last_deep_scrub_stamp":"2024-03-09T15:36:26.129236+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.129236+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:37:12.973344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":86,"num_write":38,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.c","version":"261'1","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:36.219508+0000","last_change":"2024-03-09T15:09:50.070468+0000","last_active":"2024-03-09T15:34:36.219508+0000","last_peered":"2024-03-09T15:34:36.219508+0000","last_clean":"2024-03-09T15:34:36.219508+0000","last_became_active":"2024-03-09T15:09:50.069324+0000","last_became_peered":"2024-03-09T15:09:50.069324+0000","last_unstale":"2024-03-09T15:34:36.219508+0000","last_undegraded":"2024-03-09T15:34:36.219508+0000","last_fullsized":"2024-03-09T15:34:36.219508+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.b","version":"258'3","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:55.992308+0000","last_change":"2024-03-09T14:46:26.805564+0000","last_active":"2024-03-09T15:35:55.992308+0000","last_peered":"2024-03-09T15:35:55.992308+0000","last_clean":"2024-03-09T15:35:55.992308+0000","last_became_active":"2024-03-09T14:46:26.804615+0000","last_became_peered":"2024-03-09T14:46:26.804615+0000","last_unstale":"2024-03-09T15:35:55.992308+0000","last_undegraded":"2024-03-09T15:35:55.992308+0000","last_fullsized":"2024-03-09T15:35:55.992308+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.a","version":"258'3","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.202430+0000","last_change":"2024-03-09T15:36:11.202430+0000","last_active":"2024-03-09T15:36:11.202430+0000","last_peered":"2024-03-09T15:36:11.202430+0000","last_clean":"2024-03-09T15:36:11.202430+0000","last_became_active":"2024-03-09T14:46:18.768431+0000","last_became_peered":"2024-03-09T14:46:18.768431+0000","last_unstale":"2024-03-09T15:36:11.202430+0000","last_undegraded":"2024-03-09T15:36:11.202430+0000","last_fullsized":"2024-03-09T15:36:11.202430+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:11.202170+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:11.202170+0000","last_clean_scrub_stamp":"2024-03-09T15:36:11.202170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:12:25.295523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.9","version":"261'43","reported_seq":166,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.428686+0000","last_change":"2024-03-09T15:36:18.428750+0000","last_active":"2024-03-09T15:36:18.428686+0000","last_peered":"2024-03-09T15:36:18.428686+0000","last_clean":"2024-03-09T15:36:18.428686+0000","last_became_active":"2024-03-09T14:39:13.259975+0000","last_became_peered":"2024-03-09T14:39:13.259975+0000","last_unstale":"2024-03-09T15:36:18.428686+0000","last_undegraded":"2024-03-09T15:36:18.428686+0000","last_fullsized":"2024-03-09T15:36:18.428686+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:18.428488+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:18.428488+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.428488+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:15:47.696457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":169,"num_read_kb":119,"num_write":74,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.d","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:36.026315+0000","last_change":"2024-03-09T15:09:50.084646+0000","last_active":"2024-03-09T15:34:36.026315+0000","last_peered":"2024-03-09T15:34:36.026315+0000","last_clean":"2024-03-09T15:34:36.026315+0000","last_became_active":"2024-03-09T15:09:50.084074+0000","last_became_peered":"2024-03-09T15:09:50.084074+0000","last_unstale":"2024-03-09T15:34:36.026315+0000","last_undegraded":"2024-03-09T15:34:36.026315+0000","last_fullsized":"2024-03-09T15:34:36.026315+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.a","version":"258'3","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.196067+0000","last_change":"2024-03-09T15:36:21.196067+0000","last_active":"2024-03-09T15:36:21.196067+0000","last_peered":"2024-03-09T15:36:21.196067+0000","last_clean":"2024-03-09T15:36:21.196067+0000","last_became_active":"2024-03-09T14:46:26.825949+0000","last_became_peered":"2024-03-09T14:46:26.825949+0000","last_unstale":"2024-03-09T15:36:21.196067+0000","last_undegraded":"2024-03-09T15:36:21.196067+0000","last_fullsized":"2024-03-09T15:36:21.196067+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:21.195795+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:21.195795+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.195795+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:51.756963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.b","version":"261'3","reported_seq":64,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.180486+0000","last_change":"2024-03-09T15:36:29.180546+0000","last_active":"2024-03-09T15:36:29.180486+0000","last_peered":"2024-03-09T15:36:29.180486+0000","last_clean":"2024-03-09T15:36:29.180486+0000","last_became_active":"2024-03-09T14:46:18.784869+0000","last_became_peered":"2024-03-09T14:46:18.784869+0000","last_unstale":"2024-03-09T15:36:29.180486+0000","last_undegraded":"2024-03-09T15:36:29.180486+0000","last_fullsized":"2024-03-09T15:36:29.180486+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:29.180292+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:29.180292+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.180292+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:09:41.856539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.8","version":"258'10","reported_seq":95,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.980744+0000","last_change":"2024-03-09T15:36:16.980807+0000","last_active":"2024-03-09T15:36:16.980744+0000","last_peered":"2024-03-09T15:36:16.980744+0000","last_clean":"2024-03-09T15:36:16.980744+0000","last_became_active":"2024-03-09T14:39:13.264006+0000","last_became_peered":"2024-03-09T14:39:13.264006+0000","last_unstale":"2024-03-09T15:36:16.980744+0000","last_undegraded":"2024-03-09T15:36:16.980744+0000","last_fullsized":"2024-03-09T15:36:16.980744+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:16.980539+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:16.980539+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.980539+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:50:44.529795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.2","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.222739+0000","last_change":"2024-03-09T15:09:50.066069+0000","last_active":"2024-03-09T15:34:51.222739+0000","last_peered":"2024-03-09T15:34:51.222739+0000","last_clean":"2024-03-09T15:34:51.222739+0000","last_became_active":"2024-03-09T15:09:50.065487+0000","last_became_peered":"2024-03-09T15:09:50.065487+0000","last_unstale":"2024-03-09T15:34:51.222739+0000","last_undegraded":"2024-03-09T15:34:51.222739+0000","last_fullsized":"2024-03-09T15:34:51.222739+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.5","version":"258'2","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:36.026359+0000","last_change":"2024-03-09T14:46:26.813029+0000","last_active":"2024-03-09T15:34:36.026359+0000","last_peered":"2024-03-09T15:34:36.026359+0000","last_clean":"2024-03-09T15:34:36.026359+0000","last_became_active":"2024-03-09T14:46:26.812137+0000","last_became_peered":"2024-03-09T14:46:26.812137+0000","last_unstale":"2024-03-09T15:34:36.026359+0000","last_undegraded":"2024-03-09T15:34:36.026359+0000","last_fullsized":"2024-03-09T15:34:36.026359+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4096,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.4","version":"0'0","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.218589+0000","last_change":"2024-03-09T15:36:19.218589+0000","last_active":"2024-03-09T15:36:19.218589+0000","last_peered":"2024-03-09T15:36:19.218589+0000","last_clean":"2024-03-09T15:36:19.218589+0000","last_became_active":"2024-03-09T14:46:18.751608+0000","last_became_peered":"2024-03-09T14:46:18.751608+0000","last_unstale":"2024-03-09T15:36:19.218589+0000","last_undegraded":"2024-03-09T15:36:19.218589+0000","last_fullsized":"2024-03-09T15:36:19.218589+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:19.218339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:19.218339+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.218339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:28:04.515161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.7","version":"261'8","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.227007+0000","last_change":"2024-03-09T15:36:14.227039+0000","last_active":"2024-03-09T15:36:14.227007+0000","last_peered":"2024-03-09T15:36:14.227007+0000","last_clean":"2024-03-09T15:36:14.227007+0000","last_became_active":"2024-03-09T14:39:13.273363+0000","last_became_peered":"2024-03-09T14:39:13.273363+0000","last_unstale":"2024-03-09T15:36:14.227007+0000","last_undegraded":"2024-03-09T15:36:14.227007+0000","last_fullsized":"2024-03-09T15:36:14.227007+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:14.226893+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:14.226893+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.226893+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:21:38.723375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.3","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.211395+0000","last_change":"2024-03-09T15:09:50.078106+0000","last_active":"2024-03-09T15:34:51.211395+0000","last_peered":"2024-03-09T15:34:51.211395+0000","last_clean":"2024-03-09T15:34:51.211395+0000","last_became_active":"2024-03-09T15:09:50.072488+0000","last_became_peered":"2024-03-09T15:09:50.072488+0000","last_unstale":"2024-03-09T15:34:51.211395+0000","last_undegraded":"2024-03-09T15:34:51.211395+0000","last_fullsized":"2024-03-09T15:34:51.211395+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.4","version":"258'2","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.740325+0000","last_change":"2024-03-09T14:46:26.895288+0000","last_active":"2024-03-09T15:36:10.740325+0000","last_peered":"2024-03-09T15:36:10.740325+0000","last_clean":"2024-03-09T15:36:10.740325+0000","last_became_active":"2024-03-09T14:46:26.894506+0000","last_became_peered":"2024-03-09T14:46:26.894506+0000","last_unstale":"2024-03-09T15:36:10.740325+0000","last_undegraded":"2024-03-09T15:36:10.740325+0000","last_fullsized":"2024-03-09T15:36:10.740325+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":23,"num_write":2,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.5","version":"258'5","reported_seq":67,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:30.987675+0000","last_change":"2024-03-09T14:46:18.771982+0000","last_active":"2024-03-09T15:34:30.987675+0000","last_peered":"2024-03-09T15:34:30.987675+0000","last_clean":"2024-03-09T15:34:30.987675+0000","last_became_active":"2024-03-09T14:46:18.770799+0000","last_became_peered":"2024-03-09T14:46:18.770799+0000","last_unstale":"2024-03-09T15:34:30.987675+0000","last_undegraded":"2024-03-09T15:34:30.987675+0000","last_fullsized":"2024-03-09T15:34:30.987675+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.6","version":"261'22","reported_seq":121,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.972029+0000","last_change":"2024-03-09T15:36:14.972165+0000","last_active":"2024-03-09T15:36:14.972029+0000","last_peered":"2024-03-09T15:36:14.972029+0000","last_clean":"2024-03-09T15:36:14.972029+0000","last_became_active":"2024-03-09T14:39:13.266246+0000","last_became_peered":"2024-03-09T14:39:13.266246+0000","last_unstale":"2024-03-09T15:36:14.972029+0000","last_undegraded":"2024-03-09T15:36:14.972029+0000","last_fullsized":"2024-03-09T15:36:14.972029+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'22","last_scrub_stamp":"2024-03-09T15:36:14.971789+0000","last_deep_scrub":"261'22","last_deep_scrub_stamp":"2024-03-09T15:36:14.971789+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.971789+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:50:18.362986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":65,"num_write":41,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.0","version":"0'0","reported_seq":26,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:00.993222+0000","last_change":"2024-03-09T15:09:50.064673+0000","last_active":"2024-03-09T15:36:00.993222+0000","last_peered":"2024-03-09T15:36:00.993222+0000","last_clean":"2024-03-09T15:36:00.993222+0000","last_became_active":"2024-03-09T15:09:50.063350+0000","last_became_peered":"2024-03-09T15:09:50.063350+0000","last_unstale":"2024-03-09T15:36:00.993222+0000","last_undegraded":"2024-03-09T15:36:00.993222+0000","last_fullsized":"2024-03-09T15:36:00.993222+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.7","version":"0'0","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.151944+0000","last_change":"2024-03-09T15:36:29.151997+0000","last_active":"2024-03-09T15:36:29.151944+0000","last_peered":"2024-03-09T15:36:29.151944+0000","last_clean":"2024-03-09T15:36:29.151944+0000","last_became_active":"2024-03-09T14:46:26.896503+0000","last_became_peered":"2024-03-09T14:46:26.896503+0000","last_unstale":"2024-03-09T15:36:29.151944+0000","last_undegraded":"2024-03-09T15:36:29.151944+0000","last_fullsized":"2024-03-09T15:36:29.151944+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:29.151756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:29.151756+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.151756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:12:40.874461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.6","version":"261'2","reported_seq":63,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.752529+0000","last_change":"2024-03-09T15:36:26.752595+0000","last_active":"2024-03-09T15:36:26.752529+0000","last_peered":"2024-03-09T15:36:26.752529+0000","last_clean":"2024-03-09T15:36:26.752529+0000","last_became_active":"2024-03-09T14:46:18.778572+0000","last_became_peered":"2024-03-09T14:46:18.778572+0000","last_unstale":"2024-03-09T15:36:26.752529+0000","last_undegraded":"2024-03-09T15:36:26.752529+0000","last_fullsized":"2024-03-09T15:36:26.752529+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:26.752326+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:26.752326+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.752326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:06:26.783165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.5","version":"258'18","reported_seq":116,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.164653+0000","last_change":"2024-03-09T15:36:23.164716+0000","last_active":"2024-03-09T15:36:23.164653+0000","last_peered":"2024-03-09T15:36:23.164653+0000","last_clean":"2024-03-09T15:36:23.164653+0000","last_became_active":"2024-03-09T14:39:13.263833+0000","last_became_peered":"2024-03-09T14:39:13.263833+0000","last_unstale":"2024-03-09T15:36:23.164653+0000","last_undegraded":"2024-03-09T15:36:23.164653+0000","last_fullsized":"2024-03-09T15:36:23.164653+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'18","last_scrub_stamp":"2024-03-09T15:36:23.164449+0000","last_deep_scrub":"258'18","last_deep_scrub_stamp":"2024-03-09T15:36:23.164449+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.164449+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:22:04.204143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":58,"num_write":29,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.1","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.006085+0000","last_change":"2024-03-09T15:09:50.069405+0000","last_active":"2024-03-09T15:34:51.006085+0000","last_peered":"2024-03-09T15:34:51.006085+0000","last_clean":"2024-03-09T15:34:51.006085+0000","last_became_active":"2024-03-09T15:09:50.067947+0000","last_became_peered":"2024-03-09T15:09:50.067947+0000","last_unstale":"2024-03-09T15:34:51.006085+0000","last_undegraded":"2024-03-09T15:34:51.006085+0000","last_fullsized":"2024-03-09T15:34:51.006085+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"67.6","version":"261'4","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.650346+0000","last_change":"2024-03-09T14:46:26.809411+0000","last_active":"2024-03-09T15:36:12.650346+0000","last_peered":"2024-03-09T15:36:12.650346+0000","last_clean":"2024-03-09T15:36:12.650346+0000","last_became_active":"2024-03-09T14:46:26.808384+0000","last_became_peered":"2024-03-09T14:46:26.808384+0000","last_unstale":"2024-03-09T15:36:12.650346+0000","last_undegraded":"2024-03-09T15:36:12.650346+0000","last_fullsized":"2024-03-09T15:36:12.650346+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":180,"num_write":4,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.7","version":"261'9","reported_seq":83,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.291656+0000","last_change":"2024-03-09T15:36:14.291833+0000","last_active":"2024-03-09T15:36:14.291656+0000","last_peered":"2024-03-09T15:36:14.291656+0000","last_clean":"2024-03-09T15:36:14.291656+0000","last_became_active":"2024-03-09T14:46:18.762472+0000","last_became_peered":"2024-03-09T14:46:18.762472+0000","last_unstale":"2024-03-09T15:36:14.291656+0000","last_undegraded":"2024-03-09T15:36:14.291656+0000","last_fullsized":"2024-03-09T15:36:14.291656+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-03-09T15:36:14.291396+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-03-09T15:36:14.291396+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.291396+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:45:34.816033+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":30722,"num_write":17,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.4","version":"258'10","reported_seq":96,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.555803+0000","last_change":"2024-03-09T15:36:14.555905+0000","last_active":"2024-03-09T15:36:14.555803+0000","last_peered":"2024-03-09T15:36:14.555803+0000","last_clean":"2024-03-09T15:36:14.555803+0000","last_became_active":"2024-03-09T14:39:13.270238+0000","last_became_peered":"2024-03-09T14:39:13.270238+0000","last_unstale":"2024-03-09T15:36:14.555803+0000","last_undegraded":"2024-03-09T15:36:14.555803+0000","last_fullsized":"2024-03-09T15:36:14.555803+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:14.555648+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:14.555648+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.555648+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:38:18.000083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.6","version":"260'1","reported_seq":26,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.004181+0000","last_change":"2024-03-09T15:09:50.072623+0000","last_active":"2024-03-09T15:35:51.004181+0000","last_peered":"2024-03-09T15:35:51.004181+0000","last_clean":"2024-03-09T15:35:51.004181+0000","last_became_active":"2024-03-09T15:09:50.072180+0000","last_became_peered":"2024-03-09T15:09:50.072180+0000","last_unstale":"2024-03-09T15:35:51.004181+0000","last_undegraded":"2024-03-09T15:35:51.004181+0000","last_fullsized":"2024-03-09T15:35:51.004181+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1","version":"0'0","reported_seq":55,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.542310+0000","last_change":"2024-03-09T15:36:21.542310+0000","last_active":"2024-03-09T15:36:21.542310+0000","last_peered":"2024-03-09T15:36:21.542310+0000","last_clean":"2024-03-09T15:36:21.542310+0000","last_became_active":"2024-03-09T14:46:26.799019+0000","last_became_peered":"2024-03-09T14:46:26.799019+0000","last_unstale":"2024-03-09T15:36:21.542310+0000","last_undegraded":"2024-03-09T15:36:21.542310+0000","last_fullsized":"2024-03-09T15:36:21.542310+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.542044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.542044+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.542044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:24:20.252130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.0","version":"0'0","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.229238+0000","last_change":"2024-03-09T15:36:18.229238+0000","last_active":"2024-03-09T15:36:18.229238+0000","last_peered":"2024-03-09T15:36:18.229238+0000","last_clean":"2024-03-09T15:36:18.229238+0000","last_became_active":"2024-03-09T14:46:18.752290+0000","last_became_peered":"2024-03-09T14:46:18.752290+0000","last_unstale":"2024-03-09T15:36:18.229238+0000","last_undegraded":"2024-03-09T15:36:18.229238+0000","last_fullsized":"2024-03-09T15:36:18.229238+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:18.228896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:18.228896+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.228896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:58:17.281185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.3","version":"261'61","reported_seq":224,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.587549+0000","last_change":"2024-03-09T15:36:14.587616+0000","last_active":"2024-03-09T15:36:14.587549+0000","last_peered":"2024-03-09T15:36:14.587549+0000","last_clean":"2024-03-09T15:36:14.587549+0000","last_became_active":"2024-03-09T14:39:13.259125+0000","last_became_peered":"2024-03-09T14:39:13.259125+0000","last_unstale":"2024-03-09T15:36:14.587549+0000","last_undegraded":"2024-03-09T15:36:14.587549+0000","last_fullsized":"2024-03-09T15:36:14.587549+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'61","last_scrub_stamp":"2024-03-09T15:36:14.587351+0000","last_deep_scrub":"261'61","last_deep_scrub_stamp":"2024-03-09T15:36:14.587351+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.587351+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:00:04.188736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":281,"num_read_kb":205,"num_write":98,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.7","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.044007+0000","last_change":"2024-03-09T15:09:50.064879+0000","last_active":"2024-03-09T15:35:51.044007+0000","last_peered":"2024-03-09T15:35:51.044007+0000","last_clean":"2024-03-09T15:35:51.044007+0000","last_became_active":"2024-03-09T15:09:50.062429+0000","last_became_peered":"2024-03-09T15:09:50.062429+0000","last_unstale":"2024-03-09T15:35:51.044007+0000","last_undegraded":"2024-03-09T15:35:51.044007+0000","last_fullsized":"2024-03-09T15:35:51.044007+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.0","version":"261'6","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.318433+0000","last_change":"2024-03-09T14:46:26.816110+0000","last_active":"2024-03-09T15:36:06.318433+0000","last_peered":"2024-03-09T15:36:06.318433+0000","last_clean":"2024-03-09T15:36:06.318433+0000","last_became_active":"2024-03-09T14:46:26.815311+0000","last_became_peered":"2024-03-09T14:46:26.815311+0000","last_unstale":"2024-03-09T15:36:06.318433+0000","last_undegraded":"2024-03-09T15:36:06.318433+0000","last_fullsized":"2024-03-09T15:36:06.318433+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":16384,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1","version":"261'6","reported_seq":73,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.249215+0000","last_change":"2024-03-09T15:36:15.249215+0000","last_active":"2024-03-09T15:36:15.249215+0000","last_peered":"2024-03-09T15:36:15.249215+0000","last_clean":"2024-03-09T15:36:15.249215+0000","last_became_active":"2024-03-09T14:46:18.757694+0000","last_became_peered":"2024-03-09T14:46:18.757694+0000","last_unstale":"2024-03-09T15:36:15.249215+0000","last_undegraded":"2024-03-09T15:36:15.249215+0000","last_fullsized":"2024-03-09T15:36:15.249215+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:15.248968+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:15.248968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.248968+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:50:59.398908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":4098,"num_write":9,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.2","version":"258'20","reported_seq":119,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.118799+0000","last_change":"2024-03-09T15:36:13.118868+0000","last_active":"2024-03-09T15:36:13.118799+0000","last_peered":"2024-03-09T15:36:13.118799+0000","last_clean":"2024-03-09T15:36:13.118799+0000","last_became_active":"2024-03-09T14:39:13.264203+0000","last_became_peered":"2024-03-09T14:39:13.264203+0000","last_unstale":"2024-03-09T15:36:13.118799+0000","last_undegraded":"2024-03-09T15:36:13.118799+0000","last_fullsized":"2024-03-09T15:36:13.118799+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'20","last_scrub_stamp":"2024-03-09T15:36:13.118589+0000","last_deep_scrub":"258'20","last_deep_scrub_stamp":"2024-03-09T15:36:13.118589+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.118589+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:29:35.065434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":65,"num_write":31,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.4","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.030150+0000","last_change":"2024-03-09T15:09:50.070703+0000","last_active":"2024-03-09T15:34:51.030150+0000","last_peered":"2024-03-09T15:34:51.030150+0000","last_clean":"2024-03-09T15:34:51.030150+0000","last_became_active":"2024-03-09T15:09:50.066222+0000","last_became_peered":"2024-03-09T15:09:50.066222+0000","last_unstale":"2024-03-09T15:34:51.030150+0000","last_undegraded":"2024-03-09T15:34:51.030150+0000","last_fullsized":"2024-03-09T15:34:51.030150+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.3","version":"0'0","reported_seq":47,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.613968+0000","last_change":"2024-03-09T14:46:26.826931+0000","last_active":"2024-03-09T15:36:13.613968+0000","last_peered":"2024-03-09T15:36:13.613968+0000","last_clean":"2024-03-09T15:36:13.613968+0000","last_became_active":"2024-03-09T14:46:26.826105+0000","last_became_peered":"2024-03-09T14:46:26.826105+0000","last_unstale":"2024-03-09T15:36:13.613968+0000","last_undegraded":"2024-03-09T15:36:13.613968+0000","last_fullsized":"2024-03-09T15:36:13.613968+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.2","version":"0'0","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.469191+0000","last_change":"2024-03-09T15:36:12.469191+0000","last_active":"2024-03-09T15:36:12.469191+0000","last_peered":"2024-03-09T15:36:12.469191+0000","last_clean":"2024-03-09T15:36:12.469191+0000","last_became_active":"2024-03-09T14:46:18.749091+0000","last_became_peered":"2024-03-09T14:46:18.749091+0000","last_unstale":"2024-03-09T15:36:12.469191+0000","last_undegraded":"2024-03-09T15:36:12.469191+0000","last_fullsized":"2024-03-09T15:36:12.469191+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:12.468992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:12.468992+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.468992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:06:25.628906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1","version":"261'25","reported_seq":134,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.079889+0000","last_change":"2024-03-09T15:36:19.079952+0000","last_active":"2024-03-09T15:36:19.079889+0000","last_peered":"2024-03-09T15:36:19.079889+0000","last_clean":"2024-03-09T15:36:19.079889+0000","last_became_active":"2024-03-09T14:39:13.255589+0000","last_became_peered":"2024-03-09T14:39:13.255589+0000","last_unstale":"2024-03-09T15:36:19.079889+0000","last_undegraded":"2024-03-09T15:36:19.079889+0000","last_fullsized":"2024-03-09T15:36:19.079889+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'25","last_scrub_stamp":"2024-03-09T15:36:19.079692+0000","last_deep_scrub":"261'25","last_deep_scrub_stamp":"2024-03-09T15:36:19.079692+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.079692+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:25:37.071627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":79,"num_write":35,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.5","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:56.223957+0000","last_change":"2024-03-09T15:09:50.083562+0000","last_active":"2024-03-09T15:34:56.223957+0000","last_peered":"2024-03-09T15:34:56.223957+0000","last_clean":"2024-03-09T15:34:56.223957+0000","last_became_active":"2024-03-09T15:09:50.083064+0000","last_became_peered":"2024-03-09T15:09:50.083064+0000","last_unstale":"2024-03-09T15:34:56.223957+0000","last_undegraded":"2024-03-09T15:34:56.223957+0000","last_fullsized":"2024-03-09T15:34:56.223957+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.2","version":"258'3","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.650587+0000","last_change":"2024-03-09T14:46:26.808768+0000","last_active":"2024-03-09T15:36:12.650587+0000","last_peered":"2024-03-09T15:36:12.650587+0000","last_clean":"2024-03-09T15:36:12.650587+0000","last_became_active":"2024-03-09T14:46:26.807746+0000","last_became_peered":"2024-03-09T14:46:26.807746+0000","last_unstale":"2024-03-09T15:36:12.650587+0000","last_undegraded":"2024-03-09T15:36:12.650587+0000","last_fullsized":"2024-03-09T15:36:12.650587+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1024,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.3","version":"0'0","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.030314+0000","last_change":"2024-03-09T15:36:18.030314+0000","last_active":"2024-03-09T15:36:18.030314+0000","last_peered":"2024-03-09T15:36:18.030314+0000","last_clean":"2024-03-09T15:36:18.030314+0000","last_became_active":"2024-03-09T14:46:18.765077+0000","last_became_peered":"2024-03-09T14:46:18.765077+0000","last_unstale":"2024-03-09T15:36:18.030314+0000","last_undegraded":"2024-03-09T15:36:18.030314+0000","last_fullsized":"2024-03-09T15:36:18.030314+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:18.030067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:18.030067+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.030067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:44:34.102574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.0","version":"250'25","reported_seq":139,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.180831+0000","last_change":"2024-03-09T15:36:20.180899+0000","last_active":"2024-03-09T15:36:20.180831+0000","last_peered":"2024-03-09T15:36:20.180831+0000","last_clean":"2024-03-09T15:36:20.180831+0000","last_became_active":"2024-03-09T14:39:13.268553+0000","last_became_peered":"2024-03-09T14:39:13.268553+0000","last_unstale":"2024-03-09T15:36:20.180831+0000","last_undegraded":"2024-03-09T15:36:20.180831+0000","last_fullsized":"2024-03-09T15:36:20.180831+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-03-09T15:36:20.180625+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-03-09T15:36:20.180625+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.180625+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:48:22.579836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":97,"num_write":32,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"261'114","reported_seq":672,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.486191+0000","last_change":"2024-03-09T14:28:32.340306+0000","last_active":"2024-03-09T15:36:06.486191+0000","last_peered":"2024-03-09T15:36:06.486191+0000","last_clean":"2024-03-09T15:36:06.486191+0000","last_became_active":"2024-03-09T14:28:32.339785+0000","last_became_peered":"2024-03-09T14:28:32.339785+0000","last_unstale":"2024-03-09T15:36:06.486191+0000","last_undegraded":"2024-03-09T15:36:06.486191+0000","last_fullsized":"2024-03-09T15:36:06.486191+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":130,"num_read_kb":5377,"num_write":373,"num_write_kb":16955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"261'146","reported_seq":688,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.227641+0000","last_change":"2024-03-09T14:28:34.304676+0000","last_active":"2024-03-09T15:36:06.227641+0000","last_peered":"2024-03-09T15:36:06.227641+0000","last_clean":"2024-03-09T15:36:06.227641+0000","last_became_active":"2024-03-09T14:28:34.303980+0000","last_became_peered":"2024-03-09T14:28:34.303980+0000","last_unstale":"2024-03-09T15:36:06.227641+0000","last_undegraded":"2024-03-09T15:36:06.227641+0000","last_fullsized":"2024-03-09T15:36:06.227641+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":546,"num_read_kb":417,"num_write":228,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"261'135","reported_seq":688,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.522120+0000","last_change":"2024-03-09T14:28:32.307225+0000","last_active":"2024-03-09T15:36:06.522120+0000","last_peered":"2024-03-09T15:36:06.522120+0000","last_clean":"2024-03-09T15:36:06.522120+0000","last_became_active":"2024-03-09T14:28:32.306840+0000","last_became_peered":"2024-03-09T14:28:32.306840+0000","last_unstale":"2024-03-09T15:36:06.522120+0000","last_undegraded":"2024-03-09T15:36:06.522120+0000","last_fullsized":"2024-03-09T15:36:06.522120+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":100,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3389647,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":156,"num_read_kb":3004,"num_write":523,"num_write_kb":19720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"261'89","reported_seq":656,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.239609+0000","last_change":"2024-03-09T14:28:34.324788+0000","last_active":"2024-03-09T15:36:06.239609+0000","last_peered":"2024-03-09T15:36:06.239609+0000","last_clean":"2024-03-09T15:36:06.239609+0000","last_became_active":"2024-03-09T14:28:34.324053+0000","last_became_peered":"2024-03-09T14:28:34.324053+0000","last_unstale":"2024-03-09T15:36:06.239609+0000","last_undegraded":"2024-03-09T15:36:06.239609+0000","last_fullsized":"2024-03-09T15:36:06.239609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":372,"num_read_kb":300,"num_write":141,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3d","version":"261'158","reported_seq":719,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.527086+0000","last_change":"2024-03-09T14:28:32.292874+0000","last_active":"2024-03-09T15:36:06.527086+0000","last_peered":"2024-03-09T15:36:06.527086+0000","last_clean":"2024-03-09T15:36:06.527086+0000","last_became_active":"2024-03-09T14:28:32.286966+0000","last_became_peered":"2024-03-09T14:28:32.286966+0000","last_unstale":"2024-03-09T15:36:06.527086+0000","last_undegraded":"2024-03-09T15:36:06.527086+0000","last_fullsized":"2024-03-09T15:36:06.527086+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":100,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":14171,"num_write":610,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"261'157","reported_seq":719,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.022871+0000","last_change":"2024-03-09T14:28:34.307655+0000","last_active":"2024-03-09T15:36:06.022871+0000","last_peered":"2024-03-09T15:36:06.022871+0000","last_clean":"2024-03-09T15:36:06.022871+0000","last_became_active":"2024-03-09T14:28:34.306764+0000","last_became_peered":"2024-03-09T14:28:34.306764+0000","last_unstale":"2024-03-09T15:36:06.022871+0000","last_undegraded":"2024-03-09T15:36:06.022871+0000","last_fullsized":"2024-03-09T15:36:06.022871+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":481,"num_write":261,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3c","version":"261'190","reported_seq":783,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.491016+0000","last_change":"2024-03-09T14:28:32.310328+0000","last_active":"2024-03-09T15:36:06.491016+0000","last_peered":"2024-03-09T15:36:06.491016+0000","last_clean":"2024-03-09T15:36:06.491016+0000","last_became_active":"2024-03-09T14:28:32.305615+0000","last_became_peered":"2024-03-09T14:28:32.305615+0000","last_unstale":"2024-03-09T15:36:06.491016+0000","last_undegraded":"2024-03-09T15:36:06.491016+0000","last_fullsized":"2024-03-09T15:36:06.491016+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":100,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5326894,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":256,"num_read_kb":28937,"num_write":797,"num_write_kb":45835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"261'156","reported_seq":733,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.046701+0000","last_change":"2024-03-09T14:28:34.316937+0000","last_active":"2024-03-09T15:36:06.046701+0000","last_peered":"2024-03-09T15:36:06.046701+0000","last_clean":"2024-03-09T15:36:06.046701+0000","last_became_active":"2024-03-09T14:28:34.316500+0000","last_became_peered":"2024-03-09T14:28:34.316500+0000","last_unstale":"2024-03-09T15:36:06.046701+0000","last_undegraded":"2024-03-09T15:36:06.046701+0000","last_fullsized":"2024-03-09T15:36:06.046701+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":100,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":622,"num_read_kb":488,"num_write":257,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3b","version":"261'150","reported_seq":813,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.509451+0000","last_change":"2024-03-09T14:28:32.317675+0000","last_active":"2024-03-09T15:36:06.509451+0000","last_peered":"2024-03-09T15:36:06.509451+0000","last_clean":"2024-03-09T15:36:06.509451+0000","last_became_active":"2024-03-09T14:28:32.314923+0000","last_became_peered":"2024-03-09T14:28:32.314923+0000","last_unstale":"2024-03-09T15:36:06.509451+0000","last_undegraded":"2024-03-09T15:36:06.509451+0000","last_fullsized":"2024-03-09T15:36:06.509451+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1917997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":297,"num_read_kb":17282,"num_write":517,"num_write_kb":14747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3c","version":"261'124","reported_seq":707,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:55.806374+0000","last_change":"2024-03-09T14:28:34.315343+0000","last_active":"2024-03-09T15:34:55.806374+0000","last_peered":"2024-03-09T15:34:55.806374+0000","last_clean":"2024-03-09T15:34:55.806374+0000","last_became_active":"2024-03-09T14:28:34.314617+0000","last_became_peered":"2024-03-09T14:28:34.314617+0000","last_unstale":"2024-03-09T15:34:55.806374+0000","last_undegraded":"2024-03-09T15:34:55.806374+0000","last_fullsized":"2024-03-09T15:34:55.806374+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":493,"num_read_kb":395,"num_write":185,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3a","version":"261'150","reported_seq":692,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.165942+0000","last_change":"2024-03-09T15:36:14.166015+0000","last_active":"2024-03-09T15:36:14.165942+0000","last_peered":"2024-03-09T15:36:14.165942+0000","last_clean":"2024-03-09T15:36:14.165942+0000","last_became_active":"2024-03-09T14:29:06.433145+0000","last_became_peered":"2024-03-09T14:29:06.433145+0000","last_unstale":"2024-03-09T15:36:14.165942+0000","last_undegraded":"2024-03-09T15:36:14.165942+0000","last_fullsized":"2024-03-09T15:36:14.165942+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'150","last_scrub_stamp":"2024-03-09T15:36:14.165814+0000","last_deep_scrub":"261'150","last_deep_scrub_stamp":"2024-03-09T15:36:14.165814+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.165814+0000","objects_scrubbed":3,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:29:26.186154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":6443,"num_write":604,"num_write_kb":33049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"261'274","reported_seq":940,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:16.011571+0000","last_change":"2024-03-09T14:28:34.314501+0000","last_active":"2024-03-09T15:35:16.011571+0000","last_peered":"2024-03-09T15:35:16.011571+0000","last_clean":"2024-03-09T15:35:16.011571+0000","last_became_active":"2024-03-09T14:28:34.310343+0000","last_became_peered":"2024-03-09T14:28:34.310343+0000","last_unstale":"2024-03-09T15:35:16.011571+0000","last_undegraded":"2024-03-09T15:35:16.011571+0000","last_fullsized":"2024-03-09T15:35:16.011571+0000","mapping_epoch":22,"log_start":"247'200","ondisk_log_start":"247'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":200,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1205,"num_read_kb":1028,"num_write":418,"num_write_kb":253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"261'133","reported_seq":690,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.495607+0000","last_change":"2024-03-09T14:28:32.314711+0000","last_active":"2024-03-09T15:36:06.495607+0000","last_peered":"2024-03-09T15:36:06.495607+0000","last_clean":"2024-03-09T15:36:06.495607+0000","last_became_active":"2024-03-09T14:28:32.314258+0000","last_became_peered":"2024-03-09T14:28:32.314258+0000","last_unstale":"2024-03-09T15:36:06.495607+0000","last_undegraded":"2024-03-09T15:36:06.495607+0000","last_fullsized":"2024-03-09T15:36:06.495607+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":169,"num_read_kb":17581,"num_write":466,"num_write_kb":22929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3e","version":"261'44","reported_seq":619,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.733708+0000","last_change":"2024-03-09T15:36:29.733770+0000","last_active":"2024-03-09T15:36:29.733708+0000","last_peered":"2024-03-09T15:36:29.733708+0000","last_clean":"2024-03-09T15:36:29.733708+0000","last_became_active":"2024-03-09T14:28:34.322206+0000","last_became_peered":"2024-03-09T14:28:34.322206+0000","last_unstale":"2024-03-09T15:36:29.733708+0000","last_undegraded":"2024-03-09T15:36:29.733708+0000","last_fullsized":"2024-03-09T15:36:29.733708+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:29.733482+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:29.733482+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.733482+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:58.948789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":201,"num_read_kb":171,"num_write":60,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"261'150","reported_seq":697,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.522393+0000","last_change":"2024-03-09T14:28:32.308873+0000","last_active":"2024-03-09T15:36:06.522393+0000","last_peered":"2024-03-09T15:36:06.522393+0000","last_clean":"2024-03-09T15:36:06.522393+0000","last_became_active":"2024-03-09T14:28:32.308522+0000","last_became_peered":"2024-03-09T14:28:32.308522+0000","last_unstale":"2024-03-09T15:36:06.522393+0000","last_undegraded":"2024-03-09T15:36:06.522393+0000","last_fullsized":"2024-03-09T15:36:06.522393+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525396,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":175,"num_read_kb":14130,"num_write":546,"num_write_kb":36844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"261'52","reported_seq":604,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:15.997010+0000","last_change":"2024-03-09T14:28:34.321740+0000","last_active":"2024-03-09T15:35:15.997010+0000","last_peered":"2024-03-09T15:35:15.997010+0000","last_clean":"2024-03-09T15:35:15.997010+0000","last_became_active":"2024-03-09T14:28:34.320646+0000","last_became_peered":"2024-03-09T14:28:34.320646+0000","last_unstale":"2024-03-09T15:35:15.997010+0000","last_undegraded":"2024-03-09T15:35:15.997010+0000","last_fullsized":"2024-03-09T15:35:15.997010+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":174,"num_write":79,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"261'157","reported_seq":743,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.649512+0000","last_change":"2024-03-09T14:28:34.307366+0000","last_active":"2024-03-09T15:36:12.649512+0000","last_peered":"2024-03-09T15:36:12.649512+0000","last_clean":"2024-03-09T15:36:12.649512+0000","last_became_active":"2024-03-09T14:28:34.306592+0000","last_became_peered":"2024-03-09T14:28:34.306592+0000","last_unstale":"2024-03-09T15:36:12.649512+0000","last_undegraded":"2024-03-09T15:36:12.649512+0000","last_fullsized":"2024-03-09T15:36:12.649512+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":599,"num_read_kb":473,"num_write":232,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"261'165","reported_seq":726,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.220156+0000","last_change":"2024-03-09T15:36:16.220340+0000","last_active":"2024-03-09T15:36:16.220156+0000","last_peered":"2024-03-09T15:36:16.220156+0000","last_clean":"2024-03-09T15:36:16.220156+0000","last_became_active":"2024-03-09T14:29:06.438523+0000","last_became_peered":"2024-03-09T14:29:06.438523+0000","last_unstale":"2024-03-09T15:36:16.220156+0000","last_undegraded":"2024-03-09T15:36:16.220156+0000","last_fullsized":"2024-03-09T15:36:16.220156+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:16.219968+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:16.219968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.219968+0000","objects_scrubbed":4,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:42:29.990282+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728801,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":293,"num_read_kb":1034,"num_write":597,"num_write_kb":27895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.31","version":"261'88","reported_seq":644,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.102086+0000","last_change":"2024-03-09T14:28:34.322326+0000","last_active":"2024-03-09T15:36:14.102086+0000","last_peered":"2024-03-09T15:36:14.102086+0000","last_clean":"2024-03-09T15:36:14.102086+0000","last_became_active":"2024-03-09T14:28:34.321352+0000","last_became_peered":"2024-03-09T14:28:34.321352+0000","last_unstale":"2024-03-09T15:36:14.102086+0000","last_undegraded":"2024-03-09T15:36:14.102086+0000","last_fullsized":"2024-03-09T15:36:14.102086+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":342,"num_read_kb":270,"num_write":126,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"261'131","reported_seq":674,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.612948+0000","last_change":"2024-03-09T14:28:32.337035+0000","last_active":"2024-03-09T15:36:13.612948+0000","last_peered":"2024-03-09T15:36:13.612948+0000","last_clean":"2024-03-09T15:36:13.612948+0000","last_became_active":"2024-03-09T14:28:32.334208+0000","last_became_peered":"2024-03-09T14:28:32.334208+0000","last_unstale":"2024-03-09T15:36:13.612948+0000","last_undegraded":"2024-03-09T15:36:13.612948+0000","last_fullsized":"2024-03-09T15:36:13.612948+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":100,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262185,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":143,"num_read_kb":128,"num_write":408,"num_write_kb":22112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"261'62","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.124373+0000","last_change":"2024-03-09T14:28:34.299187+0000","last_active":"2024-03-09T15:36:13.124373+0000","last_peered":"2024-03-09T15:36:13.124373+0000","last_clean":"2024-03-09T15:36:13.124373+0000","last_became_active":"2024-03-09T14:28:34.298428+0000","last_became_peered":"2024-03-09T14:28:34.298428+0000","last_unstale":"2024-03-09T15:36:13.124373+0000","last_undegraded":"2024-03-09T15:36:13.124373+0000","last_fullsized":"2024-03-09T15:36:13.124373+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":202,"num_write":92,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"261'114","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.612834+0000","last_change":"2024-03-09T14:28:32.333004+0000","last_active":"2024-03-09T15:36:13.612834+0000","last_peered":"2024-03-09T15:36:13.612834+0000","last_clean":"2024-03-09T15:36:13.612834+0000","last_became_active":"2024-03-09T14:28:32.318963+0000","last_became_peered":"2024-03-09T14:28:32.318963+0000","last_unstale":"2024-03-09T15:36:13.612834+0000","last_undegraded":"2024-03-09T15:36:13.612834+0000","last_fullsized":"2024-03-09T15:36:13.612834+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":100,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170436,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":164,"num_read_kb":1700,"num_write":435,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.33","version":"261'102","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.709272+0000","last_change":"2024-03-09T14:28:34.320630+0000","last_active":"2024-03-09T15:36:11.709272+0000","last_peered":"2024-03-09T15:36:11.709272+0000","last_clean":"2024-03-09T15:36:11.709272+0000","last_became_active":"2024-03-09T14:28:34.320331+0000","last_became_peered":"2024-03-09T14:28:34.320331+0000","last_unstale":"2024-03-09T15:36:11.709272+0000","last_undegraded":"2024-03-09T15:36:11.709272+0000","last_fullsized":"2024-03-09T15:36:11.709272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":320,"num_write":144,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"261'130","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.649099+0000","last_change":"2024-03-09T14:28:32.338458+0000","last_active":"2024-03-09T15:36:12.649099+0000","last_peered":"2024-03-09T15:36:12.649099+0000","last_clean":"2024-03-09T15:36:12.649099+0000","last_became_active":"2024-03-09T14:28:32.337934+0000","last_became_peered":"2024-03-09T14:28:32.337934+0000","last_unstale":"2024-03-09T15:36:12.649099+0000","last_undegraded":"2024-03-09T15:36:12.649099+0000","last_fullsized":"2024-03-09T15:36:12.649099+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263250,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":163,"num_read_kb":12763,"num_write":465,"num_write_kb":21372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"261'69","reported_seq":607,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.709290+0000","last_change":"2024-03-09T14:28:34.311072+0000","last_active":"2024-03-09T15:36:11.709290+0000","last_peered":"2024-03-09T15:36:11.709290+0000","last_clean":"2024-03-09T15:36:11.709290+0000","last_became_active":"2024-03-09T14:28:34.310703+0000","last_became_peered":"2024-03-09T14:28:34.310703+0000","last_unstale":"2024-03-09T15:36:11.709290+0000","last_undegraded":"2024-03-09T15:36:11.709290+0000","last_fullsized":"2024-03-09T15:36:11.709290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":272,"num_read_kb":220,"num_write":109,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"261'157","reported_seq":725,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.185814+0000","last_change":"2024-03-09T14:28:32.337031+0000","last_active":"2024-03-09T15:36:12.185814+0000","last_peered":"2024-03-09T15:36:12.185814+0000","last_clean":"2024-03-09T15:36:12.185814+0000","last_became_active":"2024-03-09T14:28:32.336636+0000","last_became_peered":"2024-03-09T14:28:32.336636+0000","last_unstale":"2024-03-09T15:36:12.185814+0000","last_undegraded":"2024-03-09T15:36:12.185814+0000","last_fullsized":"2024-03-09T15:36:12.185814+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262241,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":231,"num_read_kb":12490,"num_write":578,"num_write_kb":28462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"261'87","reported_seq":639,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.649448+0000","last_change":"2024-03-09T14:28:34.316215+0000","last_active":"2024-03-09T15:36:12.649448+0000","last_peered":"2024-03-09T15:36:12.649448+0000","last_clean":"2024-03-09T15:36:12.649448+0000","last_became_active":"2024-03-09T14:28:34.314665+0000","last_became_peered":"2024-03-09T14:28:34.314665+0000","last_unstale":"2024-03-09T15:36:12.649448+0000","last_undegraded":"2024-03-09T15:36:12.649448+0000","last_fullsized":"2024-03-09T15:36:12.649448+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":335,"num_read_kb":265,"num_write":124,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"261'120","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739013+0000","last_change":"2024-03-09T14:28:32.298846+0000","last_active":"2024-03-09T15:36:10.739013+0000","last_peered":"2024-03-09T15:36:10.739013+0000","last_clean":"2024-03-09T15:36:10.739013+0000","last_became_active":"2024-03-09T14:28:32.298428+0000","last_became_peered":"2024-03-09T14:28:32.298428+0000","last_unstale":"2024-03-09T15:36:10.739013+0000","last_undegraded":"2024-03-09T15:36:10.739013+0000","last_fullsized":"2024-03-09T15:36:10.739013+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":191,"num_read_kb":31150,"num_write":464,"num_write_kb":37586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"261'76","reported_seq":591,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.170281+0000","last_change":"2024-03-09T15:36:18.170281+0000","last_active":"2024-03-09T15:36:18.170281+0000","last_peered":"2024-03-09T15:36:18.170281+0000","last_clean":"2024-03-09T15:36:18.170281+0000","last_became_active":"2024-03-09T14:28:42.403553+0000","last_became_peered":"2024-03-09T14:28:42.403553+0000","last_unstale":"2024-03-09T15:36:18.170281+0000","last_undegraded":"2024-03-09T15:36:18.170281+0000","last_fullsized":"2024-03-09T15:36:18.170281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'76","last_scrub_stamp":"2024-03-09T15:36:18.170018+0000","last_deep_scrub":"261'76","last_deep_scrub_stamp":"2024-03-09T15:36:18.170018+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.170018+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:55:35.040311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":84,"num_write":184,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"261'98","reported_seq":674,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.480604+0000","last_change":"2024-03-09T15:36:20.480604+0000","last_active":"2024-03-09T15:36:20.480604+0000","last_peered":"2024-03-09T15:36:20.480604+0000","last_clean":"2024-03-09T15:36:20.480604+0000","last_became_active":"2024-03-09T14:29:06.439359+0000","last_became_peered":"2024-03-09T14:29:06.439359+0000","last_unstale":"2024-03-09T15:36:20.480604+0000","last_undegraded":"2024-03-09T15:36:20.480604+0000","last_fullsized":"2024-03-09T15:36:20.480604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'98","last_scrub_stamp":"2024-03-09T15:36:20.480339+0000","last_deep_scrub":"261'98","last_deep_scrub_stamp":"2024-03-09T15:36:20.480339+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.480339+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:32:16.216352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":413,"num_read_kb":333,"num_write":149,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"261'165","reported_seq":788,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101468+0000","last_change":"2024-03-09T14:28:32.330235+0000","last_active":"2024-03-09T15:36:14.101468+0000","last_peered":"2024-03-09T15:36:14.101468+0000","last_clean":"2024-03-09T15:36:14.101468+0000","last_became_active":"2024-03-09T14:28:32.329457+0000","last_became_peered":"2024-03-09T14:28:32.329457+0000","last_unstale":"2024-03-09T15:36:14.101468+0000","last_undegraded":"2024-03-09T15:36:14.101468+0000","last_fullsized":"2024-03-09T15:36:14.101468+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506100,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":263,"num_read_kb":141328,"num_write":603,"num_write_kb":37022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.124506+0000","last_change":"2024-03-09T14:28:36.321332+0000","last_active":"2024-03-09T15:36:13.124506+0000","last_peered":"2024-03-09T15:36:13.124506+0000","last_clean":"2024-03-09T15:36:13.124506+0000","last_became_active":"2024-03-09T14:28:36.320603+0000","last_became_peered":"2024-03-09T14:28:36.320603+0000","last_unstale":"2024-03-09T15:36:13.124506+0000","last_undegraded":"2024-03-09T15:36:13.124506+0000","last_fullsized":"2024-03-09T15:36:13.124506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"261'229","reported_seq":841,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739506+0000","last_change":"2024-03-09T14:28:38.327507+0000","last_active":"2024-03-09T15:36:10.739506+0000","last_peered":"2024-03-09T15:36:10.739506+0000","last_clean":"2024-03-09T15:36:10.739506+0000","last_became_active":"2024-03-09T14:28:38.326826+0000","last_became_peered":"2024-03-09T14:28:38.326826+0000","last_unstale":"2024-03-09T15:36:10.739506+0000","last_undegraded":"2024-03-09T15:36:10.739506+0000","last_fullsized":"2024-03-09T15:36:10.739506+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":200,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11101,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":365,"num_read_kb":621,"num_write":250,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.102509+0000","last_change":"2024-03-09T14:28:40.359830+0000","last_active":"2024-03-09T15:36:14.102509+0000","last_peered":"2024-03-09T15:36:14.102509+0000","last_clean":"2024-03-09T15:36:14.102509+0000","last_became_active":"2024-03-09T14:28:40.358954+0000","last_became_peered":"2024-03-09T14:28:40.358954+0000","last_unstale":"2024-03-09T15:36:14.102509+0000","last_undegraded":"2024-03-09T15:36:14.102509+0000","last_fullsized":"2024-03-09T15:36:14.102509+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"261'772","reported_seq":2394,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.694618+0000","last_change":"2024-03-09T15:36:20.694693+0000","last_active":"2024-03-09T15:36:20.694618+0000","last_peered":"2024-03-09T15:36:20.694618+0000","last_clean":"2024-03-09T15:36:20.694618+0000","last_became_active":"2024-03-09T14:28:42.394691+0000","last_became_peered":"2024-03-09T14:28:42.394691+0000","last_unstale":"2024-03-09T15:36:20.694618+0000","last_undegraded":"2024-03-09T15:36:20.694618+0000","last_fullsized":"2024-03-09T15:36:20.694618+0000","mapping_epoch":30,"log_start":"261'700","ondisk_log_start":"261'700","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'772","last_scrub_stamp":"2024-03-09T15:36:20.694425+0000","last_deep_scrub":"261'772","last_deep_scrub_stamp":"2024-03-09T15:36:20.694425+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.694425+0000","objects_scrubbed":3,"log_size":72,"log_dups_size":700,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:30:23.952683+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":612,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2611,"num_read_kb":2601,"num_write":1560,"num_write_kb":569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":376,"num_omap_keys":2,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"261'40","reported_seq":580,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.124275+0000","last_change":"2024-03-09T14:28:34.326083+0000","last_active":"2024-03-09T15:36:13.124275+0000","last_peered":"2024-03-09T15:36:13.124275+0000","last_clean":"2024-03-09T15:36:13.124275+0000","last_became_active":"2024-03-09T14:28:34.324569+0000","last_became_peered":"2024-03-09T14:28:34.324569+0000","last_unstale":"2024-03-09T15:36:13.124275+0000","last_undegraded":"2024-03-09T15:36:13.124275+0000","last_fullsized":"2024-03-09T15:36:13.124275+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":168,"num_read_kb":136,"num_write":58,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"261'169","reported_seq":741,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.738625+0000","last_change":"2024-03-09T14:28:32.310039+0000","last_active":"2024-03-09T15:36:10.738625+0000","last_peered":"2024-03-09T15:36:10.738625+0000","last_clean":"2024-03-09T15:36:10.738625+0000","last_became_active":"2024-03-09T14:28:32.309536+0000","last_became_peered":"2024-03-09T14:28:32.309536+0000","last_unstale":"2024-03-09T15:36:10.738625+0000","last_undegraded":"2024-03-09T15:36:10.738625+0000","last_fullsized":"2024-03-09T15:36:10.738625+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":100,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":29533,"num_write":652,"num_write_kb":30801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.186114+0000","last_change":"2024-03-09T14:28:36.320450+0000","last_active":"2024-03-09T15:36:12.186114+0000","last_peered":"2024-03-09T15:36:12.186114+0000","last_clean":"2024-03-09T15:36:12.186114+0000","last_became_active":"2024-03-09T14:28:36.320079+0000","last_became_peered":"2024-03-09T14:28:36.320079+0000","last_unstale":"2024-03-09T15:36:12.186114+0000","last_undegraded":"2024-03-09T15:36:12.186114+0000","last_fullsized":"2024-03-09T15:36:12.186114+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"261'90","reported_seq":636,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.122254+0000","last_change":"2024-03-09T15:36:27.122328+0000","last_active":"2024-03-09T15:36:27.122254+0000","last_peered":"2024-03-09T15:36:27.122254+0000","last_clean":"2024-03-09T15:36:27.122254+0000","last_became_active":"2024-03-09T14:28:38.340896+0000","last_became_peered":"2024-03-09T14:28:38.340896+0000","last_unstale":"2024-03-09T15:36:27.122254+0000","last_undegraded":"2024-03-09T15:36:27.122254+0000","last_fullsized":"2024-03-09T15:36:27.122254+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'90","last_scrub_stamp":"2024-03-09T15:36:27.122054+0000","last_deep_scrub":"261'90","last_deep_scrub_stamp":"2024-03-09T15:36:27.122054+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.122054+0000","objects_scrubbed":3,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:23:13.339583+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":101,"num_write":68,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.119178+0000","last_change":"2024-03-09T15:36:23.119232+0000","last_active":"2024-03-09T15:36:23.119178+0000","last_peered":"2024-03-09T15:36:23.119178+0000","last_clean":"2024-03-09T15:36:23.119178+0000","last_became_active":"2024-03-09T14:28:40.370788+0000","last_became_peered":"2024-03-09T14:28:40.370788+0000","last_unstale":"2024-03-09T15:36:23.119178+0000","last_undegraded":"2024-03-09T15:36:23.119178+0000","last_fullsized":"2024-03-09T15:36:23.119178+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:23.118978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:23.118978+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.118978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:04:36.080050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"261'64","reported_seq":576,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.282044+0000","last_change":"2024-03-09T15:36:21.282129+0000","last_active":"2024-03-09T15:36:21.282044+0000","last_peered":"2024-03-09T15:36:21.282044+0000","last_clean":"2024-03-09T15:36:21.282044+0000","last_became_active":"2024-03-09T14:28:42.408738+0000","last_became_peered":"2024-03-09T14:28:42.408738+0000","last_unstale":"2024-03-09T15:36:21.282044+0000","last_undegraded":"2024-03-09T15:36:21.282044+0000","last_fullsized":"2024-03-09T15:36:21.282044+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'64","last_scrub_stamp":"2024-03-09T15:36:21.281839+0000","last_deep_scrub":"261'64","last_deep_scrub_stamp":"2024-03-09T15:36:21.281839+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.281839+0000","objects_scrubbed":5,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:53:36.132862+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":681,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":76,"num_read_kb":75,"num_write":140,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"261'157","reported_seq":747,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739077+0000","last_change":"2024-03-09T14:28:34.305220+0000","last_active":"2024-03-09T15:36:10.739077+0000","last_peered":"2024-03-09T15:36:10.739077+0000","last_clean":"2024-03-09T15:36:10.739077+0000","last_became_active":"2024-03-09T14:28:34.304892+0000","last_became_peered":"2024-03-09T14:28:34.304892+0000","last_unstale":"2024-03-09T15:36:10.739077+0000","last_undegraded":"2024-03-09T15:36:10.739077+0000","last_fullsized":"2024-03-09T15:36:10.739077+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":622,"num_read_kb":492,"num_write":239,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"261'161","reported_seq":863,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101697+0000","last_change":"2024-03-09T14:28:32.319206+0000","last_active":"2024-03-09T15:36:14.101697+0000","last_peered":"2024-03-09T15:36:14.101697+0000","last_clean":"2024-03-09T15:36:14.101697+0000","last_became_active":"2024-03-09T14:28:32.318428+0000","last_became_peered":"2024-03-09T14:28:32.318428+0000","last_unstale":"2024-03-09T15:36:14.101697+0000","last_undegraded":"2024-03-09T15:36:14.101697+0000","last_fullsized":"2024-03-09T15:36:14.101697+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505054,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":544,"num_read_kb":113871,"num_write":619,"num_write_kb":46622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":498,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.186163+0000","last_change":"2024-03-09T14:28:36.320826+0000","last_active":"2024-03-09T15:36:12.186163+0000","last_peered":"2024-03-09T15:36:12.186163+0000","last_clean":"2024-03-09T15:36:12.186163+0000","last_became_active":"2024-03-09T14:28:36.320095+0000","last_became_peered":"2024-03-09T14:28:36.320095+0000","last_unstale":"2024-03-09T15:36:12.186163+0000","last_undegraded":"2024-03-09T15:36:12.186163+0000","last_fullsized":"2024-03-09T15:36:12.186163+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"261'108","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.164895+0000","last_change":"2024-03-09T15:36:17.165000+0000","last_active":"2024-03-09T15:36:17.164895+0000","last_peered":"2024-03-09T15:36:17.164895+0000","last_clean":"2024-03-09T15:36:17.164895+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:36:17.164895+0000","last_undegraded":"2024-03-09T15:36:17.164895+0000","last_fullsized":"2024-03-09T15:36:17.164895+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'108","last_scrub_stamp":"2024-03-09T15:36:17.164692+0000","last_deep_scrub":"261'108","last_deep_scrub_stamp":"2024-03-09T15:36:17.164692+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.164692+0000","objects_scrubbed":5,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:41:19.308226+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"261'505","reported_seq":2293,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.103072+0000","last_change":"2024-03-09T14:28:40.373218+0000","last_active":"2024-03-09T15:36:14.103072+0000","last_peered":"2024-03-09T15:36:14.103072+0000","last_clean":"2024-03-09T15:36:14.103072+0000","last_became_active":"2024-03-09T14:28:40.371507+0000","last_became_peered":"2024-03-09T14:28:40.371507+0000","last_unstale":"2024-03-09T15:36:14.103072+0000","last_undegraded":"2024-03-09T15:36:14.103072+0000","last_fullsized":"2024-03-09T15:36:14.103072+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":400,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"261'138","reported_seq":953,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.158161+0000","last_change":"2024-03-09T15:36:19.158227+0000","last_active":"2024-03-09T15:36:19.158161+0000","last_peered":"2024-03-09T15:36:19.158161+0000","last_clean":"2024-03-09T15:36:19.158161+0000","last_became_active":"2024-03-09T14:28:42.398498+0000","last_became_peered":"2024-03-09T14:28:42.398498+0000","last_unstale":"2024-03-09T15:36:19.158161+0000","last_undegraded":"2024-03-09T15:36:19.158161+0000","last_fullsized":"2024-03-09T15:36:19.158161+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'138","last_scrub_stamp":"2024-03-09T15:36:19.157968+0000","last_deep_scrub":"261'138","last_deep_scrub_stamp":"2024-03-09T15:36:19.157968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.157968+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":100,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:49:21.053160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":498,"num_read_kb":498,"num_write":277,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":234,"num_omap_keys":2,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"261'53","reported_seq":594,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.185931+0000","last_change":"2024-03-09T14:28:34.330792+0000","last_active":"2024-03-09T15:36:12.185931+0000","last_peered":"2024-03-09T15:36:12.185931+0000","last_clean":"2024-03-09T15:36:12.185931+0000","last_became_active":"2024-03-09T14:28:34.330347+0000","last_became_peered":"2024-03-09T14:28:34.330347+0000","last_unstale":"2024-03-09T15:36:12.185931+0000","last_undegraded":"2024-03-09T15:36:12.185931+0000","last_fullsized":"2024-03-09T15:36:12.185931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":212,"num_read_kb":171,"num_write":77,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"261'166","reported_seq":740,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.612752+0000","last_change":"2024-03-09T14:28:32.332979+0000","last_active":"2024-03-09T15:36:13.612752+0000","last_peered":"2024-03-09T15:36:13.612752+0000","last_clean":"2024-03-09T15:36:13.612752+0000","last_became_active":"2024-03-09T14:28:32.319819+0000","last_became_peered":"2024-03-09T14:28:32.319819+0000","last_unstale":"2024-03-09T15:36:13.612752+0000","last_undegraded":"2024-03-09T15:36:13.612752+0000","last_fullsized":"2024-03-09T15:36:13.612752+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":100,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5244508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":231,"num_read_kb":15567,"num_write":579,"num_write_kb":34195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.124600+0000","last_change":"2024-03-09T14:28:36.311535+0000","last_active":"2024-03-09T15:36:13.124600+0000","last_peered":"2024-03-09T15:36:13.124600+0000","last_clean":"2024-03-09T15:36:13.124600+0000","last_became_active":"2024-03-09T14:28:36.310231+0000","last_became_peered":"2024-03-09T14:28:36.310231+0000","last_unstale":"2024-03-09T15:36:13.124600+0000","last_undegraded":"2024-03-09T15:36:13.124600+0000","last_fullsized":"2024-03-09T15:36:13.124600+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"261'170","reported_seq":755,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.102850+0000","last_change":"2024-03-09T14:28:38.599078+0000","last_active":"2024-03-09T15:36:14.102850+0000","last_peered":"2024-03-09T15:36:14.102850+0000","last_clean":"2024-03-09T15:36:14.102850+0000","last_became_active":"2024-03-09T14:28:38.598308+0000","last_became_peered":"2024-03-09T14:28:38.598308+0000","last_unstale":"2024-03-09T15:36:14.102850+0000","last_undegraded":"2024-03-09T15:36:14.102850+0000","last_fullsized":"2024-03-09T15:36:14.102850+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":100,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8724,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":532,"num_write":190,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"261'513","reported_seq":2500,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.540756+0000","last_change":"2024-03-09T14:28:40.388710+0000","last_active":"2024-03-09T15:36:06.540756+0000","last_peered":"2024-03-09T15:36:06.540756+0000","last_clean":"2024-03-09T15:36:06.540756+0000","last_became_active":"2024-03-09T14:28:40.387990+0000","last_became_peered":"2024-03-09T14:28:40.387990+0000","last_unstale":"2024-03-09T15:36:06.540756+0000","last_undegraded":"2024-03-09T15:36:06.540756+0000","last_fullsized":"2024-03-09T15:36:06.540756+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":500,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"261'101","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.740079+0000","last_change":"2024-03-09T15:36:28.740141+0000","last_active":"2024-03-09T15:36:28.740079+0000","last_peered":"2024-03-09T15:36:28.740079+0000","last_clean":"2024-03-09T15:36:28.740079+0000","last_became_active":"2024-03-09T14:28:34.322731+0000","last_became_peered":"2024-03-09T14:28:34.322731+0000","last_unstale":"2024-03-09T15:36:28.740079+0000","last_undegraded":"2024-03-09T15:36:28.740079+0000","last_fullsized":"2024-03-09T15:36:28.740079+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'101","last_scrub_stamp":"2024-03-09T15:36:28.739877+0000","last_deep_scrub":"261'101","last_deep_scrub_stamp":"2024-03-09T15:36:28.739877+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.739877+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:43:42.753565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":330,"num_write":141,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"261'123","reported_seq":673,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101424+0000","last_change":"2024-03-09T14:28:32.329755+0000","last_active":"2024-03-09T15:36:14.101424+0000","last_peered":"2024-03-09T15:36:14.101424+0000","last_clean":"2024-03-09T15:36:14.101424+0000","last_became_active":"2024-03-09T14:28:32.329148+0000","last_became_peered":"2024-03-09T14:28:32.329148+0000","last_unstale":"2024-03-09T15:36:14.101424+0000","last_undegraded":"2024-03-09T15:36:14.101424+0000","last_fullsized":"2024-03-09T15:36:14.101424+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":100,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7947490,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":1766,"num_write":513,"num_write_kb":18289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:46.235429+0000","last_change":"2024-03-09T14:28:36.314780+0000","last_active":"2024-03-09T15:35:46.235429+0000","last_peered":"2024-03-09T15:35:46.235429+0000","last_clean":"2024-03-09T15:35:46.235429+0000","last_became_active":"2024-03-09T14:28:36.313904+0000","last_became_peered":"2024-03-09T14:28:36.313904+0000","last_unstale":"2024-03-09T15:35:46.235429+0000","last_undegraded":"2024-03-09T15:35:46.235429+0000","last_fullsized":"2024-03-09T15:35:46.235429+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"261'276","reported_seq":920,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.746370+0000","last_change":"2024-03-09T15:36:24.746541+0000","last_active":"2024-03-09T15:36:24.746370+0000","last_peered":"2024-03-09T15:36:24.746370+0000","last_clean":"2024-03-09T15:36:24.746370+0000","last_became_active":"2024-03-09T14:28:38.597247+0000","last_became_peered":"2024-03-09T14:28:38.597247+0000","last_unstale":"2024-03-09T15:36:24.746370+0000","last_undegraded":"2024-03-09T15:36:24.746370+0000","last_fullsized":"2024-03-09T15:36:24.746370+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'276","last_scrub_stamp":"2024-03-09T15:36:24.746130+0000","last_deep_scrub":"261'276","last_deep_scrub_stamp":"2024-03-09T15:36:24.746130+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.746130+0000","objects_scrubbed":9,"log_size":76,"log_dups_size":200,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:53:37.971114+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9398,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":421,"num_read_kb":664,"num_write":284,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.994762+0000","last_change":"2024-03-09T14:28:40.360962+0000","last_active":"2024-03-09T15:36:10.994762+0000","last_peered":"2024-03-09T15:36:10.994762+0000","last_clean":"2024-03-09T15:36:10.994762+0000","last_became_active":"2024-03-09T14:28:40.359906+0000","last_became_peered":"2024-03-09T14:28:40.359906+0000","last_unstale":"2024-03-09T15:36:10.994762+0000","last_undegraded":"2024-03-09T15:36:10.994762+0000","last_fullsized":"2024-03-09T15:36:10.994762+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"261'522","reported_seq":1369,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.664753+0000","last_change":"2024-03-09T15:36:16.664829+0000","last_active":"2024-03-09T15:36:16.664753+0000","last_peered":"2024-03-09T15:36:16.664753+0000","last_clean":"2024-03-09T15:36:16.664753+0000","last_became_active":"2024-03-09T14:28:42.402739+0000","last_became_peered":"2024-03-09T14:28:42.402739+0000","last_unstale":"2024-03-09T15:36:16.664753+0000","last_undegraded":"2024-03-09T15:36:16.664753+0000","last_fullsized":"2024-03-09T15:36:16.664753+0000","mapping_epoch":30,"log_start":"261'500","ondisk_log_start":"261'500","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'522","last_scrub_stamp":"2024-03-09T15:36:16.664560+0000","last_deep_scrub":"261'522","last_deep_scrub_stamp":"2024-03-09T15:36:16.664560+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.664560+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":500,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:05:16.194957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1336,"num_read_kb":1311,"num_write":1083,"num_write_kb":376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":381,"num_omap_keys":2,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"261'92","reported_seq":663,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:46.017837+0000","last_change":"2024-03-09T14:28:34.302153+0000","last_active":"2024-03-09T15:35:46.017837+0000","last_peered":"2024-03-09T15:35:46.017837+0000","last_clean":"2024-03-09T15:35:46.017837+0000","last_became_active":"2024-03-09T14:28:34.300795+0000","last_became_peered":"2024-03-09T14:28:34.300795+0000","last_unstale":"2024-03-09T15:35:46.017837+0000","last_undegraded":"2024-03-09T15:35:46.017837+0000","last_fullsized":"2024-03-09T15:35:46.017837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":361,"num_read_kb":288,"num_write":150,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"261'142","reported_seq":716,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.452480+0000","last_change":"2024-03-09T14:28:32.318853+0000","last_active":"2024-03-09T15:36:06.452480+0000","last_peered":"2024-03-09T15:36:06.452480+0000","last_clean":"2024-03-09T15:36:06.452480+0000","last_became_active":"2024-03-09T14:28:32.318111+0000","last_became_peered":"2024-03-09T14:28:32.318111+0000","last_unstale":"2024-03-09T15:36:06.452480+0000","last_undegraded":"2024-03-09T15:36:06.452480+0000","last_fullsized":"2024-03-09T15:36:06.452480+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":100,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242921,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":6230,"num_write":597,"num_write_kb":19779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.197656+0000","last_change":"2024-03-09T15:36:27.197765+0000","last_active":"2024-03-09T15:36:27.197656+0000","last_peered":"2024-03-09T15:36:27.197656+0000","last_clean":"2024-03-09T15:36:27.197656+0000","last_became_active":"2024-03-09T14:28:36.301227+0000","last_became_peered":"2024-03-09T14:28:36.301227+0000","last_unstale":"2024-03-09T15:36:27.197656+0000","last_undegraded":"2024-03-09T15:36:27.197656+0000","last_fullsized":"2024-03-09T15:36:27.197656+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:27.197420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:27.197420+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.197420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:56:31.881994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"261'173","reported_seq":751,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.837083+0000","last_change":"2024-03-09T14:28:38.347678+0000","last_active":"2024-03-09T15:36:10.837083+0000","last_peered":"2024-03-09T15:36:10.837083+0000","last_clean":"2024-03-09T15:36:10.837083+0000","last_became_active":"2024-03-09T14:28:38.346837+0000","last_became_peered":"2024-03-09T14:28:38.346837+0000","last_unstale":"2024-03-09T15:36:10.837083+0000","last_undegraded":"2024-03-09T15:36:10.837083+0000","last_fullsized":"2024-03-09T15:36:10.837083+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":100,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16267,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":276,"num_read_kb":528,"num_write":188,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.485270+0000","last_change":"2024-03-09T15:36:30.485329+0000","last_active":"2024-03-09T15:36:30.485270+0000","last_peered":"2024-03-09T15:36:30.485270+0000","last_clean":"2024-03-09T15:36:30.485270+0000","last_became_active":"2024-03-09T14:28:40.389175+0000","last_became_peered":"2024-03-09T14:28:40.389175+0000","last_unstale":"2024-03-09T15:36:30.485270+0000","last_undegraded":"2024-03-09T15:36:30.485270+0000","last_fullsized":"2024-03-09T15:36:30.485270+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:30.485071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:30.485071+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.485071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:54:36.706597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"261'58","reported_seq":554,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:45.816898+0000","last_change":"2024-03-09T14:28:42.401990+0000","last_active":"2024-03-09T15:35:45.816898+0000","last_peered":"2024-03-09T15:35:45.816898+0000","last_clean":"2024-03-09T15:35:45.816898+0000","last_became_active":"2024-03-09T14:28:42.401214+0000","last_became_peered":"2024-03-09T14:28:42.401214+0000","last_unstale":"2024-03-09T15:35:45.816898+0000","last_undegraded":"2024-03-09T15:35:45.816898+0000","last_fullsized":"2024-03-09T15:35:45.816898+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":47,"num_write":140,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"261'78","reported_seq":635,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.820891+0000","last_change":"2024-03-09T14:28:34.317931+0000","last_active":"2024-03-09T15:36:05.820891+0000","last_peered":"2024-03-09T15:36:05.820891+0000","last_clean":"2024-03-09T15:36:05.820891+0000","last_became_active":"2024-03-09T14:28:34.316812+0000","last_became_peered":"2024-03-09T14:28:34.316812+0000","last_unstale":"2024-03-09T15:36:05.820891+0000","last_undegraded":"2024-03-09T15:36:05.820891+0000","last_fullsized":"2024-03-09T15:36:05.820891+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":320,"num_read_kb":256,"num_write":116,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"261'111","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.172728+0000","last_change":"2024-03-09T15:36:27.172803+0000","last_active":"2024-03-09T15:36:27.172728+0000","last_peered":"2024-03-09T15:36:27.172728+0000","last_clean":"2024-03-09T15:36:27.172728+0000","last_became_active":"2024-03-09T14:29:06.446333+0000","last_became_peered":"2024-03-09T14:29:06.446333+0000","last_unstale":"2024-03-09T15:36:27.172728+0000","last_undegraded":"2024-03-09T15:36:27.172728+0000","last_fullsized":"2024-03-09T15:36:27.172728+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'111","last_scrub_stamp":"2024-03-09T15:36:27.172540+0000","last_deep_scrub":"261'111","last_deep_scrub_stamp":"2024-03-09T15:36:27.172540+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.172540+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:12:44.004446+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":147,"num_read_kb":1025,"num_write":402,"num_write_kb":11167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":515,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.252418+0000","last_change":"2024-03-09T15:36:25.252530+0000","last_active":"2024-03-09T15:36:25.252418+0000","last_peered":"2024-03-09T15:36:25.252418+0000","last_clean":"2024-03-09T15:36:25.252418+0000","last_became_active":"2024-03-09T14:28:36.303695+0000","last_became_peered":"2024-03-09T14:28:36.303695+0000","last_unstale":"2024-03-09T15:36:25.252418+0000","last_undegraded":"2024-03-09T15:36:25.252418+0000","last_fullsized":"2024-03-09T15:36:25.252418+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.252128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.252128+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.252128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:47:06.758741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"261'220","reported_seq":829,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.463646+0000","last_change":"2024-03-09T14:28:38.340119+0000","last_active":"2024-03-09T15:36:06.463646+0000","last_peered":"2024-03-09T15:36:06.463646+0000","last_clean":"2024-03-09T15:36:06.463646+0000","last_became_active":"2024-03-09T14:28:38.339300+0000","last_became_peered":"2024-03-09T14:28:38.339300+0000","last_unstale":"2024-03-09T15:36:06.463646+0000","last_undegraded":"2024-03-09T15:36:06.463646+0000","last_fullsized":"2024-03-09T15:36:06.463646+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":200,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8477,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":353,"num_read_kb":624,"num_write":244,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"261'501","reported_seq":2511,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.541827+0000","last_change":"2024-03-09T14:28:40.368533+0000","last_active":"2024-03-09T15:36:06.541827+0000","last_peered":"2024-03-09T15:36:06.541827+0000","last_clean":"2024-03-09T15:36:06.541827+0000","last_became_active":"2024-03-09T14:28:40.367119+0000","last_became_peered":"2024-03-09T14:28:40.367119+0000","last_unstale":"2024-03-09T15:36:06.541827+0000","last_undegraded":"2024-03-09T15:36:06.541827+0000","last_fullsized":"2024-03-09T15:36:06.541827+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":400,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"261'82","reported_seq":585,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.245845+0000","last_change":"2024-03-09T15:36:28.245919+0000","last_active":"2024-03-09T15:36:28.245845+0000","last_peered":"2024-03-09T15:36:28.245845+0000","last_clean":"2024-03-09T15:36:28.245845+0000","last_became_active":"2024-03-09T14:28:42.396389+0000","last_became_peered":"2024-03-09T14:28:42.396389+0000","last_unstale":"2024-03-09T15:36:28.245845+0000","last_undegraded":"2024-03-09T15:36:28.245845+0000","last_fullsized":"2024-03-09T15:36:28.245845+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'82","last_scrub_stamp":"2024-03-09T15:36:28.245628+0000","last_deep_scrub":"261'82","last_deep_scrub_stamp":"2024-03-09T15:36:28.245628+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.245628+0000","objects_scrubbed":1,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:26:51.961711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":65,"num_write":200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"261'44","reported_seq":583,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.046758+0000","last_change":"2024-03-09T14:28:34.298294+0000","last_active":"2024-03-09T15:36:06.046758+0000","last_peered":"2024-03-09T15:36:06.046758+0000","last_clean":"2024-03-09T15:36:06.046758+0000","last_became_active":"2024-03-09T14:28:34.297080+0000","last_became_peered":"2024-03-09T14:28:34.297080+0000","last_unstale":"2024-03-09T15:36:06.046758+0000","last_undegraded":"2024-03-09T15:36:06.046758+0000","last_fullsized":"2024-03-09T15:36:06.046758+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":189,"num_read_kb":155,"num_write":63,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"261'152","reported_seq":698,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.200175+0000","last_change":"2024-03-09T15:36:20.200259+0000","last_active":"2024-03-09T15:36:20.200175+0000","last_peered":"2024-03-09T15:36:20.200175+0000","last_clean":"2024-03-09T15:36:20.200175+0000","last_became_active":"2024-03-09T14:29:06.445573+0000","last_became_peered":"2024-03-09T14:29:06.445573+0000","last_unstale":"2024-03-09T15:36:20.200175+0000","last_undegraded":"2024-03-09T15:36:20.200175+0000","last_fullsized":"2024-03-09T15:36:20.200175+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'152","last_scrub_stamp":"2024-03-09T15:36:20.199970+0000","last_deep_scrub":"261'152","last_deep_scrub_stamp":"2024-03-09T15:36:20.199970+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.199970+0000","objects_scrubbed":5,"log_size":52,"log_dups_size":100,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:59:48.356486+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":204,"num_read_kb":157,"num_write":458,"num_write_kb":32761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.186224+0000","last_change":"2024-03-09T15:36:19.186332+0000","last_active":"2024-03-09T15:36:19.186224+0000","last_peered":"2024-03-09T15:36:19.186224+0000","last_clean":"2024-03-09T15:36:19.186224+0000","last_became_active":"2024-03-09T14:28:36.322638+0000","last_became_peered":"2024-03-09T14:28:36.322638+0000","last_unstale":"2024-03-09T15:36:19.186224+0000","last_undegraded":"2024-03-09T15:36:19.186224+0000","last_fullsized":"2024-03-09T15:36:19.186224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:19.185963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:19.185963+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.185963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:52:57.100559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"261'318","reported_seq":962,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.499196+0000","last_change":"2024-03-09T14:28:38.329540+0000","last_active":"2024-03-09T15:36:06.499196+0000","last_peered":"2024-03-09T15:36:06.499196+0000","last_clean":"2024-03-09T15:36:06.499196+0000","last_became_active":"2024-03-09T14:28:38.328897+0000","last_became_peered":"2024-03-09T14:28:38.328897+0000","last_unstale":"2024-03-09T15:36:06.499196+0000","last_undegraded":"2024-03-09T15:36:06.499196+0000","last_fullsized":"2024-03-09T15:36:06.499196+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":300,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":543,"num_read_kb":1162,"num_write":388,"num_write_kb":277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.192323+0000","last_change":"2024-03-09T15:36:12.192350+0000","last_active":"2024-03-09T15:36:12.192323+0000","last_peered":"2024-03-09T15:36:12.192323+0000","last_clean":"2024-03-09T15:36:12.192323+0000","last_became_active":"2024-03-09T14:28:40.383758+0000","last_became_peered":"2024-03-09T14:28:40.383758+0000","last_unstale":"2024-03-09T15:36:12.192323+0000","last_undegraded":"2024-03-09T15:36:12.192323+0000","last_fullsized":"2024-03-09T15:36:12.192323+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:12.192210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:12.192210+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.192210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:18:21.362249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"261'56","reported_seq":576,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.525835+0000","last_change":"2024-03-09T15:36:19.525925+0000","last_active":"2024-03-09T15:36:19.525835+0000","last_peered":"2024-03-09T15:36:19.525835+0000","last_clean":"2024-03-09T15:36:19.525835+0000","last_became_active":"2024-03-09T14:28:42.401880+0000","last_became_peered":"2024-03-09T14:28:42.401880+0000","last_unstale":"2024-03-09T15:36:19.525835+0000","last_undegraded":"2024-03-09T15:36:19.525835+0000","last_fullsized":"2024-03-09T15:36:19.525835+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:19.525622+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:19.525622+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.525622+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:58:32.566894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":69,"num_write":120,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"261'79","reported_seq":640,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.022991+0000","last_change":"2024-03-09T14:28:34.316156+0000","last_active":"2024-03-09T15:36:06.022991+0000","last_peered":"2024-03-09T15:36:06.022991+0000","last_clean":"2024-03-09T15:36:06.022991+0000","last_became_active":"2024-03-09T14:28:34.315432+0000","last_became_peered":"2024-03-09T14:28:34.315432+0000","last_unstale":"2024-03-09T15:36:06.022991+0000","last_undegraded":"2024-03-09T15:36:06.022991+0000","last_fullsized":"2024-03-09T15:36:06.022991+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":315,"num_read_kb":251,"num_write":126,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"261'157","reported_seq":696,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.528298+0000","last_change":"2024-03-09T14:28:32.300326+0000","last_active":"2024-03-09T15:36:06.528298+0000","last_peered":"2024-03-09T15:36:06.528298+0000","last_clean":"2024-03-09T15:36:06.528298+0000","last_became_active":"2024-03-09T14:28:32.299897+0000","last_became_peered":"2024-03-09T14:28:32.299897+0000","last_unstale":"2024-03-09T15:36:06.528298+0000","last_undegraded":"2024-03-09T15:36:06.528298+0000","last_fullsized":"2024-03-09T15:36:06.528298+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506226,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":160,"num_read_kb":225,"num_write":553,"num_write_kb":26448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":515,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.168556+0000","last_change":"2024-03-09T15:36:21.168616+0000","last_active":"2024-03-09T15:36:21.168556+0000","last_peered":"2024-03-09T15:36:21.168556+0000","last_clean":"2024-03-09T15:36:21.168556+0000","last_became_active":"2024-03-09T14:28:36.320029+0000","last_became_peered":"2024-03-09T14:28:36.320029+0000","last_unstale":"2024-03-09T15:36:21.168556+0000","last_undegraded":"2024-03-09T15:36:21.168556+0000","last_fullsized":"2024-03-09T15:36:21.168556+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.168322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.168322+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.168322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:54:01.071807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"261'213","reported_seq":814,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.154985+0000","last_change":"2024-03-09T15:36:28.155065+0000","last_active":"2024-03-09T15:36:28.154985+0000","last_peered":"2024-03-09T15:36:28.154985+0000","last_clean":"2024-03-09T15:36:28.154985+0000","last_became_active":"2024-03-09T14:28:38.338658+0000","last_became_peered":"2024-03-09T14:28:38.338658+0000","last_unstale":"2024-03-09T15:36:28.154985+0000","last_undegraded":"2024-03-09T15:36:28.154985+0000","last_fullsized":"2024-03-09T15:36:28.154985+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'213","last_scrub_stamp":"2024-03-09T15:36:28.154798+0000","last_deep_scrub":"261'213","last_deep_scrub_stamp":"2024-03-09T15:36:28.154798+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.154798+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":200,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:48:17.369213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12662,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":352,"num_read_kb":613,"num_write":292,"num_write_kb":188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13964,"num_omap_keys":34,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.231293+0000","last_change":"2024-03-09T15:36:30.231399+0000","last_active":"2024-03-09T15:36:30.231293+0000","last_peered":"2024-03-09T15:36:30.231293+0000","last_clean":"2024-03-09T15:36:30.231293+0000","last_became_active":"2024-03-09T14:28:40.378973+0000","last_became_peered":"2024-03-09T14:28:40.378973+0000","last_unstale":"2024-03-09T15:36:30.231293+0000","last_undegraded":"2024-03-09T15:36:30.231293+0000","last_fullsized":"2024-03-09T15:36:30.231293+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:30.231002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:30.231002+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.231002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:08:44.672009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"261'44","reported_seq":546,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.449604+0000","last_change":"2024-03-09T15:36:23.449669+0000","last_active":"2024-03-09T15:36:23.449604+0000","last_peered":"2024-03-09T15:36:23.449604+0000","last_clean":"2024-03-09T15:36:23.449604+0000","last_became_active":"2024-03-09T14:28:42.397533+0000","last_became_peered":"2024-03-09T14:28:42.397533+0000","last_unstale":"2024-03-09T15:36:23.449604+0000","last_undegraded":"2024-03-09T15:36:23.449604+0000","last_fullsized":"2024-03-09T15:36:23.449604+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:23.449413+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:23.449413+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.449413+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:46:36.909599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":91,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"261'56","reported_seq":617,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.023086+0000","last_change":"2024-03-09T14:28:34.315224+0000","last_active":"2024-03-09T15:36:06.023086+0000","last_peered":"2024-03-09T15:36:06.023086+0000","last_clean":"2024-03-09T15:36:06.023086+0000","last_became_active":"2024-03-09T14:28:34.313475+0000","last_became_peered":"2024-03-09T14:28:34.313475+0000","last_unstale":"2024-03-09T15:36:06.023086+0000","last_undegraded":"2024-03-09T15:36:06.023086+0000","last_fullsized":"2024-03-09T15:36:06.023086+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":198,"num_write":78,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"261'129","reported_seq":682,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.495789+0000","last_change":"2024-03-09T14:28:32.319640+0000","last_active":"2024-03-09T15:36:06.495789+0000","last_peered":"2024-03-09T15:36:06.495789+0000","last_clean":"2024-03-09T15:36:06.495789+0000","last_became_active":"2024-03-09T14:28:32.316406+0000","last_became_peered":"2024-03-09T14:28:32.316406+0000","last_unstale":"2024-03-09T15:36:06.495789+0000","last_undegraded":"2024-03-09T15:36:06.495789+0000","last_fullsized":"2024-03-09T15:36:06.495789+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655835,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":162,"num_read_kb":15736,"num_write":466,"num_write_kb":13578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.511704+0000","last_change":"2024-03-09T15:36:27.511852+0000","last_active":"2024-03-09T15:36:27.511704+0000","last_peered":"2024-03-09T15:36:27.511704+0000","last_clean":"2024-03-09T15:36:27.511704+0000","last_became_active":"2024-03-09T14:28:36.319210+0000","last_became_peered":"2024-03-09T14:28:36.319210+0000","last_unstale":"2024-03-09T15:36:27.511704+0000","last_undegraded":"2024-03-09T15:36:27.511704+0000","last_fullsized":"2024-03-09T15:36:27.511704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:27.511458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:27.511458+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.511458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:07.969688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"261'223","reported_seq":837,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.196282+0000","last_change":"2024-03-09T15:36:15.196282+0000","last_active":"2024-03-09T15:36:15.196282+0000","last_peered":"2024-03-09T15:36:15.196282+0000","last_clean":"2024-03-09T15:36:15.196282+0000","last_became_active":"2024-03-09T14:28:38.350390+0000","last_became_peered":"2024-03-09T14:28:38.350390+0000","last_unstale":"2024-03-09T15:36:15.196282+0000","last_undegraded":"2024-03-09T15:36:15.196282+0000","last_fullsized":"2024-03-09T15:36:15.196282+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'223","last_scrub_stamp":"2024-03-09T15:36:15.195858+0000","last_deep_scrub":"261'223","last_deep_scrub_stamp":"2024-03-09T15:36:15.195858+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.195858+0000","objects_scrubbed":7,"log_size":23,"log_dups_size":200,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:47:25.524775+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9540,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":338,"num_read_kb":580,"num_write":230,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"261'474","reported_seq":2368,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.200778+0000","last_change":"2024-03-09T15:36:13.200827+0000","last_active":"2024-03-09T15:36:13.200778+0000","last_peered":"2024-03-09T15:36:13.200778+0000","last_clean":"2024-03-09T15:36:13.200778+0000","last_became_active":"2024-03-09T14:28:40.373360+0000","last_became_peered":"2024-03-09T14:28:40.373360+0000","last_unstale":"2024-03-09T15:36:13.200778+0000","last_undegraded":"2024-03-09T15:36:13.200778+0000","last_fullsized":"2024-03-09T15:36:13.200778+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'474","last_scrub_stamp":"2024-03-09T15:36:13.200687+0000","last_deep_scrub":"261'474","last_deep_scrub_stamp":"2024-03-09T15:36:13.200687+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.200687+0000","objects_scrubbed":1,"log_size":74,"log_dups_size":400,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:36:38.570841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"261'108","reported_seq":663,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.587316+0000","last_change":"2024-03-09T15:36:16.587396+0000","last_active":"2024-03-09T15:36:16.587316+0000","last_peered":"2024-03-09T15:36:16.587316+0000","last_clean":"2024-03-09T15:36:16.587316+0000","last_became_active":"2024-03-09T14:28:42.408233+0000","last_became_peered":"2024-03-09T14:28:42.408233+0000","last_unstale":"2024-03-09T15:36:16.587316+0000","last_undegraded":"2024-03-09T15:36:16.587316+0000","last_fullsized":"2024-03-09T15:36:16.587316+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'108","last_scrub_stamp":"2024-03-09T15:36:16.587096+0000","last_deep_scrub":"261'108","last_deep_scrub_stamp":"2024-03-09T15:36:16.587096+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.587096+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:31:09.055955+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":274,"num_read_kb":227,"num_write":264,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"261'72","reported_seq":622,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.023174+0000","last_change":"2024-03-09T14:28:34.310932+0000","last_active":"2024-03-09T15:36:06.023174+0000","last_peered":"2024-03-09T15:36:06.023174+0000","last_clean":"2024-03-09T15:36:06.023174+0000","last_became_active":"2024-03-09T14:28:34.310061+0000","last_became_peered":"2024-03-09T14:28:34.310061+0000","last_unstale":"2024-03-09T15:36:06.023174+0000","last_undegraded":"2024-03-09T15:36:06.023174+0000","last_fullsized":"2024-03-09T15:36:06.023174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":294,"num_read_kb":233,"num_write":114,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"261'166","reported_seq":774,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.526499+0000","last_change":"2024-03-09T14:28:32.313964+0000","last_active":"2024-03-09T15:36:06.526499+0000","last_peered":"2024-03-09T15:36:06.526499+0000","last_clean":"2024-03-09T15:36:06.526499+0000","last_became_active":"2024-03-09T14:28:32.313566+0000","last_became_peered":"2024-03-09T14:28:32.313566+0000","last_unstale":"2024-03-09T15:36:06.526499+0000","last_undegraded":"2024-03-09T15:36:06.526499+0000","last_fullsized":"2024-03-09T15:36:06.526499+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":100,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":245,"num_read_kb":84880,"num_write":567,"num_write_kb":19718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"32'7","reported_seq":1461,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.124817+0000","last_change":"2024-03-09T14:28:36.327203+0000","last_active":"2024-03-09T15:36:13.124817+0000","last_peered":"2024-03-09T15:36:13.124817+0000","last_clean":"2024-03-09T15:36:13.124817+0000","last_became_active":"2024-03-09T14:28:36.325820+0000","last_became_peered":"2024-03-09T14:28:36.325820+0000","last_unstale":"2024-03-09T15:36:13.124817+0000","last_undegraded":"2024-03-09T15:36:13.124817+0000","last_fullsized":"2024-03-09T15:36:13.124817+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1195,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1118,"num_read_kb":1421,"num_write":14,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"261'86","reported_seq":617,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.837134+0000","last_change":"2024-03-09T14:28:38.596269+0000","last_active":"2024-03-09T15:36:10.837134+0000","last_peered":"2024-03-09T15:36:10.837134+0000","last_clean":"2024-03-09T15:36:10.837134+0000","last_became_active":"2024-03-09T14:28:38.595478+0000","last_became_peered":"2024-03-09T14:28:38.595478+0000","last_unstale":"2024-03-09T15:36:10.837134+0000","last_undegraded":"2024-03-09T15:36:10.837134+0000","last_fullsized":"2024-03-09T15:36:10.837134+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":105,"num_read_kb":100,"num_write":70,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"261'489","reported_seq":2304,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.167232+0000","last_change":"2024-03-09T15:36:26.167298+0000","last_active":"2024-03-09T15:36:26.167232+0000","last_peered":"2024-03-09T15:36:26.167232+0000","last_clean":"2024-03-09T15:36:26.167232+0000","last_became_active":"2024-03-09T14:28:40.374408+0000","last_became_peered":"2024-03-09T14:28:40.374408+0000","last_unstale":"2024-03-09T15:36:26.167232+0000","last_undegraded":"2024-03-09T15:36:26.167232+0000","last_fullsized":"2024-03-09T15:36:26.167232+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'489","last_scrub_stamp":"2024-03-09T15:36:26.166904+0000","last_deep_scrub":"261'489","last_deep_scrub_stamp":"2024-03-09T15:36:26.166904+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.166904+0000","objects_scrubbed":1,"log_size":89,"log_dups_size":400,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:49:54.207124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"261'94","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.145360+0000","last_change":"2024-03-09T15:36:16.145423+0000","last_active":"2024-03-09T15:36:16.145360+0000","last_peered":"2024-03-09T15:36:16.145360+0000","last_clean":"2024-03-09T15:36:16.145360+0000","last_became_active":"2024-03-09T14:28:42.411523+0000","last_became_peered":"2024-03-09T14:28:42.411523+0000","last_unstale":"2024-03-09T15:36:16.145360+0000","last_undegraded":"2024-03-09T15:36:16.145360+0000","last_fullsized":"2024-03-09T15:36:16.145360+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'94","last_scrub_stamp":"2024-03-09T15:36:16.145166+0000","last_deep_scrub":"261'94","last_deep_scrub_stamp":"2024-03-09T15:36:16.145166+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.145166+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:56:22.170750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":72,"num_write":221,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"261'42","reported_seq":605,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.506243+0000","last_change":"2024-03-09T15:36:28.506314+0000","last_active":"2024-03-09T15:36:28.506243+0000","last_peered":"2024-03-09T15:36:28.506243+0000","last_clean":"2024-03-09T15:36:28.506243+0000","last_became_active":"2024-03-09T14:28:34.317226+0000","last_became_peered":"2024-03-09T14:28:34.317226+0000","last_unstale":"2024-03-09T15:36:28.506243+0000","last_undegraded":"2024-03-09T15:36:28.506243+0000","last_fullsized":"2024-03-09T15:36:28.506243+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'42","last_scrub_stamp":"2024-03-09T15:36:28.505993+0000","last_deep_scrub":"261'42","last_deep_scrub_stamp":"2024-03-09T15:36:28.505993+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.505993+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:14:19.130657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":188,"num_read_kb":158,"num_write":62,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"261'116","reported_seq":653,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.527014+0000","last_change":"2024-03-09T14:28:32.295693+0000","last_active":"2024-03-09T15:36:06.527014+0000","last_peered":"2024-03-09T15:36:06.527014+0000","last_clean":"2024-03-09T15:36:06.527014+0000","last_became_active":"2024-03-09T14:28:32.295179+0000","last_became_peered":"2024-03-09T14:28:32.295179+0000","last_unstale":"2024-03-09T15:36:06.527014+0000","last_undegraded":"2024-03-09T15:36:06.527014+0000","last_fullsized":"2024-03-09T15:36:06.527014+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":100,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262159,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":143,"num_read_kb":124,"num_write":390,"num_write_kb":5920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.023143+0000","last_change":"2024-03-09T14:28:36.319885+0000","last_active":"2024-03-09T15:36:06.023143+0000","last_peered":"2024-03-09T15:36:06.023143+0000","last_clean":"2024-03-09T15:36:06.023143+0000","last_became_active":"2024-03-09T14:28:36.319102+0000","last_became_peered":"2024-03-09T14:28:36.319102+0000","last_unstale":"2024-03-09T15:36:06.023143+0000","last_undegraded":"2024-03-09T15:36:06.023143+0000","last_fullsized":"2024-03-09T15:36:06.023143+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"261'321","reported_seq":958,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.774275+0000","last_change":"2024-03-09T15:36:27.774372+0000","last_active":"2024-03-09T15:36:27.774275+0000","last_peered":"2024-03-09T15:36:27.774275+0000","last_clean":"2024-03-09T15:36:27.774275+0000","last_became_active":"2024-03-09T14:28:38.337254+0000","last_became_peered":"2024-03-09T14:28:38.337254+0000","last_unstale":"2024-03-09T15:36:27.774275+0000","last_undegraded":"2024-03-09T15:36:27.774275+0000","last_fullsized":"2024-03-09T15:36:27.774275+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'321","last_scrub_stamp":"2024-03-09T15:36:27.774086+0000","last_deep_scrub":"261'321","last_deep_scrub_stamp":"2024-03-09T15:36:27.774086+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.774086+0000","objects_scrubbed":9,"log_size":21,"log_dups_size":300,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:34:50.482182+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13936,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":673,"num_read_kb":965,"num_write":788,"num_write_kb":629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":141964,"num_omap_keys":356,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.219910+0000","last_change":"2024-03-09T15:36:30.219963+0000","last_active":"2024-03-09T15:36:30.219910+0000","last_peered":"2024-03-09T15:36:30.219910+0000","last_clean":"2024-03-09T15:36:30.219910+0000","last_became_active":"2024-03-09T14:28:40.377714+0000","last_became_peered":"2024-03-09T14:28:40.377714+0000","last_unstale":"2024-03-09T15:36:30.219910+0000","last_undegraded":"2024-03-09T15:36:30.219910+0000","last_fullsized":"2024-03-09T15:36:30.219910+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:30.219708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:30.219708+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.219708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:30:50.699663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"261'69","reported_seq":577,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.545201+0000","last_change":"2024-03-09T15:36:20.545264+0000","last_active":"2024-03-09T15:36:20.545201+0000","last_peered":"2024-03-09T15:36:20.545201+0000","last_clean":"2024-03-09T15:36:20.545201+0000","last_became_active":"2024-03-09T14:28:42.401775+0000","last_became_peered":"2024-03-09T14:28:42.401775+0000","last_unstale":"2024-03-09T15:36:20.545201+0000","last_undegraded":"2024-03-09T15:36:20.545201+0000","last_fullsized":"2024-03-09T15:36:20.545201+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'69","last_scrub_stamp":"2024-03-09T15:36:20.545001+0000","last_deep_scrub":"261'69","last_deep_scrub_stamp":"2024-03-09T15:36:20.545001+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.545001+0000","objects_scrubbed":1,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:08:40.010383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":77,"num_write":153,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"261'174","reported_seq":834,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.140904+0000","last_change":"2024-03-09T15:36:27.140963+0000","last_active":"2024-03-09T15:36:27.140904+0000","last_peered":"2024-03-09T15:36:27.140904+0000","last_clean":"2024-03-09T15:36:27.140904+0000","last_became_active":"2024-03-09T14:28:34.326919+0000","last_became_peered":"2024-03-09T14:28:34.326919+0000","last_unstale":"2024-03-09T15:36:27.140904+0000","last_undegraded":"2024-03-09T15:36:27.140904+0000","last_fullsized":"2024-03-09T15:36:27.140904+0000","mapping_epoch":22,"log_start":"34'100","ondisk_log_start":"34'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'174","last_scrub_stamp":"2024-03-09T15:36:27.140715+0000","last_deep_scrub":"261'174","last_deep_scrub_stamp":"2024-03-09T15:36:27.140715+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.140715+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:06:14.462396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":914,"num_read_kb":819,"num_write":300,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101288+0000","last_change":"2024-03-09T14:28:12.265518+0000","last_active":"2024-03-09T15:36:14.101288+0000","last_peered":"2024-03-09T15:36:14.101288+0000","last_clean":"2024-03-09T15:36:14.101288+0000","last_became_active":"2024-03-09T14:28:12.262430+0000","last_became_peered":"2024-03-09T14:28:12.262430+0000","last_unstale":"2024-03-09T15:36:14.101288+0000","last_undegraded":"2024-03-09T15:36:14.101288+0000","last_fullsized":"2024-03-09T15:36:14.101288+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_clean_scrub_stamp":"2024-03-09T14:28:11.179640+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:01.213523+0000","last_change":"2024-03-09T14:28:19.233336+0000","last_active":"2024-03-09T15:35:01.213523+0000","last_peered":"2024-03-09T15:35:01.213523+0000","last_clean":"2024-03-09T15:35:01.213523+0000","last_became_active":"2024-03-09T14:28:19.232517+0000","last_became_peered":"2024-03-09T14:28:19.232517+0000","last_unstale":"2024-03-09T15:35:01.213523+0000","last_undegraded":"2024-03-09T15:35:01.213523+0000","last_fullsized":"2024-03-09T15:35:01.213523+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"261'130","reported_seq":690,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.491713+0000","last_change":"2024-03-09T14:28:32.300845+0000","last_active":"2024-03-09T15:36:06.491713+0000","last_peered":"2024-03-09T15:36:06.491713+0000","last_clean":"2024-03-09T15:36:06.491713+0000","last_became_active":"2024-03-09T14:28:32.300468+0000","last_became_peered":"2024-03-09T14:28:32.300468+0000","last_unstale":"2024-03-09T15:36:06.491713+0000","last_undegraded":"2024-03-09T15:36:06.491713+0000","last_fullsized":"2024-03-09T15:36:06.491713+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311644,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":173,"num_read_kb":6450,"num_write":457,"num_write_kb":21101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":511,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:16.228443+0000","last_change":"2024-03-09T14:28:36.318953+0000","last_active":"2024-03-09T15:35:16.228443+0000","last_peered":"2024-03-09T15:35:16.228443+0000","last_clean":"2024-03-09T15:35:16.228443+0000","last_became_active":"2024-03-09T14:28:36.318172+0000","last_became_peered":"2024-03-09T14:28:36.318172+0000","last_unstale":"2024-03-09T15:35:16.228443+0000","last_undegraded":"2024-03-09T15:35:16.228443+0000","last_fullsized":"2024-03-09T15:35:16.228443+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"261'502","reported_seq":1226,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.737856+0000","last_change":"2024-03-09T15:36:25.737931+0000","last_active":"2024-03-09T15:36:25.737856+0000","last_peered":"2024-03-09T15:36:25.737856+0000","last_clean":"2024-03-09T15:36:25.737856+0000","last_became_active":"2024-03-09T14:28:38.348941+0000","last_became_peered":"2024-03-09T14:28:38.348941+0000","last_unstale":"2024-03-09T15:36:25.737856+0000","last_undegraded":"2024-03-09T15:36:25.737856+0000","last_fullsized":"2024-03-09T15:36:25.737856+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'502","last_scrub_stamp":"2024-03-09T15:36:25.737665+0000","last_deep_scrub":"261'502","last_deep_scrub_stamp":"2024-03-09T15:36:25.737665+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.737665+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":400,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:22:57.116693+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37400,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":987,"num_read_kb":1910,"num_write":934,"num_write_kb":767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":90684,"num_omap_keys":220,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":497,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.043973+0000","last_change":"2024-03-09T14:28:40.364392+0000","last_active":"2024-03-09T15:35:51.043973+0000","last_peered":"2024-03-09T15:35:51.043973+0000","last_clean":"2024-03-09T15:35:51.043973+0000","last_became_active":"2024-03-09T14:28:40.362781+0000","last_became_peered":"2024-03-09T14:28:40.362781+0000","last_unstale":"2024-03-09T15:35:51.043973+0000","last_undegraded":"2024-03-09T15:35:51.043973+0000","last_fullsized":"2024-03-09T15:35:51.043973+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"261'118","reported_seq":818,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:55.818857+0000","last_change":"2024-03-09T14:28:42.399378+0000","last_active":"2024-03-09T15:35:55.818857+0000","last_peered":"2024-03-09T15:35:55.818857+0000","last_clean":"2024-03-09T15:35:55.818857+0000","last_became_active":"2024-03-09T14:28:42.398626+0000","last_became_peered":"2024-03-09T14:28:42.398626+0000","last_unstale":"2024-03-09T15:35:55.818857+0000","last_undegraded":"2024-03-09T15:35:55.818857+0000","last_fullsized":"2024-03-09T15:35:55.818857+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":356,"num_read_kb":343,"num_write":215,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"261'90","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:46.042921+0000","last_change":"2024-03-09T14:28:34.297905+0000","last_active":"2024-03-09T15:35:46.042921+0000","last_peered":"2024-03-09T15:35:46.042921+0000","last_clean":"2024-03-09T15:35:46.042921+0000","last_became_active":"2024-03-09T14:28:34.297472+0000","last_became_peered":"2024-03-09T14:28:34.297472+0000","last_unstale":"2024-03-09T15:35:46.042921+0000","last_undegraded":"2024-03-09T15:35:46.042921+0000","last_fullsized":"2024-03-09T15:35:46.042921+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":356,"num_read_kb":292,"num_write":141,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":527,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101340+0000","last_change":"2024-03-09T14:28:19.242176+0000","last_active":"2024-03-09T15:36:14.101340+0000","last_peered":"2024-03-09T15:36:14.101340+0000","last_clean":"2024-03-09T15:36:14.101340+0000","last_became_active":"2024-03-09T14:28:19.241110+0000","last_became_peered":"2024-03-09T14:28:19.241110+0000","last_unstale":"2024-03-09T15:36:14.101340+0000","last_undegraded":"2024-03-09T15:36:14.101340+0000","last_fullsized":"2024-03-09T15:36:14.101340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"261'138","reported_seq":711,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.525842+0000","last_change":"2024-03-09T14:28:32.312953+0000","last_active":"2024-03-09T15:36:06.525842+0000","last_peered":"2024-03-09T15:36:06.525842+0000","last_clean":"2024-03-09T15:36:06.525842+0000","last_became_active":"2024-03-09T14:28:32.312536+0000","last_became_peered":"2024-03-09T14:28:32.312536+0000","last_unstale":"2024-03-09T15:36:06.525842+0000","last_undegraded":"2024-03-09T15:36:06.525842+0000","last_fullsized":"2024-03-09T15:36:06.525842+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":100,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":219,"num_read_kb":12489,"num_write":525,"num_write_kb":17953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:46.223756+0000","last_change":"2024-03-09T14:28:36.307721+0000","last_active":"2024-03-09T15:35:46.223756+0000","last_peered":"2024-03-09T15:35:46.223756+0000","last_clean":"2024-03-09T15:35:46.223756+0000","last_became_active":"2024-03-09T14:28:36.306912+0000","last_became_peered":"2024-03-09T14:28:36.306912+0000","last_unstale":"2024-03-09T15:35:46.223756+0000","last_undegraded":"2024-03-09T15:35:46.223756+0000","last_fullsized":"2024-03-09T15:35:46.223756+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"261'847","reported_seq":2525,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.533064+0000","last_change":"2024-03-09T14:28:38.329490+0000","last_active":"2024-03-09T15:36:06.533064+0000","last_peered":"2024-03-09T15:36:06.533064+0000","last_clean":"2024-03-09T15:36:06.533064+0000","last_became_active":"2024-03-09T14:28:38.328738+0000","last_became_peered":"2024-03-09T14:28:38.328738+0000","last_unstale":"2024-03-09T15:36:06.533064+0000","last_undegraded":"2024-03-09T15:36:06.533064+0000","last_fullsized":"2024-03-09T15:36:06.533064+0000","mapping_epoch":26,"log_start":"261'800","ondisk_log_start":"261'800","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":800,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34683,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1323,"num_read_kb":2148,"num_write":662,"num_write_kb":462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:31.232104+0000","last_change":"2024-03-09T14:28:40.387247+0000","last_active":"2024-03-09T15:35:31.232104+0000","last_peered":"2024-03-09T15:35:31.232104+0000","last_clean":"2024-03-09T15:35:31.232104+0000","last_became_active":"2024-03-09T14:28:40.386451+0000","last_became_peered":"2024-03-09T14:28:40.386451+0000","last_unstale":"2024-03-09T15:35:31.232104+0000","last_undegraded":"2024-03-09T15:35:31.232104+0000","last_fullsized":"2024-03-09T15:35:31.232104+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"261'129","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.567045+0000","last_change":"2024-03-09T15:36:18.567214+0000","last_active":"2024-03-09T15:36:18.567045+0000","last_peered":"2024-03-09T15:36:18.567045+0000","last_clean":"2024-03-09T15:36:18.567045+0000","last_became_active":"2024-03-09T14:28:42.403013+0000","last_became_peered":"2024-03-09T14:28:42.403013+0000","last_unstale":"2024-03-09T15:36:18.567045+0000","last_undegraded":"2024-03-09T15:36:18.567045+0000","last_fullsized":"2024-03-09T15:36:18.567045+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'129","last_scrub_stamp":"2024-03-09T15:36:18.566788+0000","last_deep_scrub":"261'129","last_deep_scrub_stamp":"2024-03-09T15:36:18.566788+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.566788+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:45:17.069116+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1248,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":261,"num_read_kb":241,"num_write":300,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"261'48","reported_seq":602,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:45.990476+0000","last_change":"2024-03-09T14:28:34.297150+0000","last_active":"2024-03-09T15:35:45.990476+0000","last_peered":"2024-03-09T15:35:45.990476+0000","last_clean":"2024-03-09T15:35:45.990476+0000","last_became_active":"2024-03-09T14:28:34.296733+0000","last_became_peered":"2024-03-09T14:28:34.296733+0000","last_unstale":"2024-03-09T15:35:45.990476+0000","last_undegraded":"2024-03-09T15:35:45.990476+0000","last_fullsized":"2024-03-09T15:35:45.990476+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":200,"num_read_kb":166,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":531,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.234263+0000","last_change":"2024-03-09T15:36:28.234318+0000","last_active":"2024-03-09T15:36:28.234263+0000","last_peered":"2024-03-09T15:36:28.234263+0000","last_clean":"2024-03-09T15:36:28.234263+0000","last_became_active":"2024-03-09T14:28:19.436985+0000","last_became_peered":"2024-03-09T14:28:19.436985+0000","last_unstale":"2024-03-09T15:36:28.234263+0000","last_undegraded":"2024-03-09T15:36:28.234263+0000","last_fullsized":"2024-03-09T15:36:28.234263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:28.234071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:28.234071+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.234071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:27:16.042052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"261'152","reported_seq":810,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.519928+0000","last_change":"2024-03-09T14:28:32.303463+0000","last_active":"2024-03-09T15:36:06.519928+0000","last_peered":"2024-03-09T15:36:06.519928+0000","last_clean":"2024-03-09T15:36:06.519928+0000","last_became_active":"2024-03-09T14:28:32.303176+0000","last_became_peered":"2024-03-09T14:28:32.303176+0000","last_unstale":"2024-03-09T15:36:06.519928+0000","last_undegraded":"2024-03-09T15:36:06.519928+0000","last_fullsized":"2024-03-09T15:36:06.519928+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":100,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8554724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":486,"num_read_kb":3616,"num_write":592,"num_write_kb":29269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"261'2","reported_seq":518,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.203415+0000","last_change":"2024-03-09T15:36:23.203478+0000","last_active":"2024-03-09T15:36:23.203415+0000","last_peered":"2024-03-09T15:36:23.203415+0000","last_clean":"2024-03-09T15:36:23.203415+0000","last_became_active":"2024-03-09T14:28:36.305971+0000","last_became_peered":"2024-03-09T14:28:36.305971+0000","last_unstale":"2024-03-09T15:36:23.203415+0000","last_undegraded":"2024-03-09T15:36:23.203415+0000","last_fullsized":"2024-03-09T15:36:23.203415+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:23.203217+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:23.203217+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.203217+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:42:55.897785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"261'230","reported_seq":1599,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.976894+0000","last_change":"2024-03-09T14:28:38.598201+0000","last_active":"2024-03-09T15:35:58.976894+0000","last_peered":"2024-03-09T15:35:58.976894+0000","last_clean":"2024-03-09T15:35:58.976894+0000","last_became_active":"2024-03-09T14:28:38.597403+0000","last_became_peered":"2024-03-09T14:28:38.597403+0000","last_unstale":"2024-03-09T15:35:58.976894+0000","last_undegraded":"2024-03-09T15:35:58.976894+0000","last_fullsized":"2024-03-09T15:35:58.976894+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":200,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":226,"num_read_kb":216,"num_write":195,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":498,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:21.037287+0000","last_change":"2024-03-09T14:28:40.372884+0000","last_active":"2024-03-09T15:35:21.037287+0000","last_peered":"2024-03-09T15:35:21.037287+0000","last_clean":"2024-03-09T15:35:21.037287+0000","last_became_active":"2024-03-09T14:28:40.371383+0000","last_became_peered":"2024-03-09T14:28:40.371383+0000","last_unstale":"2024-03-09T15:35:21.037287+0000","last_undegraded":"2024-03-09T15:35:21.037287+0000","last_fullsized":"2024-03-09T15:35:21.037287+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"261'203","reported_seq":803,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.435372+0000","last_change":"2024-03-09T15:36:24.435440+0000","last_active":"2024-03-09T15:36:24.435372+0000","last_peered":"2024-03-09T15:36:24.435372+0000","last_clean":"2024-03-09T15:36:24.435372+0000","last_became_active":"2024-03-09T14:28:42.395781+0000","last_became_peered":"2024-03-09T14:28:42.395781+0000","last_unstale":"2024-03-09T15:36:24.435372+0000","last_undegraded":"2024-03-09T15:36:24.435372+0000","last_fullsized":"2024-03-09T15:36:24.435372+0000","mapping_epoch":30,"log_start":"247'100","ondisk_log_start":"247'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'203","last_scrub_stamp":"2024-03-09T15:36:24.435182+0000","last_deep_scrub":"261'203","last_deep_scrub_stamp":"2024-03-09T15:36:24.435182+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.435182+0000","objects_scrubbed":2,"log_size":103,"log_dups_size":100,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:28.404974+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":435,"num_read_kb":456,"num_write":425,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":851,"num_omap_keys":5,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"261'74","reported_seq":619,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:15.983964+0000","last_change":"2024-03-09T14:28:34.313660+0000","last_active":"2024-03-09T15:35:15.983964+0000","last_peered":"2024-03-09T15:35:15.983964+0000","last_clean":"2024-03-09T15:35:15.983964+0000","last_became_active":"2024-03-09T14:28:34.313302+0000","last_became_peered":"2024-03-09T14:28:34.313302+0000","last_unstale":"2024-03-09T15:35:15.983964+0000","last_undegraded":"2024-03-09T15:35:15.983964+0000","last_fullsized":"2024-03-09T15:35:15.983964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":225,"num_write":104,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":523,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:01.213454+0000","last_change":"2024-03-09T14:28:19.238720+0000","last_active":"2024-03-09T15:35:01.213454+0000","last_peered":"2024-03-09T15:35:01.213454+0000","last_clean":"2024-03-09T15:35:01.213454+0000","last_became_active":"2024-03-09T14:28:19.237937+0000","last_became_peered":"2024-03-09T14:28:19.237937+0000","last_unstale":"2024-03-09T15:35:01.213454+0000","last_undegraded":"2024-03-09T15:35:01.213454+0000","last_fullsized":"2024-03-09T15:35:01.213454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"261'127","reported_seq":703,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.518582+0000","last_change":"2024-03-09T14:28:32.312937+0000","last_active":"2024-03-09T15:36:06.518582+0000","last_peered":"2024-03-09T15:36:06.518582+0000","last_clean":"2024-03-09T15:36:06.518582+0000","last_became_active":"2024-03-09T14:28:32.303963+0000","last_became_peered":"2024-03-09T14:28:32.303963+0000","last_unstale":"2024-03-09T15:36:06.518582+0000","last_undegraded":"2024-03-09T15:36:06.518582+0000","last_fullsized":"2024-03-09T15:36:06.518582+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":100,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6898850,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":175,"num_read_kb":15115,"num_write":430,"num_write_kb":32085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:31.039898+0000","last_change":"2024-03-09T14:28:36.318128+0000","last_active":"2024-03-09T15:35:31.039898+0000","last_peered":"2024-03-09T15:35:31.039898+0000","last_clean":"2024-03-09T15:35:31.039898+0000","last_became_active":"2024-03-09T14:28:36.317586+0000","last_became_peered":"2024-03-09T14:28:36.317586+0000","last_unstale":"2024-03-09T15:35:31.039898+0000","last_undegraded":"2024-03-09T15:35:31.039898+0000","last_fullsized":"2024-03-09T15:35:31.039898+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"261'129","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:50.832966+0000","last_change":"2024-03-09T14:28:38.335628+0000","last_active":"2024-03-09T15:35:50.832966+0000","last_peered":"2024-03-09T15:35:50.832966+0000","last_clean":"2024-03-09T15:35:50.832966+0000","last_became_active":"2024-03-09T14:28:38.334463+0000","last_became_peered":"2024-03-09T14:28:38.334463+0000","last_unstale":"2024-03-09T15:35:50.832966+0000","last_undegraded":"2024-03-09T15:35:50.832966+0000","last_fullsized":"2024-03-09T15:35:50.832966+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":178,"num_read_kb":172,"num_write":132,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.103290+0000","last_change":"2024-03-09T15:36:20.103341+0000","last_active":"2024-03-09T15:36:20.103290+0000","last_peered":"2024-03-09T15:36:20.103290+0000","last_clean":"2024-03-09T15:36:20.103290+0000","last_became_active":"2024-03-09T14:28:40.370436+0000","last_became_peered":"2024-03-09T14:28:40.370436+0000","last_unstale":"2024-03-09T15:36:20.103290+0000","last_undegraded":"2024-03-09T15:36:20.103290+0000","last_fullsized":"2024-03-09T15:36:20.103290+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:20.103091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:20.103091+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.103091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:29:38.107241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"261'48","reported_seq":573,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.682061+0000","last_change":"2024-03-09T15:36:17.682122+0000","last_active":"2024-03-09T15:36:17.682061+0000","last_peered":"2024-03-09T15:36:17.682061+0000","last_clean":"2024-03-09T15:36:17.682061+0000","last_became_active":"2024-03-09T14:28:42.397058+0000","last_became_peered":"2024-03-09T14:28:42.397058+0000","last_unstale":"2024-03-09T15:36:17.682061+0000","last_undegraded":"2024-03-09T15:36:17.682061+0000","last_fullsized":"2024-03-09T15:36:17.682061+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:17.681861+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:17.681861+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.681861+0000","objects_scrubbed":1,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:38:01.033648+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":100,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"261'43","reported_seq":594,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:15.983866+0000","last_change":"2024-03-09T14:28:34.297850+0000","last_active":"2024-03-09T15:35:15.983866+0000","last_peered":"2024-03-09T15:35:15.983866+0000","last_clean":"2024-03-09T15:35:15.983866+0000","last_became_active":"2024-03-09T14:28:34.296953+0000","last_became_peered":"2024-03-09T14:28:34.296953+0000","last_unstale":"2024-03-09T15:35:15.983866+0000","last_undegraded":"2024-03-09T15:35:15.983866+0000","last_fullsized":"2024-03-09T15:35:15.983866+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":162,"num_write":75,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"261'163","reported_seq":798,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.490972+0000","last_change":"2024-03-09T14:28:32.303697+0000","last_active":"2024-03-09T15:36:06.490972+0000","last_peered":"2024-03-09T15:36:06.490972+0000","last_clean":"2024-03-09T15:36:06.490972+0000","last_became_active":"2024-03-09T14:28:32.302970+0000","last_became_peered":"2024-03-09T14:28:32.302970+0000","last_unstale":"2024-03-09T15:36:06.490972+0000","last_undegraded":"2024-03-09T15:36:06.490972+0000","last_fullsized":"2024-03-09T15:36:06.490972+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":100,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655822,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":94521,"num_write":632,"num_write_kb":45805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":521,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:00.980726+0000","last_change":"2024-03-09T14:28:19.236670+0000","last_active":"2024-03-09T15:35:00.980726+0000","last_peered":"2024-03-09T15:35:00.980726+0000","last_clean":"2024-03-09T15:35:00.980726+0000","last_became_active":"2024-03-09T14:28:19.236240+0000","last_became_peered":"2024-03-09T14:28:19.236240+0000","last_unstale":"2024-03-09T15:35:00.980726+0000","last_undegraded":"2024-03-09T15:35:00.980726+0000","last_fullsized":"2024-03-09T15:35:00.980726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:16.035977+0000","last_change":"2024-03-09T14:28:36.315028+0000","last_active":"2024-03-09T15:35:16.035977+0000","last_peered":"2024-03-09T15:35:16.035977+0000","last_clean":"2024-03-09T15:35:16.035977+0000","last_became_active":"2024-03-09T14:28:36.314526+0000","last_became_peered":"2024-03-09T14:28:36.314526+0000","last_unstale":"2024-03-09T15:35:16.035977+0000","last_undegraded":"2024-03-09T15:35:16.035977+0000","last_fullsized":"2024-03-09T15:35:16.035977+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"261'384","reported_seq":1067,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.295054+0000","last_change":"2024-03-09T14:28:38.335922+0000","last_active":"2024-03-09T15:36:06.295054+0000","last_peered":"2024-03-09T15:36:06.295054+0000","last_clean":"2024-03-09T15:36:06.295054+0000","last_became_active":"2024-03-09T14:28:38.334976+0000","last_became_peered":"2024-03-09T14:28:38.334976+0000","last_unstale":"2024-03-09T15:36:06.295054+0000","last_undegraded":"2024-03-09T15:36:06.295054+0000","last_fullsized":"2024-03-09T15:36:06.295054+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":300,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21042,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":640,"num_read_kb":1204,"num_write":456,"num_write_kb":268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.131101+0000","last_change":"2024-03-09T15:36:20.131154+0000","last_active":"2024-03-09T15:36:20.131101+0000","last_peered":"2024-03-09T15:36:20.131101+0000","last_clean":"2024-03-09T15:36:20.131101+0000","last_became_active":"2024-03-09T14:28:40.379108+0000","last_became_peered":"2024-03-09T14:28:40.379108+0000","last_unstale":"2024-03-09T15:36:20.131101+0000","last_undegraded":"2024-03-09T15:36:20.131101+0000","last_fullsized":"2024-03-09T15:36:20.131101+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:20.130903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:20.130903+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.130903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:41:11.167873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"261'104","reported_seq":639,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.264151+0000","last_change":"2024-03-09T15:36:26.264345+0000","last_active":"2024-03-09T15:36:26.264151+0000","last_peered":"2024-03-09T15:36:26.264151+0000","last_clean":"2024-03-09T15:36:26.264151+0000","last_became_active":"2024-03-09T14:28:42.405277+0000","last_became_peered":"2024-03-09T14:28:42.405277+0000","last_unstale":"2024-03-09T15:36:26.264151+0000","last_undegraded":"2024-03-09T15:36:26.264151+0000","last_fullsized":"2024-03-09T15:36:26.264151+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'104","last_scrub_stamp":"2024-03-09T15:36:26.263887+0000","last_deep_scrub":"261'104","last_deep_scrub_stamp":"2024-03-09T15:36:26.263887+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.263887+0000","objects_scrubbed":3,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:52.786785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":717,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185,"num_read_kb":160,"num_write":261,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"261'43","reported_seq":576,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:16.036023+0000","last_change":"2024-03-09T14:28:34.298018+0000","last_active":"2024-03-09T15:35:16.036023+0000","last_peered":"2024-03-09T15:35:16.036023+0000","last_clean":"2024-03-09T15:35:16.036023+0000","last_became_active":"2024-03-09T14:28:34.296660+0000","last_became_peered":"2024-03-09T14:28:34.296660+0000","last_unstale":"2024-03-09T15:35:16.036023+0000","last_undegraded":"2024-03-09T15:35:16.036023+0000","last_fullsized":"2024-03-09T15:35:16.036023+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":129,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"261'148","reported_seq":779,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.491461+0000","last_change":"2024-03-09T14:28:32.305873+0000","last_active":"2024-03-09T15:36:06.491461+0000","last_peered":"2024-03-09T15:36:06.491461+0000","last_clean":"2024-03-09T15:36:06.491461+0000","last_became_active":"2024-03-09T14:28:32.305318+0000","last_became_peered":"2024-03-09T14:28:32.305318+0000","last_unstale":"2024-03-09T15:36:06.491461+0000","last_undegraded":"2024-03-09T15:36:06.491461+0000","last_fullsized":"2024-03-09T15:36:06.491461+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":100,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262245,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":357,"num_read_kb":10578,"num_write":456,"num_write_kb":20505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":525,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:11.227202+0000","last_change":"2024-03-09T14:28:19.241475+0000","last_active":"2024-03-09T15:35:11.227202+0000","last_peered":"2024-03-09T15:35:11.227202+0000","last_clean":"2024-03-09T15:35:11.227202+0000","last_became_active":"2024-03-09T14:28:19.240704+0000","last_became_peered":"2024-03-09T14:28:19.240704+0000","last_unstale":"2024-03-09T15:35:11.227202+0000","last_undegraded":"2024-03-09T15:35:11.227202+0000","last_fullsized":"2024-03-09T15:35:11.227202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.191317+0000","last_change":"2024-03-09T15:36:25.191369+0000","last_active":"2024-03-09T15:36:25.191317+0000","last_peered":"2024-03-09T15:36:25.191317+0000","last_clean":"2024-03-09T15:36:25.191317+0000","last_became_active":"2024-03-09T14:28:36.314676+0000","last_became_peered":"2024-03-09T14:28:36.314676+0000","last_unstale":"2024-03-09T15:36:25.191317+0000","last_undegraded":"2024-03-09T15:36:25.191317+0000","last_fullsized":"2024-03-09T15:36:25.191317+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.191100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.191100+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.191100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:12:04.217794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"261'197","reported_seq":779,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.982322+0000","last_change":"2024-03-09T14:28:38.595995+0000","last_active":"2024-03-09T15:35:58.982322+0000","last_peered":"2024-03-09T15:35:58.982322+0000","last_clean":"2024-03-09T15:35:58.982322+0000","last_became_active":"2024-03-09T14:28:38.595028+0000","last_became_peered":"2024-03-09T14:28:38.595028+0000","last_unstale":"2024-03-09T15:35:58.982322+0000","last_undegraded":"2024-03-09T15:35:58.982322+0000","last_fullsized":"2024-03-09T15:35:58.982322+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":100,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":269,"num_read_kb":262,"num_write":230,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"261'501","reported_seq":2306,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.727120+0000","last_change":"2024-03-09T15:36:23.727185+0000","last_active":"2024-03-09T15:36:23.727120+0000","last_peered":"2024-03-09T15:36:23.727120+0000","last_clean":"2024-03-09T15:36:23.727120+0000","last_became_active":"2024-03-09T14:28:40.369165+0000","last_became_peered":"2024-03-09T14:28:40.369165+0000","last_unstale":"2024-03-09T15:36:23.727120+0000","last_undegraded":"2024-03-09T15:36:23.727120+0000","last_fullsized":"2024-03-09T15:36:23.727120+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'501","last_scrub_stamp":"2024-03-09T15:36:23.726932+0000","last_deep_scrub":"261'501","last_deep_scrub_stamp":"2024-03-09T15:36:23.726932+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.726932+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":400,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:05:11.145403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"261'79","reported_seq":577,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.487857+0000","last_change":"2024-03-09T15:36:27.487857+0000","last_active":"2024-03-09T15:36:27.487857+0000","last_peered":"2024-03-09T15:36:27.487857+0000","last_clean":"2024-03-09T15:36:27.487857+0000","last_became_active":"2024-03-09T14:28:42.392402+0000","last_became_peered":"2024-03-09T14:28:42.392402+0000","last_unstale":"2024-03-09T15:36:27.487857+0000","last_undegraded":"2024-03-09T15:36:27.487857+0000","last_fullsized":"2024-03-09T15:36:27.487857+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'79","last_scrub_stamp":"2024-03-09T15:36:27.487564+0000","last_deep_scrub":"261'79","last_deep_scrub_stamp":"2024-03-09T15:36:27.487564+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.487564+0000","objects_scrubbed":3,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:09:13.446303+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":57,"num_write":172,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"261'48","reported_seq":598,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.188879+0000","last_change":"2024-03-09T15:36:18.188944+0000","last_active":"2024-03-09T15:36:18.188879+0000","last_peered":"2024-03-09T15:36:18.188879+0000","last_clean":"2024-03-09T15:36:18.188879+0000","last_became_active":"2024-03-09T14:29:06.440556+0000","last_became_peered":"2024-03-09T14:29:06.440556+0000","last_unstale":"2024-03-09T15:36:18.188879+0000","last_undegraded":"2024-03-09T15:36:18.188879+0000","last_fullsized":"2024-03-09T15:36:18.188879+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:18.188663+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:18.188663+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.188663+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:03:02.670504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":216,"num_read_kb":182,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"261'145","reported_seq":691,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.492729+0000","last_change":"2024-03-09T14:28:32.319314+0000","last_active":"2024-03-09T15:36:06.492729+0000","last_peered":"2024-03-09T15:36:06.492729+0000","last_clean":"2024-03-09T15:36:06.492729+0000","last_became_active":"2024-03-09T14:28:32.318971+0000","last_became_peered":"2024-03-09T14:28:32.318971+0000","last_unstale":"2024-03-09T15:36:06.492729+0000","last_undegraded":"2024-03-09T15:36:06.492729+0000","last_fullsized":"2024-03-09T15:36:06.492729+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":151,"num_read_kb":8647,"num_write":476,"num_write_kb":27350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":520,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:30.987444+0000","last_change":"2024-03-09T14:28:19.238362+0000","last_active":"2024-03-09T15:35:30.987444+0000","last_peered":"2024-03-09T15:35:30.987444+0000","last_clean":"2024-03-09T15:35:30.987444+0000","last_became_active":"2024-03-09T14:28:19.237750+0000","last_became_peered":"2024-03-09T14:28:19.237750+0000","last_unstale":"2024-03-09T15:35:30.987444+0000","last_undegraded":"2024-03-09T15:35:30.987444+0000","last_fullsized":"2024-03-09T15:35:30.987444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:16.011425+0000","last_change":"2024-03-09T14:28:36.328518+0000","last_active":"2024-03-09T15:35:16.011425+0000","last_peered":"2024-03-09T15:35:16.011425+0000","last_clean":"2024-03-09T15:35:16.011425+0000","last_became_active":"2024-03-09T14:28:36.327601+0000","last_became_peered":"2024-03-09T14:28:36.327601+0000","last_unstale":"2024-03-09T15:35:16.011425+0000","last_undegraded":"2024-03-09T15:35:16.011425+0000","last_fullsized":"2024-03-09T15:35:16.011425+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"261'111","reported_seq":675,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.173092+0000","last_change":"2024-03-09T15:36:26.173180+0000","last_active":"2024-03-09T15:36:26.173092+0000","last_peered":"2024-03-09T15:36:26.173092+0000","last_clean":"2024-03-09T15:36:26.173092+0000","last_became_active":"2024-03-09T14:28:38.347750+0000","last_became_peered":"2024-03-09T14:28:38.347750+0000","last_unstale":"2024-03-09T15:36:26.173092+0000","last_undegraded":"2024-03-09T15:36:26.173092+0000","last_fullsized":"2024-03-09T15:36:26.173092+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'111","last_scrub_stamp":"2024-03-09T15:36:26.172900+0000","last_deep_scrub":"261'111","last_deep_scrub_stamp":"2024-03-09T15:36:26.172900+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.172900+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:00:14.752406+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"261'514","reported_seq":2487,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.539757+0000","last_change":"2024-03-09T14:28:40.387580+0000","last_active":"2024-03-09T15:36:06.539757+0000","last_peered":"2024-03-09T15:36:06.539757+0000","last_clean":"2024-03-09T15:36:06.539757+0000","last_became_active":"2024-03-09T14:28:40.386859+0000","last_became_peered":"2024-03-09T14:28:40.386859+0000","last_unstale":"2024-03-09T15:36:06.539757+0000","last_undegraded":"2024-03-09T15:36:06.539757+0000","last_fullsized":"2024-03-09T15:36:06.539757+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":500,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"261'108","reported_seq":618,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.271001+0000","last_change":"2024-03-09T15:36:25.271065+0000","last_active":"2024-03-09T15:36:25.271001+0000","last_peered":"2024-03-09T15:36:25.271001+0000","last_clean":"2024-03-09T15:36:25.271001+0000","last_became_active":"2024-03-09T14:28:42.406139+0000","last_became_peered":"2024-03-09T14:28:42.406139+0000","last_unstale":"2024-03-09T15:36:25.271001+0000","last_undegraded":"2024-03-09T15:36:25.271001+0000","last_fullsized":"2024-03-09T15:36:25.271001+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'108","last_scrub_stamp":"2024-03-09T15:36:25.270811+0000","last_deep_scrub":"261'108","last_deep_scrub_stamp":"2024-03-09T15:36:25.270811+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.270811+0000","objects_scrubbed":2,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:05:43.169466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":65,"num_write":235,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"261'216","reported_seq":788,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:16.011393+0000","last_change":"2024-03-09T14:28:34.301025+0000","last_active":"2024-03-09T15:35:16.011393+0000","last_peered":"2024-03-09T15:35:16.011393+0000","last_clean":"2024-03-09T15:35:16.011393+0000","last_became_active":"2024-03-09T14:28:34.299988+0000","last_became_peered":"2024-03-09T14:28:34.299988+0000","last_unstale":"2024-03-09T15:35:16.011393+0000","last_undegraded":"2024-03-09T15:35:16.011393+0000","last_fullsized":"2024-03-09T15:35:16.011393+0000","mapping_epoch":22,"log_start":"258'200","ondisk_log_start":"258'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":200,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":843,"num_read_kb":645,"num_write":377,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"261'150","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.454415+0000","last_change":"2024-03-09T14:28:32.299582+0000","last_active":"2024-03-09T15:36:06.454415+0000","last_peered":"2024-03-09T15:36:06.454415+0000","last_clean":"2024-03-09T15:36:06.454415+0000","last_became_active":"2024-03-09T14:28:32.298977+0000","last_became_peered":"2024-03-09T14:28:32.298977+0000","last_unstale":"2024-03-09T15:36:06.454415+0000","last_undegraded":"2024-03-09T15:36:06.454415+0000","last_fullsized":"2024-03-09T15:36:06.454415+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242905,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":173,"num_read_kb":5781,"num_write":510,"num_write_kb":37567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":519,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:30.813886+0000","last_change":"2024-03-09T14:28:19.242561+0000","last_active":"2024-03-09T15:35:30.813886+0000","last_peered":"2024-03-09T15:35:30.813886+0000","last_clean":"2024-03-09T15:35:30.813886+0000","last_became_active":"2024-03-09T14:28:19.241700+0000","last_became_peered":"2024-03-09T14:28:19.241700+0000","last_unstale":"2024-03-09T15:35:30.813886+0000","last_undegraded":"2024-03-09T15:35:30.813886+0000","last_fullsized":"2024-03-09T15:35:30.813886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"32'3","reported_seq":823,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.707998+0000","last_change":"2024-03-09T14:28:36.317874+0000","last_active":"2024-03-09T15:36:05.707998+0000","last_peered":"2024-03-09T15:36:05.707998+0000","last_clean":"2024-03-09T15:36:05.707998+0000","last_became_active":"2024-03-09T14:28:36.317327+0000","last_became_peered":"2024-03-09T14:28:36.317327+0000","last_unstale":"2024-03-09T15:36:05.707998+0000","last_undegraded":"2024-03-09T15:36:05.707998+0000","last_fullsized":"2024-03-09T15:36:05.707998+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":484,"num_read_kb":482,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"261'470","reported_seq":1210,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.347690+0000","last_change":"2024-03-09T14:28:38.603808+0000","last_active":"2024-03-09T15:36:06.347690+0000","last_peered":"2024-03-09T15:36:06.347690+0000","last_clean":"2024-03-09T15:36:06.347690+0000","last_became_active":"2024-03-09T14:28:38.602908+0000","last_became_peered":"2024-03-09T14:28:38.602908+0000","last_unstale":"2024-03-09T15:36:06.347690+0000","last_undegraded":"2024-03-09T15:36:06.347690+0000","last_fullsized":"2024-03-09T15:36:06.347690+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":400,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29305,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":797,"num_read_kb":1561,"num_write":550,"num_write_kb":344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:20.826861+0000","last_change":"2024-03-09T14:28:40.376657+0000","last_active":"2024-03-09T15:35:20.826861+0000","last_peered":"2024-03-09T15:35:20.826861+0000","last_clean":"2024-03-09T15:35:20.826861+0000","last_became_active":"2024-03-09T14:28:40.375884+0000","last_became_peered":"2024-03-09T14:28:40.375884+0000","last_unstale":"2024-03-09T15:35:20.826861+0000","last_undegraded":"2024-03-09T15:35:20.826861+0000","last_fullsized":"2024-03-09T15:35:20.826861+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"261'56","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.577103+0000","last_change":"2024-03-09T15:36:13.577103+0000","last_active":"2024-03-09T15:36:13.577103+0000","last_peered":"2024-03-09T15:36:13.577103+0000","last_clean":"2024-03-09T15:36:13.577103+0000","last_became_active":"2024-03-09T14:28:42.405998+0000","last_became_peered":"2024-03-09T14:28:42.405998+0000","last_unstale":"2024-03-09T15:36:13.577103+0000","last_undegraded":"2024-03-09T15:36:13.577103+0000","last_fullsized":"2024-03-09T15:36:13.577103+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:13.576825+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:13.576825+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.576825+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:29:57.437161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":52,"num_write":123,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1f","version":"261'149","reported_seq":735,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.301854+0000","last_change":"2024-03-09T15:36:24.301934+0000","last_active":"2024-03-09T15:36:24.301854+0000","last_peered":"2024-03-09T15:36:24.301854+0000","last_clean":"2024-03-09T15:36:24.301854+0000","last_became_active":"2024-03-09T14:28:42.408774+0000","last_became_peered":"2024-03-09T14:28:42.408774+0000","last_unstale":"2024-03-09T15:36:24.301854+0000","last_undegraded":"2024-03-09T15:36:24.301854+0000","last_fullsized":"2024-03-09T15:36:24.301854+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'149","last_scrub_stamp":"2024-03-09T15:36:24.301670+0000","last_deep_scrub":"261'149","last_deep_scrub_stamp":"2024-03-09T15:36:24.301670+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.301670+0000","objects_scrubbed":3,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:31:20.109756+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":256,"num_read_kb":233,"num_write":254,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"261'183","reported_seq":745,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.649044+0000","last_change":"2024-03-09T14:28:34.315742+0000","last_active":"2024-03-09T15:36:12.649044+0000","last_peered":"2024-03-09T15:36:12.649044+0000","last_clean":"2024-03-09T15:36:12.649044+0000","last_became_active":"2024-03-09T14:28:34.314893+0000","last_became_peered":"2024-03-09T14:28:34.314893+0000","last_unstale":"2024-03-09T15:36:12.649044+0000","last_undegraded":"2024-03-09T15:36:12.649044+0000","last_fullsized":"2024-03-09T15:36:12.649044+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":100,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":706,"num_read_kb":592,"num_write":291,"num_write_kb":155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"261'129","reported_seq":675,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.648862+0000","last_change":"2024-03-09T14:28:32.322792+0000","last_active":"2024-03-09T15:36:12.648862+0000","last_peered":"2024-03-09T15:36:12.648862+0000","last_clean":"2024-03-09T15:36:12.648862+0000","last_became_active":"2024-03-09T14:28:32.322364+0000","last_became_peered":"2024-03-09T14:28:32.322364+0000","last_unstale":"2024-03-09T15:36:12.648862+0000","last_undegraded":"2024-03-09T15:36:12.648862+0000","last_fullsized":"2024-03-09T15:36:12.648862+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":128,"num_write":389,"num_write_kb":27813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":512,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.576225+0000","last_change":"2024-03-09T15:36:25.576277+0000","last_active":"2024-03-09T15:36:25.576225+0000","last_peered":"2024-03-09T15:36:25.576225+0000","last_clean":"2024-03-09T15:36:25.576225+0000","last_became_active":"2024-03-09T14:28:36.313984+0000","last_became_peered":"2024-03-09T14:28:36.313984+0000","last_unstale":"2024-03-09T15:36:25.576225+0000","last_undegraded":"2024-03-09T15:36:25.576225+0000","last_fullsized":"2024-03-09T15:36:25.576225+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.576027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.576027+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.576027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:16:03.346315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"261'191","reported_seq":775,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.186294+0000","last_change":"2024-03-09T14:28:38.345412+0000","last_active":"2024-03-09T15:36:12.186294+0000","last_peered":"2024-03-09T15:36:12.186294+0000","last_clean":"2024-03-09T15:36:12.186294+0000","last_became_active":"2024-03-09T14:28:38.338262+0000","last_became_peered":"2024-03-09T14:28:38.338262+0000","last_unstale":"2024-03-09T15:36:12.186294+0000","last_undegraded":"2024-03-09T15:36:12.186294+0000","last_fullsized":"2024-03-09T15:36:12.186294+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":100,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15994,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":315,"num_read_kb":601,"num_write":220,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":491,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.186465+0000","last_change":"2024-03-09T14:28:40.390067+0000","last_active":"2024-03-09T15:36:12.186465+0000","last_peered":"2024-03-09T15:36:12.186465+0000","last_clean":"2024-03-09T15:36:12.186465+0000","last_became_active":"2024-03-09T14:28:40.389277+0000","last_became_peered":"2024-03-09T14:28:40.389277+0000","last_unstale":"2024-03-09T15:36:12.186465+0000","last_undegraded":"2024-03-09T15:36:12.186465+0000","last_fullsized":"2024-03-09T15:36:12.186465+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"261'56","reported_seq":560,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.467550+0000","last_change":"2024-03-09T15:36:22.467616+0000","last_active":"2024-03-09T15:36:22.467550+0000","last_peered":"2024-03-09T15:36:22.467550+0000","last_clean":"2024-03-09T15:36:22.467550+0000","last_became_active":"2024-03-09T14:28:42.409229+0000","last_became_peered":"2024-03-09T14:28:42.409229+0000","last_unstale":"2024-03-09T15:36:22.467550+0000","last_undegraded":"2024-03-09T15:36:22.467550+0000","last_fullsized":"2024-03-09T15:36:22.467550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:22.467355+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:22.467355+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.467355+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:46:09.238574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":828,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":44,"num_write":116,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"261'78","reported_seq":640,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.558934+0000","last_change":"2024-03-09T15:36:26.558992+0000","last_active":"2024-03-09T15:36:26.558934+0000","last_peered":"2024-03-09T15:36:26.558934+0000","last_clean":"2024-03-09T15:36:26.558934+0000","last_became_active":"2024-03-09T14:28:34.320474+0000","last_became_peered":"2024-03-09T14:28:34.320474+0000","last_unstale":"2024-03-09T15:36:26.558934+0000","last_undegraded":"2024-03-09T15:36:26.558934+0000","last_fullsized":"2024-03-09T15:36:26.558934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'78","last_scrub_stamp":"2024-03-09T15:36:26.558741+0000","last_deep_scrub":"261'78","last_deep_scrub_stamp":"2024-03-09T15:36:26.558741+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.558741+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:58:25.865037+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":231,"num_write":128,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"261'159","reported_seq":793,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101678+0000","last_change":"2024-03-09T14:28:32.317900+0000","last_active":"2024-03-09T15:36:14.101678+0000","last_peered":"2024-03-09T15:36:14.101678+0000","last_clean":"2024-03-09T15:36:14.101678+0000","last_became_active":"2024-03-09T14:28:32.315355+0000","last_became_peered":"2024-03-09T14:28:32.315355+0000","last_unstale":"2024-03-09T15:36:14.101678+0000","last_undegraded":"2024-03-09T15:36:14.101678+0000","last_fullsized":"2024-03-09T15:36:14.101678+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":325,"num_read_kb":85989,"num_write":581,"num_write_kb":44811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.186025+0000","last_change":"2024-03-09T14:28:36.311910+0000","last_active":"2024-03-09T15:36:12.186025+0000","last_peered":"2024-03-09T15:36:12.186025+0000","last_clean":"2024-03-09T15:36:12.186025+0000","last_became_active":"2024-03-09T14:28:36.311487+0000","last_became_peered":"2024-03-09T14:28:36.311487+0000","last_unstale":"2024-03-09T15:36:12.186025+0000","last_undegraded":"2024-03-09T15:36:12.186025+0000","last_fullsized":"2024-03-09T15:36:12.186025+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"261'39","reported_seq":552,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739505+0000","last_change":"2024-03-09T14:28:38.333369+0000","last_active":"2024-03-09T15:36:10.739505+0000","last_peered":"2024-03-09T15:36:10.739505+0000","last_clean":"2024-03-09T15:36:10.739505+0000","last_became_active":"2024-03-09T14:28:38.332610+0000","last_became_peered":"2024-03-09T14:28:38.332610+0000","last_unstale":"2024-03-09T15:36:10.739505+0000","last_undegraded":"2024-03-09T15:36:10.739505+0000","last_fullsized":"2024-03-09T15:36:10.739505+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":41,"num_write":30,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":500,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.102405+0000","last_change":"2024-03-09T14:28:40.368708+0000","last_active":"2024-03-09T15:36:14.102405+0000","last_peered":"2024-03-09T15:36:14.102405+0000","last_clean":"2024-03-09T15:36:14.102405+0000","last_became_active":"2024-03-09T14:28:40.367795+0000","last_became_peered":"2024-03-09T14:28:40.367795+0000","last_unstale":"2024-03-09T15:36:14.102405+0000","last_undegraded":"2024-03-09T15:36:14.102405+0000","last_fullsized":"2024-03-09T15:36:14.102405+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"261'172","reported_seq":700,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.304337+0000","last_change":"2024-03-09T15:36:22.304413+0000","last_active":"2024-03-09T15:36:22.304337+0000","last_peered":"2024-03-09T15:36:22.304337+0000","last_clean":"2024-03-09T15:36:22.304337+0000","last_became_active":"2024-03-09T14:28:42.408913+0000","last_became_peered":"2024-03-09T14:28:42.408913+0000","last_unstale":"2024-03-09T15:36:22.304337+0000","last_undegraded":"2024-03-09T15:36:22.304337+0000","last_fullsized":"2024-03-09T15:36:22.304337+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'172","last_scrub_stamp":"2024-03-09T15:36:22.304104+0000","last_deep_scrub":"261'172","last_deep_scrub_stamp":"2024-03-09T15:36:22.304104+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.304104+0000","objects_scrubbed":2,"log_size":72,"log_dups_size":100,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:45:45.379355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":208,"num_write":351,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":398,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"261'54","reported_seq":595,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.185823+0000","last_change":"2024-03-09T14:28:34.331192+0000","last_active":"2024-03-09T15:36:12.185823+0000","last_peered":"2024-03-09T15:36:12.185823+0000","last_clean":"2024-03-09T15:36:12.185823+0000","last_became_active":"2024-03-09T14:28:34.330813+0000","last_became_peered":"2024-03-09T14:28:34.330813+0000","last_unstale":"2024-03-09T15:36:12.185823+0000","last_undegraded":"2024-03-09T15:36:12.185823+0000","last_fullsized":"2024-03-09T15:36:12.185823+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":220,"num_read_kb":178,"num_write":76,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"261'127","reported_seq":685,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.123789+0000","last_change":"2024-03-09T14:28:32.313709+0000","last_active":"2024-03-09T15:36:13.123789+0000","last_peered":"2024-03-09T15:36:13.123789+0000","last_clean":"2024-03-09T15:36:13.123789+0000","last_became_active":"2024-03-09T14:28:32.311843+0000","last_became_peered":"2024-03-09T14:28:32.311843+0000","last_unstale":"2024-03-09T15:36:13.123789+0000","last_undegraded":"2024-03-09T15:36:13.123789+0000","last_fullsized":"2024-03-09T15:36:13.123789+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":100,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655838,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":157,"num_read_kb":5348,"num_write":475,"num_write_kb":35385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.102229+0000","last_change":"2024-03-09T14:28:36.324124+0000","last_active":"2024-03-09T15:36:14.102229+0000","last_peered":"2024-03-09T15:36:14.102229+0000","last_clean":"2024-03-09T15:36:14.102229+0000","last_became_active":"2024-03-09T14:28:36.323335+0000","last_became_peered":"2024-03-09T14:28:36.323335+0000","last_unstale":"2024-03-09T15:36:14.102229+0000","last_undegraded":"2024-03-09T15:36:14.102229+0000","last_fullsized":"2024-03-09T15:36:14.102229+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"261'222","reported_seq":821,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.186232+0000","last_change":"2024-03-09T14:28:38.345760+0000","last_active":"2024-03-09T15:36:12.186232+0000","last_peered":"2024-03-09T15:36:12.186232+0000","last_clean":"2024-03-09T15:36:12.186232+0000","last_became_active":"2024-03-09T14:28:38.344951+0000","last_became_peered":"2024-03-09T14:28:38.344951+0000","last_unstale":"2024-03-09T15:36:12.186232+0000","last_undegraded":"2024-03-09T15:36:12.186232+0000","last_fullsized":"2024-03-09T15:36:12.186232+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":200,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7791,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":353,"num_read_kb":629,"num_write":244,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.128626+0000","last_change":"2024-03-09T15:36:22.128680+0000","last_active":"2024-03-09T15:36:22.128626+0000","last_peered":"2024-03-09T15:36:22.128626+0000","last_clean":"2024-03-09T15:36:22.128626+0000","last_became_active":"2024-03-09T14:28:40.370313+0000","last_became_peered":"2024-03-09T14:28:40.370313+0000","last_unstale":"2024-03-09T15:36:22.128626+0000","last_undegraded":"2024-03-09T15:36:22.128626+0000","last_fullsized":"2024-03-09T15:36:22.128626+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.128436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.128436+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.128436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:31:27.821903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"261'47","reported_seq":550,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.629026+0000","last_change":"2024-03-09T15:36:15.629026+0000","last_active":"2024-03-09T15:36:15.629026+0000","last_peered":"2024-03-09T15:36:15.629026+0000","last_clean":"2024-03-09T15:36:15.629026+0000","last_became_active":"2024-03-09T14:28:42.404360+0000","last_became_peered":"2024-03-09T14:28:42.404360+0000","last_unstale":"2024-03-09T15:36:15.629026+0000","last_undegraded":"2024-03-09T15:36:15.629026+0000","last_fullsized":"2024-03-09T15:36:15.629026+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'47","last_scrub_stamp":"2024-03-09T15:36:15.628774+0000","last_deep_scrub":"261'47","last_deep_scrub_stamp":"2024-03-09T15:36:15.628774+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.628774+0000","objects_scrubbed":2,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:04:15.376099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":43,"num_write":116,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"261'87","reported_seq":642,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.185857+0000","last_change":"2024-03-09T14:28:34.313014+0000","last_active":"2024-03-09T15:36:12.185857+0000","last_peered":"2024-03-09T15:36:12.185857+0000","last_clean":"2024-03-09T15:36:12.185857+0000","last_became_active":"2024-03-09T14:28:34.305301+0000","last_became_peered":"2024-03-09T14:28:34.305301+0000","last_unstale":"2024-03-09T15:36:12.185857+0000","last_undegraded":"2024-03-09T15:36:12.185857+0000","last_fullsized":"2024-03-09T15:36:12.185857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":317,"num_write":145,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"261'143","reported_seq":701,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.738692+0000","last_change":"2024-03-09T14:28:32.293199+0000","last_active":"2024-03-09T15:36:10.738692+0000","last_peered":"2024-03-09T15:36:10.738692+0000","last_clean":"2024-03-09T15:36:10.738692+0000","last_became_active":"2024-03-09T14:28:32.292393+0000","last_became_peered":"2024-03-09T14:28:32.292393+0000","last_unstale":"2024-03-09T15:36:10.738692+0000","last_undegraded":"2024-03-09T15:36:10.738692+0000","last_fullsized":"2024-03-09T15:36:10.738692+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":100,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6900333,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":188,"num_read_kb":14616,"num_write":468,"num_write_kb":23484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":511,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.124875+0000","last_change":"2024-03-09T15:36:21.124931+0000","last_active":"2024-03-09T15:36:21.124875+0000","last_peered":"2024-03-09T15:36:21.124875+0000","last_clean":"2024-03-09T15:36:21.124875+0000","last_became_active":"2024-03-09T14:28:36.328946+0000","last_became_peered":"2024-03-09T14:28:36.328946+0000","last_unstale":"2024-03-09T15:36:21.124875+0000","last_undegraded":"2024-03-09T15:36:21.124875+0000","last_fullsized":"2024-03-09T15:36:21.124875+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.124678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.124678+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.124678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:16:00.704022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"261'168","reported_seq":754,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.102589+0000","last_change":"2024-03-09T14:28:38.336043+0000","last_active":"2024-03-09T15:36:14.102589+0000","last_peered":"2024-03-09T15:36:14.102589+0000","last_clean":"2024-03-09T15:36:14.102589+0000","last_became_active":"2024-03-09T14:28:38.335107+0000","last_became_peered":"2024-03-09T14:28:38.335107+0000","last_unstale":"2024-03-09T15:36:14.102589+0000","last_undegraded":"2024-03-09T15:36:14.102589+0000","last_fullsized":"2024-03-09T15:36:14.102589+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":100,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9765,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":273,"num_read_kb":518,"num_write":186,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":495,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739754+0000","last_change":"2024-03-09T14:28:40.364509+0000","last_active":"2024-03-09T15:36:10.739754+0000","last_peered":"2024-03-09T15:36:10.739754+0000","last_clean":"2024-03-09T15:36:10.739754+0000","last_became_active":"2024-03-09T14:28:40.363778+0000","last_became_peered":"2024-03-09T14:28:40.363778+0000","last_unstale":"2024-03-09T15:36:10.739754+0000","last_undegraded":"2024-03-09T15:36:10.739754+0000","last_fullsized":"2024-03-09T15:36:10.739754+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"261'159","reported_seq":742,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.206354+0000","last_change":"2024-03-09T15:36:29.206413+0000","last_active":"2024-03-09T15:36:29.206354+0000","last_peered":"2024-03-09T15:36:29.206354+0000","last_clean":"2024-03-09T15:36:29.206354+0000","last_became_active":"2024-03-09T14:28:34.325031+0000","last_became_peered":"2024-03-09T14:28:34.325031+0000","last_unstale":"2024-03-09T15:36:29.206354+0000","last_undegraded":"2024-03-09T15:36:29.206354+0000","last_fullsized":"2024-03-09T15:36:29.206354+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'159","last_scrub_stamp":"2024-03-09T15:36:29.206170+0000","last_deep_scrub":"261'159","last_deep_scrub_stamp":"2024-03-09T15:36:29.206170+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.206170+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:05:19.071599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":584,"num_read_kb":461,"num_write":254,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"261'111","reported_seq":652,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.185582+0000","last_change":"2024-03-09T14:28:32.296634+0000","last_active":"2024-03-09T15:36:12.185582+0000","last_peered":"2024-03-09T15:36:12.185582+0000","last_clean":"2024-03-09T15:36:12.185582+0000","last_became_active":"2024-03-09T14:28:32.296179+0000","last_became_peered":"2024-03-09T14:28:32.296179+0000","last_unstale":"2024-03-09T15:36:12.185582+0000","last_undegraded":"2024-03-09T15:36:12.185582+0000","last_fullsized":"2024-03-09T15:36:12.185582+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263265,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":136,"num_read_kb":11385,"num_write":371,"num_write_kb":17181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.030256+0000","last_change":"2024-03-09T14:28:36.320059+0000","last_active":"2024-03-09T15:34:51.030256+0000","last_peered":"2024-03-09T15:34:51.030256+0000","last_clean":"2024-03-09T15:34:51.030256+0000","last_became_active":"2024-03-09T14:28:36.319702+0000","last_became_peered":"2024-03-09T14:28:36.319702+0000","last_unstale":"2024-03-09T15:34:51.030256+0000","last_undegraded":"2024-03-09T15:34:51.030256+0000","last_fullsized":"2024-03-09T15:34:51.030256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"261'312","reported_seq":953,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.994801+0000","last_change":"2024-03-09T14:28:38.333414+0000","last_active":"2024-03-09T15:36:10.994801+0000","last_peered":"2024-03-09T15:36:10.994801+0000","last_clean":"2024-03-09T15:36:10.994801+0000","last_became_active":"2024-03-09T14:28:38.332661+0000","last_became_peered":"2024-03-09T14:28:38.332661+0000","last_unstale":"2024-03-09T15:36:10.994801+0000","last_undegraded":"2024-03-09T15:36:10.994801+0000","last_fullsized":"2024-03-09T15:36:10.994801+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":300,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16427,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":565,"num_read_kb":1160,"num_write":446,"num_write_kb":361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.679444+0000","last_change":"2024-03-09T15:36:21.679497+0000","last_active":"2024-03-09T15:36:21.679444+0000","last_peered":"2024-03-09T15:36:21.679444+0000","last_clean":"2024-03-09T15:36:21.679444+0000","last_became_active":"2024-03-09T14:28:40.378829+0000","last_became_peered":"2024-03-09T14:28:40.378829+0000","last_unstale":"2024-03-09T15:36:21.679444+0000","last_undegraded":"2024-03-09T15:36:21.679444+0000","last_fullsized":"2024-03-09T15:36:21.679444+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.679245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.679245+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.679245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:45:45.724495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"261'126","reported_seq":635,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.487636+0000","last_change":"2024-03-09T15:36:26.487767+0000","last_active":"2024-03-09T15:36:26.487636+0000","last_peered":"2024-03-09T15:36:26.487636+0000","last_clean":"2024-03-09T15:36:26.487636+0000","last_became_active":"2024-03-09T14:28:42.392509+0000","last_became_peered":"2024-03-09T14:28:42.392509+0000","last_unstale":"2024-03-09T15:36:26.487636+0000","last_undegraded":"2024-03-09T15:36:26.487636+0000","last_fullsized":"2024-03-09T15:36:26.487636+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'126","last_scrub_stamp":"2024-03-09T15:36:26.487384+0000","last_deep_scrub":"261'126","last_deep_scrub_stamp":"2024-03-09T15:36:26.487384+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.487384+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":100,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:33:29.950896+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":643,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":173,"num_write":275,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"261'132","reported_seq":685,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.030230+0000","last_change":"2024-03-09T14:28:34.330979+0000","last_active":"2024-03-09T15:34:51.030230+0000","last_peered":"2024-03-09T15:34:51.030230+0000","last_clean":"2024-03-09T15:34:51.030230+0000","last_became_active":"2024-03-09T14:28:34.330686+0000","last_became_peered":"2024-03-09T14:28:34.330686+0000","last_unstale":"2024-03-09T15:34:51.030230+0000","last_undegraded":"2024-03-09T15:34:51.030230+0000","last_fullsized":"2024-03-09T15:34:51.030230+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":527,"num_read_kb":451,"num_write":212,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"261'133","reported_seq":673,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.223417+0000","last_change":"2024-03-09T14:28:32.301602+0000","last_active":"2024-03-09T15:36:11.223417+0000","last_peered":"2024-03-09T15:36:11.223417+0000","last_clean":"2024-03-09T15:36:11.223417+0000","last_became_active":"2024-03-09T14:28:32.301087+0000","last_became_peered":"2024-03-09T14:28:32.301087+0000","last_unstale":"2024-03-09T15:36:11.223417+0000","last_undegraded":"2024-03-09T15:36:11.223417+0000","last_fullsized":"2024-03-09T15:36:11.223417+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":137,"num_read_kb":3393,"num_write":460,"num_write_kb":23270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.130798+0000","last_change":"2024-03-09T15:36:23.130851+0000","last_active":"2024-03-09T15:36:23.130798+0000","last_peered":"2024-03-09T15:36:23.130798+0000","last_clean":"2024-03-09T15:36:23.130798+0000","last_became_active":"2024-03-09T14:28:36.328814+0000","last_became_peered":"2024-03-09T14:28:36.328814+0000","last_unstale":"2024-03-09T15:36:23.130798+0000","last_undegraded":"2024-03-09T15:36:23.130798+0000","last_fullsized":"2024-03-09T15:36:23.130798+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:23.130603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:23.130603+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.130603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:54:38.685950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"261'252","reported_seq":857,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.240670+0000","last_change":"2024-03-09T14:28:38.332894+0000","last_active":"2024-03-09T15:36:11.240670+0000","last_peered":"2024-03-09T15:36:11.240670+0000","last_clean":"2024-03-09T15:36:11.240670+0000","last_became_active":"2024-03-09T14:28:38.331930+0000","last_became_peered":"2024-03-09T14:28:38.331930+0000","last_unstale":"2024-03-09T15:36:11.240670+0000","last_undegraded":"2024-03-09T15:36:11.240670+0000","last_fullsized":"2024-03-09T15:36:11.240670+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":200,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21830,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":424,"num_read_kb":798,"num_write":342,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.994792+0000","last_change":"2024-03-09T14:28:40.367660+0000","last_active":"2024-03-09T15:36:10.994792+0000","last_peered":"2024-03-09T15:36:10.994792+0000","last_clean":"2024-03-09T15:36:10.994792+0000","last_became_active":"2024-03-09T14:28:40.366723+0000","last_became_peered":"2024-03-09T14:28:40.366723+0000","last_unstale":"2024-03-09T15:36:10.994792+0000","last_undegraded":"2024-03-09T15:36:10.994792+0000","last_fullsized":"2024-03-09T15:36:10.994792+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"261'108","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.613500+0000","last_change":"2024-03-09T14:28:42.395993+0000","last_active":"2024-03-09T15:36:13.613500+0000","last_peered":"2024-03-09T15:36:13.613500+0000","last_clean":"2024-03-09T15:36:13.613500+0000","last_became_active":"2024-03-09T14:28:42.395375+0000","last_became_peered":"2024-03-09T14:28:42.395375+0000","last_unstale":"2024-03-09T15:36:13.613500+0000","last_undegraded":"2024-03-09T15:36:13.613500+0000","last_fullsized":"2024-03-09T15:36:13.613500+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:41.350663+0000","last_clean_scrub_stamp":"2024-03-09T14:28:41.350663+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1199,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":139,"num_read_kb":136,"num_write":341,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"261'99","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.836325+0000","last_change":"2024-03-09T14:28:34.312670+0000","last_active":"2024-03-09T15:36:05.836325+0000","last_peered":"2024-03-09T15:36:05.836325+0000","last_clean":"2024-03-09T15:36:05.836325+0000","last_became_active":"2024-03-09T14:28:34.312297+0000","last_became_peered":"2024-03-09T14:28:34.312297+0000","last_unstale":"2024-03-09T15:36:05.836325+0000","last_undegraded":"2024-03-09T15:36:05.836325+0000","last_fullsized":"2024-03-09T15:36:05.836325+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":415,"num_read_kb":347,"num_write":146,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"261'116","reported_seq":685,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.525310+0000","last_change":"2024-03-09T14:28:32.314186+0000","last_active":"2024-03-09T15:36:06.525310+0000","last_peered":"2024-03-09T15:36:06.525310+0000","last_clean":"2024-03-09T15:36:06.525310+0000","last_became_active":"2024-03-09T14:28:32.312937+0000","last_became_peered":"2024-03-09T14:28:32.312937+0000","last_unstale":"2024-03-09T15:36:06.525310+0000","last_undegraded":"2024-03-09T15:36:06.525310+0000","last_fullsized":"2024-03-09T15:36:06.525310+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":100,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262175,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":159,"num_read_kb":47424,"num_write":430,"num_write_kb":24173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.023049+0000","last_change":"2024-03-09T14:28:36.316936+0000","last_active":"2024-03-09T15:36:06.023049+0000","last_peered":"2024-03-09T15:36:06.023049+0000","last_clean":"2024-03-09T15:36:06.023049+0000","last_became_active":"2024-03-09T14:28:36.316094+0000","last_became_peered":"2024-03-09T14:28:36.316094+0000","last_unstale":"2024-03-09T15:36:06.023049+0000","last_undegraded":"2024-03-09T15:36:06.023049+0000","last_fullsized":"2024-03-09T15:36:06.023049+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"261'191","reported_seq":778,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.513146+0000","last_change":"2024-03-09T14:28:38.596294+0000","last_active":"2024-03-09T15:36:06.513146+0000","last_peered":"2024-03-09T15:36:06.513146+0000","last_clean":"2024-03-09T15:36:06.513146+0000","last_became_active":"2024-03-09T14:28:38.595522+0000","last_became_peered":"2024-03-09T14:28:38.595522+0000","last_unstale":"2024-03-09T15:36:06.513146+0000","last_undegraded":"2024-03-09T15:36:06.513146+0000","last_fullsized":"2024-03-09T15:36:06.513146+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":100,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12058,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":311,"num_read_kb":573,"num_write":216,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.585640+0000","last_change":"2024-03-09T15:36:22.585690+0000","last_active":"2024-03-09T15:36:22.585640+0000","last_peered":"2024-03-09T15:36:22.585640+0000","last_clean":"2024-03-09T15:36:22.585640+0000","last_became_active":"2024-03-09T14:28:40.376549+0000","last_became_peered":"2024-03-09T14:28:40.376549+0000","last_unstale":"2024-03-09T15:36:22.585640+0000","last_undegraded":"2024-03-09T15:36:22.585640+0000","last_fullsized":"2024-03-09T15:36:22.585640+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.585447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.585447+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.585447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:45:35.078062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"261'45","reported_seq":550,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.666451+0000","last_change":"2024-03-09T15:36:19.666451+0000","last_active":"2024-03-09T15:36:19.666451+0000","last_peered":"2024-03-09T15:36:19.666451+0000","last_clean":"2024-03-09T15:36:19.666451+0000","last_became_active":"2024-03-09T14:28:42.394780+0000","last_became_peered":"2024-03-09T14:28:42.394780+0000","last_unstale":"2024-03-09T15:36:19.666451+0000","last_undegraded":"2024-03-09T15:36:19.666451+0000","last_fullsized":"2024-03-09T15:36:19.666451+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'45","last_scrub_stamp":"2024-03-09T15:36:19.666178+0000","last_deep_scrub":"261'45","last_deep_scrub_stamp":"2024-03-09T15:36:19.666178+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.666178+0000","objects_scrubbed":3,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:53:32.256553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":84,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"261'68","reported_seq":616,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.006577+0000","last_change":"2024-03-09T14:28:34.313519+0000","last_active":"2024-03-09T15:36:06.006577+0000","last_peered":"2024-03-09T15:36:06.006577+0000","last_clean":"2024-03-09T15:36:06.006577+0000","last_became_active":"2024-03-09T14:28:34.313125+0000","last_became_peered":"2024-03-09T14:28:34.313125+0000","last_unstale":"2024-03-09T15:36:06.006577+0000","last_undegraded":"2024-03-09T15:36:06.006577+0000","last_fullsized":"2024-03-09T15:36:06.006577+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"261'187","reported_seq":792,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.486344+0000","last_change":"2024-03-09T14:28:32.304248+0000","last_active":"2024-03-09T15:36:06.486344+0000","last_peered":"2024-03-09T15:36:06.486344+0000","last_clean":"2024-03-09T15:36:06.486344+0000","last_became_active":"2024-03-09T14:28:32.303835+0000","last_became_peered":"2024-03-09T14:28:32.303835+0000","last_unstale":"2024-03-09T15:36:06.486344+0000","last_undegraded":"2024-03-09T15:36:06.486344+0000","last_fullsized":"2024-03-09T15:36:06.486344+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":100,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":371,"num_read_kb":1431,"num_write":999,"num_write_kb":8832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.488481+0000","last_change":"2024-03-09T15:36:29.488628+0000","last_active":"2024-03-09T15:36:29.488481+0000","last_peered":"2024-03-09T15:36:29.488481+0000","last_clean":"2024-03-09T15:36:29.488481+0000","last_became_active":"2024-03-09T14:28:36.322060+0000","last_became_peered":"2024-03-09T14:28:36.322060+0000","last_unstale":"2024-03-09T15:36:29.488481+0000","last_undegraded":"2024-03-09T15:36:29.488481+0000","last_fullsized":"2024-03-09T15:36:29.488481+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:29.488248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:29.488248+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.488248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:40:34.293730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"261'96","reported_seq":646,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.151380+0000","last_change":"2024-03-09T15:36:24.151474+0000","last_active":"2024-03-09T15:36:24.151380+0000","last_peered":"2024-03-09T15:36:24.151380+0000","last_clean":"2024-03-09T15:36:24.151380+0000","last_became_active":"2024-03-09T14:28:38.595749+0000","last_became_peered":"2024-03-09T14:28:38.595749+0000","last_unstale":"2024-03-09T15:36:24.151380+0000","last_undegraded":"2024-03-09T15:36:24.151380+0000","last_fullsized":"2024-03-09T15:36:24.151380+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'96","last_scrub_stamp":"2024-03-09T15:36:24.151188+0000","last_deep_scrub":"261'96","last_deep_scrub_stamp":"2024-03-09T15:36:24.151188+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.151188+0000","objects_scrubbed":7,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:06:38.216926+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":132,"num_read_kb":127,"num_write":124,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11222,"num_omap_keys":30,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"261'489","reported_seq":2266,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.168974+0000","last_change":"2024-03-09T15:36:28.169035+0000","last_active":"2024-03-09T15:36:28.168974+0000","last_peered":"2024-03-09T15:36:28.168974+0000","last_clean":"2024-03-09T15:36:28.168974+0000","last_became_active":"2024-03-09T14:28:40.376946+0000","last_became_peered":"2024-03-09T14:28:40.376946+0000","last_unstale":"2024-03-09T15:36:28.168974+0000","last_undegraded":"2024-03-09T15:36:28.168974+0000","last_fullsized":"2024-03-09T15:36:28.168974+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'489","last_scrub_stamp":"2024-03-09T15:36:28.168784+0000","last_deep_scrub":"261'489","last_deep_scrub_stamp":"2024-03-09T15:36:28.168784+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.168784+0000","objects_scrubbed":1,"log_size":89,"log_dups_size":400,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:26:51.629900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.10","version":"261'45","reported_seq":559,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.174006+0000","last_change":"2024-03-09T15:36:31.174074+0000","last_active":"2024-03-09T15:36:31.174006+0000","last_peered":"2024-03-09T15:36:31.174006+0000","last_clean":"2024-03-09T15:36:31.174006+0000","last_became_active":"2024-03-09T14:28:42.397829+0000","last_became_peered":"2024-03-09T14:28:42.397829+0000","last_unstale":"2024-03-09T15:36:31.174006+0000","last_undegraded":"2024-03-09T15:36:31.174006+0000","last_fullsized":"2024-03-09T15:36:31.174006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'45","last_scrub_stamp":"2024-03-09T15:36:31.173791+0000","last_deep_scrub":"261'45","last_deep_scrub_stamp":"2024-03-09T15:36:31.173791+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.173791+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:44.989801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":97,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"261'144","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.022937+0000","last_change":"2024-03-09T14:28:34.321426+0000","last_active":"2024-03-09T15:36:06.022937+0000","last_peered":"2024-03-09T15:36:06.022937+0000","last_clean":"2024-03-09T15:36:06.022937+0000","last_became_active":"2024-03-09T14:28:34.319891+0000","last_became_peered":"2024-03-09T14:28:34.319891+0000","last_unstale":"2024-03-09T15:36:06.022937+0000","last_undegraded":"2024-03-09T15:36:06.022937+0000","last_fullsized":"2024-03-09T15:36:06.022937+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":571,"num_read_kb":433,"num_write":263,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"261'161","reported_seq":808,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.528583+0000","last_change":"2024-03-09T14:28:32.323880+0000","last_active":"2024-03-09T15:36:06.528583+0000","last_peered":"2024-03-09T15:36:06.528583+0000","last_clean":"2024-03-09T15:36:06.528583+0000","last_became_active":"2024-03-09T14:28:32.323198+0000","last_became_peered":"2024-03-09T14:28:32.323198+0000","last_unstale":"2024-03-09T15:36:06.528583+0000","last_undegraded":"2024-03-09T15:36:06.528583+0000","last_fullsized":"2024-03-09T15:36:06.528583+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4801550,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":406,"num_read_kb":2095,"num_write":652,"num_write_kb":25158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.239640+0000","last_change":"2024-03-09T14:28:36.329314+0000","last_active":"2024-03-09T15:36:06.239640+0000","last_peered":"2024-03-09T15:36:06.239640+0000","last_clean":"2024-03-09T15:36:06.239640+0000","last_became_active":"2024-03-09T14:28:36.328529+0000","last_became_peered":"2024-03-09T14:28:36.328529+0000","last_unstale":"2024-03-09T15:36:06.239640+0000","last_undegraded":"2024-03-09T15:36:06.239640+0000","last_fullsized":"2024-03-09T15:36:06.239640+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"261'224","reported_seq":828,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.124658+0000","last_change":"2024-03-09T14:28:38.335758+0000","last_active":"2024-03-09T15:36:13.124658+0000","last_peered":"2024-03-09T15:36:13.124658+0000","last_clean":"2024-03-09T15:36:13.124658+0000","last_became_active":"2024-03-09T14:28:38.334991+0000","last_became_peered":"2024-03-09T14:28:38.334991+0000","last_unstale":"2024-03-09T15:36:13.124658+0000","last_undegraded":"2024-03-09T15:36:13.124658+0000","last_fullsized":"2024-03-09T15:36:13.124658+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":200,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13314,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":373,"num_read_kb":687,"num_write":264,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":494,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739734+0000","last_change":"2024-03-09T14:28:40.368865+0000","last_active":"2024-03-09T15:36:10.739734+0000","last_peered":"2024-03-09T15:36:10.739734+0000","last_clean":"2024-03-09T15:36:10.739734+0000","last_became_active":"2024-03-09T14:28:40.368039+0000","last_became_peered":"2024-03-09T14:28:40.368039+0000","last_unstale":"2024-03-09T15:36:10.739734+0000","last_undegraded":"2024-03-09T15:36:10.739734+0000","last_fullsized":"2024-03-09T15:36:10.739734+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:39.328032+0000","last_clean_scrub_stamp":"2024-03-09T14:28:39.328032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"261'79","reported_seq":609,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.203923+0000","last_change":"2024-03-09T15:36:28.203997+0000","last_active":"2024-03-09T15:36:28.203923+0000","last_peered":"2024-03-09T15:36:28.203923+0000","last_clean":"2024-03-09T15:36:28.203923+0000","last_became_active":"2024-03-09T14:28:42.413840+0000","last_became_peered":"2024-03-09T14:28:42.413840+0000","last_unstale":"2024-03-09T15:36:28.203923+0000","last_undegraded":"2024-03-09T15:36:28.203923+0000","last_fullsized":"2024-03-09T15:36:28.203923+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'79","last_scrub_stamp":"2024-03-09T15:36:28.203746+0000","last_deep_scrub":"261'79","last_deep_scrub_stamp":"2024-03-09T15:36:28.203746+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.203746+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:20:00.519578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":129,"num_write":187,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"261'36","reported_seq":574,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.022834+0000","last_change":"2024-03-09T14:28:34.318737+0000","last_active":"2024-03-09T15:36:06.022834+0000","last_peered":"2024-03-09T15:36:06.022834+0000","last_clean":"2024-03-09T15:36:06.022834+0000","last_became_active":"2024-03-09T14:28:34.314194+0000","last_became_peered":"2024-03-09T14:28:34.314194+0000","last_unstale":"2024-03-09T15:36:06.022834+0000","last_undegraded":"2024-03-09T15:36:06.022834+0000","last_fullsized":"2024-03-09T15:36:06.022834+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":120,"num_write":54,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"261'147","reported_seq":801,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.526571+0000","last_change":"2024-03-09T14:28:32.309431+0000","last_active":"2024-03-09T15:36:06.526571+0000","last_peered":"2024-03-09T15:36:06.526571+0000","last_clean":"2024-03-09T15:36:06.526571+0000","last_became_active":"2024-03-09T14:28:32.307702+0000","last_became_peered":"2024-03-09T14:28:32.307702+0000","last_unstale":"2024-03-09T15:36:06.526571+0000","last_undegraded":"2024-03-09T15:36:06.526571+0000","last_fullsized":"2024-03-09T15:36:06.526571+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":100,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":273,"num_read_kb":1758,"num_write":483,"num_write_kb":23078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.227766+0000","last_change":"2024-03-09T14:28:36.305006+0000","last_active":"2024-03-09T15:36:06.227766+0000","last_peered":"2024-03-09T15:36:06.227766+0000","last_clean":"2024-03-09T15:36:06.227766+0000","last_became_active":"2024-03-09T14:28:36.304241+0000","last_became_peered":"2024-03-09T14:28:36.304241+0000","last_unstale":"2024-03-09T15:36:06.227766+0000","last_undegraded":"2024-03-09T15:36:06.227766+0000","last_fullsized":"2024-03-09T15:36:06.227766+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"261'178","reported_seq":759,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739605+0000","last_change":"2024-03-09T14:28:38.340000+0000","last_active":"2024-03-09T15:36:10.739605+0000","last_peered":"2024-03-09T15:36:10.739605+0000","last_clean":"2024-03-09T15:36:10.739605+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:36:10.739605+0000","last_undegraded":"2024-03-09T15:36:10.739605+0000","last_fullsized":"2024-03-09T15:36:10.739605+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":100,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":521,"num_write":190,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.170977+0000","last_change":"2024-03-09T15:36:24.171034+0000","last_active":"2024-03-09T15:36:24.170977+0000","last_peered":"2024-03-09T15:36:24.170977+0000","last_clean":"2024-03-09T15:36:24.170977+0000","last_became_active":"2024-03-09T14:28:40.371820+0000","last_became_peered":"2024-03-09T14:28:40.371820+0000","last_unstale":"2024-03-09T15:36:24.170977+0000","last_undegraded":"2024-03-09T15:36:24.170977+0000","last_fullsized":"2024-03-09T15:36:24.170977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:24.170785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:24.170785+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.170785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:48:51.479283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.16","version":"261'56","reported_seq":580,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.501906+0000","last_change":"2024-03-09T15:36:28.501999+0000","last_active":"2024-03-09T15:36:28.501906+0000","last_peered":"2024-03-09T15:36:28.501906+0000","last_clean":"2024-03-09T15:36:28.501906+0000","last_became_active":"2024-03-09T14:28:42.392326+0000","last_became_peered":"2024-03-09T14:28:42.392326+0000","last_unstale":"2024-03-09T15:36:28.501906+0000","last_undegraded":"2024-03-09T15:36:28.501906+0000","last_fullsized":"2024-03-09T15:36:28.501906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:28.501691+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:28.501691+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.501691+0000","objects_scrubbed":3,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:17:33.874116+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":772,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":99,"num_write":117,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"261'70","reported_seq":634,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.185883+0000","last_change":"2024-03-09T14:28:34.324293+0000","last_active":"2024-03-09T15:36:12.185883+0000","last_peered":"2024-03-09T15:36:12.185883+0000","last_clean":"2024-03-09T15:36:12.185883+0000","last_became_active":"2024-03-09T14:28:34.323969+0000","last_became_peered":"2024-03-09T14:28:34.323969+0000","last_unstale":"2024-03-09T15:36:12.185883+0000","last_undegraded":"2024-03-09T15:36:12.185883+0000","last_fullsized":"2024-03-09T15:36:12.185883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":256,"num_write":98,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"261'176","reported_seq":768,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.521720+0000","last_change":"2024-03-09T14:28:32.338014+0000","last_active":"2024-03-09T15:36:06.521720+0000","last_peered":"2024-03-09T15:36:06.521720+0000","last_clean":"2024-03-09T15:36:06.521720+0000","last_became_active":"2024-03-09T14:28:32.337755+0000","last_became_peered":"2024-03-09T14:28:32.337755+0000","last_unstale":"2024-03-09T15:36:06.521720+0000","last_undegraded":"2024-03-09T15:36:06.521720+0000","last_fullsized":"2024-03-09T15:36:06.521720+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":100,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663535,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":299,"num_read_kb":3602,"num_write":667,"num_write_kb":11566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.649782+0000","last_change":"2024-03-09T14:28:36.318326+0000","last_active":"2024-03-09T15:36:12.649782+0000","last_peered":"2024-03-09T15:36:12.649782+0000","last_clean":"2024-03-09T15:36:12.649782+0000","last_became_active":"2024-03-09T14:28:36.317569+0000","last_became_peered":"2024-03-09T14:28:36.317569+0000","last_unstale":"2024-03-09T15:36:12.649782+0000","last_undegraded":"2024-03-09T15:36:12.649782+0000","last_fullsized":"2024-03-09T15:36:12.649782+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"261'152","reported_seq":713,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.837107+0000","last_change":"2024-03-09T14:28:38.348095+0000","last_active":"2024-03-09T15:36:10.837107+0000","last_peered":"2024-03-09T15:36:10.837107+0000","last_clean":"2024-03-09T15:36:10.837107+0000","last_became_active":"2024-03-09T14:28:38.347370+0000","last_became_peered":"2024-03-09T14:28:38.347370+0000","last_unstale":"2024-03-09T15:36:10.837107+0000","last_undegraded":"2024-03-09T15:36:10.837107+0000","last_fullsized":"2024-03-09T15:36:10.837107+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":100,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18257,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":531,"num_write":172,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.105659+0000","last_change":"2024-03-09T15:36:22.105715+0000","last_active":"2024-03-09T15:36:22.105659+0000","last_peered":"2024-03-09T15:36:22.105659+0000","last_clean":"2024-03-09T15:36:22.105659+0000","last_became_active":"2024-03-09T14:28:40.376513+0000","last_became_peered":"2024-03-09T14:28:40.376513+0000","last_unstale":"2024-03-09T15:36:22.105659+0000","last_undegraded":"2024-03-09T15:36:22.105659+0000","last_fullsized":"2024-03-09T15:36:22.105659+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.105441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.105441+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.105441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:26:26.970816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"261'96","reported_seq":988,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.177168+0000","last_change":"2024-03-09T15:36:30.177269+0000","last_active":"2024-03-09T15:36:30.177168+0000","last_peered":"2024-03-09T15:36:30.177168+0000","last_clean":"2024-03-09T15:36:30.177168+0000","last_became_active":"2024-03-09T14:28:42.409717+0000","last_became_peered":"2024-03-09T14:28:42.409717+0000","last_unstale":"2024-03-09T15:36:30.177168+0000","last_undegraded":"2024-03-09T15:36:30.177168+0000","last_fullsized":"2024-03-09T15:36:30.177168+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'96","last_scrub_stamp":"2024-03-09T15:36:30.176968+0000","last_deep_scrub":"261'96","last_deep_scrub_stamp":"2024-03-09T15:36:30.176968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.176968+0000","objects_scrubbed":4,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:07:18.249596+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":500,"num_read_kb":493,"num_write":215,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"261'53","reported_seq":587,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.649047+0000","last_change":"2024-03-09T14:28:34.314180+0000","last_active":"2024-03-09T15:36:12.649047+0000","last_peered":"2024-03-09T15:36:12.649047+0000","last_clean":"2024-03-09T15:36:12.649047+0000","last_became_active":"2024-03-09T14:28:34.313380+0000","last_became_peered":"2024-03-09T14:28:34.313380+0000","last_unstale":"2024-03-09T15:36:12.649047+0000","last_undegraded":"2024-03-09T15:36:12.649047+0000","last_fullsized":"2024-03-09T15:36:12.649047+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":197,"num_read_kb":156,"num_write":74,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"261'161","reported_seq":749,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.518065+0000","last_change":"2024-03-09T14:28:32.293553+0000","last_active":"2024-03-09T15:36:06.518065+0000","last_peered":"2024-03-09T15:36:06.518065+0000","last_clean":"2024-03-09T15:36:06.518065+0000","last_became_active":"2024-03-09T14:28:32.292862+0000","last_became_peered":"2024-03-09T14:28:32.292862+0000","last_unstale":"2024-03-09T15:36:06.518065+0000","last_undegraded":"2024-03-09T15:36:06.518065+0000","last_fullsized":"2024-03-09T15:36:06.518065+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":264,"num_read_kb":35693,"num_write":668,"num_write_kb":24072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"261'2","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.597930+0000","last_change":"2024-03-09T15:36:23.597930+0000","last_active":"2024-03-09T15:36:23.597930+0000","last_peered":"2024-03-09T15:36:23.597930+0000","last_clean":"2024-03-09T15:36:23.597930+0000","last_became_active":"2024-03-09T14:28:36.313405+0000","last_became_peered":"2024-03-09T14:28:36.313405+0000","last_unstale":"2024-03-09T15:36:23.597930+0000","last_undegraded":"2024-03-09T15:36:23.597930+0000","last_fullsized":"2024-03-09T15:36:23.597930+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:23.597678+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:23.597678+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.597678+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:11:34.369475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"261'213","reported_seq":819,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.171733+0000","last_change":"2024-03-09T15:36:30.171809+0000","last_active":"2024-03-09T15:36:30.171733+0000","last_peered":"2024-03-09T15:36:30.171733+0000","last_clean":"2024-03-09T15:36:30.171733+0000","last_became_active":"2024-03-09T14:28:38.330571+0000","last_became_peered":"2024-03-09T14:28:38.330571+0000","last_unstale":"2024-03-09T15:36:30.171733+0000","last_undegraded":"2024-03-09T15:36:30.171733+0000","last_fullsized":"2024-03-09T15:36:30.171733+0000","mapping_epoch":26,"log_start":"261'201","ondisk_log_start":"261'201","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'213","last_scrub_stamp":"2024-03-09T15:36:30.171534+0000","last_deep_scrub":"261'213","last_deep_scrub_stamp":"2024-03-09T15:36:30.171534+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.171534+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":201,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:50.639816+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11093,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":349,"num_read_kb":659,"num_write":246,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.213225+0000","last_change":"2024-03-09T15:36:29.213283+0000","last_active":"2024-03-09T15:36:29.213225+0000","last_peered":"2024-03-09T15:36:29.213225+0000","last_clean":"2024-03-09T15:36:29.213225+0000","last_became_active":"2024-03-09T14:28:40.385080+0000","last_became_peered":"2024-03-09T14:28:40.385080+0000","last_unstale":"2024-03-09T15:36:29.213225+0000","last_undegraded":"2024-03-09T15:36:29.213225+0000","last_fullsized":"2024-03-09T15:36:29.213225+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:29.213014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:29.213014+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.213014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:47.618516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"261'69","reported_seq":572,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.140878+0000","last_change":"2024-03-09T15:36:15.140935+0000","last_active":"2024-03-09T15:36:15.140878+0000","last_peered":"2024-03-09T15:36:15.140878+0000","last_clean":"2024-03-09T15:36:15.140878+0000","last_became_active":"2024-03-09T14:28:42.412616+0000","last_became_peered":"2024-03-09T14:28:42.412616+0000","last_unstale":"2024-03-09T15:36:15.140878+0000","last_undegraded":"2024-03-09T15:36:15.140878+0000","last_fullsized":"2024-03-09T15:36:15.140878+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'69","last_scrub_stamp":"2024-03-09T15:36:15.140656+0000","last_deep_scrub":"261'69","last_deep_scrub_stamp":"2024-03-09T15:36:15.140656+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.140656+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:26:05.406444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":49,"num_write":163,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.27","version":"261'50","reported_seq":598,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:45.990449+0000","last_change":"2024-03-09T14:28:34.307137+0000","last_active":"2024-03-09T15:35:45.990449+0000","last_peered":"2024-03-09T15:35:45.990449+0000","last_clean":"2024-03-09T15:35:45.990449+0000","last_became_active":"2024-03-09T14:28:34.306685+0000","last_became_peered":"2024-03-09T14:28:34.306685+0000","last_unstale":"2024-03-09T15:35:45.990449+0000","last_undegraded":"2024-03-09T15:35:45.990449+0000","last_fullsized":"2024-03-09T15:35:45.990449+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":204,"num_read_kb":164,"num_write":70,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"261'140","reported_seq":724,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.472400+0000","last_change":"2024-03-09T14:28:32.334098+0000","last_active":"2024-03-09T15:36:06.472400+0000","last_peered":"2024-03-09T15:36:06.472400+0000","last_clean":"2024-03-09T15:36:06.472400+0000","last_became_active":"2024-03-09T14:28:32.333008+0000","last_became_peered":"2024-03-09T14:28:32.333008+0000","last_unstale":"2024-03-09T15:36:06.472400+0000","last_undegraded":"2024-03-09T15:36:06.472400+0000","last_fullsized":"2024-03-09T15:36:06.472400+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":100,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655838,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":237,"num_read_kb":10390,"num_write":554,"num_write_kb":33266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.26","version":"261'181","reported_seq":781,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.520994+0000","last_change":"2024-03-09T15:36:30.521058+0000","last_active":"2024-03-09T15:36:30.520994+0000","last_peered":"2024-03-09T15:36:30.520994+0000","last_clean":"2024-03-09T15:36:30.520994+0000","last_became_active":"2024-03-09T14:28:34.314007+0000","last_became_peered":"2024-03-09T14:28:34.314007+0000","last_unstale":"2024-03-09T15:36:30.520994+0000","last_undegraded":"2024-03-09T15:36:30.520994+0000","last_fullsized":"2024-03-09T15:36:30.520994+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'181","last_scrub_stamp":"2024-03-09T15:36:30.520807+0000","last_deep_scrub":"261'181","last_deep_scrub_stamp":"2024-03-09T15:36:30.520807+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.520807+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":100,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:13:34.179786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":729,"num_read_kb":586,"num_write":272,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"261'155","reported_seq":713,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.244531+0000","last_change":"2024-03-09T15:36:29.244629+0000","last_active":"2024-03-09T15:36:29.244531+0000","last_peered":"2024-03-09T15:36:29.244531+0000","last_clean":"2024-03-09T15:36:29.244531+0000","last_became_active":"2024-03-09T14:28:32.286609+0000","last_became_peered":"2024-03-09T14:28:32.286609+0000","last_unstale":"2024-03-09T15:36:29.244531+0000","last_undegraded":"2024-03-09T15:36:29.244531+0000","last_fullsized":"2024-03-09T15:36:29.244531+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'155","last_scrub_stamp":"2024-03-09T15:36:29.244332+0000","last_deep_scrub":"261'155","last_deep_scrub_stamp":"2024-03-09T15:36:29.244332+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.244332+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":100,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:53:34.232406+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":810211,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":147,"num_read_kb":1907,"num_write":492,"num_write_kb":32192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"261'94","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.704648+0000","last_change":"2024-03-09T15:36:30.704712+0000","last_active":"2024-03-09T15:36:30.704648+0000","last_peered":"2024-03-09T15:36:30.704648+0000","last_clean":"2024-03-09T15:36:30.704648+0000","last_became_active":"2024-03-09T14:28:34.323541+0000","last_became_peered":"2024-03-09T14:28:34.323541+0000","last_unstale":"2024-03-09T15:36:30.704648+0000","last_undegraded":"2024-03-09T15:36:30.704648+0000","last_fullsized":"2024-03-09T15:36:30.704648+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'94","last_scrub_stamp":"2024-03-09T15:36:30.704449+0000","last_deep_scrub":"261'94","last_deep_scrub_stamp":"2024-03-09T15:36:30.704449+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.704449+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:11.739435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":384,"num_read_kb":306,"num_write":146,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":230,"num_omap_keys":1,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"261'171","reported_seq":768,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.519303+0000","last_change":"2024-03-09T14:28:32.299059+0000","last_active":"2024-03-09T15:36:06.519303+0000","last_peered":"2024-03-09T15:36:06.519303+0000","last_clean":"2024-03-09T15:36:06.519303+0000","last_became_active":"2024-03-09T14:28:32.295699+0000","last_became_peered":"2024-03-09T14:28:32.295699+0000","last_unstale":"2024-03-09T15:36:06.519303+0000","last_undegraded":"2024-03-09T15:36:06.519303+0000","last_fullsized":"2024-03-09T15:36:06.519303+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":100,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":297,"num_read_kb":14189,"num_write":716,"num_write_kb":33521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"261'101","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.552706+0000","last_change":"2024-03-09T15:36:24.552770+0000","last_active":"2024-03-09T15:36:24.552706+0000","last_peered":"2024-03-09T15:36:24.552706+0000","last_clean":"2024-03-09T15:36:24.552706+0000","last_became_active":"2024-03-09T14:28:34.321647+0000","last_became_peered":"2024-03-09T14:28:34.321647+0000","last_unstale":"2024-03-09T15:36:24.552706+0000","last_undegraded":"2024-03-09T15:36:24.552706+0000","last_fullsized":"2024-03-09T15:36:24.552706+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'101","last_scrub_stamp":"2024-03-09T15:36:24.552512+0000","last_deep_scrub":"261'101","last_deep_scrub_stamp":"2024-03-09T15:36:24.552512+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.552512+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:59:27.278086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":329,"num_write":161,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"261'128","reported_seq":681,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.494943+0000","last_change":"2024-03-09T14:28:32.298327+0000","last_active":"2024-03-09T15:36:06.494943+0000","last_peered":"2024-03-09T15:36:06.494943+0000","last_clean":"2024-03-09T15:36:06.494943+0000","last_became_active":"2024-03-09T14:28:32.297866+0000","last_became_peered":"2024-03-09T14:28:32.297866+0000","last_unstale":"2024-03-09T15:36:06.494943+0000","last_undegraded":"2024-03-09T15:36:06.494943+0000","last_fullsized":"2024-03-09T15:36:06.494943+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":163,"num_read_kb":3467,"num_write":495,"num_write_kb":23815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"261'52","reported_seq":602,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:30.828803+0000","last_change":"2024-03-09T14:28:34.330916+0000","last_active":"2024-03-09T15:35:30.828803+0000","last_peered":"2024-03-09T15:35:30.828803+0000","last_clean":"2024-03-09T15:35:30.828803+0000","last_became_active":"2024-03-09T14:28:34.330511+0000","last_became_peered":"2024-03-09T14:28:34.330511+0000","last_unstale":"2024-03-09T15:35:30.828803+0000","last_undegraded":"2024-03-09T15:35:30.828803+0000","last_fullsized":"2024-03-09T15:35:30.828803+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":225,"num_read_kb":185,"num_write":82,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"261'118","reported_seq":668,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.475009+0000","last_change":"2024-03-09T14:28:32.326489+0000","last_active":"2024-03-09T15:36:06.475009+0000","last_peered":"2024-03-09T15:36:06.475009+0000","last_clean":"2024-03-09T15:36:06.475009+0000","last_became_active":"2024-03-09T14:28:32.326148+0000","last_became_peered":"2024-03-09T14:28:32.326148+0000","last_unstale":"2024-03-09T15:36:06.475009+0000","last_undegraded":"2024-03-09T15:36:06.475009+0000","last_fullsized":"2024-03-09T15:36:06.475009+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":173,"num_read_kb":5614,"num_write":451,"num_write_kb":21169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"261'134","reported_seq":686,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:16.011461+0000","last_change":"2024-03-09T14:28:34.323012+0000","last_active":"2024-03-09T15:35:16.011461+0000","last_peered":"2024-03-09T15:35:16.011461+0000","last_clean":"2024-03-09T15:35:16.011461+0000","last_became_active":"2024-03-09T14:28:34.322225+0000","last_became_peered":"2024-03-09T14:28:34.322225+0000","last_unstale":"2024-03-09T15:35:16.011461+0000","last_undegraded":"2024-03-09T15:35:16.011461+0000","last_fullsized":"2024-03-09T15:35:16.011461+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":498,"num_read_kb":399,"num_write":194,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"261'171","reported_seq":738,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.526920+0000","last_change":"2024-03-09T14:28:32.300246+0000","last_active":"2024-03-09T15:36:06.526920+0000","last_peered":"2024-03-09T15:36:06.526920+0000","last_clean":"2024-03-09T15:36:06.526920+0000","last_became_active":"2024-03-09T14:28:32.299839+0000","last_became_peered":"2024-03-09T14:28:32.299839+0000","last_unstale":"2024-03-09T15:36:06.526920+0000","last_undegraded":"2024-03-09T15:36:06.526920+0000","last_fullsized":"2024-03-09T15:36:06.526920+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":100,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242922,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":18833,"num_write":606,"num_write_kb":39169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"261'74","reported_seq":609,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:15.825716+0000","last_change":"2024-03-09T14:28:34.315670+0000","last_active":"2024-03-09T15:35:15.825716+0000","last_peered":"2024-03-09T15:35:15.825716+0000","last_clean":"2024-03-09T15:35:15.825716+0000","last_became_active":"2024-03-09T14:28:34.315302+0000","last_became_peered":"2024-03-09T14:28:34.315302+0000","last_unstale":"2024-03-09T15:35:15.825716+0000","last_undegraded":"2024-03-09T15:35:15.825716+0000","last_fullsized":"2024-03-09T15:35:15.825716+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":224,"num_write":116,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"261'149","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.505120+0000","last_change":"2024-03-09T14:28:32.336827+0000","last_active":"2024-03-09T15:36:06.505120+0000","last_peered":"2024-03-09T15:36:06.505120+0000","last_clean":"2024-03-09T15:36:06.505120+0000","last_became_active":"2024-03-09T14:28:32.334117+0000","last_became_peered":"2024-03-09T14:28:32.334117+0000","last_unstale":"2024-03-09T15:36:06.505120+0000","last_undegraded":"2024-03-09T15:36:06.505120+0000","last_fullsized":"2024-03-09T15:36:06.505120+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":182,"num_read_kb":192,"num_write":563,"num_write_kb":14566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"261'48","reported_seq":607,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:15.810338+0000","last_change":"2024-03-09T14:28:34.323974+0000","last_active":"2024-03-09T15:35:15.810338+0000","last_peered":"2024-03-09T15:35:15.810338+0000","last_clean":"2024-03-09T15:35:15.810338+0000","last_became_active":"2024-03-09T14:28:34.323202+0000","last_became_peered":"2024-03-09T14:28:34.323202+0000","last_unstale":"2024-03-09T15:35:15.810338+0000","last_undegraded":"2024-03-09T15:35:15.810338+0000","last_fullsized":"2024-03-09T15:35:15.810338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":219,"num_read_kb":183,"num_write":70,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"261'138","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.519238+0000","last_change":"2024-03-09T14:28:32.300258+0000","last_active":"2024-03-09T15:36:06.519238+0000","last_peered":"2024-03-09T15:36:06.519238+0000","last_clean":"2024-03-09T15:36:06.519238+0000","last_became_active":"2024-03-09T14:28:32.299737+0000","last_became_peered":"2024-03-09T14:28:32.299737+0000","last_unstale":"2024-03-09T15:36:06.519238+0000","last_undegraded":"2024-03-09T15:36:06.519238+0000","last_fullsized":"2024-03-09T15:36:06.519238+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":100,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":168,"num_read_kb":12613,"num_write":521,"num_write_kb":31618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"261'94","reported_seq":646,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739086+0000","last_change":"2024-03-09T14:28:34.315866+0000","last_active":"2024-03-09T15:36:10.739086+0000","last_peered":"2024-03-09T15:36:10.739086+0000","last_clean":"2024-03-09T15:36:10.739086+0000","last_became_active":"2024-03-09T14:28:34.315451+0000","last_became_peered":"2024-03-09T14:28:34.315451+0000","last_unstale":"2024-03-09T15:36:10.739086+0000","last_undegraded":"2024-03-09T15:36:10.739086+0000","last_fullsized":"2024-03-09T15:36:10.739086+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":374,"num_read_kb":293,"num_write":145,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"261'144","reported_seq":711,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.509009+0000","last_change":"2024-03-09T14:28:32.305393+0000","last_active":"2024-03-09T15:36:06.509009+0000","last_peered":"2024-03-09T15:36:06.509009+0000","last_clean":"2024-03-09T15:36:06.509009+0000","last_became_active":"2024-03-09T14:28:32.304540+0000","last_became_peered":"2024-03-09T14:28:32.304540+0000","last_unstale":"2024-03-09T15:36:06.509009+0000","last_undegraded":"2024-03-09T15:36:06.509009+0000","last_fullsized":"2024-03-09T15:36:06.509009+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655899,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":200,"num_read_kb":1806,"num_write":509,"num_write_kb":18066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2e","version":"261'39","reported_seq":601,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.649349+0000","last_change":"2024-03-09T14:28:34.314876+0000","last_active":"2024-03-09T15:36:12.649349+0000","last_peered":"2024-03-09T15:36:12.649349+0000","last_clean":"2024-03-09T15:36:12.649349+0000","last_became_active":"2024-03-09T14:28:34.312399+0000","last_became_peered":"2024-03-09T14:28:34.312399+0000","last_unstale":"2024-03-09T15:36:12.649349+0000","last_undegraded":"2024-03-09T15:36:12.649349+0000","last_fullsized":"2024-03-09T15:36:12.649349+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":182,"num_read_kb":156,"num_write":57,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"261'145","reported_seq":689,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.495921+0000","last_change":"2024-03-09T14:28:32.314908+0000","last_active":"2024-03-09T15:36:06.495921+0000","last_peered":"2024-03-09T15:36:06.495921+0000","last_clean":"2024-03-09T15:36:06.495921+0000","last_became_active":"2024-03-09T14:28:32.314596+0000","last_became_peered":"2024-03-09T14:28:32.314596+0000","last_unstale":"2024-03-09T15:36:06.495921+0000","last_undegraded":"2024-03-09T15:36:06.495921+0000","last_fullsized":"2024-03-09T15:36:06.495921+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263241,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":148,"num_read_kb":12793,"num_write":461,"num_write_kb":35350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"261'64","reported_seq":627,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.709374+0000","last_change":"2024-03-09T14:28:34.325871+0000","last_active":"2024-03-09T15:36:11.709374+0000","last_peered":"2024-03-09T15:36:11.709374+0000","last_clean":"2024-03-09T15:36:11.709374+0000","last_became_active":"2024-03-09T14:28:34.325461+0000","last_became_peered":"2024-03-09T14:28:34.325461+0000","last_unstale":"2024-03-09T15:36:11.709374+0000","last_undegraded":"2024-03-09T15:36:11.709374+0000","last_fullsized":"2024-03-09T15:36:11.709374+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":264,"num_read_kb":214,"num_write":89,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"261'159","reported_seq":736,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.522767+0000","last_change":"2024-03-09T14:28:32.313085+0000","last_active":"2024-03-09T15:36:06.522767+0000","last_peered":"2024-03-09T15:36:06.522767+0000","last_clean":"2024-03-09T15:36:06.522767+0000","last_became_active":"2024-03-09T14:28:32.311750+0000","last_became_peered":"2024-03-09T14:28:32.311750+0000","last_unstale":"2024-03-09T15:36:06.522767+0000","last_undegraded":"2024-03-09T15:36:06.522767+0000","last_fullsized":"2024-03-09T15:36:06.522767+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":213,"num_read_kb":14896,"num_write":533,"num_write_kb":41774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"261'162","reported_seq":775,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.023023+0000","last_change":"2024-03-09T14:28:34.313344+0000","last_active":"2024-03-09T15:36:06.023023+0000","last_peered":"2024-03-09T15:36:06.023023+0000","last_clean":"2024-03-09T15:36:06.023023+0000","last_became_active":"2024-03-09T14:28:34.312212+0000","last_became_peered":"2024-03-09T14:28:34.312212+0000","last_unstale":"2024-03-09T15:36:06.023023+0000","last_undegraded":"2024-03-09T15:36:06.023023+0000","last_fullsized":"2024-03-09T15:36:06.023023+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":100,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":664,"num_read_kb":532,"num_write":243,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2b","version":"261'152","reported_seq":725,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.528226+0000","last_change":"2024-03-09T14:28:32.309913+0000","last_active":"2024-03-09T15:36:06.528226+0000","last_peered":"2024-03-09T15:36:06.528226+0000","last_clean":"2024-03-09T15:36:06.528226+0000","last_became_active":"2024-03-09T14:28:32.309562+0000","last_became_peered":"2024-03-09T14:28:32.309562+0000","last_unstale":"2024-03-09T15:36:06.528226+0000","last_undegraded":"2024-03-09T15:36:06.528226+0000","last_fullsized":"2024-03-09T15:36:06.528226+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":100,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":246,"num_read_kb":5348,"num_write":573,"num_write_kb":20887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"261'117","reported_seq":711,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.239657+0000","last_change":"2024-03-09T14:28:34.322800+0000","last_active":"2024-03-09T15:36:06.239657+0000","last_peered":"2024-03-09T15:36:06.239657+0000","last_clean":"2024-03-09T15:36:06.239657+0000","last_became_active":"2024-03-09T14:28:34.321424+0000","last_became_peered":"2024-03-09T14:28:34.321424+0000","last_unstale":"2024-03-09T15:36:06.239657+0000","last_undegraded":"2024-03-09T15:36:06.239657+0000","last_fullsized":"2024-03-09T15:36:06.239657+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":513,"num_read_kb":409,"num_write":169,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"261'162","reported_seq":731,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.507907+0000","last_change":"2024-03-09T14:28:32.334610+0000","last_active":"2024-03-09T15:36:06.507907+0000","last_peered":"2024-03-09T15:36:06.507907+0000","last_clean":"2024-03-09T15:36:06.507907+0000","last_became_active":"2024-03-09T14:28:32.333838+0000","last_became_peered":"2024-03-09T14:28:32.333838+0000","last_unstale":"2024-03-09T15:36:06.507907+0000","last_undegraded":"2024-03-09T15:36:06.507907+0000","last_fullsized":"2024-03-09T15:36:06.507907+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":100,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":238,"num_read_kb":3315,"num_write":673,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2a","version":"261'71","reported_seq":636,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.046616+0000","last_change":"2024-03-09T14:28:34.297744+0000","last_active":"2024-03-09T15:36:06.046616+0000","last_peered":"2024-03-09T15:36:06.046616+0000","last_clean":"2024-03-09T15:36:06.046616+0000","last_became_active":"2024-03-09T14:28:34.297042+0000","last_became_peered":"2024-03-09T14:28:34.297042+0000","last_unstale":"2024-03-09T15:36:06.046616+0000","last_undegraded":"2024-03-09T15:36:06.046616+0000","last_fullsized":"2024-03-09T15:36:06.046616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":242,"num_write":99,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"261'122","reported_seq":696,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101796+0000","last_change":"2024-03-09T14:28:32.320093+0000","last_active":"2024-03-09T15:36:14.101796+0000","last_peered":"2024-03-09T15:36:14.101796+0000","last_clean":"2024-03-09T15:36:14.101796+0000","last_became_active":"2024-03-09T14:28:32.317797+0000","last_became_peered":"2024-03-09T14:28:32.317797+0000","last_unstale":"2024-03-09T15:36:14.101796+0000","last_undegraded":"2024-03-09T15:36:14.101796+0000","last_fullsized":"2024-03-09T15:36:14.101796+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263201,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":187,"num_read_kb":4139,"num_write":475,"num_write_kb":19145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"261'113","reported_seq":669,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.836154+0000","last_change":"2024-03-09T14:28:34.314621+0000","last_active":"2024-03-09T15:36:05.836154+0000","last_peered":"2024-03-09T15:36:05.836154+0000","last_clean":"2024-03-09T15:36:05.836154+0000","last_became_active":"2024-03-09T14:28:34.314073+0000","last_became_peered":"2024-03-09T14:28:34.314073+0000","last_unstale":"2024-03-09T15:36:05.836154+0000","last_undegraded":"2024-03-09T15:36:05.836154+0000","last_fullsized":"2024-03-09T15:36:05.836154+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":100,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":445,"num_read_kb":351,"num_write":167,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"261'140","reported_seq":726,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.283437+0000","last_change":"2024-03-09T15:36:30.283533+0000","last_active":"2024-03-09T15:36:30.283437+0000","last_peered":"2024-03-09T15:36:30.283437+0000","last_clean":"2024-03-09T15:36:30.283437+0000","last_became_active":"2024-03-09T14:28:32.334187+0000","last_became_peered":"2024-03-09T14:28:32.334187+0000","last_unstale":"2024-03-09T15:36:30.283437+0000","last_undegraded":"2024-03-09T15:36:30.283437+0000","last_fullsized":"2024-03-09T15:36:30.283437+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'140","last_scrub_stamp":"2024-03-09T15:36:30.283226+0000","last_deep_scrub":"261'140","last_deep_scrub_stamp":"2024-03-09T15:36:30.283226+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.283226+0000","objects_scrubbed":6,"log_size":40,"log_dups_size":100,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:10:30.041775+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2181199,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":224,"num_read_kb":1842,"num_write":561,"num_write_kb":7884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"261'4028","reported_seq":4580,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:56.237823+0000","last_change":"2024-03-09T14:28:34.324195+0000","last_active":"2024-03-09T15:35:56.237823+0000","last_peered":"2024-03-09T15:35:56.237823+0000","last_clean":"2024-03-09T15:35:56.237823+0000","last_became_active":"2024-03-09T14:28:34.322421+0000","last_became_peered":"2024-03-09T14:28:34.322421+0000","last_unstale":"2024-03-09T15:35:56.237823+0000","last_undegraded":"2024-03-09T15:35:56.237823+0000","last_fullsized":"2024-03-09T15:35:56.237823+0000","mapping_epoch":22,"log_start":"261'4000","ondisk_log_start":"261'4000","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14144,"num_read_kb":11461,"num_write":6038,"num_write_kb":3015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"261'130","reported_seq":701,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.123960+0000","last_change":"2024-03-09T14:28:32.324991+0000","last_active":"2024-03-09T15:36:13.123960+0000","last_peered":"2024-03-09T15:36:13.123960+0000","last_clean":"2024-03-09T15:36:13.123960+0000","last_became_active":"2024-03-09T14:28:32.324166+0000","last_became_peered":"2024-03-09T14:28:32.324166+0000","last_unstale":"2024-03-09T15:36:13.123960+0000","last_undegraded":"2024-03-09T15:36:13.123960+0000","last_fullsized":"2024-03-09T15:36:13.123960+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269861,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":9428,"num_write":515,"num_write_kb":30713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"261'141","reported_seq":721,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.123943+0000","last_change":"2024-03-09T14:28:32.288697+0000","last_active":"2024-03-09T15:36:13.123943+0000","last_peered":"2024-03-09T15:36:13.123943+0000","last_clean":"2024-03-09T15:36:13.123943+0000","last_became_active":"2024-03-09T14:28:32.287937+0000","last_became_peered":"2024-03-09T14:28:32.287937+0000","last_unstale":"2024-03-09T15:36:13.123943+0000","last_undegraded":"2024-03-09T15:36:13.123943+0000","last_fullsized":"2024-03-09T15:36:13.123943+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":100,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243043,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":214,"num_read_kb":48289,"num_write":459,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"261'161","reported_seq":728,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739182+0000","last_change":"2024-03-09T14:28:34.317008+0000","last_active":"2024-03-09T15:36:10.739182+0000","last_peered":"2024-03-09T15:36:10.739182+0000","last_clean":"2024-03-09T15:36:10.739182+0000","last_became_active":"2024-03-09T14:28:34.316689+0000","last_became_peered":"2024-03-09T14:28:34.316689+0000","last_unstale":"2024-03-09T15:36:10.739182+0000","last_undegraded":"2024-03-09T15:36:10.739182+0000","last_fullsized":"2024-03-09T15:36:10.739182+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":634,"num_read_kb":490,"num_write":268,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"261'157","reported_seq":737,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.223488+0000","last_change":"2024-03-09T14:28:32.313446+0000","last_active":"2024-03-09T15:36:11.223488+0000","last_peered":"2024-03-09T15:36:11.223488+0000","last_clean":"2024-03-09T15:36:11.223488+0000","last_became_active":"2024-03-09T14:28:32.313132+0000","last_became_peered":"2024-03-09T14:28:32.313132+0000","last_unstale":"2024-03-09T15:36:11.223488+0000","last_undegraded":"2024-03-09T15:36:11.223488+0000","last_fullsized":"2024-03-09T15:36:11.223488+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262227,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":255,"num_read_kb":5401,"num_write":635,"num_write_kb":32648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"261'41","reported_seq":588,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.124418+0000","last_change":"2024-03-09T14:28:34.304440+0000","last_active":"2024-03-09T15:36:13.124418+0000","last_peered":"2024-03-09T15:36:13.124418+0000","last_clean":"2024-03-09T15:36:13.124418+0000","last_became_active":"2024-03-09T14:28:34.303701+0000","last_became_peered":"2024-03-09T14:28:34.303701+0000","last_unstale":"2024-03-09T15:36:13.124418+0000","last_undegraded":"2024-03-09T15:36:13.124418+0000","last_fullsized":"2024-03-09T15:36:13.124418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":166,"num_read_kb":138,"num_write":64,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":65,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3686,"num_read_kb":2614,"num_write":1236,"num_write_kb":514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6449,"num_omap_keys":13,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":13176,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":8518,"num_read_kb":8294,"num_write":8580,"num_write_kb":2546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2240,"num_omap_keys":13,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":26352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2076,"ondisk_log_size":2076,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":64,"acting":64,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":377960,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":12181,"num_read_kb":21090,"num_write":9023,"num_write_kb":5457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":257834,"num_omap_keys":640,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1734,"ondisk_log_size":1734,"up":64,"acting":64,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1719,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1602,"num_read_kb":1903,"num_write":22,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":38590,"num_read_kb":31236,"num_write":15314,"num_write_kb":7493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":230,"num_omap_keys":1,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3942,"ondisk_log_size":3942,"up":128,"acting":128,"num_store_stats":8},{"poolid":68,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":135300586,"num_objects":320,"num_object_clones":0,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":14057,"num_read_kb":1121153,"num_write":34295,"num_write_kb":1701153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":271892480,"data_stored":270601172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2879,"ondisk_log_size":2879,"up":128,"acting":128,"num_store_stats":8},{"poolid":67,"num_pg":32,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":140,"num_read_kb":192739,"num_write":156,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":66,"num_pg":32,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":138,"num_read_kb":192649,"num_write":156,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575676,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293320,"kb_used_data":45568,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":94078520,"statfs":{"total":96636764160,"available":96336404480,"internally_reserved":0,"allocated":46661632,"data_stored":45012346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":253688266},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575671,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":229800,"kb_used_data":22304,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":94142040,"statfs":{"total":96636764160,"available":96401448960,"internally_reserved":0,"allocated":22839296,"data_stored":21261512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575672,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132532,"kb_used_data":61164,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94239308,"statfs":{"total":96636764160,"available":96501051392,"internally_reserved":0,"allocated":62631936,"data_stored":60939198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575676,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100148,"kb_used_data":55084,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94271692,"statfs":{"total":96636764160,"available":96534212608,"internally_reserved":0,"allocated":56406016,"data_stored":54802077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575671,"num_pgs":118,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252776,"kb_used_data":41120,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":94119064,"statfs":{"total":96636764160,"available":96377921536,"internally_reserved":0,"allocated":42106880,"data_stored":40381748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":216725959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575673,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91428,"kb_used_data":38044,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94280412,"statfs":{"total":96636764160,"available":96543141888,"internally_reserved":0,"allocated":38957056,"data_stored":37358501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575672,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131376,"kb_used_data":41960,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94240464,"statfs":{"total":96636764160,"available":96502235136,"internally_reserved":0,"allocated":42967040,"data_stored":41342836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575674,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244644,"kb_used_data":55196,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":94127196,"statfs":{"total":96636764160,"available":96386248704,"internally_reserved":0,"allocated":56520704,"data_stored":54866964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40763392,"data_stored":40607042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23621632,"data_stored":23479247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30126080,"data_stored":29981692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31137792,"data_stored":30907070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":42364928,"data_stored":42183762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":48488448,"data_stored":48318049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":19247104,"data_stored":19128319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36143104,"data_stored":35995991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":3901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":2727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10485760,"data_stored":10485760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":1048576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15728640,"data_stored":15728640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":1048576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10485760,"data_stored":10485760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T15:36:35.046 INFO:tasks.ceph:pgid 68.1a last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.046 INFO:tasks.ceph:pgid 67.1d last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.046 INFO:tasks.ceph:pgid 68.1b last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.046 INFO:tasks.ceph:pgid 67.1c last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.047 INFO:tasks.ceph:pgid 68.18 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.047 INFO:tasks.ceph:pgid 67.1f last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.047 INFO:tasks.ceph:pgid 66.1e last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.047 INFO:tasks.ceph:pgid 68.19 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.047 INFO:tasks.ceph:pgid 67.1e last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.048 INFO:tasks.ceph:pgid 68.1e last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.048 INFO:tasks.ceph:pgid 67.19 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.048 INFO:tasks.ceph:pgid 68.1f last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.048 INFO:tasks.ceph:pgid 67.18 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.048 INFO:tasks.ceph:pgid 68.1c last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.048 INFO:tasks.ceph:pgid 67.1b last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.049 INFO:tasks.ceph:pgid 66.1a last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.049 INFO:tasks.ceph:pgid 68.1d last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.049 INFO:tasks.ceph:pgid 68.12 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.049 INFO:tasks.ceph:pgid 67.15 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.049 INFO:tasks.ceph:pgid 68.13 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.049 INFO:tasks.ceph:pgid 67.14 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.050 INFO:tasks.ceph:pgid 68.10 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.050 INFO:tasks.ceph:pgid 67.17 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.050 INFO:tasks.ceph:pgid 66.16 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.050 INFO:tasks.ceph:pgid 68.11 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.050 INFO:tasks.ceph:pgid 67.16 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.050 INFO:tasks.ceph:pgid 66.17 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.051 INFO:tasks.ceph:pgid 68.16 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.051 INFO:tasks.ceph:pgid 67.11 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.051 INFO:tasks.ceph:pgid 68.17 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.051 INFO:tasks.ceph:pgid 67.10 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.051 INFO:tasks.ceph:pgid 68.14 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.052 INFO:tasks.ceph:pgid 67.13 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.052 INFO:tasks.ceph:pgid 66.12 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.053 INFO:tasks.ceph:pgid 68.15 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.053 INFO:tasks.ceph:pgid 67.12 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.053 INFO:tasks.ceph:pgid 66.13 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.054 INFO:tasks.ceph:pgid 68.a last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.054 INFO:tasks.ceph:pgid 68.b last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.054 INFO:tasks.ceph:pgid 66.d last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.054 INFO:tasks.ceph:pgid 68.8 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.054 INFO:tasks.ceph:pgid 67.f last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.055 INFO:tasks.ceph:pgid 68.9 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.055 INFO:tasks.ceph:pgid 67.e last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.055 INFO:tasks.ceph:pgid 68.e last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.055 INFO:tasks.ceph:pgid 67.9 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.055 INFO:tasks.ceph:pgid 68.f last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.056 INFO:tasks.ceph:pgid 68.c last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.056 INFO:tasks.ceph:pgid 67.b last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.056 INFO:tasks.ceph:pgid 68.d last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.056 INFO:tasks.ceph:pgid 68.2 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.056 INFO:tasks.ceph:pgid 67.5 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.056 INFO:tasks.ceph:pgid 68.3 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.057 INFO:tasks.ceph:pgid 67.4 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.057 INFO:tasks.ceph:pgid 66.5 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.057 INFO:tasks.ceph:pgid 68.0 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.057 INFO:tasks.ceph:pgid 68.1 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.057 INFO:tasks.ceph:pgid 67.6 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.058 INFO:tasks.ceph:pgid 68.6 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.058 INFO:tasks.ceph:pgid 68.7 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.058 INFO:tasks.ceph:pgid 67.0 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.058 INFO:tasks.ceph:pgid 68.4 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.058 INFO:tasks.ceph:pgid 67.3 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.059 INFO:tasks.ceph:pgid 68.5 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.059 INFO:tasks.ceph:pgid 67.2 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.059 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.059 INFO:tasks.ceph:pgid 4.38 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.059 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.059 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.060 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.060 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.060 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.060 INFO:tasks.ceph:pgid 4.3b last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.060 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.060 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.061 INFO:tasks.ceph:pgid 4.3d last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.061 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.061 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.061 INFO:tasks.ceph:pgid 4.3f last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.061 INFO:tasks.ceph:pgid 4.30 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.061 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.062 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.062 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.062 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.062 INFO:tasks.ceph:pgid 4.33 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.062 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.062 INFO:tasks.ceph:pgid 4.34 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.063 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.063 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.063 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.063 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.063 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.064 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.064 INFO:tasks.ceph:pgid 7.17 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.064 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.064 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.064 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.064 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.065 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.065 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.065 INFO:tasks.ceph:pgid 7.15 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.065 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.065 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.065 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.066 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.066 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.066 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.066 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.066 INFO:tasks.ceph:pgid 7.b last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.067 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.067 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.067 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.067 INFO:tasks.ceph:pgid 8.5 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.067 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.068 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.068 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.068 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.068 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.068 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.069 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.069 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.069 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.069 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.069 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.070 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.070 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.070 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.070 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.070 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-09T14:28:11.179640+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=11, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.071 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.071 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.071 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.071 INFO:tasks.ceph:pgid 7.6 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.071 INFO:tasks.ceph:pgid 8.9 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.072 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.072 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.072 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.072 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.072 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.073 INFO:tasks.ceph:pgid 7.5 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.073 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.073 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.073 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.074 INFO:tasks.ceph:pgid 7.4 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.074 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.074 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.078 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.078 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.078 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.078 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.079 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.079 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.079 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.079 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.079 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.080 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.080 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.080 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.080 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.080 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.080 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.081 INFO:tasks.ceph:pgid 7.3 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.081 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.081 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.081 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.081 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.082 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.082 INFO:tasks.ceph:pgid 7.2 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.082 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.082 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.083 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.083 INFO:tasks.ceph:pgid 7.10 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.083 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.083 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.083 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.083 INFO:tasks.ceph:pgid 7.11 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.084 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.084 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.084 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.084 INFO:tasks.ceph:pgid 6.13 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.084 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.085 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.085 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.085 INFO:tasks.ceph:pgid 7.13 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.085 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.086 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.086 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.086 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.086 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.086 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.087 INFO:tasks.ceph:pgid 7.1d last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.087 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2024-03-09T14:28:41.350663+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=41, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.087 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.087 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.087 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.087 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.088 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.088 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.088 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.088 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.088 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.088 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.089 INFO:tasks.ceph:pgid 7.18 last_scrub_stamp 2024-03-09T14:28:39.328032+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=39, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.089 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.089 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.089 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.089 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.089 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.090 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.090 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.090 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.090 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.090 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.091 INFO:tasks.ceph:pgid 4.27 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.091 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.091 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.091 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.091 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.091 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.092 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.092 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.092 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.092 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.092 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.092 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.093 INFO:tasks.ceph:pgid 4.2f last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.093 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.093 INFO:tasks.ceph:pgid 4.2e last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.093 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.093 INFO:tasks.ceph:pgid 4.2d last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.093 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.094 INFO:tasks.ceph:pgid 4.2c last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.094 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.094 INFO:tasks.ceph:pgid 4.2b last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.094 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.094 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.094 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.095 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.095 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.095 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.095 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.095 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.095 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.096 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:35.096 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-09T15:36:55.097 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T15:36:55.488 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:36:55.501 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-09T15:36:55.510 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":2315,"stamp":"2024-03-09T15:36:55.282671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":169838241,"num_objects":629,"num_object_clones":0,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":0,"num_read":79040,"num_read_kb":1571790,"num_write":68959,"num_write_kb":1988151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":341156,"num_omap_keys":891,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12400,"ondisk_log_size":12400,"up":786,"acting":786,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":786,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1480184,"kb_used_data":360440,"kb_used_omap":12,"kb_used_meta":1119667,"kb_avail":753494536,"statfs":{"total":773094113280,"available":771578404864,"internally_reserved":0,"allocated":369090560,"data_stored":355965182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1146539568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17654,"num_omap_keys":57,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007972"},"pg_stats":[{"pgid":"68.1a","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.211347+0000","last_change":"2024-03-09T15:09:50.085348+0000","last_active":"2024-03-09T15:34:51.211347+0000","last_peered":"2024-03-09T15:34:51.211347+0000","last_clean":"2024-03-09T15:34:51.211347+0000","last_became_active":"2024-03-09T15:09:50.084784+0000","last_became_peered":"2024-03-09T15:09:50.084784+0000","last_unstale":"2024-03-09T15:34:51.211347+0000","last_undegraded":"2024-03-09T15:34:51.211347+0000","last_fullsized":"2024-03-09T15:34:51.211347+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.1d","version":"258'7","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.125221+0000","last_change":"2024-03-09T14:46:26.826294+0000","last_active":"2024-03-09T15:36:13.125221+0000","last_peered":"2024-03-09T15:36:13.125221+0000","last_clean":"2024-03-09T15:36:13.125221+0000","last_became_active":"2024-03-09T14:46:26.825404+0000","last_became_peered":"2024-03-09T14:46:26.825404+0000","last_unstale":"2024-03-09T15:36:13.125221+0000","last_undegraded":"2024-03-09T15:36:13.125221+0000","last_fullsized":"2024-03-09T15:36:13.125221+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20498,"num_write":6,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1c","version":"261'2","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.429218+0000","last_change":"2024-03-09T15:36:17.429218+0000","last_active":"2024-03-09T15:36:17.429218+0000","last_peered":"2024-03-09T15:36:17.429218+0000","last_clean":"2024-03-09T15:36:17.429218+0000","last_became_active":"2024-03-09T14:46:18.768383+0000","last_became_peered":"2024-03-09T14:46:18.768383+0000","last_unstale":"2024-03-09T15:36:17.429218+0000","last_undegraded":"2024-03-09T15:36:17.429218+0000","last_fullsized":"2024-03-09T15:36:17.429218+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:17.428960+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:17.428960+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.428960+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:03:37.237056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.1f","version":"258'45","reported_seq":176,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.184209+0000","last_change":"2024-03-09T15:36:12.184266+0000","last_active":"2024-03-09T15:36:12.184209+0000","last_peered":"2024-03-09T15:36:12.184209+0000","last_clean":"2024-03-09T15:36:12.184209+0000","last_became_active":"2024-03-09T14:39:13.281020+0000","last_became_peered":"2024-03-09T14:39:13.281020+0000","last_unstale":"2024-03-09T15:36:12.184209+0000","last_undegraded":"2024-03-09T15:36:12.184209+0000","last_fullsized":"2024-03-09T15:36:12.184209+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'45","last_scrub_stamp":"2024-03-09T15:36:12.183988+0000","last_deep_scrub":"258'45","last_deep_scrub_stamp":"2024-03-09T15:36:12.183988+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.183988+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:21:51.000354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":134,"num_write":70,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1b","version":"0'0","reported_seq":31,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.284563+0000","last_change":"2024-03-09T15:36:43.284617+0000","last_active":"2024-03-09T15:36:43.284563+0000","last_peered":"2024-03-09T15:36:43.284563+0000","last_clean":"2024-03-09T15:36:43.284563+0000","last_became_active":"2024-03-09T15:09:50.082690+0000","last_became_peered":"2024-03-09T15:09:50.082690+0000","last_unstale":"2024-03-09T15:36:43.284563+0000","last_undegraded":"2024-03-09T15:36:43.284563+0000","last_fullsized":"2024-03-09T15:36:43.284563+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:43.284378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:43.284378+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.284378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:49:16.785671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1c","version":"258'5","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.650521+0000","last_change":"2024-03-09T14:46:26.807670+0000","last_active":"2024-03-09T15:36:12.650521+0000","last_peered":"2024-03-09T15:36:12.650521+0000","last_clean":"2024-03-09T15:36:12.650521+0000","last_became_active":"2024-03-09T14:46:26.806865+0000","last_became_peered":"2024-03-09T14:46:26.806865+0000","last_unstale":"2024-03-09T15:36:12.650521+0000","last_undegraded":"2024-03-09T15:36:12.650521+0000","last_fullsized":"2024-03-09T15:36:12.650521+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.1d","version":"261'2","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.601517+0000","last_change":"2024-03-09T15:36:17.601517+0000","last_active":"2024-03-09T15:36:17.601517+0000","last_peered":"2024-03-09T15:36:17.601517+0000","last_clean":"2024-03-09T15:36:17.601517+0000","last_became_active":"2024-03-09T14:46:18.779760+0000","last_became_peered":"2024-03-09T14:46:18.779760+0000","last_unstale":"2024-03-09T15:36:17.601517+0000","last_undegraded":"2024-03-09T15:36:17.601517+0000","last_fullsized":"2024-03-09T15:36:17.601517+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:17.601237+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:17.601237+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.601237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:48:08.721381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1e","version":"0'0","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.210922+0000","last_change":"2024-03-09T15:36:22.210983+0000","last_active":"2024-03-09T15:36:22.210922+0000","last_peered":"2024-03-09T15:36:22.210922+0000","last_clean":"2024-03-09T15:36:22.210922+0000","last_became_active":"2024-03-09T14:39:13.259731+0000","last_became_peered":"2024-03-09T14:39:13.259731+0000","last_unstale":"2024-03-09T15:36:22.210922+0000","last_undegraded":"2024-03-09T15:36:22.210922+0000","last_fullsized":"2024-03-09T15:36:22.210922+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.210700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.210700+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.210700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:46:50.001547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.18","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.614119+0000","last_change":"2024-03-09T15:09:50.081224+0000","last_active":"2024-03-09T15:36:13.614119+0000","last_peered":"2024-03-09T15:36:13.614119+0000","last_clean":"2024-03-09T15:36:13.614119+0000","last_became_active":"2024-03-09T15:09:50.080578+0000","last_became_peered":"2024-03-09T15:09:50.080578+0000","last_unstale":"2024-03-09T15:36:13.614119+0000","last_undegraded":"2024-03-09T15:36:13.614119+0000","last_fullsized":"2024-03-09T15:36:13.614119+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.1f","version":"261'5","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.740388+0000","last_change":"2024-03-09T14:46:26.896619+0000","last_active":"2024-03-09T15:36:10.740388+0000","last_peered":"2024-03-09T15:36:10.740388+0000","last_clean":"2024-03-09T15:36:10.740388+0000","last_became_active":"2024-03-09T14:46:26.895608+0000","last_became_peered":"2024-03-09T14:46:26.895608+0000","last_unstale":"2024-03-09T15:36:10.740388+0000","last_undegraded":"2024-03-09T15:36:10.740388+0000","last_fullsized":"2024-03-09T15:36:10.740388+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":12288,"num_write":5,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1e","version":"0'0","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.247327+0000","last_change":"2024-03-09T15:36:39.247436+0000","last_active":"2024-03-09T15:36:39.247327+0000","last_peered":"2024-03-09T15:36:39.247327+0000","last_clean":"2024-03-09T15:36:39.247327+0000","last_became_active":"2024-03-09T14:46:18.781856+0000","last_became_peered":"2024-03-09T14:46:18.781856+0000","last_unstale":"2024-03-09T15:36:39.247327+0000","last_undegraded":"2024-03-09T15:36:39.247327+0000","last_fullsized":"2024-03-09T15:36:39.247327+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:39.247089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:39.247089+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.247089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:10:03.506121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.1d","version":"258'35","reported_seq":152,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.273912+0000","last_change":"2024-03-09T15:36:17.273912+0000","last_active":"2024-03-09T15:36:17.273912+0000","last_peered":"2024-03-09T15:36:17.273912+0000","last_clean":"2024-03-09T15:36:17.273912+0000","last_became_active":"2024-03-09T14:39:13.258351+0000","last_became_peered":"2024-03-09T14:39:13.258351+0000","last_unstale":"2024-03-09T15:36:17.273912+0000","last_undegraded":"2024-03-09T15:36:17.273912+0000","last_fullsized":"2024-03-09T15:36:17.273912+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'35","last_scrub_stamp":"2024-03-09T15:36:17.273654+0000","last_deep_scrub":"258'35","last_deep_scrub_stamp":"2024-03-09T15:36:17.273654+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.273654+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:22:59.569323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":152,"num_read_kb":109,"num_write":57,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.19","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.187003+0000","last_change":"2024-03-09T15:09:50.083658+0000","last_active":"2024-03-09T15:36:12.187003+0000","last_peered":"2024-03-09T15:36:12.187003+0000","last_clean":"2024-03-09T15:36:12.187003+0000","last_became_active":"2024-03-09T15:09:50.081680+0000","last_became_peered":"2024-03-09T15:09:50.081680+0000","last_unstale":"2024-03-09T15:36:12.187003+0000","last_undegraded":"2024-03-09T15:36:12.187003+0000","last_fullsized":"2024-03-09T15:36:12.187003+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.1e","version":"261'6","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.485897+0000","last_change":"2024-03-09T14:47:07.798858+0000","last_active":"2024-03-09T15:36:06.485897+0000","last_peered":"2024-03-09T15:36:06.485897+0000","last_clean":"2024-03-09T15:36:06.485897+0000","last_became_active":"2024-03-09T14:47:07.798247+0000","last_became_peered":"2024-03-09T14:47:07.798247+0000","last_unstale":"2024-03-09T15:36:06.485897+0000","last_undegraded":"2024-03-09T15:36:06.485897+0000","last_fullsized":"2024-03-09T15:36:06.485897+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":16384,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1f","version":"258'5","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.180362+0000","last_change":"2024-03-09T15:36:25.180427+0000","last_active":"2024-03-09T15:36:25.180362+0000","last_peered":"2024-03-09T15:36:25.180362+0000","last_clean":"2024-03-09T15:36:25.180362+0000","last_became_active":"2024-03-09T14:46:18.771731+0000","last_became_peered":"2024-03-09T14:46:18.771731+0000","last_unstale":"2024-03-09T15:36:25.180362+0000","last_undegraded":"2024-03-09T15:36:25.180362+0000","last_fullsized":"2024-03-09T15:36:25.180362+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'5","last_scrub_stamp":"2024-03-09T15:36:25.180164+0000","last_deep_scrub":"258'5","last_deep_scrub_stamp":"2024-03-09T15:36:25.180164+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.180164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:55:32.192979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5136,"num_write":5,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1c","version":"261'13","reported_seq":99,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.507564+0000","last_change":"2024-03-09T15:36:15.507564+0000","last_active":"2024-03-09T15:36:15.507564+0000","last_peered":"2024-03-09T15:36:15.507564+0000","last_clean":"2024-03-09T15:36:15.507564+0000","last_became_active":"2024-03-09T14:39:13.260945+0000","last_became_peered":"2024-03-09T14:39:13.260945+0000","last_unstale":"2024-03-09T15:36:15.507564+0000","last_undegraded":"2024-03-09T15:36:15.507564+0000","last_fullsized":"2024-03-09T15:36:15.507564+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'13","last_scrub_stamp":"2024-03-09T15:36:15.507291+0000","last_deep_scrub":"261'13","last_deep_scrub_stamp":"2024-03-09T15:36:15.507291+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.507291+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:29:48.463973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":27,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1e","version":"0'0","reported_seq":26,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:55.834070+0000","last_change":"2024-03-09T15:09:50.088230+0000","last_active":"2024-03-09T15:35:55.834070+0000","last_peered":"2024-03-09T15:35:55.834070+0000","last_clean":"2024-03-09T15:35:55.834070+0000","last_became_active":"2024-03-09T15:09:50.086789+0000","last_became_peered":"2024-03-09T15:09:50.086789+0000","last_unstale":"2024-03-09T15:35:55.834070+0000","last_undegraded":"2024-03-09T15:35:55.834070+0000","last_fullsized":"2024-03-09T15:35:55.834070+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.19","version":"261'6","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.157492+0000","last_change":"2024-03-09T14:46:26.819845+0000","last_active":"2024-03-09T15:36:06.157492+0000","last_peered":"2024-03-09T15:36:06.157492+0000","last_clean":"2024-03-09T15:36:06.157492+0000","last_became_active":"2024-03-09T14:46:26.819085+0000","last_became_peered":"2024-03-09T14:46:26.819085+0000","last_unstale":"2024-03-09T15:36:06.157492+0000","last_undegraded":"2024-03-09T15:36:06.157492+0000","last_fullsized":"2024-03-09T15:36:06.157492+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":12288,"num_write":11,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.18","version":"261'14","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.270716+0000","last_change":"2024-03-09T15:36:27.270917+0000","last_active":"2024-03-09T15:36:27.270716+0000","last_peered":"2024-03-09T15:36:27.270716+0000","last_clean":"2024-03-09T15:36:27.270716+0000","last_became_active":"2024-03-09T14:46:18.776161+0000","last_became_peered":"2024-03-09T14:46:18.776161+0000","last_unstale":"2024-03-09T15:36:27.270716+0000","last_undegraded":"2024-03-09T15:36:27.270716+0000","last_fullsized":"2024-03-09T15:36:27.270716+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'14","last_scrub_stamp":"2024-03-09T15:36:27.270455+0000","last_deep_scrub":"261'14","last_deep_scrub_stamp":"2024-03-09T15:36:27.270455+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.270455+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:23:13.389149+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":36889,"num_write":17,"num_write_kb":16407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.1b","version":"258'10","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.553151+0000","last_change":"2024-03-09T15:36:14.553151+0000","last_active":"2024-03-09T15:36:14.553151+0000","last_peered":"2024-03-09T15:36:14.553151+0000","last_clean":"2024-03-09T15:36:14.553151+0000","last_became_active":"2024-03-09T14:39:13.268179+0000","last_became_peered":"2024-03-09T14:39:13.268179+0000","last_unstale":"2024-03-09T15:36:14.553151+0000","last_undegraded":"2024-03-09T15:36:14.553151+0000","last_fullsized":"2024-03-09T15:36:14.553151+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:14.552885+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:14.552885+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.552885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:39.435007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1f","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:50.978735+0000","last_change":"2024-03-09T15:09:50.064176+0000","last_active":"2024-03-09T15:34:50.978735+0000","last_peered":"2024-03-09T15:34:50.978735+0000","last_clean":"2024-03-09T15:34:50.978735+0000","last_became_active":"2024-03-09T15:09:50.063705+0000","last_became_peered":"2024-03-09T15:09:50.063705+0000","last_unstale":"2024-03-09T15:34:50.978735+0000","last_undegraded":"2024-03-09T15:34:50.978735+0000","last_fullsized":"2024-03-09T15:34:50.978735+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.18","version":"261'3","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.314926+0000","last_change":"2024-03-09T15:36:48.314986+0000","last_active":"2024-03-09T15:36:48.314926+0000","last_peered":"2024-03-09T15:36:48.314926+0000","last_clean":"2024-03-09T15:36:48.314926+0000","last_became_active":"2024-03-09T14:46:26.824522+0000","last_became_peered":"2024-03-09T14:46:26.824522+0000","last_unstale":"2024-03-09T15:36:48.314926+0000","last_undegraded":"2024-03-09T15:36:48.314926+0000","last_fullsized":"2024-03-09T15:36:48.314926+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:48.314729+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:48.314729+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.314729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:53:25.103105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.19","version":"261'9","reported_seq":75,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.166246+0000","last_change":"2024-03-09T15:36:17.166321+0000","last_active":"2024-03-09T15:36:17.166246+0000","last_peered":"2024-03-09T15:36:17.166246+0000","last_clean":"2024-03-09T15:36:17.166246+0000","last_became_active":"2024-03-09T14:46:18.769730+0000","last_became_peered":"2024-03-09T14:46:18.769730+0000","last_unstale":"2024-03-09T15:36:17.166246+0000","last_undegraded":"2024-03-09T15:36:17.166246+0000","last_fullsized":"2024-03-09T15:36:17.166246+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-03-09T15:36:17.165994+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-03-09T15:36:17.165994+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.165994+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:56:10.015397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":19456,"num_write":19,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1a","version":"261'44","reported_seq":174,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.008442+0000","last_change":"2024-03-09T15:36:14.008442+0000","last_active":"2024-03-09T15:36:14.008442+0000","last_peered":"2024-03-09T15:36:14.008442+0000","last_clean":"2024-03-09T15:36:14.008442+0000","last_became_active":"2024-03-09T14:39:13.273298+0000","last_became_peered":"2024-03-09T14:39:13.273298+0000","last_unstale":"2024-03-09T15:36:14.008442+0000","last_undegraded":"2024-03-09T15:36:14.008442+0000","last_fullsized":"2024-03-09T15:36:14.008442+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:14.008154+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:14.008154+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.008154+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:19:56.477799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":138,"num_write":72,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.1c","version":"261'1","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.222862+0000","last_change":"2024-03-09T15:09:50.075720+0000","last_active":"2024-03-09T15:34:51.222862+0000","last_peered":"2024-03-09T15:34:51.222862+0000","last_clean":"2024-03-09T15:34:51.222862+0000","last_became_active":"2024-03-09T15:09:50.074952+0000","last_became_peered":"2024-03-09T15:09:50.074952+0000","last_unstale":"2024-03-09T15:34:51.222862+0000","last_undegraded":"2024-03-09T15:34:51.222862+0000","last_fullsized":"2024-03-09T15:34:51.222862+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1b","version":"261'3","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.334633+0000","last_change":"2024-03-09T15:36:46.334695+0000","last_active":"2024-03-09T15:36:46.334633+0000","last_peered":"2024-03-09T15:36:46.334633+0000","last_clean":"2024-03-09T15:36:46.334633+0000","last_became_active":"2024-03-09T14:46:26.804622+0000","last_became_peered":"2024-03-09T14:46:26.804622+0000","last_unstale":"2024-03-09T15:36:46.334633+0000","last_undegraded":"2024-03-09T15:36:46.334633+0000","last_fullsized":"2024-03-09T15:36:46.334633+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:46.334433+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:46.334433+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.334433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:30:10.876583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.1a","version":"261'5","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.650388+0000","last_change":"2024-03-09T14:46:18.785531+0000","last_active":"2024-03-09T15:36:12.650388+0000","last_peered":"2024-03-09T15:36:12.650388+0000","last_clean":"2024-03-09T15:36:12.650388+0000","last_became_active":"2024-03-09T14:46:18.784821+0000","last_became_peered":"2024-03-09T14:46:18.784821+0000","last_unstale":"2024-03-09T15:36:12.650388+0000","last_undegraded":"2024-03-09T15:36:12.650388+0000","last_fullsized":"2024-03-09T15:36:12.650388+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4096,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.19","version":"258'28","reported_seq":129,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.263878+0000","last_change":"2024-03-09T15:36:18.263945+0000","last_active":"2024-03-09T15:36:18.263878+0000","last_peered":"2024-03-09T15:36:18.263878+0000","last_clean":"2024-03-09T15:36:18.263878+0000","last_became_active":"2024-03-09T14:39:13.257473+0000","last_became_peered":"2024-03-09T14:39:13.257473+0000","last_unstale":"2024-03-09T15:36:18.263878+0000","last_undegraded":"2024-03-09T15:36:18.263878+0000","last_fullsized":"2024-03-09T15:36:18.263878+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'28","last_scrub_stamp":"2024-03-09T15:36:18.263646+0000","last_deep_scrub":"258'28","last_deep_scrub_stamp":"2024-03-09T15:36:18.263646+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.263646+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:59:14.805307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":78,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1d","version":"0'0","reported_seq":23,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.222939+0000","last_change":"2024-03-09T15:09:50.075294+0000","last_active":"2024-03-09T15:34:51.222939+0000","last_peered":"2024-03-09T15:34:51.222939+0000","last_clean":"2024-03-09T15:34:51.222939+0000","last_became_active":"2024-03-09T15:09:50.074875+0000","last_became_peered":"2024-03-09T15:09:50.074875+0000","last_unstale":"2024-03-09T15:34:51.222939+0000","last_undegraded":"2024-03-09T15:34:51.222939+0000","last_fullsized":"2024-03-09T15:34:51.222939+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1a","version":"261'11","reported_seq":72,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.456175+0000","last_change":"2024-03-09T15:36:21.456175+0000","last_active":"2024-03-09T15:36:21.456175+0000","last_peered":"2024-03-09T15:36:21.456175+0000","last_clean":"2024-03-09T15:36:21.456175+0000","last_became_active":"2024-03-09T14:46:26.812889+0000","last_became_peered":"2024-03-09T14:46:26.812889+0000","last_unstale":"2024-03-09T15:36:21.456175+0000","last_undegraded":"2024-03-09T15:36:21.456175+0000","last_fullsized":"2024-03-09T15:36:21.456175+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'11","last_scrub_stamp":"2024-03-09T15:36:21.455916+0000","last_deep_scrub":"261'11","last_deep_scrub_stamp":"2024-03-09T15:36:21.455916+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.455916+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:07:00.143287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":12290,"num_write":14,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.1b","version":"261'2","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.253249+0000","last_change":"2024-03-09T15:36:19.253249+0000","last_active":"2024-03-09T15:36:19.253249+0000","last_peered":"2024-03-09T15:36:19.253249+0000","last_clean":"2024-03-09T15:36:19.253249+0000","last_became_active":"2024-03-09T14:46:18.756394+0000","last_became_peered":"2024-03-09T14:46:18.756394+0000","last_unstale":"2024-03-09T15:36:19.253249+0000","last_undegraded":"2024-03-09T15:36:19.253249+0000","last_fullsized":"2024-03-09T15:36:19.253249+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:19.252989+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:19.252989+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.252989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:28:28.982640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":40,"num_write":2,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.18","version":"261'8","reported_seq":86,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.579241+0000","last_change":"2024-03-09T15:36:15.579306+0000","last_active":"2024-03-09T15:36:15.579241+0000","last_peered":"2024-03-09T15:36:15.579241+0000","last_clean":"2024-03-09T15:36:15.579241+0000","last_became_active":"2024-03-09T14:39:13.257996+0000","last_became_peered":"2024-03-09T14:39:13.257996+0000","last_unstale":"2024-03-09T15:36:15.579241+0000","last_undegraded":"2024-03-09T15:36:15.579241+0000","last_fullsized":"2024-03-09T15:36:15.579241+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:15.579034+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:15.579034+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.579034+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:42:49.650613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.12","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:30.987310+0000","last_change":"2024-03-09T15:09:50.070950+0000","last_active":"2024-03-09T15:35:30.987310+0000","last_peered":"2024-03-09T15:35:30.987310+0000","last_clean":"2024-03-09T15:35:30.987310+0000","last_became_active":"2024-03-09T15:09:50.070345+0000","last_became_peered":"2024-03-09T15:09:50.070345+0000","last_unstale":"2024-03-09T15:35:30.987310+0000","last_undegraded":"2024-03-09T15:35:30.987310+0000","last_fullsized":"2024-03-09T15:35:30.987310+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.15","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.552761+0000","last_change":"2024-03-09T15:36:41.552821+0000","last_active":"2024-03-09T15:36:41.552761+0000","last_peered":"2024-03-09T15:36:41.552761+0000","last_clean":"2024-03-09T15:36:41.552761+0000","last_became_active":"2024-03-09T14:46:26.890992+0000","last_became_peered":"2024-03-09T14:46:26.890992+0000","last_unstale":"2024-03-09T15:36:41.552761+0000","last_undegraded":"2024-03-09T15:36:41.552761+0000","last_fullsized":"2024-03-09T15:36:41.552761+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:41.552560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:41.552560+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.552560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:48:25.323567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.14","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.701752+0000","last_change":"2024-03-09T15:36:22.701809+0000","last_active":"2024-03-09T15:36:22.701752+0000","last_peered":"2024-03-09T15:36:22.701752+0000","last_clean":"2024-03-09T15:36:22.701752+0000","last_became_active":"2024-03-09T14:46:18.766777+0000","last_became_peered":"2024-03-09T14:46:18.766777+0000","last_unstale":"2024-03-09T15:36:22.701752+0000","last_undegraded":"2024-03-09T15:36:22.701752+0000","last_fullsized":"2024-03-09T15:36:22.701752+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.701528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.701528+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.701528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:03:22.022639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.17","version":"258'19","reported_seq":115,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.216612+0000","last_change":"2024-03-09T15:36:13.216647+0000","last_active":"2024-03-09T15:36:13.216612+0000","last_peered":"2024-03-09T15:36:13.216612+0000","last_clean":"2024-03-09T15:36:13.216612+0000","last_became_active":"2024-03-09T14:39:13.279919+0000","last_became_peered":"2024-03-09T14:39:13.279919+0000","last_unstale":"2024-03-09T15:36:13.216612+0000","last_undegraded":"2024-03-09T15:36:13.216612+0000","last_fullsized":"2024-03-09T15:36:13.216612+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'19","last_scrub_stamp":"2024-03-09T15:36:13.216485+0000","last_deep_scrub":"258'19","last_deep_scrub_stamp":"2024-03-09T15:36:13.216485+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.216485+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:22:26.193181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":64,"num_write":32,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.13","version":"261'2","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.296653+0000","last_change":"2024-03-09T15:36:41.296724+0000","last_active":"2024-03-09T15:36:41.296653+0000","last_peered":"2024-03-09T15:36:41.296653+0000","last_clean":"2024-03-09T15:36:41.296653+0000","last_became_active":"2024-03-09T15:09:50.079604+0000","last_became_peered":"2024-03-09T15:09:50.079604+0000","last_unstale":"2024-03-09T15:36:41.296653+0000","last_undegraded":"2024-03-09T15:36:41.296653+0000","last_fullsized":"2024-03-09T15:36:41.296653+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:41.296454+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:41.296454+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.296454+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:36:03.858346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.14","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:40.830834+0000","last_change":"2024-03-09T14:46:26.894898+0000","last_active":"2024-03-09T15:35:40.830834+0000","last_peered":"2024-03-09T15:35:40.830834+0000","last_clean":"2024-03-09T15:35:40.830834+0000","last_became_active":"2024-03-09T14:46:26.894088+0000","last_became_peered":"2024-03-09T14:46:26.894088+0000","last_unstale":"2024-03-09T15:35:40.830834+0000","last_undegraded":"2024-03-09T15:35:40.830834+0000","last_fullsized":"2024-03-09T15:35:40.830834+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.15","version":"0'0","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.232191+0000","last_change":"2024-03-09T15:36:16.232245+0000","last_active":"2024-03-09T15:36:16.232191+0000","last_peered":"2024-03-09T15:36:16.232191+0000","last_clean":"2024-03-09T15:36:16.232191+0000","last_became_active":"2024-03-09T14:46:18.752413+0000","last_became_peered":"2024-03-09T14:46:18.752413+0000","last_unstale":"2024-03-09T15:36:16.232191+0000","last_undegraded":"2024-03-09T15:36:16.232191+0000","last_fullsized":"2024-03-09T15:36:16.232191+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:16.232005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:16.232005+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.232005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:59:24.622129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.16","version":"261'52","reported_seq":193,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.459135+0000","last_change":"2024-03-09T15:36:16.459199+0000","last_active":"2024-03-09T15:36:16.459135+0000","last_peered":"2024-03-09T15:36:16.459135+0000","last_clean":"2024-03-09T15:36:16.459135+0000","last_became_active":"2024-03-09T14:39:13.262413+0000","last_became_peered":"2024-03-09T14:39:13.262413+0000","last_unstale":"2024-03-09T15:36:16.459135+0000","last_undegraded":"2024-03-09T15:36:16.459135+0000","last_fullsized":"2024-03-09T15:36:16.459135+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'52","last_scrub_stamp":"2024-03-09T15:36:16.458932+0000","last_deep_scrub":"261'52","last_deep_scrub_stamp":"2024-03-09T15:36:16.458932+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.458932+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:28:33.068618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":163,"num_write":78,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.10","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:31.231826+0000","last_change":"2024-03-09T15:09:50.075112+0000","last_active":"2024-03-09T15:35:31.231826+0000","last_peered":"2024-03-09T15:35:31.231826+0000","last_clean":"2024-03-09T15:35:31.231826+0000","last_became_active":"2024-03-09T15:09:50.074274+0000","last_became_peered":"2024-03-09T15:09:50.074274+0000","last_unstale":"2024-03-09T15:35:31.231826+0000","last_undegraded":"2024-03-09T15:35:31.231826+0000","last_fullsized":"2024-03-09T15:35:31.231826+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.17","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:50.820667+0000","last_change":"2024-03-09T14:46:26.894297+0000","last_active":"2024-03-09T15:34:50.820667+0000","last_peered":"2024-03-09T15:34:50.820667+0000","last_clean":"2024-03-09T15:34:50.820667+0000","last_became_active":"2024-03-09T14:46:26.893514+0000","last_became_peered":"2024-03-09T14:46:26.893514+0000","last_unstale":"2024-03-09T15:34:50.820667+0000","last_undegraded":"2024-03-09T15:34:50.820667+0000","last_fullsized":"2024-03-09T15:34:50.820667+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.16","version":"0'0","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.231978+0000","last_change":"2024-03-09T15:36:38.232040+0000","last_active":"2024-03-09T15:36:38.231978+0000","last_peered":"2024-03-09T15:36:38.231978+0000","last_clean":"2024-03-09T15:36:38.231978+0000","last_became_active":"2024-03-09T14:46:18.769274+0000","last_became_peered":"2024-03-09T14:46:18.769274+0000","last_unstale":"2024-03-09T15:36:38.231978+0000","last_undegraded":"2024-03-09T15:36:38.231978+0000","last_fullsized":"2024-03-09T15:36:38.231978+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:38.231785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:38.231785+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.231785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:50:40.645273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.15","version":"0'0","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.220144+0000","last_change":"2024-03-09T15:36:24.220202+0000","last_active":"2024-03-09T15:36:24.220144+0000","last_peered":"2024-03-09T15:36:24.220144+0000","last_clean":"2024-03-09T15:36:24.220144+0000","last_became_active":"2024-03-09T14:39:13.262066+0000","last_became_peered":"2024-03-09T14:39:13.262066+0000","last_unstale":"2024-03-09T15:36:24.220144+0000","last_undegraded":"2024-03-09T15:36:24.220144+0000","last_fullsized":"2024-03-09T15:36:24.220144+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:24.219947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:24.219947+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.219947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:56:44.705423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.11","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:31.220191+0000","last_change":"2024-03-09T15:09:50.086149+0000","last_active":"2024-03-09T15:35:31.220191+0000","last_peered":"2024-03-09T15:35:31.220191+0000","last_clean":"2024-03-09T15:35:31.220191+0000","last_became_active":"2024-03-09T15:09:50.085346+0000","last_became_peered":"2024-03-09T15:09:50.085346+0000","last_unstale":"2024-03-09T15:35:31.220191+0000","last_undegraded":"2024-03-09T15:35:31.220191+0000","last_fullsized":"2024-03-09T15:35:31.220191+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.16","version":"0'0","reported_seq":43,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:46.005141+0000","last_change":"2024-03-09T14:46:26.818659+0000","last_active":"2024-03-09T15:34:46.005141+0000","last_peered":"2024-03-09T15:34:46.005141+0000","last_clean":"2024-03-09T15:34:46.005141+0000","last_became_active":"2024-03-09T14:46:26.817864+0000","last_became_peered":"2024-03-09T14:46:26.817864+0000","last_unstale":"2024-03-09T15:34:46.005141+0000","last_undegraded":"2024-03-09T15:34:46.005141+0000","last_fullsized":"2024-03-09T15:34:46.005141+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.17","version":"0'0","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.208378+0000","last_change":"2024-03-09T15:36:31.208514+0000","last_active":"2024-03-09T15:36:31.208378+0000","last_peered":"2024-03-09T15:36:31.208378+0000","last_clean":"2024-03-09T15:36:31.208378+0000","last_became_active":"2024-03-09T14:46:18.776430+0000","last_became_peered":"2024-03-09T14:46:18.776430+0000","last_unstale":"2024-03-09T15:36:31.208378+0000","last_undegraded":"2024-03-09T15:36:31.208378+0000","last_fullsized":"2024-03-09T15:36:31.208378+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:31.208147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:31.208147+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.208147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:52:30.734801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.14","version":"261'68","reported_seq":280,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.178145+0000","last_change":"2024-03-09T15:36:24.178221+0000","last_active":"2024-03-09T15:36:24.178145+0000","last_peered":"2024-03-09T15:36:24.178145+0000","last_clean":"2024-03-09T15:36:24.178145+0000","last_became_active":"2024-03-09T14:39:13.253007+0000","last_became_peered":"2024-03-09T14:39:13.253007+0000","last_unstale":"2024-03-09T15:36:24.178145+0000","last_undegraded":"2024-03-09T15:36:24.178145+0000","last_fullsized":"2024-03-09T15:36:24.178145+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'68","last_scrub_stamp":"2024-03-09T15:36:24.177894+0000","last_deep_scrub":"261'68","last_deep_scrub_stamp":"2024-03-09T15:36:24.177894+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.177894+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:09:13.568231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":411,"num_read_kb":279,"num_write":87,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4473,"num_omap_keys":9,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.16","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.187173+0000","last_change":"2024-03-09T15:09:50.087752+0000","last_active":"2024-03-09T15:36:12.187173+0000","last_peered":"2024-03-09T15:36:12.187173+0000","last_clean":"2024-03-09T15:36:12.187173+0000","last_became_active":"2024-03-09T15:09:50.086173+0000","last_became_peered":"2024-03-09T15:09:50.086173+0000","last_unstale":"2024-03-09T15:36:12.187173+0000","last_undegraded":"2024-03-09T15:36:12.187173+0000","last_fullsized":"2024-03-09T15:36:12.187173+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.11","version":"261'8","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.163105+0000","last_change":"2024-03-09T15:36:50.163269+0000","last_active":"2024-03-09T15:36:50.163105+0000","last_peered":"2024-03-09T15:36:50.163105+0000","last_clean":"2024-03-09T15:36:50.163105+0000","last_became_active":"2024-03-09T14:46:26.817014+0000","last_became_peered":"2024-03-09T14:46:26.817014+0000","last_unstale":"2024-03-09T15:36:50.163105+0000","last_undegraded":"2024-03-09T15:36:50.163105+0000","last_fullsized":"2024-03-09T15:36:50.163105+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:50.162856+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:50.162856+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.162856+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:04:50.027298+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":23554,"num_write":11,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.10","version":"261'3","reported_seq":63,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.121627+0000","last_change":"2024-03-09T15:36:21.121690+0000","last_active":"2024-03-09T15:36:21.121627+0000","last_peered":"2024-03-09T15:36:21.121627+0000","last_clean":"2024-03-09T15:36:21.121627+0000","last_became_active":"2024-03-09T14:46:18.770936+0000","last_became_peered":"2024-03-09T14:46:18.770936+0000","last_unstale":"2024-03-09T15:36:21.121627+0000","last_undegraded":"2024-03-09T15:36:21.121627+0000","last_fullsized":"2024-03-09T15:36:21.121627+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:21.121408+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:21.121408+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.121408+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:44:45.507890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.13","version":"258'16","reported_seq":104,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.204733+0000","last_change":"2024-03-09T15:36:22.204801+0000","last_active":"2024-03-09T15:36:22.204733+0000","last_peered":"2024-03-09T15:36:22.204733+0000","last_clean":"2024-03-09T15:36:22.204733+0000","last_became_active":"2024-03-09T14:39:13.266228+0000","last_became_peered":"2024-03-09T14:39:13.266228+0000","last_unstale":"2024-03-09T15:36:22.204733+0000","last_undegraded":"2024-03-09T15:36:22.204733+0000","last_fullsized":"2024-03-09T15:36:22.204733+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'16","last_scrub_stamp":"2024-03-09T15:36:22.204502+0000","last_deep_scrub":"258'16","last_deep_scrub_stamp":"2024-03-09T15:36:22.204502+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.204502+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:33:31.523340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":45,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.17","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.740548+0000","last_change":"2024-03-09T15:09:50.075655+0000","last_active":"2024-03-09T15:36:10.740548+0000","last_peered":"2024-03-09T15:36:10.740548+0000","last_clean":"2024-03-09T15:36:10.740548+0000","last_became_active":"2024-03-09T15:09:50.075187+0000","last_became_peered":"2024-03-09T15:09:50.075187+0000","last_unstale":"2024-03-09T15:36:10.740548+0000","last_undegraded":"2024-03-09T15:36:10.740548+0000","last_fullsized":"2024-03-09T15:36:10.740548+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.10","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.190342+0000","last_change":"2024-03-09T15:36:48.190396+0000","last_active":"2024-03-09T15:36:48.190342+0000","last_peered":"2024-03-09T15:36:48.190342+0000","last_clean":"2024-03-09T15:36:48.190342+0000","last_became_active":"2024-03-09T14:46:26.802476+0000","last_became_peered":"2024-03-09T14:46:26.802476+0000","last_unstale":"2024-03-09T15:36:48.190342+0000","last_undegraded":"2024-03-09T15:36:48.190342+0000","last_fullsized":"2024-03-09T15:36:48.190342+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:48.190138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:48.190138+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.190138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:25.403833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.11","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.548131+0000","last_change":"2024-03-09T15:36:13.548167+0000","last_active":"2024-03-09T15:36:13.548131+0000","last_peered":"2024-03-09T15:36:13.548131+0000","last_clean":"2024-03-09T15:36:13.548131+0000","last_became_active":"2024-03-09T14:46:18.760959+0000","last_became_peered":"2024-03-09T14:46:18.760959+0000","last_unstale":"2024-03-09T15:36:13.548131+0000","last_undegraded":"2024-03-09T15:36:13.548131+0000","last_fullsized":"2024-03-09T15:36:13.548131+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:13.548016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:13.548016+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.548016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:46:22.640051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.12","version":"258'7","reported_seq":86,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.252904+0000","last_change":"2024-03-09T15:36:16.252969+0000","last_active":"2024-03-09T15:36:16.252904+0000","last_peered":"2024-03-09T15:36:16.252904+0000","last_clean":"2024-03-09T15:36:16.252904+0000","last_became_active":"2024-03-09T14:39:13.273740+0000","last_became_peered":"2024-03-09T14:39:13.273740+0000","last_unstale":"2024-03-09T15:36:16.252904+0000","last_undegraded":"2024-03-09T15:36:16.252904+0000","last_fullsized":"2024-03-09T15:36:16.252904+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'7","last_scrub_stamp":"2024-03-09T15:36:16.252708+0000","last_deep_scrub":"258'7","last_deep_scrub_stamp":"2024-03-09T15:36:16.252708+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.252708+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:42:28.881579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":26,"num_write":13,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.14","version":"260'1","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:36.040914+0000","last_change":"2024-03-09T15:09:50.065240+0000","last_active":"2024-03-09T15:35:36.040914+0000","last_peered":"2024-03-09T15:35:36.040914+0000","last_clean":"2024-03-09T15:35:36.040914+0000","last_became_active":"2024-03-09T15:09:50.064428+0000","last_became_peered":"2024-03-09T15:09:50.064428+0000","last_unstale":"2024-03-09T15:35:36.040914+0000","last_undegraded":"2024-03-09T15:35:36.040914+0000","last_fullsized":"2024-03-09T15:35:36.040914+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.13","version":"261'3","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.462155+0000","last_change":"2024-03-09T15:36:33.462284+0000","last_active":"2024-03-09T15:36:33.462155+0000","last_peered":"2024-03-09T15:36:33.462155+0000","last_clean":"2024-03-09T15:36:33.462155+0000","last_became_active":"2024-03-09T14:46:26.813088+0000","last_became_peered":"2024-03-09T14:46:26.813088+0000","last_unstale":"2024-03-09T15:36:33.462155+0000","last_undegraded":"2024-03-09T15:36:33.462155+0000","last_fullsized":"2024-03-09T15:36:33.462155+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:33.461831+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:33.461831+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.461831+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:23:33.077781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.12","version":"261'6","reported_seq":68,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.313727+0000","last_change":"2024-03-09T15:36:45.313892+0000","last_active":"2024-03-09T15:36:45.313727+0000","last_peered":"2024-03-09T15:36:45.313727+0000","last_clean":"2024-03-09T15:36:45.313727+0000","last_became_active":"2024-03-09T14:46:18.782158+0000","last_became_peered":"2024-03-09T14:46:18.782158+0000","last_unstale":"2024-03-09T15:36:45.313727+0000","last_undegraded":"2024-03-09T15:36:45.313727+0000","last_fullsized":"2024-03-09T15:36:45.313727+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:45.313488+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:45.313488+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.313488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:37.339820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":11264,"num_write":6,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.11","version":"258'36","reported_seq":157,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.155312+0000","last_change":"2024-03-09T15:36:14.155374+0000","last_active":"2024-03-09T15:36:14.155312+0000","last_peered":"2024-03-09T15:36:14.155312+0000","last_clean":"2024-03-09T15:36:14.155312+0000","last_became_active":"2024-03-09T14:39:13.260219+0000","last_became_peered":"2024-03-09T14:39:13.260219+0000","last_unstale":"2024-03-09T15:36:14.155312+0000","last_undegraded":"2024-03-09T15:36:14.155312+0000","last_fullsized":"2024-03-09T15:36:14.155312+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'36","last_scrub_stamp":"2024-03-09T15:36:14.155108+0000","last_deep_scrub":"258'36","last_deep_scrub_stamp":"2024-03-09T15:36:14.155108+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.155108+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:26:18.598712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":118,"num_write":57,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.15","version":"261'1","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.187041+0000","last_change":"2024-03-09T15:09:50.082323+0000","last_active":"2024-03-09T15:36:12.187041+0000","last_peered":"2024-03-09T15:36:12.187041+0000","last_clean":"2024-03-09T15:36:12.187041+0000","last_became_active":"2024-03-09T15:09:50.081688+0000","last_became_peered":"2024-03-09T15:09:50.081688+0000","last_unstale":"2024-03-09T15:36:12.187041+0000","last_undegraded":"2024-03-09T15:36:12.187041+0000","last_fullsized":"2024-03-09T15:36:12.187041+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.12","version":"261'2","reported_seq":55,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.350486+0000","last_change":"2024-03-09T15:36:47.350548+0000","last_active":"2024-03-09T15:36:47.350486+0000","last_peered":"2024-03-09T15:36:47.350486+0000","last_clean":"2024-03-09T15:36:47.350486+0000","last_became_active":"2024-03-09T14:46:26.806155+0000","last_became_peered":"2024-03-09T14:46:26.806155+0000","last_unstale":"2024-03-09T15:36:47.350486+0000","last_undegraded":"2024-03-09T15:36:47.350486+0000","last_fullsized":"2024-03-09T15:36:47.350486+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:47.350284+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:47.350284+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.350284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:49:11.128257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.13","version":"261'2","reported_seq":53,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.155249+0000","last_change":"2024-03-09T14:46:18.783226+0000","last_active":"2024-03-09T15:36:06.155249+0000","last_peered":"2024-03-09T15:36:06.155249+0000","last_clean":"2024-03-09T15:36:06.155249+0000","last_became_active":"2024-03-09T14:46:18.782340+0000","last_became_peered":"2024-03-09T14:46:18.782340+0000","last_unstale":"2024-03-09T15:36:06.155249+0000","last_undegraded":"2024-03-09T15:36:06.155249+0000","last_fullsized":"2024-03-09T15:36:06.155249+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:17.715683+0000","last_clean_scrub_stamp":"2024-03-09T14:46:17.715683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":50,"num_write":2,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.10","version":"261'68","reported_seq":266,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.437896+0000","last_change":"2024-03-09T15:36:19.437984+0000","last_active":"2024-03-09T15:36:19.437896+0000","last_peered":"2024-03-09T15:36:19.437896+0000","last_clean":"2024-03-09T15:36:19.437896+0000","last_became_active":"2024-03-09T14:39:13.245107+0000","last_became_peered":"2024-03-09T14:39:13.245107+0000","last_unstale":"2024-03-09T15:36:19.437896+0000","last_undegraded":"2024-03-09T15:36:19.437896+0000","last_fullsized":"2024-03-09T15:36:19.437896+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'68","last_scrub_stamp":"2024-03-09T15:36:19.437698+0000","last_deep_scrub":"261'68","last_deep_scrub_stamp":"2024-03-09T15:36:19.437698+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.437698+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:55:08.019711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":382,"num_read_kb":270,"num_write":93,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1976,"num_omap_keys":4,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.a","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.327061+0000","last_change":"2024-03-09T15:36:42.327115+0000","last_active":"2024-03-09T15:36:42.327061+0000","last_peered":"2024-03-09T15:36:42.327061+0000","last_clean":"2024-03-09T15:36:42.327061+0000","last_became_active":"2024-03-09T15:09:50.079554+0000","last_became_peered":"2024-03-09T15:09:50.079554+0000","last_unstale":"2024-03-09T15:36:42.327061+0000","last_undegraded":"2024-03-09T15:36:42.327061+0000","last_fullsized":"2024-03-09T15:36:42.327061+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:42.326870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:42.326870+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.326870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:01:30.715008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.d","version":"261'5","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.470652+0000","last_change":"2024-03-09T15:36:25.470799+0000","last_active":"2024-03-09T15:36:25.470652+0000","last_peered":"2024-03-09T15:36:25.470652+0000","last_clean":"2024-03-09T15:36:25.470652+0000","last_became_active":"2024-03-09T14:46:26.812859+0000","last_became_peered":"2024-03-09T14:46:26.812859+0000","last_unstale":"2024-03-09T15:36:25.470652+0000","last_undegraded":"2024-03-09T15:36:25.470652+0000","last_fullsized":"2024-03-09T15:36:25.470652+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'5","last_scrub_stamp":"2024-03-09T15:36:25.470415+0000","last_deep_scrub":"261'5","last_deep_scrub_stamp":"2024-03-09T15:36:25.470415+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.470415+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:20:04.740693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":8192,"num_write":10,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.c","version":"261'11","reported_seq":79,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.263874+0000","last_change":"2024-03-09T15:36:17.264004+0000","last_active":"2024-03-09T15:36:17.263874+0000","last_peered":"2024-03-09T15:36:17.263874+0000","last_clean":"2024-03-09T15:36:17.263874+0000","last_became_active":"2024-03-09T14:46:18.766370+0000","last_became_peered":"2024-03-09T14:46:18.766370+0000","last_unstale":"2024-03-09T15:36:17.263874+0000","last_undegraded":"2024-03-09T15:36:17.263874+0000","last_fullsized":"2024-03-09T15:36:17.263874+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'11","last_scrub_stamp":"2024-03-09T15:36:17.263641+0000","last_deep_scrub":"261'11","last_deep_scrub_stamp":"2024-03-09T15:36:17.263641+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.263641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:28:25.038359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":20480,"num_write":16,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.f","version":"258'18","reported_seq":108,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.541046+0000","last_change":"2024-03-09T15:36:12.541106+0000","last_active":"2024-03-09T15:36:12.541046+0000","last_peered":"2024-03-09T15:36:12.541046+0000","last_clean":"2024-03-09T15:36:12.541046+0000","last_became_active":"2024-03-09T14:39:13.268971+0000","last_became_peered":"2024-03-09T14:39:13.268971+0000","last_unstale":"2024-03-09T15:36:12.541046+0000","last_undegraded":"2024-03-09T15:36:12.541046+0000","last_fullsized":"2024-03-09T15:36:12.541046+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'18","last_scrub_stamp":"2024-03-09T15:36:12.540848+0000","last_deep_scrub":"258'18","last_deep_scrub_stamp":"2024-03-09T15:36:12.540848+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.540848+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:51.392572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.b","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.229996+0000","last_change":"2024-03-09T15:36:37.229996+0000","last_active":"2024-03-09T15:36:37.229996+0000","last_peered":"2024-03-09T15:36:37.229996+0000","last_clean":"2024-03-09T15:36:37.229996+0000","last_became_active":"2024-03-09T15:09:50.069508+0000","last_became_peered":"2024-03-09T15:09:50.069508+0000","last_unstale":"2024-03-09T15:36:37.229996+0000","last_undegraded":"2024-03-09T15:36:37.229996+0000","last_fullsized":"2024-03-09T15:36:37.229996+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:37.229744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:37.229744+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.229744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:14:18.843785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.c","version":"261'3","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.457377+0000","last_change":"2024-03-09T15:36:29.457444+0000","last_active":"2024-03-09T15:36:29.457377+0000","last_peered":"2024-03-09T15:36:29.457377+0000","last_clean":"2024-03-09T15:36:29.457377+0000","last_became_active":"2024-03-09T14:46:26.813009+0000","last_became_peered":"2024-03-09T14:46:26.813009+0000","last_unstale":"2024-03-09T15:36:29.457377+0000","last_undegraded":"2024-03-09T15:36:29.457377+0000","last_fullsized":"2024-03-09T15:36:29.457377+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:29.457166+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:29.457166+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.457166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:35:35.350489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2048,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.d","version":"261'3","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.167902+0000","last_change":"2024-03-09T15:36:36.167961+0000","last_active":"2024-03-09T15:36:36.167902+0000","last_peered":"2024-03-09T15:36:36.167902+0000","last_clean":"2024-03-09T15:36:36.167902+0000","last_became_active":"2024-03-09T14:46:18.770732+0000","last_became_peered":"2024-03-09T14:46:18.770732+0000","last_unstale":"2024-03-09T15:36:36.167902+0000","last_undegraded":"2024-03-09T15:36:36.167902+0000","last_fullsized":"2024-03-09T15:36:36.167902+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:36.167630+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:36.167630+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.167630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:24.866080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.e","version":"261'8","reported_seq":88,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.655370+0000","last_change":"2024-03-09T15:36:18.655429+0000","last_active":"2024-03-09T15:36:18.655370+0000","last_peered":"2024-03-09T15:36:18.655370+0000","last_clean":"2024-03-09T15:36:18.655370+0000","last_became_active":"2024-03-09T14:39:13.271672+0000","last_became_peered":"2024-03-09T14:39:13.271672+0000","last_unstale":"2024-03-09T15:36:18.655370+0000","last_undegraded":"2024-03-09T15:36:18.655370+0000","last_fullsized":"2024-03-09T15:36:18.655370+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:18.655167+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:18.655167+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.655167+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:25:51.301220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.8","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.370543+0000","last_change":"2024-03-09T15:36:50.370653+0000","last_active":"2024-03-09T15:36:50.370543+0000","last_peered":"2024-03-09T15:36:50.370543+0000","last_clean":"2024-03-09T15:36:50.370543+0000","last_became_active":"2024-03-09T15:09:50.061662+0000","last_became_peered":"2024-03-09T15:09:50.061662+0000","last_unstale":"2024-03-09T15:36:50.370543+0000","last_undegraded":"2024-03-09T15:36:50.370543+0000","last_fullsized":"2024-03-09T15:36:50.370543+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:50.370325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:50.370325+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.370325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:31:38.698974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.f","version":"261'6","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.109316+0000","last_change":"2024-03-09T14:46:26.900254+0000","last_active":"2024-03-09T15:36:06.109316+0000","last_peered":"2024-03-09T15:36:06.109316+0000","last_clean":"2024-03-09T15:36:06.109316+0000","last_became_active":"2024-03-09T14:46:26.899430+0000","last_became_peered":"2024-03-09T14:46:26.899430+0000","last_unstale":"2024-03-09T15:36:06.109316+0000","last_undegraded":"2024-03-09T15:36:06.109316+0000","last_fullsized":"2024-03-09T15:36:06.109316+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5120,"num_write":11,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.e","version":"258'3","reported_seq":65,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.271195+0000","last_change":"2024-03-09T15:36:15.271259+0000","last_active":"2024-03-09T15:36:15.271195+0000","last_peered":"2024-03-09T15:36:15.271195+0000","last_clean":"2024-03-09T15:36:15.271195+0000","last_became_active":"2024-03-09T14:46:18.768694+0000","last_became_peered":"2024-03-09T14:46:18.768694+0000","last_unstale":"2024-03-09T15:36:15.271195+0000","last_undegraded":"2024-03-09T15:36:15.271195+0000","last_fullsized":"2024-03-09T15:36:15.271195+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:15.270937+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:15.270937+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.270937+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:49:15.823574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.d","version":"261'43","reported_seq":168,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.940181+0000","last_change":"2024-03-09T15:36:15.940345+0000","last_active":"2024-03-09T15:36:15.940181+0000","last_peered":"2024-03-09T15:36:15.940181+0000","last_clean":"2024-03-09T15:36:15.940181+0000","last_became_active":"2024-03-09T14:39:13.257275+0000","last_became_peered":"2024-03-09T14:39:13.257275+0000","last_unstale":"2024-03-09T15:36:15.940181+0000","last_undegraded":"2024-03-09T15:36:15.940181+0000","last_fullsized":"2024-03-09T15:36:15.940181+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:15.939952+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:15.939952+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.939952+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:40:15.055734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":175,"num_read_kb":123,"num_write":70,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.9","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.236580+0000","last_change":"2024-03-09T15:09:50.077371+0000","last_active":"2024-03-09T15:35:51.236580+0000","last_peered":"2024-03-09T15:35:51.236580+0000","last_clean":"2024-03-09T15:35:51.236580+0000","last_became_active":"2024-03-09T15:09:50.076702+0000","last_became_peered":"2024-03-09T15:09:50.076702+0000","last_unstale":"2024-03-09T15:35:51.236580+0000","last_undegraded":"2024-03-09T15:35:51.236580+0000","last_fullsized":"2024-03-09T15:35:51.236580+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.e","version":"261'6","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.497081+0000","last_change":"2024-03-09T15:36:37.497144+0000","last_active":"2024-03-09T15:36:37.497081+0000","last_peered":"2024-03-09T15:36:37.497081+0000","last_clean":"2024-03-09T15:36:37.497081+0000","last_became_active":"2024-03-09T14:46:26.814709+0000","last_became_peered":"2024-03-09T14:46:26.814709+0000","last_unstale":"2024-03-09T15:36:37.497081+0000","last_undegraded":"2024-03-09T15:36:37.497081+0000","last_fullsized":"2024-03-09T15:36:37.497081+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:37.496875+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:37.496875+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.496875+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:32:45.549662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.f","version":"261'5","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.512325+0000","last_change":"2024-03-09T15:36:13.512372+0000","last_active":"2024-03-09T15:36:13.512325+0000","last_peered":"2024-03-09T15:36:13.512325+0000","last_clean":"2024-03-09T15:36:13.512325+0000","last_became_active":"2024-03-09T14:46:18.766000+0000","last_became_peered":"2024-03-09T14:46:18.766000+0000","last_unstale":"2024-03-09T15:36:13.512325+0000","last_undegraded":"2024-03-09T15:36:13.512325+0000","last_fullsized":"2024-03-09T15:36:13.512325+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'5","last_scrub_stamp":"2024-03-09T15:36:13.512169+0000","last_deep_scrub":"261'5","last_deep_scrub_stamp":"2024-03-09T15:36:13.512169+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.512169+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:07:18.735860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4096,"num_write":10,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.c","version":"0'0","reported_seq":68,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.206724+0000","last_change":"2024-03-09T15:36:26.206778+0000","last_active":"2024-03-09T15:36:26.206724+0000","last_peered":"2024-03-09T15:36:26.206724+0000","last_clean":"2024-03-09T15:36:26.206724+0000","last_became_active":"2024-03-09T14:39:13.270527+0000","last_became_peered":"2024-03-09T14:39:13.270527+0000","last_unstale":"2024-03-09T15:36:26.206724+0000","last_undegraded":"2024-03-09T15:36:26.206724+0000","last_fullsized":"2024-03-09T15:36:26.206724+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:26.206522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:26.206522+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.206522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:18:22.897656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.e","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.187069+0000","last_change":"2024-03-09T15:09:50.087358+0000","last_active":"2024-03-09T15:36:12.187069+0000","last_peered":"2024-03-09T15:36:12.187069+0000","last_clean":"2024-03-09T15:36:12.187069+0000","last_became_active":"2024-03-09T15:09:50.085548+0000","last_became_peered":"2024-03-09T15:09:50.085548+0000","last_unstale":"2024-03-09T15:36:12.187069+0000","last_undegraded":"2024-03-09T15:36:12.187069+0000","last_fullsized":"2024-03-09T15:36:12.187069+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.9","version":"0'0","reported_seq":46,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.650389+0000","last_change":"2024-03-09T14:46:26.802054+0000","last_active":"2024-03-09T15:36:12.650389+0000","last_peered":"2024-03-09T15:36:12.650389+0000","last_clean":"2024-03-09T15:36:12.650389+0000","last_became_active":"2024-03-09T14:46:26.801235+0000","last_became_peered":"2024-03-09T14:46:26.801235+0000","last_unstale":"2024-03-09T15:36:12.650389+0000","last_undegraded":"2024-03-09T15:36:12.650389+0000","last_fullsized":"2024-03-09T15:36:12.650389+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.8","version":"0'0","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.316818+0000","last_change":"2024-03-09T15:36:23.316871+0000","last_active":"2024-03-09T15:36:23.316818+0000","last_peered":"2024-03-09T15:36:23.316818+0000","last_clean":"2024-03-09T15:36:23.316818+0000","last_became_active":"2024-03-09T14:46:18.774547+0000","last_became_peered":"2024-03-09T14:46:18.774547+0000","last_unstale":"2024-03-09T15:36:23.316818+0000","last_undegraded":"2024-03-09T15:36:23.316818+0000","last_fullsized":"2024-03-09T15:36:23.316818+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:23.316611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:23.316611+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.316611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:09:52.305120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.b","version":"258'10","reported_seq":94,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.250282+0000","last_change":"2024-03-09T15:36:20.250347+0000","last_active":"2024-03-09T15:36:20.250282+0000","last_peered":"2024-03-09T15:36:20.250282+0000","last_clean":"2024-03-09T15:36:20.250282+0000","last_became_active":"2024-03-09T14:39:13.257238+0000","last_became_peered":"2024-03-09T14:39:13.257238+0000","last_unstale":"2024-03-09T15:36:20.250282+0000","last_undegraded":"2024-03-09T15:36:20.250282+0000","last_fullsized":"2024-03-09T15:36:20.250282+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:20.250091+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:20.250091+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.250091+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:55:12.169918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.f","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:41.220718+0000","last_change":"2024-03-09T15:09:50.071187+0000","last_active":"2024-03-09T15:34:41.220718+0000","last_peered":"2024-03-09T15:34:41.220718+0000","last_clean":"2024-03-09T15:34:41.220718+0000","last_became_active":"2024-03-09T15:09:50.070526+0000","last_became_peered":"2024-03-09T15:09:50.070526+0000","last_unstale":"2024-03-09T15:34:41.220718+0000","last_undegraded":"2024-03-09T15:34:41.220718+0000","last_fullsized":"2024-03-09T15:34:41.220718+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.8","version":"0'0","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.159602+0000","last_change":"2024-03-09T15:36:25.159656+0000","last_active":"2024-03-09T15:36:25.159602+0000","last_peered":"2024-03-09T15:36:25.159602+0000","last_clean":"2024-03-09T15:36:25.159602+0000","last_became_active":"2024-03-09T14:46:26.893908+0000","last_became_peered":"2024-03-09T14:46:26.893908+0000","last_unstale":"2024-03-09T15:36:25.159602+0000","last_undegraded":"2024-03-09T15:36:25.159602+0000","last_fullsized":"2024-03-09T15:36:25.159602+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.159401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.159401+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.159401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:51:03.092803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.9","version":"261'3","reported_seq":63,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.495189+0000","last_change":"2024-03-09T15:36:11.495211+0000","last_active":"2024-03-09T15:36:11.495189+0000","last_peered":"2024-03-09T15:36:11.495189+0000","last_clean":"2024-03-09T15:36:11.495189+0000","last_became_active":"2024-03-09T14:46:18.761367+0000","last_became_peered":"2024-03-09T14:46:18.761367+0000","last_unstale":"2024-03-09T15:36:11.495189+0000","last_undegraded":"2024-03-09T15:36:11.495189+0000","last_fullsized":"2024-03-09T15:36:11.495189+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:11.495111+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:11.495111+0000","last_clean_scrub_stamp":"2024-03-09T15:36:11.495111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:55:49.995490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.a","version":"258'23","reported_seq":132,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.129434+0000","last_change":"2024-03-09T15:36:26.129498+0000","last_active":"2024-03-09T15:36:26.129434+0000","last_peered":"2024-03-09T15:36:26.129434+0000","last_clean":"2024-03-09T15:36:26.129434+0000","last_became_active":"2024-03-09T14:39:13.252547+0000","last_became_peered":"2024-03-09T14:39:13.252547+0000","last_unstale":"2024-03-09T15:36:26.129434+0000","last_undegraded":"2024-03-09T15:36:26.129434+0000","last_fullsized":"2024-03-09T15:36:26.129434+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'23","last_scrub_stamp":"2024-03-09T15:36:26.129236+0000","last_deep_scrub":"258'23","last_deep_scrub_stamp":"2024-03-09T15:36:26.129236+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.129236+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:37:12.973344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":86,"num_write":38,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.c","version":"261'1","reported_seq":26,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.104285+0000","last_change":"2024-03-09T15:09:50.070468+0000","last_active":"2024-03-09T15:36:14.104285+0000","last_peered":"2024-03-09T15:36:14.104285+0000","last_clean":"2024-03-09T15:36:14.104285+0000","last_became_active":"2024-03-09T15:09:50.069324+0000","last_became_peered":"2024-03-09T15:09:50.069324+0000","last_unstale":"2024-03-09T15:36:14.104285+0000","last_undegraded":"2024-03-09T15:36:14.104285+0000","last_fullsized":"2024-03-09T15:36:14.104285+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.b","version":"258'3","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.176799+0000","last_change":"2024-03-09T15:36:49.176973+0000","last_active":"2024-03-09T15:36:49.176799+0000","last_peered":"2024-03-09T15:36:49.176799+0000","last_clean":"2024-03-09T15:36:49.176799+0000","last_became_active":"2024-03-09T14:46:26.804615+0000","last_became_peered":"2024-03-09T14:46:26.804615+0000","last_unstale":"2024-03-09T15:36:49.176799+0000","last_undegraded":"2024-03-09T15:36:49.176799+0000","last_fullsized":"2024-03-09T15:36:49.176799+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:49.176568+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:49.176568+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.176568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:20:16.951902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.a","version":"258'3","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.202430+0000","last_change":"2024-03-09T15:36:11.202430+0000","last_active":"2024-03-09T15:36:11.202430+0000","last_peered":"2024-03-09T15:36:11.202430+0000","last_clean":"2024-03-09T15:36:11.202430+0000","last_became_active":"2024-03-09T14:46:18.768431+0000","last_became_peered":"2024-03-09T14:46:18.768431+0000","last_unstale":"2024-03-09T15:36:11.202430+0000","last_undegraded":"2024-03-09T15:36:11.202430+0000","last_fullsized":"2024-03-09T15:36:11.202430+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:11.202170+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:11.202170+0000","last_clean_scrub_stamp":"2024-03-09T15:36:11.202170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:12:25.295523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.9","version":"261'43","reported_seq":166,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.428686+0000","last_change":"2024-03-09T15:36:18.428750+0000","last_active":"2024-03-09T15:36:18.428686+0000","last_peered":"2024-03-09T15:36:18.428686+0000","last_clean":"2024-03-09T15:36:18.428686+0000","last_became_active":"2024-03-09T14:39:13.259975+0000","last_became_peered":"2024-03-09T14:39:13.259975+0000","last_unstale":"2024-03-09T15:36:18.428686+0000","last_undegraded":"2024-03-09T15:36:18.428686+0000","last_fullsized":"2024-03-09T15:36:18.428686+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:18.428488+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:18.428488+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.428488+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:15:47.696457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":169,"num_read_kb":119,"num_write":74,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.d","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.740595+0000","last_change":"2024-03-09T15:09:50.084646+0000","last_active":"2024-03-09T15:36:10.740595+0000","last_peered":"2024-03-09T15:36:10.740595+0000","last_clean":"2024-03-09T15:36:10.740595+0000","last_became_active":"2024-03-09T15:09:50.084074+0000","last_became_peered":"2024-03-09T15:09:50.084074+0000","last_unstale":"2024-03-09T15:36:10.740595+0000","last_undegraded":"2024-03-09T15:36:10.740595+0000","last_fullsized":"2024-03-09T15:36:10.740595+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.a","version":"258'3","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.196067+0000","last_change":"2024-03-09T15:36:21.196067+0000","last_active":"2024-03-09T15:36:21.196067+0000","last_peered":"2024-03-09T15:36:21.196067+0000","last_clean":"2024-03-09T15:36:21.196067+0000","last_became_active":"2024-03-09T14:46:26.825949+0000","last_became_peered":"2024-03-09T14:46:26.825949+0000","last_unstale":"2024-03-09T15:36:21.196067+0000","last_undegraded":"2024-03-09T15:36:21.196067+0000","last_fullsized":"2024-03-09T15:36:21.196067+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:21.195795+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:21.195795+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.195795+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:51.756963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.b","version":"261'3","reported_seq":64,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.180486+0000","last_change":"2024-03-09T15:36:29.180546+0000","last_active":"2024-03-09T15:36:29.180486+0000","last_peered":"2024-03-09T15:36:29.180486+0000","last_clean":"2024-03-09T15:36:29.180486+0000","last_became_active":"2024-03-09T14:46:18.784869+0000","last_became_peered":"2024-03-09T14:46:18.784869+0000","last_unstale":"2024-03-09T15:36:29.180486+0000","last_undegraded":"2024-03-09T15:36:29.180486+0000","last_fullsized":"2024-03-09T15:36:29.180486+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:29.180292+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:29.180292+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.180292+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:09:41.856539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.8","version":"258'10","reported_seq":95,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.980744+0000","last_change":"2024-03-09T15:36:16.980807+0000","last_active":"2024-03-09T15:36:16.980744+0000","last_peered":"2024-03-09T15:36:16.980744+0000","last_clean":"2024-03-09T15:36:16.980744+0000","last_became_active":"2024-03-09T14:39:13.264006+0000","last_became_peered":"2024-03-09T14:39:13.264006+0000","last_unstale":"2024-03-09T15:36:16.980744+0000","last_undegraded":"2024-03-09T15:36:16.980744+0000","last_fullsized":"2024-03-09T15:36:16.980744+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:16.980539+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:16.980539+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.980539+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:50:44.529795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.2","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.235992+0000","last_change":"2024-03-09T15:36:33.236049+0000","last_active":"2024-03-09T15:36:33.235992+0000","last_peered":"2024-03-09T15:36:33.235992+0000","last_clean":"2024-03-09T15:36:33.235992+0000","last_became_active":"2024-03-09T15:09:50.065487+0000","last_became_peered":"2024-03-09T15:09:50.065487+0000","last_unstale":"2024-03-09T15:36:33.235992+0000","last_undegraded":"2024-03-09T15:36:33.235992+0000","last_fullsized":"2024-03-09T15:36:33.235992+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:33.235751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:33.235751+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.235751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:45.773762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.5","version":"258'2","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.230666+0000","last_change":"2024-03-09T15:36:35.230666+0000","last_active":"2024-03-09T15:36:35.230666+0000","last_peered":"2024-03-09T15:36:35.230666+0000","last_clean":"2024-03-09T15:36:35.230666+0000","last_became_active":"2024-03-09T14:46:26.812137+0000","last_became_peered":"2024-03-09T14:46:26.812137+0000","last_unstale":"2024-03-09T15:36:35.230666+0000","last_undegraded":"2024-03-09T15:36:35.230666+0000","last_fullsized":"2024-03-09T15:36:35.230666+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'2","last_scrub_stamp":"2024-03-09T15:36:35.230397+0000","last_deep_scrub":"258'2","last_deep_scrub_stamp":"2024-03-09T15:36:35.230397+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.230397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:52:41.341440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4096,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.4","version":"0'0","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.218589+0000","last_change":"2024-03-09T15:36:19.218589+0000","last_active":"2024-03-09T15:36:19.218589+0000","last_peered":"2024-03-09T15:36:19.218589+0000","last_clean":"2024-03-09T15:36:19.218589+0000","last_became_active":"2024-03-09T14:46:18.751608+0000","last_became_peered":"2024-03-09T14:46:18.751608+0000","last_unstale":"2024-03-09T15:36:19.218589+0000","last_undegraded":"2024-03-09T15:36:19.218589+0000","last_fullsized":"2024-03-09T15:36:19.218589+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:19.218339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:19.218339+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.218339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:28:04.515161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.7","version":"261'8","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.227007+0000","last_change":"2024-03-09T15:36:14.227039+0000","last_active":"2024-03-09T15:36:14.227007+0000","last_peered":"2024-03-09T15:36:14.227007+0000","last_clean":"2024-03-09T15:36:14.227007+0000","last_became_active":"2024-03-09T14:39:13.273363+0000","last_became_peered":"2024-03-09T14:39:13.273363+0000","last_unstale":"2024-03-09T15:36:14.227007+0000","last_undegraded":"2024-03-09T15:36:14.227007+0000","last_fullsized":"2024-03-09T15:36:14.227007+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:14.226893+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:14.226893+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.226893+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:21:38.723375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.3","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.211395+0000","last_change":"2024-03-09T15:09:50.078106+0000","last_active":"2024-03-09T15:34:51.211395+0000","last_peered":"2024-03-09T15:34:51.211395+0000","last_clean":"2024-03-09T15:34:51.211395+0000","last_became_active":"2024-03-09T15:09:50.072488+0000","last_became_peered":"2024-03-09T15:09:50.072488+0000","last_unstale":"2024-03-09T15:34:51.211395+0000","last_undegraded":"2024-03-09T15:34:51.211395+0000","last_fullsized":"2024-03-09T15:34:51.211395+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.4","version":"258'2","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.256968+0000","last_change":"2024-03-09T15:36:39.257131+0000","last_active":"2024-03-09T15:36:39.256968+0000","last_peered":"2024-03-09T15:36:39.256968+0000","last_clean":"2024-03-09T15:36:39.256968+0000","last_became_active":"2024-03-09T14:46:26.894506+0000","last_became_peered":"2024-03-09T14:46:26.894506+0000","last_unstale":"2024-03-09T15:36:39.256968+0000","last_undegraded":"2024-03-09T15:36:39.256968+0000","last_fullsized":"2024-03-09T15:36:39.256968+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'2","last_scrub_stamp":"2024-03-09T15:36:39.256730+0000","last_deep_scrub":"258'2","last_deep_scrub_stamp":"2024-03-09T15:36:39.256730+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.256730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:48:26.210989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":23,"num_write":2,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.5","version":"261'6","reported_seq":77,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.217768+0000","last_change":"2024-03-09T15:36:37.217842+0000","last_active":"2024-03-09T15:36:37.217768+0000","last_peered":"2024-03-09T15:36:37.217768+0000","last_clean":"2024-03-09T15:36:37.217768+0000","last_became_active":"2024-03-09T14:46:18.770799+0000","last_became_peered":"2024-03-09T14:46:18.770799+0000","last_unstale":"2024-03-09T15:36:37.217768+0000","last_undegraded":"2024-03-09T15:36:37.217768+0000","last_fullsized":"2024-03-09T15:36:37.217768+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:37.217579+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:37.217579+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.217579+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:32:01.418959+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.6","version":"261'22","reported_seq":121,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.972029+0000","last_change":"2024-03-09T15:36:14.972165+0000","last_active":"2024-03-09T15:36:14.972029+0000","last_peered":"2024-03-09T15:36:14.972029+0000","last_clean":"2024-03-09T15:36:14.972029+0000","last_became_active":"2024-03-09T14:39:13.266246+0000","last_became_peered":"2024-03-09T14:39:13.266246+0000","last_unstale":"2024-03-09T15:36:14.972029+0000","last_undegraded":"2024-03-09T15:36:14.972029+0000","last_fullsized":"2024-03-09T15:36:14.972029+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'22","last_scrub_stamp":"2024-03-09T15:36:14.971789+0000","last_deep_scrub":"261'22","last_deep_scrub_stamp":"2024-03-09T15:36:14.971789+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.971789+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:50:18.362986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":65,"num_write":41,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.0","version":"0'0","reported_seq":26,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:00.993222+0000","last_change":"2024-03-09T15:09:50.064673+0000","last_active":"2024-03-09T15:36:00.993222+0000","last_peered":"2024-03-09T15:36:00.993222+0000","last_clean":"2024-03-09T15:36:00.993222+0000","last_became_active":"2024-03-09T15:09:50.063350+0000","last_became_peered":"2024-03-09T15:09:50.063350+0000","last_unstale":"2024-03-09T15:36:00.993222+0000","last_undegraded":"2024-03-09T15:36:00.993222+0000","last_fullsized":"2024-03-09T15:36:00.993222+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.7","version":"0'0","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.151944+0000","last_change":"2024-03-09T15:36:29.151997+0000","last_active":"2024-03-09T15:36:29.151944+0000","last_peered":"2024-03-09T15:36:29.151944+0000","last_clean":"2024-03-09T15:36:29.151944+0000","last_became_active":"2024-03-09T14:46:26.896503+0000","last_became_peered":"2024-03-09T14:46:26.896503+0000","last_unstale":"2024-03-09T15:36:29.151944+0000","last_undegraded":"2024-03-09T15:36:29.151944+0000","last_fullsized":"2024-03-09T15:36:29.151944+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:29.151756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:29.151756+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.151756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:12:40.874461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.6","version":"261'2","reported_seq":63,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.752529+0000","last_change":"2024-03-09T15:36:26.752595+0000","last_active":"2024-03-09T15:36:26.752529+0000","last_peered":"2024-03-09T15:36:26.752529+0000","last_clean":"2024-03-09T15:36:26.752529+0000","last_became_active":"2024-03-09T14:46:18.778572+0000","last_became_peered":"2024-03-09T14:46:18.778572+0000","last_unstale":"2024-03-09T15:36:26.752529+0000","last_undegraded":"2024-03-09T15:36:26.752529+0000","last_fullsized":"2024-03-09T15:36:26.752529+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:26.752326+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:26.752326+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.752326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:06:26.783165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.5","version":"258'18","reported_seq":116,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.164653+0000","last_change":"2024-03-09T15:36:23.164716+0000","last_active":"2024-03-09T15:36:23.164653+0000","last_peered":"2024-03-09T15:36:23.164653+0000","last_clean":"2024-03-09T15:36:23.164653+0000","last_became_active":"2024-03-09T14:39:13.263833+0000","last_became_peered":"2024-03-09T14:39:13.263833+0000","last_unstale":"2024-03-09T15:36:23.164653+0000","last_undegraded":"2024-03-09T15:36:23.164653+0000","last_fullsized":"2024-03-09T15:36:23.164653+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'18","last_scrub_stamp":"2024-03-09T15:36:23.164449+0000","last_deep_scrub":"258'18","last_deep_scrub_stamp":"2024-03-09T15:36:23.164449+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.164449+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:22:04.204143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":58,"num_write":29,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.1","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.006085+0000","last_change":"2024-03-09T15:09:50.069405+0000","last_active":"2024-03-09T15:34:51.006085+0000","last_peered":"2024-03-09T15:34:51.006085+0000","last_clean":"2024-03-09T15:34:51.006085+0000","last_became_active":"2024-03-09T15:09:50.067947+0000","last_became_peered":"2024-03-09T15:09:50.067947+0000","last_unstale":"2024-03-09T15:34:51.006085+0000","last_undegraded":"2024-03-09T15:34:51.006085+0000","last_fullsized":"2024-03-09T15:34:51.006085+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"67.6","version":"261'4","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.650346+0000","last_change":"2024-03-09T14:46:26.809411+0000","last_active":"2024-03-09T15:36:12.650346+0000","last_peered":"2024-03-09T15:36:12.650346+0000","last_clean":"2024-03-09T15:36:12.650346+0000","last_became_active":"2024-03-09T14:46:26.808384+0000","last_became_peered":"2024-03-09T14:46:26.808384+0000","last_unstale":"2024-03-09T15:36:12.650346+0000","last_undegraded":"2024-03-09T15:36:12.650346+0000","last_fullsized":"2024-03-09T15:36:12.650346+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":180,"num_write":4,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.7","version":"261'9","reported_seq":83,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.291656+0000","last_change":"2024-03-09T15:36:14.291833+0000","last_active":"2024-03-09T15:36:14.291656+0000","last_peered":"2024-03-09T15:36:14.291656+0000","last_clean":"2024-03-09T15:36:14.291656+0000","last_became_active":"2024-03-09T14:46:18.762472+0000","last_became_peered":"2024-03-09T14:46:18.762472+0000","last_unstale":"2024-03-09T15:36:14.291656+0000","last_undegraded":"2024-03-09T15:36:14.291656+0000","last_fullsized":"2024-03-09T15:36:14.291656+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-03-09T15:36:14.291396+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-03-09T15:36:14.291396+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.291396+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:45:34.816033+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":30722,"num_write":17,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.4","version":"258'10","reported_seq":96,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.555803+0000","last_change":"2024-03-09T15:36:14.555905+0000","last_active":"2024-03-09T15:36:14.555803+0000","last_peered":"2024-03-09T15:36:14.555803+0000","last_clean":"2024-03-09T15:36:14.555803+0000","last_became_active":"2024-03-09T14:39:13.270238+0000","last_became_peered":"2024-03-09T14:39:13.270238+0000","last_unstale":"2024-03-09T15:36:14.555803+0000","last_undegraded":"2024-03-09T15:36:14.555803+0000","last_fullsized":"2024-03-09T15:36:14.555803+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:14.555648+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:14.555648+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.555648+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:38:18.000083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.6","version":"261'2","reported_seq":35,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.245425+0000","last_change":"2024-03-09T15:36:40.245494+0000","last_active":"2024-03-09T15:36:40.245425+0000","last_peered":"2024-03-09T15:36:40.245425+0000","last_clean":"2024-03-09T15:36:40.245425+0000","last_became_active":"2024-03-09T15:09:50.072180+0000","last_became_peered":"2024-03-09T15:09:50.072180+0000","last_unstale":"2024-03-09T15:36:40.245425+0000","last_undegraded":"2024-03-09T15:36:40.245425+0000","last_fullsized":"2024-03-09T15:36:40.245425+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:40.245236+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:40.245236+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.245236+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:47:53.099498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1","version":"0'0","reported_seq":55,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.542310+0000","last_change":"2024-03-09T15:36:21.542310+0000","last_active":"2024-03-09T15:36:21.542310+0000","last_peered":"2024-03-09T15:36:21.542310+0000","last_clean":"2024-03-09T15:36:21.542310+0000","last_became_active":"2024-03-09T14:46:26.799019+0000","last_became_peered":"2024-03-09T14:46:26.799019+0000","last_unstale":"2024-03-09T15:36:21.542310+0000","last_undegraded":"2024-03-09T15:36:21.542310+0000","last_fullsized":"2024-03-09T15:36:21.542310+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.542044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.542044+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.542044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:24:20.252130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.0","version":"0'0","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.229238+0000","last_change":"2024-03-09T15:36:18.229238+0000","last_active":"2024-03-09T15:36:18.229238+0000","last_peered":"2024-03-09T15:36:18.229238+0000","last_clean":"2024-03-09T15:36:18.229238+0000","last_became_active":"2024-03-09T14:46:18.752290+0000","last_became_peered":"2024-03-09T14:46:18.752290+0000","last_unstale":"2024-03-09T15:36:18.229238+0000","last_undegraded":"2024-03-09T15:36:18.229238+0000","last_fullsized":"2024-03-09T15:36:18.229238+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:18.228896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:18.228896+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.228896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:58:17.281185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.3","version":"261'61","reported_seq":224,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.587549+0000","last_change":"2024-03-09T15:36:14.587616+0000","last_active":"2024-03-09T15:36:14.587549+0000","last_peered":"2024-03-09T15:36:14.587549+0000","last_clean":"2024-03-09T15:36:14.587549+0000","last_became_active":"2024-03-09T14:39:13.259125+0000","last_became_peered":"2024-03-09T14:39:13.259125+0000","last_unstale":"2024-03-09T15:36:14.587549+0000","last_undegraded":"2024-03-09T15:36:14.587549+0000","last_fullsized":"2024-03-09T15:36:14.587549+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'61","last_scrub_stamp":"2024-03-09T15:36:14.587351+0000","last_deep_scrub":"261'61","last_deep_scrub_stamp":"2024-03-09T15:36:14.587351+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.587351+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:00:04.188736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":281,"num_read_kb":205,"num_write":98,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.7","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:51.044007+0000","last_change":"2024-03-09T15:09:50.064879+0000","last_active":"2024-03-09T15:35:51.044007+0000","last_peered":"2024-03-09T15:35:51.044007+0000","last_clean":"2024-03-09T15:35:51.044007+0000","last_became_active":"2024-03-09T15:09:50.062429+0000","last_became_peered":"2024-03-09T15:09:50.062429+0000","last_unstale":"2024-03-09T15:35:51.044007+0000","last_undegraded":"2024-03-09T15:35:51.044007+0000","last_fullsized":"2024-03-09T15:35:51.044007+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.0","version":"261'6","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.125140+0000","last_change":"2024-03-09T14:46:26.816110+0000","last_active":"2024-03-09T15:36:13.125140+0000","last_peered":"2024-03-09T15:36:13.125140+0000","last_clean":"2024-03-09T15:36:13.125140+0000","last_became_active":"2024-03-09T14:46:26.815311+0000","last_became_peered":"2024-03-09T14:46:26.815311+0000","last_unstale":"2024-03-09T15:36:13.125140+0000","last_undegraded":"2024-03-09T15:36:13.125140+0000","last_fullsized":"2024-03-09T15:36:13.125140+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":16384,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1","version":"261'6","reported_seq":73,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.249215+0000","last_change":"2024-03-09T15:36:15.249215+0000","last_active":"2024-03-09T15:36:15.249215+0000","last_peered":"2024-03-09T15:36:15.249215+0000","last_clean":"2024-03-09T15:36:15.249215+0000","last_became_active":"2024-03-09T14:46:18.757694+0000","last_became_peered":"2024-03-09T14:46:18.757694+0000","last_unstale":"2024-03-09T15:36:15.249215+0000","last_undegraded":"2024-03-09T15:36:15.249215+0000","last_fullsized":"2024-03-09T15:36:15.249215+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:15.248968+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:15.248968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.248968+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:50:59.398908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":4098,"num_write":9,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.2","version":"258'20","reported_seq":119,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.118799+0000","last_change":"2024-03-09T15:36:13.118868+0000","last_active":"2024-03-09T15:36:13.118799+0000","last_peered":"2024-03-09T15:36:13.118799+0000","last_clean":"2024-03-09T15:36:13.118799+0000","last_became_active":"2024-03-09T14:39:13.264203+0000","last_became_peered":"2024-03-09T14:39:13.264203+0000","last_unstale":"2024-03-09T15:36:13.118799+0000","last_undegraded":"2024-03-09T15:36:13.118799+0000","last_fullsized":"2024-03-09T15:36:13.118799+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'20","last_scrub_stamp":"2024-03-09T15:36:13.118589+0000","last_deep_scrub":"258'20","last_deep_scrub_stamp":"2024-03-09T15:36:13.118589+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.118589+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:29:35.065434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":65,"num_write":31,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.4","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:51.030150+0000","last_change":"2024-03-09T15:09:50.070703+0000","last_active":"2024-03-09T15:34:51.030150+0000","last_peered":"2024-03-09T15:34:51.030150+0000","last_clean":"2024-03-09T15:34:51.030150+0000","last_became_active":"2024-03-09T15:09:50.066222+0000","last_became_peered":"2024-03-09T15:09:50.066222+0000","last_unstale":"2024-03-09T15:34:51.030150+0000","last_undegraded":"2024-03-09T15:34:51.030150+0000","last_fullsized":"2024-03-09T15:34:51.030150+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.3","version":"0'0","reported_seq":55,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.342569+0000","last_change":"2024-03-09T15:36:49.342621+0000","last_active":"2024-03-09T15:36:49.342569+0000","last_peered":"2024-03-09T15:36:49.342569+0000","last_clean":"2024-03-09T15:36:49.342569+0000","last_became_active":"2024-03-09T14:46:26.826105+0000","last_became_peered":"2024-03-09T14:46:26.826105+0000","last_unstale":"2024-03-09T15:36:49.342569+0000","last_undegraded":"2024-03-09T15:36:49.342569+0000","last_fullsized":"2024-03-09T15:36:49.342569+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:49.342369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:49.342369+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.342369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:03:37.308549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.2","version":"0'0","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.469191+0000","last_change":"2024-03-09T15:36:12.469191+0000","last_active":"2024-03-09T15:36:12.469191+0000","last_peered":"2024-03-09T15:36:12.469191+0000","last_clean":"2024-03-09T15:36:12.469191+0000","last_became_active":"2024-03-09T14:46:18.749091+0000","last_became_peered":"2024-03-09T14:46:18.749091+0000","last_unstale":"2024-03-09T15:36:12.469191+0000","last_undegraded":"2024-03-09T15:36:12.469191+0000","last_fullsized":"2024-03-09T15:36:12.469191+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:12.468992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:12.468992+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.468992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:06:25.628906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1","version":"261'25","reported_seq":134,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.079889+0000","last_change":"2024-03-09T15:36:19.079952+0000","last_active":"2024-03-09T15:36:19.079889+0000","last_peered":"2024-03-09T15:36:19.079889+0000","last_clean":"2024-03-09T15:36:19.079889+0000","last_became_active":"2024-03-09T14:39:13.255589+0000","last_became_peered":"2024-03-09T14:39:13.255589+0000","last_unstale":"2024-03-09T15:36:19.079889+0000","last_undegraded":"2024-03-09T15:36:19.079889+0000","last_fullsized":"2024-03-09T15:36:19.079889+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'25","last_scrub_stamp":"2024-03-09T15:36:19.079692+0000","last_deep_scrub":"261'25","last_deep_scrub_stamp":"2024-03-09T15:36:19.079692+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.079692+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:25:37.071627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":79,"num_write":35,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.5","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:34:56.223957+0000","last_change":"2024-03-09T15:09:50.083562+0000","last_active":"2024-03-09T15:34:56.223957+0000","last_peered":"2024-03-09T15:34:56.223957+0000","last_clean":"2024-03-09T15:34:56.223957+0000","last_became_active":"2024-03-09T15:09:50.083064+0000","last_became_peered":"2024-03-09T15:09:50.083064+0000","last_unstale":"2024-03-09T15:34:56.223957+0000","last_undegraded":"2024-03-09T15:34:56.223957+0000","last_fullsized":"2024-03-09T15:34:56.223957+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.2","version":"258'3","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.650587+0000","last_change":"2024-03-09T14:46:26.808768+0000","last_active":"2024-03-09T15:36:12.650587+0000","last_peered":"2024-03-09T15:36:12.650587+0000","last_clean":"2024-03-09T15:36:12.650587+0000","last_became_active":"2024-03-09T14:46:26.807746+0000","last_became_peered":"2024-03-09T14:46:26.807746+0000","last_unstale":"2024-03-09T15:36:12.650587+0000","last_undegraded":"2024-03-09T15:36:12.650587+0000","last_fullsized":"2024-03-09T15:36:12.650587+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:46:25.763492+0000","last_clean_scrub_stamp":"2024-03-09T14:46:25.763492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1024,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.3","version":"0'0","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.030314+0000","last_change":"2024-03-09T15:36:18.030314+0000","last_active":"2024-03-09T15:36:18.030314+0000","last_peered":"2024-03-09T15:36:18.030314+0000","last_clean":"2024-03-09T15:36:18.030314+0000","last_became_active":"2024-03-09T14:46:18.765077+0000","last_became_peered":"2024-03-09T14:46:18.765077+0000","last_unstale":"2024-03-09T15:36:18.030314+0000","last_undegraded":"2024-03-09T15:36:18.030314+0000","last_fullsized":"2024-03-09T15:36:18.030314+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:18.030067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:18.030067+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.030067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:44:34.102574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.0","version":"250'25","reported_seq":139,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.180831+0000","last_change":"2024-03-09T15:36:20.180899+0000","last_active":"2024-03-09T15:36:20.180831+0000","last_peered":"2024-03-09T15:36:20.180831+0000","last_clean":"2024-03-09T15:36:20.180831+0000","last_became_active":"2024-03-09T14:39:13.268553+0000","last_became_peered":"2024-03-09T14:39:13.268553+0000","last_unstale":"2024-03-09T15:36:20.180831+0000","last_undegraded":"2024-03-09T15:36:20.180831+0000","last_fullsized":"2024-03-09T15:36:20.180831+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-03-09T15:36:20.180625+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-03-09T15:36:20.180625+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.180625+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:48:22.579836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":97,"num_write":32,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"261'118","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.121631+0000","last_change":"2024-03-09T15:36:48.121725+0000","last_active":"2024-03-09T15:36:48.121631+0000","last_peered":"2024-03-09T15:36:48.121631+0000","last_clean":"2024-03-09T15:36:48.121631+0000","last_became_active":"2024-03-09T14:28:32.339785+0000","last_became_peered":"2024-03-09T14:28:32.339785+0000","last_unstale":"2024-03-09T15:36:48.121631+0000","last_undegraded":"2024-03-09T15:36:48.121631+0000","last_fullsized":"2024-03-09T15:36:48.121631+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'118","last_scrub_stamp":"2024-03-09T15:36:48.121408+0000","last_deep_scrub":"261'118","last_deep_scrub_stamp":"2024-03-09T15:36:48.121408+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.121408+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:07:57.417207+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":130,"num_read_kb":5377,"num_write":373,"num_write_kb":16955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"261'146","reported_seq":696,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.743200+0000","last_change":"2024-03-09T15:36:31.743261+0000","last_active":"2024-03-09T15:36:31.743200+0000","last_peered":"2024-03-09T15:36:31.743200+0000","last_clean":"2024-03-09T15:36:31.743200+0000","last_became_active":"2024-03-09T14:28:34.303980+0000","last_became_peered":"2024-03-09T14:28:34.303980+0000","last_unstale":"2024-03-09T15:36:31.743200+0000","last_undegraded":"2024-03-09T15:36:31.743200+0000","last_fullsized":"2024-03-09T15:36:31.743200+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'146","last_scrub_stamp":"2024-03-09T15:36:31.743004+0000","last_deep_scrub":"261'146","last_deep_scrub_stamp":"2024-03-09T15:36:31.743004+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.743004+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:05:36.454535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":546,"num_read_kb":417,"num_write":228,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"261'143","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.213927+0000","last_change":"2024-03-09T15:36:45.214056+0000","last_active":"2024-03-09T15:36:45.213927+0000","last_peered":"2024-03-09T15:36:45.213927+0000","last_clean":"2024-03-09T15:36:45.213927+0000","last_became_active":"2024-03-09T14:28:32.306840+0000","last_became_peered":"2024-03-09T14:28:32.306840+0000","last_unstale":"2024-03-09T15:36:45.213927+0000","last_undegraded":"2024-03-09T15:36:45.213927+0000","last_fullsized":"2024-03-09T15:36:45.213927+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'143","last_scrub_stamp":"2024-03-09T15:36:45.213735+0000","last_deep_scrub":"261'143","last_deep_scrub_stamp":"2024-03-09T15:36:45.213735+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.213735+0000","objects_scrubbed":8,"log_size":43,"log_dups_size":100,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:08:41.879574+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3389647,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":156,"num_read_kb":3004,"num_write":523,"num_write_kb":19720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"261'90","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.206902+0000","last_change":"2024-03-09T15:36:40.206967+0000","last_active":"2024-03-09T15:36:40.206902+0000","last_peered":"2024-03-09T15:36:40.206902+0000","last_clean":"2024-03-09T15:36:40.206902+0000","last_became_active":"2024-03-09T14:28:34.324053+0000","last_became_peered":"2024-03-09T14:28:34.324053+0000","last_unstale":"2024-03-09T15:36:40.206902+0000","last_undegraded":"2024-03-09T15:36:40.206902+0000","last_fullsized":"2024-03-09T15:36:40.206902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'90","last_scrub_stamp":"2024-03-09T15:36:40.206700+0000","last_deep_scrub":"261'90","last_deep_scrub_stamp":"2024-03-09T15:36:40.206700+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.206700+0000","objects_scrubbed":1,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:36:21.366820+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":372,"num_read_kb":300,"num_write":141,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3d","version":"261'158","reported_seq":719,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.527086+0000","last_change":"2024-03-09T14:28:32.292874+0000","last_active":"2024-03-09T15:36:06.527086+0000","last_peered":"2024-03-09T15:36:06.527086+0000","last_clean":"2024-03-09T15:36:06.527086+0000","last_became_active":"2024-03-09T14:28:32.286966+0000","last_became_peered":"2024-03-09T14:28:32.286966+0000","last_unstale":"2024-03-09T15:36:06.527086+0000","last_undegraded":"2024-03-09T15:36:06.527086+0000","last_fullsized":"2024-03-09T15:36:06.527086+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":100,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":14171,"num_write":610,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"261'157","reported_seq":727,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.378778+0000","last_change":"2024-03-09T15:36:46.378837+0000","last_active":"2024-03-09T15:36:46.378778+0000","last_peered":"2024-03-09T15:36:46.378778+0000","last_clean":"2024-03-09T15:36:46.378778+0000","last_became_active":"2024-03-09T14:28:34.306764+0000","last_became_peered":"2024-03-09T14:28:34.306764+0000","last_unstale":"2024-03-09T15:36:46.378778+0000","last_undegraded":"2024-03-09T15:36:46.378778+0000","last_fullsized":"2024-03-09T15:36:46.378778+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'157","last_scrub_stamp":"2024-03-09T15:36:46.378588+0000","last_deep_scrub":"261'157","last_deep_scrub_stamp":"2024-03-09T15:36:46.378588+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.378588+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:55:21.945660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":481,"num_write":261,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3c","version":"261'190","reported_seq":784,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.124229+0000","last_change":"2024-03-09T14:28:32.310328+0000","last_active":"2024-03-09T15:36:13.124229+0000","last_peered":"2024-03-09T15:36:13.124229+0000","last_clean":"2024-03-09T15:36:13.124229+0000","last_became_active":"2024-03-09T14:28:32.305615+0000","last_became_peered":"2024-03-09T14:28:32.305615+0000","last_unstale":"2024-03-09T15:36:13.124229+0000","last_undegraded":"2024-03-09T15:36:13.124229+0000","last_fullsized":"2024-03-09T15:36:13.124229+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":100,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5326894,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":256,"num_read_kb":28937,"num_write":797,"num_write_kb":45835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"261'157","reported_seq":742,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.270196+0000","last_change":"2024-03-09T15:36:41.270258+0000","last_active":"2024-03-09T15:36:41.270196+0000","last_peered":"2024-03-09T15:36:41.270196+0000","last_clean":"2024-03-09T15:36:41.270196+0000","last_became_active":"2024-03-09T14:28:34.316500+0000","last_became_peered":"2024-03-09T14:28:34.316500+0000","last_unstale":"2024-03-09T15:36:41.270196+0000","last_undegraded":"2024-03-09T15:36:41.270196+0000","last_fullsized":"2024-03-09T15:36:41.270196+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'157","last_scrub_stamp":"2024-03-09T15:36:41.269997+0000","last_deep_scrub":"261'157","last_deep_scrub_stamp":"2024-03-09T15:36:41.269997+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.269997+0000","objects_scrubbed":1,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:09:23.890486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":622,"num_read_kb":488,"num_write":257,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3b","version":"261'150","reported_seq":813,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.509451+0000","last_change":"2024-03-09T14:28:32.317675+0000","last_active":"2024-03-09T15:36:06.509451+0000","last_peered":"2024-03-09T15:36:06.509451+0000","last_clean":"2024-03-09T15:36:06.509451+0000","last_became_active":"2024-03-09T14:28:32.314923+0000","last_became_peered":"2024-03-09T14:28:32.314923+0000","last_unstale":"2024-03-09T15:36:06.509451+0000","last_undegraded":"2024-03-09T15:36:06.509451+0000","last_fullsized":"2024-03-09T15:36:06.509451+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1917997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":297,"num_read_kb":17282,"num_write":517,"num_write_kb":14747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3c","version":"261'124","reported_seq":715,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.314456+0000","last_change":"2024-03-09T15:36:35.314515+0000","last_active":"2024-03-09T15:36:35.314456+0000","last_peered":"2024-03-09T15:36:35.314456+0000","last_clean":"2024-03-09T15:36:35.314456+0000","last_became_active":"2024-03-09T14:28:34.314617+0000","last_became_peered":"2024-03-09T14:28:34.314617+0000","last_unstale":"2024-03-09T15:36:35.314456+0000","last_undegraded":"2024-03-09T15:36:35.314456+0000","last_fullsized":"2024-03-09T15:36:35.314456+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'124","last_scrub_stamp":"2024-03-09T15:36:35.314270+0000","last_deep_scrub":"261'124","last_deep_scrub_stamp":"2024-03-09T15:36:35.314270+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.314270+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:54:11.780427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":493,"num_read_kb":395,"num_write":185,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3a","version":"261'150","reported_seq":693,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.166015+0000","last_change":"2024-03-09T15:36:14.166015+0000","last_active":"2024-03-09T15:36:14.166015+0000","last_peered":"2024-03-09T15:36:14.166015+0000","last_clean":"2024-03-09T15:36:14.166015+0000","last_became_active":"2024-03-09T14:29:06.433145+0000","last_became_peered":"2024-03-09T14:29:06.433145+0000","last_unstale":"2024-03-09T15:36:14.166015+0000","last_undegraded":"2024-03-09T15:36:14.166015+0000","last_fullsized":"2024-03-09T15:36:14.166015+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'150","last_scrub_stamp":"2024-03-09T15:36:14.165814+0000","last_deep_scrub":"261'150","last_deep_scrub_stamp":"2024-03-09T15:36:14.165814+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.165814+0000","objects_scrubbed":3,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:29:26.186154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":6443,"num_write":604,"num_write_kb":33049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"261'275","reported_seq":949,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.257093+0000","last_change":"2024-03-09T15:36:38.257158+0000","last_active":"2024-03-09T15:36:38.257093+0000","last_peered":"2024-03-09T15:36:38.257093+0000","last_clean":"2024-03-09T15:36:38.257093+0000","last_became_active":"2024-03-09T14:28:34.310343+0000","last_became_peered":"2024-03-09T14:28:34.310343+0000","last_unstale":"2024-03-09T15:36:38.257093+0000","last_undegraded":"2024-03-09T15:36:38.257093+0000","last_fullsized":"2024-03-09T15:36:38.257093+0000","mapping_epoch":22,"log_start":"247'200","ondisk_log_start":"247'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'275","last_scrub_stamp":"2024-03-09T15:36:38.256877+0000","last_deep_scrub":"261'275","last_deep_scrub_stamp":"2024-03-09T15:36:38.256877+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.256877+0000","objects_scrubbed":1,"log_size":75,"log_dups_size":200,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:40:32.582105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1205,"num_read_kb":1028,"num_write":418,"num_write_kb":253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25677,"num_omap_keys":101,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"261'133","reported_seq":690,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.495607+0000","last_change":"2024-03-09T14:28:32.314711+0000","last_active":"2024-03-09T15:36:06.495607+0000","last_peered":"2024-03-09T15:36:06.495607+0000","last_clean":"2024-03-09T15:36:06.495607+0000","last_became_active":"2024-03-09T14:28:32.314258+0000","last_became_peered":"2024-03-09T14:28:32.314258+0000","last_unstale":"2024-03-09T15:36:06.495607+0000","last_undegraded":"2024-03-09T15:36:06.495607+0000","last_fullsized":"2024-03-09T15:36:06.495607+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":169,"num_read_kb":17581,"num_write":466,"num_write_kb":22929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3e","version":"261'44","reported_seq":619,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.733708+0000","last_change":"2024-03-09T15:36:29.733770+0000","last_active":"2024-03-09T15:36:29.733708+0000","last_peered":"2024-03-09T15:36:29.733708+0000","last_clean":"2024-03-09T15:36:29.733708+0000","last_became_active":"2024-03-09T14:28:34.322206+0000","last_became_peered":"2024-03-09T14:28:34.322206+0000","last_unstale":"2024-03-09T15:36:29.733708+0000","last_undegraded":"2024-03-09T15:36:29.733708+0000","last_fullsized":"2024-03-09T15:36:29.733708+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:29.733482+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:29.733482+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.733482+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:58.948789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":201,"num_read_kb":171,"num_write":60,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"261'150","reported_seq":697,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.522393+0000","last_change":"2024-03-09T14:28:32.308873+0000","last_active":"2024-03-09T15:36:06.522393+0000","last_peered":"2024-03-09T15:36:06.522393+0000","last_clean":"2024-03-09T15:36:06.522393+0000","last_became_active":"2024-03-09T14:28:32.308522+0000","last_became_peered":"2024-03-09T14:28:32.308522+0000","last_unstale":"2024-03-09T15:36:06.522393+0000","last_undegraded":"2024-03-09T15:36:06.522393+0000","last_fullsized":"2024-03-09T15:36:06.522393+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525396,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":175,"num_read_kb":14130,"num_write":546,"num_write_kb":36844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"261'52","reported_seq":612,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.219275+0000","last_change":"2024-03-09T15:36:34.219343+0000","last_active":"2024-03-09T15:36:34.219275+0000","last_peered":"2024-03-09T15:36:34.219275+0000","last_clean":"2024-03-09T15:36:34.219275+0000","last_became_active":"2024-03-09T14:28:34.320646+0000","last_became_peered":"2024-03-09T14:28:34.320646+0000","last_unstale":"2024-03-09T15:36:34.219275+0000","last_undegraded":"2024-03-09T15:36:34.219275+0000","last_fullsized":"2024-03-09T15:36:34.219275+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'52","last_scrub_stamp":"2024-03-09T15:36:34.219014+0000","last_deep_scrub":"261'52","last_deep_scrub_stamp":"2024-03-09T15:36:34.219014+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.219014+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:48:47.044377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":174,"num_write":79,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"261'157","reported_seq":743,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.649512+0000","last_change":"2024-03-09T14:28:34.307366+0000","last_active":"2024-03-09T15:36:12.649512+0000","last_peered":"2024-03-09T15:36:12.649512+0000","last_clean":"2024-03-09T15:36:12.649512+0000","last_became_active":"2024-03-09T14:28:34.306592+0000","last_became_peered":"2024-03-09T14:28:34.306592+0000","last_unstale":"2024-03-09T15:36:12.649512+0000","last_undegraded":"2024-03-09T15:36:12.649512+0000","last_fullsized":"2024-03-09T15:36:12.649512+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":599,"num_read_kb":473,"num_write":232,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"261'165","reported_seq":726,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.220156+0000","last_change":"2024-03-09T15:36:16.220340+0000","last_active":"2024-03-09T15:36:16.220156+0000","last_peered":"2024-03-09T15:36:16.220156+0000","last_clean":"2024-03-09T15:36:16.220156+0000","last_became_active":"2024-03-09T14:29:06.438523+0000","last_became_peered":"2024-03-09T14:29:06.438523+0000","last_unstale":"2024-03-09T15:36:16.220156+0000","last_undegraded":"2024-03-09T15:36:16.220156+0000","last_fullsized":"2024-03-09T15:36:16.220156+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:16.219968+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:16.219968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.219968+0000","objects_scrubbed":4,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:42:29.990282+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728801,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":293,"num_read_kb":1034,"num_write":597,"num_write_kb":27895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.31","version":"261'88","reported_seq":652,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.145113+0000","last_change":"2024-03-09T15:36:46.145171+0000","last_active":"2024-03-09T15:36:46.145113+0000","last_peered":"2024-03-09T15:36:46.145113+0000","last_clean":"2024-03-09T15:36:46.145113+0000","last_became_active":"2024-03-09T14:28:34.321352+0000","last_became_peered":"2024-03-09T14:28:34.321352+0000","last_unstale":"2024-03-09T15:36:46.145113+0000","last_undegraded":"2024-03-09T15:36:46.145113+0000","last_fullsized":"2024-03-09T15:36:46.145113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'88","last_scrub_stamp":"2024-03-09T15:36:46.144913+0000","last_deep_scrub":"261'88","last_deep_scrub_stamp":"2024-03-09T15:36:46.144913+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.144913+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:37:05.731149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":342,"num_read_kb":270,"num_write":126,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"261'133","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.262628+0000","last_change":"2024-03-09T15:36:43.262702+0000","last_active":"2024-03-09T15:36:43.262628+0000","last_peered":"2024-03-09T15:36:43.262628+0000","last_clean":"2024-03-09T15:36:43.262628+0000","last_became_active":"2024-03-09T14:28:32.334208+0000","last_became_peered":"2024-03-09T14:28:32.334208+0000","last_unstale":"2024-03-09T15:36:43.262628+0000","last_undegraded":"2024-03-09T15:36:43.262628+0000","last_fullsized":"2024-03-09T15:36:43.262628+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'133","last_scrub_stamp":"2024-03-09T15:36:43.262446+0000","last_deep_scrub":"261'133","last_deep_scrub_stamp":"2024-03-09T15:36:43.262446+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.262446+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:21:20.422366+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262185,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":143,"num_read_kb":128,"num_write":408,"num_write_kb":22112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"261'62","reported_seq":618,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.784398+0000","last_change":"2024-03-09T15:36:35.784465+0000","last_active":"2024-03-09T15:36:35.784398+0000","last_peered":"2024-03-09T15:36:35.784398+0000","last_clean":"2024-03-09T15:36:35.784398+0000","last_became_active":"2024-03-09T14:28:34.298428+0000","last_became_peered":"2024-03-09T14:28:34.298428+0000","last_unstale":"2024-03-09T15:36:35.784398+0000","last_undegraded":"2024-03-09T15:36:35.784398+0000","last_fullsized":"2024-03-09T15:36:35.784398+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'62","last_scrub_stamp":"2024-03-09T15:36:35.784200+0000","last_deep_scrub":"261'62","last_deep_scrub_stamp":"2024-03-09T15:36:35.784200+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.784200+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:32:34.722288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":202,"num_write":92,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"261'118","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.242144+0000","last_change":"2024-03-09T15:36:39.242234+0000","last_active":"2024-03-09T15:36:39.242144+0000","last_peered":"2024-03-09T15:36:39.242144+0000","last_clean":"2024-03-09T15:36:39.242144+0000","last_became_active":"2024-03-09T14:28:32.318963+0000","last_became_peered":"2024-03-09T14:28:32.318963+0000","last_unstale":"2024-03-09T15:36:39.242144+0000","last_undegraded":"2024-03-09T15:36:39.242144+0000","last_fullsized":"2024-03-09T15:36:39.242144+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'118","last_scrub_stamp":"2024-03-09T15:36:39.241932+0000","last_deep_scrub":"261'118","last_deep_scrub_stamp":"2024-03-09T15:36:39.241932+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.241932+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:06:46.492058+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170436,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":164,"num_read_kb":1700,"num_write":435,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.33","version":"261'102","reported_seq":673,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.201686+0000","last_change":"2024-03-09T15:36:33.201745+0000","last_active":"2024-03-09T15:36:33.201686+0000","last_peered":"2024-03-09T15:36:33.201686+0000","last_clean":"2024-03-09T15:36:33.201686+0000","last_became_active":"2024-03-09T14:28:34.320331+0000","last_became_peered":"2024-03-09T14:28:34.320331+0000","last_unstale":"2024-03-09T15:36:33.201686+0000","last_undegraded":"2024-03-09T15:36:33.201686+0000","last_fullsized":"2024-03-09T15:36:33.201686+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'102","last_scrub_stamp":"2024-03-09T15:36:33.201501+0000","last_deep_scrub":"261'102","last_deep_scrub_stamp":"2024-03-09T15:36:33.201501+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.201501+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:07:30.758372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":320,"num_write":144,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"261'130","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.649099+0000","last_change":"2024-03-09T14:28:32.338458+0000","last_active":"2024-03-09T15:36:12.649099+0000","last_peered":"2024-03-09T15:36:12.649099+0000","last_clean":"2024-03-09T15:36:12.649099+0000","last_became_active":"2024-03-09T14:28:32.337934+0000","last_became_peered":"2024-03-09T14:28:32.337934+0000","last_unstale":"2024-03-09T15:36:12.649099+0000","last_undegraded":"2024-03-09T15:36:12.649099+0000","last_fullsized":"2024-03-09T15:36:12.649099+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263250,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":163,"num_read_kb":12763,"num_write":465,"num_write_kb":21372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"261'69","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.229898+0000","last_change":"2024-03-09T15:36:31.229983+0000","last_active":"2024-03-09T15:36:31.229898+0000","last_peered":"2024-03-09T15:36:31.229898+0000","last_clean":"2024-03-09T15:36:31.229898+0000","last_became_active":"2024-03-09T14:28:34.310703+0000","last_became_peered":"2024-03-09T14:28:34.310703+0000","last_unstale":"2024-03-09T15:36:31.229898+0000","last_undegraded":"2024-03-09T15:36:31.229898+0000","last_fullsized":"2024-03-09T15:36:31.229898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'69","last_scrub_stamp":"2024-03-09T15:36:31.229701+0000","last_deep_scrub":"261'69","last_deep_scrub_stamp":"2024-03-09T15:36:31.229701+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.229701+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:52:58.555942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":272,"num_read_kb":220,"num_write":109,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"261'157","reported_seq":725,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.185814+0000","last_change":"2024-03-09T14:28:32.337031+0000","last_active":"2024-03-09T15:36:12.185814+0000","last_peered":"2024-03-09T15:36:12.185814+0000","last_clean":"2024-03-09T15:36:12.185814+0000","last_became_active":"2024-03-09T14:28:32.336636+0000","last_became_peered":"2024-03-09T14:28:32.336636+0000","last_unstale":"2024-03-09T15:36:12.185814+0000","last_undegraded":"2024-03-09T15:36:12.185814+0000","last_fullsized":"2024-03-09T15:36:12.185814+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262241,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":231,"num_read_kb":12490,"num_write":578,"num_write_kb":28462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"261'87","reported_seq":647,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.123887+0000","last_change":"2024-03-09T15:36:33.123953+0000","last_active":"2024-03-09T15:36:33.123887+0000","last_peered":"2024-03-09T15:36:33.123887+0000","last_clean":"2024-03-09T15:36:33.123887+0000","last_became_active":"2024-03-09T14:28:34.314665+0000","last_became_peered":"2024-03-09T14:28:34.314665+0000","last_unstale":"2024-03-09T15:36:33.123887+0000","last_undegraded":"2024-03-09T15:36:33.123887+0000","last_fullsized":"2024-03-09T15:36:33.123887+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'87","last_scrub_stamp":"2024-03-09T15:36:33.123670+0000","last_deep_scrub":"261'87","last_deep_scrub_stamp":"2024-03-09T15:36:33.123670+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.123670+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:07:39.698152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":335,"num_read_kb":265,"num_write":124,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"261'120","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739013+0000","last_change":"2024-03-09T14:28:32.298846+0000","last_active":"2024-03-09T15:36:10.739013+0000","last_peered":"2024-03-09T15:36:10.739013+0000","last_clean":"2024-03-09T15:36:10.739013+0000","last_became_active":"2024-03-09T14:28:32.298428+0000","last_became_peered":"2024-03-09T14:28:32.298428+0000","last_unstale":"2024-03-09T15:36:10.739013+0000","last_undegraded":"2024-03-09T15:36:10.739013+0000","last_fullsized":"2024-03-09T15:36:10.739013+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":191,"num_read_kb":31150,"num_write":464,"num_write_kb":37586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"261'76","reported_seq":591,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.170281+0000","last_change":"2024-03-09T15:36:18.170281+0000","last_active":"2024-03-09T15:36:18.170281+0000","last_peered":"2024-03-09T15:36:18.170281+0000","last_clean":"2024-03-09T15:36:18.170281+0000","last_became_active":"2024-03-09T14:28:42.403553+0000","last_became_peered":"2024-03-09T14:28:42.403553+0000","last_unstale":"2024-03-09T15:36:18.170281+0000","last_undegraded":"2024-03-09T15:36:18.170281+0000","last_fullsized":"2024-03-09T15:36:18.170281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'76","last_scrub_stamp":"2024-03-09T15:36:18.170018+0000","last_deep_scrub":"261'76","last_deep_scrub_stamp":"2024-03-09T15:36:18.170018+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.170018+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:55:35.040311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":84,"num_write":184,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"261'98","reported_seq":674,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.480604+0000","last_change":"2024-03-09T15:36:20.480604+0000","last_active":"2024-03-09T15:36:20.480604+0000","last_peered":"2024-03-09T15:36:20.480604+0000","last_clean":"2024-03-09T15:36:20.480604+0000","last_became_active":"2024-03-09T14:29:06.439359+0000","last_became_peered":"2024-03-09T14:29:06.439359+0000","last_unstale":"2024-03-09T15:36:20.480604+0000","last_undegraded":"2024-03-09T15:36:20.480604+0000","last_fullsized":"2024-03-09T15:36:20.480604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'98","last_scrub_stamp":"2024-03-09T15:36:20.480339+0000","last_deep_scrub":"261'98","last_deep_scrub_stamp":"2024-03-09T15:36:20.480339+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.480339+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:32:16.216352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":413,"num_read_kb":333,"num_write":149,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"261'172","reported_seq":803,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.158265+0000","last_change":"2024-03-09T15:36:50.158375+0000","last_active":"2024-03-09T15:36:50.158265+0000","last_peered":"2024-03-09T15:36:50.158265+0000","last_clean":"2024-03-09T15:36:50.158265+0000","last_became_active":"2024-03-09T14:28:32.329457+0000","last_became_peered":"2024-03-09T14:28:32.329457+0000","last_unstale":"2024-03-09T15:36:50.158265+0000","last_undegraded":"2024-03-09T15:36:50.158265+0000","last_fullsized":"2024-03-09T15:36:50.158265+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'172","last_scrub_stamp":"2024-03-09T15:36:50.157939+0000","last_deep_scrub":"261'172","last_deep_scrub_stamp":"2024-03-09T15:36:50.157939+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.157939+0000","objects_scrubbed":7,"log_size":72,"log_dups_size":100,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:28.447473+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506100,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":263,"num_read_kb":141328,"num_write":603,"num_write_kb":37022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.929658+0000","last_change":"2024-03-09T15:36:40.929714+0000","last_active":"2024-03-09T15:36:40.929658+0000","last_peered":"2024-03-09T15:36:40.929658+0000","last_clean":"2024-03-09T15:36:40.929658+0000","last_became_active":"2024-03-09T14:28:36.320603+0000","last_became_peered":"2024-03-09T14:28:36.320603+0000","last_unstale":"2024-03-09T15:36:40.929658+0000","last_undegraded":"2024-03-09T15:36:40.929658+0000","last_fullsized":"2024-03-09T15:36:40.929658+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:40.929451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:40.929451+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.929451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:14:31.906125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"261'229","reported_seq":841,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739506+0000","last_change":"2024-03-09T14:28:38.327507+0000","last_active":"2024-03-09T15:36:10.739506+0000","last_peered":"2024-03-09T15:36:10.739506+0000","last_clean":"2024-03-09T15:36:10.739506+0000","last_became_active":"2024-03-09T14:28:38.326826+0000","last_became_peered":"2024-03-09T14:28:38.326826+0000","last_unstale":"2024-03-09T15:36:10.739506+0000","last_undegraded":"2024-03-09T15:36:10.739506+0000","last_fullsized":"2024-03-09T15:36:10.739506+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":200,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11101,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":365,"num_read_kb":621,"num_write":250,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.268381+0000","last_change":"2024-03-09T15:36:36.268437+0000","last_active":"2024-03-09T15:36:36.268381+0000","last_peered":"2024-03-09T15:36:36.268381+0000","last_clean":"2024-03-09T15:36:36.268381+0000","last_became_active":"2024-03-09T14:28:40.358954+0000","last_became_peered":"2024-03-09T14:28:40.358954+0000","last_unstale":"2024-03-09T15:36:36.268381+0000","last_undegraded":"2024-03-09T15:36:36.268381+0000","last_fullsized":"2024-03-09T15:36:36.268381+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:36.268123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:36.268123+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.268123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:59:04.556053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"261'772","reported_seq":2395,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.694693+0000","last_change":"2024-03-09T15:36:20.694693+0000","last_active":"2024-03-09T15:36:20.694693+0000","last_peered":"2024-03-09T15:36:20.694693+0000","last_clean":"2024-03-09T15:36:20.694693+0000","last_became_active":"2024-03-09T14:28:42.394691+0000","last_became_peered":"2024-03-09T14:28:42.394691+0000","last_unstale":"2024-03-09T15:36:20.694693+0000","last_undegraded":"2024-03-09T15:36:20.694693+0000","last_fullsized":"2024-03-09T15:36:20.694693+0000","mapping_epoch":30,"log_start":"261'700","ondisk_log_start":"261'700","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'772","last_scrub_stamp":"2024-03-09T15:36:20.694425+0000","last_deep_scrub":"261'772","last_deep_scrub_stamp":"2024-03-09T15:36:20.694425+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.694425+0000","objects_scrubbed":3,"log_size":72,"log_dups_size":700,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:30:23.952683+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":612,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2611,"num_read_kb":2601,"num_write":1560,"num_write_kb":569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":376,"num_omap_keys":2,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"261'40","reported_seq":588,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.759957+0000","last_change":"2024-03-09T15:36:32.760017+0000","last_active":"2024-03-09T15:36:32.759957+0000","last_peered":"2024-03-09T15:36:32.759957+0000","last_clean":"2024-03-09T15:36:32.759957+0000","last_became_active":"2024-03-09T14:28:34.324569+0000","last_became_peered":"2024-03-09T14:28:34.324569+0000","last_unstale":"2024-03-09T15:36:32.759957+0000","last_undegraded":"2024-03-09T15:36:32.759957+0000","last_fullsized":"2024-03-09T15:36:32.759957+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'40","last_scrub_stamp":"2024-03-09T15:36:32.759758+0000","last_deep_scrub":"261'40","last_deep_scrub_stamp":"2024-03-09T15:36:32.759758+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.759758+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:33:41.150112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":168,"num_read_kb":136,"num_write":58,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"261'169","reported_seq":741,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.738625+0000","last_change":"2024-03-09T14:28:32.310039+0000","last_active":"2024-03-09T15:36:10.738625+0000","last_peered":"2024-03-09T15:36:10.738625+0000","last_clean":"2024-03-09T15:36:10.738625+0000","last_became_active":"2024-03-09T14:28:32.309536+0000","last_became_peered":"2024-03-09T14:28:32.309536+0000","last_unstale":"2024-03-09T15:36:10.738625+0000","last_undegraded":"2024-03-09T15:36:10.738625+0000","last_fullsized":"2024-03-09T15:36:10.738625+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":100,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":29533,"num_write":652,"num_write_kb":30801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.186114+0000","last_change":"2024-03-09T14:28:36.320450+0000","last_active":"2024-03-09T15:36:12.186114+0000","last_peered":"2024-03-09T15:36:12.186114+0000","last_clean":"2024-03-09T15:36:12.186114+0000","last_became_active":"2024-03-09T14:28:36.320079+0000","last_became_peered":"2024-03-09T14:28:36.320079+0000","last_unstale":"2024-03-09T15:36:12.186114+0000","last_undegraded":"2024-03-09T15:36:12.186114+0000","last_fullsized":"2024-03-09T15:36:12.186114+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"261'90","reported_seq":636,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.122254+0000","last_change":"2024-03-09T15:36:27.122328+0000","last_active":"2024-03-09T15:36:27.122254+0000","last_peered":"2024-03-09T15:36:27.122254+0000","last_clean":"2024-03-09T15:36:27.122254+0000","last_became_active":"2024-03-09T14:28:38.340896+0000","last_became_peered":"2024-03-09T14:28:38.340896+0000","last_unstale":"2024-03-09T15:36:27.122254+0000","last_undegraded":"2024-03-09T15:36:27.122254+0000","last_fullsized":"2024-03-09T15:36:27.122254+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'90","last_scrub_stamp":"2024-03-09T15:36:27.122054+0000","last_deep_scrub":"261'90","last_deep_scrub_stamp":"2024-03-09T15:36:27.122054+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.122054+0000","objects_scrubbed":3,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:23:13.339583+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":101,"num_write":68,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.119178+0000","last_change":"2024-03-09T15:36:23.119232+0000","last_active":"2024-03-09T15:36:23.119178+0000","last_peered":"2024-03-09T15:36:23.119178+0000","last_clean":"2024-03-09T15:36:23.119178+0000","last_became_active":"2024-03-09T14:28:40.370788+0000","last_became_peered":"2024-03-09T14:28:40.370788+0000","last_unstale":"2024-03-09T15:36:23.119178+0000","last_undegraded":"2024-03-09T15:36:23.119178+0000","last_fullsized":"2024-03-09T15:36:23.119178+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:23.118978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:23.118978+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.118978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:04:36.080050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"261'64","reported_seq":576,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.282044+0000","last_change":"2024-03-09T15:36:21.282129+0000","last_active":"2024-03-09T15:36:21.282044+0000","last_peered":"2024-03-09T15:36:21.282044+0000","last_clean":"2024-03-09T15:36:21.282044+0000","last_became_active":"2024-03-09T14:28:42.408738+0000","last_became_peered":"2024-03-09T14:28:42.408738+0000","last_unstale":"2024-03-09T15:36:21.282044+0000","last_undegraded":"2024-03-09T15:36:21.282044+0000","last_fullsized":"2024-03-09T15:36:21.282044+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'64","last_scrub_stamp":"2024-03-09T15:36:21.281839+0000","last_deep_scrub":"261'64","last_deep_scrub_stamp":"2024-03-09T15:36:21.281839+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.281839+0000","objects_scrubbed":5,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:53:36.132862+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":681,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":76,"num_read_kb":75,"num_write":140,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"261'157","reported_seq":755,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.346427+0000","last_change":"2024-03-09T15:36:44.346583+0000","last_active":"2024-03-09T15:36:44.346427+0000","last_peered":"2024-03-09T15:36:44.346427+0000","last_clean":"2024-03-09T15:36:44.346427+0000","last_became_active":"2024-03-09T14:28:34.304892+0000","last_became_peered":"2024-03-09T14:28:34.304892+0000","last_unstale":"2024-03-09T15:36:44.346427+0000","last_undegraded":"2024-03-09T15:36:44.346427+0000","last_fullsized":"2024-03-09T15:36:44.346427+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'157","last_scrub_stamp":"2024-03-09T15:36:44.346198+0000","last_deep_scrub":"261'157","last_deep_scrub_stamp":"2024-03-09T15:36:44.346198+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.346198+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:31:38.007816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":622,"num_read_kb":492,"num_write":239,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"261'161","reported_seq":863,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101697+0000","last_change":"2024-03-09T14:28:32.319206+0000","last_active":"2024-03-09T15:36:14.101697+0000","last_peered":"2024-03-09T15:36:14.101697+0000","last_clean":"2024-03-09T15:36:14.101697+0000","last_became_active":"2024-03-09T14:28:32.318428+0000","last_became_peered":"2024-03-09T14:28:32.318428+0000","last_unstale":"2024-03-09T15:36:14.101697+0000","last_undegraded":"2024-03-09T15:36:14.101697+0000","last_fullsized":"2024-03-09T15:36:14.101697+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505054,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":544,"num_read_kb":113871,"num_write":619,"num_write_kb":46622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":498,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.186163+0000","last_change":"2024-03-09T14:28:36.320826+0000","last_active":"2024-03-09T15:36:12.186163+0000","last_peered":"2024-03-09T15:36:12.186163+0000","last_clean":"2024-03-09T15:36:12.186163+0000","last_became_active":"2024-03-09T14:28:36.320095+0000","last_became_peered":"2024-03-09T14:28:36.320095+0000","last_unstale":"2024-03-09T15:36:12.186163+0000","last_undegraded":"2024-03-09T15:36:12.186163+0000","last_fullsized":"2024-03-09T15:36:12.186163+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"261'108","reported_seq":668,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.165000+0000","last_change":"2024-03-09T15:36:17.165000+0000","last_active":"2024-03-09T15:36:17.165000+0000","last_peered":"2024-03-09T15:36:17.165000+0000","last_clean":"2024-03-09T15:36:17.165000+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:36:17.165000+0000","last_undegraded":"2024-03-09T15:36:17.165000+0000","last_fullsized":"2024-03-09T15:36:17.165000+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'108","last_scrub_stamp":"2024-03-09T15:36:17.164692+0000","last_deep_scrub":"261'108","last_deep_scrub_stamp":"2024-03-09T15:36:17.164692+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.164692+0000","objects_scrubbed":5,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:41:19.308226+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"261'505","reported_seq":2301,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.252225+0000","last_change":"2024-03-09T15:36:34.252290+0000","last_active":"2024-03-09T15:36:34.252225+0000","last_peered":"2024-03-09T15:36:34.252225+0000","last_clean":"2024-03-09T15:36:34.252225+0000","last_became_active":"2024-03-09T14:28:40.371507+0000","last_became_peered":"2024-03-09T14:28:40.371507+0000","last_unstale":"2024-03-09T15:36:34.252225+0000","last_undegraded":"2024-03-09T15:36:34.252225+0000","last_fullsized":"2024-03-09T15:36:34.252225+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'505","last_scrub_stamp":"2024-03-09T15:36:34.252038+0000","last_deep_scrub":"261'505","last_deep_scrub_stamp":"2024-03-09T15:36:34.252038+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.252038+0000","objects_scrubbed":1,"log_size":105,"log_dups_size":400,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:50:10.320691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"261'138","reported_seq":953,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.158161+0000","last_change":"2024-03-09T15:36:19.158227+0000","last_active":"2024-03-09T15:36:19.158161+0000","last_peered":"2024-03-09T15:36:19.158161+0000","last_clean":"2024-03-09T15:36:19.158161+0000","last_became_active":"2024-03-09T14:28:42.398498+0000","last_became_peered":"2024-03-09T14:28:42.398498+0000","last_unstale":"2024-03-09T15:36:19.158161+0000","last_undegraded":"2024-03-09T15:36:19.158161+0000","last_fullsized":"2024-03-09T15:36:19.158161+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'138","last_scrub_stamp":"2024-03-09T15:36:19.157968+0000","last_deep_scrub":"261'138","last_deep_scrub_stamp":"2024-03-09T15:36:19.157968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.157968+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":100,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:49:21.053160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":498,"num_read_kb":498,"num_write":277,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":234,"num_omap_keys":2,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"261'53","reported_seq":602,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.567436+0000","last_change":"2024-03-09T15:36:43.567493+0000","last_active":"2024-03-09T15:36:43.567436+0000","last_peered":"2024-03-09T15:36:43.567436+0000","last_clean":"2024-03-09T15:36:43.567436+0000","last_became_active":"2024-03-09T14:28:34.330347+0000","last_became_peered":"2024-03-09T14:28:34.330347+0000","last_unstale":"2024-03-09T15:36:43.567436+0000","last_undegraded":"2024-03-09T15:36:43.567436+0000","last_fullsized":"2024-03-09T15:36:43.567436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'53","last_scrub_stamp":"2024-03-09T15:36:43.567238+0000","last_deep_scrub":"261'53","last_deep_scrub_stamp":"2024-03-09T15:36:43.567238+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.567238+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:24:51.206691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":212,"num_read_kb":171,"num_write":77,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"261'169","reported_seq":751,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.275838+0000","last_change":"2024-03-09T15:36:38.275929+0000","last_active":"2024-03-09T15:36:38.275838+0000","last_peered":"2024-03-09T15:36:38.275838+0000","last_clean":"2024-03-09T15:36:38.275838+0000","last_became_active":"2024-03-09T14:28:32.319819+0000","last_became_peered":"2024-03-09T14:28:32.319819+0000","last_unstale":"2024-03-09T15:36:38.275838+0000","last_undegraded":"2024-03-09T15:36:38.275838+0000","last_fullsized":"2024-03-09T15:36:38.275838+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'169","last_scrub_stamp":"2024-03-09T15:36:38.275636+0000","last_deep_scrub":"261'169","last_deep_scrub_stamp":"2024-03-09T15:36:38.275636+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.275636+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":100,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:18:08.037585+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5244508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":231,"num_read_kb":15567,"num_write":579,"num_write_kb":34195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.872479+0000","last_change":"2024-03-09T15:36:38.872534+0000","last_active":"2024-03-09T15:36:38.872479+0000","last_peered":"2024-03-09T15:36:38.872479+0000","last_clean":"2024-03-09T15:36:38.872479+0000","last_became_active":"2024-03-09T14:28:36.310231+0000","last_became_peered":"2024-03-09T14:28:36.310231+0000","last_unstale":"2024-03-09T15:36:38.872479+0000","last_undegraded":"2024-03-09T15:36:38.872479+0000","last_fullsized":"2024-03-09T15:36:38.872479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:38.872296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:38.872296+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.872296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:03:09.877161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"261'170","reported_seq":755,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.102850+0000","last_change":"2024-03-09T14:28:38.599078+0000","last_active":"2024-03-09T15:36:14.102850+0000","last_peered":"2024-03-09T15:36:14.102850+0000","last_clean":"2024-03-09T15:36:14.102850+0000","last_became_active":"2024-03-09T14:28:38.598308+0000","last_became_peered":"2024-03-09T14:28:38.598308+0000","last_unstale":"2024-03-09T15:36:14.102850+0000","last_undegraded":"2024-03-09T15:36:14.102850+0000","last_fullsized":"2024-03-09T15:36:14.102850+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":100,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8724,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":532,"num_write":190,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"261'513","reported_seq":2509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.458248+0000","last_change":"2024-03-09T15:36:32.458248+0000","last_active":"2024-03-09T15:36:32.458248+0000","last_peered":"2024-03-09T15:36:32.458248+0000","last_clean":"2024-03-09T15:36:32.458248+0000","last_became_active":"2024-03-09T14:28:40.387990+0000","last_became_peered":"2024-03-09T14:28:40.387990+0000","last_unstale":"2024-03-09T15:36:32.458248+0000","last_undegraded":"2024-03-09T15:36:32.458248+0000","last_fullsized":"2024-03-09T15:36:32.458248+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'513","last_scrub_stamp":"2024-03-09T15:36:32.457985+0000","last_deep_scrub":"261'513","last_deep_scrub_stamp":"2024-03-09T15:36:32.457985+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.457985+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":500,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:24:58.183827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"261'101","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.740079+0000","last_change":"2024-03-09T15:36:28.740141+0000","last_active":"2024-03-09T15:36:28.740079+0000","last_peered":"2024-03-09T15:36:28.740079+0000","last_clean":"2024-03-09T15:36:28.740079+0000","last_became_active":"2024-03-09T14:28:34.322731+0000","last_became_peered":"2024-03-09T14:28:34.322731+0000","last_unstale":"2024-03-09T15:36:28.740079+0000","last_undegraded":"2024-03-09T15:36:28.740079+0000","last_fullsized":"2024-03-09T15:36:28.740079+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'101","last_scrub_stamp":"2024-03-09T15:36:28.739877+0000","last_deep_scrub":"261'101","last_deep_scrub_stamp":"2024-03-09T15:36:28.739877+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.739877+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:43:42.753565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":330,"num_write":141,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"261'123","reported_seq":673,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101424+0000","last_change":"2024-03-09T14:28:32.329755+0000","last_active":"2024-03-09T15:36:14.101424+0000","last_peered":"2024-03-09T15:36:14.101424+0000","last_clean":"2024-03-09T15:36:14.101424+0000","last_became_active":"2024-03-09T14:28:32.329148+0000","last_became_peered":"2024-03-09T14:28:32.329148+0000","last_unstale":"2024-03-09T15:36:14.101424+0000","last_undegraded":"2024-03-09T15:36:14.101424+0000","last_fullsized":"2024-03-09T15:36:14.101424+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":100,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7947490,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":1766,"num_write":513,"num_write_kb":18289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":518,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.203914+0000","last_change":"2024-03-09T15:36:39.204023+0000","last_active":"2024-03-09T15:36:39.203914+0000","last_peered":"2024-03-09T15:36:39.203914+0000","last_clean":"2024-03-09T15:36:39.203914+0000","last_became_active":"2024-03-09T14:28:36.313904+0000","last_became_peered":"2024-03-09T14:28:36.313904+0000","last_unstale":"2024-03-09T15:36:39.203914+0000","last_undegraded":"2024-03-09T15:36:39.203914+0000","last_fullsized":"2024-03-09T15:36:39.203914+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:39.203674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:39.203674+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.203674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:18:37.090196+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"261'276","reported_seq":920,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.746370+0000","last_change":"2024-03-09T15:36:24.746541+0000","last_active":"2024-03-09T15:36:24.746370+0000","last_peered":"2024-03-09T15:36:24.746370+0000","last_clean":"2024-03-09T15:36:24.746370+0000","last_became_active":"2024-03-09T14:28:38.597247+0000","last_became_peered":"2024-03-09T14:28:38.597247+0000","last_unstale":"2024-03-09T15:36:24.746370+0000","last_undegraded":"2024-03-09T15:36:24.746370+0000","last_fullsized":"2024-03-09T15:36:24.746370+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'276","last_scrub_stamp":"2024-03-09T15:36:24.746130+0000","last_deep_scrub":"261'276","last_deep_scrub_stamp":"2024-03-09T15:36:24.746130+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.746130+0000","objects_scrubbed":9,"log_size":76,"log_dups_size":200,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:53:37.971114+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9398,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":421,"num_read_kb":664,"num_write":284,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.394226+0000","last_change":"2024-03-09T15:36:37.394277+0000","last_active":"2024-03-09T15:36:37.394226+0000","last_peered":"2024-03-09T15:36:37.394226+0000","last_clean":"2024-03-09T15:36:37.394226+0000","last_became_active":"2024-03-09T14:28:40.359906+0000","last_became_peered":"2024-03-09T14:28:40.359906+0000","last_unstale":"2024-03-09T15:36:37.394226+0000","last_undegraded":"2024-03-09T15:36:37.394226+0000","last_fullsized":"2024-03-09T15:36:37.394226+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:37.394037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:37.394037+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.394037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:42:28.463933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"261'522","reported_seq":1369,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.664753+0000","last_change":"2024-03-09T15:36:16.664829+0000","last_active":"2024-03-09T15:36:16.664753+0000","last_peered":"2024-03-09T15:36:16.664753+0000","last_clean":"2024-03-09T15:36:16.664753+0000","last_became_active":"2024-03-09T14:28:42.402739+0000","last_became_peered":"2024-03-09T14:28:42.402739+0000","last_unstale":"2024-03-09T15:36:16.664753+0000","last_undegraded":"2024-03-09T15:36:16.664753+0000","last_fullsized":"2024-03-09T15:36:16.664753+0000","mapping_epoch":30,"log_start":"261'500","ondisk_log_start":"261'500","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'522","last_scrub_stamp":"2024-03-09T15:36:16.664560+0000","last_deep_scrub":"261'522","last_deep_scrub_stamp":"2024-03-09T15:36:16.664560+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.664560+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":500,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:05:16.194957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1336,"num_read_kb":1311,"num_write":1083,"num_write_kb":376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":381,"num_omap_keys":2,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"261'92","reported_seq":663,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:46.017837+0000","last_change":"2024-03-09T14:28:34.302153+0000","last_active":"2024-03-09T15:35:46.017837+0000","last_peered":"2024-03-09T15:35:46.017837+0000","last_clean":"2024-03-09T15:35:46.017837+0000","last_became_active":"2024-03-09T14:28:34.300795+0000","last_became_peered":"2024-03-09T14:28:34.300795+0000","last_unstale":"2024-03-09T15:35:46.017837+0000","last_undegraded":"2024-03-09T15:35:46.017837+0000","last_fullsized":"2024-03-09T15:35:46.017837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":361,"num_read_kb":288,"num_write":150,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"261'142","reported_seq":716,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.452480+0000","last_change":"2024-03-09T14:28:32.318853+0000","last_active":"2024-03-09T15:36:06.452480+0000","last_peered":"2024-03-09T15:36:06.452480+0000","last_clean":"2024-03-09T15:36:06.452480+0000","last_became_active":"2024-03-09T14:28:32.318111+0000","last_became_peered":"2024-03-09T14:28:32.318111+0000","last_unstale":"2024-03-09T15:36:06.452480+0000","last_undegraded":"2024-03-09T15:36:06.452480+0000","last_fullsized":"2024-03-09T15:36:06.452480+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":100,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242921,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":6230,"num_write":597,"num_write_kb":19779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.197656+0000","last_change":"2024-03-09T15:36:27.197765+0000","last_active":"2024-03-09T15:36:27.197656+0000","last_peered":"2024-03-09T15:36:27.197656+0000","last_clean":"2024-03-09T15:36:27.197656+0000","last_became_active":"2024-03-09T14:28:36.301227+0000","last_became_peered":"2024-03-09T14:28:36.301227+0000","last_unstale":"2024-03-09T15:36:27.197656+0000","last_undegraded":"2024-03-09T15:36:27.197656+0000","last_fullsized":"2024-03-09T15:36:27.197656+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:27.197420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:27.197420+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.197420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:56:31.881994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"261'177","reported_seq":763,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.577608+0000","last_change":"2024-03-09T15:36:40.577792+0000","last_active":"2024-03-09T15:36:40.577608+0000","last_peered":"2024-03-09T15:36:40.577608+0000","last_clean":"2024-03-09T15:36:40.577608+0000","last_became_active":"2024-03-09T14:28:38.346837+0000","last_became_peered":"2024-03-09T14:28:38.346837+0000","last_unstale":"2024-03-09T15:36:40.577608+0000","last_undegraded":"2024-03-09T15:36:40.577608+0000","last_fullsized":"2024-03-09T15:36:40.577608+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'177","last_scrub_stamp":"2024-03-09T15:36:40.577391+0000","last_deep_scrub":"261'177","last_deep_scrub_stamp":"2024-03-09T15:36:40.577391+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.577391+0000","objects_scrubbed":4,"log_size":77,"log_dups_size":100,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:47:13.834091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16267,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":276,"num_read_kb":528,"num_write":188,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.485270+0000","last_change":"2024-03-09T15:36:30.485329+0000","last_active":"2024-03-09T15:36:30.485270+0000","last_peered":"2024-03-09T15:36:30.485270+0000","last_clean":"2024-03-09T15:36:30.485270+0000","last_became_active":"2024-03-09T14:28:40.389175+0000","last_became_peered":"2024-03-09T14:28:40.389175+0000","last_unstale":"2024-03-09T15:36:30.485270+0000","last_undegraded":"2024-03-09T15:36:30.485270+0000","last_fullsized":"2024-03-09T15:36:30.485270+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:30.485071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:30.485071+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.485071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:54:36.706597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"261'59","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.297133+0000","last_change":"2024-03-09T15:36:34.297200+0000","last_active":"2024-03-09T15:36:34.297133+0000","last_peered":"2024-03-09T15:36:34.297133+0000","last_clean":"2024-03-09T15:36:34.297133+0000","last_became_active":"2024-03-09T14:28:42.401214+0000","last_became_peered":"2024-03-09T14:28:42.401214+0000","last_unstale":"2024-03-09T15:36:34.297133+0000","last_undegraded":"2024-03-09T15:36:34.297133+0000","last_fullsized":"2024-03-09T15:36:34.297133+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'59","last_scrub_stamp":"2024-03-09T15:36:34.296927+0000","last_deep_scrub":"261'59","last_deep_scrub_stamp":"2024-03-09T15:36:34.296927+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.296927+0000","objects_scrubbed":1,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:18:34.284628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":47,"num_write":140,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"261'78","reported_seq":643,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.249814+0000","last_change":"2024-03-09T15:36:33.249877+0000","last_active":"2024-03-09T15:36:33.249814+0000","last_peered":"2024-03-09T15:36:33.249814+0000","last_clean":"2024-03-09T15:36:33.249814+0000","last_became_active":"2024-03-09T14:28:34.316812+0000","last_became_peered":"2024-03-09T14:28:34.316812+0000","last_unstale":"2024-03-09T15:36:33.249814+0000","last_undegraded":"2024-03-09T15:36:33.249814+0000","last_fullsized":"2024-03-09T15:36:33.249814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'78","last_scrub_stamp":"2024-03-09T15:36:33.249613+0000","last_deep_scrub":"261'78","last_deep_scrub_stamp":"2024-03-09T15:36:33.249613+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.249613+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:51.977272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":320,"num_read_kb":256,"num_write":116,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"261'111","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.172728+0000","last_change":"2024-03-09T15:36:27.172803+0000","last_active":"2024-03-09T15:36:27.172728+0000","last_peered":"2024-03-09T15:36:27.172728+0000","last_clean":"2024-03-09T15:36:27.172728+0000","last_became_active":"2024-03-09T14:29:06.446333+0000","last_became_peered":"2024-03-09T14:29:06.446333+0000","last_unstale":"2024-03-09T15:36:27.172728+0000","last_undegraded":"2024-03-09T15:36:27.172728+0000","last_fullsized":"2024-03-09T15:36:27.172728+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'111","last_scrub_stamp":"2024-03-09T15:36:27.172540+0000","last_deep_scrub":"261'111","last_deep_scrub_stamp":"2024-03-09T15:36:27.172540+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.172540+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:12:44.004446+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":147,"num_read_kb":1025,"num_write":402,"num_write_kb":11167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":515,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.252418+0000","last_change":"2024-03-09T15:36:25.252530+0000","last_active":"2024-03-09T15:36:25.252418+0000","last_peered":"2024-03-09T15:36:25.252418+0000","last_clean":"2024-03-09T15:36:25.252418+0000","last_became_active":"2024-03-09T14:28:36.303695+0000","last_became_peered":"2024-03-09T14:28:36.303695+0000","last_unstale":"2024-03-09T15:36:25.252418+0000","last_undegraded":"2024-03-09T15:36:25.252418+0000","last_fullsized":"2024-03-09T15:36:25.252418+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.252128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.252128+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.252128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:47:06.758741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"261'227","reported_seq":844,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.149330+0000","last_change":"2024-03-09T15:36:51.149432+0000","last_active":"2024-03-09T15:36:51.149330+0000","last_peered":"2024-03-09T15:36:51.149330+0000","last_clean":"2024-03-09T15:36:51.149330+0000","last_became_active":"2024-03-09T14:28:38.339300+0000","last_became_peered":"2024-03-09T14:28:38.339300+0000","last_unstale":"2024-03-09T15:36:51.149330+0000","last_undegraded":"2024-03-09T15:36:51.149330+0000","last_fullsized":"2024-03-09T15:36:51.149330+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'227","last_scrub_stamp":"2024-03-09T15:36:51.149113+0000","last_deep_scrub":"261'227","last_deep_scrub_stamp":"2024-03-09T15:36:51.149113+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.149113+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":200,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:50:51.145329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8477,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":353,"num_read_kb":624,"num_write":244,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"261'501","reported_seq":2519,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.232198+0000","last_change":"2024-03-09T15:36:34.232325+0000","last_active":"2024-03-09T15:36:34.232198+0000","last_peered":"2024-03-09T15:36:34.232198+0000","last_clean":"2024-03-09T15:36:34.232198+0000","last_became_active":"2024-03-09T14:28:40.367119+0000","last_became_peered":"2024-03-09T14:28:40.367119+0000","last_unstale":"2024-03-09T15:36:34.232198+0000","last_undegraded":"2024-03-09T15:36:34.232198+0000","last_fullsized":"2024-03-09T15:36:34.232198+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'501","last_scrub_stamp":"2024-03-09T15:36:34.231965+0000","last_deep_scrub":"261'501","last_deep_scrub_stamp":"2024-03-09T15:36:34.231965+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.231965+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":400,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:18:21.657518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"261'82","reported_seq":585,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.245845+0000","last_change":"2024-03-09T15:36:28.245919+0000","last_active":"2024-03-09T15:36:28.245845+0000","last_peered":"2024-03-09T15:36:28.245845+0000","last_clean":"2024-03-09T15:36:28.245845+0000","last_became_active":"2024-03-09T14:28:42.396389+0000","last_became_peered":"2024-03-09T14:28:42.396389+0000","last_unstale":"2024-03-09T15:36:28.245845+0000","last_undegraded":"2024-03-09T15:36:28.245845+0000","last_fullsized":"2024-03-09T15:36:28.245845+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'82","last_scrub_stamp":"2024-03-09T15:36:28.245628+0000","last_deep_scrub":"261'82","last_deep_scrub_stamp":"2024-03-09T15:36:28.245628+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.245628+0000","objects_scrubbed":1,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:26:51.961711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":65,"num_write":200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"261'44","reported_seq":591,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.354458+0000","last_change":"2024-03-09T15:36:46.354523+0000","last_active":"2024-03-09T15:36:46.354458+0000","last_peered":"2024-03-09T15:36:46.354458+0000","last_clean":"2024-03-09T15:36:46.354458+0000","last_became_active":"2024-03-09T14:28:34.297080+0000","last_became_peered":"2024-03-09T14:28:34.297080+0000","last_unstale":"2024-03-09T15:36:46.354458+0000","last_undegraded":"2024-03-09T15:36:46.354458+0000","last_fullsized":"2024-03-09T15:36:46.354458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:46.354264+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:46.354264+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.354264+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:33:23.149348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":189,"num_read_kb":155,"num_write":63,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"261'152","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.200259+0000","last_change":"2024-03-09T15:36:20.200259+0000","last_active":"2024-03-09T15:36:20.200259+0000","last_peered":"2024-03-09T15:36:20.200259+0000","last_clean":"2024-03-09T15:36:20.200259+0000","last_became_active":"2024-03-09T14:29:06.445573+0000","last_became_peered":"2024-03-09T14:29:06.445573+0000","last_unstale":"2024-03-09T15:36:20.200259+0000","last_undegraded":"2024-03-09T15:36:20.200259+0000","last_fullsized":"2024-03-09T15:36:20.200259+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'152","last_scrub_stamp":"2024-03-09T15:36:20.199970+0000","last_deep_scrub":"261'152","last_deep_scrub_stamp":"2024-03-09T15:36:20.199970+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.199970+0000","objects_scrubbed":5,"log_size":52,"log_dups_size":100,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:59:48.356486+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":204,"num_read_kb":157,"num_write":458,"num_write_kb":32761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.186224+0000","last_change":"2024-03-09T15:36:19.186332+0000","last_active":"2024-03-09T15:36:19.186224+0000","last_peered":"2024-03-09T15:36:19.186224+0000","last_clean":"2024-03-09T15:36:19.186224+0000","last_became_active":"2024-03-09T14:28:36.322638+0000","last_became_peered":"2024-03-09T14:28:36.322638+0000","last_unstale":"2024-03-09T15:36:19.186224+0000","last_undegraded":"2024-03-09T15:36:19.186224+0000","last_fullsized":"2024-03-09T15:36:19.186224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:19.185963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:19.185963+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.185963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:52:57.100559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"261'318","reported_seq":962,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.499196+0000","last_change":"2024-03-09T14:28:38.329540+0000","last_active":"2024-03-09T15:36:06.499196+0000","last_peered":"2024-03-09T15:36:06.499196+0000","last_clean":"2024-03-09T15:36:06.499196+0000","last_became_active":"2024-03-09T14:28:38.328897+0000","last_became_peered":"2024-03-09T14:28:38.328897+0000","last_unstale":"2024-03-09T15:36:06.499196+0000","last_undegraded":"2024-03-09T15:36:06.499196+0000","last_fullsized":"2024-03-09T15:36:06.499196+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":300,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":543,"num_read_kb":1162,"num_write":388,"num_write_kb":277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.192323+0000","last_change":"2024-03-09T15:36:12.192350+0000","last_active":"2024-03-09T15:36:12.192323+0000","last_peered":"2024-03-09T15:36:12.192323+0000","last_clean":"2024-03-09T15:36:12.192323+0000","last_became_active":"2024-03-09T14:28:40.383758+0000","last_became_peered":"2024-03-09T14:28:40.383758+0000","last_unstale":"2024-03-09T15:36:12.192323+0000","last_undegraded":"2024-03-09T15:36:12.192323+0000","last_fullsized":"2024-03-09T15:36:12.192323+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:12.192210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:12.192210+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.192210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:18:21.362249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"261'56","reported_seq":576,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.525835+0000","last_change":"2024-03-09T15:36:19.525925+0000","last_active":"2024-03-09T15:36:19.525835+0000","last_peered":"2024-03-09T15:36:19.525835+0000","last_clean":"2024-03-09T15:36:19.525835+0000","last_became_active":"2024-03-09T14:28:42.401880+0000","last_became_peered":"2024-03-09T14:28:42.401880+0000","last_unstale":"2024-03-09T15:36:19.525835+0000","last_undegraded":"2024-03-09T15:36:19.525835+0000","last_fullsized":"2024-03-09T15:36:19.525835+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:19.525622+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:19.525622+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.525622+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:58:32.566894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":69,"num_write":120,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"261'81","reported_seq":650,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.131661+0000","last_change":"2024-03-09T15:36:31.131726+0000","last_active":"2024-03-09T15:36:31.131661+0000","last_peered":"2024-03-09T15:36:31.131661+0000","last_clean":"2024-03-09T15:36:31.131661+0000","last_became_active":"2024-03-09T14:28:34.315432+0000","last_became_peered":"2024-03-09T14:28:34.315432+0000","last_unstale":"2024-03-09T15:36:31.131661+0000","last_undegraded":"2024-03-09T15:36:31.131661+0000","last_fullsized":"2024-03-09T15:36:31.131661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'81","last_scrub_stamp":"2024-03-09T15:36:31.131473+0000","last_deep_scrub":"261'81","last_deep_scrub_stamp":"2024-03-09T15:36:31.131473+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.131473+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:42:11.454205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":315,"num_read_kb":251,"num_write":126,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":466,"num_omap_keys":2,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"261'157","reported_seq":696,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.528298+0000","last_change":"2024-03-09T14:28:32.300326+0000","last_active":"2024-03-09T15:36:06.528298+0000","last_peered":"2024-03-09T15:36:06.528298+0000","last_clean":"2024-03-09T15:36:06.528298+0000","last_became_active":"2024-03-09T14:28:32.299897+0000","last_became_peered":"2024-03-09T14:28:32.299897+0000","last_unstale":"2024-03-09T15:36:06.528298+0000","last_undegraded":"2024-03-09T15:36:06.528298+0000","last_fullsized":"2024-03-09T15:36:06.528298+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506226,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":160,"num_read_kb":225,"num_write":553,"num_write_kb":26448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":515,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.168556+0000","last_change":"2024-03-09T15:36:21.168616+0000","last_active":"2024-03-09T15:36:21.168556+0000","last_peered":"2024-03-09T15:36:21.168556+0000","last_clean":"2024-03-09T15:36:21.168556+0000","last_became_active":"2024-03-09T14:28:36.320029+0000","last_became_peered":"2024-03-09T14:28:36.320029+0000","last_unstale":"2024-03-09T15:36:21.168556+0000","last_undegraded":"2024-03-09T15:36:21.168556+0000","last_fullsized":"2024-03-09T15:36:21.168556+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.168322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.168322+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.168322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:54:01.071807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"261'213","reported_seq":814,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.154985+0000","last_change":"2024-03-09T15:36:28.155065+0000","last_active":"2024-03-09T15:36:28.154985+0000","last_peered":"2024-03-09T15:36:28.154985+0000","last_clean":"2024-03-09T15:36:28.154985+0000","last_became_active":"2024-03-09T14:28:38.338658+0000","last_became_peered":"2024-03-09T14:28:38.338658+0000","last_unstale":"2024-03-09T15:36:28.154985+0000","last_undegraded":"2024-03-09T15:36:28.154985+0000","last_fullsized":"2024-03-09T15:36:28.154985+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'213","last_scrub_stamp":"2024-03-09T15:36:28.154798+0000","last_deep_scrub":"261'213","last_deep_scrub_stamp":"2024-03-09T15:36:28.154798+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.154798+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":200,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:48:17.369213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12662,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":352,"num_read_kb":613,"num_write":292,"num_write_kb":188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13964,"num_omap_keys":34,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.231293+0000","last_change":"2024-03-09T15:36:30.231399+0000","last_active":"2024-03-09T15:36:30.231293+0000","last_peered":"2024-03-09T15:36:30.231293+0000","last_clean":"2024-03-09T15:36:30.231293+0000","last_became_active":"2024-03-09T14:28:40.378973+0000","last_became_peered":"2024-03-09T14:28:40.378973+0000","last_unstale":"2024-03-09T15:36:30.231293+0000","last_undegraded":"2024-03-09T15:36:30.231293+0000","last_fullsized":"2024-03-09T15:36:30.231293+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:30.231002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:30.231002+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.231002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:08:44.672009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"261'44","reported_seq":546,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.449604+0000","last_change":"2024-03-09T15:36:23.449669+0000","last_active":"2024-03-09T15:36:23.449604+0000","last_peered":"2024-03-09T15:36:23.449604+0000","last_clean":"2024-03-09T15:36:23.449604+0000","last_became_active":"2024-03-09T14:28:42.397533+0000","last_became_peered":"2024-03-09T14:28:42.397533+0000","last_unstale":"2024-03-09T15:36:23.449604+0000","last_undegraded":"2024-03-09T15:36:23.449604+0000","last_fullsized":"2024-03-09T15:36:23.449604+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:23.449413+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:23.449413+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.449413+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:46:36.909599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":91,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"261'56","reported_seq":625,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.166995+0000","last_change":"2024-03-09T15:36:34.167057+0000","last_active":"2024-03-09T15:36:34.166995+0000","last_peered":"2024-03-09T15:36:34.166995+0000","last_clean":"2024-03-09T15:36:34.166995+0000","last_became_active":"2024-03-09T14:28:34.313475+0000","last_became_peered":"2024-03-09T14:28:34.313475+0000","last_unstale":"2024-03-09T15:36:34.166995+0000","last_undegraded":"2024-03-09T15:36:34.166995+0000","last_fullsized":"2024-03-09T15:36:34.166995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:34.166804+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:34.166804+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.166804+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:15:44.814256+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":198,"num_write":78,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"261'136","reported_seq":697,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.314272+0000","last_change":"2024-03-09T15:36:37.314511+0000","last_active":"2024-03-09T15:36:37.314272+0000","last_peered":"2024-03-09T15:36:37.314272+0000","last_clean":"2024-03-09T15:36:37.314272+0000","last_became_active":"2024-03-09T14:28:32.316406+0000","last_became_peered":"2024-03-09T14:28:32.316406+0000","last_unstale":"2024-03-09T15:36:37.314272+0000","last_undegraded":"2024-03-09T15:36:37.314272+0000","last_fullsized":"2024-03-09T15:36:37.314272+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'136","last_scrub_stamp":"2024-03-09T15:36:37.314063+0000","last_deep_scrub":"261'136","last_deep_scrub_stamp":"2024-03-09T15:36:37.314063+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.314063+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:50:27.588191+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655835,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":162,"num_read_kb":15736,"num_write":466,"num_write_kb":13578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.511704+0000","last_change":"2024-03-09T15:36:27.511852+0000","last_active":"2024-03-09T15:36:27.511704+0000","last_peered":"2024-03-09T15:36:27.511704+0000","last_clean":"2024-03-09T15:36:27.511704+0000","last_became_active":"2024-03-09T14:28:36.319210+0000","last_became_peered":"2024-03-09T14:28:36.319210+0000","last_unstale":"2024-03-09T15:36:27.511704+0000","last_undegraded":"2024-03-09T15:36:27.511704+0000","last_fullsized":"2024-03-09T15:36:27.511704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:27.511458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:27.511458+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.511458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:07.969688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"261'223","reported_seq":837,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.196282+0000","last_change":"2024-03-09T15:36:15.196282+0000","last_active":"2024-03-09T15:36:15.196282+0000","last_peered":"2024-03-09T15:36:15.196282+0000","last_clean":"2024-03-09T15:36:15.196282+0000","last_became_active":"2024-03-09T14:28:38.350390+0000","last_became_peered":"2024-03-09T14:28:38.350390+0000","last_unstale":"2024-03-09T15:36:15.196282+0000","last_undegraded":"2024-03-09T15:36:15.196282+0000","last_fullsized":"2024-03-09T15:36:15.196282+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'223","last_scrub_stamp":"2024-03-09T15:36:15.195858+0000","last_deep_scrub":"261'223","last_deep_scrub_stamp":"2024-03-09T15:36:15.195858+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.195858+0000","objects_scrubbed":7,"log_size":23,"log_dups_size":200,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:47:25.524775+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9540,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":338,"num_read_kb":580,"num_write":230,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"261'474","reported_seq":2368,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.200778+0000","last_change":"2024-03-09T15:36:13.200827+0000","last_active":"2024-03-09T15:36:13.200778+0000","last_peered":"2024-03-09T15:36:13.200778+0000","last_clean":"2024-03-09T15:36:13.200778+0000","last_became_active":"2024-03-09T14:28:40.373360+0000","last_became_peered":"2024-03-09T14:28:40.373360+0000","last_unstale":"2024-03-09T15:36:13.200778+0000","last_undegraded":"2024-03-09T15:36:13.200778+0000","last_fullsized":"2024-03-09T15:36:13.200778+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'474","last_scrub_stamp":"2024-03-09T15:36:13.200687+0000","last_deep_scrub":"261'474","last_deep_scrub_stamp":"2024-03-09T15:36:13.200687+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.200687+0000","objects_scrubbed":1,"log_size":74,"log_dups_size":400,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:36:38.570841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"261'108","reported_seq":663,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.587316+0000","last_change":"2024-03-09T15:36:16.587396+0000","last_active":"2024-03-09T15:36:16.587316+0000","last_peered":"2024-03-09T15:36:16.587316+0000","last_clean":"2024-03-09T15:36:16.587316+0000","last_became_active":"2024-03-09T14:28:42.408233+0000","last_became_peered":"2024-03-09T14:28:42.408233+0000","last_unstale":"2024-03-09T15:36:16.587316+0000","last_undegraded":"2024-03-09T15:36:16.587316+0000","last_fullsized":"2024-03-09T15:36:16.587316+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'108","last_scrub_stamp":"2024-03-09T15:36:16.587096+0000","last_deep_scrub":"261'108","last_deep_scrub_stamp":"2024-03-09T15:36:16.587096+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.587096+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:31:09.055955+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":274,"num_read_kb":227,"num_write":264,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"261'72","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.342615+0000","last_change":"2024-03-09T15:36:44.342675+0000","last_active":"2024-03-09T15:36:44.342615+0000","last_peered":"2024-03-09T15:36:44.342615+0000","last_clean":"2024-03-09T15:36:44.342615+0000","last_became_active":"2024-03-09T14:28:34.310061+0000","last_became_peered":"2024-03-09T14:28:34.310061+0000","last_unstale":"2024-03-09T15:36:44.342615+0000","last_undegraded":"2024-03-09T15:36:44.342615+0000","last_fullsized":"2024-03-09T15:36:44.342615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-09T15:36:44.342422+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-09T15:36:44.342422+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.342422+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:25:32.650543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":294,"num_read_kb":233,"num_write":114,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"261'174","reported_seq":790,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.437240+0000","last_change":"2024-03-09T15:36:48.437332+0000","last_active":"2024-03-09T15:36:48.437240+0000","last_peered":"2024-03-09T15:36:48.437240+0000","last_clean":"2024-03-09T15:36:48.437240+0000","last_became_active":"2024-03-09T14:28:32.313566+0000","last_became_peered":"2024-03-09T14:28:32.313566+0000","last_unstale":"2024-03-09T15:36:48.437240+0000","last_undegraded":"2024-03-09T15:36:48.437240+0000","last_fullsized":"2024-03-09T15:36:48.437240+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'174","last_scrub_stamp":"2024-03-09T15:36:48.437063+0000","last_deep_scrub":"261'174","last_deep_scrub_stamp":"2024-03-09T15:36:48.437063+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.437063+0000","objects_scrubbed":8,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:03:36.289305+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":245,"num_read_kb":84880,"num_write":567,"num_write_kb":19718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"261'10","reported_seq":1472,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.904683+0000","last_change":"2024-03-09T15:36:42.904753+0000","last_active":"2024-03-09T15:36:42.904683+0000","last_peered":"2024-03-09T15:36:42.904683+0000","last_clean":"2024-03-09T15:36:42.904683+0000","last_became_active":"2024-03-09T14:28:36.325820+0000","last_became_peered":"2024-03-09T14:28:36.325820+0000","last_unstale":"2024-03-09T15:36:42.904683+0000","last_undegraded":"2024-03-09T15:36:42.904683+0000","last_fullsized":"2024-03-09T15:36:42.904683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'10","last_scrub_stamp":"2024-03-09T15:36:42.904497+0000","last_deep_scrub":"261'10","last_deep_scrub_stamp":"2024-03-09T15:36:42.904497+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.904497+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:01:23.989269+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1195,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1118,"num_read_kb":1421,"num_write":14,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"261'91","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.533379+0000","last_change":"2024-03-09T15:36:38.533460+0000","last_active":"2024-03-09T15:36:38.533379+0000","last_peered":"2024-03-09T15:36:38.533379+0000","last_clean":"2024-03-09T15:36:38.533379+0000","last_became_active":"2024-03-09T14:28:38.595478+0000","last_became_peered":"2024-03-09T14:28:38.595478+0000","last_unstale":"2024-03-09T15:36:38.533379+0000","last_undegraded":"2024-03-09T15:36:38.533379+0000","last_fullsized":"2024-03-09T15:36:38.533379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'91","last_scrub_stamp":"2024-03-09T15:36:38.533177+0000","last_deep_scrub":"261'91","last_deep_scrub_stamp":"2024-03-09T15:36:38.533177+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.533177+0000","objects_scrubbed":5,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:59:32.253642+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":105,"num_read_kb":100,"num_write":70,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"261'489","reported_seq":2304,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.167232+0000","last_change":"2024-03-09T15:36:26.167298+0000","last_active":"2024-03-09T15:36:26.167232+0000","last_peered":"2024-03-09T15:36:26.167232+0000","last_clean":"2024-03-09T15:36:26.167232+0000","last_became_active":"2024-03-09T14:28:40.374408+0000","last_became_peered":"2024-03-09T14:28:40.374408+0000","last_unstale":"2024-03-09T15:36:26.167232+0000","last_undegraded":"2024-03-09T15:36:26.167232+0000","last_fullsized":"2024-03-09T15:36:26.167232+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'489","last_scrub_stamp":"2024-03-09T15:36:26.166904+0000","last_deep_scrub":"261'489","last_deep_scrub_stamp":"2024-03-09T15:36:26.166904+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.166904+0000","objects_scrubbed":1,"log_size":89,"log_dups_size":400,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:49:54.207124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"261'94","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.145360+0000","last_change":"2024-03-09T15:36:16.145423+0000","last_active":"2024-03-09T15:36:16.145360+0000","last_peered":"2024-03-09T15:36:16.145360+0000","last_clean":"2024-03-09T15:36:16.145360+0000","last_became_active":"2024-03-09T14:28:42.411523+0000","last_became_peered":"2024-03-09T14:28:42.411523+0000","last_unstale":"2024-03-09T15:36:16.145360+0000","last_undegraded":"2024-03-09T15:36:16.145360+0000","last_fullsized":"2024-03-09T15:36:16.145360+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'94","last_scrub_stamp":"2024-03-09T15:36:16.145166+0000","last_deep_scrub":"261'94","last_deep_scrub_stamp":"2024-03-09T15:36:16.145166+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.145166+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:56:22.170750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":72,"num_write":221,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"261'42","reported_seq":605,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.506243+0000","last_change":"2024-03-09T15:36:28.506314+0000","last_active":"2024-03-09T15:36:28.506243+0000","last_peered":"2024-03-09T15:36:28.506243+0000","last_clean":"2024-03-09T15:36:28.506243+0000","last_became_active":"2024-03-09T14:28:34.317226+0000","last_became_peered":"2024-03-09T14:28:34.317226+0000","last_unstale":"2024-03-09T15:36:28.506243+0000","last_undegraded":"2024-03-09T15:36:28.506243+0000","last_fullsized":"2024-03-09T15:36:28.506243+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'42","last_scrub_stamp":"2024-03-09T15:36:28.505993+0000","last_deep_scrub":"261'42","last_deep_scrub_stamp":"2024-03-09T15:36:28.505993+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.505993+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:14:19.130657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":188,"num_read_kb":158,"num_write":62,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"261'116","reported_seq":653,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.527014+0000","last_change":"2024-03-09T14:28:32.295693+0000","last_active":"2024-03-09T15:36:06.527014+0000","last_peered":"2024-03-09T15:36:06.527014+0000","last_clean":"2024-03-09T15:36:06.527014+0000","last_became_active":"2024-03-09T14:28:32.295179+0000","last_became_peered":"2024-03-09T14:28:32.295179+0000","last_unstale":"2024-03-09T15:36:06.527014+0000","last_undegraded":"2024-03-09T15:36:06.527014+0000","last_fullsized":"2024-03-09T15:36:06.527014+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":100,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262159,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":143,"num_read_kb":124,"num_write":390,"num_write_kb":5920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.379645+0000","last_change":"2024-03-09T15:36:45.379697+0000","last_active":"2024-03-09T15:36:45.379645+0000","last_peered":"2024-03-09T15:36:45.379645+0000","last_clean":"2024-03-09T15:36:45.379645+0000","last_became_active":"2024-03-09T14:28:36.319102+0000","last_became_peered":"2024-03-09T14:28:36.319102+0000","last_unstale":"2024-03-09T15:36:45.379645+0000","last_undegraded":"2024-03-09T15:36:45.379645+0000","last_fullsized":"2024-03-09T15:36:45.379645+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:45.379458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:45.379458+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.379458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:10:12.455227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"261'321","reported_seq":958,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.774275+0000","last_change":"2024-03-09T15:36:27.774372+0000","last_active":"2024-03-09T15:36:27.774275+0000","last_peered":"2024-03-09T15:36:27.774275+0000","last_clean":"2024-03-09T15:36:27.774275+0000","last_became_active":"2024-03-09T14:28:38.337254+0000","last_became_peered":"2024-03-09T14:28:38.337254+0000","last_unstale":"2024-03-09T15:36:27.774275+0000","last_undegraded":"2024-03-09T15:36:27.774275+0000","last_fullsized":"2024-03-09T15:36:27.774275+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'321","last_scrub_stamp":"2024-03-09T15:36:27.774086+0000","last_deep_scrub":"261'321","last_deep_scrub_stamp":"2024-03-09T15:36:27.774086+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.774086+0000","objects_scrubbed":9,"log_size":21,"log_dups_size":300,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:34:50.482182+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13936,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":673,"num_read_kb":965,"num_write":788,"num_write_kb":629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":141964,"num_omap_keys":356,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.219910+0000","last_change":"2024-03-09T15:36:30.219963+0000","last_active":"2024-03-09T15:36:30.219910+0000","last_peered":"2024-03-09T15:36:30.219910+0000","last_clean":"2024-03-09T15:36:30.219910+0000","last_became_active":"2024-03-09T14:28:40.377714+0000","last_became_peered":"2024-03-09T14:28:40.377714+0000","last_unstale":"2024-03-09T15:36:30.219910+0000","last_undegraded":"2024-03-09T15:36:30.219910+0000","last_fullsized":"2024-03-09T15:36:30.219910+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:30.219708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:30.219708+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.219708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:30:50.699663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"261'69","reported_seq":577,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.545201+0000","last_change":"2024-03-09T15:36:20.545264+0000","last_active":"2024-03-09T15:36:20.545201+0000","last_peered":"2024-03-09T15:36:20.545201+0000","last_clean":"2024-03-09T15:36:20.545201+0000","last_became_active":"2024-03-09T14:28:42.401775+0000","last_became_peered":"2024-03-09T14:28:42.401775+0000","last_unstale":"2024-03-09T15:36:20.545201+0000","last_undegraded":"2024-03-09T15:36:20.545201+0000","last_fullsized":"2024-03-09T15:36:20.545201+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'69","last_scrub_stamp":"2024-03-09T15:36:20.545001+0000","last_deep_scrub":"261'69","last_deep_scrub_stamp":"2024-03-09T15:36:20.545001+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.545001+0000","objects_scrubbed":1,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:08:40.010383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":77,"num_write":153,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"261'174","reported_seq":834,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.140904+0000","last_change":"2024-03-09T15:36:27.140963+0000","last_active":"2024-03-09T15:36:27.140904+0000","last_peered":"2024-03-09T15:36:27.140904+0000","last_clean":"2024-03-09T15:36:27.140904+0000","last_became_active":"2024-03-09T14:28:34.326919+0000","last_became_peered":"2024-03-09T14:28:34.326919+0000","last_unstale":"2024-03-09T15:36:27.140904+0000","last_undegraded":"2024-03-09T15:36:27.140904+0000","last_fullsized":"2024-03-09T15:36:27.140904+0000","mapping_epoch":22,"log_start":"34'100","ondisk_log_start":"34'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'174","last_scrub_stamp":"2024-03-09T15:36:27.140715+0000","last_deep_scrub":"261'174","last_deep_scrub_stamp":"2024-03-09T15:36:27.140715+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.140715+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:06:14.462396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":914,"num_read_kb":819,"num_write":300,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101288+0000","last_change":"2024-03-09T14:28:12.265518+0000","last_active":"2024-03-09T15:36:14.101288+0000","last_peered":"2024-03-09T15:36:14.101288+0000","last_clean":"2024-03-09T15:36:14.101288+0000","last_became_active":"2024-03-09T14:28:12.262430+0000","last_became_peered":"2024-03-09T14:28:12.262430+0000","last_unstale":"2024-03-09T15:36:14.101288+0000","last_undegraded":"2024-03-09T15:36:14.101288+0000","last_fullsized":"2024-03-09T15:36:14.101288+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:11.179640+0000","last_clean_scrub_stamp":"2024-03-09T14:28:11.179640+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":531,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.969790+0000","last_change":"2024-03-09T15:36:44.969844+0000","last_active":"2024-03-09T15:36:44.969790+0000","last_peered":"2024-03-09T15:36:44.969790+0000","last_clean":"2024-03-09T15:36:44.969790+0000","last_became_active":"2024-03-09T14:28:19.232517+0000","last_became_peered":"2024-03-09T14:28:19.232517+0000","last_unstale":"2024-03-09T15:36:44.969790+0000","last_undegraded":"2024-03-09T15:36:44.969790+0000","last_fullsized":"2024-03-09T15:36:44.969790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:44.969581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:44.969581+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.969581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:43:52.001009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"261'130","reported_seq":690,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.491713+0000","last_change":"2024-03-09T14:28:32.300845+0000","last_active":"2024-03-09T15:36:06.491713+0000","last_peered":"2024-03-09T15:36:06.491713+0000","last_clean":"2024-03-09T15:36:06.491713+0000","last_became_active":"2024-03-09T14:28:32.300468+0000","last_became_peered":"2024-03-09T14:28:32.300468+0000","last_unstale":"2024-03-09T15:36:06.491713+0000","last_undegraded":"2024-03-09T15:36:06.491713+0000","last_fullsized":"2024-03-09T15:36:06.491713+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311644,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":173,"num_read_kb":6450,"num_write":457,"num_write_kb":21101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":519,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.163673+0000","last_change":"2024-03-09T15:36:41.163783+0000","last_active":"2024-03-09T15:36:41.163673+0000","last_peered":"2024-03-09T15:36:41.163673+0000","last_clean":"2024-03-09T15:36:41.163673+0000","last_became_active":"2024-03-09T14:28:36.318172+0000","last_became_peered":"2024-03-09T14:28:36.318172+0000","last_unstale":"2024-03-09T15:36:41.163673+0000","last_undegraded":"2024-03-09T15:36:41.163673+0000","last_fullsized":"2024-03-09T15:36:41.163673+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:41.163438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:41.163438+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.163438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:55:32.987190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"261'502","reported_seq":1226,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.737856+0000","last_change":"2024-03-09T15:36:25.737931+0000","last_active":"2024-03-09T15:36:25.737856+0000","last_peered":"2024-03-09T15:36:25.737856+0000","last_clean":"2024-03-09T15:36:25.737856+0000","last_became_active":"2024-03-09T14:28:38.348941+0000","last_became_peered":"2024-03-09T14:28:38.348941+0000","last_unstale":"2024-03-09T15:36:25.737856+0000","last_undegraded":"2024-03-09T15:36:25.737856+0000","last_fullsized":"2024-03-09T15:36:25.737856+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'502","last_scrub_stamp":"2024-03-09T15:36:25.737665+0000","last_deep_scrub":"261'502","last_deep_scrub_stamp":"2024-03-09T15:36:25.737665+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.737665+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":400,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:22:57.116693+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37400,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":987,"num_read_kb":1910,"num_write":934,"num_write_kb":767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":90684,"num_omap_keys":220,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.239905+0000","last_change":"2024-03-09T15:36:37.239957+0000","last_active":"2024-03-09T15:36:37.239905+0000","last_peered":"2024-03-09T15:36:37.239905+0000","last_clean":"2024-03-09T15:36:37.239905+0000","last_became_active":"2024-03-09T14:28:40.362781+0000","last_became_peered":"2024-03-09T14:28:40.362781+0000","last_unstale":"2024-03-09T15:36:37.239905+0000","last_undegraded":"2024-03-09T15:36:37.239905+0000","last_fullsized":"2024-03-09T15:36:37.239905+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:37.239705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:37.239705+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.239705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:28.768337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"261'121","reported_seq":829,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.279007+0000","last_change":"2024-03-09T15:36:44.279073+0000","last_active":"2024-03-09T15:36:44.279007+0000","last_peered":"2024-03-09T15:36:44.279007+0000","last_clean":"2024-03-09T15:36:44.279007+0000","last_became_active":"2024-03-09T14:28:42.398626+0000","last_became_peered":"2024-03-09T14:28:42.398626+0000","last_unstale":"2024-03-09T15:36:44.279007+0000","last_undegraded":"2024-03-09T15:36:44.279007+0000","last_fullsized":"2024-03-09T15:36:44.279007+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'121","last_scrub_stamp":"2024-03-09T15:36:44.278808+0000","last_deep_scrub":"261'121","last_deep_scrub_stamp":"2024-03-09T15:36:44.278808+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.278808+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":100,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:44:17.595367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":356,"num_read_kb":343,"num_write":215,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":66,"num_omap_keys":1,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"261'90","reported_seq":679,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.266824+0000","last_change":"2024-03-09T15:36:50.266889+0000","last_active":"2024-03-09T15:36:50.266824+0000","last_peered":"2024-03-09T15:36:50.266824+0000","last_clean":"2024-03-09T15:36:50.266824+0000","last_became_active":"2024-03-09T14:28:34.297472+0000","last_became_peered":"2024-03-09T14:28:34.297472+0000","last_unstale":"2024-03-09T15:36:50.266824+0000","last_undegraded":"2024-03-09T15:36:50.266824+0000","last_fullsized":"2024-03-09T15:36:50.266824+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'90","last_scrub_stamp":"2024-03-09T15:36:50.266621+0000","last_deep_scrub":"261'90","last_deep_scrub_stamp":"2024-03-09T15:36:50.266621+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.266621+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:34:30.319896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":356,"num_read_kb":292,"num_write":141,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":527,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101340+0000","last_change":"2024-03-09T14:28:19.242176+0000","last_active":"2024-03-09T15:36:14.101340+0000","last_peered":"2024-03-09T15:36:14.101340+0000","last_clean":"2024-03-09T15:36:14.101340+0000","last_became_active":"2024-03-09T14:28:19.241110+0000","last_became_peered":"2024-03-09T14:28:19.241110+0000","last_unstale":"2024-03-09T15:36:14.101340+0000","last_undegraded":"2024-03-09T15:36:14.101340+0000","last_fullsized":"2024-03-09T15:36:14.101340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"261'138","reported_seq":711,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.525842+0000","last_change":"2024-03-09T14:28:32.312953+0000","last_active":"2024-03-09T15:36:06.525842+0000","last_peered":"2024-03-09T15:36:06.525842+0000","last_clean":"2024-03-09T15:36:06.525842+0000","last_became_active":"2024-03-09T14:28:32.312536+0000","last_became_peered":"2024-03-09T14:28:32.312536+0000","last_unstale":"2024-03-09T15:36:06.525842+0000","last_undegraded":"2024-03-09T15:36:06.525842+0000","last_fullsized":"2024-03-09T15:36:06.525842+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":100,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":219,"num_read_kb":12489,"num_write":525,"num_write_kb":17953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.821324+0000","last_change":"2024-03-09T15:36:36.821377+0000","last_active":"2024-03-09T15:36:36.821324+0000","last_peered":"2024-03-09T15:36:36.821324+0000","last_clean":"2024-03-09T15:36:36.821324+0000","last_became_active":"2024-03-09T14:28:36.306912+0000","last_became_peered":"2024-03-09T14:28:36.306912+0000","last_unstale":"2024-03-09T15:36:36.821324+0000","last_undegraded":"2024-03-09T15:36:36.821324+0000","last_fullsized":"2024-03-09T15:36:36.821324+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:36.821128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:36.821128+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.821128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:16:15.465239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"261'854","reported_seq":2540,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.493222+0000","last_change":"2024-03-09T15:36:31.493336+0000","last_active":"2024-03-09T15:36:31.493222+0000","last_peered":"2024-03-09T15:36:31.493222+0000","last_clean":"2024-03-09T15:36:31.493222+0000","last_became_active":"2024-03-09T14:28:38.328738+0000","last_became_peered":"2024-03-09T14:28:38.328738+0000","last_unstale":"2024-03-09T15:36:31.493222+0000","last_undegraded":"2024-03-09T15:36:31.493222+0000","last_fullsized":"2024-03-09T15:36:31.493222+0000","mapping_epoch":26,"log_start":"261'800","ondisk_log_start":"261'800","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'854","last_scrub_stamp":"2024-03-09T15:36:31.493032+0000","last_deep_scrub":"261'854","last_deep_scrub_stamp":"2024-03-09T15:36:31.493032+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.493032+0000","objects_scrubbed":11,"log_size":54,"log_dups_size":800,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:44:58.935541+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34683,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1323,"num_read_kb":2148,"num_write":662,"num_write_kb":462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5354,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":511,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.192845+0000","last_change":"2024-03-09T15:36:32.192899+0000","last_active":"2024-03-09T15:36:32.192845+0000","last_peered":"2024-03-09T15:36:32.192845+0000","last_clean":"2024-03-09T15:36:32.192845+0000","last_became_active":"2024-03-09T14:28:40.386451+0000","last_became_peered":"2024-03-09T14:28:40.386451+0000","last_unstale":"2024-03-09T15:36:32.192845+0000","last_undegraded":"2024-03-09T15:36:32.192845+0000","last_fullsized":"2024-03-09T15:36:32.192845+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:32.192627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:32.192627+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.192627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:09:46.269539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"261'129","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.567045+0000","last_change":"2024-03-09T15:36:18.567214+0000","last_active":"2024-03-09T15:36:18.567045+0000","last_peered":"2024-03-09T15:36:18.567045+0000","last_clean":"2024-03-09T15:36:18.567045+0000","last_became_active":"2024-03-09T14:28:42.403013+0000","last_became_peered":"2024-03-09T14:28:42.403013+0000","last_unstale":"2024-03-09T15:36:18.567045+0000","last_undegraded":"2024-03-09T15:36:18.567045+0000","last_fullsized":"2024-03-09T15:36:18.567045+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'129","last_scrub_stamp":"2024-03-09T15:36:18.566788+0000","last_deep_scrub":"261'129","last_deep_scrub_stamp":"2024-03-09T15:36:18.566788+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.566788+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:45:17.069116+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1248,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":261,"num_read_kb":241,"num_write":300,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"261'48","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.363144+0000","last_change":"2024-03-09T15:36:38.363206+0000","last_active":"2024-03-09T15:36:38.363144+0000","last_peered":"2024-03-09T15:36:38.363144+0000","last_clean":"2024-03-09T15:36:38.363144+0000","last_became_active":"2024-03-09T14:28:34.296733+0000","last_became_peered":"2024-03-09T14:28:34.296733+0000","last_unstale":"2024-03-09T15:36:38.363144+0000","last_undegraded":"2024-03-09T15:36:38.363144+0000","last_fullsized":"2024-03-09T15:36:38.363144+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:38.362948+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:38.362948+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.362948+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:32:01.096416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":200,"num_read_kb":166,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":531,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.234263+0000","last_change":"2024-03-09T15:36:28.234318+0000","last_active":"2024-03-09T15:36:28.234263+0000","last_peered":"2024-03-09T15:36:28.234263+0000","last_clean":"2024-03-09T15:36:28.234263+0000","last_became_active":"2024-03-09T14:28:19.436985+0000","last_became_peered":"2024-03-09T14:28:19.436985+0000","last_unstale":"2024-03-09T15:36:28.234263+0000","last_undegraded":"2024-03-09T15:36:28.234263+0000","last_fullsized":"2024-03-09T15:36:28.234263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:28.234071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:28.234071+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.234071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:27:16.042052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"261'159","reported_seq":826,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.221405+0000","last_change":"2024-03-09T15:36:43.221405+0000","last_active":"2024-03-09T15:36:43.221405+0000","last_peered":"2024-03-09T15:36:43.221405+0000","last_clean":"2024-03-09T15:36:43.221405+0000","last_became_active":"2024-03-09T14:28:32.303176+0000","last_became_peered":"2024-03-09T14:28:32.303176+0000","last_unstale":"2024-03-09T15:36:43.221405+0000","last_undegraded":"2024-03-09T15:36:43.221405+0000","last_fullsized":"2024-03-09T15:36:43.221405+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'159","last_scrub_stamp":"2024-03-09T15:36:43.221036+0000","last_deep_scrub":"261'159","last_deep_scrub_stamp":"2024-03-09T15:36:43.221036+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.221036+0000","objects_scrubbed":7,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:18:01.135873+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8554724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":486,"num_read_kb":3616,"num_write":592,"num_write_kb":29269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"261'2","reported_seq":518,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.203415+0000","last_change":"2024-03-09T15:36:23.203478+0000","last_active":"2024-03-09T15:36:23.203415+0000","last_peered":"2024-03-09T15:36:23.203415+0000","last_clean":"2024-03-09T15:36:23.203415+0000","last_became_active":"2024-03-09T14:28:36.305971+0000","last_became_peered":"2024-03-09T14:28:36.305971+0000","last_unstale":"2024-03-09T15:36:23.203415+0000","last_undegraded":"2024-03-09T15:36:23.203415+0000","last_fullsized":"2024-03-09T15:36:23.203415+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:23.203217+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:23.203217+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.203217+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:42:55.897785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"261'230","reported_seq":1599,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:58.976894+0000","last_change":"2024-03-09T14:28:38.598201+0000","last_active":"2024-03-09T15:35:58.976894+0000","last_peered":"2024-03-09T15:35:58.976894+0000","last_clean":"2024-03-09T15:35:58.976894+0000","last_became_active":"2024-03-09T14:28:38.597403+0000","last_became_peered":"2024-03-09T14:28:38.597403+0000","last_unstale":"2024-03-09T15:35:58.976894+0000","last_undegraded":"2024-03-09T15:35:58.976894+0000","last_fullsized":"2024-03-09T15:35:58.976894+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":200,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":226,"num_read_kb":216,"num_write":195,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.247829+0000","last_change":"2024-03-09T15:36:32.247884+0000","last_active":"2024-03-09T15:36:32.247829+0000","last_peered":"2024-03-09T15:36:32.247829+0000","last_clean":"2024-03-09T15:36:32.247829+0000","last_became_active":"2024-03-09T14:28:40.371383+0000","last_became_peered":"2024-03-09T14:28:40.371383+0000","last_unstale":"2024-03-09T15:36:32.247829+0000","last_undegraded":"2024-03-09T15:36:32.247829+0000","last_fullsized":"2024-03-09T15:36:32.247829+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:32.247639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:32.247639+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.247639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:52:44.454327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"261'203","reported_seq":803,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.435372+0000","last_change":"2024-03-09T15:36:24.435440+0000","last_active":"2024-03-09T15:36:24.435372+0000","last_peered":"2024-03-09T15:36:24.435372+0000","last_clean":"2024-03-09T15:36:24.435372+0000","last_became_active":"2024-03-09T14:28:42.395781+0000","last_became_peered":"2024-03-09T14:28:42.395781+0000","last_unstale":"2024-03-09T15:36:24.435372+0000","last_undegraded":"2024-03-09T15:36:24.435372+0000","last_fullsized":"2024-03-09T15:36:24.435372+0000","mapping_epoch":30,"log_start":"247'100","ondisk_log_start":"247'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'203","last_scrub_stamp":"2024-03-09T15:36:24.435182+0000","last_deep_scrub":"261'203","last_deep_scrub_stamp":"2024-03-09T15:36:24.435182+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.435182+0000","objects_scrubbed":2,"log_size":103,"log_dups_size":100,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:28.404974+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":435,"num_read_kb":456,"num_write":425,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":851,"num_omap_keys":5,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"261'74","reported_seq":627,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.454421+0000","last_change":"2024-03-09T15:36:32.454571+0000","last_active":"2024-03-09T15:36:32.454421+0000","last_peered":"2024-03-09T15:36:32.454421+0000","last_clean":"2024-03-09T15:36:32.454421+0000","last_became_active":"2024-03-09T14:28:34.313302+0000","last_became_peered":"2024-03-09T14:28:34.313302+0000","last_unstale":"2024-03-09T15:36:32.454421+0000","last_undegraded":"2024-03-09T15:36:32.454421+0000","last_fullsized":"2024-03-09T15:36:32.454421+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'74","last_scrub_stamp":"2024-03-09T15:36:32.454104+0000","last_deep_scrub":"261'74","last_deep_scrub_stamp":"2024-03-09T15:36:32.454104+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.454104+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:47:12.348471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":225,"num_write":104,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":531,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.939037+0000","last_change":"2024-03-09T15:36:45.939088+0000","last_active":"2024-03-09T15:36:45.939037+0000","last_peered":"2024-03-09T15:36:45.939037+0000","last_clean":"2024-03-09T15:36:45.939037+0000","last_became_active":"2024-03-09T14:28:19.237937+0000","last_became_peered":"2024-03-09T14:28:19.237937+0000","last_unstale":"2024-03-09T15:36:45.939037+0000","last_undegraded":"2024-03-09T15:36:45.939037+0000","last_fullsized":"2024-03-09T15:36:45.939037+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:45.938842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:45.938842+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.938842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:04:17.332833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"261'130","reported_seq":714,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.409386+0000","last_change":"2024-03-09T15:36:47.409498+0000","last_active":"2024-03-09T15:36:47.409386+0000","last_peered":"2024-03-09T15:36:47.409386+0000","last_clean":"2024-03-09T15:36:47.409386+0000","last_became_active":"2024-03-09T14:28:32.303963+0000","last_became_peered":"2024-03-09T14:28:32.303963+0000","last_unstale":"2024-03-09T15:36:47.409386+0000","last_undegraded":"2024-03-09T15:36:47.409386+0000","last_fullsized":"2024-03-09T15:36:47.409386+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'130","last_scrub_stamp":"2024-03-09T15:36:47.409169+0000","last_deep_scrub":"261'130","last_deep_scrub_stamp":"2024-03-09T15:36:47.409169+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.409169+0000","objects_scrubbed":3,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:17:16.816469+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6898850,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":175,"num_read_kb":15115,"num_write":430,"num_write_kb":32085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.253103+0000","last_change":"2024-03-09T15:36:49.253156+0000","last_active":"2024-03-09T15:36:49.253103+0000","last_peered":"2024-03-09T15:36:49.253103+0000","last_clean":"2024-03-09T15:36:49.253103+0000","last_became_active":"2024-03-09T14:28:36.317586+0000","last_became_peered":"2024-03-09T14:28:36.317586+0000","last_unstale":"2024-03-09T15:36:49.253103+0000","last_undegraded":"2024-03-09T15:36:49.253103+0000","last_fullsized":"2024-03-09T15:36:49.253103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:49.252862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:49.252862+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.252862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:35:36.249291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"261'129","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:50.832966+0000","last_change":"2024-03-09T14:28:38.335628+0000","last_active":"2024-03-09T15:35:50.832966+0000","last_peered":"2024-03-09T15:35:50.832966+0000","last_clean":"2024-03-09T15:35:50.832966+0000","last_became_active":"2024-03-09T14:28:38.334463+0000","last_became_peered":"2024-03-09T14:28:38.334463+0000","last_unstale":"2024-03-09T15:35:50.832966+0000","last_undegraded":"2024-03-09T15:35:50.832966+0000","last_fullsized":"2024-03-09T15:35:50.832966+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":178,"num_read_kb":172,"num_write":132,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.103290+0000","last_change":"2024-03-09T15:36:20.103341+0000","last_active":"2024-03-09T15:36:20.103290+0000","last_peered":"2024-03-09T15:36:20.103290+0000","last_clean":"2024-03-09T15:36:20.103290+0000","last_became_active":"2024-03-09T14:28:40.370436+0000","last_became_peered":"2024-03-09T14:28:40.370436+0000","last_unstale":"2024-03-09T15:36:20.103290+0000","last_undegraded":"2024-03-09T15:36:20.103290+0000","last_fullsized":"2024-03-09T15:36:20.103290+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:20.103091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:20.103091+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.103091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:29:38.107241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"261'48","reported_seq":573,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.682061+0000","last_change":"2024-03-09T15:36:17.682122+0000","last_active":"2024-03-09T15:36:17.682061+0000","last_peered":"2024-03-09T15:36:17.682061+0000","last_clean":"2024-03-09T15:36:17.682061+0000","last_became_active":"2024-03-09T14:28:42.397058+0000","last_became_peered":"2024-03-09T14:28:42.397058+0000","last_unstale":"2024-03-09T15:36:17.682061+0000","last_undegraded":"2024-03-09T15:36:17.682061+0000","last_fullsized":"2024-03-09T15:36:17.682061+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:17.681861+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:17.681861+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.681861+0000","objects_scrubbed":1,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:38:01.033648+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":100,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"261'43","reported_seq":602,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.426247+0000","last_change":"2024-03-09T15:36:36.426328+0000","last_active":"2024-03-09T15:36:36.426247+0000","last_peered":"2024-03-09T15:36:36.426247+0000","last_clean":"2024-03-09T15:36:36.426247+0000","last_became_active":"2024-03-09T14:28:34.296953+0000","last_became_peered":"2024-03-09T14:28:34.296953+0000","last_unstale":"2024-03-09T15:36:36.426247+0000","last_undegraded":"2024-03-09T15:36:36.426247+0000","last_fullsized":"2024-03-09T15:36:36.426247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:36.426051+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:36.426051+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.426051+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:43:59.353686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":162,"num_write":75,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"261'165","reported_seq":808,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.889335+0000","last_change":"2024-03-09T15:36:48.889523+0000","last_active":"2024-03-09T15:36:48.889335+0000","last_peered":"2024-03-09T15:36:48.889335+0000","last_clean":"2024-03-09T15:36:48.889335+0000","last_became_active":"2024-03-09T14:28:32.302970+0000","last_became_peered":"2024-03-09T14:28:32.302970+0000","last_unstale":"2024-03-09T15:36:48.889335+0000","last_undegraded":"2024-03-09T15:36:48.889335+0000","last_fullsized":"2024-03-09T15:36:48.889335+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:48.889098+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:48.889098+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.889098+0000","objects_scrubbed":2,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:53:35.895952+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655822,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":94521,"num_write":632,"num_write_kb":45805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":529,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.321200+0000","last_change":"2024-03-09T15:36:39.321259+0000","last_active":"2024-03-09T15:36:39.321200+0000","last_peered":"2024-03-09T15:36:39.321200+0000","last_clean":"2024-03-09T15:36:39.321200+0000","last_became_active":"2024-03-09T14:28:19.236240+0000","last_became_peered":"2024-03-09T14:28:19.236240+0000","last_unstale":"2024-03-09T15:36:39.321200+0000","last_undegraded":"2024-03-09T15:36:39.321200+0000","last_fullsized":"2024-03-09T15:36:39.321200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:39.321005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:39.321005+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.321005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:47:02.712539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.328428+0000","last_change":"2024-03-09T15:36:47.328482+0000","last_active":"2024-03-09T15:36:47.328428+0000","last_peered":"2024-03-09T15:36:47.328428+0000","last_clean":"2024-03-09T15:36:47.328428+0000","last_became_active":"2024-03-09T14:28:36.314526+0000","last_became_peered":"2024-03-09T14:28:36.314526+0000","last_unstale":"2024-03-09T15:36:47.328428+0000","last_undegraded":"2024-03-09T15:36:47.328428+0000","last_fullsized":"2024-03-09T15:36:47.328428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:47.328224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:47.328224+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.328224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:38:07.502391+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"261'393","reported_seq":1084,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.271897+0000","last_change":"2024-03-09T15:36:40.271984+0000","last_active":"2024-03-09T15:36:40.271897+0000","last_peered":"2024-03-09T15:36:40.271897+0000","last_clean":"2024-03-09T15:36:40.271897+0000","last_became_active":"2024-03-09T14:28:38.334976+0000","last_became_peered":"2024-03-09T14:28:38.334976+0000","last_unstale":"2024-03-09T15:36:40.271897+0000","last_undegraded":"2024-03-09T15:36:40.271897+0000","last_fullsized":"2024-03-09T15:36:40.271897+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'393","last_scrub_stamp":"2024-03-09T15:36:40.271702+0000","last_deep_scrub":"261'393","last_deep_scrub_stamp":"2024-03-09T15:36:40.271702+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.271702+0000","objects_scrubbed":9,"log_size":93,"log_dups_size":300,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:45:05.080884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21042,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":640,"num_read_kb":1204,"num_write":456,"num_write_kb":268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2284,"num_omap_keys":4,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.131101+0000","last_change":"2024-03-09T15:36:20.131154+0000","last_active":"2024-03-09T15:36:20.131101+0000","last_peered":"2024-03-09T15:36:20.131101+0000","last_clean":"2024-03-09T15:36:20.131101+0000","last_became_active":"2024-03-09T14:28:40.379108+0000","last_became_peered":"2024-03-09T14:28:40.379108+0000","last_unstale":"2024-03-09T15:36:20.131101+0000","last_undegraded":"2024-03-09T15:36:20.131101+0000","last_fullsized":"2024-03-09T15:36:20.131101+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:20.130903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:20.130903+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.130903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:41:11.167873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"261'104","reported_seq":639,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.264151+0000","last_change":"2024-03-09T15:36:26.264345+0000","last_active":"2024-03-09T15:36:26.264151+0000","last_peered":"2024-03-09T15:36:26.264151+0000","last_clean":"2024-03-09T15:36:26.264151+0000","last_became_active":"2024-03-09T14:28:42.405277+0000","last_became_peered":"2024-03-09T14:28:42.405277+0000","last_unstale":"2024-03-09T15:36:26.264151+0000","last_undegraded":"2024-03-09T15:36:26.264151+0000","last_fullsized":"2024-03-09T15:36:26.264151+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'104","last_scrub_stamp":"2024-03-09T15:36:26.263887+0000","last_deep_scrub":"261'104","last_deep_scrub_stamp":"2024-03-09T15:36:26.263887+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.263887+0000","objects_scrubbed":3,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:52.786785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":717,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185,"num_read_kb":160,"num_write":261,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"261'43","reported_seq":584,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.293213+0000","last_change":"2024-03-09T15:36:48.293348+0000","last_active":"2024-03-09T15:36:48.293213+0000","last_peered":"2024-03-09T15:36:48.293213+0000","last_clean":"2024-03-09T15:36:48.293213+0000","last_became_active":"2024-03-09T14:28:34.296660+0000","last_became_peered":"2024-03-09T14:28:34.296660+0000","last_unstale":"2024-03-09T15:36:48.293213+0000","last_undegraded":"2024-03-09T15:36:48.293213+0000","last_fullsized":"2024-03-09T15:36:48.293213+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:48.292984+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:48.292984+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.292984+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:58:19.981031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":129,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"261'157","reported_seq":796,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.182597+0000","last_change":"2024-03-09T15:36:44.182693+0000","last_active":"2024-03-09T15:36:44.182597+0000","last_peered":"2024-03-09T15:36:44.182597+0000","last_clean":"2024-03-09T15:36:44.182597+0000","last_became_active":"2024-03-09T14:28:32.305318+0000","last_became_peered":"2024-03-09T14:28:32.305318+0000","last_unstale":"2024-03-09T15:36:44.182597+0000","last_undegraded":"2024-03-09T15:36:44.182597+0000","last_fullsized":"2024-03-09T15:36:44.182597+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'157","last_scrub_stamp":"2024-03-09T15:36:44.182405+0000","last_deep_scrub":"261'157","last_deep_scrub_stamp":"2024-03-09T15:36:44.182405+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.182405+0000","objects_scrubbed":9,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:14:02.068526+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262245,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":357,"num_read_kb":10578,"num_write":456,"num_write_kb":20505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":525,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:11.227202+0000","last_change":"2024-03-09T14:28:19.241475+0000","last_active":"2024-03-09T15:35:11.227202+0000","last_peered":"2024-03-09T15:35:11.227202+0000","last_clean":"2024-03-09T15:35:11.227202+0000","last_became_active":"2024-03-09T14:28:19.240704+0000","last_became_peered":"2024-03-09T14:28:19.240704+0000","last_unstale":"2024-03-09T15:35:11.227202+0000","last_undegraded":"2024-03-09T15:35:11.227202+0000","last_fullsized":"2024-03-09T15:35:11.227202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:18.207340+0000","last_clean_scrub_stamp":"2024-03-09T14:28:18.207340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.191317+0000","last_change":"2024-03-09T15:36:25.191369+0000","last_active":"2024-03-09T15:36:25.191317+0000","last_peered":"2024-03-09T15:36:25.191317+0000","last_clean":"2024-03-09T15:36:25.191317+0000","last_became_active":"2024-03-09T14:28:36.314676+0000","last_became_peered":"2024-03-09T14:28:36.314676+0000","last_unstale":"2024-03-09T15:36:25.191317+0000","last_undegraded":"2024-03-09T15:36:25.191317+0000","last_fullsized":"2024-03-09T15:36:25.191317+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.191100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.191100+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.191100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:12:04.217794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"261'205","reported_seq":795,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.476798+0000","last_change":"2024-03-09T15:36:35.476874+0000","last_active":"2024-03-09T15:36:35.476798+0000","last_peered":"2024-03-09T15:36:35.476798+0000","last_clean":"2024-03-09T15:36:35.476798+0000","last_became_active":"2024-03-09T14:28:38.595028+0000","last_became_peered":"2024-03-09T14:28:38.595028+0000","last_unstale":"2024-03-09T15:36:35.476798+0000","last_undegraded":"2024-03-09T15:36:35.476798+0000","last_fullsized":"2024-03-09T15:36:35.476798+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'205","last_scrub_stamp":"2024-03-09T15:36:35.476600+0000","last_deep_scrub":"261'205","last_deep_scrub_stamp":"2024-03-09T15:36:35.476600+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.476600+0000","objects_scrubbed":8,"log_size":105,"log_dups_size":100,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:50:56.058991+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":269,"num_read_kb":262,"num_write":230,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":330,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"261'501","reported_seq":2307,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.727185+0000","last_change":"2024-03-09T15:36:23.727185+0000","last_active":"2024-03-09T15:36:23.727185+0000","last_peered":"2024-03-09T15:36:23.727185+0000","last_clean":"2024-03-09T15:36:23.727185+0000","last_became_active":"2024-03-09T14:28:40.369165+0000","last_became_peered":"2024-03-09T14:28:40.369165+0000","last_unstale":"2024-03-09T15:36:23.727185+0000","last_undegraded":"2024-03-09T15:36:23.727185+0000","last_fullsized":"2024-03-09T15:36:23.727185+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'501","last_scrub_stamp":"2024-03-09T15:36:23.726932+0000","last_deep_scrub":"261'501","last_deep_scrub_stamp":"2024-03-09T15:36:23.726932+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.726932+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":400,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:05:11.145403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"261'79","reported_seq":577,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.487857+0000","last_change":"2024-03-09T15:36:27.487857+0000","last_active":"2024-03-09T15:36:27.487857+0000","last_peered":"2024-03-09T15:36:27.487857+0000","last_clean":"2024-03-09T15:36:27.487857+0000","last_became_active":"2024-03-09T14:28:42.392402+0000","last_became_peered":"2024-03-09T14:28:42.392402+0000","last_unstale":"2024-03-09T15:36:27.487857+0000","last_undegraded":"2024-03-09T15:36:27.487857+0000","last_fullsized":"2024-03-09T15:36:27.487857+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'79","last_scrub_stamp":"2024-03-09T15:36:27.487564+0000","last_deep_scrub":"261'79","last_deep_scrub_stamp":"2024-03-09T15:36:27.487564+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.487564+0000","objects_scrubbed":3,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:09:13.446303+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":57,"num_write":172,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"261'48","reported_seq":598,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.188879+0000","last_change":"2024-03-09T15:36:18.188944+0000","last_active":"2024-03-09T15:36:18.188879+0000","last_peered":"2024-03-09T15:36:18.188879+0000","last_clean":"2024-03-09T15:36:18.188879+0000","last_became_active":"2024-03-09T14:29:06.440556+0000","last_became_peered":"2024-03-09T14:29:06.440556+0000","last_unstale":"2024-03-09T15:36:18.188879+0000","last_undegraded":"2024-03-09T15:36:18.188879+0000","last_fullsized":"2024-03-09T15:36:18.188879+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:18.188663+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:18.188663+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.188663+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:03:02.670504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":216,"num_read_kb":182,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"261'145","reported_seq":691,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.492729+0000","last_change":"2024-03-09T14:28:32.319314+0000","last_active":"2024-03-09T15:36:06.492729+0000","last_peered":"2024-03-09T15:36:06.492729+0000","last_clean":"2024-03-09T15:36:06.492729+0000","last_became_active":"2024-03-09T14:28:32.318971+0000","last_became_peered":"2024-03-09T14:28:32.318971+0000","last_unstale":"2024-03-09T15:36:06.492729+0000","last_undegraded":"2024-03-09T15:36:06.492729+0000","last_fullsized":"2024-03-09T15:36:06.492729+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":151,"num_read_kb":8647,"num_write":476,"num_write_kb":27350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":528,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.293647+0000","last_change":"2024-03-09T15:36:40.293706+0000","last_active":"2024-03-09T15:36:40.293647+0000","last_peered":"2024-03-09T15:36:40.293647+0000","last_clean":"2024-03-09T15:36:40.293647+0000","last_became_active":"2024-03-09T14:28:19.237750+0000","last_became_peered":"2024-03-09T14:28:19.237750+0000","last_unstale":"2024-03-09T15:36:40.293647+0000","last_undegraded":"2024-03-09T15:36:40.293647+0000","last_fullsized":"2024-03-09T15:36:40.293647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:40.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:40.293440+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:43:40.824370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:16.011425+0000","last_change":"2024-03-09T14:28:36.328518+0000","last_active":"2024-03-09T15:35:16.011425+0000","last_peered":"2024-03-09T15:35:16.011425+0000","last_clean":"2024-03-09T15:35:16.011425+0000","last_became_active":"2024-03-09T14:28:36.327601+0000","last_became_peered":"2024-03-09T14:28:36.327601+0000","last_unstale":"2024-03-09T15:35:16.011425+0000","last_undegraded":"2024-03-09T15:35:16.011425+0000","last_fullsized":"2024-03-09T15:35:16.011425+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"261'111","reported_seq":675,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.173092+0000","last_change":"2024-03-09T15:36:26.173180+0000","last_active":"2024-03-09T15:36:26.173092+0000","last_peered":"2024-03-09T15:36:26.173092+0000","last_clean":"2024-03-09T15:36:26.173092+0000","last_became_active":"2024-03-09T14:28:38.347750+0000","last_became_peered":"2024-03-09T14:28:38.347750+0000","last_unstale":"2024-03-09T15:36:26.173092+0000","last_undegraded":"2024-03-09T15:36:26.173092+0000","last_fullsized":"2024-03-09T15:36:26.173092+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'111","last_scrub_stamp":"2024-03-09T15:36:26.172900+0000","last_deep_scrub":"261'111","last_deep_scrub_stamp":"2024-03-09T15:36:26.172900+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.172900+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:00:14.752406+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"261'514","reported_seq":2495,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.486338+0000","last_change":"2024-03-09T15:36:34.486405+0000","last_active":"2024-03-09T15:36:34.486338+0000","last_peered":"2024-03-09T15:36:34.486338+0000","last_clean":"2024-03-09T15:36:34.486338+0000","last_became_active":"2024-03-09T14:28:40.386859+0000","last_became_peered":"2024-03-09T14:28:40.386859+0000","last_unstale":"2024-03-09T15:36:34.486338+0000","last_undegraded":"2024-03-09T15:36:34.486338+0000","last_fullsized":"2024-03-09T15:36:34.486338+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'514","last_scrub_stamp":"2024-03-09T15:36:34.486115+0000","last_deep_scrub":"261'514","last_deep_scrub_stamp":"2024-03-09T15:36:34.486115+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.486115+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":500,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:43:00.093432+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"261'108","reported_seq":618,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.271001+0000","last_change":"2024-03-09T15:36:25.271065+0000","last_active":"2024-03-09T15:36:25.271001+0000","last_peered":"2024-03-09T15:36:25.271001+0000","last_clean":"2024-03-09T15:36:25.271001+0000","last_became_active":"2024-03-09T14:28:42.406139+0000","last_became_peered":"2024-03-09T14:28:42.406139+0000","last_unstale":"2024-03-09T15:36:25.271001+0000","last_undegraded":"2024-03-09T15:36:25.271001+0000","last_fullsized":"2024-03-09T15:36:25.271001+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'108","last_scrub_stamp":"2024-03-09T15:36:25.270811+0000","last_deep_scrub":"261'108","last_deep_scrub_stamp":"2024-03-09T15:36:25.270811+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.270811+0000","objects_scrubbed":2,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:05:43.169466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":65,"num_write":235,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"261'216","reported_seq":788,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:16.011393+0000","last_change":"2024-03-09T14:28:34.301025+0000","last_active":"2024-03-09T15:35:16.011393+0000","last_peered":"2024-03-09T15:35:16.011393+0000","last_clean":"2024-03-09T15:35:16.011393+0000","last_became_active":"2024-03-09T14:28:34.299988+0000","last_became_peered":"2024-03-09T14:28:34.299988+0000","last_unstale":"2024-03-09T15:35:16.011393+0000","last_undegraded":"2024-03-09T15:35:16.011393+0000","last_fullsized":"2024-03-09T15:35:16.011393+0000","mapping_epoch":22,"log_start":"258'200","ondisk_log_start":"258'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":200,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":843,"num_read_kb":645,"num_write":377,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"261'150","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.454415+0000","last_change":"2024-03-09T14:28:32.299582+0000","last_active":"2024-03-09T15:36:06.454415+0000","last_peered":"2024-03-09T15:36:06.454415+0000","last_clean":"2024-03-09T15:36:06.454415+0000","last_became_active":"2024-03-09T14:28:32.298977+0000","last_became_peered":"2024-03-09T14:28:32.298977+0000","last_unstale":"2024-03-09T15:36:06.454415+0000","last_undegraded":"2024-03-09T15:36:06.454415+0000","last_fullsized":"2024-03-09T15:36:06.454415+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242905,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":173,"num_read_kb":5781,"num_write":510,"num_write_kb":37567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.336163+0000","last_change":"2024-03-09T15:36:36.336215+0000","last_active":"2024-03-09T15:36:36.336163+0000","last_peered":"2024-03-09T15:36:36.336163+0000","last_clean":"2024-03-09T15:36:36.336163+0000","last_became_active":"2024-03-09T14:28:19.241700+0000","last_became_peered":"2024-03-09T14:28:19.241700+0000","last_unstale":"2024-03-09T15:36:36.336163+0000","last_undegraded":"2024-03-09T15:36:36.336163+0000","last_fullsized":"2024-03-09T15:36:36.336163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:36.335976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:36.335976+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.335976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:25:25.910931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"32'3","reported_seq":824,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.186014+0000","last_change":"2024-03-09T14:28:36.317874+0000","last_active":"2024-03-09T15:36:12.186014+0000","last_peered":"2024-03-09T15:36:12.186014+0000","last_clean":"2024-03-09T15:36:12.186014+0000","last_became_active":"2024-03-09T14:28:36.317327+0000","last_became_peered":"2024-03-09T14:28:36.317327+0000","last_unstale":"2024-03-09T15:36:12.186014+0000","last_undegraded":"2024-03-09T15:36:12.186014+0000","last_fullsized":"2024-03-09T15:36:12.186014+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":484,"num_read_kb":482,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"261'478","reported_seq":1226,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.248700+0000","last_change":"2024-03-09T15:36:38.248788+0000","last_active":"2024-03-09T15:36:38.248700+0000","last_peered":"2024-03-09T15:36:38.248700+0000","last_clean":"2024-03-09T15:36:38.248700+0000","last_became_active":"2024-03-09T14:28:38.602908+0000","last_became_peered":"2024-03-09T14:28:38.602908+0000","last_unstale":"2024-03-09T15:36:38.248700+0000","last_undegraded":"2024-03-09T15:36:38.248700+0000","last_fullsized":"2024-03-09T15:36:38.248700+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'478","last_scrub_stamp":"2024-03-09T15:36:38.248482+0000","last_deep_scrub":"261'478","last_deep_scrub_stamp":"2024-03-09T15:36:38.248482+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.248482+0000","objects_scrubbed":8,"log_size":78,"log_dups_size":400,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:50:57.022028+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29305,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":797,"num_read_kb":1561,"num_write":550,"num_write_kb":344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.482306+0000","last_change":"2024-03-09T15:36:35.482326+0000","last_active":"2024-03-09T15:36:35.482306+0000","last_peered":"2024-03-09T15:36:35.482306+0000","last_clean":"2024-03-09T15:36:35.482306+0000","last_became_active":"2024-03-09T14:28:40.375884+0000","last_became_peered":"2024-03-09T14:28:40.375884+0000","last_unstale":"2024-03-09T15:36:35.482306+0000","last_undegraded":"2024-03-09T15:36:35.482306+0000","last_fullsized":"2024-03-09T15:36:35.482306+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:35.482232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:35.482232+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.482232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:57:53.662951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"261'56","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.577103+0000","last_change":"2024-03-09T15:36:13.577103+0000","last_active":"2024-03-09T15:36:13.577103+0000","last_peered":"2024-03-09T15:36:13.577103+0000","last_clean":"2024-03-09T15:36:13.577103+0000","last_became_active":"2024-03-09T14:28:42.405998+0000","last_became_peered":"2024-03-09T14:28:42.405998+0000","last_unstale":"2024-03-09T15:36:13.577103+0000","last_undegraded":"2024-03-09T15:36:13.577103+0000","last_fullsized":"2024-03-09T15:36:13.577103+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:13.576825+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:13.576825+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.576825+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:29:57.437161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":52,"num_write":123,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1f","version":"261'149","reported_seq":735,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.301854+0000","last_change":"2024-03-09T15:36:24.301934+0000","last_active":"2024-03-09T15:36:24.301854+0000","last_peered":"2024-03-09T15:36:24.301854+0000","last_clean":"2024-03-09T15:36:24.301854+0000","last_became_active":"2024-03-09T14:28:42.408774+0000","last_became_peered":"2024-03-09T14:28:42.408774+0000","last_unstale":"2024-03-09T15:36:24.301854+0000","last_undegraded":"2024-03-09T15:36:24.301854+0000","last_fullsized":"2024-03-09T15:36:24.301854+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'149","last_scrub_stamp":"2024-03-09T15:36:24.301670+0000","last_deep_scrub":"261'149","last_deep_scrub_stamp":"2024-03-09T15:36:24.301670+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.301670+0000","objects_scrubbed":3,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:31:20.109756+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":256,"num_read_kb":233,"num_write":254,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"261'183","reported_seq":753,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.152146+0000","last_change":"2024-03-09T15:36:32.152205+0000","last_active":"2024-03-09T15:36:32.152146+0000","last_peered":"2024-03-09T15:36:32.152146+0000","last_clean":"2024-03-09T15:36:32.152146+0000","last_became_active":"2024-03-09T14:28:34.314893+0000","last_became_peered":"2024-03-09T14:28:34.314893+0000","last_unstale":"2024-03-09T15:36:32.152146+0000","last_undegraded":"2024-03-09T15:36:32.152146+0000","last_fullsized":"2024-03-09T15:36:32.152146+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'183","last_scrub_stamp":"2024-03-09T15:36:32.151949+0000","last_deep_scrub":"261'183","last_deep_scrub_stamp":"2024-03-09T15:36:32.151949+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.151949+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":100,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:40:37.495594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":706,"num_read_kb":592,"num_write":291,"num_write_kb":155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"261'132","reported_seq":686,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.480459+0000","last_change":"2024-03-09T15:36:50.480551+0000","last_active":"2024-03-09T15:36:50.480459+0000","last_peered":"2024-03-09T15:36:50.480459+0000","last_clean":"2024-03-09T15:36:50.480459+0000","last_became_active":"2024-03-09T14:28:32.322364+0000","last_became_peered":"2024-03-09T14:28:32.322364+0000","last_unstale":"2024-03-09T15:36:50.480459+0000","last_undegraded":"2024-03-09T15:36:50.480459+0000","last_fullsized":"2024-03-09T15:36:50.480459+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'132","last_scrub_stamp":"2024-03-09T15:36:50.480252+0000","last_deep_scrub":"261'132","last_deep_scrub_stamp":"2024-03-09T15:36:50.480252+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.480252+0000","objects_scrubbed":3,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:46:23.715589+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":128,"num_write":389,"num_write_kb":27813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":512,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.576225+0000","last_change":"2024-03-09T15:36:25.576277+0000","last_active":"2024-03-09T15:36:25.576225+0000","last_peered":"2024-03-09T15:36:25.576225+0000","last_clean":"2024-03-09T15:36:25.576225+0000","last_became_active":"2024-03-09T14:28:36.313984+0000","last_became_peered":"2024-03-09T14:28:36.313984+0000","last_unstale":"2024-03-09T15:36:25.576225+0000","last_undegraded":"2024-03-09T15:36:25.576225+0000","last_fullsized":"2024-03-09T15:36:25.576225+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.576027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.576027+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.576027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:16:03.346315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"261'191","reported_seq":775,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.186294+0000","last_change":"2024-03-09T14:28:38.345412+0000","last_active":"2024-03-09T15:36:12.186294+0000","last_peered":"2024-03-09T15:36:12.186294+0000","last_clean":"2024-03-09T15:36:12.186294+0000","last_became_active":"2024-03-09T14:28:38.338262+0000","last_became_peered":"2024-03-09T14:28:38.338262+0000","last_unstale":"2024-03-09T15:36:12.186294+0000","last_undegraded":"2024-03-09T15:36:12.186294+0000","last_fullsized":"2024-03-09T15:36:12.186294+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":100,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15994,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":315,"num_read_kb":601,"num_write":220,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.450333+0000","last_change":"2024-03-09T15:36:31.450390+0000","last_active":"2024-03-09T15:36:31.450333+0000","last_peered":"2024-03-09T15:36:31.450333+0000","last_clean":"2024-03-09T15:36:31.450333+0000","last_became_active":"2024-03-09T14:28:40.389277+0000","last_became_peered":"2024-03-09T14:28:40.389277+0000","last_unstale":"2024-03-09T15:36:31.450333+0000","last_undegraded":"2024-03-09T15:36:31.450333+0000","last_fullsized":"2024-03-09T15:36:31.450333+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:31.450128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:31.450128+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.450128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:52:04.464244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"261'56","reported_seq":560,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.467550+0000","last_change":"2024-03-09T15:36:22.467616+0000","last_active":"2024-03-09T15:36:22.467550+0000","last_peered":"2024-03-09T15:36:22.467550+0000","last_clean":"2024-03-09T15:36:22.467550+0000","last_became_active":"2024-03-09T14:28:42.409229+0000","last_became_peered":"2024-03-09T14:28:42.409229+0000","last_unstale":"2024-03-09T15:36:22.467550+0000","last_undegraded":"2024-03-09T15:36:22.467550+0000","last_fullsized":"2024-03-09T15:36:22.467550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:22.467355+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:22.467355+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.467355+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:46:09.238574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":828,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":44,"num_write":116,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"261'78","reported_seq":640,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.558934+0000","last_change":"2024-03-09T15:36:26.558992+0000","last_active":"2024-03-09T15:36:26.558934+0000","last_peered":"2024-03-09T15:36:26.558934+0000","last_clean":"2024-03-09T15:36:26.558934+0000","last_became_active":"2024-03-09T14:28:34.320474+0000","last_became_peered":"2024-03-09T14:28:34.320474+0000","last_unstale":"2024-03-09T15:36:26.558934+0000","last_undegraded":"2024-03-09T15:36:26.558934+0000","last_fullsized":"2024-03-09T15:36:26.558934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'78","last_scrub_stamp":"2024-03-09T15:36:26.558741+0000","last_deep_scrub":"261'78","last_deep_scrub_stamp":"2024-03-09T15:36:26.558741+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.558741+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:58:25.865037+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":231,"num_write":128,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"261'159","reported_seq":793,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101678+0000","last_change":"2024-03-09T14:28:32.317900+0000","last_active":"2024-03-09T15:36:14.101678+0000","last_peered":"2024-03-09T15:36:14.101678+0000","last_clean":"2024-03-09T15:36:14.101678+0000","last_became_active":"2024-03-09T14:28:32.315355+0000","last_became_peered":"2024-03-09T14:28:32.315355+0000","last_unstale":"2024-03-09T15:36:14.101678+0000","last_undegraded":"2024-03-09T15:36:14.101678+0000","last_fullsized":"2024-03-09T15:36:14.101678+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":325,"num_read_kb":85989,"num_write":581,"num_write_kb":44811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.649651+0000","last_change":"2024-03-09T15:36:49.649702+0000","last_active":"2024-03-09T15:36:49.649651+0000","last_peered":"2024-03-09T15:36:49.649651+0000","last_clean":"2024-03-09T15:36:49.649651+0000","last_became_active":"2024-03-09T14:28:36.311487+0000","last_became_peered":"2024-03-09T14:28:36.311487+0000","last_unstale":"2024-03-09T15:36:49.649651+0000","last_undegraded":"2024-03-09T15:36:49.649651+0000","last_fullsized":"2024-03-09T15:36:49.649651+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:49.649459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:49.649459+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.649459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:35:37.031126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"261'39","reported_seq":552,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739505+0000","last_change":"2024-03-09T14:28:38.333369+0000","last_active":"2024-03-09T15:36:10.739505+0000","last_peered":"2024-03-09T15:36:10.739505+0000","last_clean":"2024-03-09T15:36:10.739505+0000","last_became_active":"2024-03-09T14:28:38.332610+0000","last_became_peered":"2024-03-09T14:28:38.332610+0000","last_unstale":"2024-03-09T15:36:10.739505+0000","last_undegraded":"2024-03-09T15:36:10.739505+0000","last_fullsized":"2024-03-09T15:36:10.739505+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":41,"num_write":30,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.258023+0000","last_change":"2024-03-09T15:36:35.258136+0000","last_active":"2024-03-09T15:36:35.258023+0000","last_peered":"2024-03-09T15:36:35.258023+0000","last_clean":"2024-03-09T15:36:35.258023+0000","last_became_active":"2024-03-09T14:28:40.367795+0000","last_became_peered":"2024-03-09T14:28:40.367795+0000","last_unstale":"2024-03-09T15:36:35.258023+0000","last_undegraded":"2024-03-09T15:36:35.258023+0000","last_fullsized":"2024-03-09T15:36:35.258023+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:35.257744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:35.257744+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.257744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:42:18.958156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"261'172","reported_seq":700,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.304337+0000","last_change":"2024-03-09T15:36:22.304413+0000","last_active":"2024-03-09T15:36:22.304337+0000","last_peered":"2024-03-09T15:36:22.304337+0000","last_clean":"2024-03-09T15:36:22.304337+0000","last_became_active":"2024-03-09T14:28:42.408913+0000","last_became_peered":"2024-03-09T14:28:42.408913+0000","last_unstale":"2024-03-09T15:36:22.304337+0000","last_undegraded":"2024-03-09T15:36:22.304337+0000","last_fullsized":"2024-03-09T15:36:22.304337+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'172","last_scrub_stamp":"2024-03-09T15:36:22.304104+0000","last_deep_scrub":"261'172","last_deep_scrub_stamp":"2024-03-09T15:36:22.304104+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.304104+0000","objects_scrubbed":2,"log_size":72,"log_dups_size":100,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:45:45.379355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":208,"num_write":351,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":398,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"261'54","reported_seq":603,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.672674+0000","last_change":"2024-03-09T15:36:47.672732+0000","last_active":"2024-03-09T15:36:47.672674+0000","last_peered":"2024-03-09T15:36:47.672674+0000","last_clean":"2024-03-09T15:36:47.672674+0000","last_became_active":"2024-03-09T14:28:34.330813+0000","last_became_peered":"2024-03-09T14:28:34.330813+0000","last_unstale":"2024-03-09T15:36:47.672674+0000","last_undegraded":"2024-03-09T15:36:47.672674+0000","last_fullsized":"2024-03-09T15:36:47.672674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'54","last_scrub_stamp":"2024-03-09T15:36:47.672474+0000","last_deep_scrub":"261'54","last_deep_scrub_stamp":"2024-03-09T15:36:47.672474+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.672474+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:48:25.059611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":220,"num_read_kb":178,"num_write":76,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"261'133","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.908045+0000","last_change":"2024-03-09T15:36:41.908262+0000","last_active":"2024-03-09T15:36:41.908045+0000","last_peered":"2024-03-09T15:36:41.908045+0000","last_clean":"2024-03-09T15:36:41.908045+0000","last_became_active":"2024-03-09T14:28:32.311843+0000","last_became_peered":"2024-03-09T14:28:32.311843+0000","last_unstale":"2024-03-09T15:36:41.908045+0000","last_undegraded":"2024-03-09T15:36:41.908045+0000","last_fullsized":"2024-03-09T15:36:41.908045+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'133","last_scrub_stamp":"2024-03-09T15:36:41.907782+0000","last_deep_scrub":"261'133","last_deep_scrub_stamp":"2024-03-09T15:36:41.907782+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.907782+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:05:53.440636+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655838,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":157,"num_read_kb":5348,"num_write":475,"num_write_kb":35385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.141589+0000","last_change":"2024-03-09T15:36:43.141654+0000","last_active":"2024-03-09T15:36:43.141589+0000","last_peered":"2024-03-09T15:36:43.141589+0000","last_clean":"2024-03-09T15:36:43.141589+0000","last_became_active":"2024-03-09T14:28:36.323335+0000","last_became_peered":"2024-03-09T14:28:36.323335+0000","last_unstale":"2024-03-09T15:36:43.141589+0000","last_undegraded":"2024-03-09T15:36:43.141589+0000","last_fullsized":"2024-03-09T15:36:43.141589+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:43.141360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:43.141360+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.141360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:29:28.525098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"261'227","reported_seq":834,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.585747+0000","last_change":"2024-03-09T15:36:42.585928+0000","last_active":"2024-03-09T15:36:42.585747+0000","last_peered":"2024-03-09T15:36:42.585747+0000","last_clean":"2024-03-09T15:36:42.585747+0000","last_became_active":"2024-03-09T14:28:38.344951+0000","last_became_peered":"2024-03-09T14:28:38.344951+0000","last_unstale":"2024-03-09T15:36:42.585747+0000","last_undegraded":"2024-03-09T15:36:42.585747+0000","last_fullsized":"2024-03-09T15:36:42.585747+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'227","last_scrub_stamp":"2024-03-09T15:36:42.585503+0000","last_deep_scrub":"261'227","last_deep_scrub_stamp":"2024-03-09T15:36:42.585503+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.585503+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":200,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:09:44.534942+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7791,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":353,"num_read_kb":629,"num_write":244,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.128626+0000","last_change":"2024-03-09T15:36:22.128680+0000","last_active":"2024-03-09T15:36:22.128626+0000","last_peered":"2024-03-09T15:36:22.128626+0000","last_clean":"2024-03-09T15:36:22.128626+0000","last_became_active":"2024-03-09T14:28:40.370313+0000","last_became_peered":"2024-03-09T14:28:40.370313+0000","last_unstale":"2024-03-09T15:36:22.128626+0000","last_undegraded":"2024-03-09T15:36:22.128626+0000","last_fullsized":"2024-03-09T15:36:22.128626+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.128436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.128436+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.128436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:31:27.821903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"261'47","reported_seq":550,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.629026+0000","last_change":"2024-03-09T15:36:15.629026+0000","last_active":"2024-03-09T15:36:15.629026+0000","last_peered":"2024-03-09T15:36:15.629026+0000","last_clean":"2024-03-09T15:36:15.629026+0000","last_became_active":"2024-03-09T14:28:42.404360+0000","last_became_peered":"2024-03-09T14:28:42.404360+0000","last_unstale":"2024-03-09T15:36:15.629026+0000","last_undegraded":"2024-03-09T15:36:15.629026+0000","last_fullsized":"2024-03-09T15:36:15.629026+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'47","last_scrub_stamp":"2024-03-09T15:36:15.628774+0000","last_deep_scrub":"261'47","last_deep_scrub_stamp":"2024-03-09T15:36:15.628774+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.628774+0000","objects_scrubbed":2,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:04:15.376099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":43,"num_write":116,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"261'88","reported_seq":651,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.697326+0000","last_change":"2024-03-09T15:36:50.697389+0000","last_active":"2024-03-09T15:36:50.697326+0000","last_peered":"2024-03-09T15:36:50.697326+0000","last_clean":"2024-03-09T15:36:50.697326+0000","last_became_active":"2024-03-09T14:28:34.305301+0000","last_became_peered":"2024-03-09T14:28:34.305301+0000","last_unstale":"2024-03-09T15:36:50.697326+0000","last_undegraded":"2024-03-09T15:36:50.697326+0000","last_fullsized":"2024-03-09T15:36:50.697326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'88","last_scrub_stamp":"2024-03-09T15:36:50.697133+0000","last_deep_scrub":"261'88","last_deep_scrub_stamp":"2024-03-09T15:36:50.697133+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.697133+0000","objects_scrubbed":1,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:31:45.483994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":317,"num_write":145,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2876,"num_omap_keys":12,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"261'143","reported_seq":701,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.738692+0000","last_change":"2024-03-09T14:28:32.293199+0000","last_active":"2024-03-09T15:36:10.738692+0000","last_peered":"2024-03-09T15:36:10.738692+0000","last_clean":"2024-03-09T15:36:10.738692+0000","last_became_active":"2024-03-09T14:28:32.292393+0000","last_became_peered":"2024-03-09T14:28:32.292393+0000","last_unstale":"2024-03-09T15:36:10.738692+0000","last_undegraded":"2024-03-09T15:36:10.738692+0000","last_fullsized":"2024-03-09T15:36:10.738692+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":100,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6900333,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":188,"num_read_kb":14616,"num_write":468,"num_write_kb":23484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":511,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.124875+0000","last_change":"2024-03-09T15:36:21.124931+0000","last_active":"2024-03-09T15:36:21.124875+0000","last_peered":"2024-03-09T15:36:21.124875+0000","last_clean":"2024-03-09T15:36:21.124875+0000","last_became_active":"2024-03-09T14:28:36.328946+0000","last_became_peered":"2024-03-09T14:28:36.328946+0000","last_unstale":"2024-03-09T15:36:21.124875+0000","last_undegraded":"2024-03-09T15:36:21.124875+0000","last_fullsized":"2024-03-09T15:36:21.124875+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.124678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.124678+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.124678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:16:00.704022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"261'172","reported_seq":766,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.110832+0000","last_change":"2024-03-09T15:36:49.110993+0000","last_active":"2024-03-09T15:36:49.110832+0000","last_peered":"2024-03-09T15:36:49.110832+0000","last_clean":"2024-03-09T15:36:49.110832+0000","last_became_active":"2024-03-09T14:28:38.335107+0000","last_became_peered":"2024-03-09T14:28:38.335107+0000","last_unstale":"2024-03-09T15:36:49.110832+0000","last_undegraded":"2024-03-09T15:36:49.110832+0000","last_fullsized":"2024-03-09T15:36:49.110832+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'172","last_scrub_stamp":"2024-03-09T15:36:49.110544+0000","last_deep_scrub":"261'172","last_deep_scrub_stamp":"2024-03-09T15:36:49.110544+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.110544+0000","objects_scrubbed":4,"log_size":72,"log_dups_size":100,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:18:50.261252+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9765,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":273,"num_read_kb":518,"num_write":186,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.274828+0000","last_change":"2024-03-09T15:36:36.274880+0000","last_active":"2024-03-09T15:36:36.274828+0000","last_peered":"2024-03-09T15:36:36.274828+0000","last_clean":"2024-03-09T15:36:36.274828+0000","last_became_active":"2024-03-09T14:28:40.363778+0000","last_became_peered":"2024-03-09T14:28:40.363778+0000","last_unstale":"2024-03-09T15:36:36.274828+0000","last_undegraded":"2024-03-09T15:36:36.274828+0000","last_fullsized":"2024-03-09T15:36:36.274828+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:36.274642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:36.274642+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.274642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:53:47.883218+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"261'159","reported_seq":742,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.206354+0000","last_change":"2024-03-09T15:36:29.206413+0000","last_active":"2024-03-09T15:36:29.206354+0000","last_peered":"2024-03-09T15:36:29.206354+0000","last_clean":"2024-03-09T15:36:29.206354+0000","last_became_active":"2024-03-09T14:28:34.325031+0000","last_became_peered":"2024-03-09T14:28:34.325031+0000","last_unstale":"2024-03-09T15:36:29.206354+0000","last_undegraded":"2024-03-09T15:36:29.206354+0000","last_fullsized":"2024-03-09T15:36:29.206354+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'159","last_scrub_stamp":"2024-03-09T15:36:29.206170+0000","last_deep_scrub":"261'159","last_deep_scrub_stamp":"2024-03-09T15:36:29.206170+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.206170+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:05:19.071599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":584,"num_read_kb":461,"num_write":254,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"261'111","reported_seq":652,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.185582+0000","last_change":"2024-03-09T14:28:32.296634+0000","last_active":"2024-03-09T15:36:12.185582+0000","last_peered":"2024-03-09T15:36:12.185582+0000","last_clean":"2024-03-09T15:36:12.185582+0000","last_became_active":"2024-03-09T14:28:32.296179+0000","last_became_peered":"2024-03-09T14:28:32.296179+0000","last_unstale":"2024-03-09T15:36:12.185582+0000","last_undegraded":"2024-03-09T15:36:12.185582+0000","last_fullsized":"2024-03-09T15:36:12.185582+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263265,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":136,"num_read_kb":11385,"num_write":371,"num_write_kb":17181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.739314+0000","last_change":"2024-03-09T14:28:36.320059+0000","last_active":"2024-03-09T15:36:10.739314+0000","last_peered":"2024-03-09T15:36:10.739314+0000","last_clean":"2024-03-09T15:36:10.739314+0000","last_became_active":"2024-03-09T14:28:36.319702+0000","last_became_peered":"2024-03-09T14:28:36.319702+0000","last_unstale":"2024-03-09T15:36:10.739314+0000","last_undegraded":"2024-03-09T15:36:10.739314+0000","last_fullsized":"2024-03-09T15:36:10.739314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:35.280359+0000","last_clean_scrub_stamp":"2024-03-09T14:28:35.280359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"261'316","reported_seq":965,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.480382+0000","last_change":"2024-03-09T15:36:33.480469+0000","last_active":"2024-03-09T15:36:33.480382+0000","last_peered":"2024-03-09T15:36:33.480382+0000","last_clean":"2024-03-09T15:36:33.480382+0000","last_became_active":"2024-03-09T14:28:38.332661+0000","last_became_peered":"2024-03-09T14:28:38.332661+0000","last_unstale":"2024-03-09T15:36:33.480382+0000","last_undegraded":"2024-03-09T15:36:33.480382+0000","last_fullsized":"2024-03-09T15:36:33.480382+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'316","last_scrub_stamp":"2024-03-09T15:36:33.480185+0000","last_deep_scrub":"261'316","last_deep_scrub_stamp":"2024-03-09T15:36:33.480185+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.480185+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":300,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:48:54.047973+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16427,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":565,"num_read_kb":1160,"num_write":446,"num_write_kb":361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22450,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.679444+0000","last_change":"2024-03-09T15:36:21.679497+0000","last_active":"2024-03-09T15:36:21.679444+0000","last_peered":"2024-03-09T15:36:21.679444+0000","last_clean":"2024-03-09T15:36:21.679444+0000","last_became_active":"2024-03-09T14:28:40.378829+0000","last_became_peered":"2024-03-09T14:28:40.378829+0000","last_unstale":"2024-03-09T15:36:21.679444+0000","last_undegraded":"2024-03-09T15:36:21.679444+0000","last_fullsized":"2024-03-09T15:36:21.679444+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.679245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.679245+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.679245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:45:45.724495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"261'126","reported_seq":635,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.487636+0000","last_change":"2024-03-09T15:36:26.487767+0000","last_active":"2024-03-09T15:36:26.487636+0000","last_peered":"2024-03-09T15:36:26.487636+0000","last_clean":"2024-03-09T15:36:26.487636+0000","last_became_active":"2024-03-09T14:28:42.392509+0000","last_became_peered":"2024-03-09T14:28:42.392509+0000","last_unstale":"2024-03-09T15:36:26.487636+0000","last_undegraded":"2024-03-09T15:36:26.487636+0000","last_fullsized":"2024-03-09T15:36:26.487636+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'126","last_scrub_stamp":"2024-03-09T15:36:26.487384+0000","last_deep_scrub":"261'126","last_deep_scrub_stamp":"2024-03-09T15:36:26.487384+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.487384+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":100,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:33:29.950896+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":643,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":173,"num_write":275,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"261'132","reported_seq":694,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.361023+0000","last_change":"2024-03-09T15:36:45.361023+0000","last_active":"2024-03-09T15:36:45.361023+0000","last_peered":"2024-03-09T15:36:45.361023+0000","last_clean":"2024-03-09T15:36:45.361023+0000","last_became_active":"2024-03-09T14:28:34.330686+0000","last_became_peered":"2024-03-09T14:28:34.330686+0000","last_unstale":"2024-03-09T15:36:45.361023+0000","last_undegraded":"2024-03-09T15:36:45.361023+0000","last_fullsized":"2024-03-09T15:36:45.361023+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'132","last_scrub_stamp":"2024-03-09T15:36:45.360646+0000","last_deep_scrub":"261'132","last_deep_scrub_stamp":"2024-03-09T15:36:45.360646+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.360646+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:37:31.175012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":527,"num_read_kb":451,"num_write":212,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"261'143","reported_seq":691,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.218846+0000","last_change":"2024-03-09T15:36:42.218957+0000","last_active":"2024-03-09T15:36:42.218846+0000","last_peered":"2024-03-09T15:36:42.218846+0000","last_clean":"2024-03-09T15:36:42.218846+0000","last_became_active":"2024-03-09T14:28:32.301087+0000","last_became_peered":"2024-03-09T14:28:32.301087+0000","last_unstale":"2024-03-09T15:36:42.218846+0000","last_undegraded":"2024-03-09T15:36:42.218846+0000","last_fullsized":"2024-03-09T15:36:42.218846+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'143","last_scrub_stamp":"2024-03-09T15:36:42.218633+0000","last_deep_scrub":"261'143","last_deep_scrub_stamp":"2024-03-09T15:36:42.218633+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.218633+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":100,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:38:14.730766+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":137,"num_read_kb":3393,"num_write":460,"num_write_kb":23270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.130798+0000","last_change":"2024-03-09T15:36:23.130851+0000","last_active":"2024-03-09T15:36:23.130798+0000","last_peered":"2024-03-09T15:36:23.130798+0000","last_clean":"2024-03-09T15:36:23.130798+0000","last_became_active":"2024-03-09T14:28:36.328814+0000","last_became_peered":"2024-03-09T14:28:36.328814+0000","last_unstale":"2024-03-09T15:36:23.130798+0000","last_undegraded":"2024-03-09T15:36:23.130798+0000","last_fullsized":"2024-03-09T15:36:23.130798+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:23.130603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:23.130603+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.130603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:54:38.685950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"261'260","reported_seq":873,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.173698+0000","last_change":"2024-03-09T15:36:47.173796+0000","last_active":"2024-03-09T15:36:47.173698+0000","last_peered":"2024-03-09T15:36:47.173698+0000","last_clean":"2024-03-09T15:36:47.173698+0000","last_became_active":"2024-03-09T14:28:38.331930+0000","last_became_peered":"2024-03-09T14:28:38.331930+0000","last_unstale":"2024-03-09T15:36:47.173698+0000","last_undegraded":"2024-03-09T15:36:47.173698+0000","last_fullsized":"2024-03-09T15:36:47.173698+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'260","last_scrub_stamp":"2024-03-09T15:36:47.173452+0000","last_deep_scrub":"261'260","last_deep_scrub_stamp":"2024-03-09T15:36:47.173452+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.173452+0000","objects_scrubbed":8,"log_size":60,"log_dups_size":200,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:18:24.859389+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21830,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":424,"num_read_kb":798,"num_write":342,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11460,"num_omap_keys":24,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.208618+0000","last_change":"2024-03-09T15:36:47.208675+0000","last_active":"2024-03-09T15:36:47.208618+0000","last_peered":"2024-03-09T15:36:47.208618+0000","last_clean":"2024-03-09T15:36:47.208618+0000","last_became_active":"2024-03-09T14:28:40.366723+0000","last_became_peered":"2024-03-09T14:28:40.366723+0000","last_unstale":"2024-03-09T15:36:47.208618+0000","last_undegraded":"2024-03-09T15:36:47.208618+0000","last_fullsized":"2024-03-09T15:36:47.208618+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:47.208400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:47.208400+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.208400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:22:48.196708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"261'112","reported_seq":627,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.220706+0000","last_change":"2024-03-09T15:36:32.220786+0000","last_active":"2024-03-09T15:36:32.220706+0000","last_peered":"2024-03-09T15:36:32.220706+0000","last_clean":"2024-03-09T15:36:32.220706+0000","last_became_active":"2024-03-09T14:28:42.395375+0000","last_became_peered":"2024-03-09T14:28:42.395375+0000","last_unstale":"2024-03-09T15:36:32.220706+0000","last_undegraded":"2024-03-09T15:36:32.220706+0000","last_fullsized":"2024-03-09T15:36:32.220706+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'112","last_scrub_stamp":"2024-03-09T15:36:32.220503+0000","last_deep_scrub":"261'112","last_deep_scrub_stamp":"2024-03-09T15:36:32.220503+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.220503+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":100,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:58:59.546341+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1199,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":139,"num_read_kb":136,"num_write":341,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"261'99","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:05.836325+0000","last_change":"2024-03-09T14:28:34.312670+0000","last_active":"2024-03-09T15:36:05.836325+0000","last_peered":"2024-03-09T15:36:05.836325+0000","last_clean":"2024-03-09T15:36:05.836325+0000","last_became_active":"2024-03-09T14:28:34.312297+0000","last_became_peered":"2024-03-09T14:28:34.312297+0000","last_unstale":"2024-03-09T15:36:05.836325+0000","last_undegraded":"2024-03-09T15:36:05.836325+0000","last_fullsized":"2024-03-09T15:36:05.836325+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":415,"num_read_kb":347,"num_write":146,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"261'120","reported_seq":697,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.470441+0000","last_change":"2024-03-09T15:36:49.470646+0000","last_active":"2024-03-09T15:36:49.470441+0000","last_peered":"2024-03-09T15:36:49.470441+0000","last_clean":"2024-03-09T15:36:49.470441+0000","last_became_active":"2024-03-09T14:28:32.312937+0000","last_became_peered":"2024-03-09T14:28:32.312937+0000","last_unstale":"2024-03-09T15:36:49.470441+0000","last_undegraded":"2024-03-09T15:36:49.470441+0000","last_fullsized":"2024-03-09T15:36:49.470441+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'120","last_scrub_stamp":"2024-03-09T15:36:49.470219+0000","last_deep_scrub":"261'120","last_deep_scrub_stamp":"2024-03-09T15:36:49.470219+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.470219+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:58:33.393108+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262175,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":159,"num_read_kb":47424,"num_write":430,"num_write_kb":24173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.273419+0000","last_change":"2024-03-09T15:36:41.273535+0000","last_active":"2024-03-09T15:36:41.273419+0000","last_peered":"2024-03-09T15:36:41.273419+0000","last_clean":"2024-03-09T15:36:41.273419+0000","last_became_active":"2024-03-09T14:28:36.316094+0000","last_became_peered":"2024-03-09T14:28:36.316094+0000","last_unstale":"2024-03-09T15:36:41.273419+0000","last_undegraded":"2024-03-09T15:36:41.273419+0000","last_fullsized":"2024-03-09T15:36:41.273419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:41.273162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:41.273162+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.273162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:10:54.703371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"261'195","reported_seq":790,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.519051+0000","last_change":"2024-03-09T15:36:36.519208+0000","last_active":"2024-03-09T15:36:36.519051+0000","last_peered":"2024-03-09T15:36:36.519051+0000","last_clean":"2024-03-09T15:36:36.519051+0000","last_became_active":"2024-03-09T14:28:38.595522+0000","last_became_peered":"2024-03-09T14:28:38.595522+0000","last_unstale":"2024-03-09T15:36:36.519051+0000","last_undegraded":"2024-03-09T15:36:36.519051+0000","last_fullsized":"2024-03-09T15:36:36.519051+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'195","last_scrub_stamp":"2024-03-09T15:36:36.518818+0000","last_deep_scrub":"261'195","last_deep_scrub_stamp":"2024-03-09T15:36:36.518818+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.518818+0000","objects_scrubbed":5,"log_size":95,"log_dups_size":100,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:00:11.684059+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12058,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":311,"num_read_kb":573,"num_write":216,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.585690+0000","last_change":"2024-03-09T15:36:22.585690+0000","last_active":"2024-03-09T15:36:22.585690+0000","last_peered":"2024-03-09T15:36:22.585690+0000","last_clean":"2024-03-09T15:36:22.585690+0000","last_became_active":"2024-03-09T14:28:40.376549+0000","last_became_peered":"2024-03-09T14:28:40.376549+0000","last_unstale":"2024-03-09T15:36:22.585690+0000","last_undegraded":"2024-03-09T15:36:22.585690+0000","last_fullsized":"2024-03-09T15:36:22.585690+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.585447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.585447+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.585447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:45:35.078062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"261'45","reported_seq":550,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.666451+0000","last_change":"2024-03-09T15:36:19.666451+0000","last_active":"2024-03-09T15:36:19.666451+0000","last_peered":"2024-03-09T15:36:19.666451+0000","last_clean":"2024-03-09T15:36:19.666451+0000","last_became_active":"2024-03-09T14:28:42.394780+0000","last_became_peered":"2024-03-09T14:28:42.394780+0000","last_unstale":"2024-03-09T15:36:19.666451+0000","last_undegraded":"2024-03-09T15:36:19.666451+0000","last_fullsized":"2024-03-09T15:36:19.666451+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'45","last_scrub_stamp":"2024-03-09T15:36:19.666178+0000","last_deep_scrub":"261'45","last_deep_scrub_stamp":"2024-03-09T15:36:19.666178+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.666178+0000","objects_scrubbed":3,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:53:32.256553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":84,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"261'68","reported_seq":624,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.190901+0000","last_change":"2024-03-09T15:36:32.190971+0000","last_active":"2024-03-09T15:36:32.190901+0000","last_peered":"2024-03-09T15:36:32.190901+0000","last_clean":"2024-03-09T15:36:32.190901+0000","last_became_active":"2024-03-09T14:28:34.313125+0000","last_became_peered":"2024-03-09T14:28:34.313125+0000","last_unstale":"2024-03-09T15:36:32.190901+0000","last_undegraded":"2024-03-09T15:36:32.190901+0000","last_fullsized":"2024-03-09T15:36:32.190901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'68","last_scrub_stamp":"2024-03-09T15:36:32.190702+0000","last_deep_scrub":"261'68","last_deep_scrub_stamp":"2024-03-09T15:36:32.190702+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.190702+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:14:08.005357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"261'187","reported_seq":792,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.486344+0000","last_change":"2024-03-09T14:28:32.304248+0000","last_active":"2024-03-09T15:36:06.486344+0000","last_peered":"2024-03-09T15:36:06.486344+0000","last_clean":"2024-03-09T15:36:06.486344+0000","last_became_active":"2024-03-09T14:28:32.303835+0000","last_became_peered":"2024-03-09T14:28:32.303835+0000","last_unstale":"2024-03-09T15:36:06.486344+0000","last_undegraded":"2024-03-09T15:36:06.486344+0000","last_fullsized":"2024-03-09T15:36:06.486344+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":100,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":371,"num_read_kb":1431,"num_write":999,"num_write_kb":8832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.488481+0000","last_change":"2024-03-09T15:36:29.488628+0000","last_active":"2024-03-09T15:36:29.488481+0000","last_peered":"2024-03-09T15:36:29.488481+0000","last_clean":"2024-03-09T15:36:29.488481+0000","last_became_active":"2024-03-09T14:28:36.322060+0000","last_became_peered":"2024-03-09T14:28:36.322060+0000","last_unstale":"2024-03-09T15:36:29.488481+0000","last_undegraded":"2024-03-09T15:36:29.488481+0000","last_fullsized":"2024-03-09T15:36:29.488481+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:29.488248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:29.488248+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.488248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:40:34.293730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"261'96","reported_seq":647,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.151474+0000","last_change":"2024-03-09T15:36:24.151474+0000","last_active":"2024-03-09T15:36:24.151474+0000","last_peered":"2024-03-09T15:36:24.151474+0000","last_clean":"2024-03-09T15:36:24.151474+0000","last_became_active":"2024-03-09T14:28:38.595749+0000","last_became_peered":"2024-03-09T14:28:38.595749+0000","last_unstale":"2024-03-09T15:36:24.151474+0000","last_undegraded":"2024-03-09T15:36:24.151474+0000","last_fullsized":"2024-03-09T15:36:24.151474+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'96","last_scrub_stamp":"2024-03-09T15:36:24.151188+0000","last_deep_scrub":"261'96","last_deep_scrub_stamp":"2024-03-09T15:36:24.151188+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.151188+0000","objects_scrubbed":7,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:06:38.216926+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":132,"num_read_kb":127,"num_write":124,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11222,"num_omap_keys":30,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"261'489","reported_seq":2266,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.168974+0000","last_change":"2024-03-09T15:36:28.169035+0000","last_active":"2024-03-09T15:36:28.168974+0000","last_peered":"2024-03-09T15:36:28.168974+0000","last_clean":"2024-03-09T15:36:28.168974+0000","last_became_active":"2024-03-09T14:28:40.376946+0000","last_became_peered":"2024-03-09T14:28:40.376946+0000","last_unstale":"2024-03-09T15:36:28.168974+0000","last_undegraded":"2024-03-09T15:36:28.168974+0000","last_fullsized":"2024-03-09T15:36:28.168974+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'489","last_scrub_stamp":"2024-03-09T15:36:28.168784+0000","last_deep_scrub":"261'489","last_deep_scrub_stamp":"2024-03-09T15:36:28.168784+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.168784+0000","objects_scrubbed":1,"log_size":89,"log_dups_size":400,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:26:51.629900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.10","version":"261'45","reported_seq":559,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.174006+0000","last_change":"2024-03-09T15:36:31.174074+0000","last_active":"2024-03-09T15:36:31.174006+0000","last_peered":"2024-03-09T15:36:31.174006+0000","last_clean":"2024-03-09T15:36:31.174006+0000","last_became_active":"2024-03-09T14:28:42.397829+0000","last_became_peered":"2024-03-09T14:28:42.397829+0000","last_unstale":"2024-03-09T15:36:31.174006+0000","last_undegraded":"2024-03-09T15:36:31.174006+0000","last_fullsized":"2024-03-09T15:36:31.174006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'45","last_scrub_stamp":"2024-03-09T15:36:31.173791+0000","last_deep_scrub":"261'45","last_deep_scrub_stamp":"2024-03-09T15:36:31.173791+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.173791+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:44.989801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":97,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"261'144","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.216346+0000","last_change":"2024-03-09T15:36:37.216431+0000","last_active":"2024-03-09T15:36:37.216346+0000","last_peered":"2024-03-09T15:36:37.216346+0000","last_clean":"2024-03-09T15:36:37.216346+0000","last_became_active":"2024-03-09T14:28:34.319891+0000","last_became_peered":"2024-03-09T14:28:34.319891+0000","last_unstale":"2024-03-09T15:36:37.216346+0000","last_undegraded":"2024-03-09T15:36:37.216346+0000","last_fullsized":"2024-03-09T15:36:37.216346+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'144","last_scrub_stamp":"2024-03-09T15:36:37.216142+0000","last_deep_scrub":"261'144","last_deep_scrub_stamp":"2024-03-09T15:36:37.216142+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.216142+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:10:29.932991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":571,"num_read_kb":433,"num_write":263,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"261'165","reported_seq":820,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.905667+0000","last_change":"2024-03-09T15:36:39.905809+0000","last_active":"2024-03-09T15:36:39.905667+0000","last_peered":"2024-03-09T15:36:39.905667+0000","last_clean":"2024-03-09T15:36:39.905667+0000","last_became_active":"2024-03-09T14:28:32.323198+0000","last_became_peered":"2024-03-09T14:28:32.323198+0000","last_unstale":"2024-03-09T15:36:39.905667+0000","last_undegraded":"2024-03-09T15:36:39.905667+0000","last_fullsized":"2024-03-09T15:36:39.905667+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:39.905460+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:39.905460+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.905460+0000","objects_scrubbed":4,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:16:12.321789+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4801550,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":406,"num_read_kb":2095,"num_write":652,"num_write_kb":25158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.154703+0000","last_change":"2024-03-09T15:36:45.154756+0000","last_active":"2024-03-09T15:36:45.154703+0000","last_peered":"2024-03-09T15:36:45.154703+0000","last_clean":"2024-03-09T15:36:45.154703+0000","last_became_active":"2024-03-09T14:28:36.328529+0000","last_became_peered":"2024-03-09T14:28:36.328529+0000","last_unstale":"2024-03-09T15:36:45.154703+0000","last_undegraded":"2024-03-09T15:36:45.154703+0000","last_fullsized":"2024-03-09T15:36:45.154703+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:45.154500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:45.154500+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.154500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:30:04.214748+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"261'224","reported_seq":828,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.124658+0000","last_change":"2024-03-09T14:28:38.335758+0000","last_active":"2024-03-09T15:36:13.124658+0000","last_peered":"2024-03-09T15:36:13.124658+0000","last_clean":"2024-03-09T15:36:13.124658+0000","last_became_active":"2024-03-09T14:28:38.334991+0000","last_became_peered":"2024-03-09T14:28:38.334991+0000","last_unstale":"2024-03-09T15:36:13.124658+0000","last_undegraded":"2024-03-09T15:36:13.124658+0000","last_fullsized":"2024-03-09T15:36:13.124658+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:37.297771+0000","last_clean_scrub_stamp":"2024-03-09T14:28:37.297771+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":200,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13314,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":373,"num_read_kb":687,"num_write":264,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.226908+0000","last_change":"2024-03-09T15:36:33.226963+0000","last_active":"2024-03-09T15:36:33.226908+0000","last_peered":"2024-03-09T15:36:33.226908+0000","last_clean":"2024-03-09T15:36:33.226908+0000","last_became_active":"2024-03-09T14:28:40.368039+0000","last_became_peered":"2024-03-09T14:28:40.368039+0000","last_unstale":"2024-03-09T15:36:33.226908+0000","last_undegraded":"2024-03-09T15:36:33.226908+0000","last_fullsized":"2024-03-09T15:36:33.226908+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:33.226717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:33.226717+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.226717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:17.480958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"261'79","reported_seq":609,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.203923+0000","last_change":"2024-03-09T15:36:28.203997+0000","last_active":"2024-03-09T15:36:28.203923+0000","last_peered":"2024-03-09T15:36:28.203923+0000","last_clean":"2024-03-09T15:36:28.203923+0000","last_became_active":"2024-03-09T14:28:42.413840+0000","last_became_peered":"2024-03-09T14:28:42.413840+0000","last_unstale":"2024-03-09T15:36:28.203923+0000","last_undegraded":"2024-03-09T15:36:28.203923+0000","last_fullsized":"2024-03-09T15:36:28.203923+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'79","last_scrub_stamp":"2024-03-09T15:36:28.203746+0000","last_deep_scrub":"261'79","last_deep_scrub_stamp":"2024-03-09T15:36:28.203746+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.203746+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:20:00.519578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":129,"num_write":187,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"261'36","reported_seq":582,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.213368+0000","last_change":"2024-03-09T15:36:35.213428+0000","last_active":"2024-03-09T15:36:35.213368+0000","last_peered":"2024-03-09T15:36:35.213368+0000","last_clean":"2024-03-09T15:36:35.213368+0000","last_became_active":"2024-03-09T14:28:34.314194+0000","last_became_peered":"2024-03-09T14:28:34.314194+0000","last_unstale":"2024-03-09T15:36:35.213368+0000","last_undegraded":"2024-03-09T15:36:35.213368+0000","last_fullsized":"2024-03-09T15:36:35.213368+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'36","last_scrub_stamp":"2024-03-09T15:36:35.213167+0000","last_deep_scrub":"261'36","last_deep_scrub_stamp":"2024-03-09T15:36:35.213167+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.213167+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:30.268110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":120,"num_write":54,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"261'150","reported_seq":812,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.865057+0000","last_change":"2024-03-09T15:36:50.865141+0000","last_active":"2024-03-09T15:36:50.865057+0000","last_peered":"2024-03-09T15:36:50.865057+0000","last_clean":"2024-03-09T15:36:50.865057+0000","last_became_active":"2024-03-09T14:28:32.307702+0000","last_became_peered":"2024-03-09T14:28:32.307702+0000","last_unstale":"2024-03-09T15:36:50.865057+0000","last_undegraded":"2024-03-09T15:36:50.865057+0000","last_fullsized":"2024-03-09T15:36:50.865057+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'150","last_scrub_stamp":"2024-03-09T15:36:50.864867+0000","last_deep_scrub":"261'150","last_deep_scrub_stamp":"2024-03-09T15:36:50.864867+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.864867+0000","objects_scrubbed":3,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:16:58.825247+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":273,"num_read_kb":1758,"num_write":483,"num_write_kb":23078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":512,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.746052+0000","last_change":"2024-03-09T15:36:34.746107+0000","last_active":"2024-03-09T15:36:34.746052+0000","last_peered":"2024-03-09T15:36:34.746052+0000","last_clean":"2024-03-09T15:36:34.746052+0000","last_became_active":"2024-03-09T14:28:36.304241+0000","last_became_peered":"2024-03-09T14:28:36.304241+0000","last_unstale":"2024-03-09T15:36:34.746052+0000","last_undegraded":"2024-03-09T15:36:34.746052+0000","last_fullsized":"2024-03-09T15:36:34.746052+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:34.745840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:34.745840+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.745840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:07:47.752362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"261'182","reported_seq":771,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.223598+0000","last_change":"2024-03-09T15:36:38.223787+0000","last_active":"2024-03-09T15:36:38.223598+0000","last_peered":"2024-03-09T15:36:38.223598+0000","last_clean":"2024-03-09T15:36:38.223598+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:36:38.223598+0000","last_undegraded":"2024-03-09T15:36:38.223598+0000","last_fullsized":"2024-03-09T15:36:38.223598+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'182","last_scrub_stamp":"2024-03-09T15:36:38.223354+0000","last_deep_scrub":"261'182","last_deep_scrub_stamp":"2024-03-09T15:36:38.223354+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.223354+0000","objects_scrubbed":4,"log_size":82,"log_dups_size":100,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:01:53.396420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":521,"num_write":190,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.171034+0000","last_change":"2024-03-09T15:36:24.171034+0000","last_active":"2024-03-09T15:36:24.171034+0000","last_peered":"2024-03-09T15:36:24.171034+0000","last_clean":"2024-03-09T15:36:24.171034+0000","last_became_active":"2024-03-09T14:28:40.371820+0000","last_became_peered":"2024-03-09T14:28:40.371820+0000","last_unstale":"2024-03-09T15:36:24.171034+0000","last_undegraded":"2024-03-09T15:36:24.171034+0000","last_fullsized":"2024-03-09T15:36:24.171034+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:24.170785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:24.170785+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.170785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:48:51.479283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.16","version":"261'56","reported_seq":580,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.501906+0000","last_change":"2024-03-09T15:36:28.501999+0000","last_active":"2024-03-09T15:36:28.501906+0000","last_peered":"2024-03-09T15:36:28.501906+0000","last_clean":"2024-03-09T15:36:28.501906+0000","last_became_active":"2024-03-09T14:28:42.392326+0000","last_became_peered":"2024-03-09T14:28:42.392326+0000","last_unstale":"2024-03-09T15:36:28.501906+0000","last_undegraded":"2024-03-09T15:36:28.501906+0000","last_fullsized":"2024-03-09T15:36:28.501906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:28.501691+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:28.501691+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.501691+0000","objects_scrubbed":3,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:17:33.874116+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":772,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":99,"num_write":117,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"261'70","reported_seq":642,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.589053+0000","last_change":"2024-03-09T15:36:44.589114+0000","last_active":"2024-03-09T15:36:44.589053+0000","last_peered":"2024-03-09T15:36:44.589053+0000","last_clean":"2024-03-09T15:36:44.589053+0000","last_became_active":"2024-03-09T14:28:34.323969+0000","last_became_peered":"2024-03-09T14:28:34.323969+0000","last_unstale":"2024-03-09T15:36:44.589053+0000","last_undegraded":"2024-03-09T15:36:44.589053+0000","last_fullsized":"2024-03-09T15:36:44.589053+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'70","last_scrub_stamp":"2024-03-09T15:36:44.588877+0000","last_deep_scrub":"261'70","last_deep_scrub_stamp":"2024-03-09T15:36:44.588877+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.588877+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:27:16.122474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":256,"num_write":98,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"261'176","reported_seq":769,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.185642+0000","last_change":"2024-03-09T14:28:32.338014+0000","last_active":"2024-03-09T15:36:12.185642+0000","last_peered":"2024-03-09T15:36:12.185642+0000","last_clean":"2024-03-09T15:36:12.185642+0000","last_became_active":"2024-03-09T14:28:32.337755+0000","last_became_peered":"2024-03-09T14:28:32.337755+0000","last_unstale":"2024-03-09T15:36:12.185642+0000","last_undegraded":"2024-03-09T15:36:12.185642+0000","last_fullsized":"2024-03-09T15:36:12.185642+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":100,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663535,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":299,"num_read_kb":3602,"num_write":667,"num_write_kb":11566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.298938+0000","last_change":"2024-03-09T15:36:43.298995+0000","last_active":"2024-03-09T15:36:43.298938+0000","last_peered":"2024-03-09T15:36:43.298938+0000","last_clean":"2024-03-09T15:36:43.298938+0000","last_became_active":"2024-03-09T14:28:36.317569+0000","last_became_peered":"2024-03-09T14:28:36.317569+0000","last_unstale":"2024-03-09T15:36:43.298938+0000","last_undegraded":"2024-03-09T15:36:43.298938+0000","last_fullsized":"2024-03-09T15:36:43.298938+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:43.298743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:43.298743+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.298743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:54:29.675835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"261'154","reported_seq":723,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.559070+0000","last_change":"2024-03-09T15:36:39.559219+0000","last_active":"2024-03-09T15:36:39.559070+0000","last_peered":"2024-03-09T15:36:39.559070+0000","last_clean":"2024-03-09T15:36:39.559070+0000","last_became_active":"2024-03-09T14:28:38.347370+0000","last_became_peered":"2024-03-09T14:28:38.347370+0000","last_unstale":"2024-03-09T15:36:39.559070+0000","last_undegraded":"2024-03-09T15:36:39.559070+0000","last_fullsized":"2024-03-09T15:36:39.559070+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'154","last_scrub_stamp":"2024-03-09T15:36:39.558842+0000","last_deep_scrub":"261'154","last_deep_scrub_stamp":"2024-03-09T15:36:39.558842+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.558842+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":100,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:06:29.281642+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18257,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":531,"num_write":172,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.105659+0000","last_change":"2024-03-09T15:36:22.105715+0000","last_active":"2024-03-09T15:36:22.105659+0000","last_peered":"2024-03-09T15:36:22.105659+0000","last_clean":"2024-03-09T15:36:22.105659+0000","last_became_active":"2024-03-09T14:28:40.376513+0000","last_became_peered":"2024-03-09T14:28:40.376513+0000","last_unstale":"2024-03-09T15:36:22.105659+0000","last_undegraded":"2024-03-09T15:36:22.105659+0000","last_fullsized":"2024-03-09T15:36:22.105659+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.105441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.105441+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.105441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:26:26.970816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"261'96","reported_seq":988,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.177168+0000","last_change":"2024-03-09T15:36:30.177269+0000","last_active":"2024-03-09T15:36:30.177168+0000","last_peered":"2024-03-09T15:36:30.177168+0000","last_clean":"2024-03-09T15:36:30.177168+0000","last_became_active":"2024-03-09T14:28:42.409717+0000","last_became_peered":"2024-03-09T14:28:42.409717+0000","last_unstale":"2024-03-09T15:36:30.177168+0000","last_undegraded":"2024-03-09T15:36:30.177168+0000","last_fullsized":"2024-03-09T15:36:30.177168+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'96","last_scrub_stamp":"2024-03-09T15:36:30.176968+0000","last_deep_scrub":"261'96","last_deep_scrub_stamp":"2024-03-09T15:36:30.176968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.176968+0000","objects_scrubbed":4,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:07:18.249596+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":500,"num_read_kb":493,"num_write":215,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"261'53","reported_seq":595,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.270125+0000","last_change":"2024-03-09T15:36:40.270186+0000","last_active":"2024-03-09T15:36:40.270125+0000","last_peered":"2024-03-09T15:36:40.270125+0000","last_clean":"2024-03-09T15:36:40.270125+0000","last_became_active":"2024-03-09T14:28:34.313380+0000","last_became_peered":"2024-03-09T14:28:34.313380+0000","last_unstale":"2024-03-09T15:36:40.270125+0000","last_undegraded":"2024-03-09T15:36:40.270125+0000","last_fullsized":"2024-03-09T15:36:40.270125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'53","last_scrub_stamp":"2024-03-09T15:36:40.269935+0000","last_deep_scrub":"261'53","last_deep_scrub_stamp":"2024-03-09T15:36:40.269935+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.269935+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:22:56.243161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":197,"num_read_kb":156,"num_write":74,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"261'161","reported_seq":749,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.518065+0000","last_change":"2024-03-09T14:28:32.293553+0000","last_active":"2024-03-09T15:36:06.518065+0000","last_peered":"2024-03-09T15:36:06.518065+0000","last_clean":"2024-03-09T15:36:06.518065+0000","last_became_active":"2024-03-09T14:28:32.292862+0000","last_became_peered":"2024-03-09T14:28:32.292862+0000","last_unstale":"2024-03-09T15:36:06.518065+0000","last_undegraded":"2024-03-09T15:36:06.518065+0000","last_fullsized":"2024-03-09T15:36:06.518065+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":264,"num_read_kb":35693,"num_write":668,"num_write_kb":24072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"261'2","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.597930+0000","last_change":"2024-03-09T15:36:23.597930+0000","last_active":"2024-03-09T15:36:23.597930+0000","last_peered":"2024-03-09T15:36:23.597930+0000","last_clean":"2024-03-09T15:36:23.597930+0000","last_became_active":"2024-03-09T14:28:36.313405+0000","last_became_peered":"2024-03-09T14:28:36.313405+0000","last_unstale":"2024-03-09T15:36:23.597930+0000","last_undegraded":"2024-03-09T15:36:23.597930+0000","last_fullsized":"2024-03-09T15:36:23.597930+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:23.597678+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:23.597678+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.597678+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:11:34.369475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"261'213","reported_seq":819,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.171733+0000","last_change":"2024-03-09T15:36:30.171809+0000","last_active":"2024-03-09T15:36:30.171733+0000","last_peered":"2024-03-09T15:36:30.171733+0000","last_clean":"2024-03-09T15:36:30.171733+0000","last_became_active":"2024-03-09T14:28:38.330571+0000","last_became_peered":"2024-03-09T14:28:38.330571+0000","last_unstale":"2024-03-09T15:36:30.171733+0000","last_undegraded":"2024-03-09T15:36:30.171733+0000","last_fullsized":"2024-03-09T15:36:30.171733+0000","mapping_epoch":26,"log_start":"261'201","ondisk_log_start":"261'201","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'213","last_scrub_stamp":"2024-03-09T15:36:30.171534+0000","last_deep_scrub":"261'213","last_deep_scrub_stamp":"2024-03-09T15:36:30.171534+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.171534+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":201,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:50.639816+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11093,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":349,"num_read_kb":659,"num_write":246,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.213225+0000","last_change":"2024-03-09T15:36:29.213283+0000","last_active":"2024-03-09T15:36:29.213225+0000","last_peered":"2024-03-09T15:36:29.213225+0000","last_clean":"2024-03-09T15:36:29.213225+0000","last_became_active":"2024-03-09T14:28:40.385080+0000","last_became_peered":"2024-03-09T14:28:40.385080+0000","last_unstale":"2024-03-09T15:36:29.213225+0000","last_undegraded":"2024-03-09T15:36:29.213225+0000","last_fullsized":"2024-03-09T15:36:29.213225+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:29.213014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:29.213014+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.213014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:47.618516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"261'69","reported_seq":573,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.140935+0000","last_change":"2024-03-09T15:36:15.140935+0000","last_active":"2024-03-09T15:36:15.140935+0000","last_peered":"2024-03-09T15:36:15.140935+0000","last_clean":"2024-03-09T15:36:15.140935+0000","last_became_active":"2024-03-09T14:28:42.412616+0000","last_became_peered":"2024-03-09T14:28:42.412616+0000","last_unstale":"2024-03-09T15:36:15.140935+0000","last_undegraded":"2024-03-09T15:36:15.140935+0000","last_fullsized":"2024-03-09T15:36:15.140935+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'69","last_scrub_stamp":"2024-03-09T15:36:15.140656+0000","last_deep_scrub":"261'69","last_deep_scrub_stamp":"2024-03-09T15:36:15.140656+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.140656+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:26:05.406444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":49,"num_write":163,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.27","version":"261'50","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.426480+0000","last_change":"2024-03-09T15:36:34.426540+0000","last_active":"2024-03-09T15:36:34.426480+0000","last_peered":"2024-03-09T15:36:34.426480+0000","last_clean":"2024-03-09T15:36:34.426480+0000","last_became_active":"2024-03-09T14:28:34.306685+0000","last_became_peered":"2024-03-09T14:28:34.306685+0000","last_unstale":"2024-03-09T15:36:34.426480+0000","last_undegraded":"2024-03-09T15:36:34.426480+0000","last_fullsized":"2024-03-09T15:36:34.426480+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'50","last_scrub_stamp":"2024-03-09T15:36:34.426288+0000","last_deep_scrub":"261'50","last_deep_scrub_stamp":"2024-03-09T15:36:34.426288+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.426288+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:49:13.124210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":204,"num_read_kb":164,"num_write":70,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"261'146","reported_seq":738,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.283440+0000","last_change":"2024-03-09T15:36:40.283538+0000","last_active":"2024-03-09T15:36:40.283440+0000","last_peered":"2024-03-09T15:36:40.283440+0000","last_clean":"2024-03-09T15:36:40.283440+0000","last_became_active":"2024-03-09T14:28:32.333008+0000","last_became_peered":"2024-03-09T14:28:32.333008+0000","last_unstale":"2024-03-09T15:36:40.283440+0000","last_undegraded":"2024-03-09T15:36:40.283440+0000","last_fullsized":"2024-03-09T15:36:40.283440+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'146","last_scrub_stamp":"2024-03-09T15:36:40.283252+0000","last_deep_scrub":"261'146","last_deep_scrub_stamp":"2024-03-09T15:36:40.283252+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.283252+0000","objects_scrubbed":6,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:58:17.359879+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655838,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":237,"num_read_kb":10390,"num_write":554,"num_write_kb":33266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.26","version":"261'181","reported_seq":781,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.520994+0000","last_change":"2024-03-09T15:36:30.521058+0000","last_active":"2024-03-09T15:36:30.520994+0000","last_peered":"2024-03-09T15:36:30.520994+0000","last_clean":"2024-03-09T15:36:30.520994+0000","last_became_active":"2024-03-09T14:28:34.314007+0000","last_became_peered":"2024-03-09T14:28:34.314007+0000","last_unstale":"2024-03-09T15:36:30.520994+0000","last_undegraded":"2024-03-09T15:36:30.520994+0000","last_fullsized":"2024-03-09T15:36:30.520994+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'181","last_scrub_stamp":"2024-03-09T15:36:30.520807+0000","last_deep_scrub":"261'181","last_deep_scrub_stamp":"2024-03-09T15:36:30.520807+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.520807+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":100,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:13:34.179786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":729,"num_read_kb":586,"num_write":272,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"261'155","reported_seq":713,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.244531+0000","last_change":"2024-03-09T15:36:29.244629+0000","last_active":"2024-03-09T15:36:29.244531+0000","last_peered":"2024-03-09T15:36:29.244531+0000","last_clean":"2024-03-09T15:36:29.244531+0000","last_became_active":"2024-03-09T14:28:32.286609+0000","last_became_peered":"2024-03-09T14:28:32.286609+0000","last_unstale":"2024-03-09T15:36:29.244531+0000","last_undegraded":"2024-03-09T15:36:29.244531+0000","last_fullsized":"2024-03-09T15:36:29.244531+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'155","last_scrub_stamp":"2024-03-09T15:36:29.244332+0000","last_deep_scrub":"261'155","last_deep_scrub_stamp":"2024-03-09T15:36:29.244332+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.244332+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":100,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:53:34.232406+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":810211,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":147,"num_read_kb":1907,"num_write":492,"num_write_kb":32192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"261'94","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.704648+0000","last_change":"2024-03-09T15:36:30.704712+0000","last_active":"2024-03-09T15:36:30.704648+0000","last_peered":"2024-03-09T15:36:30.704648+0000","last_clean":"2024-03-09T15:36:30.704648+0000","last_became_active":"2024-03-09T14:28:34.323541+0000","last_became_peered":"2024-03-09T14:28:34.323541+0000","last_unstale":"2024-03-09T15:36:30.704648+0000","last_undegraded":"2024-03-09T15:36:30.704648+0000","last_fullsized":"2024-03-09T15:36:30.704648+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'94","last_scrub_stamp":"2024-03-09T15:36:30.704449+0000","last_deep_scrub":"261'94","last_deep_scrub_stamp":"2024-03-09T15:36:30.704449+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.704449+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:11.739435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":384,"num_read_kb":306,"num_write":146,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":230,"num_omap_keys":1,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"261'173","reported_seq":778,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.935595+0000","last_change":"2024-03-09T15:36:47.935747+0000","last_active":"2024-03-09T15:36:47.935595+0000","last_peered":"2024-03-09T15:36:47.935595+0000","last_clean":"2024-03-09T15:36:47.935595+0000","last_became_active":"2024-03-09T14:28:32.295699+0000","last_became_peered":"2024-03-09T14:28:32.295699+0000","last_unstale":"2024-03-09T15:36:47.935595+0000","last_undegraded":"2024-03-09T15:36:47.935595+0000","last_fullsized":"2024-03-09T15:36:47.935595+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'173","last_scrub_stamp":"2024-03-09T15:36:47.935356+0000","last_deep_scrub":"261'173","last_deep_scrub_stamp":"2024-03-09T15:36:47.935356+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.935356+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":100,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:15:12.568313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":297,"num_read_kb":14189,"num_write":716,"num_write_kb":33521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"261'101","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.552706+0000","last_change":"2024-03-09T15:36:24.552770+0000","last_active":"2024-03-09T15:36:24.552706+0000","last_peered":"2024-03-09T15:36:24.552706+0000","last_clean":"2024-03-09T15:36:24.552706+0000","last_became_active":"2024-03-09T14:28:34.321647+0000","last_became_peered":"2024-03-09T14:28:34.321647+0000","last_unstale":"2024-03-09T15:36:24.552706+0000","last_undegraded":"2024-03-09T15:36:24.552706+0000","last_fullsized":"2024-03-09T15:36:24.552706+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'101","last_scrub_stamp":"2024-03-09T15:36:24.552512+0000","last_deep_scrub":"261'101","last_deep_scrub_stamp":"2024-03-09T15:36:24.552512+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.552512+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:59:27.278086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":329,"num_write":161,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"261'128","reported_seq":681,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.494943+0000","last_change":"2024-03-09T14:28:32.298327+0000","last_active":"2024-03-09T15:36:06.494943+0000","last_peered":"2024-03-09T15:36:06.494943+0000","last_clean":"2024-03-09T15:36:06.494943+0000","last_became_active":"2024-03-09T14:28:32.297866+0000","last_became_peered":"2024-03-09T14:28:32.297866+0000","last_unstale":"2024-03-09T15:36:06.494943+0000","last_undegraded":"2024-03-09T15:36:06.494943+0000","last_fullsized":"2024-03-09T15:36:06.494943+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":100,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":163,"num_read_kb":3467,"num_write":495,"num_write_kb":23815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"261'52","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.607074+0000","last_change":"2024-03-09T15:36:45.607269+0000","last_active":"2024-03-09T15:36:45.607074+0000","last_peered":"2024-03-09T15:36:45.607074+0000","last_clean":"2024-03-09T15:36:45.607074+0000","last_became_active":"2024-03-09T14:28:34.330511+0000","last_became_peered":"2024-03-09T14:28:34.330511+0000","last_unstale":"2024-03-09T15:36:45.607074+0000","last_undegraded":"2024-03-09T15:36:45.607074+0000","last_fullsized":"2024-03-09T15:36:45.607074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'52","last_scrub_stamp":"2024-03-09T15:36:45.606838+0000","last_deep_scrub":"261'52","last_deep_scrub_stamp":"2024-03-09T15:36:45.606838+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.606838+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:14:12.247135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":225,"num_read_kb":185,"num_write":82,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"261'118","reported_seq":668,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.475009+0000","last_change":"2024-03-09T14:28:32.326489+0000","last_active":"2024-03-09T15:36:06.475009+0000","last_peered":"2024-03-09T15:36:06.475009+0000","last_clean":"2024-03-09T15:36:06.475009+0000","last_became_active":"2024-03-09T14:28:32.326148+0000","last_became_peered":"2024-03-09T14:28:32.326148+0000","last_unstale":"2024-03-09T15:36:06.475009+0000","last_undegraded":"2024-03-09T15:36:06.475009+0000","last_fullsized":"2024-03-09T15:36:06.475009+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":173,"num_read_kb":5614,"num_write":451,"num_write_kb":21169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"261'134","reported_seq":694,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.224786+0000","last_change":"2024-03-09T15:36:39.224849+0000","last_active":"2024-03-09T15:36:39.224786+0000","last_peered":"2024-03-09T15:36:39.224786+0000","last_clean":"2024-03-09T15:36:39.224786+0000","last_became_active":"2024-03-09T14:28:34.322225+0000","last_became_peered":"2024-03-09T14:28:34.322225+0000","last_unstale":"2024-03-09T15:36:39.224786+0000","last_undegraded":"2024-03-09T15:36:39.224786+0000","last_fullsized":"2024-03-09T15:36:39.224786+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'134","last_scrub_stamp":"2024-03-09T15:36:39.224588+0000","last_deep_scrub":"261'134","last_deep_scrub_stamp":"2024-03-09T15:36:39.224588+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.224588+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:04:21.612595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":498,"num_read_kb":399,"num_write":194,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"261'177","reported_seq":752,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.267800+0000","last_change":"2024-03-09T15:36:41.267910+0000","last_active":"2024-03-09T15:36:41.267800+0000","last_peered":"2024-03-09T15:36:41.267800+0000","last_clean":"2024-03-09T15:36:41.267800+0000","last_became_active":"2024-03-09T14:28:32.299839+0000","last_became_peered":"2024-03-09T14:28:32.299839+0000","last_unstale":"2024-03-09T15:36:41.267800+0000","last_undegraded":"2024-03-09T15:36:41.267800+0000","last_fullsized":"2024-03-09T15:36:41.267800+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'177","last_scrub_stamp":"2024-03-09T15:36:41.267607+0000","last_deep_scrub":"261'177","last_deep_scrub_stamp":"2024-03-09T15:36:41.267607+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.267607+0000","objects_scrubbed":6,"log_size":77,"log_dups_size":100,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:54:41.435261+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242922,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":18833,"num_write":606,"num_write_kb":39169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"261'74","reported_seq":617,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.623011+0000","last_change":"2024-03-09T15:36:46.623071+0000","last_active":"2024-03-09T15:36:46.623011+0000","last_peered":"2024-03-09T15:36:46.623011+0000","last_clean":"2024-03-09T15:36:46.623011+0000","last_became_active":"2024-03-09T14:28:34.315302+0000","last_became_peered":"2024-03-09T14:28:34.315302+0000","last_unstale":"2024-03-09T15:36:46.623011+0000","last_undegraded":"2024-03-09T15:36:46.623011+0000","last_fullsized":"2024-03-09T15:36:46.623011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'74","last_scrub_stamp":"2024-03-09T15:36:46.622798+0000","last_deep_scrub":"261'74","last_deep_scrub_stamp":"2024-03-09T15:36:46.622798+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.622798+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:10:27.130072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":224,"num_write":116,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"261'156","reported_seq":719,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.267202+0000","last_change":"2024-03-09T15:36:42.267361+0000","last_active":"2024-03-09T15:36:42.267202+0000","last_peered":"2024-03-09T15:36:42.267202+0000","last_clean":"2024-03-09T15:36:42.267202+0000","last_became_active":"2024-03-09T14:28:32.334117+0000","last_became_peered":"2024-03-09T14:28:32.334117+0000","last_unstale":"2024-03-09T15:36:42.267202+0000","last_undegraded":"2024-03-09T15:36:42.267202+0000","last_fullsized":"2024-03-09T15:36:42.267202+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'156","last_scrub_stamp":"2024-03-09T15:36:42.266976+0000","last_deep_scrub":"261'156","last_deep_scrub_stamp":"2024-03-09T15:36:42.266976+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.266976+0000","objects_scrubbed":7,"log_size":56,"log_dups_size":100,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:04:14.319419+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":182,"num_read_kb":192,"num_write":563,"num_write_kb":14566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"261'48","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.189108+0000","last_change":"2024-03-09T15:36:31.189173+0000","last_active":"2024-03-09T15:36:31.189108+0000","last_peered":"2024-03-09T15:36:31.189108+0000","last_clean":"2024-03-09T15:36:31.189108+0000","last_became_active":"2024-03-09T14:28:34.323202+0000","last_became_peered":"2024-03-09T14:28:34.323202+0000","last_unstale":"2024-03-09T15:36:31.189108+0000","last_undegraded":"2024-03-09T15:36:31.189108+0000","last_fullsized":"2024-03-09T15:36:31.189108+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:31.188911+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:31.188911+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.188911+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:36:34.360701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":219,"num_read_kb":183,"num_write":70,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"261'138","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.519238+0000","last_change":"2024-03-09T14:28:32.300258+0000","last_active":"2024-03-09T15:36:06.519238+0000","last_peered":"2024-03-09T15:36:06.519238+0000","last_clean":"2024-03-09T15:36:06.519238+0000","last_became_active":"2024-03-09T14:28:32.299737+0000","last_became_peered":"2024-03-09T14:28:32.299737+0000","last_unstale":"2024-03-09T15:36:06.519238+0000","last_undegraded":"2024-03-09T15:36:06.519238+0000","last_fullsized":"2024-03-09T15:36:06.519238+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":100,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":168,"num_read_kb":12613,"num_write":521,"num_write_kb":31618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"261'94","reported_seq":654,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.306693+0000","last_change":"2024-03-09T15:36:42.306758+0000","last_active":"2024-03-09T15:36:42.306693+0000","last_peered":"2024-03-09T15:36:42.306693+0000","last_clean":"2024-03-09T15:36:42.306693+0000","last_became_active":"2024-03-09T14:28:34.315451+0000","last_became_peered":"2024-03-09T14:28:34.315451+0000","last_unstale":"2024-03-09T15:36:42.306693+0000","last_undegraded":"2024-03-09T15:36:42.306693+0000","last_fullsized":"2024-03-09T15:36:42.306693+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'94","last_scrub_stamp":"2024-03-09T15:36:42.306481+0000","last_deep_scrub":"261'94","last_deep_scrub_stamp":"2024-03-09T15:36:42.306481+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.306481+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:34:40.259583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":374,"num_read_kb":293,"num_write":145,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"261'150","reported_seq":725,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.844582+0000","last_change":"2024-03-09T15:36:49.844668+0000","last_active":"2024-03-09T15:36:49.844582+0000","last_peered":"2024-03-09T15:36:49.844582+0000","last_clean":"2024-03-09T15:36:49.844582+0000","last_became_active":"2024-03-09T14:28:32.304540+0000","last_became_peered":"2024-03-09T14:28:32.304540+0000","last_unstale":"2024-03-09T15:36:49.844582+0000","last_undegraded":"2024-03-09T15:36:49.844582+0000","last_fullsized":"2024-03-09T15:36:49.844582+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'150","last_scrub_stamp":"2024-03-09T15:36:49.844389+0000","last_deep_scrub":"261'150","last_deep_scrub_stamp":"2024-03-09T15:36:49.844389+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.844389+0000","objects_scrubbed":6,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:30:01.179531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655899,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":200,"num_read_kb":1806,"num_write":509,"num_write_kb":18066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2e","version":"261'40","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.169829+0000","last_change":"2024-03-09T15:36:36.169895+0000","last_active":"2024-03-09T15:36:36.169829+0000","last_peered":"2024-03-09T15:36:36.169829+0000","last_clean":"2024-03-09T15:36:36.169829+0000","last_became_active":"2024-03-09T14:28:34.312399+0000","last_became_peered":"2024-03-09T14:28:34.312399+0000","last_unstale":"2024-03-09T15:36:36.169829+0000","last_undegraded":"2024-03-09T15:36:36.169829+0000","last_fullsized":"2024-03-09T15:36:36.169829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'40","last_scrub_stamp":"2024-03-09T15:36:36.169631+0000","last_deep_scrub":"261'40","last_deep_scrub_stamp":"2024-03-09T15:36:36.169631+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.169631+0000","objects_scrubbed":1,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:31:09.147800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":182,"num_read_kb":156,"num_write":57,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"261'145","reported_seq":689,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.495921+0000","last_change":"2024-03-09T14:28:32.314908+0000","last_active":"2024-03-09T15:36:06.495921+0000","last_peered":"2024-03-09T15:36:06.495921+0000","last_clean":"2024-03-09T15:36:06.495921+0000","last_became_active":"2024-03-09T14:28:32.314596+0000","last_became_peered":"2024-03-09T14:28:32.314596+0000","last_unstale":"2024-03-09T15:36:06.495921+0000","last_undegraded":"2024-03-09T15:36:06.495921+0000","last_fullsized":"2024-03-09T15:36:06.495921+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":100,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263241,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":148,"num_read_kb":12793,"num_write":461,"num_write_kb":35350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"261'64","reported_seq":635,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.186824+0000","last_change":"2024-03-09T15:36:35.186888+0000","last_active":"2024-03-09T15:36:35.186824+0000","last_peered":"2024-03-09T15:36:35.186824+0000","last_clean":"2024-03-09T15:36:35.186824+0000","last_became_active":"2024-03-09T14:28:34.325461+0000","last_became_peered":"2024-03-09T14:28:34.325461+0000","last_unstale":"2024-03-09T15:36:35.186824+0000","last_undegraded":"2024-03-09T15:36:35.186824+0000","last_fullsized":"2024-03-09T15:36:35.186824+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'64","last_scrub_stamp":"2024-03-09T15:36:35.186622+0000","last_deep_scrub":"261'64","last_deep_scrub_stamp":"2024-03-09T15:36:35.186622+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.186622+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:12:15.358732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":264,"num_read_kb":214,"num_write":89,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"261'165","reported_seq":750,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.949321+0000","last_change":"2024-03-09T15:36:43.949499+0000","last_active":"2024-03-09T15:36:43.949321+0000","last_peered":"2024-03-09T15:36:43.949321+0000","last_clean":"2024-03-09T15:36:43.949321+0000","last_became_active":"2024-03-09T14:28:32.311750+0000","last_became_peered":"2024-03-09T14:28:32.311750+0000","last_unstale":"2024-03-09T15:36:43.949321+0000","last_undegraded":"2024-03-09T15:36:43.949321+0000","last_fullsized":"2024-03-09T15:36:43.949321+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:43.949104+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:43.949104+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.949104+0000","objects_scrubbed":6,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:52:33.164001+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":213,"num_read_kb":14896,"num_write":533,"num_write_kb":41774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"261'162","reported_seq":783,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.302576+0000","last_change":"2024-03-09T15:36:42.302635+0000","last_active":"2024-03-09T15:36:42.302576+0000","last_peered":"2024-03-09T15:36:42.302576+0000","last_clean":"2024-03-09T15:36:42.302576+0000","last_became_active":"2024-03-09T14:28:34.312212+0000","last_became_peered":"2024-03-09T14:28:34.312212+0000","last_unstale":"2024-03-09T15:36:42.302576+0000","last_undegraded":"2024-03-09T15:36:42.302576+0000","last_fullsized":"2024-03-09T15:36:42.302576+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'162","last_scrub_stamp":"2024-03-09T15:36:42.302384+0000","last_deep_scrub":"261'162","last_deep_scrub_stamp":"2024-03-09T15:36:42.302384+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.302384+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":100,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:11:17.535660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":664,"num_read_kb":532,"num_write":243,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2b","version":"261'152","reported_seq":725,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.528226+0000","last_change":"2024-03-09T14:28:32.309913+0000","last_active":"2024-03-09T15:36:06.528226+0000","last_peered":"2024-03-09T15:36:06.528226+0000","last_clean":"2024-03-09T15:36:06.528226+0000","last_became_active":"2024-03-09T14:28:32.309562+0000","last_became_peered":"2024-03-09T14:28:32.309562+0000","last_unstale":"2024-03-09T15:36:06.528226+0000","last_undegraded":"2024-03-09T15:36:06.528226+0000","last_fullsized":"2024-03-09T15:36:06.528226+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":100,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":246,"num_read_kb":5348,"num_write":573,"num_write_kb":20887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"261'118","reported_seq":720,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.176542+0000","last_change":"2024-03-09T15:36:42.176738+0000","last_active":"2024-03-09T15:36:42.176542+0000","last_peered":"2024-03-09T15:36:42.176542+0000","last_clean":"2024-03-09T15:36:42.176542+0000","last_became_active":"2024-03-09T14:28:34.321424+0000","last_became_peered":"2024-03-09T14:28:34.321424+0000","last_unstale":"2024-03-09T15:36:42.176542+0000","last_undegraded":"2024-03-09T15:36:42.176542+0000","last_fullsized":"2024-03-09T15:36:42.176542+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'118","last_scrub_stamp":"2024-03-09T15:36:42.176313+0000","last_deep_scrub":"261'118","last_deep_scrub_stamp":"2024-03-09T15:36:42.176313+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.176313+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:02:02.079925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":513,"num_read_kb":409,"num_write":169,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3064,"num_omap_keys":18,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"261'167","reported_seq":744,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.264592+0000","last_change":"2024-03-09T15:36:41.264697+0000","last_active":"2024-03-09T15:36:41.264592+0000","last_peered":"2024-03-09T15:36:41.264592+0000","last_clean":"2024-03-09T15:36:41.264592+0000","last_became_active":"2024-03-09T14:28:32.333838+0000","last_became_peered":"2024-03-09T14:28:32.333838+0000","last_unstale":"2024-03-09T15:36:41.264592+0000","last_undegraded":"2024-03-09T15:36:41.264592+0000","last_fullsized":"2024-03-09T15:36:41.264592+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'167","last_scrub_stamp":"2024-03-09T15:36:41.264378+0000","last_deep_scrub":"261'167","last_deep_scrub_stamp":"2024-03-09T15:36:41.264378+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.264378+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":100,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:23:18.360963+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":238,"num_read_kb":3315,"num_write":673,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2a","version":"261'71","reported_seq":636,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:06.046616+0000","last_change":"2024-03-09T14:28:34.297744+0000","last_active":"2024-03-09T15:36:06.046616+0000","last_peered":"2024-03-09T15:36:06.046616+0000","last_clean":"2024-03-09T15:36:06.046616+0000","last_became_active":"2024-03-09T14:28:34.297042+0000","last_became_peered":"2024-03-09T14:28:34.297042+0000","last_unstale":"2024-03-09T15:36:06.046616+0000","last_undegraded":"2024-03-09T15:36:06.046616+0000","last_fullsized":"2024-03-09T15:36:06.046616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:33.273735+0000","last_clean_scrub_stamp":"2024-03-09T14:28:33.273735+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":242,"num_write":99,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"261'122","reported_seq":696,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.101796+0000","last_change":"2024-03-09T14:28:32.320093+0000","last_active":"2024-03-09T15:36:14.101796+0000","last_peered":"2024-03-09T15:36:14.101796+0000","last_clean":"2024-03-09T15:36:14.101796+0000","last_became_active":"2024-03-09T14:28:32.317797+0000","last_became_peered":"2024-03-09T14:28:32.317797+0000","last_unstale":"2024-03-09T15:36:14.101796+0000","last_undegraded":"2024-03-09T15:36:14.101796+0000","last_fullsized":"2024-03-09T15:36:14.101796+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T14:28:31.262301+0000","last_clean_scrub_stamp":"2024-03-09T14:28:31.262301+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263201,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":187,"num_read_kb":4139,"num_write":475,"num_write_kb":19145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"261'113","reported_seq":677,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.688184+0000","last_change":"2024-03-09T15:36:48.688250+0000","last_active":"2024-03-09T15:36:48.688184+0000","last_peered":"2024-03-09T15:36:48.688184+0000","last_clean":"2024-03-09T15:36:48.688184+0000","last_became_active":"2024-03-09T14:28:34.314073+0000","last_became_peered":"2024-03-09T14:28:34.314073+0000","last_unstale":"2024-03-09T15:36:48.688184+0000","last_undegraded":"2024-03-09T15:36:48.688184+0000","last_fullsized":"2024-03-09T15:36:48.688184+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'113","last_scrub_stamp":"2024-03-09T15:36:48.687973+0000","last_deep_scrub":"261'113","last_deep_scrub_stamp":"2024-03-09T15:36:48.687973+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.687973+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":100,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:23:16.042642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":445,"num_read_kb":351,"num_write":167,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"261'140","reported_seq":726,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.283437+0000","last_change":"2024-03-09T15:36:30.283533+0000","last_active":"2024-03-09T15:36:30.283437+0000","last_peered":"2024-03-09T15:36:30.283437+0000","last_clean":"2024-03-09T15:36:30.283437+0000","last_became_active":"2024-03-09T14:28:32.334187+0000","last_became_peered":"2024-03-09T14:28:32.334187+0000","last_unstale":"2024-03-09T15:36:30.283437+0000","last_undegraded":"2024-03-09T15:36:30.283437+0000","last_fullsized":"2024-03-09T15:36:30.283437+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'140","last_scrub_stamp":"2024-03-09T15:36:30.283226+0000","last_deep_scrub":"261'140","last_deep_scrub_stamp":"2024-03-09T15:36:30.283226+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.283226+0000","objects_scrubbed":6,"log_size":40,"log_dups_size":100,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:10:30.041775+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2181199,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":224,"num_read_kb":1842,"num_write":561,"num_write_kb":7884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"261'4028","reported_seq":4588,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.151978+0000","last_change":"2024-03-09T15:36:44.152037+0000","last_active":"2024-03-09T15:36:44.151978+0000","last_peered":"2024-03-09T15:36:44.151978+0000","last_clean":"2024-03-09T15:36:44.151978+0000","last_became_active":"2024-03-09T14:28:34.322421+0000","last_became_peered":"2024-03-09T14:28:34.322421+0000","last_unstale":"2024-03-09T15:36:44.151978+0000","last_undegraded":"2024-03-09T15:36:44.151978+0000","last_fullsized":"2024-03-09T15:36:44.151978+0000","mapping_epoch":22,"log_start":"261'4000","ondisk_log_start":"261'4000","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'4028","last_scrub_stamp":"2024-03-09T15:36:44.151779+0000","last_deep_scrub":"261'4028","last_deep_scrub_stamp":"2024-03-09T15:36:44.151779+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.151779+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:57:46.039852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14144,"num_read_kb":11461,"num_write":6038,"num_write_kb":3015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"261'136","reported_seq":715,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.873937+0000","last_change":"2024-03-09T15:36:37.874066+0000","last_active":"2024-03-09T15:36:37.873937+0000","last_peered":"2024-03-09T15:36:37.873937+0000","last_clean":"2024-03-09T15:36:37.873937+0000","last_became_active":"2024-03-09T14:28:32.324166+0000","last_became_peered":"2024-03-09T14:28:32.324166+0000","last_unstale":"2024-03-09T15:36:37.873937+0000","last_undegraded":"2024-03-09T15:36:37.873937+0000","last_fullsized":"2024-03-09T15:36:37.873937+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'136","last_scrub_stamp":"2024-03-09T15:36:37.873720+0000","last_deep_scrub":"261'136","last_deep_scrub_stamp":"2024-03-09T15:36:37.873720+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.873720+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:39:18.821060+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269861,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":9428,"num_write":515,"num_write_kb":30713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"261'144","reported_seq":732,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.957343+0000","last_change":"2024-03-09T15:36:46.957426+0000","last_active":"2024-03-09T15:36:46.957343+0000","last_peered":"2024-03-09T15:36:46.957343+0000","last_clean":"2024-03-09T15:36:46.957343+0000","last_became_active":"2024-03-09T14:28:32.287937+0000","last_became_peered":"2024-03-09T14:28:32.287937+0000","last_unstale":"2024-03-09T15:36:46.957343+0000","last_undegraded":"2024-03-09T15:36:46.957343+0000","last_fullsized":"2024-03-09T15:36:46.957343+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'144","last_scrub_stamp":"2024-03-09T15:36:46.957162+0000","last_deep_scrub":"261'144","last_deep_scrub_stamp":"2024-03-09T15:36:46.957162+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.957162+0000","objects_scrubbed":3,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:17:49.579394+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243043,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":214,"num_read_kb":48289,"num_write":459,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"261'161","reported_seq":736,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.317920+0000","last_change":"2024-03-09T15:36:43.318022+0000","last_active":"2024-03-09T15:36:43.317920+0000","last_peered":"2024-03-09T15:36:43.317920+0000","last_clean":"2024-03-09T15:36:43.317920+0000","last_became_active":"2024-03-09T14:28:34.316689+0000","last_became_peered":"2024-03-09T14:28:34.316689+0000","last_unstale":"2024-03-09T15:36:43.317920+0000","last_undegraded":"2024-03-09T15:36:43.317920+0000","last_fullsized":"2024-03-09T15:36:43.317920+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'161","last_scrub_stamp":"2024-03-09T15:36:43.317727+0000","last_deep_scrub":"261'161","last_deep_scrub_stamp":"2024-03-09T15:36:43.317727+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.317727+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:41.055202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":634,"num_read_kb":490,"num_write":268,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"261'163","reported_seq":751,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.213002+0000","last_change":"2024-03-09T15:36:46.213092+0000","last_active":"2024-03-09T15:36:46.213002+0000","last_peered":"2024-03-09T15:36:46.213002+0000","last_clean":"2024-03-09T15:36:46.213002+0000","last_became_active":"2024-03-09T14:28:32.313132+0000","last_became_peered":"2024-03-09T14:28:32.313132+0000","last_unstale":"2024-03-09T15:36:46.213002+0000","last_undegraded":"2024-03-09T15:36:46.213002+0000","last_fullsized":"2024-03-09T15:36:46.213002+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'163","last_scrub_stamp":"2024-03-09T15:36:46.212812+0000","last_deep_scrub":"261'163","last_deep_scrub_stamp":"2024-03-09T15:36:46.212812+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.212812+0000","objects_scrubbed":6,"log_size":63,"log_dups_size":100,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:51:57.319048+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262227,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":255,"num_read_kb":5401,"num_write":635,"num_write_kb":32648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"261'41","reported_seq":596,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.761668+0000","last_change":"2024-03-09T15:36:33.761734+0000","last_active":"2024-03-09T15:36:33.761668+0000","last_peered":"2024-03-09T15:36:33.761668+0000","last_clean":"2024-03-09T15:36:33.761668+0000","last_became_active":"2024-03-09T14:28:34.303701+0000","last_became_peered":"2024-03-09T14:28:34.303701+0000","last_unstale":"2024-03-09T15:36:33.761668+0000","last_undegraded":"2024-03-09T15:36:33.761668+0000","last_fullsized":"2024-03-09T15:36:33.761668+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'41","last_scrub_stamp":"2024-03-09T15:36:33.761432+0000","last_deep_scrub":"261'41","last_deep_scrub_stamp":"2024-03-09T15:36:33.761432+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.761432+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:47:50.469481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":166,"num_read_kb":138,"num_write":64,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":65,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3686,"num_read_kb":2614,"num_write":1236,"num_write_kb":514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6449,"num_omap_keys":13,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":13176,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":8518,"num_read_kb":8294,"num_write":8580,"num_write_kb":2546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2306,"num_omap_keys":14,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":26352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1984,"ondisk_log_size":1984,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":64,"acting":64,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":377960,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":12181,"num_read_kb":21090,"num_write":9023,"num_write_kb":5457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":299712,"num_omap_keys":726,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1813,"ondisk_log_size":1813,"up":64,"acting":64,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1719,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1602,"num_read_kb":1903,"num_write":22,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":38590,"num_read_kb":31236,"num_write":15314,"num_write_kb":7493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":32313,"num_omap_keys":134,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3950,"ondisk_log_size":3950,"up":128,"acting":128,"num_store_stats":8},{"poolid":68,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":376,"num_omap_keys":4,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":135300586,"num_objects":320,"num_object_clones":0,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":14057,"num_read_kb":1121153,"num_write":34295,"num_write_kb":1701153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":271892480,"data_stored":270601172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3026,"ondisk_log_size":3026,"up":128,"acting":128,"num_store_stats":8},{"poolid":67,"num_pg":32,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":140,"num_read_kb":192739,"num_write":156,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":66,"num_pg":32,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":138,"num_read_kb":192649,"num_write":156,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575680,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293320,"kb_used_data":45568,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":94078520,"statfs":{"total":96636764160,"available":96336404480,"internally_reserved":0,"allocated":46661632,"data_stored":45012346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":253688266},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575675,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233960,"kb_used_data":22304,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":94137880,"statfs":{"total":96636764160,"available":96397189120,"internally_reserved":0,"allocated":22839296,"data_stored":21261512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575676,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132532,"kb_used_data":61164,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94239308,"statfs":{"total":96636764160,"available":96501051392,"internally_reserved":0,"allocated":62631936,"data_stored":60939198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575680,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100148,"kb_used_data":55084,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94271692,"statfs":{"total":96636764160,"available":96534212608,"internally_reserved":0,"allocated":56406016,"data_stored":54802077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575675,"num_pgs":118,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252776,"kb_used_data":41120,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":94119064,"statfs":{"total":96636764160,"available":96377921536,"internally_reserved":0,"allocated":42106880,"data_stored":40381748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":216725959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575677,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91428,"kb_used_data":38044,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94280412,"statfs":{"total":96636764160,"available":96543141888,"internally_reserved":0,"allocated":38957056,"data_stored":37358501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575676,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131376,"kb_used_data":41960,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94240464,"statfs":{"total":96636764160,"available":96502235136,"internally_reserved":0,"allocated":42967040,"data_stored":41342836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575678,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244644,"kb_used_data":55196,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":94127196,"statfs":{"total":96636764160,"available":96386248704,"internally_reserved":0,"allocated":56520704,"data_stored":54866964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40763392,"data_stored":40607042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23621632,"data_stored":23479247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30126080,"data_stored":29981692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31137792,"data_stored":30907070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":42364928,"data_stored":42183762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":48488448,"data_stored":48318049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":19247104,"data_stored":19128319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36143104,"data_stored":35995991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":3901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":2727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10485760,"data_stored":10485760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":1048576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15728640,"data_stored":15728640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":1048576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10485760,"data_stored":10485760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T15:36:55.530 INFO:tasks.ceph:pgid 68.1a last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.531 INFO:tasks.ceph:pgid 67.1d last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.531 INFO:tasks.ceph:pgid 67.1c last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.531 INFO:tasks.ceph:pgid 68.18 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.531 INFO:tasks.ceph:pgid 67.1f last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.531 INFO:tasks.ceph:pgid 68.19 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.532 INFO:tasks.ceph:pgid 67.1e last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.532 INFO:tasks.ceph:pgid 68.1e last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.532 INFO:tasks.ceph:pgid 67.19 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.532 INFO:tasks.ceph:pgid 68.1f last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.532 INFO:tasks.ceph:pgid 68.1c last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.533 INFO:tasks.ceph:pgid 66.1a last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.533 INFO:tasks.ceph:pgid 68.1d last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.533 INFO:tasks.ceph:pgid 68.12 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.533 INFO:tasks.ceph:pgid 67.14 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.533 INFO:tasks.ceph:pgid 68.10 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.534 INFO:tasks.ceph:pgid 67.17 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.534 INFO:tasks.ceph:pgid 68.11 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.534 INFO:tasks.ceph:pgid 67.16 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.534 INFO:tasks.ceph:pgid 68.16 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.534 INFO:tasks.ceph:pgid 68.17 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.535 INFO:tasks.ceph:pgid 68.14 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.535 INFO:tasks.ceph:pgid 68.15 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.535 INFO:tasks.ceph:pgid 66.13 last_scrub_stamp 2024-03-09T14:46:17.715683+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=17, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.535 INFO:tasks.ceph:pgid 67.f last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.535 INFO:tasks.ceph:pgid 68.9 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.536 INFO:tasks.ceph:pgid 68.e last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.536 INFO:tasks.ceph:pgid 67.9 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.536 INFO:tasks.ceph:pgid 68.f last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.536 INFO:tasks.ceph:pgid 68.c last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.536 INFO:tasks.ceph:pgid 68.d last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.537 INFO:tasks.ceph:pgid 68.3 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.537 INFO:tasks.ceph:pgid 68.0 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.537 INFO:tasks.ceph:pgid 68.1 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.537 INFO:tasks.ceph:pgid 67.6 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.538 INFO:tasks.ceph:pgid 68.7 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.538 INFO:tasks.ceph:pgid 67.0 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.538 INFO:tasks.ceph:pgid 68.4 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.538 INFO:tasks.ceph:pgid 68.5 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.538 INFO:tasks.ceph:pgid 67.2 last_scrub_stamp 2024-03-09T14:46:25.763492+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=46, tm_sec=25, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.539 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.540 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.540 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.540 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.540 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.541 INFO:tasks.ceph:pgid 4.30 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.541 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.541 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.542 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.542 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.543 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.543 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.544 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.544 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.545 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.545 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.545 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.546 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.546 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.547 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.547 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.547 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-09T14:28:11.179640+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=11, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.547 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.548 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.548 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.548 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.548 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.549 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-09T14:28:18.207340+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.549 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.550 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.550 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.550 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.550 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.550 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.551 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.551 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.551 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.551 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.551 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-03-09T14:28:35.280359+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=35, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.552 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.552 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.552 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-03-09T14:28:37.297771+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=37, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.552 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.553 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.553 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.553 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.553 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.554 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.554 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.554 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2024-03-09T14:28:33.273735+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.554 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-09T14:28:31.262301+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=14, tm_min=28, tm_sec=31, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:36:55.554 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-09T15:37:15.556 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T15:37:15.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:37:15.949 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-09T15:37:15.963 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":2325,"stamp":"2024-03-09T15:37:15.296841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":169838241,"num_objects":629,"num_object_clones":0,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":0,"num_read":79040,"num_read_kb":1571790,"num_write":68959,"num_write_kb":1988151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":349853,"num_omap_keys":920,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12598,"ondisk_log_size":12598,"up":786,"acting":786,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":786,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1480184,"kb_used_data":360440,"kb_used_omap":12,"kb_used_meta":1119667,"kb_avail":753494536,"statfs":{"total":773094113280,"available":771578404864,"internally_reserved":0,"allocated":369090560,"data_stored":355965182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1146539568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1046,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008179"},"pg_stats":[{"pgid":"68.1a","version":"0'0","reported_seq":31,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:58.883130+0000","last_change":"2024-03-09T15:36:58.883184+0000","last_active":"2024-03-09T15:36:58.883130+0000","last_peered":"2024-03-09T15:36:58.883130+0000","last_clean":"2024-03-09T15:36:58.883130+0000","last_became_active":"2024-03-09T15:09:50.084784+0000","last_became_peered":"2024-03-09T15:09:50.084784+0000","last_unstale":"2024-03-09T15:36:58.883130+0000","last_undegraded":"2024-03-09T15:36:58.883130+0000","last_fullsized":"2024-03-09T15:36:58.883130+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:58.882933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:58.882933+0000","last_clean_scrub_stamp":"2024-03-09T15:36:58.882933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:28:18.565529+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.1d","version":"261'8","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:56.901811+0000","last_change":"2024-03-09T15:36:56.901918+0000","last_active":"2024-03-09T15:36:56.901811+0000","last_peered":"2024-03-09T15:36:56.901811+0000","last_clean":"2024-03-09T15:36:56.901811+0000","last_became_active":"2024-03-09T14:46:26.825404+0000","last_became_peered":"2024-03-09T14:46:26.825404+0000","last_unstale":"2024-03-09T15:36:56.901811+0000","last_undegraded":"2024-03-09T15:36:56.901811+0000","last_fullsized":"2024-03-09T15:36:56.901811+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:56.901625+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:56.901625+0000","last_clean_scrub_stamp":"2024-03-09T15:36:56.901625+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:57:06.137533+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20498,"num_write":6,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1c","version":"261'2","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.429218+0000","last_change":"2024-03-09T15:36:17.429218+0000","last_active":"2024-03-09T15:36:17.429218+0000","last_peered":"2024-03-09T15:36:17.429218+0000","last_clean":"2024-03-09T15:36:17.429218+0000","last_became_active":"2024-03-09T14:46:18.768383+0000","last_became_peered":"2024-03-09T14:46:18.768383+0000","last_unstale":"2024-03-09T15:36:17.429218+0000","last_undegraded":"2024-03-09T15:36:17.429218+0000","last_fullsized":"2024-03-09T15:36:17.429218+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:17.428960+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:17.428960+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.428960+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:03:37.237056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.1f","version":"258'45","reported_seq":176,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.184209+0000","last_change":"2024-03-09T15:36:12.184266+0000","last_active":"2024-03-09T15:36:12.184209+0000","last_peered":"2024-03-09T15:36:12.184209+0000","last_clean":"2024-03-09T15:36:12.184209+0000","last_became_active":"2024-03-09T14:39:13.281020+0000","last_became_peered":"2024-03-09T14:39:13.281020+0000","last_unstale":"2024-03-09T15:36:12.184209+0000","last_undegraded":"2024-03-09T15:36:12.184209+0000","last_fullsized":"2024-03-09T15:36:12.184209+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'45","last_scrub_stamp":"2024-03-09T15:36:12.183988+0000","last_deep_scrub":"258'45","last_deep_scrub_stamp":"2024-03-09T15:36:12.183988+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.183988+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:21:51.000354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":134,"num_write":70,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1b","version":"0'0","reported_seq":31,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.284563+0000","last_change":"2024-03-09T15:36:43.284617+0000","last_active":"2024-03-09T15:36:43.284563+0000","last_peered":"2024-03-09T15:36:43.284563+0000","last_clean":"2024-03-09T15:36:43.284563+0000","last_became_active":"2024-03-09T15:09:50.082690+0000","last_became_peered":"2024-03-09T15:09:50.082690+0000","last_unstale":"2024-03-09T15:36:43.284563+0000","last_undegraded":"2024-03-09T15:36:43.284563+0000","last_fullsized":"2024-03-09T15:36:43.284563+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:43.284378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:43.284378+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.284378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:49:16.785671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1c","version":"261'6","reported_seq":67,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:58.594971+0000","last_change":"2024-03-09T15:36:58.595054+0000","last_active":"2024-03-09T15:36:58.594971+0000","last_peered":"2024-03-09T15:36:58.594971+0000","last_clean":"2024-03-09T15:36:58.594971+0000","last_became_active":"2024-03-09T14:46:26.806865+0000","last_became_peered":"2024-03-09T14:46:26.806865+0000","last_unstale":"2024-03-09T15:36:58.594971+0000","last_undegraded":"2024-03-09T15:36:58.594971+0000","last_fullsized":"2024-03-09T15:36:58.594971+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:58.594767+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:58.594767+0000","last_clean_scrub_stamp":"2024-03-09T15:36:58.594767+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:15.479100+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.1d","version":"261'2","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.601517+0000","last_change":"2024-03-09T15:36:17.601517+0000","last_active":"2024-03-09T15:36:17.601517+0000","last_peered":"2024-03-09T15:36:17.601517+0000","last_clean":"2024-03-09T15:36:17.601517+0000","last_became_active":"2024-03-09T14:46:18.779760+0000","last_became_peered":"2024-03-09T14:46:18.779760+0000","last_unstale":"2024-03-09T15:36:17.601517+0000","last_undegraded":"2024-03-09T15:36:17.601517+0000","last_fullsized":"2024-03-09T15:36:17.601517+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:17.601237+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:17.601237+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.601237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:48:08.721381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1e","version":"0'0","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.210922+0000","last_change":"2024-03-09T15:36:22.210983+0000","last_active":"2024-03-09T15:36:22.210922+0000","last_peered":"2024-03-09T15:36:22.210922+0000","last_clean":"2024-03-09T15:36:22.210922+0000","last_became_active":"2024-03-09T14:39:13.259731+0000","last_became_peered":"2024-03-09T14:39:13.259731+0000","last_unstale":"2024-03-09T15:36:22.210922+0000","last_undegraded":"2024-03-09T15:36:22.210922+0000","last_fullsized":"2024-03-09T15:36:22.210922+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.210700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.210700+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.210700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:46:50.001547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.18","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.409171+0000","last_change":"2024-03-09T15:36:51.409229+0000","last_active":"2024-03-09T15:36:51.409171+0000","last_peered":"2024-03-09T15:36:51.409171+0000","last_clean":"2024-03-09T15:36:51.409171+0000","last_became_active":"2024-03-09T15:09:50.080578+0000","last_became_peered":"2024-03-09T15:09:50.080578+0000","last_unstale":"2024-03-09T15:36:51.409171+0000","last_undegraded":"2024-03-09T15:36:51.409171+0000","last_fullsized":"2024-03-09T15:36:51.409171+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:51.408977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:51.408977+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.408977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:39:57.431783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.1f","version":"261'5","reported_seq":65,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:06.047166+0000","last_change":"2024-03-09T15:37:06.047229+0000","last_active":"2024-03-09T15:37:06.047166+0000","last_peered":"2024-03-09T15:37:06.047166+0000","last_clean":"2024-03-09T15:37:06.047166+0000","last_became_active":"2024-03-09T14:46:26.895608+0000","last_became_peered":"2024-03-09T14:46:26.895608+0000","last_unstale":"2024-03-09T15:37:06.047166+0000","last_undegraded":"2024-03-09T15:37:06.047166+0000","last_fullsized":"2024-03-09T15:37:06.047166+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'5","last_scrub_stamp":"2024-03-09T15:37:06.046969+0000","last_deep_scrub":"261'5","last_deep_scrub_stamp":"2024-03-09T15:37:06.046969+0000","last_clean_scrub_stamp":"2024-03-09T15:37:06.046969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:09:38.936416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":12288,"num_write":5,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1e","version":"0'0","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.247327+0000","last_change":"2024-03-09T15:36:39.247436+0000","last_active":"2024-03-09T15:36:39.247327+0000","last_peered":"2024-03-09T15:36:39.247327+0000","last_clean":"2024-03-09T15:36:39.247327+0000","last_became_active":"2024-03-09T14:46:18.781856+0000","last_became_peered":"2024-03-09T14:46:18.781856+0000","last_unstale":"2024-03-09T15:36:39.247327+0000","last_undegraded":"2024-03-09T15:36:39.247327+0000","last_fullsized":"2024-03-09T15:36:39.247327+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:39.247089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:39.247089+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.247089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:10:03.506121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.1d","version":"258'35","reported_seq":152,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.273912+0000","last_change":"2024-03-09T15:36:17.273912+0000","last_active":"2024-03-09T15:36:17.273912+0000","last_peered":"2024-03-09T15:36:17.273912+0000","last_clean":"2024-03-09T15:36:17.273912+0000","last_became_active":"2024-03-09T14:39:13.258351+0000","last_became_peered":"2024-03-09T14:39:13.258351+0000","last_unstale":"2024-03-09T15:36:17.273912+0000","last_undegraded":"2024-03-09T15:36:17.273912+0000","last_fullsized":"2024-03-09T15:36:17.273912+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'35","last_scrub_stamp":"2024-03-09T15:36:17.273654+0000","last_deep_scrub":"258'35","last_deep_scrub_stamp":"2024-03-09T15:36:17.273654+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.273654+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:22:59.569323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":152,"num_read_kb":109,"num_write":57,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.19","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.187003+0000","last_change":"2024-03-09T15:09:50.083658+0000","last_active":"2024-03-09T15:36:12.187003+0000","last_peered":"2024-03-09T15:36:12.187003+0000","last_clean":"2024-03-09T15:36:12.187003+0000","last_became_active":"2024-03-09T15:09:50.081680+0000","last_became_peered":"2024-03-09T15:09:50.081680+0000","last_unstale":"2024-03-09T15:36:12.187003+0000","last_undegraded":"2024-03-09T15:36:12.187003+0000","last_fullsized":"2024-03-09T15:36:12.187003+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.1e","version":"261'6","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:07.013711+0000","last_change":"2024-03-09T15:37:07.013901+0000","last_active":"2024-03-09T15:37:07.013711+0000","last_peered":"2024-03-09T15:37:07.013711+0000","last_clean":"2024-03-09T15:37:07.013711+0000","last_became_active":"2024-03-09T14:47:07.798247+0000","last_became_peered":"2024-03-09T14:47:07.798247+0000","last_unstale":"2024-03-09T15:37:07.013711+0000","last_undegraded":"2024-03-09T15:37:07.013711+0000","last_fullsized":"2024-03-09T15:37:07.013711+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:37:07.013452+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:37:07.013452+0000","last_clean_scrub_stamp":"2024-03-09T15:37:07.013452+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:56:28.347945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":16384,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1f","version":"258'5","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.180362+0000","last_change":"2024-03-09T15:36:25.180427+0000","last_active":"2024-03-09T15:36:25.180362+0000","last_peered":"2024-03-09T15:36:25.180362+0000","last_clean":"2024-03-09T15:36:25.180362+0000","last_became_active":"2024-03-09T14:46:18.771731+0000","last_became_peered":"2024-03-09T14:46:18.771731+0000","last_unstale":"2024-03-09T15:36:25.180362+0000","last_undegraded":"2024-03-09T15:36:25.180362+0000","last_fullsized":"2024-03-09T15:36:25.180362+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'5","last_scrub_stamp":"2024-03-09T15:36:25.180164+0000","last_deep_scrub":"258'5","last_deep_scrub_stamp":"2024-03-09T15:36:25.180164+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.180164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:55:32.192979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5136,"num_write":5,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1c","version":"261'13","reported_seq":99,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.507564+0000","last_change":"2024-03-09T15:36:15.507564+0000","last_active":"2024-03-09T15:36:15.507564+0000","last_peered":"2024-03-09T15:36:15.507564+0000","last_clean":"2024-03-09T15:36:15.507564+0000","last_became_active":"2024-03-09T14:39:13.260945+0000","last_became_peered":"2024-03-09T14:39:13.260945+0000","last_unstale":"2024-03-09T15:36:15.507564+0000","last_undegraded":"2024-03-09T15:36:15.507564+0000","last_fullsized":"2024-03-09T15:36:15.507564+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'13","last_scrub_stamp":"2024-03-09T15:36:15.507291+0000","last_deep_scrub":"261'13","last_deep_scrub_stamp":"2024-03-09T15:36:15.507291+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.507291+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:29:48.463973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":27,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1e","version":"0'0","reported_seq":26,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:35:55.834070+0000","last_change":"2024-03-09T15:09:50.088230+0000","last_active":"2024-03-09T15:35:55.834070+0000","last_peered":"2024-03-09T15:35:55.834070+0000","last_clean":"2024-03-09T15:35:55.834070+0000","last_became_active":"2024-03-09T15:09:50.086789+0000","last_became_peered":"2024-03-09T15:09:50.086789+0000","last_unstale":"2024-03-09T15:35:55.834070+0000","last_undegraded":"2024-03-09T15:35:55.834070+0000","last_fullsized":"2024-03-09T15:35:55.834070+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.19","version":"261'6","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:55.893770+0000","last_change":"2024-03-09T15:36:55.893852+0000","last_active":"2024-03-09T15:36:55.893770+0000","last_peered":"2024-03-09T15:36:55.893770+0000","last_clean":"2024-03-09T15:36:55.893770+0000","last_became_active":"2024-03-09T14:46:26.819085+0000","last_became_peered":"2024-03-09T14:46:26.819085+0000","last_unstale":"2024-03-09T15:36:55.893770+0000","last_undegraded":"2024-03-09T15:36:55.893770+0000","last_fullsized":"2024-03-09T15:36:55.893770+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:55.893565+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:55.893565+0000","last_clean_scrub_stamp":"2024-03-09T15:36:55.893565+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:27:43.536475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":12288,"num_write":11,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.18","version":"261'14","reported_seq":90,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.270917+0000","last_change":"2024-03-09T15:36:27.270917+0000","last_active":"2024-03-09T15:36:27.270917+0000","last_peered":"2024-03-09T15:36:27.270917+0000","last_clean":"2024-03-09T15:36:27.270917+0000","last_became_active":"2024-03-09T14:46:18.776161+0000","last_became_peered":"2024-03-09T14:46:18.776161+0000","last_unstale":"2024-03-09T15:36:27.270917+0000","last_undegraded":"2024-03-09T15:36:27.270917+0000","last_fullsized":"2024-03-09T15:36:27.270917+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'14","last_scrub_stamp":"2024-03-09T15:36:27.270455+0000","last_deep_scrub":"261'14","last_deep_scrub_stamp":"2024-03-09T15:36:27.270455+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.270455+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:23:13.389149+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":36889,"num_write":17,"num_write_kb":16407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.1b","version":"258'10","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.553151+0000","last_change":"2024-03-09T15:36:14.553151+0000","last_active":"2024-03-09T15:36:14.553151+0000","last_peered":"2024-03-09T15:36:14.553151+0000","last_clean":"2024-03-09T15:36:14.553151+0000","last_became_active":"2024-03-09T14:39:13.268179+0000","last_became_peered":"2024-03-09T14:39:13.268179+0000","last_unstale":"2024-03-09T15:36:14.553151+0000","last_undegraded":"2024-03-09T15:36:14.553151+0000","last_fullsized":"2024-03-09T15:36:14.553151+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:14.552885+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:14.552885+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.552885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:39.435007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1f","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:52.182480+0000","last_change":"2024-03-09T15:36:52.182480+0000","last_active":"2024-03-09T15:36:52.182480+0000","last_peered":"2024-03-09T15:36:52.182480+0000","last_clean":"2024-03-09T15:36:52.182480+0000","last_became_active":"2024-03-09T15:09:50.063705+0000","last_became_peered":"2024-03-09T15:09:50.063705+0000","last_unstale":"2024-03-09T15:36:52.182480+0000","last_undegraded":"2024-03-09T15:36:52.182480+0000","last_fullsized":"2024-03-09T15:36:52.182480+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:52.182104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:52.182104+0000","last_clean_scrub_stamp":"2024-03-09T15:36:52.182104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:42:53.890576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.18","version":"261'3","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.314926+0000","last_change":"2024-03-09T15:36:48.314986+0000","last_active":"2024-03-09T15:36:48.314926+0000","last_peered":"2024-03-09T15:36:48.314926+0000","last_clean":"2024-03-09T15:36:48.314926+0000","last_became_active":"2024-03-09T14:46:26.824522+0000","last_became_peered":"2024-03-09T14:46:26.824522+0000","last_unstale":"2024-03-09T15:36:48.314926+0000","last_undegraded":"2024-03-09T15:36:48.314926+0000","last_fullsized":"2024-03-09T15:36:48.314926+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:48.314729+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:48.314729+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.314729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:53:25.103105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.19","version":"261'9","reported_seq":75,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.166246+0000","last_change":"2024-03-09T15:36:17.166321+0000","last_active":"2024-03-09T15:36:17.166246+0000","last_peered":"2024-03-09T15:36:17.166246+0000","last_clean":"2024-03-09T15:36:17.166246+0000","last_became_active":"2024-03-09T14:46:18.769730+0000","last_became_peered":"2024-03-09T14:46:18.769730+0000","last_unstale":"2024-03-09T15:36:17.166246+0000","last_undegraded":"2024-03-09T15:36:17.166246+0000","last_fullsized":"2024-03-09T15:36:17.166246+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-03-09T15:36:17.165994+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-03-09T15:36:17.165994+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.165994+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:56:10.015397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":19456,"num_write":19,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1a","version":"261'44","reported_seq":174,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.008442+0000","last_change":"2024-03-09T15:36:14.008442+0000","last_active":"2024-03-09T15:36:14.008442+0000","last_peered":"2024-03-09T15:36:14.008442+0000","last_clean":"2024-03-09T15:36:14.008442+0000","last_became_active":"2024-03-09T14:39:13.273298+0000","last_became_peered":"2024-03-09T14:39:13.273298+0000","last_unstale":"2024-03-09T15:36:14.008442+0000","last_undegraded":"2024-03-09T15:36:14.008442+0000","last_fullsized":"2024-03-09T15:36:14.008442+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:14.008154+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:14.008154+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.008154+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:19:56.477799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":138,"num_write":72,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.1c","version":"261'2","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:08.153921+0000","last_change":"2024-03-09T15:37:08.153982+0000","last_active":"2024-03-09T15:37:08.153921+0000","last_peered":"2024-03-09T15:37:08.153921+0000","last_clean":"2024-03-09T15:37:08.153921+0000","last_became_active":"2024-03-09T15:09:50.074952+0000","last_became_peered":"2024-03-09T15:09:50.074952+0000","last_unstale":"2024-03-09T15:37:08.153921+0000","last_undegraded":"2024-03-09T15:37:08.153921+0000","last_fullsized":"2024-03-09T15:37:08.153921+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:37:08.153733+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:37:08.153733+0000","last_clean_scrub_stamp":"2024-03-09T15:37:08.153733+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:29:59.718406+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1b","version":"261'3","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.334633+0000","last_change":"2024-03-09T15:36:46.334695+0000","last_active":"2024-03-09T15:36:46.334633+0000","last_peered":"2024-03-09T15:36:46.334633+0000","last_clean":"2024-03-09T15:36:46.334633+0000","last_became_active":"2024-03-09T14:46:26.804622+0000","last_became_peered":"2024-03-09T14:46:26.804622+0000","last_unstale":"2024-03-09T15:36:46.334633+0000","last_undegraded":"2024-03-09T15:36:46.334633+0000","last_fullsized":"2024-03-09T15:36:46.334633+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:46.334433+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:46.334433+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.334433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:30:10.876583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.1a","version":"261'5","reported_seq":65,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:56.540292+0000","last_change":"2024-03-09T15:36:56.540355+0000","last_active":"2024-03-09T15:36:56.540292+0000","last_peered":"2024-03-09T15:36:56.540292+0000","last_clean":"2024-03-09T15:36:56.540292+0000","last_became_active":"2024-03-09T14:46:18.784821+0000","last_became_peered":"2024-03-09T14:46:18.784821+0000","last_unstale":"2024-03-09T15:36:56.540292+0000","last_undegraded":"2024-03-09T15:36:56.540292+0000","last_fullsized":"2024-03-09T15:36:56.540292+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'5","last_scrub_stamp":"2024-03-09T15:36:56.540098+0000","last_deep_scrub":"261'5","last_deep_scrub_stamp":"2024-03-09T15:36:56.540098+0000","last_clean_scrub_stamp":"2024-03-09T15:36:56.540098+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:38:53.526961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4096,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.19","version":"258'28","reported_seq":129,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.263878+0000","last_change":"2024-03-09T15:36:18.263945+0000","last_active":"2024-03-09T15:36:18.263878+0000","last_peered":"2024-03-09T15:36:18.263878+0000","last_clean":"2024-03-09T15:36:18.263878+0000","last_became_active":"2024-03-09T14:39:13.257473+0000","last_became_peered":"2024-03-09T14:39:13.257473+0000","last_unstale":"2024-03-09T15:36:18.263878+0000","last_undegraded":"2024-03-09T15:36:18.263878+0000","last_fullsized":"2024-03-09T15:36:18.263878+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'28","last_scrub_stamp":"2024-03-09T15:36:18.263646+0000","last_deep_scrub":"258'28","last_deep_scrub_stamp":"2024-03-09T15:36:18.263646+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.263646+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:59:14.805307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":78,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1d","version":"0'0","reported_seq":31,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:07.155568+0000","last_change":"2024-03-09T15:37:07.155622+0000","last_active":"2024-03-09T15:37:07.155568+0000","last_peered":"2024-03-09T15:37:07.155568+0000","last_clean":"2024-03-09T15:37:07.155568+0000","last_became_active":"2024-03-09T15:09:50.074875+0000","last_became_peered":"2024-03-09T15:09:50.074875+0000","last_unstale":"2024-03-09T15:37:07.155568+0000","last_undegraded":"2024-03-09T15:37:07.155568+0000","last_fullsized":"2024-03-09T15:37:07.155568+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:07.155375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:07.155375+0000","last_clean_scrub_stamp":"2024-03-09T15:37:07.155375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:01:55.131735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1a","version":"261'11","reported_seq":72,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.456175+0000","last_change":"2024-03-09T15:36:21.456175+0000","last_active":"2024-03-09T15:36:21.456175+0000","last_peered":"2024-03-09T15:36:21.456175+0000","last_clean":"2024-03-09T15:36:21.456175+0000","last_became_active":"2024-03-09T14:46:26.812889+0000","last_became_peered":"2024-03-09T14:46:26.812889+0000","last_unstale":"2024-03-09T15:36:21.456175+0000","last_undegraded":"2024-03-09T15:36:21.456175+0000","last_fullsized":"2024-03-09T15:36:21.456175+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'11","last_scrub_stamp":"2024-03-09T15:36:21.455916+0000","last_deep_scrub":"261'11","last_deep_scrub_stamp":"2024-03-09T15:36:21.455916+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.455916+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:07:00.143287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":12290,"num_write":14,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.1b","version":"261'2","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.253249+0000","last_change":"2024-03-09T15:36:19.253249+0000","last_active":"2024-03-09T15:36:19.253249+0000","last_peered":"2024-03-09T15:36:19.253249+0000","last_clean":"2024-03-09T15:36:19.253249+0000","last_became_active":"2024-03-09T14:46:18.756394+0000","last_became_peered":"2024-03-09T14:46:18.756394+0000","last_unstale":"2024-03-09T15:36:19.253249+0000","last_undegraded":"2024-03-09T15:36:19.253249+0000","last_fullsized":"2024-03-09T15:36:19.253249+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:19.252989+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:19.252989+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.252989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:28:28.982640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":40,"num_write":2,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.18","version":"261'8","reported_seq":86,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.579241+0000","last_change":"2024-03-09T15:36:15.579306+0000","last_active":"2024-03-09T15:36:15.579241+0000","last_peered":"2024-03-09T15:36:15.579241+0000","last_clean":"2024-03-09T15:36:15.579241+0000","last_became_active":"2024-03-09T14:39:13.257996+0000","last_became_peered":"2024-03-09T14:39:13.257996+0000","last_unstale":"2024-03-09T15:36:15.579241+0000","last_undegraded":"2024-03-09T15:36:15.579241+0000","last_fullsized":"2024-03-09T15:36:15.579241+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:15.579034+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:15.579034+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.579034+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:42:49.650613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.12","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:54.156296+0000","last_change":"2024-03-09T15:36:54.156410+0000","last_active":"2024-03-09T15:36:54.156296+0000","last_peered":"2024-03-09T15:36:54.156296+0000","last_clean":"2024-03-09T15:36:54.156296+0000","last_became_active":"2024-03-09T15:09:50.070345+0000","last_became_peered":"2024-03-09T15:09:50.070345+0000","last_unstale":"2024-03-09T15:36:54.156296+0000","last_undegraded":"2024-03-09T15:36:54.156296+0000","last_fullsized":"2024-03-09T15:36:54.156296+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:54.156066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:54.156066+0000","last_clean_scrub_stamp":"2024-03-09T15:36:54.156066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:09:57.089544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.15","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.552761+0000","last_change":"2024-03-09T15:36:41.552821+0000","last_active":"2024-03-09T15:36:41.552761+0000","last_peered":"2024-03-09T15:36:41.552761+0000","last_clean":"2024-03-09T15:36:41.552761+0000","last_became_active":"2024-03-09T14:46:26.890992+0000","last_became_peered":"2024-03-09T14:46:26.890992+0000","last_unstale":"2024-03-09T15:36:41.552761+0000","last_undegraded":"2024-03-09T15:36:41.552761+0000","last_fullsized":"2024-03-09T15:36:41.552761+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:41.552560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:41.552560+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.552560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:48:25.323567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.14","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.701752+0000","last_change":"2024-03-09T15:36:22.701809+0000","last_active":"2024-03-09T15:36:22.701752+0000","last_peered":"2024-03-09T15:36:22.701752+0000","last_clean":"2024-03-09T15:36:22.701752+0000","last_became_active":"2024-03-09T14:46:18.766777+0000","last_became_peered":"2024-03-09T14:46:18.766777+0000","last_unstale":"2024-03-09T15:36:22.701752+0000","last_undegraded":"2024-03-09T15:36:22.701752+0000","last_fullsized":"2024-03-09T15:36:22.701752+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.701528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.701528+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.701528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:03:22.022639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.17","version":"258'19","reported_seq":115,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.216612+0000","last_change":"2024-03-09T15:36:13.216647+0000","last_active":"2024-03-09T15:36:13.216612+0000","last_peered":"2024-03-09T15:36:13.216612+0000","last_clean":"2024-03-09T15:36:13.216612+0000","last_became_active":"2024-03-09T14:39:13.279919+0000","last_became_peered":"2024-03-09T14:39:13.279919+0000","last_unstale":"2024-03-09T15:36:13.216612+0000","last_undegraded":"2024-03-09T15:36:13.216612+0000","last_fullsized":"2024-03-09T15:36:13.216612+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'19","last_scrub_stamp":"2024-03-09T15:36:13.216485+0000","last_deep_scrub":"258'19","last_deep_scrub_stamp":"2024-03-09T15:36:13.216485+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.216485+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:22:26.193181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":64,"num_write":32,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.13","version":"261'2","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.296653+0000","last_change":"2024-03-09T15:36:41.296724+0000","last_active":"2024-03-09T15:36:41.296653+0000","last_peered":"2024-03-09T15:36:41.296653+0000","last_clean":"2024-03-09T15:36:41.296653+0000","last_became_active":"2024-03-09T15:09:50.079604+0000","last_became_peered":"2024-03-09T15:09:50.079604+0000","last_unstale":"2024-03-09T15:36:41.296653+0000","last_undegraded":"2024-03-09T15:36:41.296653+0000","last_fullsized":"2024-03-09T15:36:41.296653+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:41.296454+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:41.296454+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.296454+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:36:03.858346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.14","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:09.702163+0000","last_change":"2024-03-09T15:37:09.702224+0000","last_active":"2024-03-09T15:37:09.702163+0000","last_peered":"2024-03-09T15:37:09.702163+0000","last_clean":"2024-03-09T15:37:09.702163+0000","last_became_active":"2024-03-09T14:46:26.894088+0000","last_became_peered":"2024-03-09T14:46:26.894088+0000","last_unstale":"2024-03-09T15:37:09.702163+0000","last_undegraded":"2024-03-09T15:37:09.702163+0000","last_fullsized":"2024-03-09T15:37:09.702163+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:09.701920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:09.701920+0000","last_clean_scrub_stamp":"2024-03-09T15:37:09.701920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:49:58.460178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.15","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.232245+0000","last_change":"2024-03-09T15:36:16.232245+0000","last_active":"2024-03-09T15:36:16.232245+0000","last_peered":"2024-03-09T15:36:16.232245+0000","last_clean":"2024-03-09T15:36:16.232245+0000","last_became_active":"2024-03-09T14:46:18.752413+0000","last_became_peered":"2024-03-09T14:46:18.752413+0000","last_unstale":"2024-03-09T15:36:16.232245+0000","last_undegraded":"2024-03-09T15:36:16.232245+0000","last_fullsized":"2024-03-09T15:36:16.232245+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:16.232005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:16.232005+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.232005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:59:24.622129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.16","version":"261'52","reported_seq":193,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.459135+0000","last_change":"2024-03-09T15:36:16.459199+0000","last_active":"2024-03-09T15:36:16.459135+0000","last_peered":"2024-03-09T15:36:16.459135+0000","last_clean":"2024-03-09T15:36:16.459135+0000","last_became_active":"2024-03-09T14:39:13.262413+0000","last_became_peered":"2024-03-09T14:39:13.262413+0000","last_unstale":"2024-03-09T15:36:16.459135+0000","last_undegraded":"2024-03-09T15:36:16.459135+0000","last_fullsized":"2024-03-09T15:36:16.459135+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'52","last_scrub_stamp":"2024-03-09T15:36:16.458932+0000","last_deep_scrub":"261'52","last_deep_scrub_stamp":"2024-03-09T15:36:16.458932+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.458932+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:28:33.068618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":163,"num_write":78,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.10","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:06.105683+0000","last_change":"2024-03-09T15:37:06.105847+0000","last_active":"2024-03-09T15:37:06.105683+0000","last_peered":"2024-03-09T15:37:06.105683+0000","last_clean":"2024-03-09T15:37:06.105683+0000","last_became_active":"2024-03-09T15:09:50.074274+0000","last_became_peered":"2024-03-09T15:09:50.074274+0000","last_unstale":"2024-03-09T15:37:06.105683+0000","last_undegraded":"2024-03-09T15:37:06.105683+0000","last_fullsized":"2024-03-09T15:37:06.105683+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:06.105130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:06.105130+0000","last_clean_scrub_stamp":"2024-03-09T15:37:06.105130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:58:01.351666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.17","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:08.730823+0000","last_change":"2024-03-09T15:37:08.730877+0000","last_active":"2024-03-09T15:37:08.730823+0000","last_peered":"2024-03-09T15:37:08.730823+0000","last_clean":"2024-03-09T15:37:08.730823+0000","last_became_active":"2024-03-09T14:46:26.893514+0000","last_became_peered":"2024-03-09T14:46:26.893514+0000","last_unstale":"2024-03-09T15:37:08.730823+0000","last_undegraded":"2024-03-09T15:37:08.730823+0000","last_fullsized":"2024-03-09T15:37:08.730823+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:08.730628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:08.730628+0000","last_clean_scrub_stamp":"2024-03-09T15:37:08.730628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:26:35.643642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.16","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.232040+0000","last_change":"2024-03-09T15:36:38.232040+0000","last_active":"2024-03-09T15:36:38.232040+0000","last_peered":"2024-03-09T15:36:38.232040+0000","last_clean":"2024-03-09T15:36:38.232040+0000","last_became_active":"2024-03-09T14:46:18.769274+0000","last_became_peered":"2024-03-09T14:46:18.769274+0000","last_unstale":"2024-03-09T15:36:38.232040+0000","last_undegraded":"2024-03-09T15:36:38.232040+0000","last_fullsized":"2024-03-09T15:36:38.232040+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:38.231785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:38.231785+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.231785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:50:40.645273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.15","version":"0'0","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.220144+0000","last_change":"2024-03-09T15:36:24.220202+0000","last_active":"2024-03-09T15:36:24.220144+0000","last_peered":"2024-03-09T15:36:24.220144+0000","last_clean":"2024-03-09T15:36:24.220144+0000","last_became_active":"2024-03-09T14:39:13.262066+0000","last_became_peered":"2024-03-09T14:39:13.262066+0000","last_unstale":"2024-03-09T15:36:24.220144+0000","last_undegraded":"2024-03-09T15:36:24.220144+0000","last_fullsized":"2024-03-09T15:36:24.220144+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:24.219947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:24.219947+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.219947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:56:44.705423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.11","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:59.918041+0000","last_change":"2024-03-09T15:36:59.918095+0000","last_active":"2024-03-09T15:36:59.918041+0000","last_peered":"2024-03-09T15:36:59.918041+0000","last_clean":"2024-03-09T15:36:59.918041+0000","last_became_active":"2024-03-09T15:09:50.085346+0000","last_became_peered":"2024-03-09T15:09:50.085346+0000","last_unstale":"2024-03-09T15:36:59.918041+0000","last_undegraded":"2024-03-09T15:36:59.918041+0000","last_fullsized":"2024-03-09T15:36:59.918041+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:59.917834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:59.917834+0000","last_clean_scrub_stamp":"2024-03-09T15:36:59.917834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:16:55.219020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.16","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:01.614261+0000","last_change":"2024-03-09T15:37:01.614261+0000","last_active":"2024-03-09T15:37:01.614261+0000","last_peered":"2024-03-09T15:37:01.614261+0000","last_clean":"2024-03-09T15:37:01.614261+0000","last_became_active":"2024-03-09T14:46:26.817864+0000","last_became_peered":"2024-03-09T14:46:26.817864+0000","last_unstale":"2024-03-09T15:37:01.614261+0000","last_undegraded":"2024-03-09T15:37:01.614261+0000","last_fullsized":"2024-03-09T15:37:01.614261+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:01.614008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:01.614008+0000","last_clean_scrub_stamp":"2024-03-09T15:37:01.614008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:43:47.987698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.17","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.208514+0000","last_change":"2024-03-09T15:36:31.208514+0000","last_active":"2024-03-09T15:36:31.208514+0000","last_peered":"2024-03-09T15:36:31.208514+0000","last_clean":"2024-03-09T15:36:31.208514+0000","last_became_active":"2024-03-09T14:46:18.776430+0000","last_became_peered":"2024-03-09T14:46:18.776430+0000","last_unstale":"2024-03-09T15:36:31.208514+0000","last_undegraded":"2024-03-09T15:36:31.208514+0000","last_fullsized":"2024-03-09T15:36:31.208514+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:31.208147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:31.208147+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.208147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:52:30.734801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.14","version":"261'68","reported_seq":280,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.178145+0000","last_change":"2024-03-09T15:36:24.178221+0000","last_active":"2024-03-09T15:36:24.178145+0000","last_peered":"2024-03-09T15:36:24.178145+0000","last_clean":"2024-03-09T15:36:24.178145+0000","last_became_active":"2024-03-09T14:39:13.253007+0000","last_became_peered":"2024-03-09T14:39:13.253007+0000","last_unstale":"2024-03-09T15:36:24.178145+0000","last_undegraded":"2024-03-09T15:36:24.178145+0000","last_fullsized":"2024-03-09T15:36:24.178145+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'68","last_scrub_stamp":"2024-03-09T15:36:24.177894+0000","last_deep_scrub":"261'68","last_deep_scrub_stamp":"2024-03-09T15:36:24.177894+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.177894+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:09:13.568231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":411,"num_read_kb":279,"num_write":87,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4473,"num_omap_keys":9,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.16","version":"0'0","reported_seq":24,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.187173+0000","last_change":"2024-03-09T15:09:50.087752+0000","last_active":"2024-03-09T15:36:12.187173+0000","last_peered":"2024-03-09T15:36:12.187173+0000","last_clean":"2024-03-09T15:36:12.187173+0000","last_became_active":"2024-03-09T15:09:50.086173+0000","last_became_peered":"2024-03-09T15:09:50.086173+0000","last_unstale":"2024-03-09T15:36:12.187173+0000","last_undegraded":"2024-03-09T15:36:12.187173+0000","last_fullsized":"2024-03-09T15:36:12.187173+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.11","version":"261'8","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.163105+0000","last_change":"2024-03-09T15:36:50.163269+0000","last_active":"2024-03-09T15:36:50.163105+0000","last_peered":"2024-03-09T15:36:50.163105+0000","last_clean":"2024-03-09T15:36:50.163105+0000","last_became_active":"2024-03-09T14:46:26.817014+0000","last_became_peered":"2024-03-09T14:46:26.817014+0000","last_unstale":"2024-03-09T15:36:50.163105+0000","last_undegraded":"2024-03-09T15:36:50.163105+0000","last_fullsized":"2024-03-09T15:36:50.163105+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:50.162856+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:50.162856+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.162856+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:04:50.027298+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":23554,"num_write":11,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.10","version":"261'3","reported_seq":63,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.121627+0000","last_change":"2024-03-09T15:36:21.121690+0000","last_active":"2024-03-09T15:36:21.121627+0000","last_peered":"2024-03-09T15:36:21.121627+0000","last_clean":"2024-03-09T15:36:21.121627+0000","last_became_active":"2024-03-09T14:46:18.770936+0000","last_became_peered":"2024-03-09T14:46:18.770936+0000","last_unstale":"2024-03-09T15:36:21.121627+0000","last_undegraded":"2024-03-09T15:36:21.121627+0000","last_fullsized":"2024-03-09T15:36:21.121627+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:21.121408+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:21.121408+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.121408+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:44:45.507890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.13","version":"258'16","reported_seq":104,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.204733+0000","last_change":"2024-03-09T15:36:22.204801+0000","last_active":"2024-03-09T15:36:22.204733+0000","last_peered":"2024-03-09T15:36:22.204733+0000","last_clean":"2024-03-09T15:36:22.204733+0000","last_became_active":"2024-03-09T14:39:13.266228+0000","last_became_peered":"2024-03-09T14:39:13.266228+0000","last_unstale":"2024-03-09T15:36:22.204733+0000","last_undegraded":"2024-03-09T15:36:22.204733+0000","last_fullsized":"2024-03-09T15:36:22.204733+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'16","last_scrub_stamp":"2024-03-09T15:36:22.204502+0000","last_deep_scrub":"258'16","last_deep_scrub_stamp":"2024-03-09T15:36:22.204502+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.204502+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:33:31.523340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":45,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.17","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:11.014947+0000","last_change":"2024-03-09T15:37:11.015000+0000","last_active":"2024-03-09T15:37:11.014947+0000","last_peered":"2024-03-09T15:37:11.014947+0000","last_clean":"2024-03-09T15:37:11.014947+0000","last_became_active":"2024-03-09T15:09:50.075187+0000","last_became_peered":"2024-03-09T15:09:50.075187+0000","last_unstale":"2024-03-09T15:37:11.014947+0000","last_undegraded":"2024-03-09T15:37:11.014947+0000","last_fullsized":"2024-03-09T15:37:11.014947+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:11.014747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:11.014747+0000","last_clean_scrub_stamp":"2024-03-09T15:37:11.014747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:23:32.716146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.10","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.190342+0000","last_change":"2024-03-09T15:36:48.190396+0000","last_active":"2024-03-09T15:36:48.190342+0000","last_peered":"2024-03-09T15:36:48.190342+0000","last_clean":"2024-03-09T15:36:48.190342+0000","last_became_active":"2024-03-09T14:46:26.802476+0000","last_became_peered":"2024-03-09T14:46:26.802476+0000","last_unstale":"2024-03-09T15:36:48.190342+0000","last_undegraded":"2024-03-09T15:36:48.190342+0000","last_fullsized":"2024-03-09T15:36:48.190342+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:48.190138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:48.190138+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.190138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:25.403833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.11","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.548131+0000","last_change":"2024-03-09T15:36:13.548167+0000","last_active":"2024-03-09T15:36:13.548131+0000","last_peered":"2024-03-09T15:36:13.548131+0000","last_clean":"2024-03-09T15:36:13.548131+0000","last_became_active":"2024-03-09T14:46:18.760959+0000","last_became_peered":"2024-03-09T14:46:18.760959+0000","last_unstale":"2024-03-09T15:36:13.548131+0000","last_undegraded":"2024-03-09T15:36:13.548131+0000","last_fullsized":"2024-03-09T15:36:13.548131+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:13.548016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:13.548016+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.548016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:46:22.640051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.12","version":"258'7","reported_seq":86,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.252904+0000","last_change":"2024-03-09T15:36:16.252969+0000","last_active":"2024-03-09T15:36:16.252904+0000","last_peered":"2024-03-09T15:36:16.252904+0000","last_clean":"2024-03-09T15:36:16.252904+0000","last_became_active":"2024-03-09T14:39:13.273740+0000","last_became_peered":"2024-03-09T14:39:13.273740+0000","last_unstale":"2024-03-09T15:36:16.252904+0000","last_undegraded":"2024-03-09T15:36:16.252904+0000","last_fullsized":"2024-03-09T15:36:16.252904+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'7","last_scrub_stamp":"2024-03-09T15:36:16.252708+0000","last_deep_scrub":"258'7","last_deep_scrub_stamp":"2024-03-09T15:36:16.252708+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.252708+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:42:28.881579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":26,"num_write":13,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.14","version":"261'2","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:08.964088+0000","last_change":"2024-03-09T15:37:08.964222+0000","last_active":"2024-03-09T15:37:08.964088+0000","last_peered":"2024-03-09T15:37:08.964088+0000","last_clean":"2024-03-09T15:37:08.964088+0000","last_became_active":"2024-03-09T15:09:50.064428+0000","last_became_peered":"2024-03-09T15:09:50.064428+0000","last_unstale":"2024-03-09T15:37:08.964088+0000","last_undegraded":"2024-03-09T15:37:08.964088+0000","last_fullsized":"2024-03-09T15:37:08.964088+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:37:08.963834+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:37:08.963834+0000","last_clean_scrub_stamp":"2024-03-09T15:37:08.963834+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:30:24.868391+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.13","version":"261'3","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.462155+0000","last_change":"2024-03-09T15:36:33.462284+0000","last_active":"2024-03-09T15:36:33.462155+0000","last_peered":"2024-03-09T15:36:33.462155+0000","last_clean":"2024-03-09T15:36:33.462155+0000","last_became_active":"2024-03-09T14:46:26.813088+0000","last_became_peered":"2024-03-09T14:46:26.813088+0000","last_unstale":"2024-03-09T15:36:33.462155+0000","last_undegraded":"2024-03-09T15:36:33.462155+0000","last_fullsized":"2024-03-09T15:36:33.462155+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:33.461831+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:33.461831+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.461831+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:23:33.077781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.12","version":"261'6","reported_seq":68,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.313727+0000","last_change":"2024-03-09T15:36:45.313892+0000","last_active":"2024-03-09T15:36:45.313727+0000","last_peered":"2024-03-09T15:36:45.313727+0000","last_clean":"2024-03-09T15:36:45.313727+0000","last_became_active":"2024-03-09T14:46:18.782158+0000","last_became_peered":"2024-03-09T14:46:18.782158+0000","last_unstale":"2024-03-09T15:36:45.313727+0000","last_undegraded":"2024-03-09T15:36:45.313727+0000","last_fullsized":"2024-03-09T15:36:45.313727+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:45.313488+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:45.313488+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.313488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:37.339820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":11264,"num_write":6,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.11","version":"258'36","reported_seq":157,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.155312+0000","last_change":"2024-03-09T15:36:14.155374+0000","last_active":"2024-03-09T15:36:14.155312+0000","last_peered":"2024-03-09T15:36:14.155312+0000","last_clean":"2024-03-09T15:36:14.155312+0000","last_became_active":"2024-03-09T14:39:13.260219+0000","last_became_peered":"2024-03-09T14:39:13.260219+0000","last_unstale":"2024-03-09T15:36:14.155312+0000","last_undegraded":"2024-03-09T15:36:14.155312+0000","last_fullsized":"2024-03-09T15:36:14.155312+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'36","last_scrub_stamp":"2024-03-09T15:36:14.155108+0000","last_deep_scrub":"258'36","last_deep_scrub_stamp":"2024-03-09T15:36:14.155108+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.155108+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:26:18.598712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":118,"num_write":57,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.15","version":"261'2","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:10.665106+0000","last_change":"2024-03-09T15:37:10.665177+0000","last_active":"2024-03-09T15:37:10.665106+0000","last_peered":"2024-03-09T15:37:10.665106+0000","last_clean":"2024-03-09T15:37:10.665106+0000","last_became_active":"2024-03-09T15:09:50.081688+0000","last_became_peered":"2024-03-09T15:09:50.081688+0000","last_unstale":"2024-03-09T15:37:10.665106+0000","last_undegraded":"2024-03-09T15:37:10.665106+0000","last_fullsized":"2024-03-09T15:37:10.665106+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:37:10.664908+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:37:10.664908+0000","last_clean_scrub_stamp":"2024-03-09T15:37:10.664908+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:59:48.186267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.12","version":"261'2","reported_seq":55,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.350486+0000","last_change":"2024-03-09T15:36:47.350548+0000","last_active":"2024-03-09T15:36:47.350486+0000","last_peered":"2024-03-09T15:36:47.350486+0000","last_clean":"2024-03-09T15:36:47.350486+0000","last_became_active":"2024-03-09T14:46:26.806155+0000","last_became_peered":"2024-03-09T14:46:26.806155+0000","last_unstale":"2024-03-09T15:36:47.350486+0000","last_undegraded":"2024-03-09T15:36:47.350486+0000","last_fullsized":"2024-03-09T15:36:47.350486+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:47.350284+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:47.350284+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.350284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:49:11.128257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.13","version":"261'2","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:53.834708+0000","last_change":"2024-03-09T15:36:53.834774+0000","last_active":"2024-03-09T15:36:53.834708+0000","last_peered":"2024-03-09T15:36:53.834708+0000","last_clean":"2024-03-09T15:36:53.834708+0000","last_became_active":"2024-03-09T14:46:18.782340+0000","last_became_peered":"2024-03-09T14:46:18.782340+0000","last_unstale":"2024-03-09T15:36:53.834708+0000","last_undegraded":"2024-03-09T15:36:53.834708+0000","last_fullsized":"2024-03-09T15:36:53.834708+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:53.834512+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:53.834512+0000","last_clean_scrub_stamp":"2024-03-09T15:36:53.834512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:38:35.688017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":50,"num_write":2,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.10","version":"261'68","reported_seq":266,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.437896+0000","last_change":"2024-03-09T15:36:19.437984+0000","last_active":"2024-03-09T15:36:19.437896+0000","last_peered":"2024-03-09T15:36:19.437896+0000","last_clean":"2024-03-09T15:36:19.437896+0000","last_became_active":"2024-03-09T14:39:13.245107+0000","last_became_peered":"2024-03-09T14:39:13.245107+0000","last_unstale":"2024-03-09T15:36:19.437896+0000","last_undegraded":"2024-03-09T15:36:19.437896+0000","last_fullsized":"2024-03-09T15:36:19.437896+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'68","last_scrub_stamp":"2024-03-09T15:36:19.437698+0000","last_deep_scrub":"261'68","last_deep_scrub_stamp":"2024-03-09T15:36:19.437698+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.437698+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:55:08.019711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":382,"num_read_kb":270,"num_write":93,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1976,"num_omap_keys":4,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.a","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.327061+0000","last_change":"2024-03-09T15:36:42.327115+0000","last_active":"2024-03-09T15:36:42.327061+0000","last_peered":"2024-03-09T15:36:42.327061+0000","last_clean":"2024-03-09T15:36:42.327061+0000","last_became_active":"2024-03-09T15:09:50.079554+0000","last_became_peered":"2024-03-09T15:09:50.079554+0000","last_unstale":"2024-03-09T15:36:42.327061+0000","last_undegraded":"2024-03-09T15:36:42.327061+0000","last_fullsized":"2024-03-09T15:36:42.327061+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:42.326870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:42.326870+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.326870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:01:30.715008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.d","version":"261'5","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.470652+0000","last_change":"2024-03-09T15:36:25.470799+0000","last_active":"2024-03-09T15:36:25.470652+0000","last_peered":"2024-03-09T15:36:25.470652+0000","last_clean":"2024-03-09T15:36:25.470652+0000","last_became_active":"2024-03-09T14:46:26.812859+0000","last_became_peered":"2024-03-09T14:46:26.812859+0000","last_unstale":"2024-03-09T15:36:25.470652+0000","last_undegraded":"2024-03-09T15:36:25.470652+0000","last_fullsized":"2024-03-09T15:36:25.470652+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'5","last_scrub_stamp":"2024-03-09T15:36:25.470415+0000","last_deep_scrub":"261'5","last_deep_scrub_stamp":"2024-03-09T15:36:25.470415+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.470415+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:20:04.740693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":8192,"num_write":10,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.c","version":"261'11","reported_seq":79,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.263874+0000","last_change":"2024-03-09T15:36:17.264004+0000","last_active":"2024-03-09T15:36:17.263874+0000","last_peered":"2024-03-09T15:36:17.263874+0000","last_clean":"2024-03-09T15:36:17.263874+0000","last_became_active":"2024-03-09T14:46:18.766370+0000","last_became_peered":"2024-03-09T14:46:18.766370+0000","last_unstale":"2024-03-09T15:36:17.263874+0000","last_undegraded":"2024-03-09T15:36:17.263874+0000","last_fullsized":"2024-03-09T15:36:17.263874+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'11","last_scrub_stamp":"2024-03-09T15:36:17.263641+0000","last_deep_scrub":"261'11","last_deep_scrub_stamp":"2024-03-09T15:36:17.263641+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.263641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:28:25.038359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":20480,"num_write":16,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.f","version":"258'18","reported_seq":108,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.541046+0000","last_change":"2024-03-09T15:36:12.541106+0000","last_active":"2024-03-09T15:36:12.541046+0000","last_peered":"2024-03-09T15:36:12.541046+0000","last_clean":"2024-03-09T15:36:12.541046+0000","last_became_active":"2024-03-09T14:39:13.268971+0000","last_became_peered":"2024-03-09T14:39:13.268971+0000","last_unstale":"2024-03-09T15:36:12.541046+0000","last_undegraded":"2024-03-09T15:36:12.541046+0000","last_fullsized":"2024-03-09T15:36:12.541046+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'18","last_scrub_stamp":"2024-03-09T15:36:12.540848+0000","last_deep_scrub":"258'18","last_deep_scrub_stamp":"2024-03-09T15:36:12.540848+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.540848+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:51.392572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.b","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.229996+0000","last_change":"2024-03-09T15:36:37.229996+0000","last_active":"2024-03-09T15:36:37.229996+0000","last_peered":"2024-03-09T15:36:37.229996+0000","last_clean":"2024-03-09T15:36:37.229996+0000","last_became_active":"2024-03-09T15:09:50.069508+0000","last_became_peered":"2024-03-09T15:09:50.069508+0000","last_unstale":"2024-03-09T15:36:37.229996+0000","last_undegraded":"2024-03-09T15:36:37.229996+0000","last_fullsized":"2024-03-09T15:36:37.229996+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:37.229744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:37.229744+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.229744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:14:18.843785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.c","version":"261'3","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.457377+0000","last_change":"2024-03-09T15:36:29.457444+0000","last_active":"2024-03-09T15:36:29.457377+0000","last_peered":"2024-03-09T15:36:29.457377+0000","last_clean":"2024-03-09T15:36:29.457377+0000","last_became_active":"2024-03-09T14:46:26.813009+0000","last_became_peered":"2024-03-09T14:46:26.813009+0000","last_unstale":"2024-03-09T15:36:29.457377+0000","last_undegraded":"2024-03-09T15:36:29.457377+0000","last_fullsized":"2024-03-09T15:36:29.457377+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:29.457166+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:29.457166+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.457166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:35:35.350489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2048,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.d","version":"261'3","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.167902+0000","last_change":"2024-03-09T15:36:36.167961+0000","last_active":"2024-03-09T15:36:36.167902+0000","last_peered":"2024-03-09T15:36:36.167902+0000","last_clean":"2024-03-09T15:36:36.167902+0000","last_became_active":"2024-03-09T14:46:18.770732+0000","last_became_peered":"2024-03-09T14:46:18.770732+0000","last_unstale":"2024-03-09T15:36:36.167902+0000","last_undegraded":"2024-03-09T15:36:36.167902+0000","last_fullsized":"2024-03-09T15:36:36.167902+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:36.167630+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:36.167630+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.167630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:24.866080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.e","version":"261'8","reported_seq":88,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.655370+0000","last_change":"2024-03-09T15:36:18.655429+0000","last_active":"2024-03-09T15:36:18.655370+0000","last_peered":"2024-03-09T15:36:18.655370+0000","last_clean":"2024-03-09T15:36:18.655370+0000","last_became_active":"2024-03-09T14:39:13.271672+0000","last_became_peered":"2024-03-09T14:39:13.271672+0000","last_unstale":"2024-03-09T15:36:18.655370+0000","last_undegraded":"2024-03-09T15:36:18.655370+0000","last_fullsized":"2024-03-09T15:36:18.655370+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:18.655167+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:18.655167+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.655167+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:25:51.301220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.8","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.370543+0000","last_change":"2024-03-09T15:36:50.370653+0000","last_active":"2024-03-09T15:36:50.370543+0000","last_peered":"2024-03-09T15:36:50.370543+0000","last_clean":"2024-03-09T15:36:50.370543+0000","last_became_active":"2024-03-09T15:09:50.061662+0000","last_became_peered":"2024-03-09T15:09:50.061662+0000","last_unstale":"2024-03-09T15:36:50.370543+0000","last_undegraded":"2024-03-09T15:36:50.370543+0000","last_fullsized":"2024-03-09T15:36:50.370543+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:50.370325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:50.370325+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.370325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:31:38.698974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.f","version":"261'6","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.198675+0000","last_change":"2024-03-09T15:36:51.198827+0000","last_active":"2024-03-09T15:36:51.198675+0000","last_peered":"2024-03-09T15:36:51.198675+0000","last_clean":"2024-03-09T15:36:51.198675+0000","last_became_active":"2024-03-09T14:46:26.899430+0000","last_became_peered":"2024-03-09T14:46:26.899430+0000","last_unstale":"2024-03-09T15:36:51.198675+0000","last_undegraded":"2024-03-09T15:36:51.198675+0000","last_fullsized":"2024-03-09T15:36:51.198675+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:51.198415+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:51.198415+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.198415+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:37:54.991742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5120,"num_write":11,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.e","version":"258'3","reported_seq":65,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.271195+0000","last_change":"2024-03-09T15:36:15.271259+0000","last_active":"2024-03-09T15:36:15.271195+0000","last_peered":"2024-03-09T15:36:15.271195+0000","last_clean":"2024-03-09T15:36:15.271195+0000","last_became_active":"2024-03-09T14:46:18.768694+0000","last_became_peered":"2024-03-09T14:46:18.768694+0000","last_unstale":"2024-03-09T15:36:15.271195+0000","last_undegraded":"2024-03-09T15:36:15.271195+0000","last_fullsized":"2024-03-09T15:36:15.271195+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:15.270937+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:15.270937+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.270937+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:49:15.823574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.d","version":"261'43","reported_seq":168,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.940181+0000","last_change":"2024-03-09T15:36:15.940345+0000","last_active":"2024-03-09T15:36:15.940181+0000","last_peered":"2024-03-09T15:36:15.940181+0000","last_clean":"2024-03-09T15:36:15.940181+0000","last_became_active":"2024-03-09T14:39:13.257275+0000","last_became_peered":"2024-03-09T14:39:13.257275+0000","last_unstale":"2024-03-09T15:36:15.940181+0000","last_undegraded":"2024-03-09T15:36:15.940181+0000","last_fullsized":"2024-03-09T15:36:15.940181+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:15.939952+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:15.939952+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.939952+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:40:15.055734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":175,"num_read_kb":123,"num_write":70,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.9","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:09.188836+0000","last_change":"2024-03-09T15:37:09.188890+0000","last_active":"2024-03-09T15:37:09.188836+0000","last_peered":"2024-03-09T15:37:09.188836+0000","last_clean":"2024-03-09T15:37:09.188836+0000","last_became_active":"2024-03-09T15:09:50.076702+0000","last_became_peered":"2024-03-09T15:09:50.076702+0000","last_unstale":"2024-03-09T15:37:09.188836+0000","last_undegraded":"2024-03-09T15:37:09.188836+0000","last_fullsized":"2024-03-09T15:37:09.188836+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:09.188634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:09.188634+0000","last_clean_scrub_stamp":"2024-03-09T15:37:09.188634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:40:20.717469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.e","version":"261'6","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.497081+0000","last_change":"2024-03-09T15:36:37.497144+0000","last_active":"2024-03-09T15:36:37.497081+0000","last_peered":"2024-03-09T15:36:37.497081+0000","last_clean":"2024-03-09T15:36:37.497081+0000","last_became_active":"2024-03-09T14:46:26.814709+0000","last_became_peered":"2024-03-09T14:46:26.814709+0000","last_unstale":"2024-03-09T15:36:37.497081+0000","last_undegraded":"2024-03-09T15:36:37.497081+0000","last_fullsized":"2024-03-09T15:36:37.497081+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:37.496875+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:37.496875+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.496875+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:32:45.549662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.f","version":"261'5","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.512325+0000","last_change":"2024-03-09T15:36:13.512372+0000","last_active":"2024-03-09T15:36:13.512325+0000","last_peered":"2024-03-09T15:36:13.512325+0000","last_clean":"2024-03-09T15:36:13.512325+0000","last_became_active":"2024-03-09T14:46:18.766000+0000","last_became_peered":"2024-03-09T14:46:18.766000+0000","last_unstale":"2024-03-09T15:36:13.512325+0000","last_undegraded":"2024-03-09T15:36:13.512325+0000","last_fullsized":"2024-03-09T15:36:13.512325+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'5","last_scrub_stamp":"2024-03-09T15:36:13.512169+0000","last_deep_scrub":"261'5","last_deep_scrub_stamp":"2024-03-09T15:36:13.512169+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.512169+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:07:18.735860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4096,"num_write":10,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.c","version":"0'0","reported_seq":68,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.206724+0000","last_change":"2024-03-09T15:36:26.206778+0000","last_active":"2024-03-09T15:36:26.206724+0000","last_peered":"2024-03-09T15:36:26.206724+0000","last_clean":"2024-03-09T15:36:26.206724+0000","last_became_active":"2024-03-09T14:39:13.270527+0000","last_became_peered":"2024-03-09T14:39:13.270527+0000","last_unstale":"2024-03-09T15:36:26.206724+0000","last_undegraded":"2024-03-09T15:36:26.206724+0000","last_fullsized":"2024-03-09T15:36:26.206724+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:26.206522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:26.206522+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.206522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:18:22.897656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.e","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.187069+0000","last_change":"2024-03-09T15:09:50.087358+0000","last_active":"2024-03-09T15:36:12.187069+0000","last_peered":"2024-03-09T15:36:12.187069+0000","last_clean":"2024-03-09T15:36:12.187069+0000","last_became_active":"2024-03-09T15:09:50.085548+0000","last_became_peered":"2024-03-09T15:09:50.085548+0000","last_unstale":"2024-03-09T15:36:12.187069+0000","last_undegraded":"2024-03-09T15:36:12.187069+0000","last_fullsized":"2024-03-09T15:36:12.187069+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.9","version":"0'0","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:57.571078+0000","last_change":"2024-03-09T15:36:57.571134+0000","last_active":"2024-03-09T15:36:57.571078+0000","last_peered":"2024-03-09T15:36:57.571078+0000","last_clean":"2024-03-09T15:36:57.571078+0000","last_became_active":"2024-03-09T14:46:26.801235+0000","last_became_peered":"2024-03-09T14:46:26.801235+0000","last_unstale":"2024-03-09T15:36:57.571078+0000","last_undegraded":"2024-03-09T15:36:57.571078+0000","last_fullsized":"2024-03-09T15:36:57.571078+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:57.570883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:57.570883+0000","last_clean_scrub_stamp":"2024-03-09T15:36:57.570883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:56:17.614656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.8","version":"0'0","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.316818+0000","last_change":"2024-03-09T15:36:23.316871+0000","last_active":"2024-03-09T15:36:23.316818+0000","last_peered":"2024-03-09T15:36:23.316818+0000","last_clean":"2024-03-09T15:36:23.316818+0000","last_became_active":"2024-03-09T14:46:18.774547+0000","last_became_peered":"2024-03-09T14:46:18.774547+0000","last_unstale":"2024-03-09T15:36:23.316818+0000","last_undegraded":"2024-03-09T15:36:23.316818+0000","last_fullsized":"2024-03-09T15:36:23.316818+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:23.316611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:23.316611+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.316611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:09:52.305120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.b","version":"258'10","reported_seq":94,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.250282+0000","last_change":"2024-03-09T15:36:20.250347+0000","last_active":"2024-03-09T15:36:20.250282+0000","last_peered":"2024-03-09T15:36:20.250282+0000","last_clean":"2024-03-09T15:36:20.250282+0000","last_became_active":"2024-03-09T14:39:13.257238+0000","last_became_peered":"2024-03-09T14:39:13.257238+0000","last_unstale":"2024-03-09T15:36:20.250282+0000","last_undegraded":"2024-03-09T15:36:20.250282+0000","last_fullsized":"2024-03-09T15:36:20.250282+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:20.250091+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:20.250091+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.250091+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:55:12.169918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.f","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:05.135984+0000","last_change":"2024-03-09T15:37:05.136040+0000","last_active":"2024-03-09T15:37:05.135984+0000","last_peered":"2024-03-09T15:37:05.135984+0000","last_clean":"2024-03-09T15:37:05.135984+0000","last_became_active":"2024-03-09T15:09:50.070526+0000","last_became_peered":"2024-03-09T15:09:50.070526+0000","last_unstale":"2024-03-09T15:37:05.135984+0000","last_undegraded":"2024-03-09T15:37:05.135984+0000","last_fullsized":"2024-03-09T15:37:05.135984+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:05.135769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:05.135769+0000","last_clean_scrub_stamp":"2024-03-09T15:37:05.135769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:09:12.888354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.8","version":"0'0","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.159602+0000","last_change":"2024-03-09T15:36:25.159656+0000","last_active":"2024-03-09T15:36:25.159602+0000","last_peered":"2024-03-09T15:36:25.159602+0000","last_clean":"2024-03-09T15:36:25.159602+0000","last_became_active":"2024-03-09T14:46:26.893908+0000","last_became_peered":"2024-03-09T14:46:26.893908+0000","last_unstale":"2024-03-09T15:36:25.159602+0000","last_undegraded":"2024-03-09T15:36:25.159602+0000","last_fullsized":"2024-03-09T15:36:25.159602+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.159401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.159401+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.159401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:51:03.092803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.9","version":"261'3","reported_seq":63,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.495189+0000","last_change":"2024-03-09T15:36:11.495211+0000","last_active":"2024-03-09T15:36:11.495189+0000","last_peered":"2024-03-09T15:36:11.495189+0000","last_clean":"2024-03-09T15:36:11.495189+0000","last_became_active":"2024-03-09T14:46:18.761367+0000","last_became_peered":"2024-03-09T14:46:18.761367+0000","last_unstale":"2024-03-09T15:36:11.495189+0000","last_undegraded":"2024-03-09T15:36:11.495189+0000","last_fullsized":"2024-03-09T15:36:11.495189+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:11.495111+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:11.495111+0000","last_clean_scrub_stamp":"2024-03-09T15:36:11.495111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:55:49.995490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.a","version":"258'23","reported_seq":132,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.129434+0000","last_change":"2024-03-09T15:36:26.129498+0000","last_active":"2024-03-09T15:36:26.129434+0000","last_peered":"2024-03-09T15:36:26.129434+0000","last_clean":"2024-03-09T15:36:26.129434+0000","last_became_active":"2024-03-09T14:39:13.252547+0000","last_became_peered":"2024-03-09T14:39:13.252547+0000","last_unstale":"2024-03-09T15:36:26.129434+0000","last_undegraded":"2024-03-09T15:36:26.129434+0000","last_fullsized":"2024-03-09T15:36:26.129434+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'23","last_scrub_stamp":"2024-03-09T15:36:26.129236+0000","last_deep_scrub":"258'23","last_deep_scrub_stamp":"2024-03-09T15:36:26.129236+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.129236+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:37:12.973344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":86,"num_write":38,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.c","version":"261'2","reported_seq":35,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:04.138010+0000","last_change":"2024-03-09T15:37:04.138094+0000","last_active":"2024-03-09T15:37:04.138010+0000","last_peered":"2024-03-09T15:37:04.138010+0000","last_clean":"2024-03-09T15:37:04.138010+0000","last_became_active":"2024-03-09T15:09:50.069324+0000","last_became_peered":"2024-03-09T15:09:50.069324+0000","last_unstale":"2024-03-09T15:37:04.138010+0000","last_undegraded":"2024-03-09T15:37:04.138010+0000","last_fullsized":"2024-03-09T15:37:04.138010+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:37:04.137795+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:37:04.137795+0000","last_clean_scrub_stamp":"2024-03-09T15:37:04.137795+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:45:38.998974+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.b","version":"258'3","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.176799+0000","last_change":"2024-03-09T15:36:49.176973+0000","last_active":"2024-03-09T15:36:49.176799+0000","last_peered":"2024-03-09T15:36:49.176799+0000","last_clean":"2024-03-09T15:36:49.176799+0000","last_became_active":"2024-03-09T14:46:26.804615+0000","last_became_peered":"2024-03-09T14:46:26.804615+0000","last_unstale":"2024-03-09T15:36:49.176799+0000","last_undegraded":"2024-03-09T15:36:49.176799+0000","last_fullsized":"2024-03-09T15:36:49.176799+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:49.176568+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:49.176568+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.176568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:20:16.951902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.a","version":"258'3","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.202430+0000","last_change":"2024-03-09T15:36:11.202430+0000","last_active":"2024-03-09T15:36:11.202430+0000","last_peered":"2024-03-09T15:36:11.202430+0000","last_clean":"2024-03-09T15:36:11.202430+0000","last_became_active":"2024-03-09T14:46:18.768431+0000","last_became_peered":"2024-03-09T14:46:18.768431+0000","last_unstale":"2024-03-09T15:36:11.202430+0000","last_undegraded":"2024-03-09T15:36:11.202430+0000","last_fullsized":"2024-03-09T15:36:11.202430+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:11.202170+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:11.202170+0000","last_clean_scrub_stamp":"2024-03-09T15:36:11.202170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:12:25.295523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.9","version":"261'43","reported_seq":166,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.428686+0000","last_change":"2024-03-09T15:36:18.428750+0000","last_active":"2024-03-09T15:36:18.428686+0000","last_peered":"2024-03-09T15:36:18.428686+0000","last_clean":"2024-03-09T15:36:18.428686+0000","last_became_active":"2024-03-09T14:39:13.259975+0000","last_became_peered":"2024-03-09T14:39:13.259975+0000","last_unstale":"2024-03-09T15:36:18.428686+0000","last_undegraded":"2024-03-09T15:36:18.428686+0000","last_fullsized":"2024-03-09T15:36:18.428686+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:18.428488+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:18.428488+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.428488+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:15:47.696457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":169,"num_read_kb":119,"num_write":74,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.d","version":"0'0","reported_seq":25,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:10.740595+0000","last_change":"2024-03-09T15:09:50.084646+0000","last_active":"2024-03-09T15:36:10.740595+0000","last_peered":"2024-03-09T15:36:10.740595+0000","last_clean":"2024-03-09T15:36:10.740595+0000","last_became_active":"2024-03-09T15:09:50.084074+0000","last_became_peered":"2024-03-09T15:09:50.084074+0000","last_unstale":"2024-03-09T15:36:10.740595+0000","last_undegraded":"2024-03-09T15:36:10.740595+0000","last_fullsized":"2024-03-09T15:36:10.740595+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:09:49.016993+0000","last_clean_scrub_stamp":"2024-03-09T15:09:49.016993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.a","version":"258'3","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.196067+0000","last_change":"2024-03-09T15:36:21.196067+0000","last_active":"2024-03-09T15:36:21.196067+0000","last_peered":"2024-03-09T15:36:21.196067+0000","last_clean":"2024-03-09T15:36:21.196067+0000","last_became_active":"2024-03-09T14:46:26.825949+0000","last_became_peered":"2024-03-09T14:46:26.825949+0000","last_unstale":"2024-03-09T15:36:21.196067+0000","last_undegraded":"2024-03-09T15:36:21.196067+0000","last_fullsized":"2024-03-09T15:36:21.196067+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:21.195795+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:21.195795+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.195795+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:51.756963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.b","version":"261'3","reported_seq":64,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.180486+0000","last_change":"2024-03-09T15:36:29.180546+0000","last_active":"2024-03-09T15:36:29.180486+0000","last_peered":"2024-03-09T15:36:29.180486+0000","last_clean":"2024-03-09T15:36:29.180486+0000","last_became_active":"2024-03-09T14:46:18.784869+0000","last_became_peered":"2024-03-09T14:46:18.784869+0000","last_unstale":"2024-03-09T15:36:29.180486+0000","last_undegraded":"2024-03-09T15:36:29.180486+0000","last_fullsized":"2024-03-09T15:36:29.180486+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:29.180292+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:29.180292+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.180292+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:09:41.856539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.8","version":"258'10","reported_seq":95,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.980744+0000","last_change":"2024-03-09T15:36:16.980807+0000","last_active":"2024-03-09T15:36:16.980744+0000","last_peered":"2024-03-09T15:36:16.980744+0000","last_clean":"2024-03-09T15:36:16.980744+0000","last_became_active":"2024-03-09T14:39:13.264006+0000","last_became_peered":"2024-03-09T14:39:13.264006+0000","last_unstale":"2024-03-09T15:36:16.980744+0000","last_undegraded":"2024-03-09T15:36:16.980744+0000","last_fullsized":"2024-03-09T15:36:16.980744+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:16.980539+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:16.980539+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.980539+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:50:44.529795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.2","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.236049+0000","last_change":"2024-03-09T15:36:33.236049+0000","last_active":"2024-03-09T15:36:33.236049+0000","last_peered":"2024-03-09T15:36:33.236049+0000","last_clean":"2024-03-09T15:36:33.236049+0000","last_became_active":"2024-03-09T15:09:50.065487+0000","last_became_peered":"2024-03-09T15:09:50.065487+0000","last_unstale":"2024-03-09T15:36:33.236049+0000","last_undegraded":"2024-03-09T15:36:33.236049+0000","last_fullsized":"2024-03-09T15:36:33.236049+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:33.235751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:33.235751+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.235751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:45.773762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.5","version":"258'2","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.230666+0000","last_change":"2024-03-09T15:36:35.230666+0000","last_active":"2024-03-09T15:36:35.230666+0000","last_peered":"2024-03-09T15:36:35.230666+0000","last_clean":"2024-03-09T15:36:35.230666+0000","last_became_active":"2024-03-09T14:46:26.812137+0000","last_became_peered":"2024-03-09T14:46:26.812137+0000","last_unstale":"2024-03-09T15:36:35.230666+0000","last_undegraded":"2024-03-09T15:36:35.230666+0000","last_fullsized":"2024-03-09T15:36:35.230666+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'2","last_scrub_stamp":"2024-03-09T15:36:35.230397+0000","last_deep_scrub":"258'2","last_deep_scrub_stamp":"2024-03-09T15:36:35.230397+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.230397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:52:41.341440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4096,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.4","version":"0'0","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.218589+0000","last_change":"2024-03-09T15:36:19.218589+0000","last_active":"2024-03-09T15:36:19.218589+0000","last_peered":"2024-03-09T15:36:19.218589+0000","last_clean":"2024-03-09T15:36:19.218589+0000","last_became_active":"2024-03-09T14:46:18.751608+0000","last_became_peered":"2024-03-09T14:46:18.751608+0000","last_unstale":"2024-03-09T15:36:19.218589+0000","last_undegraded":"2024-03-09T15:36:19.218589+0000","last_fullsized":"2024-03-09T15:36:19.218589+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:19.218339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:19.218339+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.218339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:28:04.515161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.7","version":"261'8","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.227007+0000","last_change":"2024-03-09T15:36:14.227039+0000","last_active":"2024-03-09T15:36:14.227007+0000","last_peered":"2024-03-09T15:36:14.227007+0000","last_clean":"2024-03-09T15:36:14.227007+0000","last_became_active":"2024-03-09T14:39:13.273363+0000","last_became_peered":"2024-03-09T14:39:13.273363+0000","last_unstale":"2024-03-09T15:36:14.227007+0000","last_undegraded":"2024-03-09T15:36:14.227007+0000","last_fullsized":"2024-03-09T15:36:14.227007+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:14.226893+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:14.226893+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.226893+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:21:38.723375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.3","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:57.884969+0000","last_change":"2024-03-09T15:36:57.885023+0000","last_active":"2024-03-09T15:36:57.884969+0000","last_peered":"2024-03-09T15:36:57.884969+0000","last_clean":"2024-03-09T15:36:57.884969+0000","last_became_active":"2024-03-09T15:09:50.072488+0000","last_became_peered":"2024-03-09T15:09:50.072488+0000","last_unstale":"2024-03-09T15:36:57.884969+0000","last_undegraded":"2024-03-09T15:36:57.884969+0000","last_fullsized":"2024-03-09T15:36:57.884969+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:57.884771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:57.884771+0000","last_clean_scrub_stamp":"2024-03-09T15:36:57.884771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:47:08.805239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.4","version":"258'2","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.256968+0000","last_change":"2024-03-09T15:36:39.257131+0000","last_active":"2024-03-09T15:36:39.256968+0000","last_peered":"2024-03-09T15:36:39.256968+0000","last_clean":"2024-03-09T15:36:39.256968+0000","last_became_active":"2024-03-09T14:46:26.894506+0000","last_became_peered":"2024-03-09T14:46:26.894506+0000","last_unstale":"2024-03-09T15:36:39.256968+0000","last_undegraded":"2024-03-09T15:36:39.256968+0000","last_fullsized":"2024-03-09T15:36:39.256968+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'2","last_scrub_stamp":"2024-03-09T15:36:39.256730+0000","last_deep_scrub":"258'2","last_deep_scrub_stamp":"2024-03-09T15:36:39.256730+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.256730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:48:26.210989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":23,"num_write":2,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.5","version":"261'6","reported_seq":77,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.217768+0000","last_change":"2024-03-09T15:36:37.217842+0000","last_active":"2024-03-09T15:36:37.217768+0000","last_peered":"2024-03-09T15:36:37.217768+0000","last_clean":"2024-03-09T15:36:37.217768+0000","last_became_active":"2024-03-09T14:46:18.770799+0000","last_became_peered":"2024-03-09T14:46:18.770799+0000","last_unstale":"2024-03-09T15:36:37.217768+0000","last_undegraded":"2024-03-09T15:36:37.217768+0000","last_fullsized":"2024-03-09T15:36:37.217768+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:37.217579+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:37.217579+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.217579+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:32:01.418959+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.6","version":"261'22","reported_seq":121,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.972029+0000","last_change":"2024-03-09T15:36:14.972165+0000","last_active":"2024-03-09T15:36:14.972029+0000","last_peered":"2024-03-09T15:36:14.972029+0000","last_clean":"2024-03-09T15:36:14.972029+0000","last_became_active":"2024-03-09T14:39:13.266246+0000","last_became_peered":"2024-03-09T14:39:13.266246+0000","last_unstale":"2024-03-09T15:36:14.972029+0000","last_undegraded":"2024-03-09T15:36:14.972029+0000","last_fullsized":"2024-03-09T15:36:14.972029+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'22","last_scrub_stamp":"2024-03-09T15:36:14.971789+0000","last_deep_scrub":"261'22","last_deep_scrub_stamp":"2024-03-09T15:36:14.971789+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.971789+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:50:18.362986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":65,"num_write":41,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.0","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:53.136614+0000","last_change":"2024-03-09T15:36:53.136668+0000","last_active":"2024-03-09T15:36:53.136614+0000","last_peered":"2024-03-09T15:36:53.136614+0000","last_clean":"2024-03-09T15:36:53.136614+0000","last_became_active":"2024-03-09T15:09:50.063350+0000","last_became_peered":"2024-03-09T15:09:50.063350+0000","last_unstale":"2024-03-09T15:36:53.136614+0000","last_undegraded":"2024-03-09T15:36:53.136614+0000","last_fullsized":"2024-03-09T15:36:53.136614+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:53.136413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:53.136413+0000","last_clean_scrub_stamp":"2024-03-09T15:36:53.136413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:11:03.539751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.7","version":"0'0","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.151944+0000","last_change":"2024-03-09T15:36:29.151997+0000","last_active":"2024-03-09T15:36:29.151944+0000","last_peered":"2024-03-09T15:36:29.151944+0000","last_clean":"2024-03-09T15:36:29.151944+0000","last_became_active":"2024-03-09T14:46:26.896503+0000","last_became_peered":"2024-03-09T14:46:26.896503+0000","last_unstale":"2024-03-09T15:36:29.151944+0000","last_undegraded":"2024-03-09T15:36:29.151944+0000","last_fullsized":"2024-03-09T15:36:29.151944+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:29.151756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:29.151756+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.151756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:12:40.874461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.6","version":"261'2","reported_seq":63,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.752529+0000","last_change":"2024-03-09T15:36:26.752595+0000","last_active":"2024-03-09T15:36:26.752529+0000","last_peered":"2024-03-09T15:36:26.752529+0000","last_clean":"2024-03-09T15:36:26.752529+0000","last_became_active":"2024-03-09T14:46:18.778572+0000","last_became_peered":"2024-03-09T14:46:18.778572+0000","last_unstale":"2024-03-09T15:36:26.752529+0000","last_undegraded":"2024-03-09T15:36:26.752529+0000","last_fullsized":"2024-03-09T15:36:26.752529+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:26.752326+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:26.752326+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.752326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:06:26.783165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.5","version":"258'18","reported_seq":116,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.164653+0000","last_change":"2024-03-09T15:36:23.164716+0000","last_active":"2024-03-09T15:36:23.164653+0000","last_peered":"2024-03-09T15:36:23.164653+0000","last_clean":"2024-03-09T15:36:23.164653+0000","last_became_active":"2024-03-09T14:39:13.263833+0000","last_became_peered":"2024-03-09T14:39:13.263833+0000","last_unstale":"2024-03-09T15:36:23.164653+0000","last_undegraded":"2024-03-09T15:36:23.164653+0000","last_fullsized":"2024-03-09T15:36:23.164653+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'18","last_scrub_stamp":"2024-03-09T15:36:23.164449+0000","last_deep_scrub":"258'18","last_deep_scrub_stamp":"2024-03-09T15:36:23.164449+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.164449+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:22:04.204143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":58,"num_write":29,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.1","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:02.565044+0000","last_change":"2024-03-09T15:37:02.565097+0000","last_active":"2024-03-09T15:37:02.565044+0000","last_peered":"2024-03-09T15:37:02.565044+0000","last_clean":"2024-03-09T15:37:02.565044+0000","last_became_active":"2024-03-09T15:09:50.067947+0000","last_became_peered":"2024-03-09T15:09:50.067947+0000","last_unstale":"2024-03-09T15:37:02.565044+0000","last_undegraded":"2024-03-09T15:37:02.565044+0000","last_fullsized":"2024-03-09T15:37:02.565044+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:02.564849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:02.564849+0000","last_clean_scrub_stamp":"2024-03-09T15:37:02.564849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:16.040745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"67.6","version":"261'4","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:00.640406+0000","last_change":"2024-03-09T15:37:00.640539+0000","last_active":"2024-03-09T15:37:00.640406+0000","last_peered":"2024-03-09T15:37:00.640406+0000","last_clean":"2024-03-09T15:37:00.640406+0000","last_became_active":"2024-03-09T14:46:26.808384+0000","last_became_peered":"2024-03-09T14:46:26.808384+0000","last_unstale":"2024-03-09T15:37:00.640406+0000","last_undegraded":"2024-03-09T15:37:00.640406+0000","last_fullsized":"2024-03-09T15:37:00.640406+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'4","last_scrub_stamp":"2024-03-09T15:37:00.640182+0000","last_deep_scrub":"261'4","last_deep_scrub_stamp":"2024-03-09T15:37:00.640182+0000","last_clean_scrub_stamp":"2024-03-09T15:37:00.640182+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:28:34.063992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":180,"num_write":4,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.7","version":"261'9","reported_seq":83,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.291656+0000","last_change":"2024-03-09T15:36:14.291833+0000","last_active":"2024-03-09T15:36:14.291656+0000","last_peered":"2024-03-09T15:36:14.291656+0000","last_clean":"2024-03-09T15:36:14.291656+0000","last_became_active":"2024-03-09T14:46:18.762472+0000","last_became_peered":"2024-03-09T14:46:18.762472+0000","last_unstale":"2024-03-09T15:36:14.291656+0000","last_undegraded":"2024-03-09T15:36:14.291656+0000","last_fullsized":"2024-03-09T15:36:14.291656+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-03-09T15:36:14.291396+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-03-09T15:36:14.291396+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.291396+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:45:34.816033+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":30722,"num_write":17,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.4","version":"258'10","reported_seq":96,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.555803+0000","last_change":"2024-03-09T15:36:14.555905+0000","last_active":"2024-03-09T15:36:14.555803+0000","last_peered":"2024-03-09T15:36:14.555803+0000","last_clean":"2024-03-09T15:36:14.555803+0000","last_became_active":"2024-03-09T14:39:13.270238+0000","last_became_peered":"2024-03-09T14:39:13.270238+0000","last_unstale":"2024-03-09T15:36:14.555803+0000","last_undegraded":"2024-03-09T15:36:14.555803+0000","last_fullsized":"2024-03-09T15:36:14.555803+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:14.555648+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:14.555648+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.555648+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:38:18.000083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.6","version":"261'2","reported_seq":35,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.245425+0000","last_change":"2024-03-09T15:36:40.245494+0000","last_active":"2024-03-09T15:36:40.245425+0000","last_peered":"2024-03-09T15:36:40.245425+0000","last_clean":"2024-03-09T15:36:40.245425+0000","last_became_active":"2024-03-09T15:09:50.072180+0000","last_became_peered":"2024-03-09T15:09:50.072180+0000","last_unstale":"2024-03-09T15:36:40.245425+0000","last_undegraded":"2024-03-09T15:36:40.245425+0000","last_fullsized":"2024-03-09T15:36:40.245425+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:40.245236+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:40.245236+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.245236+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:47:53.099498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1","version":"0'0","reported_seq":55,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.542310+0000","last_change":"2024-03-09T15:36:21.542310+0000","last_active":"2024-03-09T15:36:21.542310+0000","last_peered":"2024-03-09T15:36:21.542310+0000","last_clean":"2024-03-09T15:36:21.542310+0000","last_became_active":"2024-03-09T14:46:26.799019+0000","last_became_peered":"2024-03-09T14:46:26.799019+0000","last_unstale":"2024-03-09T15:36:21.542310+0000","last_undegraded":"2024-03-09T15:36:21.542310+0000","last_fullsized":"2024-03-09T15:36:21.542310+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.542044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.542044+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.542044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:24:20.252130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.0","version":"0'0","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.229238+0000","last_change":"2024-03-09T15:36:18.229238+0000","last_active":"2024-03-09T15:36:18.229238+0000","last_peered":"2024-03-09T15:36:18.229238+0000","last_clean":"2024-03-09T15:36:18.229238+0000","last_became_active":"2024-03-09T14:46:18.752290+0000","last_became_peered":"2024-03-09T14:46:18.752290+0000","last_unstale":"2024-03-09T15:36:18.229238+0000","last_undegraded":"2024-03-09T15:36:18.229238+0000","last_fullsized":"2024-03-09T15:36:18.229238+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:18.228896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:18.228896+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.228896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:58:17.281185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.3","version":"261'61","reported_seq":224,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.587549+0000","last_change":"2024-03-09T15:36:14.587616+0000","last_active":"2024-03-09T15:36:14.587549+0000","last_peered":"2024-03-09T15:36:14.587549+0000","last_clean":"2024-03-09T15:36:14.587549+0000","last_became_active":"2024-03-09T14:39:13.259125+0000","last_became_peered":"2024-03-09T14:39:13.259125+0000","last_unstale":"2024-03-09T15:36:14.587549+0000","last_undegraded":"2024-03-09T15:36:14.587549+0000","last_fullsized":"2024-03-09T15:36:14.587549+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'61","last_scrub_stamp":"2024-03-09T15:36:14.587351+0000","last_deep_scrub":"261'61","last_deep_scrub_stamp":"2024-03-09T15:36:14.587351+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.587351+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:00:04.188736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":281,"num_read_kb":205,"num_write":98,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.7","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:08.000781+0000","last_change":"2024-03-09T15:37:08.000838+0000","last_active":"2024-03-09T15:37:08.000781+0000","last_peered":"2024-03-09T15:37:08.000781+0000","last_clean":"2024-03-09T15:37:08.000781+0000","last_became_active":"2024-03-09T15:09:50.062429+0000","last_became_peered":"2024-03-09T15:09:50.062429+0000","last_unstale":"2024-03-09T15:37:08.000781+0000","last_undegraded":"2024-03-09T15:37:08.000781+0000","last_fullsized":"2024-03-09T15:37:08.000781+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:08.000581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:08.000581+0000","last_clean_scrub_stamp":"2024-03-09T15:37:08.000581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:04:19.242539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.0","version":"261'6","reported_seq":68,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:54.855985+0000","last_change":"2024-03-09T15:36:54.856151+0000","last_active":"2024-03-09T15:36:54.855985+0000","last_peered":"2024-03-09T15:36:54.855985+0000","last_clean":"2024-03-09T15:36:54.855985+0000","last_became_active":"2024-03-09T14:46:26.815311+0000","last_became_peered":"2024-03-09T14:46:26.815311+0000","last_unstale":"2024-03-09T15:36:54.855985+0000","last_undegraded":"2024-03-09T15:36:54.855985+0000","last_fullsized":"2024-03-09T15:36:54.855985+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:54.855723+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:54.855723+0000","last_clean_scrub_stamp":"2024-03-09T15:36:54.855723+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:25:06.455116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":16384,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1","version":"261'6","reported_seq":73,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.249215+0000","last_change":"2024-03-09T15:36:15.249215+0000","last_active":"2024-03-09T15:36:15.249215+0000","last_peered":"2024-03-09T15:36:15.249215+0000","last_clean":"2024-03-09T15:36:15.249215+0000","last_became_active":"2024-03-09T14:46:18.757694+0000","last_became_peered":"2024-03-09T14:46:18.757694+0000","last_unstale":"2024-03-09T15:36:15.249215+0000","last_undegraded":"2024-03-09T15:36:15.249215+0000","last_fullsized":"2024-03-09T15:36:15.249215+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:15.248968+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:15.248968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.248968+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:50:59.398908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":4098,"num_write":9,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.2","version":"258'20","reported_seq":119,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.118799+0000","last_change":"2024-03-09T15:36:13.118868+0000","last_active":"2024-03-09T15:36:13.118799+0000","last_peered":"2024-03-09T15:36:13.118799+0000","last_clean":"2024-03-09T15:36:13.118799+0000","last_became_active":"2024-03-09T14:39:13.264203+0000","last_became_peered":"2024-03-09T14:39:13.264203+0000","last_unstale":"2024-03-09T15:36:13.118799+0000","last_undegraded":"2024-03-09T15:36:13.118799+0000","last_fullsized":"2024-03-09T15:36:13.118799+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'20","last_scrub_stamp":"2024-03-09T15:36:13.118589+0000","last_deep_scrub":"258'20","last_deep_scrub_stamp":"2024-03-09T15:36:13.118589+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.118589+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:29:35.065434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":65,"num_write":31,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.4","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:09.986900+0000","last_change":"2024-03-09T15:37:09.986955+0000","last_active":"2024-03-09T15:37:09.986900+0000","last_peered":"2024-03-09T15:37:09.986900+0000","last_clean":"2024-03-09T15:37:09.986900+0000","last_became_active":"2024-03-09T15:09:50.066222+0000","last_became_peered":"2024-03-09T15:09:50.066222+0000","last_unstale":"2024-03-09T15:37:09.986900+0000","last_undegraded":"2024-03-09T15:37:09.986900+0000","last_fullsized":"2024-03-09T15:37:09.986900+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:09.986702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:09.986702+0000","last_clean_scrub_stamp":"2024-03-09T15:37:09.986702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:36:04.063534+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.3","version":"0'0","reported_seq":55,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.342569+0000","last_change":"2024-03-09T15:36:49.342621+0000","last_active":"2024-03-09T15:36:49.342569+0000","last_peered":"2024-03-09T15:36:49.342569+0000","last_clean":"2024-03-09T15:36:49.342569+0000","last_became_active":"2024-03-09T14:46:26.826105+0000","last_became_peered":"2024-03-09T14:46:26.826105+0000","last_unstale":"2024-03-09T15:36:49.342569+0000","last_undegraded":"2024-03-09T15:36:49.342569+0000","last_fullsized":"2024-03-09T15:36:49.342569+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:49.342369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:49.342369+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.342369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:03:37.308549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.2","version":"0'0","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.469191+0000","last_change":"2024-03-09T15:36:12.469191+0000","last_active":"2024-03-09T15:36:12.469191+0000","last_peered":"2024-03-09T15:36:12.469191+0000","last_clean":"2024-03-09T15:36:12.469191+0000","last_became_active":"2024-03-09T14:46:18.749091+0000","last_became_peered":"2024-03-09T14:46:18.749091+0000","last_unstale":"2024-03-09T15:36:12.469191+0000","last_undegraded":"2024-03-09T15:36:12.469191+0000","last_fullsized":"2024-03-09T15:36:12.469191+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:12.468992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:12.468992+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.468992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:06:25.628906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1","version":"261'25","reported_seq":134,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.079889+0000","last_change":"2024-03-09T15:36:19.079952+0000","last_active":"2024-03-09T15:36:19.079889+0000","last_peered":"2024-03-09T15:36:19.079889+0000","last_clean":"2024-03-09T15:36:19.079889+0000","last_became_active":"2024-03-09T14:39:13.255589+0000","last_became_peered":"2024-03-09T14:39:13.255589+0000","last_unstale":"2024-03-09T15:36:19.079889+0000","last_undegraded":"2024-03-09T15:36:19.079889+0000","last_fullsized":"2024-03-09T15:36:19.079889+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'25","last_scrub_stamp":"2024-03-09T15:36:19.079692+0000","last_deep_scrub":"261'25","last_deep_scrub_stamp":"2024-03-09T15:36:19.079692+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.079692+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:25:37.071627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":79,"num_write":35,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.5","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:10.238864+0000","last_change":"2024-03-09T15:37:10.239023+0000","last_active":"2024-03-09T15:37:10.238864+0000","last_peered":"2024-03-09T15:37:10.238864+0000","last_clean":"2024-03-09T15:37:10.238864+0000","last_became_active":"2024-03-09T15:09:50.083064+0000","last_became_peered":"2024-03-09T15:09:50.083064+0000","last_unstale":"2024-03-09T15:37:10.238864+0000","last_undegraded":"2024-03-09T15:37:10.238864+0000","last_fullsized":"2024-03-09T15:37:10.238864+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:10.238633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:10.238633+0000","last_clean_scrub_stamp":"2024-03-09T15:37:10.238633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:36:45.349039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.2","version":"258'3","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:59.604506+0000","last_change":"2024-03-09T15:36:59.604570+0000","last_active":"2024-03-09T15:36:59.604506+0000","last_peered":"2024-03-09T15:36:59.604506+0000","last_clean":"2024-03-09T15:36:59.604506+0000","last_became_active":"2024-03-09T14:46:26.807746+0000","last_became_peered":"2024-03-09T14:46:26.807746+0000","last_unstale":"2024-03-09T15:36:59.604506+0000","last_undegraded":"2024-03-09T15:36:59.604506+0000","last_fullsized":"2024-03-09T15:36:59.604506+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:59.604307+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:59.604307+0000","last_clean_scrub_stamp":"2024-03-09T15:36:59.604307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:45:47.163987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1024,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.3","version":"0'0","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.030314+0000","last_change":"2024-03-09T15:36:18.030314+0000","last_active":"2024-03-09T15:36:18.030314+0000","last_peered":"2024-03-09T15:36:18.030314+0000","last_clean":"2024-03-09T15:36:18.030314+0000","last_became_active":"2024-03-09T14:46:18.765077+0000","last_became_peered":"2024-03-09T14:46:18.765077+0000","last_unstale":"2024-03-09T15:36:18.030314+0000","last_undegraded":"2024-03-09T15:36:18.030314+0000","last_fullsized":"2024-03-09T15:36:18.030314+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:18.030067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:18.030067+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.030067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:44:34.102574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.0","version":"250'25","reported_seq":139,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.180831+0000","last_change":"2024-03-09T15:36:20.180899+0000","last_active":"2024-03-09T15:36:20.180831+0000","last_peered":"2024-03-09T15:36:20.180831+0000","last_clean":"2024-03-09T15:36:20.180831+0000","last_became_active":"2024-03-09T14:39:13.268553+0000","last_became_peered":"2024-03-09T14:39:13.268553+0000","last_unstale":"2024-03-09T15:36:20.180831+0000","last_undegraded":"2024-03-09T15:36:20.180831+0000","last_fullsized":"2024-03-09T15:36:20.180831+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-03-09T15:36:20.180625+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-03-09T15:36:20.180625+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.180625+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:48:22.579836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":97,"num_write":32,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"261'118","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.121631+0000","last_change":"2024-03-09T15:36:48.121725+0000","last_active":"2024-03-09T15:36:48.121631+0000","last_peered":"2024-03-09T15:36:48.121631+0000","last_clean":"2024-03-09T15:36:48.121631+0000","last_became_active":"2024-03-09T14:28:32.339785+0000","last_became_peered":"2024-03-09T14:28:32.339785+0000","last_unstale":"2024-03-09T15:36:48.121631+0000","last_undegraded":"2024-03-09T15:36:48.121631+0000","last_fullsized":"2024-03-09T15:36:48.121631+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'118","last_scrub_stamp":"2024-03-09T15:36:48.121408+0000","last_deep_scrub":"261'118","last_deep_scrub_stamp":"2024-03-09T15:36:48.121408+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.121408+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:07:57.417207+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":130,"num_read_kb":5377,"num_write":373,"num_write_kb":16955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"261'146","reported_seq":696,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.743200+0000","last_change":"2024-03-09T15:36:31.743261+0000","last_active":"2024-03-09T15:36:31.743200+0000","last_peered":"2024-03-09T15:36:31.743200+0000","last_clean":"2024-03-09T15:36:31.743200+0000","last_became_active":"2024-03-09T14:28:34.303980+0000","last_became_peered":"2024-03-09T14:28:34.303980+0000","last_unstale":"2024-03-09T15:36:31.743200+0000","last_undegraded":"2024-03-09T15:36:31.743200+0000","last_fullsized":"2024-03-09T15:36:31.743200+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'146","last_scrub_stamp":"2024-03-09T15:36:31.743004+0000","last_deep_scrub":"261'146","last_deep_scrub_stamp":"2024-03-09T15:36:31.743004+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.743004+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:05:36.454535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":546,"num_read_kb":417,"num_write":228,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"261'143","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.213927+0000","last_change":"2024-03-09T15:36:45.214056+0000","last_active":"2024-03-09T15:36:45.213927+0000","last_peered":"2024-03-09T15:36:45.213927+0000","last_clean":"2024-03-09T15:36:45.213927+0000","last_became_active":"2024-03-09T14:28:32.306840+0000","last_became_peered":"2024-03-09T14:28:32.306840+0000","last_unstale":"2024-03-09T15:36:45.213927+0000","last_undegraded":"2024-03-09T15:36:45.213927+0000","last_fullsized":"2024-03-09T15:36:45.213927+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'143","last_scrub_stamp":"2024-03-09T15:36:45.213735+0000","last_deep_scrub":"261'143","last_deep_scrub_stamp":"2024-03-09T15:36:45.213735+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.213735+0000","objects_scrubbed":8,"log_size":43,"log_dups_size":100,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:08:41.879574+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3389647,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":156,"num_read_kb":3004,"num_write":523,"num_write_kb":19720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"261'90","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.206902+0000","last_change":"2024-03-09T15:36:40.206967+0000","last_active":"2024-03-09T15:36:40.206902+0000","last_peered":"2024-03-09T15:36:40.206902+0000","last_clean":"2024-03-09T15:36:40.206902+0000","last_became_active":"2024-03-09T14:28:34.324053+0000","last_became_peered":"2024-03-09T14:28:34.324053+0000","last_unstale":"2024-03-09T15:36:40.206902+0000","last_undegraded":"2024-03-09T15:36:40.206902+0000","last_fullsized":"2024-03-09T15:36:40.206902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'90","last_scrub_stamp":"2024-03-09T15:36:40.206700+0000","last_deep_scrub":"261'90","last_deep_scrub_stamp":"2024-03-09T15:36:40.206700+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.206700+0000","objects_scrubbed":1,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:36:21.366820+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":372,"num_read_kb":300,"num_write":141,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3d","version":"261'161","reported_seq":730,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:57.183684+0000","last_change":"2024-03-09T15:36:57.183774+0000","last_active":"2024-03-09T15:36:57.183684+0000","last_peered":"2024-03-09T15:36:57.183684+0000","last_clean":"2024-03-09T15:36:57.183684+0000","last_became_active":"2024-03-09T14:28:32.286966+0000","last_became_peered":"2024-03-09T14:28:32.286966+0000","last_unstale":"2024-03-09T15:36:57.183684+0000","last_undegraded":"2024-03-09T15:36:57.183684+0000","last_fullsized":"2024-03-09T15:36:57.183684+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'161","last_scrub_stamp":"2024-03-09T15:36:57.183471+0000","last_deep_scrub":"261'161","last_deep_scrub_stamp":"2024-03-09T15:36:57.183471+0000","last_clean_scrub_stamp":"2024-03-09T15:36:57.183471+0000","objects_scrubbed":3,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:24:18.904914+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":14171,"num_write":610,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"261'157","reported_seq":727,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.378778+0000","last_change":"2024-03-09T15:36:46.378837+0000","last_active":"2024-03-09T15:36:46.378778+0000","last_peered":"2024-03-09T15:36:46.378778+0000","last_clean":"2024-03-09T15:36:46.378778+0000","last_became_active":"2024-03-09T14:28:34.306764+0000","last_became_peered":"2024-03-09T14:28:34.306764+0000","last_unstale":"2024-03-09T15:36:46.378778+0000","last_undegraded":"2024-03-09T15:36:46.378778+0000","last_fullsized":"2024-03-09T15:36:46.378778+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'157","last_scrub_stamp":"2024-03-09T15:36:46.378588+0000","last_deep_scrub":"261'157","last_deep_scrub_stamp":"2024-03-09T15:36:46.378588+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.378588+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:55:21.945660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":481,"num_write":261,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3c","version":"261'196","reported_seq":798,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.832727+0000","last_change":"2024-03-09T15:36:51.832884+0000","last_active":"2024-03-09T15:36:51.832727+0000","last_peered":"2024-03-09T15:36:51.832727+0000","last_clean":"2024-03-09T15:36:51.832727+0000","last_became_active":"2024-03-09T14:28:32.305615+0000","last_became_peered":"2024-03-09T14:28:32.305615+0000","last_unstale":"2024-03-09T15:36:51.832727+0000","last_undegraded":"2024-03-09T15:36:51.832727+0000","last_fullsized":"2024-03-09T15:36:51.832727+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'196","last_scrub_stamp":"2024-03-09T15:36:51.832524+0000","last_deep_scrub":"261'196","last_deep_scrub_stamp":"2024-03-09T15:36:51.832524+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.832524+0000","objects_scrubbed":6,"log_size":96,"log_dups_size":100,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:08:41.077484+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5326894,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":256,"num_read_kb":28937,"num_write":797,"num_write_kb":45835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"261'157","reported_seq":742,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.270196+0000","last_change":"2024-03-09T15:36:41.270258+0000","last_active":"2024-03-09T15:36:41.270196+0000","last_peered":"2024-03-09T15:36:41.270196+0000","last_clean":"2024-03-09T15:36:41.270196+0000","last_became_active":"2024-03-09T14:28:34.316500+0000","last_became_peered":"2024-03-09T14:28:34.316500+0000","last_unstale":"2024-03-09T15:36:41.270196+0000","last_undegraded":"2024-03-09T15:36:41.270196+0000","last_fullsized":"2024-03-09T15:36:41.270196+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'157","last_scrub_stamp":"2024-03-09T15:36:41.269997+0000","last_deep_scrub":"261'157","last_deep_scrub_stamp":"2024-03-09T15:36:41.269997+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.269997+0000","objects_scrubbed":1,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:09:23.890486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":622,"num_read_kb":488,"num_write":257,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3b","version":"261'156","reported_seq":827,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:59.106502+0000","last_change":"2024-03-09T15:36:59.106654+0000","last_active":"2024-03-09T15:36:59.106502+0000","last_peered":"2024-03-09T15:36:59.106502+0000","last_clean":"2024-03-09T15:36:59.106502+0000","last_became_active":"2024-03-09T14:28:32.314923+0000","last_became_peered":"2024-03-09T14:28:32.314923+0000","last_unstale":"2024-03-09T15:36:59.106502+0000","last_undegraded":"2024-03-09T15:36:59.106502+0000","last_fullsized":"2024-03-09T15:36:59.106502+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'156","last_scrub_stamp":"2024-03-09T15:36:59.106291+0000","last_deep_scrub":"261'156","last_deep_scrub_stamp":"2024-03-09T15:36:59.106291+0000","last_clean_scrub_stamp":"2024-03-09T15:36:59.106291+0000","objects_scrubbed":6,"log_size":56,"log_dups_size":100,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:56:41.396083+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1917997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":297,"num_read_kb":17282,"num_write":517,"num_write_kb":14747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3c","version":"261'124","reported_seq":716,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.314515+0000","last_change":"2024-03-09T15:36:35.314515+0000","last_active":"2024-03-09T15:36:35.314515+0000","last_peered":"2024-03-09T15:36:35.314515+0000","last_clean":"2024-03-09T15:36:35.314515+0000","last_became_active":"2024-03-09T14:28:34.314617+0000","last_became_peered":"2024-03-09T14:28:34.314617+0000","last_unstale":"2024-03-09T15:36:35.314515+0000","last_undegraded":"2024-03-09T15:36:35.314515+0000","last_fullsized":"2024-03-09T15:36:35.314515+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'124","last_scrub_stamp":"2024-03-09T15:36:35.314270+0000","last_deep_scrub":"261'124","last_deep_scrub_stamp":"2024-03-09T15:36:35.314270+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.314270+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:54:11.780427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":493,"num_read_kb":395,"num_write":185,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3a","version":"261'150","reported_seq":693,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.166015+0000","last_change":"2024-03-09T15:36:14.166015+0000","last_active":"2024-03-09T15:36:14.166015+0000","last_peered":"2024-03-09T15:36:14.166015+0000","last_clean":"2024-03-09T15:36:14.166015+0000","last_became_active":"2024-03-09T14:29:06.433145+0000","last_became_peered":"2024-03-09T14:29:06.433145+0000","last_unstale":"2024-03-09T15:36:14.166015+0000","last_undegraded":"2024-03-09T15:36:14.166015+0000","last_fullsized":"2024-03-09T15:36:14.166015+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'150","last_scrub_stamp":"2024-03-09T15:36:14.165814+0000","last_deep_scrub":"261'150","last_deep_scrub_stamp":"2024-03-09T15:36:14.165814+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.165814+0000","objects_scrubbed":3,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:29:26.186154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":6443,"num_write":604,"num_write_kb":33049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"261'275","reported_seq":949,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.257093+0000","last_change":"2024-03-09T15:36:38.257158+0000","last_active":"2024-03-09T15:36:38.257093+0000","last_peered":"2024-03-09T15:36:38.257093+0000","last_clean":"2024-03-09T15:36:38.257093+0000","last_became_active":"2024-03-09T14:28:34.310343+0000","last_became_peered":"2024-03-09T14:28:34.310343+0000","last_unstale":"2024-03-09T15:36:38.257093+0000","last_undegraded":"2024-03-09T15:36:38.257093+0000","last_fullsized":"2024-03-09T15:36:38.257093+0000","mapping_epoch":22,"log_start":"247'200","ondisk_log_start":"247'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'275","last_scrub_stamp":"2024-03-09T15:36:38.256877+0000","last_deep_scrub":"261'275","last_deep_scrub_stamp":"2024-03-09T15:36:38.256877+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.256877+0000","objects_scrubbed":1,"log_size":75,"log_dups_size":200,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:40:32.582105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1205,"num_read_kb":1028,"num_write":418,"num_write_kb":253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25677,"num_omap_keys":101,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"261'137","reported_seq":702,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:58.112254+0000","last_change":"2024-03-09T15:36:58.112328+0000","last_active":"2024-03-09T15:36:58.112254+0000","last_peered":"2024-03-09T15:36:58.112254+0000","last_clean":"2024-03-09T15:36:58.112254+0000","last_became_active":"2024-03-09T14:28:32.314258+0000","last_became_peered":"2024-03-09T14:28:32.314258+0000","last_unstale":"2024-03-09T15:36:58.112254+0000","last_undegraded":"2024-03-09T15:36:58.112254+0000","last_fullsized":"2024-03-09T15:36:58.112254+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'137","last_scrub_stamp":"2024-03-09T15:36:58.112062+0000","last_deep_scrub":"261'137","last_deep_scrub_stamp":"2024-03-09T15:36:58.112062+0000","last_clean_scrub_stamp":"2024-03-09T15:36:58.112062+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":100,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:35:31.896371+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":169,"num_read_kb":17581,"num_write":466,"num_write_kb":22929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3e","version":"261'44","reported_seq":619,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.733708+0000","last_change":"2024-03-09T15:36:29.733770+0000","last_active":"2024-03-09T15:36:29.733708+0000","last_peered":"2024-03-09T15:36:29.733708+0000","last_clean":"2024-03-09T15:36:29.733708+0000","last_became_active":"2024-03-09T14:28:34.322206+0000","last_became_peered":"2024-03-09T14:28:34.322206+0000","last_unstale":"2024-03-09T15:36:29.733708+0000","last_undegraded":"2024-03-09T15:36:29.733708+0000","last_fullsized":"2024-03-09T15:36:29.733708+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:29.733482+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:29.733482+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.733482+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:58.948789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":201,"num_read_kb":171,"num_write":60,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"261'155","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:04.787270+0000","last_change":"2024-03-09T15:37:04.787354+0000","last_active":"2024-03-09T15:37:04.787270+0000","last_peered":"2024-03-09T15:37:04.787270+0000","last_clean":"2024-03-09T15:37:04.787270+0000","last_became_active":"2024-03-09T14:28:32.308522+0000","last_became_peered":"2024-03-09T14:28:32.308522+0000","last_unstale":"2024-03-09T15:37:04.787270+0000","last_undegraded":"2024-03-09T15:37:04.787270+0000","last_fullsized":"2024-03-09T15:37:04.787270+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'155","last_scrub_stamp":"2024-03-09T15:37:04.787076+0000","last_deep_scrub":"261'155","last_deep_scrub_stamp":"2024-03-09T15:37:04.787076+0000","last_clean_scrub_stamp":"2024-03-09T15:37:04.787076+0000","objects_scrubbed":5,"log_size":55,"log_dups_size":100,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:38.097065+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525396,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":175,"num_read_kb":14130,"num_write":546,"num_write_kb":36844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"261'52","reported_seq":612,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.219275+0000","last_change":"2024-03-09T15:36:34.219343+0000","last_active":"2024-03-09T15:36:34.219275+0000","last_peered":"2024-03-09T15:36:34.219275+0000","last_clean":"2024-03-09T15:36:34.219275+0000","last_became_active":"2024-03-09T14:28:34.320646+0000","last_became_peered":"2024-03-09T14:28:34.320646+0000","last_unstale":"2024-03-09T15:36:34.219275+0000","last_undegraded":"2024-03-09T15:36:34.219275+0000","last_fullsized":"2024-03-09T15:36:34.219275+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'52","last_scrub_stamp":"2024-03-09T15:36:34.219014+0000","last_deep_scrub":"261'52","last_deep_scrub_stamp":"2024-03-09T15:36:34.219014+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.219014+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:48:47.044377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":174,"num_write":79,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"261'158","reported_seq":752,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:54.501584+0000","last_change":"2024-03-09T15:36:54.501668+0000","last_active":"2024-03-09T15:36:54.501584+0000","last_peered":"2024-03-09T15:36:54.501584+0000","last_clean":"2024-03-09T15:36:54.501584+0000","last_became_active":"2024-03-09T14:28:34.306592+0000","last_became_peered":"2024-03-09T14:28:34.306592+0000","last_unstale":"2024-03-09T15:36:54.501584+0000","last_undegraded":"2024-03-09T15:36:54.501584+0000","last_fullsized":"2024-03-09T15:36:54.501584+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'158","last_scrub_stamp":"2024-03-09T15:36:54.501377+0000","last_deep_scrub":"261'158","last_deep_scrub_stamp":"2024-03-09T15:36:54.501377+0000","last_clean_scrub_stamp":"2024-03-09T15:36:54.501377+0000","objects_scrubbed":1,"log_size":58,"log_dups_size":100,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:42:48.686833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":599,"num_read_kb":473,"num_write":232,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"261'165","reported_seq":727,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.220340+0000","last_change":"2024-03-09T15:36:16.220340+0000","last_active":"2024-03-09T15:36:16.220340+0000","last_peered":"2024-03-09T15:36:16.220340+0000","last_clean":"2024-03-09T15:36:16.220340+0000","last_became_active":"2024-03-09T14:29:06.438523+0000","last_became_peered":"2024-03-09T14:29:06.438523+0000","last_unstale":"2024-03-09T15:36:16.220340+0000","last_undegraded":"2024-03-09T15:36:16.220340+0000","last_fullsized":"2024-03-09T15:36:16.220340+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:16.219968+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:16.219968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.219968+0000","objects_scrubbed":4,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:42:29.990282+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728801,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":293,"num_read_kb":1034,"num_write":597,"num_write_kb":27895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.31","version":"261'88","reported_seq":652,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.145113+0000","last_change":"2024-03-09T15:36:46.145171+0000","last_active":"2024-03-09T15:36:46.145113+0000","last_peered":"2024-03-09T15:36:46.145113+0000","last_clean":"2024-03-09T15:36:46.145113+0000","last_became_active":"2024-03-09T14:28:34.321352+0000","last_became_peered":"2024-03-09T14:28:34.321352+0000","last_unstale":"2024-03-09T15:36:46.145113+0000","last_undegraded":"2024-03-09T15:36:46.145113+0000","last_fullsized":"2024-03-09T15:36:46.145113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'88","last_scrub_stamp":"2024-03-09T15:36:46.144913+0000","last_deep_scrub":"261'88","last_deep_scrub_stamp":"2024-03-09T15:36:46.144913+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.144913+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:37:05.731149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":342,"num_read_kb":270,"num_write":126,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"261'133","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.262628+0000","last_change":"2024-03-09T15:36:43.262702+0000","last_active":"2024-03-09T15:36:43.262628+0000","last_peered":"2024-03-09T15:36:43.262628+0000","last_clean":"2024-03-09T15:36:43.262628+0000","last_became_active":"2024-03-09T14:28:32.334208+0000","last_became_peered":"2024-03-09T14:28:32.334208+0000","last_unstale":"2024-03-09T15:36:43.262628+0000","last_undegraded":"2024-03-09T15:36:43.262628+0000","last_fullsized":"2024-03-09T15:36:43.262628+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'133","last_scrub_stamp":"2024-03-09T15:36:43.262446+0000","last_deep_scrub":"261'133","last_deep_scrub_stamp":"2024-03-09T15:36:43.262446+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.262446+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:21:20.422366+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262185,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":143,"num_read_kb":128,"num_write":408,"num_write_kb":22112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"261'62","reported_seq":618,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.784398+0000","last_change":"2024-03-09T15:36:35.784465+0000","last_active":"2024-03-09T15:36:35.784398+0000","last_peered":"2024-03-09T15:36:35.784398+0000","last_clean":"2024-03-09T15:36:35.784398+0000","last_became_active":"2024-03-09T14:28:34.298428+0000","last_became_peered":"2024-03-09T14:28:34.298428+0000","last_unstale":"2024-03-09T15:36:35.784398+0000","last_undegraded":"2024-03-09T15:36:35.784398+0000","last_fullsized":"2024-03-09T15:36:35.784398+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'62","last_scrub_stamp":"2024-03-09T15:36:35.784200+0000","last_deep_scrub":"261'62","last_deep_scrub_stamp":"2024-03-09T15:36:35.784200+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.784200+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:32:34.722288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":202,"num_write":92,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"261'118","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.242144+0000","last_change":"2024-03-09T15:36:39.242234+0000","last_active":"2024-03-09T15:36:39.242144+0000","last_peered":"2024-03-09T15:36:39.242144+0000","last_clean":"2024-03-09T15:36:39.242144+0000","last_became_active":"2024-03-09T14:28:32.318963+0000","last_became_peered":"2024-03-09T14:28:32.318963+0000","last_unstale":"2024-03-09T15:36:39.242144+0000","last_undegraded":"2024-03-09T15:36:39.242144+0000","last_fullsized":"2024-03-09T15:36:39.242144+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'118","last_scrub_stamp":"2024-03-09T15:36:39.241932+0000","last_deep_scrub":"261'118","last_deep_scrub_stamp":"2024-03-09T15:36:39.241932+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.241932+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:06:46.492058+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170436,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":164,"num_read_kb":1700,"num_write":435,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.33","version":"261'102","reported_seq":673,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.201686+0000","last_change":"2024-03-09T15:36:33.201745+0000","last_active":"2024-03-09T15:36:33.201686+0000","last_peered":"2024-03-09T15:36:33.201686+0000","last_clean":"2024-03-09T15:36:33.201686+0000","last_became_active":"2024-03-09T14:28:34.320331+0000","last_became_peered":"2024-03-09T14:28:34.320331+0000","last_unstale":"2024-03-09T15:36:33.201686+0000","last_undegraded":"2024-03-09T15:36:33.201686+0000","last_fullsized":"2024-03-09T15:36:33.201686+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'102","last_scrub_stamp":"2024-03-09T15:36:33.201501+0000","last_deep_scrub":"261'102","last_deep_scrub_stamp":"2024-03-09T15:36:33.201501+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.201501+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:07:30.758372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":320,"num_write":144,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"261'136","reported_seq":698,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.465189+0000","last_change":"2024-03-09T15:36:51.465274+0000","last_active":"2024-03-09T15:36:51.465189+0000","last_peered":"2024-03-09T15:36:51.465189+0000","last_clean":"2024-03-09T15:36:51.465189+0000","last_became_active":"2024-03-09T14:28:32.337934+0000","last_became_peered":"2024-03-09T14:28:32.337934+0000","last_unstale":"2024-03-09T15:36:51.465189+0000","last_undegraded":"2024-03-09T15:36:51.465189+0000","last_fullsized":"2024-03-09T15:36:51.465189+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'136","last_scrub_stamp":"2024-03-09T15:36:51.465001+0000","last_deep_scrub":"261'136","last_deep_scrub_stamp":"2024-03-09T15:36:51.465001+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.465001+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:36:13.771727+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263250,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":163,"num_read_kb":12763,"num_write":465,"num_write_kb":21372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"261'69","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.229898+0000","last_change":"2024-03-09T15:36:31.229983+0000","last_active":"2024-03-09T15:36:31.229898+0000","last_peered":"2024-03-09T15:36:31.229898+0000","last_clean":"2024-03-09T15:36:31.229898+0000","last_became_active":"2024-03-09T14:28:34.310703+0000","last_became_peered":"2024-03-09T14:28:34.310703+0000","last_unstale":"2024-03-09T15:36:31.229898+0000","last_undegraded":"2024-03-09T15:36:31.229898+0000","last_fullsized":"2024-03-09T15:36:31.229898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'69","last_scrub_stamp":"2024-03-09T15:36:31.229701+0000","last_deep_scrub":"261'69","last_deep_scrub_stamp":"2024-03-09T15:36:31.229701+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.229701+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:52:58.555942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":272,"num_read_kb":220,"num_write":109,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"261'162","reported_seq":738,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:56.774266+0000","last_change":"2024-03-09T15:36:56.774342+0000","last_active":"2024-03-09T15:36:56.774266+0000","last_peered":"2024-03-09T15:36:56.774266+0000","last_clean":"2024-03-09T15:36:56.774266+0000","last_became_active":"2024-03-09T14:28:32.336636+0000","last_became_peered":"2024-03-09T14:28:32.336636+0000","last_unstale":"2024-03-09T15:36:56.774266+0000","last_undegraded":"2024-03-09T15:36:56.774266+0000","last_fullsized":"2024-03-09T15:36:56.774266+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'162","last_scrub_stamp":"2024-03-09T15:36:56.774051+0000","last_deep_scrub":"261'162","last_deep_scrub_stamp":"2024-03-09T15:36:56.774051+0000","last_clean_scrub_stamp":"2024-03-09T15:36:56.774051+0000","objects_scrubbed":5,"log_size":62,"log_dups_size":100,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:51:33.272809+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262241,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":231,"num_read_kb":12490,"num_write":578,"num_write_kb":28462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"261'87","reported_seq":647,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.123887+0000","last_change":"2024-03-09T15:36:33.123953+0000","last_active":"2024-03-09T15:36:33.123887+0000","last_peered":"2024-03-09T15:36:33.123887+0000","last_clean":"2024-03-09T15:36:33.123887+0000","last_became_active":"2024-03-09T14:28:34.314665+0000","last_became_peered":"2024-03-09T14:28:34.314665+0000","last_unstale":"2024-03-09T15:36:33.123887+0000","last_undegraded":"2024-03-09T15:36:33.123887+0000","last_fullsized":"2024-03-09T15:36:33.123887+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'87","last_scrub_stamp":"2024-03-09T15:36:33.123670+0000","last_deep_scrub":"261'87","last_deep_scrub_stamp":"2024-03-09T15:36:33.123670+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.123670+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:07:39.698152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":335,"num_read_kb":265,"num_write":124,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"261'122","reported_seq":693,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:01.146225+0000","last_change":"2024-03-09T15:37:01.146373+0000","last_active":"2024-03-09T15:37:01.146225+0000","last_peered":"2024-03-09T15:37:01.146225+0000","last_clean":"2024-03-09T15:37:01.146225+0000","last_became_active":"2024-03-09T14:28:32.298428+0000","last_became_peered":"2024-03-09T14:28:32.298428+0000","last_unstale":"2024-03-09T15:37:01.146225+0000","last_undegraded":"2024-03-09T15:37:01.146225+0000","last_fullsized":"2024-03-09T15:37:01.146225+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'122","last_scrub_stamp":"2024-03-09T15:37:01.145980+0000","last_deep_scrub":"261'122","last_deep_scrub_stamp":"2024-03-09T15:37:01.145980+0000","last_clean_scrub_stamp":"2024-03-09T15:37:01.145980+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:41.749323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":191,"num_read_kb":31150,"num_write":464,"num_write_kb":37586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"261'76","reported_seq":591,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.170281+0000","last_change":"2024-03-09T15:36:18.170281+0000","last_active":"2024-03-09T15:36:18.170281+0000","last_peered":"2024-03-09T15:36:18.170281+0000","last_clean":"2024-03-09T15:36:18.170281+0000","last_became_active":"2024-03-09T14:28:42.403553+0000","last_became_peered":"2024-03-09T14:28:42.403553+0000","last_unstale":"2024-03-09T15:36:18.170281+0000","last_undegraded":"2024-03-09T15:36:18.170281+0000","last_fullsized":"2024-03-09T15:36:18.170281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'76","last_scrub_stamp":"2024-03-09T15:36:18.170018+0000","last_deep_scrub":"261'76","last_deep_scrub_stamp":"2024-03-09T15:36:18.170018+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.170018+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:55:35.040311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":84,"num_write":184,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"261'98","reported_seq":674,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.480604+0000","last_change":"2024-03-09T15:36:20.480604+0000","last_active":"2024-03-09T15:36:20.480604+0000","last_peered":"2024-03-09T15:36:20.480604+0000","last_clean":"2024-03-09T15:36:20.480604+0000","last_became_active":"2024-03-09T14:29:06.439359+0000","last_became_peered":"2024-03-09T14:29:06.439359+0000","last_unstale":"2024-03-09T15:36:20.480604+0000","last_undegraded":"2024-03-09T15:36:20.480604+0000","last_fullsized":"2024-03-09T15:36:20.480604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'98","last_scrub_stamp":"2024-03-09T15:36:20.480339+0000","last_deep_scrub":"261'98","last_deep_scrub_stamp":"2024-03-09T15:36:20.480339+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.480339+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:32:16.216352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":413,"num_read_kb":333,"num_write":149,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"261'172","reported_seq":803,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.158265+0000","last_change":"2024-03-09T15:36:50.158375+0000","last_active":"2024-03-09T15:36:50.158265+0000","last_peered":"2024-03-09T15:36:50.158265+0000","last_clean":"2024-03-09T15:36:50.158265+0000","last_became_active":"2024-03-09T14:28:32.329457+0000","last_became_peered":"2024-03-09T14:28:32.329457+0000","last_unstale":"2024-03-09T15:36:50.158265+0000","last_undegraded":"2024-03-09T15:36:50.158265+0000","last_fullsized":"2024-03-09T15:36:50.158265+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'172","last_scrub_stamp":"2024-03-09T15:36:50.157939+0000","last_deep_scrub":"261'172","last_deep_scrub_stamp":"2024-03-09T15:36:50.157939+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.157939+0000","objects_scrubbed":7,"log_size":72,"log_dups_size":100,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:28.447473+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506100,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":263,"num_read_kb":141328,"num_write":603,"num_write_kb":37022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.929658+0000","last_change":"2024-03-09T15:36:40.929714+0000","last_active":"2024-03-09T15:36:40.929658+0000","last_peered":"2024-03-09T15:36:40.929658+0000","last_clean":"2024-03-09T15:36:40.929658+0000","last_became_active":"2024-03-09T14:28:36.320603+0000","last_became_peered":"2024-03-09T14:28:36.320603+0000","last_unstale":"2024-03-09T15:36:40.929658+0000","last_undegraded":"2024-03-09T15:36:40.929658+0000","last_fullsized":"2024-03-09T15:36:40.929658+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:40.929451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:40.929451+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.929451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:14:31.906125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"261'233","reported_seq":853,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:53.265716+0000","last_change":"2024-03-09T15:36:53.265789+0000","last_active":"2024-03-09T15:36:53.265716+0000","last_peered":"2024-03-09T15:36:53.265716+0000","last_clean":"2024-03-09T15:36:53.265716+0000","last_became_active":"2024-03-09T14:28:38.326826+0000","last_became_peered":"2024-03-09T14:28:38.326826+0000","last_unstale":"2024-03-09T15:36:53.265716+0000","last_undegraded":"2024-03-09T15:36:53.265716+0000","last_fullsized":"2024-03-09T15:36:53.265716+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'233","last_scrub_stamp":"2024-03-09T15:36:53.265513+0000","last_deep_scrub":"261'233","last_deep_scrub_stamp":"2024-03-09T15:36:53.265513+0000","last_clean_scrub_stamp":"2024-03-09T15:36:53.265513+0000","objects_scrubbed":4,"log_size":33,"log_dups_size":200,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:52:51.677937+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11101,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":365,"num_read_kb":621,"num_write":250,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.268381+0000","last_change":"2024-03-09T15:36:36.268437+0000","last_active":"2024-03-09T15:36:36.268381+0000","last_peered":"2024-03-09T15:36:36.268381+0000","last_clean":"2024-03-09T15:36:36.268381+0000","last_became_active":"2024-03-09T14:28:40.358954+0000","last_became_peered":"2024-03-09T14:28:40.358954+0000","last_unstale":"2024-03-09T15:36:36.268381+0000","last_undegraded":"2024-03-09T15:36:36.268381+0000","last_fullsized":"2024-03-09T15:36:36.268381+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:36.268123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:36.268123+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.268123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:59:04.556053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"261'772","reported_seq":2395,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.694693+0000","last_change":"2024-03-09T15:36:20.694693+0000","last_active":"2024-03-09T15:36:20.694693+0000","last_peered":"2024-03-09T15:36:20.694693+0000","last_clean":"2024-03-09T15:36:20.694693+0000","last_became_active":"2024-03-09T14:28:42.394691+0000","last_became_peered":"2024-03-09T14:28:42.394691+0000","last_unstale":"2024-03-09T15:36:20.694693+0000","last_undegraded":"2024-03-09T15:36:20.694693+0000","last_fullsized":"2024-03-09T15:36:20.694693+0000","mapping_epoch":30,"log_start":"261'700","ondisk_log_start":"261'700","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'772","last_scrub_stamp":"2024-03-09T15:36:20.694425+0000","last_deep_scrub":"261'772","last_deep_scrub_stamp":"2024-03-09T15:36:20.694425+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.694425+0000","objects_scrubbed":3,"log_size":72,"log_dups_size":700,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:30:23.952683+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":612,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2611,"num_read_kb":2601,"num_write":1560,"num_write_kb":569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":376,"num_omap_keys":2,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"261'40","reported_seq":588,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.759957+0000","last_change":"2024-03-09T15:36:32.760017+0000","last_active":"2024-03-09T15:36:32.759957+0000","last_peered":"2024-03-09T15:36:32.759957+0000","last_clean":"2024-03-09T15:36:32.759957+0000","last_became_active":"2024-03-09T14:28:34.324569+0000","last_became_peered":"2024-03-09T14:28:34.324569+0000","last_unstale":"2024-03-09T15:36:32.759957+0000","last_undegraded":"2024-03-09T15:36:32.759957+0000","last_fullsized":"2024-03-09T15:36:32.759957+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'40","last_scrub_stamp":"2024-03-09T15:36:32.759758+0000","last_deep_scrub":"261'40","last_deep_scrub_stamp":"2024-03-09T15:36:32.759758+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.759758+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:33:41.150112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":168,"num_read_kb":136,"num_write":58,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"261'176","reported_seq":756,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:56.238761+0000","last_change":"2024-03-09T15:36:56.238986+0000","last_active":"2024-03-09T15:36:56.238761+0000","last_peered":"2024-03-09T15:36:56.238761+0000","last_clean":"2024-03-09T15:36:56.238761+0000","last_became_active":"2024-03-09T14:28:32.309536+0000","last_became_peered":"2024-03-09T14:28:32.309536+0000","last_unstale":"2024-03-09T15:36:56.238761+0000","last_undegraded":"2024-03-09T15:36:56.238761+0000","last_fullsized":"2024-03-09T15:36:56.238761+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'176","last_scrub_stamp":"2024-03-09T15:36:56.238518+0000","last_deep_scrub":"261'176","last_deep_scrub_stamp":"2024-03-09T15:36:56.238518+0000","last_clean_scrub_stamp":"2024-03-09T15:36:56.238518+0000","objects_scrubbed":7,"log_size":76,"log_dups_size":100,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:49:34.426356+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":29533,"num_write":652,"num_write_kb":30801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:53.738712+0000","last_change":"2024-03-09T15:36:53.738824+0000","last_active":"2024-03-09T15:36:53.738712+0000","last_peered":"2024-03-09T15:36:53.738712+0000","last_clean":"2024-03-09T15:36:53.738712+0000","last_became_active":"2024-03-09T14:28:36.320079+0000","last_became_peered":"2024-03-09T14:28:36.320079+0000","last_unstale":"2024-03-09T15:36:53.738712+0000","last_undegraded":"2024-03-09T15:36:53.738712+0000","last_fullsized":"2024-03-09T15:36:53.738712+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:53.738470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:53.738470+0000","last_clean_scrub_stamp":"2024-03-09T15:36:53.738470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:59:26.919081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"261'90","reported_seq":636,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.122254+0000","last_change":"2024-03-09T15:36:27.122328+0000","last_active":"2024-03-09T15:36:27.122254+0000","last_peered":"2024-03-09T15:36:27.122254+0000","last_clean":"2024-03-09T15:36:27.122254+0000","last_became_active":"2024-03-09T14:28:38.340896+0000","last_became_peered":"2024-03-09T14:28:38.340896+0000","last_unstale":"2024-03-09T15:36:27.122254+0000","last_undegraded":"2024-03-09T15:36:27.122254+0000","last_fullsized":"2024-03-09T15:36:27.122254+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'90","last_scrub_stamp":"2024-03-09T15:36:27.122054+0000","last_deep_scrub":"261'90","last_deep_scrub_stamp":"2024-03-09T15:36:27.122054+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.122054+0000","objects_scrubbed":3,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:23:13.339583+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":101,"num_write":68,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.119178+0000","last_change":"2024-03-09T15:36:23.119232+0000","last_active":"2024-03-09T15:36:23.119178+0000","last_peered":"2024-03-09T15:36:23.119178+0000","last_clean":"2024-03-09T15:36:23.119178+0000","last_became_active":"2024-03-09T14:28:40.370788+0000","last_became_peered":"2024-03-09T14:28:40.370788+0000","last_unstale":"2024-03-09T15:36:23.119178+0000","last_undegraded":"2024-03-09T15:36:23.119178+0000","last_fullsized":"2024-03-09T15:36:23.119178+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:23.118978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:23.118978+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.118978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:04:36.080050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"261'64","reported_seq":576,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.282044+0000","last_change":"2024-03-09T15:36:21.282129+0000","last_active":"2024-03-09T15:36:21.282044+0000","last_peered":"2024-03-09T15:36:21.282044+0000","last_clean":"2024-03-09T15:36:21.282044+0000","last_became_active":"2024-03-09T14:28:42.408738+0000","last_became_peered":"2024-03-09T14:28:42.408738+0000","last_unstale":"2024-03-09T15:36:21.282044+0000","last_undegraded":"2024-03-09T15:36:21.282044+0000","last_fullsized":"2024-03-09T15:36:21.282044+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'64","last_scrub_stamp":"2024-03-09T15:36:21.281839+0000","last_deep_scrub":"261'64","last_deep_scrub_stamp":"2024-03-09T15:36:21.281839+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.281839+0000","objects_scrubbed":5,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:53:36.132862+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":681,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":76,"num_read_kb":75,"num_write":140,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"261'157","reported_seq":755,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.346427+0000","last_change":"2024-03-09T15:36:44.346583+0000","last_active":"2024-03-09T15:36:44.346427+0000","last_peered":"2024-03-09T15:36:44.346427+0000","last_clean":"2024-03-09T15:36:44.346427+0000","last_became_active":"2024-03-09T14:28:34.304892+0000","last_became_peered":"2024-03-09T14:28:34.304892+0000","last_unstale":"2024-03-09T15:36:44.346427+0000","last_undegraded":"2024-03-09T15:36:44.346427+0000","last_fullsized":"2024-03-09T15:36:44.346427+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'157","last_scrub_stamp":"2024-03-09T15:36:44.346198+0000","last_deep_scrub":"261'157","last_deep_scrub_stamp":"2024-03-09T15:36:44.346198+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.346198+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:31:38.007816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":622,"num_read_kb":492,"num_write":239,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"261'164","reported_seq":874,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:02.159926+0000","last_change":"2024-03-09T15:37:02.160136+0000","last_active":"2024-03-09T15:37:02.159926+0000","last_peered":"2024-03-09T15:37:02.159926+0000","last_clean":"2024-03-09T15:37:02.159926+0000","last_became_active":"2024-03-09T14:28:32.318428+0000","last_became_peered":"2024-03-09T14:28:32.318428+0000","last_unstale":"2024-03-09T15:37:02.159926+0000","last_undegraded":"2024-03-09T15:37:02.159926+0000","last_fullsized":"2024-03-09T15:37:02.159926+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'164","last_scrub_stamp":"2024-03-09T15:37:02.159697+0000","last_deep_scrub":"261'164","last_deep_scrub_stamp":"2024-03-09T15:37:02.159697+0000","last_clean_scrub_stamp":"2024-03-09T15:37:02.159697+0000","objects_scrubbed":3,"log_size":64,"log_dups_size":100,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:47:12.429092+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505054,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":544,"num_read_kb":113871,"num_write":619,"num_write_kb":46622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:55.753285+0000","last_change":"2024-03-09T15:36:55.753338+0000","last_active":"2024-03-09T15:36:55.753285+0000","last_peered":"2024-03-09T15:36:55.753285+0000","last_clean":"2024-03-09T15:36:55.753285+0000","last_became_active":"2024-03-09T14:28:36.320095+0000","last_became_peered":"2024-03-09T14:28:36.320095+0000","last_unstale":"2024-03-09T15:36:55.753285+0000","last_undegraded":"2024-03-09T15:36:55.753285+0000","last_fullsized":"2024-03-09T15:36:55.753285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:55.753090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:55.753090+0000","last_clean_scrub_stamp":"2024-03-09T15:36:55.753090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:31:29.223715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"261'108","reported_seq":668,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.165000+0000","last_change":"2024-03-09T15:36:17.165000+0000","last_active":"2024-03-09T15:36:17.165000+0000","last_peered":"2024-03-09T15:36:17.165000+0000","last_clean":"2024-03-09T15:36:17.165000+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:36:17.165000+0000","last_undegraded":"2024-03-09T15:36:17.165000+0000","last_fullsized":"2024-03-09T15:36:17.165000+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'108","last_scrub_stamp":"2024-03-09T15:36:17.164692+0000","last_deep_scrub":"261'108","last_deep_scrub_stamp":"2024-03-09T15:36:17.164692+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.164692+0000","objects_scrubbed":5,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:41:19.308226+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"261'505","reported_seq":2301,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.252225+0000","last_change":"2024-03-09T15:36:34.252290+0000","last_active":"2024-03-09T15:36:34.252225+0000","last_peered":"2024-03-09T15:36:34.252225+0000","last_clean":"2024-03-09T15:36:34.252225+0000","last_became_active":"2024-03-09T14:28:40.371507+0000","last_became_peered":"2024-03-09T14:28:40.371507+0000","last_unstale":"2024-03-09T15:36:34.252225+0000","last_undegraded":"2024-03-09T15:36:34.252225+0000","last_fullsized":"2024-03-09T15:36:34.252225+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'505","last_scrub_stamp":"2024-03-09T15:36:34.252038+0000","last_deep_scrub":"261'505","last_deep_scrub_stamp":"2024-03-09T15:36:34.252038+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.252038+0000","objects_scrubbed":1,"log_size":105,"log_dups_size":400,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:50:10.320691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"261'138","reported_seq":953,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.158161+0000","last_change":"2024-03-09T15:36:19.158227+0000","last_active":"2024-03-09T15:36:19.158161+0000","last_peered":"2024-03-09T15:36:19.158161+0000","last_clean":"2024-03-09T15:36:19.158161+0000","last_became_active":"2024-03-09T14:28:42.398498+0000","last_became_peered":"2024-03-09T14:28:42.398498+0000","last_unstale":"2024-03-09T15:36:19.158161+0000","last_undegraded":"2024-03-09T15:36:19.158161+0000","last_fullsized":"2024-03-09T15:36:19.158161+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'138","last_scrub_stamp":"2024-03-09T15:36:19.157968+0000","last_deep_scrub":"261'138","last_deep_scrub_stamp":"2024-03-09T15:36:19.157968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.157968+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":100,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:49:21.053160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":498,"num_read_kb":498,"num_write":277,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":234,"num_omap_keys":2,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"261'53","reported_seq":602,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.567436+0000","last_change":"2024-03-09T15:36:43.567493+0000","last_active":"2024-03-09T15:36:43.567436+0000","last_peered":"2024-03-09T15:36:43.567436+0000","last_clean":"2024-03-09T15:36:43.567436+0000","last_became_active":"2024-03-09T14:28:34.330347+0000","last_became_peered":"2024-03-09T14:28:34.330347+0000","last_unstale":"2024-03-09T15:36:43.567436+0000","last_undegraded":"2024-03-09T15:36:43.567436+0000","last_fullsized":"2024-03-09T15:36:43.567436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'53","last_scrub_stamp":"2024-03-09T15:36:43.567238+0000","last_deep_scrub":"261'53","last_deep_scrub_stamp":"2024-03-09T15:36:43.567238+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.567238+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:24:51.206691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":212,"num_read_kb":171,"num_write":77,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"261'169","reported_seq":751,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.275838+0000","last_change":"2024-03-09T15:36:38.275929+0000","last_active":"2024-03-09T15:36:38.275838+0000","last_peered":"2024-03-09T15:36:38.275838+0000","last_clean":"2024-03-09T15:36:38.275838+0000","last_became_active":"2024-03-09T14:28:32.319819+0000","last_became_peered":"2024-03-09T14:28:32.319819+0000","last_unstale":"2024-03-09T15:36:38.275838+0000","last_undegraded":"2024-03-09T15:36:38.275838+0000","last_fullsized":"2024-03-09T15:36:38.275838+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'169","last_scrub_stamp":"2024-03-09T15:36:38.275636+0000","last_deep_scrub":"261'169","last_deep_scrub_stamp":"2024-03-09T15:36:38.275636+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.275636+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":100,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:18:08.037585+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5244508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":231,"num_read_kb":15567,"num_write":579,"num_write_kb":34195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.872479+0000","last_change":"2024-03-09T15:36:38.872534+0000","last_active":"2024-03-09T15:36:38.872479+0000","last_peered":"2024-03-09T15:36:38.872479+0000","last_clean":"2024-03-09T15:36:38.872479+0000","last_became_active":"2024-03-09T14:28:36.310231+0000","last_became_peered":"2024-03-09T14:28:36.310231+0000","last_unstale":"2024-03-09T15:36:38.872479+0000","last_undegraded":"2024-03-09T15:36:38.872479+0000","last_fullsized":"2024-03-09T15:36:38.872479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:38.872296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:38.872296+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.872296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:03:09.877161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"261'174","reported_seq":767,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:03.157279+0000","last_change":"2024-03-09T15:37:03.157372+0000","last_active":"2024-03-09T15:37:03.157279+0000","last_peered":"2024-03-09T15:37:03.157279+0000","last_clean":"2024-03-09T15:37:03.157279+0000","last_became_active":"2024-03-09T14:28:38.598308+0000","last_became_peered":"2024-03-09T14:28:38.598308+0000","last_unstale":"2024-03-09T15:37:03.157279+0000","last_undegraded":"2024-03-09T15:37:03.157279+0000","last_fullsized":"2024-03-09T15:37:03.157279+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'174","last_scrub_stamp":"2024-03-09T15:37:03.157072+0000","last_deep_scrub":"261'174","last_deep_scrub_stamp":"2024-03-09T15:37:03.157072+0000","last_clean_scrub_stamp":"2024-03-09T15:37:03.157072+0000","objects_scrubbed":4,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:09:04.298940+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8724,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":532,"num_write":190,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"261'513","reported_seq":2509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.458248+0000","last_change":"2024-03-09T15:36:32.458248+0000","last_active":"2024-03-09T15:36:32.458248+0000","last_peered":"2024-03-09T15:36:32.458248+0000","last_clean":"2024-03-09T15:36:32.458248+0000","last_became_active":"2024-03-09T14:28:40.387990+0000","last_became_peered":"2024-03-09T14:28:40.387990+0000","last_unstale":"2024-03-09T15:36:32.458248+0000","last_undegraded":"2024-03-09T15:36:32.458248+0000","last_fullsized":"2024-03-09T15:36:32.458248+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'513","last_scrub_stamp":"2024-03-09T15:36:32.457985+0000","last_deep_scrub":"261'513","last_deep_scrub_stamp":"2024-03-09T15:36:32.457985+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.457985+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":500,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:24:58.183827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"261'101","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.740079+0000","last_change":"2024-03-09T15:36:28.740141+0000","last_active":"2024-03-09T15:36:28.740079+0000","last_peered":"2024-03-09T15:36:28.740079+0000","last_clean":"2024-03-09T15:36:28.740079+0000","last_became_active":"2024-03-09T14:28:34.322731+0000","last_became_peered":"2024-03-09T14:28:34.322731+0000","last_unstale":"2024-03-09T15:36:28.740079+0000","last_undegraded":"2024-03-09T15:36:28.740079+0000","last_fullsized":"2024-03-09T15:36:28.740079+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'101","last_scrub_stamp":"2024-03-09T15:36:28.739877+0000","last_deep_scrub":"261'101","last_deep_scrub_stamp":"2024-03-09T15:36:28.739877+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.739877+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:43:42.753565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":330,"num_write":141,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"261'130","reported_seq":688,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:52.150089+0000","last_change":"2024-03-09T15:36:52.150201+0000","last_active":"2024-03-09T15:36:52.150089+0000","last_peered":"2024-03-09T15:36:52.150089+0000","last_clean":"2024-03-09T15:36:52.150089+0000","last_became_active":"2024-03-09T14:28:32.329148+0000","last_became_peered":"2024-03-09T14:28:32.329148+0000","last_unstale":"2024-03-09T15:36:52.150089+0000","last_undegraded":"2024-03-09T15:36:52.150089+0000","last_fullsized":"2024-03-09T15:36:52.150089+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'130","last_scrub_stamp":"2024-03-09T15:36:52.149897+0000","last_deep_scrub":"261'130","last_deep_scrub_stamp":"2024-03-09T15:36:52.149897+0000","last_clean_scrub_stamp":"2024-03-09T15:36:52.149897+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:56:19.005464+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7947490,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":1766,"num_write":513,"num_write_kb":18289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":518,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.203914+0000","last_change":"2024-03-09T15:36:39.204023+0000","last_active":"2024-03-09T15:36:39.203914+0000","last_peered":"2024-03-09T15:36:39.203914+0000","last_clean":"2024-03-09T15:36:39.203914+0000","last_became_active":"2024-03-09T14:28:36.313904+0000","last_became_peered":"2024-03-09T14:28:36.313904+0000","last_unstale":"2024-03-09T15:36:39.203914+0000","last_undegraded":"2024-03-09T15:36:39.203914+0000","last_fullsized":"2024-03-09T15:36:39.203914+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:39.203674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:39.203674+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.203674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:18:37.090196+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"261'276","reported_seq":920,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.746370+0000","last_change":"2024-03-09T15:36:24.746541+0000","last_active":"2024-03-09T15:36:24.746370+0000","last_peered":"2024-03-09T15:36:24.746370+0000","last_clean":"2024-03-09T15:36:24.746370+0000","last_became_active":"2024-03-09T14:28:38.597247+0000","last_became_peered":"2024-03-09T14:28:38.597247+0000","last_unstale":"2024-03-09T15:36:24.746370+0000","last_undegraded":"2024-03-09T15:36:24.746370+0000","last_fullsized":"2024-03-09T15:36:24.746370+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'276","last_scrub_stamp":"2024-03-09T15:36:24.746130+0000","last_deep_scrub":"261'276","last_deep_scrub_stamp":"2024-03-09T15:36:24.746130+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.746130+0000","objects_scrubbed":9,"log_size":76,"log_dups_size":200,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:53:37.971114+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9398,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":421,"num_read_kb":664,"num_write":284,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.394226+0000","last_change":"2024-03-09T15:36:37.394277+0000","last_active":"2024-03-09T15:36:37.394226+0000","last_peered":"2024-03-09T15:36:37.394226+0000","last_clean":"2024-03-09T15:36:37.394226+0000","last_became_active":"2024-03-09T14:28:40.359906+0000","last_became_peered":"2024-03-09T14:28:40.359906+0000","last_unstale":"2024-03-09T15:36:37.394226+0000","last_undegraded":"2024-03-09T15:36:37.394226+0000","last_fullsized":"2024-03-09T15:36:37.394226+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:37.394037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:37.394037+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.394037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:42:28.463933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"261'522","reported_seq":1369,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.664753+0000","last_change":"2024-03-09T15:36:16.664829+0000","last_active":"2024-03-09T15:36:16.664753+0000","last_peered":"2024-03-09T15:36:16.664753+0000","last_clean":"2024-03-09T15:36:16.664753+0000","last_became_active":"2024-03-09T14:28:42.402739+0000","last_became_peered":"2024-03-09T14:28:42.402739+0000","last_unstale":"2024-03-09T15:36:16.664753+0000","last_undegraded":"2024-03-09T15:36:16.664753+0000","last_fullsized":"2024-03-09T15:36:16.664753+0000","mapping_epoch":30,"log_start":"261'500","ondisk_log_start":"261'500","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'522","last_scrub_stamp":"2024-03-09T15:36:16.664560+0000","last_deep_scrub":"261'522","last_deep_scrub_stamp":"2024-03-09T15:36:16.664560+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.664560+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":500,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:05:16.194957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1336,"num_read_kb":1311,"num_write":1083,"num_write_kb":376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":381,"num_omap_keys":2,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"261'92","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:53.451238+0000","last_change":"2024-03-09T15:36:53.451297+0000","last_active":"2024-03-09T15:36:53.451238+0000","last_peered":"2024-03-09T15:36:53.451238+0000","last_clean":"2024-03-09T15:36:53.451238+0000","last_became_active":"2024-03-09T14:28:34.300795+0000","last_became_peered":"2024-03-09T14:28:34.300795+0000","last_unstale":"2024-03-09T15:36:53.451238+0000","last_undegraded":"2024-03-09T15:36:53.451238+0000","last_fullsized":"2024-03-09T15:36:53.451238+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'92","last_scrub_stamp":"2024-03-09T15:36:53.451044+0000","last_deep_scrub":"261'92","last_deep_scrub_stamp":"2024-03-09T15:36:53.451044+0000","last_clean_scrub_stamp":"2024-03-09T15:36:53.451044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:08:06.800010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":361,"num_read_kb":288,"num_write":150,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"261'148","reported_seq":730,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:01.166928+0000","last_change":"2024-03-09T15:37:01.167023+0000","last_active":"2024-03-09T15:37:01.166928+0000","last_peered":"2024-03-09T15:37:01.166928+0000","last_clean":"2024-03-09T15:37:01.166928+0000","last_became_active":"2024-03-09T14:28:32.318111+0000","last_became_peered":"2024-03-09T14:28:32.318111+0000","last_unstale":"2024-03-09T15:37:01.166928+0000","last_undegraded":"2024-03-09T15:37:01.166928+0000","last_fullsized":"2024-03-09T15:37:01.166928+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'148","last_scrub_stamp":"2024-03-09T15:37:01.166734+0000","last_deep_scrub":"261'148","last_deep_scrub_stamp":"2024-03-09T15:37:01.166734+0000","last_clean_scrub_stamp":"2024-03-09T15:37:01.166734+0000","objects_scrubbed":6,"log_size":48,"log_dups_size":100,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:17:04.846737+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242921,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":6230,"num_write":597,"num_write_kb":19779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.197656+0000","last_change":"2024-03-09T15:36:27.197765+0000","last_active":"2024-03-09T15:36:27.197656+0000","last_peered":"2024-03-09T15:36:27.197656+0000","last_clean":"2024-03-09T15:36:27.197656+0000","last_became_active":"2024-03-09T14:28:36.301227+0000","last_became_peered":"2024-03-09T14:28:36.301227+0000","last_unstale":"2024-03-09T15:36:27.197656+0000","last_undegraded":"2024-03-09T15:36:27.197656+0000","last_fullsized":"2024-03-09T15:36:27.197656+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:27.197420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:27.197420+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.197420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:56:31.881994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"261'177","reported_seq":763,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.577608+0000","last_change":"2024-03-09T15:36:40.577792+0000","last_active":"2024-03-09T15:36:40.577608+0000","last_peered":"2024-03-09T15:36:40.577608+0000","last_clean":"2024-03-09T15:36:40.577608+0000","last_became_active":"2024-03-09T14:28:38.346837+0000","last_became_peered":"2024-03-09T14:28:38.346837+0000","last_unstale":"2024-03-09T15:36:40.577608+0000","last_undegraded":"2024-03-09T15:36:40.577608+0000","last_fullsized":"2024-03-09T15:36:40.577608+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'177","last_scrub_stamp":"2024-03-09T15:36:40.577391+0000","last_deep_scrub":"261'177","last_deep_scrub_stamp":"2024-03-09T15:36:40.577391+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.577391+0000","objects_scrubbed":4,"log_size":77,"log_dups_size":100,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:47:13.834091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16267,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":276,"num_read_kb":528,"num_write":188,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.485270+0000","last_change":"2024-03-09T15:36:30.485329+0000","last_active":"2024-03-09T15:36:30.485270+0000","last_peered":"2024-03-09T15:36:30.485270+0000","last_clean":"2024-03-09T15:36:30.485270+0000","last_became_active":"2024-03-09T14:28:40.389175+0000","last_became_peered":"2024-03-09T14:28:40.389175+0000","last_unstale":"2024-03-09T15:36:30.485270+0000","last_undegraded":"2024-03-09T15:36:30.485270+0000","last_fullsized":"2024-03-09T15:36:30.485270+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:30.485071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:30.485071+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.485071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:54:36.706597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"261'59","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.297133+0000","last_change":"2024-03-09T15:36:34.297200+0000","last_active":"2024-03-09T15:36:34.297133+0000","last_peered":"2024-03-09T15:36:34.297133+0000","last_clean":"2024-03-09T15:36:34.297133+0000","last_became_active":"2024-03-09T14:28:42.401214+0000","last_became_peered":"2024-03-09T14:28:42.401214+0000","last_unstale":"2024-03-09T15:36:34.297133+0000","last_undegraded":"2024-03-09T15:36:34.297133+0000","last_fullsized":"2024-03-09T15:36:34.297133+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'59","last_scrub_stamp":"2024-03-09T15:36:34.296927+0000","last_deep_scrub":"261'59","last_deep_scrub_stamp":"2024-03-09T15:36:34.296927+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.296927+0000","objects_scrubbed":1,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:18:34.284628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":47,"num_write":140,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"261'78","reported_seq":643,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.249814+0000","last_change":"2024-03-09T15:36:33.249877+0000","last_active":"2024-03-09T15:36:33.249814+0000","last_peered":"2024-03-09T15:36:33.249814+0000","last_clean":"2024-03-09T15:36:33.249814+0000","last_became_active":"2024-03-09T14:28:34.316812+0000","last_became_peered":"2024-03-09T14:28:34.316812+0000","last_unstale":"2024-03-09T15:36:33.249814+0000","last_undegraded":"2024-03-09T15:36:33.249814+0000","last_fullsized":"2024-03-09T15:36:33.249814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'78","last_scrub_stamp":"2024-03-09T15:36:33.249613+0000","last_deep_scrub":"261'78","last_deep_scrub_stamp":"2024-03-09T15:36:33.249613+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.249613+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:51.977272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":320,"num_read_kb":256,"num_write":116,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"261'111","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.172728+0000","last_change":"2024-03-09T15:36:27.172803+0000","last_active":"2024-03-09T15:36:27.172728+0000","last_peered":"2024-03-09T15:36:27.172728+0000","last_clean":"2024-03-09T15:36:27.172728+0000","last_became_active":"2024-03-09T14:29:06.446333+0000","last_became_peered":"2024-03-09T14:29:06.446333+0000","last_unstale":"2024-03-09T15:36:27.172728+0000","last_undegraded":"2024-03-09T15:36:27.172728+0000","last_fullsized":"2024-03-09T15:36:27.172728+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'111","last_scrub_stamp":"2024-03-09T15:36:27.172540+0000","last_deep_scrub":"261'111","last_deep_scrub_stamp":"2024-03-09T15:36:27.172540+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.172540+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:12:44.004446+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":147,"num_read_kb":1025,"num_write":402,"num_write_kb":11167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":515,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.252418+0000","last_change":"2024-03-09T15:36:25.252530+0000","last_active":"2024-03-09T15:36:25.252418+0000","last_peered":"2024-03-09T15:36:25.252418+0000","last_clean":"2024-03-09T15:36:25.252418+0000","last_became_active":"2024-03-09T14:28:36.303695+0000","last_became_peered":"2024-03-09T14:28:36.303695+0000","last_unstale":"2024-03-09T15:36:25.252418+0000","last_undegraded":"2024-03-09T15:36:25.252418+0000","last_fullsized":"2024-03-09T15:36:25.252418+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.252128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.252128+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.252128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:47:06.758741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"261'227","reported_seq":844,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.149330+0000","last_change":"2024-03-09T15:36:51.149432+0000","last_active":"2024-03-09T15:36:51.149330+0000","last_peered":"2024-03-09T15:36:51.149330+0000","last_clean":"2024-03-09T15:36:51.149330+0000","last_became_active":"2024-03-09T14:28:38.339300+0000","last_became_peered":"2024-03-09T14:28:38.339300+0000","last_unstale":"2024-03-09T15:36:51.149330+0000","last_undegraded":"2024-03-09T15:36:51.149330+0000","last_fullsized":"2024-03-09T15:36:51.149330+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'227","last_scrub_stamp":"2024-03-09T15:36:51.149113+0000","last_deep_scrub":"261'227","last_deep_scrub_stamp":"2024-03-09T15:36:51.149113+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.149113+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":200,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:50:51.145329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8477,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":353,"num_read_kb":624,"num_write":244,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"261'501","reported_seq":2519,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.232198+0000","last_change":"2024-03-09T15:36:34.232325+0000","last_active":"2024-03-09T15:36:34.232198+0000","last_peered":"2024-03-09T15:36:34.232198+0000","last_clean":"2024-03-09T15:36:34.232198+0000","last_became_active":"2024-03-09T14:28:40.367119+0000","last_became_peered":"2024-03-09T14:28:40.367119+0000","last_unstale":"2024-03-09T15:36:34.232198+0000","last_undegraded":"2024-03-09T15:36:34.232198+0000","last_fullsized":"2024-03-09T15:36:34.232198+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'501","last_scrub_stamp":"2024-03-09T15:36:34.231965+0000","last_deep_scrub":"261'501","last_deep_scrub_stamp":"2024-03-09T15:36:34.231965+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.231965+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":400,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:18:21.657518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"261'82","reported_seq":585,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.245845+0000","last_change":"2024-03-09T15:36:28.245919+0000","last_active":"2024-03-09T15:36:28.245845+0000","last_peered":"2024-03-09T15:36:28.245845+0000","last_clean":"2024-03-09T15:36:28.245845+0000","last_became_active":"2024-03-09T14:28:42.396389+0000","last_became_peered":"2024-03-09T14:28:42.396389+0000","last_unstale":"2024-03-09T15:36:28.245845+0000","last_undegraded":"2024-03-09T15:36:28.245845+0000","last_fullsized":"2024-03-09T15:36:28.245845+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'82","last_scrub_stamp":"2024-03-09T15:36:28.245628+0000","last_deep_scrub":"261'82","last_deep_scrub_stamp":"2024-03-09T15:36:28.245628+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.245628+0000","objects_scrubbed":1,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:26:51.961711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":65,"num_write":200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"261'44","reported_seq":591,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.354458+0000","last_change":"2024-03-09T15:36:46.354523+0000","last_active":"2024-03-09T15:36:46.354458+0000","last_peered":"2024-03-09T15:36:46.354458+0000","last_clean":"2024-03-09T15:36:46.354458+0000","last_became_active":"2024-03-09T14:28:34.297080+0000","last_became_peered":"2024-03-09T14:28:34.297080+0000","last_unstale":"2024-03-09T15:36:46.354458+0000","last_undegraded":"2024-03-09T15:36:46.354458+0000","last_fullsized":"2024-03-09T15:36:46.354458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:46.354264+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:46.354264+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.354264+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:33:23.149348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":189,"num_read_kb":155,"num_write":63,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"261'152","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.200259+0000","last_change":"2024-03-09T15:36:20.200259+0000","last_active":"2024-03-09T15:36:20.200259+0000","last_peered":"2024-03-09T15:36:20.200259+0000","last_clean":"2024-03-09T15:36:20.200259+0000","last_became_active":"2024-03-09T14:29:06.445573+0000","last_became_peered":"2024-03-09T14:29:06.445573+0000","last_unstale":"2024-03-09T15:36:20.200259+0000","last_undegraded":"2024-03-09T15:36:20.200259+0000","last_fullsized":"2024-03-09T15:36:20.200259+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'152","last_scrub_stamp":"2024-03-09T15:36:20.199970+0000","last_deep_scrub":"261'152","last_deep_scrub_stamp":"2024-03-09T15:36:20.199970+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.199970+0000","objects_scrubbed":5,"log_size":52,"log_dups_size":100,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:59:48.356486+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":204,"num_read_kb":157,"num_write":458,"num_write_kb":32761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.186224+0000","last_change":"2024-03-09T15:36:19.186332+0000","last_active":"2024-03-09T15:36:19.186224+0000","last_peered":"2024-03-09T15:36:19.186224+0000","last_clean":"2024-03-09T15:36:19.186224+0000","last_became_active":"2024-03-09T14:28:36.322638+0000","last_became_peered":"2024-03-09T14:28:36.322638+0000","last_unstale":"2024-03-09T15:36:19.186224+0000","last_undegraded":"2024-03-09T15:36:19.186224+0000","last_fullsized":"2024-03-09T15:36:19.186224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:19.185963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:19.185963+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.185963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:52:57.100559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"261'323","reported_seq":975,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:55.211712+0000","last_change":"2024-03-09T15:36:55.211901+0000","last_active":"2024-03-09T15:36:55.211712+0000","last_peered":"2024-03-09T15:36:55.211712+0000","last_clean":"2024-03-09T15:36:55.211712+0000","last_became_active":"2024-03-09T14:28:38.328897+0000","last_became_peered":"2024-03-09T14:28:38.328897+0000","last_unstale":"2024-03-09T15:36:55.211712+0000","last_undegraded":"2024-03-09T15:36:55.211712+0000","last_fullsized":"2024-03-09T15:36:55.211712+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'323","last_scrub_stamp":"2024-03-09T15:36:55.211462+0000","last_deep_scrub":"261'323","last_deep_scrub_stamp":"2024-03-09T15:36:55.211462+0000","last_clean_scrub_stamp":"2024-03-09T15:36:55.211462+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":300,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:45:05.244251+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":543,"num_read_kb":1162,"num_write":388,"num_write_kb":277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.192323+0000","last_change":"2024-03-09T15:36:12.192350+0000","last_active":"2024-03-09T15:36:12.192323+0000","last_peered":"2024-03-09T15:36:12.192323+0000","last_clean":"2024-03-09T15:36:12.192323+0000","last_became_active":"2024-03-09T14:28:40.383758+0000","last_became_peered":"2024-03-09T14:28:40.383758+0000","last_unstale":"2024-03-09T15:36:12.192323+0000","last_undegraded":"2024-03-09T15:36:12.192323+0000","last_fullsized":"2024-03-09T15:36:12.192323+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:12.192210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:12.192210+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.192210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:18:21.362249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"261'56","reported_seq":576,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.525835+0000","last_change":"2024-03-09T15:36:19.525925+0000","last_active":"2024-03-09T15:36:19.525835+0000","last_peered":"2024-03-09T15:36:19.525835+0000","last_clean":"2024-03-09T15:36:19.525835+0000","last_became_active":"2024-03-09T14:28:42.401880+0000","last_became_peered":"2024-03-09T14:28:42.401880+0000","last_unstale":"2024-03-09T15:36:19.525835+0000","last_undegraded":"2024-03-09T15:36:19.525835+0000","last_fullsized":"2024-03-09T15:36:19.525835+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:19.525622+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:19.525622+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.525622+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:58:32.566894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":69,"num_write":120,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"261'81","reported_seq":650,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.131661+0000","last_change":"2024-03-09T15:36:31.131726+0000","last_active":"2024-03-09T15:36:31.131661+0000","last_peered":"2024-03-09T15:36:31.131661+0000","last_clean":"2024-03-09T15:36:31.131661+0000","last_became_active":"2024-03-09T14:28:34.315432+0000","last_became_peered":"2024-03-09T14:28:34.315432+0000","last_unstale":"2024-03-09T15:36:31.131661+0000","last_undegraded":"2024-03-09T15:36:31.131661+0000","last_fullsized":"2024-03-09T15:36:31.131661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'81","last_scrub_stamp":"2024-03-09T15:36:31.131473+0000","last_deep_scrub":"261'81","last_deep_scrub_stamp":"2024-03-09T15:36:31.131473+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.131473+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:42:11.454205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":315,"num_read_kb":251,"num_write":126,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":466,"num_omap_keys":2,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"261'163","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:02.770414+0000","last_change":"2024-03-09T15:37:02.770536+0000","last_active":"2024-03-09T15:37:02.770414+0000","last_peered":"2024-03-09T15:37:02.770414+0000","last_clean":"2024-03-09T15:37:02.770414+0000","last_became_active":"2024-03-09T14:28:32.299897+0000","last_became_peered":"2024-03-09T14:28:32.299897+0000","last_unstale":"2024-03-09T15:37:02.770414+0000","last_undegraded":"2024-03-09T15:37:02.770414+0000","last_fullsized":"2024-03-09T15:37:02.770414+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'163","last_scrub_stamp":"2024-03-09T15:37:02.770220+0000","last_deep_scrub":"261'163","last_deep_scrub_stamp":"2024-03-09T15:37:02.770220+0000","last_clean_scrub_stamp":"2024-03-09T15:37:02.770220+0000","objects_scrubbed":6,"log_size":63,"log_dups_size":100,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:31:20.313548+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506226,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":160,"num_read_kb":225,"num_write":553,"num_write_kb":26448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":515,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.168556+0000","last_change":"2024-03-09T15:36:21.168616+0000","last_active":"2024-03-09T15:36:21.168556+0000","last_peered":"2024-03-09T15:36:21.168556+0000","last_clean":"2024-03-09T15:36:21.168556+0000","last_became_active":"2024-03-09T14:28:36.320029+0000","last_became_peered":"2024-03-09T14:28:36.320029+0000","last_unstale":"2024-03-09T15:36:21.168556+0000","last_undegraded":"2024-03-09T15:36:21.168556+0000","last_fullsized":"2024-03-09T15:36:21.168556+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.168322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.168322+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.168322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:54:01.071807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"261'213","reported_seq":814,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.154985+0000","last_change":"2024-03-09T15:36:28.155065+0000","last_active":"2024-03-09T15:36:28.154985+0000","last_peered":"2024-03-09T15:36:28.154985+0000","last_clean":"2024-03-09T15:36:28.154985+0000","last_became_active":"2024-03-09T14:28:38.338658+0000","last_became_peered":"2024-03-09T14:28:38.338658+0000","last_unstale":"2024-03-09T15:36:28.154985+0000","last_undegraded":"2024-03-09T15:36:28.154985+0000","last_fullsized":"2024-03-09T15:36:28.154985+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'213","last_scrub_stamp":"2024-03-09T15:36:28.154798+0000","last_deep_scrub":"261'213","last_deep_scrub_stamp":"2024-03-09T15:36:28.154798+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.154798+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":200,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:48:17.369213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12662,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":352,"num_read_kb":613,"num_write":292,"num_write_kb":188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13964,"num_omap_keys":34,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.231293+0000","last_change":"2024-03-09T15:36:30.231399+0000","last_active":"2024-03-09T15:36:30.231293+0000","last_peered":"2024-03-09T15:36:30.231293+0000","last_clean":"2024-03-09T15:36:30.231293+0000","last_became_active":"2024-03-09T14:28:40.378973+0000","last_became_peered":"2024-03-09T14:28:40.378973+0000","last_unstale":"2024-03-09T15:36:30.231293+0000","last_undegraded":"2024-03-09T15:36:30.231293+0000","last_fullsized":"2024-03-09T15:36:30.231293+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:30.231002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:30.231002+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.231002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:08:44.672009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"261'44","reported_seq":547,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.449669+0000","last_change":"2024-03-09T15:36:23.449669+0000","last_active":"2024-03-09T15:36:23.449669+0000","last_peered":"2024-03-09T15:36:23.449669+0000","last_clean":"2024-03-09T15:36:23.449669+0000","last_became_active":"2024-03-09T14:28:42.397533+0000","last_became_peered":"2024-03-09T14:28:42.397533+0000","last_unstale":"2024-03-09T15:36:23.449669+0000","last_undegraded":"2024-03-09T15:36:23.449669+0000","last_fullsized":"2024-03-09T15:36:23.449669+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:23.449413+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:23.449413+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.449413+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:46:36.909599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":91,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"261'56","reported_seq":625,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.166995+0000","last_change":"2024-03-09T15:36:34.167057+0000","last_active":"2024-03-09T15:36:34.166995+0000","last_peered":"2024-03-09T15:36:34.166995+0000","last_clean":"2024-03-09T15:36:34.166995+0000","last_became_active":"2024-03-09T14:28:34.313475+0000","last_became_peered":"2024-03-09T14:28:34.313475+0000","last_unstale":"2024-03-09T15:36:34.166995+0000","last_undegraded":"2024-03-09T15:36:34.166995+0000","last_fullsized":"2024-03-09T15:36:34.166995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:34.166804+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:34.166804+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.166804+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:15:44.814256+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":198,"num_write":78,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"261'136","reported_seq":697,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.314272+0000","last_change":"2024-03-09T15:36:37.314511+0000","last_active":"2024-03-09T15:36:37.314272+0000","last_peered":"2024-03-09T15:36:37.314272+0000","last_clean":"2024-03-09T15:36:37.314272+0000","last_became_active":"2024-03-09T14:28:32.316406+0000","last_became_peered":"2024-03-09T14:28:32.316406+0000","last_unstale":"2024-03-09T15:36:37.314272+0000","last_undegraded":"2024-03-09T15:36:37.314272+0000","last_fullsized":"2024-03-09T15:36:37.314272+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'136","last_scrub_stamp":"2024-03-09T15:36:37.314063+0000","last_deep_scrub":"261'136","last_deep_scrub_stamp":"2024-03-09T15:36:37.314063+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.314063+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:50:27.588191+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655835,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":162,"num_read_kb":15736,"num_write":466,"num_write_kb":13578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.511704+0000","last_change":"2024-03-09T15:36:27.511852+0000","last_active":"2024-03-09T15:36:27.511704+0000","last_peered":"2024-03-09T15:36:27.511704+0000","last_clean":"2024-03-09T15:36:27.511704+0000","last_became_active":"2024-03-09T14:28:36.319210+0000","last_became_peered":"2024-03-09T14:28:36.319210+0000","last_unstale":"2024-03-09T15:36:27.511704+0000","last_undegraded":"2024-03-09T15:36:27.511704+0000","last_fullsized":"2024-03-09T15:36:27.511704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:27.511458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:27.511458+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.511458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:07.969688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"261'223","reported_seq":837,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.196282+0000","last_change":"2024-03-09T15:36:15.196282+0000","last_active":"2024-03-09T15:36:15.196282+0000","last_peered":"2024-03-09T15:36:15.196282+0000","last_clean":"2024-03-09T15:36:15.196282+0000","last_became_active":"2024-03-09T14:28:38.350390+0000","last_became_peered":"2024-03-09T14:28:38.350390+0000","last_unstale":"2024-03-09T15:36:15.196282+0000","last_undegraded":"2024-03-09T15:36:15.196282+0000","last_fullsized":"2024-03-09T15:36:15.196282+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'223","last_scrub_stamp":"2024-03-09T15:36:15.195858+0000","last_deep_scrub":"261'223","last_deep_scrub_stamp":"2024-03-09T15:36:15.195858+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.195858+0000","objects_scrubbed":7,"log_size":23,"log_dups_size":200,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:47:25.524775+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9540,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":338,"num_read_kb":580,"num_write":230,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"261'474","reported_seq":2368,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.200778+0000","last_change":"2024-03-09T15:36:13.200827+0000","last_active":"2024-03-09T15:36:13.200778+0000","last_peered":"2024-03-09T15:36:13.200778+0000","last_clean":"2024-03-09T15:36:13.200778+0000","last_became_active":"2024-03-09T14:28:40.373360+0000","last_became_peered":"2024-03-09T14:28:40.373360+0000","last_unstale":"2024-03-09T15:36:13.200778+0000","last_undegraded":"2024-03-09T15:36:13.200778+0000","last_fullsized":"2024-03-09T15:36:13.200778+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'474","last_scrub_stamp":"2024-03-09T15:36:13.200687+0000","last_deep_scrub":"261'474","last_deep_scrub_stamp":"2024-03-09T15:36:13.200687+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.200687+0000","objects_scrubbed":1,"log_size":74,"log_dups_size":400,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:36:38.570841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"261'108","reported_seq":663,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.587316+0000","last_change":"2024-03-09T15:36:16.587396+0000","last_active":"2024-03-09T15:36:16.587316+0000","last_peered":"2024-03-09T15:36:16.587316+0000","last_clean":"2024-03-09T15:36:16.587316+0000","last_became_active":"2024-03-09T14:28:42.408233+0000","last_became_peered":"2024-03-09T14:28:42.408233+0000","last_unstale":"2024-03-09T15:36:16.587316+0000","last_undegraded":"2024-03-09T15:36:16.587316+0000","last_fullsized":"2024-03-09T15:36:16.587316+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'108","last_scrub_stamp":"2024-03-09T15:36:16.587096+0000","last_deep_scrub":"261'108","last_deep_scrub_stamp":"2024-03-09T15:36:16.587096+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.587096+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:31:09.055955+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":274,"num_read_kb":227,"num_write":264,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"261'72","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.342615+0000","last_change":"2024-03-09T15:36:44.342675+0000","last_active":"2024-03-09T15:36:44.342615+0000","last_peered":"2024-03-09T15:36:44.342615+0000","last_clean":"2024-03-09T15:36:44.342615+0000","last_became_active":"2024-03-09T14:28:34.310061+0000","last_became_peered":"2024-03-09T14:28:34.310061+0000","last_unstale":"2024-03-09T15:36:44.342615+0000","last_undegraded":"2024-03-09T15:36:44.342615+0000","last_fullsized":"2024-03-09T15:36:44.342615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-09T15:36:44.342422+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-09T15:36:44.342422+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.342422+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:25:32.650543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":294,"num_read_kb":233,"num_write":114,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"261'174","reported_seq":790,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.437240+0000","last_change":"2024-03-09T15:36:48.437332+0000","last_active":"2024-03-09T15:36:48.437240+0000","last_peered":"2024-03-09T15:36:48.437240+0000","last_clean":"2024-03-09T15:36:48.437240+0000","last_became_active":"2024-03-09T14:28:32.313566+0000","last_became_peered":"2024-03-09T14:28:32.313566+0000","last_unstale":"2024-03-09T15:36:48.437240+0000","last_undegraded":"2024-03-09T15:36:48.437240+0000","last_fullsized":"2024-03-09T15:36:48.437240+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'174","last_scrub_stamp":"2024-03-09T15:36:48.437063+0000","last_deep_scrub":"261'174","last_deep_scrub_stamp":"2024-03-09T15:36:48.437063+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.437063+0000","objects_scrubbed":8,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:03:36.289305+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":245,"num_read_kb":84880,"num_write":567,"num_write_kb":19718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"261'10","reported_seq":1472,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.904683+0000","last_change":"2024-03-09T15:36:42.904753+0000","last_active":"2024-03-09T15:36:42.904683+0000","last_peered":"2024-03-09T15:36:42.904683+0000","last_clean":"2024-03-09T15:36:42.904683+0000","last_became_active":"2024-03-09T14:28:36.325820+0000","last_became_peered":"2024-03-09T14:28:36.325820+0000","last_unstale":"2024-03-09T15:36:42.904683+0000","last_undegraded":"2024-03-09T15:36:42.904683+0000","last_fullsized":"2024-03-09T15:36:42.904683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'10","last_scrub_stamp":"2024-03-09T15:36:42.904497+0000","last_deep_scrub":"261'10","last_deep_scrub_stamp":"2024-03-09T15:36:42.904497+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.904497+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:01:23.989269+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1195,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1118,"num_read_kb":1421,"num_write":14,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"261'91","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.533379+0000","last_change":"2024-03-09T15:36:38.533460+0000","last_active":"2024-03-09T15:36:38.533379+0000","last_peered":"2024-03-09T15:36:38.533379+0000","last_clean":"2024-03-09T15:36:38.533379+0000","last_became_active":"2024-03-09T14:28:38.595478+0000","last_became_peered":"2024-03-09T14:28:38.595478+0000","last_unstale":"2024-03-09T15:36:38.533379+0000","last_undegraded":"2024-03-09T15:36:38.533379+0000","last_fullsized":"2024-03-09T15:36:38.533379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'91","last_scrub_stamp":"2024-03-09T15:36:38.533177+0000","last_deep_scrub":"261'91","last_deep_scrub_stamp":"2024-03-09T15:36:38.533177+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.533177+0000","objects_scrubbed":5,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:59:32.253642+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":105,"num_read_kb":100,"num_write":70,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"261'489","reported_seq":2304,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.167232+0000","last_change":"2024-03-09T15:36:26.167298+0000","last_active":"2024-03-09T15:36:26.167232+0000","last_peered":"2024-03-09T15:36:26.167232+0000","last_clean":"2024-03-09T15:36:26.167232+0000","last_became_active":"2024-03-09T14:28:40.374408+0000","last_became_peered":"2024-03-09T14:28:40.374408+0000","last_unstale":"2024-03-09T15:36:26.167232+0000","last_undegraded":"2024-03-09T15:36:26.167232+0000","last_fullsized":"2024-03-09T15:36:26.167232+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'489","last_scrub_stamp":"2024-03-09T15:36:26.166904+0000","last_deep_scrub":"261'489","last_deep_scrub_stamp":"2024-03-09T15:36:26.166904+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.166904+0000","objects_scrubbed":1,"log_size":89,"log_dups_size":400,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:49:54.207124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"261'94","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.145360+0000","last_change":"2024-03-09T15:36:16.145423+0000","last_active":"2024-03-09T15:36:16.145360+0000","last_peered":"2024-03-09T15:36:16.145360+0000","last_clean":"2024-03-09T15:36:16.145360+0000","last_became_active":"2024-03-09T14:28:42.411523+0000","last_became_peered":"2024-03-09T14:28:42.411523+0000","last_unstale":"2024-03-09T15:36:16.145360+0000","last_undegraded":"2024-03-09T15:36:16.145360+0000","last_fullsized":"2024-03-09T15:36:16.145360+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'94","last_scrub_stamp":"2024-03-09T15:36:16.145166+0000","last_deep_scrub":"261'94","last_deep_scrub_stamp":"2024-03-09T15:36:16.145166+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.145166+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:56:22.170750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":72,"num_write":221,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"261'42","reported_seq":605,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.506243+0000","last_change":"2024-03-09T15:36:28.506314+0000","last_active":"2024-03-09T15:36:28.506243+0000","last_peered":"2024-03-09T15:36:28.506243+0000","last_clean":"2024-03-09T15:36:28.506243+0000","last_became_active":"2024-03-09T14:28:34.317226+0000","last_became_peered":"2024-03-09T14:28:34.317226+0000","last_unstale":"2024-03-09T15:36:28.506243+0000","last_undegraded":"2024-03-09T15:36:28.506243+0000","last_fullsized":"2024-03-09T15:36:28.506243+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'42","last_scrub_stamp":"2024-03-09T15:36:28.505993+0000","last_deep_scrub":"261'42","last_deep_scrub_stamp":"2024-03-09T15:36:28.505993+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.505993+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:14:19.130657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":188,"num_read_kb":158,"num_write":62,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"261'119","reported_seq":664,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:59.715451+0000","last_change":"2024-03-09T15:36:59.715624+0000","last_active":"2024-03-09T15:36:59.715451+0000","last_peered":"2024-03-09T15:36:59.715451+0000","last_clean":"2024-03-09T15:36:59.715451+0000","last_became_active":"2024-03-09T14:28:32.295179+0000","last_became_peered":"2024-03-09T14:28:32.295179+0000","last_unstale":"2024-03-09T15:36:59.715451+0000","last_undegraded":"2024-03-09T15:36:59.715451+0000","last_fullsized":"2024-03-09T15:36:59.715451+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'119","last_scrub_stamp":"2024-03-09T15:36:59.715201+0000","last_deep_scrub":"261'119","last_deep_scrub_stamp":"2024-03-09T15:36:59.715201+0000","last_clean_scrub_stamp":"2024-03-09T15:36:59.715201+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":100,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:06:38.255376+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262159,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":143,"num_read_kb":124,"num_write":390,"num_write_kb":5920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.379645+0000","last_change":"2024-03-09T15:36:45.379697+0000","last_active":"2024-03-09T15:36:45.379645+0000","last_peered":"2024-03-09T15:36:45.379645+0000","last_clean":"2024-03-09T15:36:45.379645+0000","last_became_active":"2024-03-09T14:28:36.319102+0000","last_became_peered":"2024-03-09T14:28:36.319102+0000","last_unstale":"2024-03-09T15:36:45.379645+0000","last_undegraded":"2024-03-09T15:36:45.379645+0000","last_fullsized":"2024-03-09T15:36:45.379645+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:45.379458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:45.379458+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.379458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:10:12.455227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"261'321","reported_seq":958,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.774275+0000","last_change":"2024-03-09T15:36:27.774372+0000","last_active":"2024-03-09T15:36:27.774275+0000","last_peered":"2024-03-09T15:36:27.774275+0000","last_clean":"2024-03-09T15:36:27.774275+0000","last_became_active":"2024-03-09T14:28:38.337254+0000","last_became_peered":"2024-03-09T14:28:38.337254+0000","last_unstale":"2024-03-09T15:36:27.774275+0000","last_undegraded":"2024-03-09T15:36:27.774275+0000","last_fullsized":"2024-03-09T15:36:27.774275+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'321","last_scrub_stamp":"2024-03-09T15:36:27.774086+0000","last_deep_scrub":"261'321","last_deep_scrub_stamp":"2024-03-09T15:36:27.774086+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.774086+0000","objects_scrubbed":9,"log_size":21,"log_dups_size":300,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:34:50.482182+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13936,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":673,"num_read_kb":965,"num_write":788,"num_write_kb":629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":141964,"num_omap_keys":356,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.219910+0000","last_change":"2024-03-09T15:36:30.219963+0000","last_active":"2024-03-09T15:36:30.219910+0000","last_peered":"2024-03-09T15:36:30.219910+0000","last_clean":"2024-03-09T15:36:30.219910+0000","last_became_active":"2024-03-09T14:28:40.377714+0000","last_became_peered":"2024-03-09T14:28:40.377714+0000","last_unstale":"2024-03-09T15:36:30.219910+0000","last_undegraded":"2024-03-09T15:36:30.219910+0000","last_fullsized":"2024-03-09T15:36:30.219910+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:30.219708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:30.219708+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.219708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:30:50.699663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"261'69","reported_seq":577,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.545201+0000","last_change":"2024-03-09T15:36:20.545264+0000","last_active":"2024-03-09T15:36:20.545201+0000","last_peered":"2024-03-09T15:36:20.545201+0000","last_clean":"2024-03-09T15:36:20.545201+0000","last_became_active":"2024-03-09T14:28:42.401775+0000","last_became_peered":"2024-03-09T14:28:42.401775+0000","last_unstale":"2024-03-09T15:36:20.545201+0000","last_undegraded":"2024-03-09T15:36:20.545201+0000","last_fullsized":"2024-03-09T15:36:20.545201+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'69","last_scrub_stamp":"2024-03-09T15:36:20.545001+0000","last_deep_scrub":"261'69","last_deep_scrub_stamp":"2024-03-09T15:36:20.545001+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.545001+0000","objects_scrubbed":1,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:08:40.010383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":77,"num_write":153,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"261'174","reported_seq":834,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.140904+0000","last_change":"2024-03-09T15:36:27.140963+0000","last_active":"2024-03-09T15:36:27.140904+0000","last_peered":"2024-03-09T15:36:27.140904+0000","last_clean":"2024-03-09T15:36:27.140904+0000","last_became_active":"2024-03-09T14:28:34.326919+0000","last_became_peered":"2024-03-09T14:28:34.326919+0000","last_unstale":"2024-03-09T15:36:27.140904+0000","last_undegraded":"2024-03-09T15:36:27.140904+0000","last_fullsized":"2024-03-09T15:36:27.140904+0000","mapping_epoch":22,"log_start":"34'100","ondisk_log_start":"34'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'174","last_scrub_stamp":"2024-03-09T15:36:27.140715+0000","last_deep_scrub":"261'174","last_deep_scrub_stamp":"2024-03-09T15:36:27.140715+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.140715+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:06:14.462396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":914,"num_read_kb":819,"num_write":300,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"261'119","reported_seq":693,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:55.122472+0000","last_change":"2024-03-09T15:36:55.122551+0000","last_active":"2024-03-09T15:36:55.122472+0000","last_peered":"2024-03-09T15:36:55.122472+0000","last_clean":"2024-03-09T15:36:55.122472+0000","last_became_active":"2024-03-09T14:28:12.262430+0000","last_became_peered":"2024-03-09T14:28:12.262430+0000","last_unstale":"2024-03-09T15:36:55.122472+0000","last_undegraded":"2024-03-09T15:36:55.122472+0000","last_fullsized":"2024-03-09T15:36:55.122472+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'119","last_scrub_stamp":"2024-03-09T15:36:55.122277+0000","last_deep_scrub":"261'119","last_deep_scrub_stamp":"2024-03-09T15:36:55.122277+0000","last_clean_scrub_stamp":"2024-03-09T15:36:55.122277+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":100,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:38:23.689863+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":532,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.969844+0000","last_change":"2024-03-09T15:36:44.969844+0000","last_active":"2024-03-09T15:36:44.969844+0000","last_peered":"2024-03-09T15:36:44.969844+0000","last_clean":"2024-03-09T15:36:44.969844+0000","last_became_active":"2024-03-09T14:28:19.232517+0000","last_became_peered":"2024-03-09T14:28:19.232517+0000","last_unstale":"2024-03-09T15:36:44.969844+0000","last_undegraded":"2024-03-09T15:36:44.969844+0000","last_fullsized":"2024-03-09T15:36:44.969844+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:44.969581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:44.969581+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.969581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:43:52.001009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"261'134","reported_seq":702,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:03.797529+0000","last_change":"2024-03-09T15:37:03.797620+0000","last_active":"2024-03-09T15:37:03.797529+0000","last_peered":"2024-03-09T15:37:03.797529+0000","last_clean":"2024-03-09T15:37:03.797529+0000","last_became_active":"2024-03-09T14:28:32.300468+0000","last_became_peered":"2024-03-09T14:28:32.300468+0000","last_unstale":"2024-03-09T15:37:03.797529+0000","last_undegraded":"2024-03-09T15:37:03.797529+0000","last_fullsized":"2024-03-09T15:37:03.797529+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'134","last_scrub_stamp":"2024-03-09T15:37:03.797338+0000","last_deep_scrub":"261'134","last_deep_scrub_stamp":"2024-03-09T15:37:03.797338+0000","last_clean_scrub_stamp":"2024-03-09T15:37:03.797338+0000","objects_scrubbed":4,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:05:05.538076+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311644,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":173,"num_read_kb":6450,"num_write":457,"num_write_kb":21101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":519,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.163673+0000","last_change":"2024-03-09T15:36:41.163783+0000","last_active":"2024-03-09T15:36:41.163673+0000","last_peered":"2024-03-09T15:36:41.163673+0000","last_clean":"2024-03-09T15:36:41.163673+0000","last_became_active":"2024-03-09T14:28:36.318172+0000","last_became_peered":"2024-03-09T14:28:36.318172+0000","last_unstale":"2024-03-09T15:36:41.163673+0000","last_undegraded":"2024-03-09T15:36:41.163673+0000","last_fullsized":"2024-03-09T15:36:41.163673+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:41.163438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:41.163438+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.163438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:55:32.987190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"261'502","reported_seq":1226,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.737856+0000","last_change":"2024-03-09T15:36:25.737931+0000","last_active":"2024-03-09T15:36:25.737856+0000","last_peered":"2024-03-09T15:36:25.737856+0000","last_clean":"2024-03-09T15:36:25.737856+0000","last_became_active":"2024-03-09T14:28:38.348941+0000","last_became_peered":"2024-03-09T14:28:38.348941+0000","last_unstale":"2024-03-09T15:36:25.737856+0000","last_undegraded":"2024-03-09T15:36:25.737856+0000","last_fullsized":"2024-03-09T15:36:25.737856+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'502","last_scrub_stamp":"2024-03-09T15:36:25.737665+0000","last_deep_scrub":"261'502","last_deep_scrub_stamp":"2024-03-09T15:36:25.737665+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.737665+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":400,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:22:57.116693+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37400,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":987,"num_read_kb":1910,"num_write":934,"num_write_kb":767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":90684,"num_omap_keys":220,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.239905+0000","last_change":"2024-03-09T15:36:37.239957+0000","last_active":"2024-03-09T15:36:37.239905+0000","last_peered":"2024-03-09T15:36:37.239905+0000","last_clean":"2024-03-09T15:36:37.239905+0000","last_became_active":"2024-03-09T14:28:40.362781+0000","last_became_peered":"2024-03-09T14:28:40.362781+0000","last_unstale":"2024-03-09T15:36:37.239905+0000","last_undegraded":"2024-03-09T15:36:37.239905+0000","last_fullsized":"2024-03-09T15:36:37.239905+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:37.239705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:37.239705+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.239705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:28.768337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"261'121","reported_seq":829,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.279007+0000","last_change":"2024-03-09T15:36:44.279073+0000","last_active":"2024-03-09T15:36:44.279007+0000","last_peered":"2024-03-09T15:36:44.279007+0000","last_clean":"2024-03-09T15:36:44.279007+0000","last_became_active":"2024-03-09T14:28:42.398626+0000","last_became_peered":"2024-03-09T14:28:42.398626+0000","last_unstale":"2024-03-09T15:36:44.279007+0000","last_undegraded":"2024-03-09T15:36:44.279007+0000","last_fullsized":"2024-03-09T15:36:44.279007+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'121","last_scrub_stamp":"2024-03-09T15:36:44.278808+0000","last_deep_scrub":"261'121","last_deep_scrub_stamp":"2024-03-09T15:36:44.278808+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.278808+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":100,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:44:17.595367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":356,"num_read_kb":343,"num_write":215,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":66,"num_omap_keys":1,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"261'90","reported_seq":679,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.266824+0000","last_change":"2024-03-09T15:36:50.266889+0000","last_active":"2024-03-09T15:36:50.266824+0000","last_peered":"2024-03-09T15:36:50.266824+0000","last_clean":"2024-03-09T15:36:50.266824+0000","last_became_active":"2024-03-09T14:28:34.297472+0000","last_became_peered":"2024-03-09T14:28:34.297472+0000","last_unstale":"2024-03-09T15:36:50.266824+0000","last_undegraded":"2024-03-09T15:36:50.266824+0000","last_fullsized":"2024-03-09T15:36:50.266824+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'90","last_scrub_stamp":"2024-03-09T15:36:50.266621+0000","last_deep_scrub":"261'90","last_deep_scrub_stamp":"2024-03-09T15:36:50.266621+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.266621+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:34:30.319896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":356,"num_read_kb":292,"num_write":141,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":535,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:57.107045+0000","last_change":"2024-03-09T15:36:57.107099+0000","last_active":"2024-03-09T15:36:57.107045+0000","last_peered":"2024-03-09T15:36:57.107045+0000","last_clean":"2024-03-09T15:36:57.107045+0000","last_became_active":"2024-03-09T14:28:19.241110+0000","last_became_peered":"2024-03-09T14:28:19.241110+0000","last_unstale":"2024-03-09T15:36:57.107045+0000","last_undegraded":"2024-03-09T15:36:57.107045+0000","last_fullsized":"2024-03-09T15:36:57.107045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:57.106845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:57.106845+0000","last_clean_scrub_stamp":"2024-03-09T15:36:57.106845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:15:58.886485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"261'139","reported_seq":720,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:54.246645+0000","last_change":"2024-03-09T15:36:54.246715+0000","last_active":"2024-03-09T15:36:54.246645+0000","last_peered":"2024-03-09T15:36:54.246645+0000","last_clean":"2024-03-09T15:36:54.246645+0000","last_became_active":"2024-03-09T14:28:32.312536+0000","last_became_peered":"2024-03-09T14:28:32.312536+0000","last_unstale":"2024-03-09T15:36:54.246645+0000","last_undegraded":"2024-03-09T15:36:54.246645+0000","last_fullsized":"2024-03-09T15:36:54.246645+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'139","last_scrub_stamp":"2024-03-09T15:36:54.246441+0000","last_deep_scrub":"261'139","last_deep_scrub_stamp":"2024-03-09T15:36:54.246441+0000","last_clean_scrub_stamp":"2024-03-09T15:36:54.246441+0000","objects_scrubbed":1,"log_size":39,"log_dups_size":100,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:59:59.656356+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":219,"num_read_kb":12489,"num_write":525,"num_write_kb":17953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.821324+0000","last_change":"2024-03-09T15:36:36.821377+0000","last_active":"2024-03-09T15:36:36.821324+0000","last_peered":"2024-03-09T15:36:36.821324+0000","last_clean":"2024-03-09T15:36:36.821324+0000","last_became_active":"2024-03-09T14:28:36.306912+0000","last_became_peered":"2024-03-09T14:28:36.306912+0000","last_unstale":"2024-03-09T15:36:36.821324+0000","last_undegraded":"2024-03-09T15:36:36.821324+0000","last_fullsized":"2024-03-09T15:36:36.821324+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:36.821128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:36.821128+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.821128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:16:15.465239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"261'854","reported_seq":2540,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.493222+0000","last_change":"2024-03-09T15:36:31.493336+0000","last_active":"2024-03-09T15:36:31.493222+0000","last_peered":"2024-03-09T15:36:31.493222+0000","last_clean":"2024-03-09T15:36:31.493222+0000","last_became_active":"2024-03-09T14:28:38.328738+0000","last_became_peered":"2024-03-09T14:28:38.328738+0000","last_unstale":"2024-03-09T15:36:31.493222+0000","last_undegraded":"2024-03-09T15:36:31.493222+0000","last_fullsized":"2024-03-09T15:36:31.493222+0000","mapping_epoch":26,"log_start":"261'800","ondisk_log_start":"261'800","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'854","last_scrub_stamp":"2024-03-09T15:36:31.493032+0000","last_deep_scrub":"261'854","last_deep_scrub_stamp":"2024-03-09T15:36:31.493032+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.493032+0000","objects_scrubbed":11,"log_size":54,"log_dups_size":800,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:44:58.935541+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34683,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1323,"num_read_kb":2148,"num_write":662,"num_write_kb":462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5354,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":511,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.192845+0000","last_change":"2024-03-09T15:36:32.192899+0000","last_active":"2024-03-09T15:36:32.192845+0000","last_peered":"2024-03-09T15:36:32.192845+0000","last_clean":"2024-03-09T15:36:32.192845+0000","last_became_active":"2024-03-09T14:28:40.386451+0000","last_became_peered":"2024-03-09T14:28:40.386451+0000","last_unstale":"2024-03-09T15:36:32.192845+0000","last_undegraded":"2024-03-09T15:36:32.192845+0000","last_fullsized":"2024-03-09T15:36:32.192845+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:32.192627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:32.192627+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.192627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:09:46.269539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"261'129","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.567045+0000","last_change":"2024-03-09T15:36:18.567214+0000","last_active":"2024-03-09T15:36:18.567045+0000","last_peered":"2024-03-09T15:36:18.567045+0000","last_clean":"2024-03-09T15:36:18.567045+0000","last_became_active":"2024-03-09T14:28:42.403013+0000","last_became_peered":"2024-03-09T14:28:42.403013+0000","last_unstale":"2024-03-09T15:36:18.567045+0000","last_undegraded":"2024-03-09T15:36:18.567045+0000","last_fullsized":"2024-03-09T15:36:18.567045+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'129","last_scrub_stamp":"2024-03-09T15:36:18.566788+0000","last_deep_scrub":"261'129","last_deep_scrub_stamp":"2024-03-09T15:36:18.566788+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.566788+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:45:17.069116+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1248,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":261,"num_read_kb":241,"num_write":300,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"261'48","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.363144+0000","last_change":"2024-03-09T15:36:38.363206+0000","last_active":"2024-03-09T15:36:38.363144+0000","last_peered":"2024-03-09T15:36:38.363144+0000","last_clean":"2024-03-09T15:36:38.363144+0000","last_became_active":"2024-03-09T14:28:34.296733+0000","last_became_peered":"2024-03-09T14:28:34.296733+0000","last_unstale":"2024-03-09T15:36:38.363144+0000","last_undegraded":"2024-03-09T15:36:38.363144+0000","last_fullsized":"2024-03-09T15:36:38.363144+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:38.362948+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:38.362948+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.362948+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:32:01.096416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":200,"num_read_kb":166,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":532,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.234318+0000","last_change":"2024-03-09T15:36:28.234318+0000","last_active":"2024-03-09T15:36:28.234318+0000","last_peered":"2024-03-09T15:36:28.234318+0000","last_clean":"2024-03-09T15:36:28.234318+0000","last_became_active":"2024-03-09T14:28:19.436985+0000","last_became_peered":"2024-03-09T14:28:19.436985+0000","last_unstale":"2024-03-09T15:36:28.234318+0000","last_undegraded":"2024-03-09T15:36:28.234318+0000","last_fullsized":"2024-03-09T15:36:28.234318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:28.234071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:28.234071+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.234071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:27:16.042052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"261'159","reported_seq":826,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.221405+0000","last_change":"2024-03-09T15:36:43.221405+0000","last_active":"2024-03-09T15:36:43.221405+0000","last_peered":"2024-03-09T15:36:43.221405+0000","last_clean":"2024-03-09T15:36:43.221405+0000","last_became_active":"2024-03-09T14:28:32.303176+0000","last_became_peered":"2024-03-09T14:28:32.303176+0000","last_unstale":"2024-03-09T15:36:43.221405+0000","last_undegraded":"2024-03-09T15:36:43.221405+0000","last_fullsized":"2024-03-09T15:36:43.221405+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'159","last_scrub_stamp":"2024-03-09T15:36:43.221036+0000","last_deep_scrub":"261'159","last_deep_scrub_stamp":"2024-03-09T15:36:43.221036+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.221036+0000","objects_scrubbed":7,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:18:01.135873+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8554724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":486,"num_read_kb":3616,"num_write":592,"num_write_kb":29269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"261'2","reported_seq":518,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.203415+0000","last_change":"2024-03-09T15:36:23.203478+0000","last_active":"2024-03-09T15:36:23.203415+0000","last_peered":"2024-03-09T15:36:23.203415+0000","last_clean":"2024-03-09T15:36:23.203415+0000","last_became_active":"2024-03-09T14:28:36.305971+0000","last_became_peered":"2024-03-09T14:28:36.305971+0000","last_unstale":"2024-03-09T15:36:23.203415+0000","last_undegraded":"2024-03-09T15:36:23.203415+0000","last_fullsized":"2024-03-09T15:36:23.203415+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:23.203217+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:23.203217+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.203217+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:42:55.897785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"261'239","reported_seq":1616,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:53.186831+0000","last_change":"2024-03-09T15:36:53.186919+0000","last_active":"2024-03-09T15:36:53.186831+0000","last_peered":"2024-03-09T15:36:53.186831+0000","last_clean":"2024-03-09T15:36:53.186831+0000","last_became_active":"2024-03-09T14:28:38.597403+0000","last_became_peered":"2024-03-09T14:28:38.597403+0000","last_unstale":"2024-03-09T15:36:53.186831+0000","last_undegraded":"2024-03-09T15:36:53.186831+0000","last_fullsized":"2024-03-09T15:36:53.186831+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'239","last_scrub_stamp":"2024-03-09T15:36:53.186633+0000","last_deep_scrub":"261'239","last_deep_scrub_stamp":"2024-03-09T15:36:53.186633+0000","last_clean_scrub_stamp":"2024-03-09T15:36:53.186633+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":200,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:34:56.366739+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":226,"num_read_kb":216,"num_write":195,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5400,"num_omap_keys":12,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.247829+0000","last_change":"2024-03-09T15:36:32.247884+0000","last_active":"2024-03-09T15:36:32.247829+0000","last_peered":"2024-03-09T15:36:32.247829+0000","last_clean":"2024-03-09T15:36:32.247829+0000","last_became_active":"2024-03-09T14:28:40.371383+0000","last_became_peered":"2024-03-09T14:28:40.371383+0000","last_unstale":"2024-03-09T15:36:32.247829+0000","last_undegraded":"2024-03-09T15:36:32.247829+0000","last_fullsized":"2024-03-09T15:36:32.247829+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:32.247639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:32.247639+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.247639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:52:44.454327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"261'203","reported_seq":803,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.435372+0000","last_change":"2024-03-09T15:36:24.435440+0000","last_active":"2024-03-09T15:36:24.435372+0000","last_peered":"2024-03-09T15:36:24.435372+0000","last_clean":"2024-03-09T15:36:24.435372+0000","last_became_active":"2024-03-09T14:28:42.395781+0000","last_became_peered":"2024-03-09T14:28:42.395781+0000","last_unstale":"2024-03-09T15:36:24.435372+0000","last_undegraded":"2024-03-09T15:36:24.435372+0000","last_fullsized":"2024-03-09T15:36:24.435372+0000","mapping_epoch":30,"log_start":"247'100","ondisk_log_start":"247'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'203","last_scrub_stamp":"2024-03-09T15:36:24.435182+0000","last_deep_scrub":"261'203","last_deep_scrub_stamp":"2024-03-09T15:36:24.435182+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.435182+0000","objects_scrubbed":2,"log_size":103,"log_dups_size":100,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:28.404974+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":435,"num_read_kb":456,"num_write":425,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":851,"num_omap_keys":5,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"261'74","reported_seq":627,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.454421+0000","last_change":"2024-03-09T15:36:32.454571+0000","last_active":"2024-03-09T15:36:32.454421+0000","last_peered":"2024-03-09T15:36:32.454421+0000","last_clean":"2024-03-09T15:36:32.454421+0000","last_became_active":"2024-03-09T14:28:34.313302+0000","last_became_peered":"2024-03-09T14:28:34.313302+0000","last_unstale":"2024-03-09T15:36:32.454421+0000","last_undegraded":"2024-03-09T15:36:32.454421+0000","last_fullsized":"2024-03-09T15:36:32.454421+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'74","last_scrub_stamp":"2024-03-09T15:36:32.454104+0000","last_deep_scrub":"261'74","last_deep_scrub_stamp":"2024-03-09T15:36:32.454104+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.454104+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:47:12.348471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":225,"num_write":104,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":532,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.939088+0000","last_change":"2024-03-09T15:36:45.939088+0000","last_active":"2024-03-09T15:36:45.939088+0000","last_peered":"2024-03-09T15:36:45.939088+0000","last_clean":"2024-03-09T15:36:45.939088+0000","last_became_active":"2024-03-09T14:28:19.237937+0000","last_became_peered":"2024-03-09T14:28:19.237937+0000","last_unstale":"2024-03-09T15:36:45.939088+0000","last_undegraded":"2024-03-09T15:36:45.939088+0000","last_fullsized":"2024-03-09T15:36:45.939088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:45.938842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:45.938842+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.938842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:04:17.332833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"261'130","reported_seq":714,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.409386+0000","last_change":"2024-03-09T15:36:47.409498+0000","last_active":"2024-03-09T15:36:47.409386+0000","last_peered":"2024-03-09T15:36:47.409386+0000","last_clean":"2024-03-09T15:36:47.409386+0000","last_became_active":"2024-03-09T14:28:32.303963+0000","last_became_peered":"2024-03-09T14:28:32.303963+0000","last_unstale":"2024-03-09T15:36:47.409386+0000","last_undegraded":"2024-03-09T15:36:47.409386+0000","last_fullsized":"2024-03-09T15:36:47.409386+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'130","last_scrub_stamp":"2024-03-09T15:36:47.409169+0000","last_deep_scrub":"261'130","last_deep_scrub_stamp":"2024-03-09T15:36:47.409169+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.409169+0000","objects_scrubbed":3,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:17:16.816469+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6898850,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":175,"num_read_kb":15115,"num_write":430,"num_write_kb":32085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.253103+0000","last_change":"2024-03-09T15:36:49.253156+0000","last_active":"2024-03-09T15:36:49.253103+0000","last_peered":"2024-03-09T15:36:49.253103+0000","last_clean":"2024-03-09T15:36:49.253103+0000","last_became_active":"2024-03-09T14:28:36.317586+0000","last_became_peered":"2024-03-09T14:28:36.317586+0000","last_unstale":"2024-03-09T15:36:49.253103+0000","last_undegraded":"2024-03-09T15:36:49.253103+0000","last_fullsized":"2024-03-09T15:36:49.253103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:49.252862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:49.252862+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.252862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:35:36.249291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"261'136","reported_seq":698,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:57.805050+0000","last_change":"2024-03-09T15:36:57.805127+0000","last_active":"2024-03-09T15:36:57.805050+0000","last_peered":"2024-03-09T15:36:57.805050+0000","last_clean":"2024-03-09T15:36:57.805050+0000","last_became_active":"2024-03-09T14:28:38.334463+0000","last_became_peered":"2024-03-09T14:28:38.334463+0000","last_unstale":"2024-03-09T15:36:57.805050+0000","last_undegraded":"2024-03-09T15:36:57.805050+0000","last_fullsized":"2024-03-09T15:36:57.805050+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'136","last_scrub_stamp":"2024-03-09T15:36:57.804850+0000","last_deep_scrub":"261'136","last_deep_scrub_stamp":"2024-03-09T15:36:57.804850+0000","last_clean_scrub_stamp":"2024-03-09T15:36:57.804850+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:28:50.226857+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":178,"num_read_kb":172,"num_write":132,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1510,"num_omap_keys":4,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.103290+0000","last_change":"2024-03-09T15:36:20.103341+0000","last_active":"2024-03-09T15:36:20.103290+0000","last_peered":"2024-03-09T15:36:20.103290+0000","last_clean":"2024-03-09T15:36:20.103290+0000","last_became_active":"2024-03-09T14:28:40.370436+0000","last_became_peered":"2024-03-09T14:28:40.370436+0000","last_unstale":"2024-03-09T15:36:20.103290+0000","last_undegraded":"2024-03-09T15:36:20.103290+0000","last_fullsized":"2024-03-09T15:36:20.103290+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:20.103091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:20.103091+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.103091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:29:38.107241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"261'48","reported_seq":573,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.682061+0000","last_change":"2024-03-09T15:36:17.682122+0000","last_active":"2024-03-09T15:36:17.682061+0000","last_peered":"2024-03-09T15:36:17.682061+0000","last_clean":"2024-03-09T15:36:17.682061+0000","last_became_active":"2024-03-09T14:28:42.397058+0000","last_became_peered":"2024-03-09T14:28:42.397058+0000","last_unstale":"2024-03-09T15:36:17.682061+0000","last_undegraded":"2024-03-09T15:36:17.682061+0000","last_fullsized":"2024-03-09T15:36:17.682061+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:17.681861+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:17.681861+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.681861+0000","objects_scrubbed":1,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:38:01.033648+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":100,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"261'43","reported_seq":602,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.426247+0000","last_change":"2024-03-09T15:36:36.426328+0000","last_active":"2024-03-09T15:36:36.426247+0000","last_peered":"2024-03-09T15:36:36.426247+0000","last_clean":"2024-03-09T15:36:36.426247+0000","last_became_active":"2024-03-09T14:28:34.296953+0000","last_became_peered":"2024-03-09T14:28:34.296953+0000","last_unstale":"2024-03-09T15:36:36.426247+0000","last_undegraded":"2024-03-09T15:36:36.426247+0000","last_fullsized":"2024-03-09T15:36:36.426247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:36.426051+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:36.426051+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.426051+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:43:59.353686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":162,"num_write":75,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"261'165","reported_seq":808,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.889335+0000","last_change":"2024-03-09T15:36:48.889523+0000","last_active":"2024-03-09T15:36:48.889335+0000","last_peered":"2024-03-09T15:36:48.889335+0000","last_clean":"2024-03-09T15:36:48.889335+0000","last_became_active":"2024-03-09T14:28:32.302970+0000","last_became_peered":"2024-03-09T14:28:32.302970+0000","last_unstale":"2024-03-09T15:36:48.889335+0000","last_undegraded":"2024-03-09T15:36:48.889335+0000","last_fullsized":"2024-03-09T15:36:48.889335+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:48.889098+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:48.889098+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.889098+0000","objects_scrubbed":2,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:53:35.895952+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655822,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":94521,"num_write":632,"num_write_kb":45805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":530,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.321259+0000","last_change":"2024-03-09T15:36:39.321259+0000","last_active":"2024-03-09T15:36:39.321259+0000","last_peered":"2024-03-09T15:36:39.321259+0000","last_clean":"2024-03-09T15:36:39.321259+0000","last_became_active":"2024-03-09T14:28:19.236240+0000","last_became_peered":"2024-03-09T14:28:19.236240+0000","last_unstale":"2024-03-09T15:36:39.321259+0000","last_undegraded":"2024-03-09T15:36:39.321259+0000","last_fullsized":"2024-03-09T15:36:39.321259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:39.321005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:39.321005+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.321005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:47:02.712539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.328428+0000","last_change":"2024-03-09T15:36:47.328482+0000","last_active":"2024-03-09T15:36:47.328428+0000","last_peered":"2024-03-09T15:36:47.328428+0000","last_clean":"2024-03-09T15:36:47.328428+0000","last_became_active":"2024-03-09T14:28:36.314526+0000","last_became_peered":"2024-03-09T14:28:36.314526+0000","last_unstale":"2024-03-09T15:36:47.328428+0000","last_undegraded":"2024-03-09T15:36:47.328428+0000","last_fullsized":"2024-03-09T15:36:47.328428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:47.328224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:47.328224+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.328224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:38:07.502391+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"261'393","reported_seq":1085,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.271984+0000","last_change":"2024-03-09T15:36:40.271984+0000","last_active":"2024-03-09T15:36:40.271984+0000","last_peered":"2024-03-09T15:36:40.271984+0000","last_clean":"2024-03-09T15:36:40.271984+0000","last_became_active":"2024-03-09T14:28:38.334976+0000","last_became_peered":"2024-03-09T14:28:38.334976+0000","last_unstale":"2024-03-09T15:36:40.271984+0000","last_undegraded":"2024-03-09T15:36:40.271984+0000","last_fullsized":"2024-03-09T15:36:40.271984+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'393","last_scrub_stamp":"2024-03-09T15:36:40.271702+0000","last_deep_scrub":"261'393","last_deep_scrub_stamp":"2024-03-09T15:36:40.271702+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.271702+0000","objects_scrubbed":9,"log_size":93,"log_dups_size":300,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:45:05.080884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21042,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":640,"num_read_kb":1204,"num_write":456,"num_write_kb":268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2284,"num_omap_keys":4,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.131101+0000","last_change":"2024-03-09T15:36:20.131154+0000","last_active":"2024-03-09T15:36:20.131101+0000","last_peered":"2024-03-09T15:36:20.131101+0000","last_clean":"2024-03-09T15:36:20.131101+0000","last_became_active":"2024-03-09T14:28:40.379108+0000","last_became_peered":"2024-03-09T14:28:40.379108+0000","last_unstale":"2024-03-09T15:36:20.131101+0000","last_undegraded":"2024-03-09T15:36:20.131101+0000","last_fullsized":"2024-03-09T15:36:20.131101+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:20.130903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:20.130903+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.130903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:41:11.167873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"261'104","reported_seq":639,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.264151+0000","last_change":"2024-03-09T15:36:26.264345+0000","last_active":"2024-03-09T15:36:26.264151+0000","last_peered":"2024-03-09T15:36:26.264151+0000","last_clean":"2024-03-09T15:36:26.264151+0000","last_became_active":"2024-03-09T14:28:42.405277+0000","last_became_peered":"2024-03-09T14:28:42.405277+0000","last_unstale":"2024-03-09T15:36:26.264151+0000","last_undegraded":"2024-03-09T15:36:26.264151+0000","last_fullsized":"2024-03-09T15:36:26.264151+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'104","last_scrub_stamp":"2024-03-09T15:36:26.263887+0000","last_deep_scrub":"261'104","last_deep_scrub_stamp":"2024-03-09T15:36:26.263887+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.263887+0000","objects_scrubbed":3,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:52.786785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":717,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185,"num_read_kb":160,"num_write":261,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"261'43","reported_seq":584,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.293213+0000","last_change":"2024-03-09T15:36:48.293348+0000","last_active":"2024-03-09T15:36:48.293213+0000","last_peered":"2024-03-09T15:36:48.293213+0000","last_clean":"2024-03-09T15:36:48.293213+0000","last_became_active":"2024-03-09T14:28:34.296660+0000","last_became_peered":"2024-03-09T14:28:34.296660+0000","last_unstale":"2024-03-09T15:36:48.293213+0000","last_undegraded":"2024-03-09T15:36:48.293213+0000","last_fullsized":"2024-03-09T15:36:48.293213+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:48.292984+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:48.292984+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.292984+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:58:19.981031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":129,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"261'157","reported_seq":796,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.182597+0000","last_change":"2024-03-09T15:36:44.182693+0000","last_active":"2024-03-09T15:36:44.182597+0000","last_peered":"2024-03-09T15:36:44.182597+0000","last_clean":"2024-03-09T15:36:44.182597+0000","last_became_active":"2024-03-09T14:28:32.305318+0000","last_became_peered":"2024-03-09T14:28:32.305318+0000","last_unstale":"2024-03-09T15:36:44.182597+0000","last_undegraded":"2024-03-09T15:36:44.182597+0000","last_fullsized":"2024-03-09T15:36:44.182597+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'157","last_scrub_stamp":"2024-03-09T15:36:44.182405+0000","last_deep_scrub":"261'157","last_deep_scrub_stamp":"2024-03-09T15:36:44.182405+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.182405+0000","objects_scrubbed":9,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:14:02.068526+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262245,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":357,"num_read_kb":10578,"num_write":456,"num_write_kb":20505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":533,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:56.079923+0000","last_change":"2024-03-09T15:36:56.079981+0000","last_active":"2024-03-09T15:36:56.079923+0000","last_peered":"2024-03-09T15:36:56.079923+0000","last_clean":"2024-03-09T15:36:56.079923+0000","last_became_active":"2024-03-09T14:28:19.240704+0000","last_became_peered":"2024-03-09T14:28:19.240704+0000","last_unstale":"2024-03-09T15:36:56.079923+0000","last_undegraded":"2024-03-09T15:36:56.079923+0000","last_fullsized":"2024-03-09T15:36:56.079923+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:56.079740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:56.079740+0000","last_clean_scrub_stamp":"2024-03-09T15:36:56.079740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:46:45.156526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.191317+0000","last_change":"2024-03-09T15:36:25.191369+0000","last_active":"2024-03-09T15:36:25.191317+0000","last_peered":"2024-03-09T15:36:25.191317+0000","last_clean":"2024-03-09T15:36:25.191317+0000","last_became_active":"2024-03-09T14:28:36.314676+0000","last_became_peered":"2024-03-09T14:28:36.314676+0000","last_unstale":"2024-03-09T15:36:25.191317+0000","last_undegraded":"2024-03-09T15:36:25.191317+0000","last_fullsized":"2024-03-09T15:36:25.191317+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.191100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.191100+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.191100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:12:04.217794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"261'205","reported_seq":795,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.476798+0000","last_change":"2024-03-09T15:36:35.476874+0000","last_active":"2024-03-09T15:36:35.476798+0000","last_peered":"2024-03-09T15:36:35.476798+0000","last_clean":"2024-03-09T15:36:35.476798+0000","last_became_active":"2024-03-09T14:28:38.595028+0000","last_became_peered":"2024-03-09T14:28:38.595028+0000","last_unstale":"2024-03-09T15:36:35.476798+0000","last_undegraded":"2024-03-09T15:36:35.476798+0000","last_fullsized":"2024-03-09T15:36:35.476798+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'205","last_scrub_stamp":"2024-03-09T15:36:35.476600+0000","last_deep_scrub":"261'205","last_deep_scrub_stamp":"2024-03-09T15:36:35.476600+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.476600+0000","objects_scrubbed":8,"log_size":105,"log_dups_size":100,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:50:56.058991+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":269,"num_read_kb":262,"num_write":230,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":330,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"261'501","reported_seq":2307,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.727185+0000","last_change":"2024-03-09T15:36:23.727185+0000","last_active":"2024-03-09T15:36:23.727185+0000","last_peered":"2024-03-09T15:36:23.727185+0000","last_clean":"2024-03-09T15:36:23.727185+0000","last_became_active":"2024-03-09T14:28:40.369165+0000","last_became_peered":"2024-03-09T14:28:40.369165+0000","last_unstale":"2024-03-09T15:36:23.727185+0000","last_undegraded":"2024-03-09T15:36:23.727185+0000","last_fullsized":"2024-03-09T15:36:23.727185+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'501","last_scrub_stamp":"2024-03-09T15:36:23.726932+0000","last_deep_scrub":"261'501","last_deep_scrub_stamp":"2024-03-09T15:36:23.726932+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.726932+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":400,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:05:11.145403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"261'79","reported_seq":577,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.487857+0000","last_change":"2024-03-09T15:36:27.487857+0000","last_active":"2024-03-09T15:36:27.487857+0000","last_peered":"2024-03-09T15:36:27.487857+0000","last_clean":"2024-03-09T15:36:27.487857+0000","last_became_active":"2024-03-09T14:28:42.392402+0000","last_became_peered":"2024-03-09T14:28:42.392402+0000","last_unstale":"2024-03-09T15:36:27.487857+0000","last_undegraded":"2024-03-09T15:36:27.487857+0000","last_fullsized":"2024-03-09T15:36:27.487857+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'79","last_scrub_stamp":"2024-03-09T15:36:27.487564+0000","last_deep_scrub":"261'79","last_deep_scrub_stamp":"2024-03-09T15:36:27.487564+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.487564+0000","objects_scrubbed":3,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:09:13.446303+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":57,"num_write":172,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"261'48","reported_seq":598,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.188879+0000","last_change":"2024-03-09T15:36:18.188944+0000","last_active":"2024-03-09T15:36:18.188879+0000","last_peered":"2024-03-09T15:36:18.188879+0000","last_clean":"2024-03-09T15:36:18.188879+0000","last_became_active":"2024-03-09T14:29:06.440556+0000","last_became_peered":"2024-03-09T14:29:06.440556+0000","last_unstale":"2024-03-09T15:36:18.188879+0000","last_undegraded":"2024-03-09T15:36:18.188879+0000","last_fullsized":"2024-03-09T15:36:18.188879+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:18.188663+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:18.188663+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.188663+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:03:02.670504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":216,"num_read_kb":182,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"261'150","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:06.778364+0000","last_change":"2024-03-09T15:37:06.778452+0000","last_active":"2024-03-09T15:37:06.778364+0000","last_peered":"2024-03-09T15:37:06.778364+0000","last_clean":"2024-03-09T15:37:06.778364+0000","last_became_active":"2024-03-09T14:28:32.318971+0000","last_became_peered":"2024-03-09T14:28:32.318971+0000","last_unstale":"2024-03-09T15:37:06.778364+0000","last_undegraded":"2024-03-09T15:37:06.778364+0000","last_fullsized":"2024-03-09T15:37:06.778364+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'150","last_scrub_stamp":"2024-03-09T15:37:06.778183+0000","last_deep_scrub":"261'150","last_deep_scrub_stamp":"2024-03-09T15:37:06.778183+0000","last_clean_scrub_stamp":"2024-03-09T15:37:06.778183+0000","objects_scrubbed":5,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:15:24.839103+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":151,"num_read_kb":8647,"num_write":476,"num_write_kb":27350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":528,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.293647+0000","last_change":"2024-03-09T15:36:40.293706+0000","last_active":"2024-03-09T15:36:40.293647+0000","last_peered":"2024-03-09T15:36:40.293647+0000","last_clean":"2024-03-09T15:36:40.293647+0000","last_became_active":"2024-03-09T14:28:19.237750+0000","last_became_peered":"2024-03-09T14:28:19.237750+0000","last_unstale":"2024-03-09T15:36:40.293647+0000","last_undegraded":"2024-03-09T15:36:40.293647+0000","last_fullsized":"2024-03-09T15:36:40.293647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:40.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:40.293440+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:43:40.824370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":518,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:55.494253+0000","last_change":"2024-03-09T15:36:55.494308+0000","last_active":"2024-03-09T15:36:55.494253+0000","last_peered":"2024-03-09T15:36:55.494253+0000","last_clean":"2024-03-09T15:36:55.494253+0000","last_became_active":"2024-03-09T14:28:36.327601+0000","last_became_peered":"2024-03-09T14:28:36.327601+0000","last_unstale":"2024-03-09T15:36:55.494253+0000","last_undegraded":"2024-03-09T15:36:55.494253+0000","last_fullsized":"2024-03-09T15:36:55.494253+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:55.494042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:55.494042+0000","last_clean_scrub_stamp":"2024-03-09T15:36:55.494042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:15:02.668845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"261'111","reported_seq":675,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.173092+0000","last_change":"2024-03-09T15:36:26.173180+0000","last_active":"2024-03-09T15:36:26.173092+0000","last_peered":"2024-03-09T15:36:26.173092+0000","last_clean":"2024-03-09T15:36:26.173092+0000","last_became_active":"2024-03-09T14:28:38.347750+0000","last_became_peered":"2024-03-09T14:28:38.347750+0000","last_unstale":"2024-03-09T15:36:26.173092+0000","last_undegraded":"2024-03-09T15:36:26.173092+0000","last_fullsized":"2024-03-09T15:36:26.173092+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'111","last_scrub_stamp":"2024-03-09T15:36:26.172900+0000","last_deep_scrub":"261'111","last_deep_scrub_stamp":"2024-03-09T15:36:26.172900+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.172900+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:00:14.752406+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"261'514","reported_seq":2495,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.486338+0000","last_change":"2024-03-09T15:36:34.486405+0000","last_active":"2024-03-09T15:36:34.486338+0000","last_peered":"2024-03-09T15:36:34.486338+0000","last_clean":"2024-03-09T15:36:34.486338+0000","last_became_active":"2024-03-09T14:28:40.386859+0000","last_became_peered":"2024-03-09T14:28:40.386859+0000","last_unstale":"2024-03-09T15:36:34.486338+0000","last_undegraded":"2024-03-09T15:36:34.486338+0000","last_fullsized":"2024-03-09T15:36:34.486338+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'514","last_scrub_stamp":"2024-03-09T15:36:34.486115+0000","last_deep_scrub":"261'514","last_deep_scrub_stamp":"2024-03-09T15:36:34.486115+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.486115+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":500,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:43:00.093432+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"261'108","reported_seq":618,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.271001+0000","last_change":"2024-03-09T15:36:25.271065+0000","last_active":"2024-03-09T15:36:25.271001+0000","last_peered":"2024-03-09T15:36:25.271001+0000","last_clean":"2024-03-09T15:36:25.271001+0000","last_became_active":"2024-03-09T14:28:42.406139+0000","last_became_peered":"2024-03-09T14:28:42.406139+0000","last_unstale":"2024-03-09T15:36:25.271001+0000","last_undegraded":"2024-03-09T15:36:25.271001+0000","last_fullsized":"2024-03-09T15:36:25.271001+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'108","last_scrub_stamp":"2024-03-09T15:36:25.270811+0000","last_deep_scrub":"261'108","last_deep_scrub_stamp":"2024-03-09T15:36:25.270811+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.270811+0000","objects_scrubbed":2,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:05:43.169466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":65,"num_write":235,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"261'217","reported_seq":797,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:52.498890+0000","last_change":"2024-03-09T15:36:52.498953+0000","last_active":"2024-03-09T15:36:52.498890+0000","last_peered":"2024-03-09T15:36:52.498890+0000","last_clean":"2024-03-09T15:36:52.498890+0000","last_became_active":"2024-03-09T14:28:34.299988+0000","last_became_peered":"2024-03-09T14:28:34.299988+0000","last_unstale":"2024-03-09T15:36:52.498890+0000","last_undegraded":"2024-03-09T15:36:52.498890+0000","last_fullsized":"2024-03-09T15:36:52.498890+0000","mapping_epoch":22,"log_start":"258'200","ondisk_log_start":"258'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'217","last_scrub_stamp":"2024-03-09T15:36:52.498694+0000","last_deep_scrub":"261'217","last_deep_scrub_stamp":"2024-03-09T15:36:52.498694+0000","last_clean_scrub_stamp":"2024-03-09T15:36:52.498694+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":200,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:46:08.054569+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":843,"num_read_kb":645,"num_write":377,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":268,"num_omap_keys":1,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"261'154","reported_seq":716,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:02.150046+0000","last_change":"2024-03-09T15:37:02.150218+0000","last_active":"2024-03-09T15:37:02.150046+0000","last_peered":"2024-03-09T15:37:02.150046+0000","last_clean":"2024-03-09T15:37:02.150046+0000","last_became_active":"2024-03-09T14:28:32.298977+0000","last_became_peered":"2024-03-09T14:28:32.298977+0000","last_unstale":"2024-03-09T15:37:02.150046+0000","last_undegraded":"2024-03-09T15:37:02.150046+0000","last_fullsized":"2024-03-09T15:37:02.150046+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'154","last_scrub_stamp":"2024-03-09T15:37:02.149792+0000","last_deep_scrub":"261'154","last_deep_scrub_stamp":"2024-03-09T15:37:02.149792+0000","last_clean_scrub_stamp":"2024-03-09T15:37:02.149792+0000","objects_scrubbed":4,"log_size":54,"log_dups_size":100,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:49:58.935130+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242905,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":173,"num_read_kb":5781,"num_write":510,"num_write_kb":37567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.336163+0000","last_change":"2024-03-09T15:36:36.336215+0000","last_active":"2024-03-09T15:36:36.336163+0000","last_peered":"2024-03-09T15:36:36.336163+0000","last_clean":"2024-03-09T15:36:36.336163+0000","last_became_active":"2024-03-09T14:28:19.241700+0000","last_became_peered":"2024-03-09T14:28:19.241700+0000","last_unstale":"2024-03-09T15:36:36.336163+0000","last_undegraded":"2024-03-09T15:36:36.336163+0000","last_fullsized":"2024-03-09T15:36:36.336163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:36.335976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:36.335976+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.335976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:25:25.910931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"261'4","reported_seq":833,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.712424+0000","last_change":"2024-03-09T15:36:51.712560+0000","last_active":"2024-03-09T15:36:51.712424+0000","last_peered":"2024-03-09T15:36:51.712424+0000","last_clean":"2024-03-09T15:36:51.712424+0000","last_became_active":"2024-03-09T14:28:36.317327+0000","last_became_peered":"2024-03-09T14:28:36.317327+0000","last_unstale":"2024-03-09T15:36:51.712424+0000","last_undegraded":"2024-03-09T15:36:51.712424+0000","last_fullsized":"2024-03-09T15:36:51.712424+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'4","last_scrub_stamp":"2024-03-09T15:36:51.712182+0000","last_deep_scrub":"261'4","last_deep_scrub_stamp":"2024-03-09T15:36:51.712182+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.712182+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:06:10.620608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":484,"num_read_kb":482,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"261'478","reported_seq":1226,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.248700+0000","last_change":"2024-03-09T15:36:38.248788+0000","last_active":"2024-03-09T15:36:38.248700+0000","last_peered":"2024-03-09T15:36:38.248700+0000","last_clean":"2024-03-09T15:36:38.248700+0000","last_became_active":"2024-03-09T14:28:38.602908+0000","last_became_peered":"2024-03-09T14:28:38.602908+0000","last_unstale":"2024-03-09T15:36:38.248700+0000","last_undegraded":"2024-03-09T15:36:38.248700+0000","last_fullsized":"2024-03-09T15:36:38.248700+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'478","last_scrub_stamp":"2024-03-09T15:36:38.248482+0000","last_deep_scrub":"261'478","last_deep_scrub_stamp":"2024-03-09T15:36:38.248482+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.248482+0000","objects_scrubbed":8,"log_size":78,"log_dups_size":400,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:50:57.022028+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29305,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":797,"num_read_kb":1561,"num_write":550,"num_write_kb":344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.482306+0000","last_change":"2024-03-09T15:36:35.482326+0000","last_active":"2024-03-09T15:36:35.482306+0000","last_peered":"2024-03-09T15:36:35.482306+0000","last_clean":"2024-03-09T15:36:35.482306+0000","last_became_active":"2024-03-09T14:28:40.375884+0000","last_became_peered":"2024-03-09T14:28:40.375884+0000","last_unstale":"2024-03-09T15:36:35.482306+0000","last_undegraded":"2024-03-09T15:36:35.482306+0000","last_fullsized":"2024-03-09T15:36:35.482306+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:35.482232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:35.482232+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.482232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:57:53.662951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"261'56","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.577103+0000","last_change":"2024-03-09T15:36:13.577103+0000","last_active":"2024-03-09T15:36:13.577103+0000","last_peered":"2024-03-09T15:36:13.577103+0000","last_clean":"2024-03-09T15:36:13.577103+0000","last_became_active":"2024-03-09T14:28:42.405998+0000","last_became_peered":"2024-03-09T14:28:42.405998+0000","last_unstale":"2024-03-09T15:36:13.577103+0000","last_undegraded":"2024-03-09T15:36:13.577103+0000","last_fullsized":"2024-03-09T15:36:13.577103+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:13.576825+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:13.576825+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.576825+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:29:57.437161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":52,"num_write":123,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1f","version":"261'149","reported_seq":735,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.301854+0000","last_change":"2024-03-09T15:36:24.301934+0000","last_active":"2024-03-09T15:36:24.301854+0000","last_peered":"2024-03-09T15:36:24.301854+0000","last_clean":"2024-03-09T15:36:24.301854+0000","last_became_active":"2024-03-09T14:28:42.408774+0000","last_became_peered":"2024-03-09T14:28:42.408774+0000","last_unstale":"2024-03-09T15:36:24.301854+0000","last_undegraded":"2024-03-09T15:36:24.301854+0000","last_fullsized":"2024-03-09T15:36:24.301854+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'149","last_scrub_stamp":"2024-03-09T15:36:24.301670+0000","last_deep_scrub":"261'149","last_deep_scrub_stamp":"2024-03-09T15:36:24.301670+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.301670+0000","objects_scrubbed":3,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:31:20.109756+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":256,"num_read_kb":233,"num_write":254,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"261'183","reported_seq":753,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.152146+0000","last_change":"2024-03-09T15:36:32.152205+0000","last_active":"2024-03-09T15:36:32.152146+0000","last_peered":"2024-03-09T15:36:32.152146+0000","last_clean":"2024-03-09T15:36:32.152146+0000","last_became_active":"2024-03-09T14:28:34.314893+0000","last_became_peered":"2024-03-09T14:28:34.314893+0000","last_unstale":"2024-03-09T15:36:32.152146+0000","last_undegraded":"2024-03-09T15:36:32.152146+0000","last_fullsized":"2024-03-09T15:36:32.152146+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'183","last_scrub_stamp":"2024-03-09T15:36:32.151949+0000","last_deep_scrub":"261'183","last_deep_scrub_stamp":"2024-03-09T15:36:32.151949+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.151949+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":100,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:40:37.495594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":706,"num_read_kb":592,"num_write":291,"num_write_kb":155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"261'132","reported_seq":686,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.480459+0000","last_change":"2024-03-09T15:36:50.480551+0000","last_active":"2024-03-09T15:36:50.480459+0000","last_peered":"2024-03-09T15:36:50.480459+0000","last_clean":"2024-03-09T15:36:50.480459+0000","last_became_active":"2024-03-09T14:28:32.322364+0000","last_became_peered":"2024-03-09T14:28:32.322364+0000","last_unstale":"2024-03-09T15:36:50.480459+0000","last_undegraded":"2024-03-09T15:36:50.480459+0000","last_fullsized":"2024-03-09T15:36:50.480459+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'132","last_scrub_stamp":"2024-03-09T15:36:50.480252+0000","last_deep_scrub":"261'132","last_deep_scrub_stamp":"2024-03-09T15:36:50.480252+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.480252+0000","objects_scrubbed":3,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:46:23.715589+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":128,"num_write":389,"num_write_kb":27813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":512,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.576225+0000","last_change":"2024-03-09T15:36:25.576277+0000","last_active":"2024-03-09T15:36:25.576225+0000","last_peered":"2024-03-09T15:36:25.576225+0000","last_clean":"2024-03-09T15:36:25.576225+0000","last_became_active":"2024-03-09T14:28:36.313984+0000","last_became_peered":"2024-03-09T14:28:36.313984+0000","last_unstale":"2024-03-09T15:36:25.576225+0000","last_undegraded":"2024-03-09T15:36:25.576225+0000","last_fullsized":"2024-03-09T15:36:25.576225+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.576027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.576027+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.576027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:16:03.346315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"261'196","reported_seq":788,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:07.771948+0000","last_change":"2024-03-09T15:37:07.772039+0000","last_active":"2024-03-09T15:37:07.771948+0000","last_peered":"2024-03-09T15:37:07.771948+0000","last_clean":"2024-03-09T15:37:07.771948+0000","last_became_active":"2024-03-09T14:28:38.338262+0000","last_became_peered":"2024-03-09T14:28:38.338262+0000","last_unstale":"2024-03-09T15:37:07.771948+0000","last_undegraded":"2024-03-09T15:37:07.771948+0000","last_fullsized":"2024-03-09T15:37:07.771948+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'196","last_scrub_stamp":"2024-03-09T15:37:07.771742+0000","last_deep_scrub":"261'196","last_deep_scrub_stamp":"2024-03-09T15:37:07.771742+0000","last_clean_scrub_stamp":"2024-03-09T15:37:07.771742+0000","objects_scrubbed":5,"log_size":96,"log_dups_size":100,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:34:19.385252+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15994,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":315,"num_read_kb":601,"num_write":220,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.450333+0000","last_change":"2024-03-09T15:36:31.450390+0000","last_active":"2024-03-09T15:36:31.450333+0000","last_peered":"2024-03-09T15:36:31.450333+0000","last_clean":"2024-03-09T15:36:31.450333+0000","last_became_active":"2024-03-09T14:28:40.389277+0000","last_became_peered":"2024-03-09T14:28:40.389277+0000","last_unstale":"2024-03-09T15:36:31.450333+0000","last_undegraded":"2024-03-09T15:36:31.450333+0000","last_fullsized":"2024-03-09T15:36:31.450333+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:31.450128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:31.450128+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.450128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:52:04.464244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"261'56","reported_seq":561,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.467616+0000","last_change":"2024-03-09T15:36:22.467616+0000","last_active":"2024-03-09T15:36:22.467616+0000","last_peered":"2024-03-09T15:36:22.467616+0000","last_clean":"2024-03-09T15:36:22.467616+0000","last_became_active":"2024-03-09T14:28:42.409229+0000","last_became_peered":"2024-03-09T14:28:42.409229+0000","last_unstale":"2024-03-09T15:36:22.467616+0000","last_undegraded":"2024-03-09T15:36:22.467616+0000","last_fullsized":"2024-03-09T15:36:22.467616+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:22.467355+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:22.467355+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.467355+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:46:09.238574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":828,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":44,"num_write":116,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"261'78","reported_seq":640,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.558934+0000","last_change":"2024-03-09T15:36:26.558992+0000","last_active":"2024-03-09T15:36:26.558934+0000","last_peered":"2024-03-09T15:36:26.558934+0000","last_clean":"2024-03-09T15:36:26.558934+0000","last_became_active":"2024-03-09T14:28:34.320474+0000","last_became_peered":"2024-03-09T14:28:34.320474+0000","last_unstale":"2024-03-09T15:36:26.558934+0000","last_undegraded":"2024-03-09T15:36:26.558934+0000","last_fullsized":"2024-03-09T15:36:26.558934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'78","last_scrub_stamp":"2024-03-09T15:36:26.558741+0000","last_deep_scrub":"261'78","last_deep_scrub_stamp":"2024-03-09T15:36:26.558741+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.558741+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:58:25.865037+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":231,"num_write":128,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"261'163","reported_seq":805,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:00.149520+0000","last_change":"2024-03-09T15:37:00.149647+0000","last_active":"2024-03-09T15:37:00.149520+0000","last_peered":"2024-03-09T15:37:00.149520+0000","last_clean":"2024-03-09T15:37:00.149520+0000","last_became_active":"2024-03-09T14:28:32.315355+0000","last_became_peered":"2024-03-09T14:28:32.315355+0000","last_unstale":"2024-03-09T15:37:00.149520+0000","last_undegraded":"2024-03-09T15:37:00.149520+0000","last_fullsized":"2024-03-09T15:37:00.149520+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'163","last_scrub_stamp":"2024-03-09T15:37:00.149328+0000","last_deep_scrub":"261'163","last_deep_scrub_stamp":"2024-03-09T15:37:00.149328+0000","last_clean_scrub_stamp":"2024-03-09T15:37:00.149328+0000","objects_scrubbed":4,"log_size":63,"log_dups_size":100,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:05:07.700021+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":325,"num_read_kb":85989,"num_write":581,"num_write_kb":44811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.649651+0000","last_change":"2024-03-09T15:36:49.649702+0000","last_active":"2024-03-09T15:36:49.649651+0000","last_peered":"2024-03-09T15:36:49.649651+0000","last_clean":"2024-03-09T15:36:49.649651+0000","last_became_active":"2024-03-09T14:28:36.311487+0000","last_became_peered":"2024-03-09T14:28:36.311487+0000","last_unstale":"2024-03-09T15:36:49.649651+0000","last_undegraded":"2024-03-09T15:36:49.649651+0000","last_fullsized":"2024-03-09T15:36:49.649651+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:49.649459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:49.649459+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.649459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:35:37.031126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"261'42","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:05.098952+0000","last_change":"2024-03-09T15:37:05.099025+0000","last_active":"2024-03-09T15:37:05.098952+0000","last_peered":"2024-03-09T15:37:05.098952+0000","last_clean":"2024-03-09T15:37:05.098952+0000","last_became_active":"2024-03-09T14:28:38.332610+0000","last_became_peered":"2024-03-09T14:28:38.332610+0000","last_unstale":"2024-03-09T15:37:05.098952+0000","last_undegraded":"2024-03-09T15:37:05.098952+0000","last_fullsized":"2024-03-09T15:37:05.098952+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'42","last_scrub_stamp":"2024-03-09T15:37:05.098734+0000","last_deep_scrub":"261'42","last_deep_scrub_stamp":"2024-03-09T15:37:05.098734+0000","last_clean_scrub_stamp":"2024-03-09T15:37:05.098734+0000","objects_scrubbed":3,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:23:34.461015+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":41,"num_write":30,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":294,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.258023+0000","last_change":"2024-03-09T15:36:35.258136+0000","last_active":"2024-03-09T15:36:35.258023+0000","last_peered":"2024-03-09T15:36:35.258023+0000","last_clean":"2024-03-09T15:36:35.258023+0000","last_became_active":"2024-03-09T14:28:40.367795+0000","last_became_peered":"2024-03-09T14:28:40.367795+0000","last_unstale":"2024-03-09T15:36:35.258023+0000","last_undegraded":"2024-03-09T15:36:35.258023+0000","last_fullsized":"2024-03-09T15:36:35.258023+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:35.257744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:35.257744+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.257744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:42:18.958156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"261'172","reported_seq":700,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.304337+0000","last_change":"2024-03-09T15:36:22.304413+0000","last_active":"2024-03-09T15:36:22.304337+0000","last_peered":"2024-03-09T15:36:22.304337+0000","last_clean":"2024-03-09T15:36:22.304337+0000","last_became_active":"2024-03-09T14:28:42.408913+0000","last_became_peered":"2024-03-09T14:28:42.408913+0000","last_unstale":"2024-03-09T15:36:22.304337+0000","last_undegraded":"2024-03-09T15:36:22.304337+0000","last_fullsized":"2024-03-09T15:36:22.304337+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'172","last_scrub_stamp":"2024-03-09T15:36:22.304104+0000","last_deep_scrub":"261'172","last_deep_scrub_stamp":"2024-03-09T15:36:22.304104+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.304104+0000","objects_scrubbed":2,"log_size":72,"log_dups_size":100,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:45:45.379355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":208,"num_write":351,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":398,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"261'54","reported_seq":603,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.672674+0000","last_change":"2024-03-09T15:36:47.672732+0000","last_active":"2024-03-09T15:36:47.672674+0000","last_peered":"2024-03-09T15:36:47.672674+0000","last_clean":"2024-03-09T15:36:47.672674+0000","last_became_active":"2024-03-09T14:28:34.330813+0000","last_became_peered":"2024-03-09T14:28:34.330813+0000","last_unstale":"2024-03-09T15:36:47.672674+0000","last_undegraded":"2024-03-09T15:36:47.672674+0000","last_fullsized":"2024-03-09T15:36:47.672674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'54","last_scrub_stamp":"2024-03-09T15:36:47.672474+0000","last_deep_scrub":"261'54","last_deep_scrub_stamp":"2024-03-09T15:36:47.672474+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.672474+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:48:25.059611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":220,"num_read_kb":178,"num_write":76,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"261'133","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.908045+0000","last_change":"2024-03-09T15:36:41.908262+0000","last_active":"2024-03-09T15:36:41.908045+0000","last_peered":"2024-03-09T15:36:41.908045+0000","last_clean":"2024-03-09T15:36:41.908045+0000","last_became_active":"2024-03-09T14:28:32.311843+0000","last_became_peered":"2024-03-09T14:28:32.311843+0000","last_unstale":"2024-03-09T15:36:41.908045+0000","last_undegraded":"2024-03-09T15:36:41.908045+0000","last_fullsized":"2024-03-09T15:36:41.908045+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'133","last_scrub_stamp":"2024-03-09T15:36:41.907782+0000","last_deep_scrub":"261'133","last_deep_scrub_stamp":"2024-03-09T15:36:41.907782+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.907782+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:05:53.440636+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655838,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":157,"num_read_kb":5348,"num_write":475,"num_write_kb":35385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.141589+0000","last_change":"2024-03-09T15:36:43.141654+0000","last_active":"2024-03-09T15:36:43.141589+0000","last_peered":"2024-03-09T15:36:43.141589+0000","last_clean":"2024-03-09T15:36:43.141589+0000","last_became_active":"2024-03-09T14:28:36.323335+0000","last_became_peered":"2024-03-09T14:28:36.323335+0000","last_unstale":"2024-03-09T15:36:43.141589+0000","last_undegraded":"2024-03-09T15:36:43.141589+0000","last_fullsized":"2024-03-09T15:36:43.141589+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:43.141360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:43.141360+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.141360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:29:28.525098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"261'227","reported_seq":834,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.585747+0000","last_change":"2024-03-09T15:36:42.585928+0000","last_active":"2024-03-09T15:36:42.585747+0000","last_peered":"2024-03-09T15:36:42.585747+0000","last_clean":"2024-03-09T15:36:42.585747+0000","last_became_active":"2024-03-09T14:28:38.344951+0000","last_became_peered":"2024-03-09T14:28:38.344951+0000","last_unstale":"2024-03-09T15:36:42.585747+0000","last_undegraded":"2024-03-09T15:36:42.585747+0000","last_fullsized":"2024-03-09T15:36:42.585747+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'227","last_scrub_stamp":"2024-03-09T15:36:42.585503+0000","last_deep_scrub":"261'227","last_deep_scrub_stamp":"2024-03-09T15:36:42.585503+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.585503+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":200,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:09:44.534942+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7791,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":353,"num_read_kb":629,"num_write":244,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.128626+0000","last_change":"2024-03-09T15:36:22.128680+0000","last_active":"2024-03-09T15:36:22.128626+0000","last_peered":"2024-03-09T15:36:22.128626+0000","last_clean":"2024-03-09T15:36:22.128626+0000","last_became_active":"2024-03-09T14:28:40.370313+0000","last_became_peered":"2024-03-09T14:28:40.370313+0000","last_unstale":"2024-03-09T15:36:22.128626+0000","last_undegraded":"2024-03-09T15:36:22.128626+0000","last_fullsized":"2024-03-09T15:36:22.128626+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.128436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.128436+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.128436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:31:27.821903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"261'47","reported_seq":550,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.629026+0000","last_change":"2024-03-09T15:36:15.629026+0000","last_active":"2024-03-09T15:36:15.629026+0000","last_peered":"2024-03-09T15:36:15.629026+0000","last_clean":"2024-03-09T15:36:15.629026+0000","last_became_active":"2024-03-09T14:28:42.404360+0000","last_became_peered":"2024-03-09T14:28:42.404360+0000","last_unstale":"2024-03-09T15:36:15.629026+0000","last_undegraded":"2024-03-09T15:36:15.629026+0000","last_fullsized":"2024-03-09T15:36:15.629026+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'47","last_scrub_stamp":"2024-03-09T15:36:15.628774+0000","last_deep_scrub":"261'47","last_deep_scrub_stamp":"2024-03-09T15:36:15.628774+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.628774+0000","objects_scrubbed":2,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:04:15.376099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":43,"num_write":116,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"261'88","reported_seq":651,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.697326+0000","last_change":"2024-03-09T15:36:50.697389+0000","last_active":"2024-03-09T15:36:50.697326+0000","last_peered":"2024-03-09T15:36:50.697326+0000","last_clean":"2024-03-09T15:36:50.697326+0000","last_became_active":"2024-03-09T14:28:34.305301+0000","last_became_peered":"2024-03-09T14:28:34.305301+0000","last_unstale":"2024-03-09T15:36:50.697326+0000","last_undegraded":"2024-03-09T15:36:50.697326+0000","last_fullsized":"2024-03-09T15:36:50.697326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'88","last_scrub_stamp":"2024-03-09T15:36:50.697133+0000","last_deep_scrub":"261'88","last_deep_scrub_stamp":"2024-03-09T15:36:50.697133+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.697133+0000","objects_scrubbed":1,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:31:45.483994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":317,"num_write":145,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2876,"num_omap_keys":12,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"261'149","reported_seq":715,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:58.170601+0000","last_change":"2024-03-09T15:36:58.170723+0000","last_active":"2024-03-09T15:36:58.170601+0000","last_peered":"2024-03-09T15:36:58.170601+0000","last_clean":"2024-03-09T15:36:58.170601+0000","last_became_active":"2024-03-09T14:28:32.292393+0000","last_became_peered":"2024-03-09T14:28:32.292393+0000","last_unstale":"2024-03-09T15:36:58.170601+0000","last_undegraded":"2024-03-09T15:36:58.170601+0000","last_fullsized":"2024-03-09T15:36:58.170601+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'149","last_scrub_stamp":"2024-03-09T15:36:58.170403+0000","last_deep_scrub":"261'149","last_deep_scrub_stamp":"2024-03-09T15:36:58.170403+0000","last_clean_scrub_stamp":"2024-03-09T15:36:58.170403+0000","objects_scrubbed":6,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:06:06.109922+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6900333,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":188,"num_read_kb":14616,"num_write":468,"num_write_kb":23484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":511,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.124875+0000","last_change":"2024-03-09T15:36:21.124931+0000","last_active":"2024-03-09T15:36:21.124875+0000","last_peered":"2024-03-09T15:36:21.124875+0000","last_clean":"2024-03-09T15:36:21.124875+0000","last_became_active":"2024-03-09T14:28:36.328946+0000","last_became_peered":"2024-03-09T14:28:36.328946+0000","last_unstale":"2024-03-09T15:36:21.124875+0000","last_undegraded":"2024-03-09T15:36:21.124875+0000","last_fullsized":"2024-03-09T15:36:21.124875+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.124678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.124678+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.124678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:16:00.704022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"261'172","reported_seq":766,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.110832+0000","last_change":"2024-03-09T15:36:49.110993+0000","last_active":"2024-03-09T15:36:49.110832+0000","last_peered":"2024-03-09T15:36:49.110832+0000","last_clean":"2024-03-09T15:36:49.110832+0000","last_became_active":"2024-03-09T14:28:38.335107+0000","last_became_peered":"2024-03-09T14:28:38.335107+0000","last_unstale":"2024-03-09T15:36:49.110832+0000","last_undegraded":"2024-03-09T15:36:49.110832+0000","last_fullsized":"2024-03-09T15:36:49.110832+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'172","last_scrub_stamp":"2024-03-09T15:36:49.110544+0000","last_deep_scrub":"261'172","last_deep_scrub_stamp":"2024-03-09T15:36:49.110544+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.110544+0000","objects_scrubbed":4,"log_size":72,"log_dups_size":100,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:18:50.261252+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9765,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":273,"num_read_kb":518,"num_write":186,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.274828+0000","last_change":"2024-03-09T15:36:36.274880+0000","last_active":"2024-03-09T15:36:36.274828+0000","last_peered":"2024-03-09T15:36:36.274828+0000","last_clean":"2024-03-09T15:36:36.274828+0000","last_became_active":"2024-03-09T14:28:40.363778+0000","last_became_peered":"2024-03-09T14:28:40.363778+0000","last_unstale":"2024-03-09T15:36:36.274828+0000","last_undegraded":"2024-03-09T15:36:36.274828+0000","last_fullsized":"2024-03-09T15:36:36.274828+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:36.274642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:36.274642+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.274642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:53:47.883218+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"261'159","reported_seq":742,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.206354+0000","last_change":"2024-03-09T15:36:29.206413+0000","last_active":"2024-03-09T15:36:29.206354+0000","last_peered":"2024-03-09T15:36:29.206354+0000","last_clean":"2024-03-09T15:36:29.206354+0000","last_became_active":"2024-03-09T14:28:34.325031+0000","last_became_peered":"2024-03-09T14:28:34.325031+0000","last_unstale":"2024-03-09T15:36:29.206354+0000","last_undegraded":"2024-03-09T15:36:29.206354+0000","last_fullsized":"2024-03-09T15:36:29.206354+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'159","last_scrub_stamp":"2024-03-09T15:36:29.206170+0000","last_deep_scrub":"261'159","last_deep_scrub_stamp":"2024-03-09T15:36:29.206170+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.206170+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:05:19.071599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":584,"num_read_kb":461,"num_write":254,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"261'118","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:00.714054+0000","last_change":"2024-03-09T15:37:00.714149+0000","last_active":"2024-03-09T15:37:00.714054+0000","last_peered":"2024-03-09T15:37:00.714054+0000","last_clean":"2024-03-09T15:37:00.714054+0000","last_became_active":"2024-03-09T14:28:32.296179+0000","last_became_peered":"2024-03-09T14:28:32.296179+0000","last_unstale":"2024-03-09T15:37:00.714054+0000","last_undegraded":"2024-03-09T15:37:00.714054+0000","last_fullsized":"2024-03-09T15:37:00.714054+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'118","last_scrub_stamp":"2024-03-09T15:37:00.713835+0000","last_deep_scrub":"261'118","last_deep_scrub_stamp":"2024-03-09T15:37:00.713835+0000","last_clean_scrub_stamp":"2024-03-09T15:37:00.713835+0000","objects_scrubbed":7,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:51:35.012606+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263265,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":136,"num_read_kb":11385,"num_write":371,"num_write_kb":17181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.269733+0000","last_change":"2024-03-09T15:36:51.269792+0000","last_active":"2024-03-09T15:36:51.269733+0000","last_peered":"2024-03-09T15:36:51.269733+0000","last_clean":"2024-03-09T15:36:51.269733+0000","last_became_active":"2024-03-09T14:28:36.319702+0000","last_became_peered":"2024-03-09T14:28:36.319702+0000","last_unstale":"2024-03-09T15:36:51.269733+0000","last_undegraded":"2024-03-09T15:36:51.269733+0000","last_fullsized":"2024-03-09T15:36:51.269733+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:51.269512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:51.269512+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.269512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:30:38.384018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"261'316","reported_seq":965,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.480382+0000","last_change":"2024-03-09T15:36:33.480469+0000","last_active":"2024-03-09T15:36:33.480382+0000","last_peered":"2024-03-09T15:36:33.480382+0000","last_clean":"2024-03-09T15:36:33.480382+0000","last_became_active":"2024-03-09T14:28:38.332661+0000","last_became_peered":"2024-03-09T14:28:38.332661+0000","last_unstale":"2024-03-09T15:36:33.480382+0000","last_undegraded":"2024-03-09T15:36:33.480382+0000","last_fullsized":"2024-03-09T15:36:33.480382+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'316","last_scrub_stamp":"2024-03-09T15:36:33.480185+0000","last_deep_scrub":"261'316","last_deep_scrub_stamp":"2024-03-09T15:36:33.480185+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.480185+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":300,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:48:54.047973+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16427,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":565,"num_read_kb":1160,"num_write":446,"num_write_kb":361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22450,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.679444+0000","last_change":"2024-03-09T15:36:21.679497+0000","last_active":"2024-03-09T15:36:21.679444+0000","last_peered":"2024-03-09T15:36:21.679444+0000","last_clean":"2024-03-09T15:36:21.679444+0000","last_became_active":"2024-03-09T14:28:40.378829+0000","last_became_peered":"2024-03-09T14:28:40.378829+0000","last_unstale":"2024-03-09T15:36:21.679444+0000","last_undegraded":"2024-03-09T15:36:21.679444+0000","last_fullsized":"2024-03-09T15:36:21.679444+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.679245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.679245+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.679245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:45:45.724495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"261'126","reported_seq":635,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.487636+0000","last_change":"2024-03-09T15:36:26.487767+0000","last_active":"2024-03-09T15:36:26.487636+0000","last_peered":"2024-03-09T15:36:26.487636+0000","last_clean":"2024-03-09T15:36:26.487636+0000","last_became_active":"2024-03-09T14:28:42.392509+0000","last_became_peered":"2024-03-09T14:28:42.392509+0000","last_unstale":"2024-03-09T15:36:26.487636+0000","last_undegraded":"2024-03-09T15:36:26.487636+0000","last_fullsized":"2024-03-09T15:36:26.487636+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'126","last_scrub_stamp":"2024-03-09T15:36:26.487384+0000","last_deep_scrub":"261'126","last_deep_scrub_stamp":"2024-03-09T15:36:26.487384+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.487384+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":100,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:33:29.950896+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":643,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":173,"num_write":275,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"261'132","reported_seq":694,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.361023+0000","last_change":"2024-03-09T15:36:45.361023+0000","last_active":"2024-03-09T15:36:45.361023+0000","last_peered":"2024-03-09T15:36:45.361023+0000","last_clean":"2024-03-09T15:36:45.361023+0000","last_became_active":"2024-03-09T14:28:34.330686+0000","last_became_peered":"2024-03-09T14:28:34.330686+0000","last_unstale":"2024-03-09T15:36:45.361023+0000","last_undegraded":"2024-03-09T15:36:45.361023+0000","last_fullsized":"2024-03-09T15:36:45.361023+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'132","last_scrub_stamp":"2024-03-09T15:36:45.360646+0000","last_deep_scrub":"261'132","last_deep_scrub_stamp":"2024-03-09T15:36:45.360646+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.360646+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:37:31.175012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":527,"num_read_kb":451,"num_write":212,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"261'143","reported_seq":691,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.218846+0000","last_change":"2024-03-09T15:36:42.218957+0000","last_active":"2024-03-09T15:36:42.218846+0000","last_peered":"2024-03-09T15:36:42.218846+0000","last_clean":"2024-03-09T15:36:42.218846+0000","last_became_active":"2024-03-09T14:28:32.301087+0000","last_became_peered":"2024-03-09T14:28:32.301087+0000","last_unstale":"2024-03-09T15:36:42.218846+0000","last_undegraded":"2024-03-09T15:36:42.218846+0000","last_fullsized":"2024-03-09T15:36:42.218846+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'143","last_scrub_stamp":"2024-03-09T15:36:42.218633+0000","last_deep_scrub":"261'143","last_deep_scrub_stamp":"2024-03-09T15:36:42.218633+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.218633+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":100,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:38:14.730766+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":137,"num_read_kb":3393,"num_write":460,"num_write_kb":23270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.130798+0000","last_change":"2024-03-09T15:36:23.130851+0000","last_active":"2024-03-09T15:36:23.130798+0000","last_peered":"2024-03-09T15:36:23.130798+0000","last_clean":"2024-03-09T15:36:23.130798+0000","last_became_active":"2024-03-09T14:28:36.328814+0000","last_became_peered":"2024-03-09T14:28:36.328814+0000","last_unstale":"2024-03-09T15:36:23.130798+0000","last_undegraded":"2024-03-09T15:36:23.130798+0000","last_fullsized":"2024-03-09T15:36:23.130798+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:23.130603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:23.130603+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.130603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:54:38.685950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"261'260","reported_seq":873,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.173698+0000","last_change":"2024-03-09T15:36:47.173796+0000","last_active":"2024-03-09T15:36:47.173698+0000","last_peered":"2024-03-09T15:36:47.173698+0000","last_clean":"2024-03-09T15:36:47.173698+0000","last_became_active":"2024-03-09T14:28:38.331930+0000","last_became_peered":"2024-03-09T14:28:38.331930+0000","last_unstale":"2024-03-09T15:36:47.173698+0000","last_undegraded":"2024-03-09T15:36:47.173698+0000","last_fullsized":"2024-03-09T15:36:47.173698+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'260","last_scrub_stamp":"2024-03-09T15:36:47.173452+0000","last_deep_scrub":"261'260","last_deep_scrub_stamp":"2024-03-09T15:36:47.173452+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.173452+0000","objects_scrubbed":8,"log_size":60,"log_dups_size":200,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:18:24.859389+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21830,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":424,"num_read_kb":798,"num_write":342,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11460,"num_omap_keys":24,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.208618+0000","last_change":"2024-03-09T15:36:47.208675+0000","last_active":"2024-03-09T15:36:47.208618+0000","last_peered":"2024-03-09T15:36:47.208618+0000","last_clean":"2024-03-09T15:36:47.208618+0000","last_became_active":"2024-03-09T14:28:40.366723+0000","last_became_peered":"2024-03-09T14:28:40.366723+0000","last_unstale":"2024-03-09T15:36:47.208618+0000","last_undegraded":"2024-03-09T15:36:47.208618+0000","last_fullsized":"2024-03-09T15:36:47.208618+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:47.208400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:47.208400+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.208400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:22:48.196708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"261'112","reported_seq":627,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.220706+0000","last_change":"2024-03-09T15:36:32.220786+0000","last_active":"2024-03-09T15:36:32.220706+0000","last_peered":"2024-03-09T15:36:32.220706+0000","last_clean":"2024-03-09T15:36:32.220706+0000","last_became_active":"2024-03-09T14:28:42.395375+0000","last_became_peered":"2024-03-09T14:28:42.395375+0000","last_unstale":"2024-03-09T15:36:32.220706+0000","last_undegraded":"2024-03-09T15:36:32.220706+0000","last_fullsized":"2024-03-09T15:36:32.220706+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'112","last_scrub_stamp":"2024-03-09T15:36:32.220503+0000","last_deep_scrub":"261'112","last_deep_scrub_stamp":"2024-03-09T15:36:32.220503+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.220503+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":100,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:58:59.546341+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1199,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":139,"num_read_kb":136,"num_write":341,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"261'99","reported_seq":695,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:52.760381+0000","last_change":"2024-03-09T15:36:52.760444+0000","last_active":"2024-03-09T15:36:52.760381+0000","last_peered":"2024-03-09T15:36:52.760381+0000","last_clean":"2024-03-09T15:36:52.760381+0000","last_became_active":"2024-03-09T14:28:34.312297+0000","last_became_peered":"2024-03-09T14:28:34.312297+0000","last_unstale":"2024-03-09T15:36:52.760381+0000","last_undegraded":"2024-03-09T15:36:52.760381+0000","last_fullsized":"2024-03-09T15:36:52.760381+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'99","last_scrub_stamp":"2024-03-09T15:36:52.760182+0000","last_deep_scrub":"261'99","last_deep_scrub_stamp":"2024-03-09T15:36:52.760182+0000","last_clean_scrub_stamp":"2024-03-09T15:36:52.760182+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:34:46.696922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":415,"num_read_kb":347,"num_write":146,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"261'120","reported_seq":697,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.470441+0000","last_change":"2024-03-09T15:36:49.470646+0000","last_active":"2024-03-09T15:36:49.470441+0000","last_peered":"2024-03-09T15:36:49.470441+0000","last_clean":"2024-03-09T15:36:49.470441+0000","last_became_active":"2024-03-09T14:28:32.312937+0000","last_became_peered":"2024-03-09T14:28:32.312937+0000","last_unstale":"2024-03-09T15:36:49.470441+0000","last_undegraded":"2024-03-09T15:36:49.470441+0000","last_fullsized":"2024-03-09T15:36:49.470441+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'120","last_scrub_stamp":"2024-03-09T15:36:49.470219+0000","last_deep_scrub":"261'120","last_deep_scrub_stamp":"2024-03-09T15:36:49.470219+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.470219+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:58:33.393108+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262175,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":159,"num_read_kb":47424,"num_write":430,"num_write_kb":24173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.273419+0000","last_change":"2024-03-09T15:36:41.273535+0000","last_active":"2024-03-09T15:36:41.273419+0000","last_peered":"2024-03-09T15:36:41.273419+0000","last_clean":"2024-03-09T15:36:41.273419+0000","last_became_active":"2024-03-09T14:28:36.316094+0000","last_became_peered":"2024-03-09T14:28:36.316094+0000","last_unstale":"2024-03-09T15:36:41.273419+0000","last_undegraded":"2024-03-09T15:36:41.273419+0000","last_fullsized":"2024-03-09T15:36:41.273419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:41.273162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:41.273162+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.273162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:10:54.703371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"261'195","reported_seq":790,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.519051+0000","last_change":"2024-03-09T15:36:36.519208+0000","last_active":"2024-03-09T15:36:36.519051+0000","last_peered":"2024-03-09T15:36:36.519051+0000","last_clean":"2024-03-09T15:36:36.519051+0000","last_became_active":"2024-03-09T14:28:38.595522+0000","last_became_peered":"2024-03-09T14:28:38.595522+0000","last_unstale":"2024-03-09T15:36:36.519051+0000","last_undegraded":"2024-03-09T15:36:36.519051+0000","last_fullsized":"2024-03-09T15:36:36.519051+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'195","last_scrub_stamp":"2024-03-09T15:36:36.518818+0000","last_deep_scrub":"261'195","last_deep_scrub_stamp":"2024-03-09T15:36:36.518818+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.518818+0000","objects_scrubbed":5,"log_size":95,"log_dups_size":100,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:00:11.684059+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12058,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":311,"num_read_kb":573,"num_write":216,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.585690+0000","last_change":"2024-03-09T15:36:22.585690+0000","last_active":"2024-03-09T15:36:22.585690+0000","last_peered":"2024-03-09T15:36:22.585690+0000","last_clean":"2024-03-09T15:36:22.585690+0000","last_became_active":"2024-03-09T14:28:40.376549+0000","last_became_peered":"2024-03-09T14:28:40.376549+0000","last_unstale":"2024-03-09T15:36:22.585690+0000","last_undegraded":"2024-03-09T15:36:22.585690+0000","last_fullsized":"2024-03-09T15:36:22.585690+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.585447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.585447+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.585447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:45:35.078062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"261'45","reported_seq":550,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.666451+0000","last_change":"2024-03-09T15:36:19.666451+0000","last_active":"2024-03-09T15:36:19.666451+0000","last_peered":"2024-03-09T15:36:19.666451+0000","last_clean":"2024-03-09T15:36:19.666451+0000","last_became_active":"2024-03-09T14:28:42.394780+0000","last_became_peered":"2024-03-09T14:28:42.394780+0000","last_unstale":"2024-03-09T15:36:19.666451+0000","last_undegraded":"2024-03-09T15:36:19.666451+0000","last_fullsized":"2024-03-09T15:36:19.666451+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'45","last_scrub_stamp":"2024-03-09T15:36:19.666178+0000","last_deep_scrub":"261'45","last_deep_scrub_stamp":"2024-03-09T15:36:19.666178+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.666178+0000","objects_scrubbed":3,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:53:32.256553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":84,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"261'68","reported_seq":624,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.190901+0000","last_change":"2024-03-09T15:36:32.190971+0000","last_active":"2024-03-09T15:36:32.190901+0000","last_peered":"2024-03-09T15:36:32.190901+0000","last_clean":"2024-03-09T15:36:32.190901+0000","last_became_active":"2024-03-09T14:28:34.313125+0000","last_became_peered":"2024-03-09T14:28:34.313125+0000","last_unstale":"2024-03-09T15:36:32.190901+0000","last_undegraded":"2024-03-09T15:36:32.190901+0000","last_fullsized":"2024-03-09T15:36:32.190901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'68","last_scrub_stamp":"2024-03-09T15:36:32.190702+0000","last_deep_scrub":"261'68","last_deep_scrub_stamp":"2024-03-09T15:36:32.190702+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.190702+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:14:08.005357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"261'191","reported_seq":804,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:03.136315+0000","last_change":"2024-03-09T15:37:03.136404+0000","last_active":"2024-03-09T15:37:03.136315+0000","last_peered":"2024-03-09T15:37:03.136315+0000","last_clean":"2024-03-09T15:37:03.136315+0000","last_became_active":"2024-03-09T14:28:32.303835+0000","last_became_peered":"2024-03-09T14:28:32.303835+0000","last_unstale":"2024-03-09T15:37:03.136315+0000","last_undegraded":"2024-03-09T15:37:03.136315+0000","last_fullsized":"2024-03-09T15:37:03.136315+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'191","last_scrub_stamp":"2024-03-09T15:37:03.136127+0000","last_deep_scrub":"261'191","last_deep_scrub_stamp":"2024-03-09T15:37:03.136127+0000","last_clean_scrub_stamp":"2024-03-09T15:37:03.136127+0000","objects_scrubbed":4,"log_size":91,"log_dups_size":100,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:18:30.887965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":371,"num_read_kb":1431,"num_write":999,"num_write_kb":8832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.488481+0000","last_change":"2024-03-09T15:36:29.488628+0000","last_active":"2024-03-09T15:36:29.488481+0000","last_peered":"2024-03-09T15:36:29.488481+0000","last_clean":"2024-03-09T15:36:29.488481+0000","last_became_active":"2024-03-09T14:28:36.322060+0000","last_became_peered":"2024-03-09T14:28:36.322060+0000","last_unstale":"2024-03-09T15:36:29.488481+0000","last_undegraded":"2024-03-09T15:36:29.488481+0000","last_fullsized":"2024-03-09T15:36:29.488481+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:29.488248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:29.488248+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.488248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:40:34.293730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"261'96","reported_seq":647,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.151474+0000","last_change":"2024-03-09T15:36:24.151474+0000","last_active":"2024-03-09T15:36:24.151474+0000","last_peered":"2024-03-09T15:36:24.151474+0000","last_clean":"2024-03-09T15:36:24.151474+0000","last_became_active":"2024-03-09T14:28:38.595749+0000","last_became_peered":"2024-03-09T14:28:38.595749+0000","last_unstale":"2024-03-09T15:36:24.151474+0000","last_undegraded":"2024-03-09T15:36:24.151474+0000","last_fullsized":"2024-03-09T15:36:24.151474+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'96","last_scrub_stamp":"2024-03-09T15:36:24.151188+0000","last_deep_scrub":"261'96","last_deep_scrub_stamp":"2024-03-09T15:36:24.151188+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.151188+0000","objects_scrubbed":7,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:06:38.216926+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":132,"num_read_kb":127,"num_write":124,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11222,"num_omap_keys":30,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"261'489","reported_seq":2266,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.168974+0000","last_change":"2024-03-09T15:36:28.169035+0000","last_active":"2024-03-09T15:36:28.168974+0000","last_peered":"2024-03-09T15:36:28.168974+0000","last_clean":"2024-03-09T15:36:28.168974+0000","last_became_active":"2024-03-09T14:28:40.376946+0000","last_became_peered":"2024-03-09T14:28:40.376946+0000","last_unstale":"2024-03-09T15:36:28.168974+0000","last_undegraded":"2024-03-09T15:36:28.168974+0000","last_fullsized":"2024-03-09T15:36:28.168974+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'489","last_scrub_stamp":"2024-03-09T15:36:28.168784+0000","last_deep_scrub":"261'489","last_deep_scrub_stamp":"2024-03-09T15:36:28.168784+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.168784+0000","objects_scrubbed":1,"log_size":89,"log_dups_size":400,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:26:51.629900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.10","version":"261'45","reported_seq":559,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.174006+0000","last_change":"2024-03-09T15:36:31.174074+0000","last_active":"2024-03-09T15:36:31.174006+0000","last_peered":"2024-03-09T15:36:31.174006+0000","last_clean":"2024-03-09T15:36:31.174006+0000","last_became_active":"2024-03-09T14:28:42.397829+0000","last_became_peered":"2024-03-09T14:28:42.397829+0000","last_unstale":"2024-03-09T15:36:31.174006+0000","last_undegraded":"2024-03-09T15:36:31.174006+0000","last_fullsized":"2024-03-09T15:36:31.174006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'45","last_scrub_stamp":"2024-03-09T15:36:31.173791+0000","last_deep_scrub":"261'45","last_deep_scrub_stamp":"2024-03-09T15:36:31.173791+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.173791+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:44.989801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":97,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"261'144","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.216346+0000","last_change":"2024-03-09T15:36:37.216431+0000","last_active":"2024-03-09T15:36:37.216346+0000","last_peered":"2024-03-09T15:36:37.216346+0000","last_clean":"2024-03-09T15:36:37.216346+0000","last_became_active":"2024-03-09T14:28:34.319891+0000","last_became_peered":"2024-03-09T14:28:34.319891+0000","last_unstale":"2024-03-09T15:36:37.216346+0000","last_undegraded":"2024-03-09T15:36:37.216346+0000","last_fullsized":"2024-03-09T15:36:37.216346+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'144","last_scrub_stamp":"2024-03-09T15:36:37.216142+0000","last_deep_scrub":"261'144","last_deep_scrub_stamp":"2024-03-09T15:36:37.216142+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.216142+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:10:29.932991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":571,"num_read_kb":433,"num_write":263,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"261'165","reported_seq":820,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.905667+0000","last_change":"2024-03-09T15:36:39.905809+0000","last_active":"2024-03-09T15:36:39.905667+0000","last_peered":"2024-03-09T15:36:39.905667+0000","last_clean":"2024-03-09T15:36:39.905667+0000","last_became_active":"2024-03-09T14:28:32.323198+0000","last_became_peered":"2024-03-09T14:28:32.323198+0000","last_unstale":"2024-03-09T15:36:39.905667+0000","last_undegraded":"2024-03-09T15:36:39.905667+0000","last_fullsized":"2024-03-09T15:36:39.905667+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:39.905460+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:39.905460+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.905460+0000","objects_scrubbed":4,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:16:12.321789+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4801550,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":406,"num_read_kb":2095,"num_write":652,"num_write_kb":25158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.154703+0000","last_change":"2024-03-09T15:36:45.154756+0000","last_active":"2024-03-09T15:36:45.154703+0000","last_peered":"2024-03-09T15:36:45.154703+0000","last_clean":"2024-03-09T15:36:45.154703+0000","last_became_active":"2024-03-09T14:28:36.328529+0000","last_became_peered":"2024-03-09T14:28:36.328529+0000","last_unstale":"2024-03-09T15:36:45.154703+0000","last_undegraded":"2024-03-09T15:36:45.154703+0000","last_fullsized":"2024-03-09T15:36:45.154703+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:45.154500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:45.154500+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.154500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:30:04.214748+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"261'231","reported_seq":843,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:52.869145+0000","last_change":"2024-03-09T15:36:52.869321+0000","last_active":"2024-03-09T15:36:52.869145+0000","last_peered":"2024-03-09T15:36:52.869145+0000","last_clean":"2024-03-09T15:36:52.869145+0000","last_became_active":"2024-03-09T14:28:38.334991+0000","last_became_peered":"2024-03-09T14:28:38.334991+0000","last_unstale":"2024-03-09T15:36:52.869145+0000","last_undegraded":"2024-03-09T15:36:52.869145+0000","last_fullsized":"2024-03-09T15:36:52.869145+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'231","last_scrub_stamp":"2024-03-09T15:36:52.868879+0000","last_deep_scrub":"261'231","last_deep_scrub_stamp":"2024-03-09T15:36:52.868879+0000","last_clean_scrub_stamp":"2024-03-09T15:36:52.868879+0000","objects_scrubbed":7,"log_size":31,"log_dups_size":200,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:38:44.778956+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13314,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":373,"num_read_kb":687,"num_write":264,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.226908+0000","last_change":"2024-03-09T15:36:33.226963+0000","last_active":"2024-03-09T15:36:33.226908+0000","last_peered":"2024-03-09T15:36:33.226908+0000","last_clean":"2024-03-09T15:36:33.226908+0000","last_became_active":"2024-03-09T14:28:40.368039+0000","last_became_peered":"2024-03-09T14:28:40.368039+0000","last_unstale":"2024-03-09T15:36:33.226908+0000","last_undegraded":"2024-03-09T15:36:33.226908+0000","last_fullsized":"2024-03-09T15:36:33.226908+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:33.226717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:33.226717+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.226717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:17.480958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"261'79","reported_seq":609,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.203923+0000","last_change":"2024-03-09T15:36:28.203997+0000","last_active":"2024-03-09T15:36:28.203923+0000","last_peered":"2024-03-09T15:36:28.203923+0000","last_clean":"2024-03-09T15:36:28.203923+0000","last_became_active":"2024-03-09T14:28:42.413840+0000","last_became_peered":"2024-03-09T14:28:42.413840+0000","last_unstale":"2024-03-09T15:36:28.203923+0000","last_undegraded":"2024-03-09T15:36:28.203923+0000","last_fullsized":"2024-03-09T15:36:28.203923+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'79","last_scrub_stamp":"2024-03-09T15:36:28.203746+0000","last_deep_scrub":"261'79","last_deep_scrub_stamp":"2024-03-09T15:36:28.203746+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.203746+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:20:00.519578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":129,"num_write":187,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"261'36","reported_seq":582,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.213368+0000","last_change":"2024-03-09T15:36:35.213428+0000","last_active":"2024-03-09T15:36:35.213368+0000","last_peered":"2024-03-09T15:36:35.213368+0000","last_clean":"2024-03-09T15:36:35.213368+0000","last_became_active":"2024-03-09T14:28:34.314194+0000","last_became_peered":"2024-03-09T14:28:34.314194+0000","last_unstale":"2024-03-09T15:36:35.213368+0000","last_undegraded":"2024-03-09T15:36:35.213368+0000","last_fullsized":"2024-03-09T15:36:35.213368+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'36","last_scrub_stamp":"2024-03-09T15:36:35.213167+0000","last_deep_scrub":"261'36","last_deep_scrub_stamp":"2024-03-09T15:36:35.213167+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.213167+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:30.268110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":120,"num_write":54,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"261'150","reported_seq":812,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.865057+0000","last_change":"2024-03-09T15:36:50.865141+0000","last_active":"2024-03-09T15:36:50.865057+0000","last_peered":"2024-03-09T15:36:50.865057+0000","last_clean":"2024-03-09T15:36:50.865057+0000","last_became_active":"2024-03-09T14:28:32.307702+0000","last_became_peered":"2024-03-09T14:28:32.307702+0000","last_unstale":"2024-03-09T15:36:50.865057+0000","last_undegraded":"2024-03-09T15:36:50.865057+0000","last_fullsized":"2024-03-09T15:36:50.865057+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'150","last_scrub_stamp":"2024-03-09T15:36:50.864867+0000","last_deep_scrub":"261'150","last_deep_scrub_stamp":"2024-03-09T15:36:50.864867+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.864867+0000","objects_scrubbed":3,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:16:58.825247+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":273,"num_read_kb":1758,"num_write":483,"num_write_kb":23078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":512,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.746052+0000","last_change":"2024-03-09T15:36:34.746107+0000","last_active":"2024-03-09T15:36:34.746052+0000","last_peered":"2024-03-09T15:36:34.746052+0000","last_clean":"2024-03-09T15:36:34.746052+0000","last_became_active":"2024-03-09T14:28:36.304241+0000","last_became_peered":"2024-03-09T14:28:36.304241+0000","last_unstale":"2024-03-09T15:36:34.746052+0000","last_undegraded":"2024-03-09T15:36:34.746052+0000","last_fullsized":"2024-03-09T15:36:34.746052+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:34.745840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:34.745840+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.745840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:07:47.752362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"261'182","reported_seq":771,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.223598+0000","last_change":"2024-03-09T15:36:38.223787+0000","last_active":"2024-03-09T15:36:38.223598+0000","last_peered":"2024-03-09T15:36:38.223598+0000","last_clean":"2024-03-09T15:36:38.223598+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:36:38.223598+0000","last_undegraded":"2024-03-09T15:36:38.223598+0000","last_fullsized":"2024-03-09T15:36:38.223598+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'182","last_scrub_stamp":"2024-03-09T15:36:38.223354+0000","last_deep_scrub":"261'182","last_deep_scrub_stamp":"2024-03-09T15:36:38.223354+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.223354+0000","objects_scrubbed":4,"log_size":82,"log_dups_size":100,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:01:53.396420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":521,"num_write":190,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.171034+0000","last_change":"2024-03-09T15:36:24.171034+0000","last_active":"2024-03-09T15:36:24.171034+0000","last_peered":"2024-03-09T15:36:24.171034+0000","last_clean":"2024-03-09T15:36:24.171034+0000","last_became_active":"2024-03-09T14:28:40.371820+0000","last_became_peered":"2024-03-09T14:28:40.371820+0000","last_unstale":"2024-03-09T15:36:24.171034+0000","last_undegraded":"2024-03-09T15:36:24.171034+0000","last_fullsized":"2024-03-09T15:36:24.171034+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:24.170785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:24.170785+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.170785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:48:51.479283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.16","version":"261'56","reported_seq":580,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.501906+0000","last_change":"2024-03-09T15:36:28.501999+0000","last_active":"2024-03-09T15:36:28.501906+0000","last_peered":"2024-03-09T15:36:28.501906+0000","last_clean":"2024-03-09T15:36:28.501906+0000","last_became_active":"2024-03-09T14:28:42.392326+0000","last_became_peered":"2024-03-09T14:28:42.392326+0000","last_unstale":"2024-03-09T15:36:28.501906+0000","last_undegraded":"2024-03-09T15:36:28.501906+0000","last_fullsized":"2024-03-09T15:36:28.501906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:28.501691+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:28.501691+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.501691+0000","objects_scrubbed":3,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:17:33.874116+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":772,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":99,"num_write":117,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"261'70","reported_seq":642,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.589053+0000","last_change":"2024-03-09T15:36:44.589114+0000","last_active":"2024-03-09T15:36:44.589053+0000","last_peered":"2024-03-09T15:36:44.589053+0000","last_clean":"2024-03-09T15:36:44.589053+0000","last_became_active":"2024-03-09T14:28:34.323969+0000","last_became_peered":"2024-03-09T14:28:34.323969+0000","last_unstale":"2024-03-09T15:36:44.589053+0000","last_undegraded":"2024-03-09T15:36:44.589053+0000","last_fullsized":"2024-03-09T15:36:44.589053+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'70","last_scrub_stamp":"2024-03-09T15:36:44.588877+0000","last_deep_scrub":"261'70","last_deep_scrub_stamp":"2024-03-09T15:36:44.588877+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.588877+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:27:16.122474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":256,"num_write":98,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"261'183","reported_seq":784,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:54.782043+0000","last_change":"2024-03-09T15:36:54.782180+0000","last_active":"2024-03-09T15:36:54.782043+0000","last_peered":"2024-03-09T15:36:54.782043+0000","last_clean":"2024-03-09T15:36:54.782043+0000","last_became_active":"2024-03-09T14:28:32.337755+0000","last_became_peered":"2024-03-09T14:28:32.337755+0000","last_unstale":"2024-03-09T15:36:54.782043+0000","last_undegraded":"2024-03-09T15:36:54.782043+0000","last_fullsized":"2024-03-09T15:36:54.782043+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'183","last_scrub_stamp":"2024-03-09T15:36:54.781815+0000","last_deep_scrub":"261'183","last_deep_scrub_stamp":"2024-03-09T15:36:54.781815+0000","last_clean_scrub_stamp":"2024-03-09T15:36:54.781815+0000","objects_scrubbed":7,"log_size":83,"log_dups_size":100,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:51:13.814026+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663535,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":299,"num_read_kb":3602,"num_write":667,"num_write_kb":11566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.298938+0000","last_change":"2024-03-09T15:36:43.298995+0000","last_active":"2024-03-09T15:36:43.298938+0000","last_peered":"2024-03-09T15:36:43.298938+0000","last_clean":"2024-03-09T15:36:43.298938+0000","last_became_active":"2024-03-09T14:28:36.317569+0000","last_became_peered":"2024-03-09T14:28:36.317569+0000","last_unstale":"2024-03-09T15:36:43.298938+0000","last_undegraded":"2024-03-09T15:36:43.298938+0000","last_fullsized":"2024-03-09T15:36:43.298938+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:43.298743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:43.298743+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.298743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:54:29.675835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"261'154","reported_seq":723,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.559070+0000","last_change":"2024-03-09T15:36:39.559219+0000","last_active":"2024-03-09T15:36:39.559070+0000","last_peered":"2024-03-09T15:36:39.559070+0000","last_clean":"2024-03-09T15:36:39.559070+0000","last_became_active":"2024-03-09T14:28:38.347370+0000","last_became_peered":"2024-03-09T14:28:38.347370+0000","last_unstale":"2024-03-09T15:36:39.559070+0000","last_undegraded":"2024-03-09T15:36:39.559070+0000","last_fullsized":"2024-03-09T15:36:39.559070+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'154","last_scrub_stamp":"2024-03-09T15:36:39.558842+0000","last_deep_scrub":"261'154","last_deep_scrub_stamp":"2024-03-09T15:36:39.558842+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.558842+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":100,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:06:29.281642+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18257,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":531,"num_write":172,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.105659+0000","last_change":"2024-03-09T15:36:22.105715+0000","last_active":"2024-03-09T15:36:22.105659+0000","last_peered":"2024-03-09T15:36:22.105659+0000","last_clean":"2024-03-09T15:36:22.105659+0000","last_became_active":"2024-03-09T14:28:40.376513+0000","last_became_peered":"2024-03-09T14:28:40.376513+0000","last_unstale":"2024-03-09T15:36:22.105659+0000","last_undegraded":"2024-03-09T15:36:22.105659+0000","last_fullsized":"2024-03-09T15:36:22.105659+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.105441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.105441+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.105441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:26:26.970816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"261'96","reported_seq":988,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.177168+0000","last_change":"2024-03-09T15:36:30.177269+0000","last_active":"2024-03-09T15:36:30.177168+0000","last_peered":"2024-03-09T15:36:30.177168+0000","last_clean":"2024-03-09T15:36:30.177168+0000","last_became_active":"2024-03-09T14:28:42.409717+0000","last_became_peered":"2024-03-09T14:28:42.409717+0000","last_unstale":"2024-03-09T15:36:30.177168+0000","last_undegraded":"2024-03-09T15:36:30.177168+0000","last_fullsized":"2024-03-09T15:36:30.177168+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'96","last_scrub_stamp":"2024-03-09T15:36:30.176968+0000","last_deep_scrub":"261'96","last_deep_scrub_stamp":"2024-03-09T15:36:30.176968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.176968+0000","objects_scrubbed":4,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:07:18.249596+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":500,"num_read_kb":493,"num_write":215,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"261'53","reported_seq":595,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.270125+0000","last_change":"2024-03-09T15:36:40.270186+0000","last_active":"2024-03-09T15:36:40.270125+0000","last_peered":"2024-03-09T15:36:40.270125+0000","last_clean":"2024-03-09T15:36:40.270125+0000","last_became_active":"2024-03-09T14:28:34.313380+0000","last_became_peered":"2024-03-09T14:28:34.313380+0000","last_unstale":"2024-03-09T15:36:40.270125+0000","last_undegraded":"2024-03-09T15:36:40.270125+0000","last_fullsized":"2024-03-09T15:36:40.270125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'53","last_scrub_stamp":"2024-03-09T15:36:40.269935+0000","last_deep_scrub":"261'53","last_deep_scrub_stamp":"2024-03-09T15:36:40.269935+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.269935+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:22:56.243161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":197,"num_read_kb":156,"num_write":74,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"261'164","reported_seq":760,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:59.110908+0000","last_change":"2024-03-09T15:36:59.111059+0000","last_active":"2024-03-09T15:36:59.110908+0000","last_peered":"2024-03-09T15:36:59.110908+0000","last_clean":"2024-03-09T15:36:59.110908+0000","last_became_active":"2024-03-09T14:28:32.292862+0000","last_became_peered":"2024-03-09T14:28:32.292862+0000","last_unstale":"2024-03-09T15:36:59.110908+0000","last_undegraded":"2024-03-09T15:36:59.110908+0000","last_fullsized":"2024-03-09T15:36:59.110908+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'164","last_scrub_stamp":"2024-03-09T15:36:59.110646+0000","last_deep_scrub":"261'164","last_deep_scrub_stamp":"2024-03-09T15:36:59.110646+0000","last_clean_scrub_stamp":"2024-03-09T15:36:59.110646+0000","objects_scrubbed":3,"log_size":64,"log_dups_size":100,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:05:14.848338+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":264,"num_read_kb":35693,"num_write":668,"num_write_kb":24072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"261'2","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.597930+0000","last_change":"2024-03-09T15:36:23.597930+0000","last_active":"2024-03-09T15:36:23.597930+0000","last_peered":"2024-03-09T15:36:23.597930+0000","last_clean":"2024-03-09T15:36:23.597930+0000","last_became_active":"2024-03-09T14:28:36.313405+0000","last_became_peered":"2024-03-09T14:28:36.313405+0000","last_unstale":"2024-03-09T15:36:23.597930+0000","last_undegraded":"2024-03-09T15:36:23.597930+0000","last_fullsized":"2024-03-09T15:36:23.597930+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:23.597678+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:23.597678+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.597678+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:11:34.369475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"261'213","reported_seq":819,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.171733+0000","last_change":"2024-03-09T15:36:30.171809+0000","last_active":"2024-03-09T15:36:30.171733+0000","last_peered":"2024-03-09T15:36:30.171733+0000","last_clean":"2024-03-09T15:36:30.171733+0000","last_became_active":"2024-03-09T14:28:38.330571+0000","last_became_peered":"2024-03-09T14:28:38.330571+0000","last_unstale":"2024-03-09T15:36:30.171733+0000","last_undegraded":"2024-03-09T15:36:30.171733+0000","last_fullsized":"2024-03-09T15:36:30.171733+0000","mapping_epoch":26,"log_start":"261'201","ondisk_log_start":"261'201","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'213","last_scrub_stamp":"2024-03-09T15:36:30.171534+0000","last_deep_scrub":"261'213","last_deep_scrub_stamp":"2024-03-09T15:36:30.171534+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.171534+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":201,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:50.639816+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11093,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":349,"num_read_kb":659,"num_write":246,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.213225+0000","last_change":"2024-03-09T15:36:29.213283+0000","last_active":"2024-03-09T15:36:29.213225+0000","last_peered":"2024-03-09T15:36:29.213225+0000","last_clean":"2024-03-09T15:36:29.213225+0000","last_became_active":"2024-03-09T14:28:40.385080+0000","last_became_peered":"2024-03-09T14:28:40.385080+0000","last_unstale":"2024-03-09T15:36:29.213225+0000","last_undegraded":"2024-03-09T15:36:29.213225+0000","last_fullsized":"2024-03-09T15:36:29.213225+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:29.213014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:29.213014+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.213014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:47.618516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"261'69","reported_seq":573,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.140935+0000","last_change":"2024-03-09T15:36:15.140935+0000","last_active":"2024-03-09T15:36:15.140935+0000","last_peered":"2024-03-09T15:36:15.140935+0000","last_clean":"2024-03-09T15:36:15.140935+0000","last_became_active":"2024-03-09T14:28:42.412616+0000","last_became_peered":"2024-03-09T14:28:42.412616+0000","last_unstale":"2024-03-09T15:36:15.140935+0000","last_undegraded":"2024-03-09T15:36:15.140935+0000","last_fullsized":"2024-03-09T15:36:15.140935+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'69","last_scrub_stamp":"2024-03-09T15:36:15.140656+0000","last_deep_scrub":"261'69","last_deep_scrub_stamp":"2024-03-09T15:36:15.140656+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.140656+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:26:05.406444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":49,"num_write":163,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.27","version":"261'50","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.426480+0000","last_change":"2024-03-09T15:36:34.426540+0000","last_active":"2024-03-09T15:36:34.426480+0000","last_peered":"2024-03-09T15:36:34.426480+0000","last_clean":"2024-03-09T15:36:34.426480+0000","last_became_active":"2024-03-09T14:28:34.306685+0000","last_became_peered":"2024-03-09T14:28:34.306685+0000","last_unstale":"2024-03-09T15:36:34.426480+0000","last_undegraded":"2024-03-09T15:36:34.426480+0000","last_fullsized":"2024-03-09T15:36:34.426480+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'50","last_scrub_stamp":"2024-03-09T15:36:34.426288+0000","last_deep_scrub":"261'50","last_deep_scrub_stamp":"2024-03-09T15:36:34.426288+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.426288+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:49:13.124210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":204,"num_read_kb":164,"num_write":70,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"261'146","reported_seq":738,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.283440+0000","last_change":"2024-03-09T15:36:40.283538+0000","last_active":"2024-03-09T15:36:40.283440+0000","last_peered":"2024-03-09T15:36:40.283440+0000","last_clean":"2024-03-09T15:36:40.283440+0000","last_became_active":"2024-03-09T14:28:32.333008+0000","last_became_peered":"2024-03-09T14:28:32.333008+0000","last_unstale":"2024-03-09T15:36:40.283440+0000","last_undegraded":"2024-03-09T15:36:40.283440+0000","last_fullsized":"2024-03-09T15:36:40.283440+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'146","last_scrub_stamp":"2024-03-09T15:36:40.283252+0000","last_deep_scrub":"261'146","last_deep_scrub_stamp":"2024-03-09T15:36:40.283252+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.283252+0000","objects_scrubbed":6,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:58:17.359879+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655838,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":237,"num_read_kb":10390,"num_write":554,"num_write_kb":33266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.26","version":"261'181","reported_seq":781,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.520994+0000","last_change":"2024-03-09T15:36:30.521058+0000","last_active":"2024-03-09T15:36:30.520994+0000","last_peered":"2024-03-09T15:36:30.520994+0000","last_clean":"2024-03-09T15:36:30.520994+0000","last_became_active":"2024-03-09T14:28:34.314007+0000","last_became_peered":"2024-03-09T14:28:34.314007+0000","last_unstale":"2024-03-09T15:36:30.520994+0000","last_undegraded":"2024-03-09T15:36:30.520994+0000","last_fullsized":"2024-03-09T15:36:30.520994+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'181","last_scrub_stamp":"2024-03-09T15:36:30.520807+0000","last_deep_scrub":"261'181","last_deep_scrub_stamp":"2024-03-09T15:36:30.520807+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.520807+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":100,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:13:34.179786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":729,"num_read_kb":586,"num_write":272,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"261'155","reported_seq":713,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.244531+0000","last_change":"2024-03-09T15:36:29.244629+0000","last_active":"2024-03-09T15:36:29.244531+0000","last_peered":"2024-03-09T15:36:29.244531+0000","last_clean":"2024-03-09T15:36:29.244531+0000","last_became_active":"2024-03-09T14:28:32.286609+0000","last_became_peered":"2024-03-09T14:28:32.286609+0000","last_unstale":"2024-03-09T15:36:29.244531+0000","last_undegraded":"2024-03-09T15:36:29.244531+0000","last_fullsized":"2024-03-09T15:36:29.244531+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'155","last_scrub_stamp":"2024-03-09T15:36:29.244332+0000","last_deep_scrub":"261'155","last_deep_scrub_stamp":"2024-03-09T15:36:29.244332+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.244332+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":100,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:53:34.232406+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":810211,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":147,"num_read_kb":1907,"num_write":492,"num_write_kb":32192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"261'94","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.704648+0000","last_change":"2024-03-09T15:36:30.704712+0000","last_active":"2024-03-09T15:36:30.704648+0000","last_peered":"2024-03-09T15:36:30.704648+0000","last_clean":"2024-03-09T15:36:30.704648+0000","last_became_active":"2024-03-09T14:28:34.323541+0000","last_became_peered":"2024-03-09T14:28:34.323541+0000","last_unstale":"2024-03-09T15:36:30.704648+0000","last_undegraded":"2024-03-09T15:36:30.704648+0000","last_fullsized":"2024-03-09T15:36:30.704648+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'94","last_scrub_stamp":"2024-03-09T15:36:30.704449+0000","last_deep_scrub":"261'94","last_deep_scrub_stamp":"2024-03-09T15:36:30.704449+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.704449+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:11.739435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":384,"num_read_kb":306,"num_write":146,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":230,"num_omap_keys":1,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"261'173","reported_seq":778,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.935595+0000","last_change":"2024-03-09T15:36:47.935747+0000","last_active":"2024-03-09T15:36:47.935595+0000","last_peered":"2024-03-09T15:36:47.935595+0000","last_clean":"2024-03-09T15:36:47.935595+0000","last_became_active":"2024-03-09T14:28:32.295699+0000","last_became_peered":"2024-03-09T14:28:32.295699+0000","last_unstale":"2024-03-09T15:36:47.935595+0000","last_undegraded":"2024-03-09T15:36:47.935595+0000","last_fullsized":"2024-03-09T15:36:47.935595+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'173","last_scrub_stamp":"2024-03-09T15:36:47.935356+0000","last_deep_scrub":"261'173","last_deep_scrub_stamp":"2024-03-09T15:36:47.935356+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.935356+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":100,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:15:12.568313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":297,"num_read_kb":14189,"num_write":716,"num_write_kb":33521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"261'101","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.552706+0000","last_change":"2024-03-09T15:36:24.552770+0000","last_active":"2024-03-09T15:36:24.552706+0000","last_peered":"2024-03-09T15:36:24.552706+0000","last_clean":"2024-03-09T15:36:24.552706+0000","last_became_active":"2024-03-09T14:28:34.321647+0000","last_became_peered":"2024-03-09T14:28:34.321647+0000","last_unstale":"2024-03-09T15:36:24.552706+0000","last_undegraded":"2024-03-09T15:36:24.552706+0000","last_fullsized":"2024-03-09T15:36:24.552706+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'101","last_scrub_stamp":"2024-03-09T15:36:24.552512+0000","last_deep_scrub":"261'101","last_deep_scrub_stamp":"2024-03-09T15:36:24.552512+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.552512+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:59:27.278086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":329,"num_write":161,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"261'137","reported_seq":698,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:00.109163+0000","last_change":"2024-03-09T15:37:00.109387+0000","last_active":"2024-03-09T15:37:00.109163+0000","last_peered":"2024-03-09T15:37:00.109163+0000","last_clean":"2024-03-09T15:37:00.109163+0000","last_became_active":"2024-03-09T14:28:32.297866+0000","last_became_peered":"2024-03-09T14:28:32.297866+0000","last_unstale":"2024-03-09T15:37:00.109163+0000","last_undegraded":"2024-03-09T15:37:00.109163+0000","last_fullsized":"2024-03-09T15:37:00.109163+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'137","last_scrub_stamp":"2024-03-09T15:37:00.108913+0000","last_deep_scrub":"261'137","last_deep_scrub_stamp":"2024-03-09T15:37:00.108913+0000","last_clean_scrub_stamp":"2024-03-09T15:37:00.108913+0000","objects_scrubbed":9,"log_size":37,"log_dups_size":100,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:00:30.085008+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":163,"num_read_kb":3467,"num_write":495,"num_write_kb":23815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"261'52","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.607074+0000","last_change":"2024-03-09T15:36:45.607269+0000","last_active":"2024-03-09T15:36:45.607074+0000","last_peered":"2024-03-09T15:36:45.607074+0000","last_clean":"2024-03-09T15:36:45.607074+0000","last_became_active":"2024-03-09T14:28:34.330511+0000","last_became_peered":"2024-03-09T14:28:34.330511+0000","last_unstale":"2024-03-09T15:36:45.607074+0000","last_undegraded":"2024-03-09T15:36:45.607074+0000","last_fullsized":"2024-03-09T15:36:45.607074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'52","last_scrub_stamp":"2024-03-09T15:36:45.606838+0000","last_deep_scrub":"261'52","last_deep_scrub_stamp":"2024-03-09T15:36:45.606838+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.606838+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:14:12.247135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":225,"num_read_kb":185,"num_write":82,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"261'120","reported_seq":678,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:58.760534+0000","last_change":"2024-03-09T15:36:58.760602+0000","last_active":"2024-03-09T15:36:58.760534+0000","last_peered":"2024-03-09T15:36:58.760534+0000","last_clean":"2024-03-09T15:36:58.760534+0000","last_became_active":"2024-03-09T14:28:32.326148+0000","last_became_peered":"2024-03-09T14:28:32.326148+0000","last_unstale":"2024-03-09T15:36:58.760534+0000","last_undegraded":"2024-03-09T15:36:58.760534+0000","last_fullsized":"2024-03-09T15:36:58.760534+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'120","last_scrub_stamp":"2024-03-09T15:36:58.760335+0000","last_deep_scrub":"261'120","last_deep_scrub_stamp":"2024-03-09T15:36:58.760335+0000","last_clean_scrub_stamp":"2024-03-09T15:36:58.760335+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:25:57.931006+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":173,"num_read_kb":5614,"num_write":451,"num_write_kb":21169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"261'134","reported_seq":694,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.224786+0000","last_change":"2024-03-09T15:36:39.224849+0000","last_active":"2024-03-09T15:36:39.224786+0000","last_peered":"2024-03-09T15:36:39.224786+0000","last_clean":"2024-03-09T15:36:39.224786+0000","last_became_active":"2024-03-09T14:28:34.322225+0000","last_became_peered":"2024-03-09T14:28:34.322225+0000","last_unstale":"2024-03-09T15:36:39.224786+0000","last_undegraded":"2024-03-09T15:36:39.224786+0000","last_fullsized":"2024-03-09T15:36:39.224786+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'134","last_scrub_stamp":"2024-03-09T15:36:39.224588+0000","last_deep_scrub":"261'134","last_deep_scrub_stamp":"2024-03-09T15:36:39.224588+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.224588+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:04:21.612595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":498,"num_read_kb":399,"num_write":194,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"261'177","reported_seq":752,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.267800+0000","last_change":"2024-03-09T15:36:41.267910+0000","last_active":"2024-03-09T15:36:41.267800+0000","last_peered":"2024-03-09T15:36:41.267800+0000","last_clean":"2024-03-09T15:36:41.267800+0000","last_became_active":"2024-03-09T14:28:32.299839+0000","last_became_peered":"2024-03-09T14:28:32.299839+0000","last_unstale":"2024-03-09T15:36:41.267800+0000","last_undegraded":"2024-03-09T15:36:41.267800+0000","last_fullsized":"2024-03-09T15:36:41.267800+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'177","last_scrub_stamp":"2024-03-09T15:36:41.267607+0000","last_deep_scrub":"261'177","last_deep_scrub_stamp":"2024-03-09T15:36:41.267607+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.267607+0000","objects_scrubbed":6,"log_size":77,"log_dups_size":100,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:54:41.435261+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242922,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":18833,"num_write":606,"num_write_kb":39169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"261'74","reported_seq":617,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.623011+0000","last_change":"2024-03-09T15:36:46.623071+0000","last_active":"2024-03-09T15:36:46.623011+0000","last_peered":"2024-03-09T15:36:46.623011+0000","last_clean":"2024-03-09T15:36:46.623011+0000","last_became_active":"2024-03-09T14:28:34.315302+0000","last_became_peered":"2024-03-09T14:28:34.315302+0000","last_unstale":"2024-03-09T15:36:46.623011+0000","last_undegraded":"2024-03-09T15:36:46.623011+0000","last_fullsized":"2024-03-09T15:36:46.623011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'74","last_scrub_stamp":"2024-03-09T15:36:46.622798+0000","last_deep_scrub":"261'74","last_deep_scrub_stamp":"2024-03-09T15:36:46.622798+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.622798+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:10:27.130072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":224,"num_write":116,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"261'156","reported_seq":719,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.267202+0000","last_change":"2024-03-09T15:36:42.267361+0000","last_active":"2024-03-09T15:36:42.267202+0000","last_peered":"2024-03-09T15:36:42.267202+0000","last_clean":"2024-03-09T15:36:42.267202+0000","last_became_active":"2024-03-09T14:28:32.334117+0000","last_became_peered":"2024-03-09T14:28:32.334117+0000","last_unstale":"2024-03-09T15:36:42.267202+0000","last_undegraded":"2024-03-09T15:36:42.267202+0000","last_fullsized":"2024-03-09T15:36:42.267202+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'156","last_scrub_stamp":"2024-03-09T15:36:42.266976+0000","last_deep_scrub":"261'156","last_deep_scrub_stamp":"2024-03-09T15:36:42.266976+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.266976+0000","objects_scrubbed":7,"log_size":56,"log_dups_size":100,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:04:14.319419+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":182,"num_read_kb":192,"num_write":563,"num_write_kb":14566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"261'48","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.189108+0000","last_change":"2024-03-09T15:36:31.189173+0000","last_active":"2024-03-09T15:36:31.189108+0000","last_peered":"2024-03-09T15:36:31.189108+0000","last_clean":"2024-03-09T15:36:31.189108+0000","last_became_active":"2024-03-09T14:28:34.323202+0000","last_became_peered":"2024-03-09T14:28:34.323202+0000","last_unstale":"2024-03-09T15:36:31.189108+0000","last_undegraded":"2024-03-09T15:36:31.189108+0000","last_fullsized":"2024-03-09T15:36:31.189108+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:31.188911+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:31.188911+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.188911+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:36:34.360701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":219,"num_read_kb":183,"num_write":70,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"261'144","reported_seq":701,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:01.743658+0000","last_change":"2024-03-09T15:37:01.743756+0000","last_active":"2024-03-09T15:37:01.743658+0000","last_peered":"2024-03-09T15:37:01.743658+0000","last_clean":"2024-03-09T15:37:01.743658+0000","last_became_active":"2024-03-09T14:28:32.299737+0000","last_became_peered":"2024-03-09T14:28:32.299737+0000","last_unstale":"2024-03-09T15:37:01.743658+0000","last_undegraded":"2024-03-09T15:37:01.743658+0000","last_fullsized":"2024-03-09T15:37:01.743658+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'144","last_scrub_stamp":"2024-03-09T15:37:01.743474+0000","last_deep_scrub":"261'144","last_deep_scrub_stamp":"2024-03-09T15:37:01.743474+0000","last_clean_scrub_stamp":"2024-03-09T15:37:01.743474+0000","objects_scrubbed":6,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:12:52.005193+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":168,"num_read_kb":12613,"num_write":521,"num_write_kb":31618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"261'94","reported_seq":654,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.306693+0000","last_change":"2024-03-09T15:36:42.306758+0000","last_active":"2024-03-09T15:36:42.306693+0000","last_peered":"2024-03-09T15:36:42.306693+0000","last_clean":"2024-03-09T15:36:42.306693+0000","last_became_active":"2024-03-09T14:28:34.315451+0000","last_became_peered":"2024-03-09T14:28:34.315451+0000","last_unstale":"2024-03-09T15:36:42.306693+0000","last_undegraded":"2024-03-09T15:36:42.306693+0000","last_fullsized":"2024-03-09T15:36:42.306693+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'94","last_scrub_stamp":"2024-03-09T15:36:42.306481+0000","last_deep_scrub":"261'94","last_deep_scrub_stamp":"2024-03-09T15:36:42.306481+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.306481+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:34:40.259583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":374,"num_read_kb":293,"num_write":145,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"261'150","reported_seq":725,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.844582+0000","last_change":"2024-03-09T15:36:49.844668+0000","last_active":"2024-03-09T15:36:49.844582+0000","last_peered":"2024-03-09T15:36:49.844582+0000","last_clean":"2024-03-09T15:36:49.844582+0000","last_became_active":"2024-03-09T14:28:32.304540+0000","last_became_peered":"2024-03-09T14:28:32.304540+0000","last_unstale":"2024-03-09T15:36:49.844582+0000","last_undegraded":"2024-03-09T15:36:49.844582+0000","last_fullsized":"2024-03-09T15:36:49.844582+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'150","last_scrub_stamp":"2024-03-09T15:36:49.844389+0000","last_deep_scrub":"261'150","last_deep_scrub_stamp":"2024-03-09T15:36:49.844389+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.844389+0000","objects_scrubbed":6,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:30:01.179531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655899,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":200,"num_read_kb":1806,"num_write":509,"num_write_kb":18066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2e","version":"261'40","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.169829+0000","last_change":"2024-03-09T15:36:36.169895+0000","last_active":"2024-03-09T15:36:36.169829+0000","last_peered":"2024-03-09T15:36:36.169829+0000","last_clean":"2024-03-09T15:36:36.169829+0000","last_became_active":"2024-03-09T14:28:34.312399+0000","last_became_peered":"2024-03-09T14:28:34.312399+0000","last_unstale":"2024-03-09T15:36:36.169829+0000","last_undegraded":"2024-03-09T15:36:36.169829+0000","last_fullsized":"2024-03-09T15:36:36.169829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'40","last_scrub_stamp":"2024-03-09T15:36:36.169631+0000","last_deep_scrub":"261'40","last_deep_scrub_stamp":"2024-03-09T15:36:36.169631+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.169631+0000","objects_scrubbed":1,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:31:09.147800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":182,"num_read_kb":156,"num_write":57,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"261'151","reported_seq":703,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:05.826947+0000","last_change":"2024-03-09T15:37:05.827117+0000","last_active":"2024-03-09T15:37:05.826947+0000","last_peered":"2024-03-09T15:37:05.826947+0000","last_clean":"2024-03-09T15:37:05.826947+0000","last_became_active":"2024-03-09T14:28:32.314596+0000","last_became_peered":"2024-03-09T14:28:32.314596+0000","last_unstale":"2024-03-09T15:37:05.826947+0000","last_undegraded":"2024-03-09T15:37:05.826947+0000","last_fullsized":"2024-03-09T15:37:05.826947+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'151","last_scrub_stamp":"2024-03-09T15:37:05.826718+0000","last_deep_scrub":"261'151","last_deep_scrub_stamp":"2024-03-09T15:37:05.826718+0000","last_clean_scrub_stamp":"2024-03-09T15:37:05.826718+0000","objects_scrubbed":6,"log_size":51,"log_dups_size":100,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:11:13.341909+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263241,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":148,"num_read_kb":12793,"num_write":461,"num_write_kb":35350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"261'64","reported_seq":635,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.186824+0000","last_change":"2024-03-09T15:36:35.186888+0000","last_active":"2024-03-09T15:36:35.186824+0000","last_peered":"2024-03-09T15:36:35.186824+0000","last_clean":"2024-03-09T15:36:35.186824+0000","last_became_active":"2024-03-09T14:28:34.325461+0000","last_became_peered":"2024-03-09T14:28:34.325461+0000","last_unstale":"2024-03-09T15:36:35.186824+0000","last_undegraded":"2024-03-09T15:36:35.186824+0000","last_fullsized":"2024-03-09T15:36:35.186824+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'64","last_scrub_stamp":"2024-03-09T15:36:35.186622+0000","last_deep_scrub":"261'64","last_deep_scrub_stamp":"2024-03-09T15:36:35.186622+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.186622+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:12:15.358732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":264,"num_read_kb":214,"num_write":89,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"261'165","reported_seq":750,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.949321+0000","last_change":"2024-03-09T15:36:43.949499+0000","last_active":"2024-03-09T15:36:43.949321+0000","last_peered":"2024-03-09T15:36:43.949321+0000","last_clean":"2024-03-09T15:36:43.949321+0000","last_became_active":"2024-03-09T14:28:32.311750+0000","last_became_peered":"2024-03-09T14:28:32.311750+0000","last_unstale":"2024-03-09T15:36:43.949321+0000","last_undegraded":"2024-03-09T15:36:43.949321+0000","last_fullsized":"2024-03-09T15:36:43.949321+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:43.949104+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:43.949104+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.949104+0000","objects_scrubbed":6,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:52:33.164001+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":213,"num_read_kb":14896,"num_write":533,"num_write_kb":41774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"261'162","reported_seq":783,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.302576+0000","last_change":"2024-03-09T15:36:42.302635+0000","last_active":"2024-03-09T15:36:42.302576+0000","last_peered":"2024-03-09T15:36:42.302576+0000","last_clean":"2024-03-09T15:36:42.302576+0000","last_became_active":"2024-03-09T14:28:34.312212+0000","last_became_peered":"2024-03-09T14:28:34.312212+0000","last_unstale":"2024-03-09T15:36:42.302576+0000","last_undegraded":"2024-03-09T15:36:42.302576+0000","last_fullsized":"2024-03-09T15:36:42.302576+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'162","last_scrub_stamp":"2024-03-09T15:36:42.302384+0000","last_deep_scrub":"261'162","last_deep_scrub_stamp":"2024-03-09T15:36:42.302384+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.302384+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":100,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:11:17.535660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":664,"num_read_kb":532,"num_write":243,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2b","version":"261'153","reported_seq":734,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:04.104837+0000","last_change":"2024-03-09T15:37:04.104909+0000","last_active":"2024-03-09T15:37:04.104837+0000","last_peered":"2024-03-09T15:37:04.104837+0000","last_clean":"2024-03-09T15:37:04.104837+0000","last_became_active":"2024-03-09T14:28:32.309562+0000","last_became_peered":"2024-03-09T14:28:32.309562+0000","last_unstale":"2024-03-09T15:37:04.104837+0000","last_undegraded":"2024-03-09T15:37:04.104837+0000","last_fullsized":"2024-03-09T15:37:04.104837+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'153","last_scrub_stamp":"2024-03-09T15:37:04.104636+0000","last_deep_scrub":"261'153","last_deep_scrub_stamp":"2024-03-09T15:37:04.104636+0000","last_clean_scrub_stamp":"2024-03-09T15:37:04.104636+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":100,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:30:43.113561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":246,"num_read_kb":5348,"num_write":573,"num_write_kb":20887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"261'118","reported_seq":720,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.176542+0000","last_change":"2024-03-09T15:36:42.176738+0000","last_active":"2024-03-09T15:36:42.176542+0000","last_peered":"2024-03-09T15:36:42.176542+0000","last_clean":"2024-03-09T15:36:42.176542+0000","last_became_active":"2024-03-09T14:28:34.321424+0000","last_became_peered":"2024-03-09T14:28:34.321424+0000","last_unstale":"2024-03-09T15:36:42.176542+0000","last_undegraded":"2024-03-09T15:36:42.176542+0000","last_fullsized":"2024-03-09T15:36:42.176542+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'118","last_scrub_stamp":"2024-03-09T15:36:42.176313+0000","last_deep_scrub":"261'118","last_deep_scrub_stamp":"2024-03-09T15:36:42.176313+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.176313+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:02:02.079925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":513,"num_read_kb":409,"num_write":169,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3064,"num_omap_keys":18,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"261'167","reported_seq":744,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.264592+0000","last_change":"2024-03-09T15:36:41.264697+0000","last_active":"2024-03-09T15:36:41.264592+0000","last_peered":"2024-03-09T15:36:41.264592+0000","last_clean":"2024-03-09T15:36:41.264592+0000","last_became_active":"2024-03-09T14:28:32.333838+0000","last_became_peered":"2024-03-09T14:28:32.333838+0000","last_unstale":"2024-03-09T15:36:41.264592+0000","last_undegraded":"2024-03-09T15:36:41.264592+0000","last_fullsized":"2024-03-09T15:36:41.264592+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'167","last_scrub_stamp":"2024-03-09T15:36:41.264378+0000","last_deep_scrub":"261'167","last_deep_scrub_stamp":"2024-03-09T15:36:41.264378+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.264378+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":100,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:23:18.360963+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":238,"num_read_kb":3315,"num_write":673,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2a","version":"261'72","reported_seq":645,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:52.300833+0000","last_change":"2024-03-09T15:36:52.300899+0000","last_active":"2024-03-09T15:36:52.300833+0000","last_peered":"2024-03-09T15:36:52.300833+0000","last_clean":"2024-03-09T15:36:52.300833+0000","last_became_active":"2024-03-09T14:28:34.297042+0000","last_became_peered":"2024-03-09T14:28:34.297042+0000","last_unstale":"2024-03-09T15:36:52.300833+0000","last_undegraded":"2024-03-09T15:36:52.300833+0000","last_fullsized":"2024-03-09T15:36:52.300833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-09T15:36:52.300623+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-09T15:36:52.300623+0000","last_clean_scrub_stamp":"2024-03-09T15:36:52.300623+0000","objects_scrubbed":1,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:15:26.404006+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":242,"num_write":99,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":473,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"261'126","reported_seq":708,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:54.140373+0000","last_change":"2024-03-09T15:36:54.140525+0000","last_active":"2024-03-09T15:36:54.140373+0000","last_peered":"2024-03-09T15:36:54.140373+0000","last_clean":"2024-03-09T15:36:54.140373+0000","last_became_active":"2024-03-09T14:28:32.317797+0000","last_became_peered":"2024-03-09T14:28:32.317797+0000","last_unstale":"2024-03-09T15:36:54.140373+0000","last_undegraded":"2024-03-09T15:36:54.140373+0000","last_fullsized":"2024-03-09T15:36:54.140373+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'126","last_scrub_stamp":"2024-03-09T15:36:54.140156+0000","last_deep_scrub":"261'126","last_deep_scrub_stamp":"2024-03-09T15:36:54.140156+0000","last_clean_scrub_stamp":"2024-03-09T15:36:54.140156+0000","objects_scrubbed":4,"log_size":26,"log_dups_size":100,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:25:48.296017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263201,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":187,"num_read_kb":4139,"num_write":475,"num_write_kb":19145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"261'113","reported_seq":677,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.688184+0000","last_change":"2024-03-09T15:36:48.688250+0000","last_active":"2024-03-09T15:36:48.688184+0000","last_peered":"2024-03-09T15:36:48.688184+0000","last_clean":"2024-03-09T15:36:48.688184+0000","last_became_active":"2024-03-09T14:28:34.314073+0000","last_became_peered":"2024-03-09T14:28:34.314073+0000","last_unstale":"2024-03-09T15:36:48.688184+0000","last_undegraded":"2024-03-09T15:36:48.688184+0000","last_fullsized":"2024-03-09T15:36:48.688184+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'113","last_scrub_stamp":"2024-03-09T15:36:48.687973+0000","last_deep_scrub":"261'113","last_deep_scrub_stamp":"2024-03-09T15:36:48.687973+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.687973+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":100,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:23:16.042642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":445,"num_read_kb":351,"num_write":167,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"261'140","reported_seq":726,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.283437+0000","last_change":"2024-03-09T15:36:30.283533+0000","last_active":"2024-03-09T15:36:30.283437+0000","last_peered":"2024-03-09T15:36:30.283437+0000","last_clean":"2024-03-09T15:36:30.283437+0000","last_became_active":"2024-03-09T14:28:32.334187+0000","last_became_peered":"2024-03-09T14:28:32.334187+0000","last_unstale":"2024-03-09T15:36:30.283437+0000","last_undegraded":"2024-03-09T15:36:30.283437+0000","last_fullsized":"2024-03-09T15:36:30.283437+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'140","last_scrub_stamp":"2024-03-09T15:36:30.283226+0000","last_deep_scrub":"261'140","last_deep_scrub_stamp":"2024-03-09T15:36:30.283226+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.283226+0000","objects_scrubbed":6,"log_size":40,"log_dups_size":100,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:10:30.041775+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2181199,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":224,"num_read_kb":1842,"num_write":561,"num_write_kb":7884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"261'4028","reported_seq":4588,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.151978+0000","last_change":"2024-03-09T15:36:44.152037+0000","last_active":"2024-03-09T15:36:44.151978+0000","last_peered":"2024-03-09T15:36:44.151978+0000","last_clean":"2024-03-09T15:36:44.151978+0000","last_became_active":"2024-03-09T14:28:34.322421+0000","last_became_peered":"2024-03-09T14:28:34.322421+0000","last_unstale":"2024-03-09T15:36:44.151978+0000","last_undegraded":"2024-03-09T15:36:44.151978+0000","last_fullsized":"2024-03-09T15:36:44.151978+0000","mapping_epoch":22,"log_start":"261'4000","ondisk_log_start":"261'4000","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'4028","last_scrub_stamp":"2024-03-09T15:36:44.151779+0000","last_deep_scrub":"261'4028","last_deep_scrub_stamp":"2024-03-09T15:36:44.151779+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.151779+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:57:46.039852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14144,"num_read_kb":11461,"num_write":6038,"num_write_kb":3015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"261'136","reported_seq":715,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.873937+0000","last_change":"2024-03-09T15:36:37.874066+0000","last_active":"2024-03-09T15:36:37.873937+0000","last_peered":"2024-03-09T15:36:37.873937+0000","last_clean":"2024-03-09T15:36:37.873937+0000","last_became_active":"2024-03-09T14:28:32.324166+0000","last_became_peered":"2024-03-09T14:28:32.324166+0000","last_unstale":"2024-03-09T15:36:37.873937+0000","last_undegraded":"2024-03-09T15:36:37.873937+0000","last_fullsized":"2024-03-09T15:36:37.873937+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'136","last_scrub_stamp":"2024-03-09T15:36:37.873720+0000","last_deep_scrub":"261'136","last_deep_scrub_stamp":"2024-03-09T15:36:37.873720+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.873720+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:39:18.821060+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269861,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":9428,"num_write":515,"num_write_kb":30713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"261'144","reported_seq":732,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.957343+0000","last_change":"2024-03-09T15:36:46.957426+0000","last_active":"2024-03-09T15:36:46.957343+0000","last_peered":"2024-03-09T15:36:46.957343+0000","last_clean":"2024-03-09T15:36:46.957343+0000","last_became_active":"2024-03-09T14:28:32.287937+0000","last_became_peered":"2024-03-09T14:28:32.287937+0000","last_unstale":"2024-03-09T15:36:46.957343+0000","last_undegraded":"2024-03-09T15:36:46.957343+0000","last_fullsized":"2024-03-09T15:36:46.957343+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'144","last_scrub_stamp":"2024-03-09T15:36:46.957162+0000","last_deep_scrub":"261'144","last_deep_scrub_stamp":"2024-03-09T15:36:46.957162+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.957162+0000","objects_scrubbed":3,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:17:49.579394+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243043,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":214,"num_read_kb":48289,"num_write":459,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"261'161","reported_seq":736,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.317920+0000","last_change":"2024-03-09T15:36:43.318022+0000","last_active":"2024-03-09T15:36:43.317920+0000","last_peered":"2024-03-09T15:36:43.317920+0000","last_clean":"2024-03-09T15:36:43.317920+0000","last_became_active":"2024-03-09T14:28:34.316689+0000","last_became_peered":"2024-03-09T14:28:34.316689+0000","last_unstale":"2024-03-09T15:36:43.317920+0000","last_undegraded":"2024-03-09T15:36:43.317920+0000","last_fullsized":"2024-03-09T15:36:43.317920+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'161","last_scrub_stamp":"2024-03-09T15:36:43.317727+0000","last_deep_scrub":"261'161","last_deep_scrub_stamp":"2024-03-09T15:36:43.317727+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.317727+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:41.055202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":634,"num_read_kb":490,"num_write":268,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"261'163","reported_seq":751,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.213002+0000","last_change":"2024-03-09T15:36:46.213092+0000","last_active":"2024-03-09T15:36:46.213002+0000","last_peered":"2024-03-09T15:36:46.213002+0000","last_clean":"2024-03-09T15:36:46.213002+0000","last_became_active":"2024-03-09T14:28:32.313132+0000","last_became_peered":"2024-03-09T14:28:32.313132+0000","last_unstale":"2024-03-09T15:36:46.213002+0000","last_undegraded":"2024-03-09T15:36:46.213002+0000","last_fullsized":"2024-03-09T15:36:46.213002+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'163","last_scrub_stamp":"2024-03-09T15:36:46.212812+0000","last_deep_scrub":"261'163","last_deep_scrub_stamp":"2024-03-09T15:36:46.212812+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.212812+0000","objects_scrubbed":6,"log_size":63,"log_dups_size":100,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:51:57.319048+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262227,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":255,"num_read_kb":5401,"num_write":635,"num_write_kb":32648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"261'41","reported_seq":596,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.761668+0000","last_change":"2024-03-09T15:36:33.761734+0000","last_active":"2024-03-09T15:36:33.761668+0000","last_peered":"2024-03-09T15:36:33.761668+0000","last_clean":"2024-03-09T15:36:33.761668+0000","last_became_active":"2024-03-09T14:28:34.303701+0000","last_became_peered":"2024-03-09T14:28:34.303701+0000","last_unstale":"2024-03-09T15:36:33.761668+0000","last_undegraded":"2024-03-09T15:36:33.761668+0000","last_fullsized":"2024-03-09T15:36:33.761668+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'41","last_scrub_stamp":"2024-03-09T15:36:33.761432+0000","last_deep_scrub":"261'41","last_deep_scrub_stamp":"2024-03-09T15:36:33.761432+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.761432+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:47:50.469481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":166,"num_read_kb":138,"num_write":64,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":65,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3686,"num_read_kb":2614,"num_write":1236,"num_write_kb":514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6449,"num_omap_keys":13,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":13176,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":8518,"num_read_kb":8294,"num_write":8580,"num_write_kb":2546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2306,"num_omap_keys":14,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":26352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1984,"ondisk_log_size":1984,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":64,"acting":64,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":377960,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":12181,"num_read_kb":21090,"num_write":9023,"num_write_kb":5457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":306916,"num_omap_keys":744,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1857,"ondisk_log_size":1857,"up":64,"acting":64,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1719,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1602,"num_read_kb":1903,"num_write":22,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":38590,"num_read_kb":31236,"num_write":15314,"num_write_kb":7493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":33054,"num_omap_keys":137,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3953,"ondisk_log_size":3953,"up":128,"acting":128,"num_store_stats":8},{"poolid":68,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1128,"num_omap_keys":12,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":135300586,"num_objects":320,"num_object_clones":0,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":14057,"num_read_kb":1121153,"num_write":34295,"num_write_kb":1701153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":271892480,"data_stored":270601172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3168,"ondisk_log_size":3168,"up":128,"acting":128,"num_store_stats":8},{"poolid":67,"num_pg":32,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":140,"num_read_kb":192739,"num_write":156,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":66,"num_pg":32,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":138,"num_read_kb":192649,"num_write":156,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575684,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293320,"kb_used_data":45568,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":94078520,"statfs":{"total":96636764160,"available":96336404480,"internally_reserved":0,"allocated":46661632,"data_stored":45012346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":253688266},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575679,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233960,"kb_used_data":22304,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":94137880,"statfs":{"total":96636764160,"available":96397189120,"internally_reserved":0,"allocated":22839296,"data_stored":21261512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575680,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132532,"kb_used_data":61164,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94239308,"statfs":{"total":96636764160,"available":96501051392,"internally_reserved":0,"allocated":62631936,"data_stored":60939198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575684,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100148,"kb_used_data":55084,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94271692,"statfs":{"total":96636764160,"available":96534212608,"internally_reserved":0,"allocated":56406016,"data_stored":54802077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575679,"num_pgs":118,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252776,"kb_used_data":41120,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":94119064,"statfs":{"total":96636764160,"available":96377921536,"internally_reserved":0,"allocated":42106880,"data_stored":40381748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":216725959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575681,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91428,"kb_used_data":38044,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94280412,"statfs":{"total":96636764160,"available":96543141888,"internally_reserved":0,"allocated":38957056,"data_stored":37358501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575680,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131376,"kb_used_data":41960,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94240464,"statfs":{"total":96636764160,"available":96502235136,"internally_reserved":0,"allocated":42967040,"data_stored":41342836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575682,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244644,"kb_used_data":55196,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":94127196,"statfs":{"total":96636764160,"available":96386248704,"internally_reserved":0,"allocated":56520704,"data_stored":54866964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40763392,"data_stored":40607042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23621632,"data_stored":23479247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30126080,"data_stored":29981692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31137792,"data_stored":30907070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":42364928,"data_stored":42183762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":48488448,"data_stored":48318049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":19247104,"data_stored":19128319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36143104,"data_stored":35995991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":3901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":2727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10485760,"data_stored":10485760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":1048576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15728640,"data_stored":15728640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":1048576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10485760,"data_stored":10485760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T15:37:15.980 INFO:tasks.ceph:pgid 68.19 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:37:15.980 INFO:tasks.ceph:pgid 68.1e last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:37:15.980 INFO:tasks.ceph:pgid 68.16 last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:37:15.981 INFO:tasks.ceph:pgid 68.e last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:37:15.981 INFO:tasks.ceph:pgid 68.d last_scrub_stamp 2024-03-09T15:09:49.016993+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=9, tm_sec=49, tm_wday=5, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=9, tm_hour=15, tm_min=36, tm_sec=9, tm_wday=5, tm_yday=69, tm_isdst=0) 2024-03-09T15:37:15.985 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-09T15:37:35.986 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T15:37:36.369 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-09T15:37:36.381 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-09T15:37:36.404 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":2335,"stamp":"2024-03-09T15:37:35.310396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":169838241,"num_objects":629,"num_object_clones":0,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":0,"num_read":79040,"num_read_kb":1571790,"num_write":68959,"num_write_kb":1988151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":349853,"num_omap_keys":920,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12598,"ondisk_log_size":12598,"up":786,"acting":786,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":786,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1480184,"kb_used_data":360440,"kb_used_omap":12,"kb_used_meta":1119667,"kb_avail":753494536,"statfs":{"total":773094113280,"available":771578404864,"internally_reserved":0,"allocated":369090560,"data_stored":355965182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1146539568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007826"},"pg_stats":[{"pgid":"68.1a","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:58.883184+0000","last_change":"2024-03-09T15:36:58.883184+0000","last_active":"2024-03-09T15:36:58.883184+0000","last_peered":"2024-03-09T15:36:58.883184+0000","last_clean":"2024-03-09T15:36:58.883184+0000","last_became_active":"2024-03-09T15:09:50.084784+0000","last_became_peered":"2024-03-09T15:09:50.084784+0000","last_unstale":"2024-03-09T15:36:58.883184+0000","last_undegraded":"2024-03-09T15:36:58.883184+0000","last_fullsized":"2024-03-09T15:36:58.883184+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:58.882933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:58.882933+0000","last_clean_scrub_stamp":"2024-03-09T15:36:58.882933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:28:18.565529+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.1d","version":"261'8","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:56.901811+0000","last_change":"2024-03-09T15:36:56.901918+0000","last_active":"2024-03-09T15:36:56.901811+0000","last_peered":"2024-03-09T15:36:56.901811+0000","last_clean":"2024-03-09T15:36:56.901811+0000","last_became_active":"2024-03-09T14:46:26.825404+0000","last_became_peered":"2024-03-09T14:46:26.825404+0000","last_unstale":"2024-03-09T15:36:56.901811+0000","last_undegraded":"2024-03-09T15:36:56.901811+0000","last_fullsized":"2024-03-09T15:36:56.901811+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:56.901625+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:56.901625+0000","last_clean_scrub_stamp":"2024-03-09T15:36:56.901625+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:57:06.137533+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20498,"num_write":6,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1c","version":"261'2","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.429218+0000","last_change":"2024-03-09T15:36:17.429218+0000","last_active":"2024-03-09T15:36:17.429218+0000","last_peered":"2024-03-09T15:36:17.429218+0000","last_clean":"2024-03-09T15:36:17.429218+0000","last_became_active":"2024-03-09T14:46:18.768383+0000","last_became_peered":"2024-03-09T14:46:18.768383+0000","last_unstale":"2024-03-09T15:36:17.429218+0000","last_undegraded":"2024-03-09T15:36:17.429218+0000","last_fullsized":"2024-03-09T15:36:17.429218+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:17.428960+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:17.428960+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.428960+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:03:37.237056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.1f","version":"258'45","reported_seq":176,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.184209+0000","last_change":"2024-03-09T15:36:12.184266+0000","last_active":"2024-03-09T15:36:12.184209+0000","last_peered":"2024-03-09T15:36:12.184209+0000","last_clean":"2024-03-09T15:36:12.184209+0000","last_became_active":"2024-03-09T14:39:13.281020+0000","last_became_peered":"2024-03-09T14:39:13.281020+0000","last_unstale":"2024-03-09T15:36:12.184209+0000","last_undegraded":"2024-03-09T15:36:12.184209+0000","last_fullsized":"2024-03-09T15:36:12.184209+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'45","last_scrub_stamp":"2024-03-09T15:36:12.183988+0000","last_deep_scrub":"258'45","last_deep_scrub_stamp":"2024-03-09T15:36:12.183988+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.183988+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:21:51.000354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":187,"num_read_kb":134,"num_write":70,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1b","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.284617+0000","last_change":"2024-03-09T15:36:43.284617+0000","last_active":"2024-03-09T15:36:43.284617+0000","last_peered":"2024-03-09T15:36:43.284617+0000","last_clean":"2024-03-09T15:36:43.284617+0000","last_became_active":"2024-03-09T15:09:50.082690+0000","last_became_peered":"2024-03-09T15:09:50.082690+0000","last_unstale":"2024-03-09T15:36:43.284617+0000","last_undegraded":"2024-03-09T15:36:43.284617+0000","last_fullsized":"2024-03-09T15:36:43.284617+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:43.284378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:43.284378+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.284378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:49:16.785671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1c","version":"261'6","reported_seq":67,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:58.594971+0000","last_change":"2024-03-09T15:36:58.595054+0000","last_active":"2024-03-09T15:36:58.594971+0000","last_peered":"2024-03-09T15:36:58.594971+0000","last_clean":"2024-03-09T15:36:58.594971+0000","last_became_active":"2024-03-09T14:46:26.806865+0000","last_became_peered":"2024-03-09T14:46:26.806865+0000","last_unstale":"2024-03-09T15:36:58.594971+0000","last_undegraded":"2024-03-09T15:36:58.594971+0000","last_fullsized":"2024-03-09T15:36:58.594971+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:58.594767+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:58.594767+0000","last_clean_scrub_stamp":"2024-03-09T15:36:58.594767+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:15.479100+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.1d","version":"261'2","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.601517+0000","last_change":"2024-03-09T15:36:17.601517+0000","last_active":"2024-03-09T15:36:17.601517+0000","last_peered":"2024-03-09T15:36:17.601517+0000","last_clean":"2024-03-09T15:36:17.601517+0000","last_became_active":"2024-03-09T14:46:18.779760+0000","last_became_peered":"2024-03-09T14:46:18.779760+0000","last_unstale":"2024-03-09T15:36:17.601517+0000","last_undegraded":"2024-03-09T15:36:17.601517+0000","last_fullsized":"2024-03-09T15:36:17.601517+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:17.601237+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:17.601237+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.601237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:48:08.721381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1e","version":"0'0","reported_seq":72,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.210983+0000","last_change":"2024-03-09T15:36:22.210983+0000","last_active":"2024-03-09T15:36:22.210983+0000","last_peered":"2024-03-09T15:36:22.210983+0000","last_clean":"2024-03-09T15:36:22.210983+0000","last_became_active":"2024-03-09T14:39:13.259731+0000","last_became_peered":"2024-03-09T14:39:13.259731+0000","last_unstale":"2024-03-09T15:36:22.210983+0000","last_undegraded":"2024-03-09T15:36:22.210983+0000","last_fullsized":"2024-03-09T15:36:22.210983+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.210700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.210700+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.210700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:46:50.001547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.18","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.409171+0000","last_change":"2024-03-09T15:36:51.409229+0000","last_active":"2024-03-09T15:36:51.409171+0000","last_peered":"2024-03-09T15:36:51.409171+0000","last_clean":"2024-03-09T15:36:51.409171+0000","last_became_active":"2024-03-09T15:09:50.080578+0000","last_became_peered":"2024-03-09T15:09:50.080578+0000","last_unstale":"2024-03-09T15:36:51.409171+0000","last_undegraded":"2024-03-09T15:36:51.409171+0000","last_fullsized":"2024-03-09T15:36:51.409171+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:51.408977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:51.408977+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.408977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:39:57.431783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.1f","version":"261'5","reported_seq":65,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:06.047166+0000","last_change":"2024-03-09T15:37:06.047229+0000","last_active":"2024-03-09T15:37:06.047166+0000","last_peered":"2024-03-09T15:37:06.047166+0000","last_clean":"2024-03-09T15:37:06.047166+0000","last_became_active":"2024-03-09T14:46:26.895608+0000","last_became_peered":"2024-03-09T14:46:26.895608+0000","last_unstale":"2024-03-09T15:37:06.047166+0000","last_undegraded":"2024-03-09T15:37:06.047166+0000","last_fullsized":"2024-03-09T15:37:06.047166+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'5","last_scrub_stamp":"2024-03-09T15:37:06.046969+0000","last_deep_scrub":"261'5","last_deep_scrub_stamp":"2024-03-09T15:37:06.046969+0000","last_clean_scrub_stamp":"2024-03-09T15:37:06.046969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:09:38.936416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":12288,"num_write":5,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1e","version":"0'0","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.247327+0000","last_change":"2024-03-09T15:36:39.247436+0000","last_active":"2024-03-09T15:36:39.247327+0000","last_peered":"2024-03-09T15:36:39.247327+0000","last_clean":"2024-03-09T15:36:39.247327+0000","last_became_active":"2024-03-09T14:46:18.781856+0000","last_became_peered":"2024-03-09T14:46:18.781856+0000","last_unstale":"2024-03-09T15:36:39.247327+0000","last_undegraded":"2024-03-09T15:36:39.247327+0000","last_fullsized":"2024-03-09T15:36:39.247327+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:39.247089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:39.247089+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.247089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:10:03.506121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.1d","version":"258'35","reported_seq":152,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.273912+0000","last_change":"2024-03-09T15:36:17.273912+0000","last_active":"2024-03-09T15:36:17.273912+0000","last_peered":"2024-03-09T15:36:17.273912+0000","last_clean":"2024-03-09T15:36:17.273912+0000","last_became_active":"2024-03-09T14:39:13.258351+0000","last_became_peered":"2024-03-09T14:39:13.258351+0000","last_unstale":"2024-03-09T15:36:17.273912+0000","last_undegraded":"2024-03-09T15:36:17.273912+0000","last_fullsized":"2024-03-09T15:36:17.273912+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'35","last_scrub_stamp":"2024-03-09T15:36:17.273654+0000","last_deep_scrub":"258'35","last_deep_scrub_stamp":"2024-03-09T15:36:17.273654+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.273654+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:22:59.569323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":152,"num_read_kb":109,"num_write":57,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.19","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:11.676155+0000","last_change":"2024-03-09T15:37:11.676212+0000","last_active":"2024-03-09T15:37:11.676155+0000","last_peered":"2024-03-09T15:37:11.676155+0000","last_clean":"2024-03-09T15:37:11.676155+0000","last_became_active":"2024-03-09T15:09:50.081680+0000","last_became_peered":"2024-03-09T15:09:50.081680+0000","last_unstale":"2024-03-09T15:37:11.676155+0000","last_undegraded":"2024-03-09T15:37:11.676155+0000","last_fullsized":"2024-03-09T15:37:11.676155+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:11.675952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:11.675952+0000","last_clean_scrub_stamp":"2024-03-09T15:37:11.675952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:26:27.513464+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.1e","version":"261'6","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:07.013901+0000","last_change":"2024-03-09T15:37:07.013901+0000","last_active":"2024-03-09T15:37:07.013901+0000","last_peered":"2024-03-09T15:37:07.013901+0000","last_clean":"2024-03-09T15:37:07.013901+0000","last_became_active":"2024-03-09T14:47:07.798247+0000","last_became_peered":"2024-03-09T14:47:07.798247+0000","last_unstale":"2024-03-09T15:37:07.013901+0000","last_undegraded":"2024-03-09T15:37:07.013901+0000","last_fullsized":"2024-03-09T15:37:07.013901+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:37:07.013452+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:37:07.013452+0000","last_clean_scrub_stamp":"2024-03-09T15:37:07.013452+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:56:28.347945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":16384,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1f","version":"258'5","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.180362+0000","last_change":"2024-03-09T15:36:25.180427+0000","last_active":"2024-03-09T15:36:25.180362+0000","last_peered":"2024-03-09T15:36:25.180362+0000","last_clean":"2024-03-09T15:36:25.180362+0000","last_became_active":"2024-03-09T14:46:18.771731+0000","last_became_peered":"2024-03-09T14:46:18.771731+0000","last_unstale":"2024-03-09T15:36:25.180362+0000","last_undegraded":"2024-03-09T15:36:25.180362+0000","last_fullsized":"2024-03-09T15:36:25.180362+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'5","last_scrub_stamp":"2024-03-09T15:36:25.180164+0000","last_deep_scrub":"258'5","last_deep_scrub_stamp":"2024-03-09T15:36:25.180164+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.180164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:55:32.192979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5136,"num_write":5,"num_write_kb":5136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1c","version":"261'13","reported_seq":99,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.507564+0000","last_change":"2024-03-09T15:36:15.507564+0000","last_active":"2024-03-09T15:36:15.507564+0000","last_peered":"2024-03-09T15:36:15.507564+0000","last_clean":"2024-03-09T15:36:15.507564+0000","last_became_active":"2024-03-09T14:39:13.260945+0000","last_became_peered":"2024-03-09T14:39:13.260945+0000","last_unstale":"2024-03-09T15:36:15.507564+0000","last_undegraded":"2024-03-09T15:36:15.507564+0000","last_fullsized":"2024-03-09T15:36:15.507564+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'13","last_scrub_stamp":"2024-03-09T15:36:15.507291+0000","last_deep_scrub":"261'13","last_deep_scrub_stamp":"2024-03-09T15:36:15.507291+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.507291+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:29:48.463973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":27,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1e","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:14.626394+0000","last_change":"2024-03-09T15:37:14.626447+0000","last_active":"2024-03-09T15:37:14.626394+0000","last_peered":"2024-03-09T15:37:14.626394+0000","last_clean":"2024-03-09T15:37:14.626394+0000","last_became_active":"2024-03-09T15:09:50.086789+0000","last_became_peered":"2024-03-09T15:09:50.086789+0000","last_unstale":"2024-03-09T15:37:14.626394+0000","last_undegraded":"2024-03-09T15:37:14.626394+0000","last_fullsized":"2024-03-09T15:37:14.626394+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:14.626200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:14.626200+0000","last_clean_scrub_stamp":"2024-03-09T15:37:14.626200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:41:24.648275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.19","version":"261'6","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:55.893852+0000","last_change":"2024-03-09T15:36:55.893852+0000","last_active":"2024-03-09T15:36:55.893852+0000","last_peered":"2024-03-09T15:36:55.893852+0000","last_clean":"2024-03-09T15:36:55.893852+0000","last_became_active":"2024-03-09T14:46:26.819085+0000","last_became_peered":"2024-03-09T14:46:26.819085+0000","last_unstale":"2024-03-09T15:36:55.893852+0000","last_undegraded":"2024-03-09T15:36:55.893852+0000","last_fullsized":"2024-03-09T15:36:55.893852+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:55.893565+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:55.893565+0000","last_clean_scrub_stamp":"2024-03-09T15:36:55.893565+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:27:43.536475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":12288,"num_write":11,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.18","version":"261'14","reported_seq":90,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.270917+0000","last_change":"2024-03-09T15:36:27.270917+0000","last_active":"2024-03-09T15:36:27.270917+0000","last_peered":"2024-03-09T15:36:27.270917+0000","last_clean":"2024-03-09T15:36:27.270917+0000","last_became_active":"2024-03-09T14:46:18.776161+0000","last_became_peered":"2024-03-09T14:46:18.776161+0000","last_unstale":"2024-03-09T15:36:27.270917+0000","last_undegraded":"2024-03-09T15:36:27.270917+0000","last_fullsized":"2024-03-09T15:36:27.270917+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'14","last_scrub_stamp":"2024-03-09T15:36:27.270455+0000","last_deep_scrub":"261'14","last_deep_scrub_stamp":"2024-03-09T15:36:27.270455+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.270455+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:23:13.389149+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":36889,"num_write":17,"num_write_kb":16407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.1b","version":"258'10","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.553151+0000","last_change":"2024-03-09T15:36:14.553151+0000","last_active":"2024-03-09T15:36:14.553151+0000","last_peered":"2024-03-09T15:36:14.553151+0000","last_clean":"2024-03-09T15:36:14.553151+0000","last_became_active":"2024-03-09T14:39:13.268179+0000","last_became_peered":"2024-03-09T14:39:13.268179+0000","last_unstale":"2024-03-09T15:36:14.553151+0000","last_undegraded":"2024-03-09T15:36:14.553151+0000","last_fullsized":"2024-03-09T15:36:14.553151+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:14.552885+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:14.552885+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.552885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:47:39.435007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.1f","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:52.182480+0000","last_change":"2024-03-09T15:36:52.182480+0000","last_active":"2024-03-09T15:36:52.182480+0000","last_peered":"2024-03-09T15:36:52.182480+0000","last_clean":"2024-03-09T15:36:52.182480+0000","last_became_active":"2024-03-09T15:09:50.063705+0000","last_became_peered":"2024-03-09T15:09:50.063705+0000","last_unstale":"2024-03-09T15:36:52.182480+0000","last_undegraded":"2024-03-09T15:36:52.182480+0000","last_fullsized":"2024-03-09T15:36:52.182480+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:52.182104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:52.182104+0000","last_clean_scrub_stamp":"2024-03-09T15:36:52.182104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:42:53.890576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.18","version":"261'3","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.314986+0000","last_change":"2024-03-09T15:36:48.314986+0000","last_active":"2024-03-09T15:36:48.314986+0000","last_peered":"2024-03-09T15:36:48.314986+0000","last_clean":"2024-03-09T15:36:48.314986+0000","last_became_active":"2024-03-09T14:46:26.824522+0000","last_became_peered":"2024-03-09T14:46:26.824522+0000","last_unstale":"2024-03-09T15:36:48.314986+0000","last_undegraded":"2024-03-09T15:36:48.314986+0000","last_fullsized":"2024-03-09T15:36:48.314986+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:48.314729+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:48.314729+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.314729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:53:25.103105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.19","version":"261'9","reported_seq":76,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.166321+0000","last_change":"2024-03-09T15:36:17.166321+0000","last_active":"2024-03-09T15:36:17.166321+0000","last_peered":"2024-03-09T15:36:17.166321+0000","last_clean":"2024-03-09T15:36:17.166321+0000","last_became_active":"2024-03-09T14:46:18.769730+0000","last_became_peered":"2024-03-09T14:46:18.769730+0000","last_unstale":"2024-03-09T15:36:17.166321+0000","last_undegraded":"2024-03-09T15:36:17.166321+0000","last_fullsized":"2024-03-09T15:36:17.166321+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-03-09T15:36:17.165994+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-03-09T15:36:17.165994+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.165994+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:56:10.015397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":19456,"num_write":19,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.1a","version":"261'44","reported_seq":174,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.008442+0000","last_change":"2024-03-09T15:36:14.008442+0000","last_active":"2024-03-09T15:36:14.008442+0000","last_peered":"2024-03-09T15:36:14.008442+0000","last_clean":"2024-03-09T15:36:14.008442+0000","last_became_active":"2024-03-09T14:39:13.273298+0000","last_became_peered":"2024-03-09T14:39:13.273298+0000","last_unstale":"2024-03-09T15:36:14.008442+0000","last_undegraded":"2024-03-09T15:36:14.008442+0000","last_fullsized":"2024-03-09T15:36:14.008442+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:14.008154+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:14.008154+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.008154+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:19:56.477799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":138,"num_write":72,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.1c","version":"261'2","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:08.153982+0000","last_change":"2024-03-09T15:37:08.153982+0000","last_active":"2024-03-09T15:37:08.153982+0000","last_peered":"2024-03-09T15:37:08.153982+0000","last_clean":"2024-03-09T15:37:08.153982+0000","last_became_active":"2024-03-09T15:09:50.074952+0000","last_became_peered":"2024-03-09T15:09:50.074952+0000","last_unstale":"2024-03-09T15:37:08.153982+0000","last_undegraded":"2024-03-09T15:37:08.153982+0000","last_fullsized":"2024-03-09T15:37:08.153982+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:37:08.153733+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:37:08.153733+0000","last_clean_scrub_stamp":"2024-03-09T15:37:08.153733+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:29:59.718406+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1b","version":"261'3","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.334633+0000","last_change":"2024-03-09T15:36:46.334695+0000","last_active":"2024-03-09T15:36:46.334633+0000","last_peered":"2024-03-09T15:36:46.334633+0000","last_clean":"2024-03-09T15:36:46.334633+0000","last_became_active":"2024-03-09T14:46:26.804622+0000","last_became_peered":"2024-03-09T14:46:26.804622+0000","last_unstale":"2024-03-09T15:36:46.334633+0000","last_undegraded":"2024-03-09T15:36:46.334633+0000","last_fullsized":"2024-03-09T15:36:46.334633+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:46.334433+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:46.334433+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.334433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:30:10.876583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.1a","version":"261'5","reported_seq":65,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:56.540292+0000","last_change":"2024-03-09T15:36:56.540355+0000","last_active":"2024-03-09T15:36:56.540292+0000","last_peered":"2024-03-09T15:36:56.540292+0000","last_clean":"2024-03-09T15:36:56.540292+0000","last_became_active":"2024-03-09T14:46:18.784821+0000","last_became_peered":"2024-03-09T14:46:18.784821+0000","last_unstale":"2024-03-09T15:36:56.540292+0000","last_undegraded":"2024-03-09T15:36:56.540292+0000","last_fullsized":"2024-03-09T15:36:56.540292+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'5","last_scrub_stamp":"2024-03-09T15:36:56.540098+0000","last_deep_scrub":"261'5","last_deep_scrub_stamp":"2024-03-09T15:36:56.540098+0000","last_clean_scrub_stamp":"2024-03-09T15:36:56.540098+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:38:53.526961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4096,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.19","version":"258'28","reported_seq":130,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.263945+0000","last_change":"2024-03-09T15:36:18.263945+0000","last_active":"2024-03-09T15:36:18.263945+0000","last_peered":"2024-03-09T15:36:18.263945+0000","last_clean":"2024-03-09T15:36:18.263945+0000","last_became_active":"2024-03-09T14:39:13.257473+0000","last_became_peered":"2024-03-09T14:39:13.257473+0000","last_unstale":"2024-03-09T15:36:18.263945+0000","last_undegraded":"2024-03-09T15:36:18.263945+0000","last_fullsized":"2024-03-09T15:36:18.263945+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'28","last_scrub_stamp":"2024-03-09T15:36:18.263646+0000","last_deep_scrub":"258'28","last_deep_scrub_stamp":"2024-03-09T15:36:18.263646+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.263646+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:59:14.805307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":78,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.1d","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:07.155622+0000","last_change":"2024-03-09T15:37:07.155622+0000","last_active":"2024-03-09T15:37:07.155622+0000","last_peered":"2024-03-09T15:37:07.155622+0000","last_clean":"2024-03-09T15:37:07.155622+0000","last_became_active":"2024-03-09T15:09:50.074875+0000","last_became_peered":"2024-03-09T15:09:50.074875+0000","last_unstale":"2024-03-09T15:37:07.155622+0000","last_undegraded":"2024-03-09T15:37:07.155622+0000","last_fullsized":"2024-03-09T15:37:07.155622+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:07.155375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:07.155375+0000","last_clean_scrub_stamp":"2024-03-09T15:37:07.155375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:01:55.131735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.1a","version":"261'11","reported_seq":72,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.456175+0000","last_change":"2024-03-09T15:36:21.456175+0000","last_active":"2024-03-09T15:36:21.456175+0000","last_peered":"2024-03-09T15:36:21.456175+0000","last_clean":"2024-03-09T15:36:21.456175+0000","last_became_active":"2024-03-09T14:46:26.812889+0000","last_became_peered":"2024-03-09T14:46:26.812889+0000","last_unstale":"2024-03-09T15:36:21.456175+0000","last_undegraded":"2024-03-09T15:36:21.456175+0000","last_fullsized":"2024-03-09T15:36:21.456175+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'11","last_scrub_stamp":"2024-03-09T15:36:21.455916+0000","last_deep_scrub":"261'11","last_deep_scrub_stamp":"2024-03-09T15:36:21.455916+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.455916+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:07:00.143287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":12290,"num_write":14,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.1b","version":"261'2","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.253249+0000","last_change":"2024-03-09T15:36:19.253249+0000","last_active":"2024-03-09T15:36:19.253249+0000","last_peered":"2024-03-09T15:36:19.253249+0000","last_clean":"2024-03-09T15:36:19.253249+0000","last_became_active":"2024-03-09T14:46:18.756394+0000","last_became_peered":"2024-03-09T14:46:18.756394+0000","last_unstale":"2024-03-09T15:36:19.253249+0000","last_undegraded":"2024-03-09T15:36:19.253249+0000","last_fullsized":"2024-03-09T15:36:19.253249+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:19.252989+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:19.252989+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.252989+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:28:28.982640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":40,"num_write":2,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.18","version":"261'8","reported_seq":87,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.579306+0000","last_change":"2024-03-09T15:36:15.579306+0000","last_active":"2024-03-09T15:36:15.579306+0000","last_peered":"2024-03-09T15:36:15.579306+0000","last_clean":"2024-03-09T15:36:15.579306+0000","last_became_active":"2024-03-09T14:39:13.257996+0000","last_became_peered":"2024-03-09T14:39:13.257996+0000","last_unstale":"2024-03-09T15:36:15.579306+0000","last_undegraded":"2024-03-09T15:36:15.579306+0000","last_fullsized":"2024-03-09T15:36:15.579306+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:15.579034+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:15.579034+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.579034+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:42:49.650613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.12","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:54.156296+0000","last_change":"2024-03-09T15:36:54.156410+0000","last_active":"2024-03-09T15:36:54.156296+0000","last_peered":"2024-03-09T15:36:54.156296+0000","last_clean":"2024-03-09T15:36:54.156296+0000","last_became_active":"2024-03-09T15:09:50.070345+0000","last_became_peered":"2024-03-09T15:09:50.070345+0000","last_unstale":"2024-03-09T15:36:54.156296+0000","last_undegraded":"2024-03-09T15:36:54.156296+0000","last_fullsized":"2024-03-09T15:36:54.156296+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:54.156066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:54.156066+0000","last_clean_scrub_stamp":"2024-03-09T15:36:54.156066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:09:57.089544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.15","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.552821+0000","last_change":"2024-03-09T15:36:41.552821+0000","last_active":"2024-03-09T15:36:41.552821+0000","last_peered":"2024-03-09T15:36:41.552821+0000","last_clean":"2024-03-09T15:36:41.552821+0000","last_became_active":"2024-03-09T14:46:26.890992+0000","last_became_peered":"2024-03-09T14:46:26.890992+0000","last_unstale":"2024-03-09T15:36:41.552821+0000","last_undegraded":"2024-03-09T15:36:41.552821+0000","last_fullsized":"2024-03-09T15:36:41.552821+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:41.552560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:41.552560+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.552560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:48:25.323567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.14","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.701752+0000","last_change":"2024-03-09T15:36:22.701809+0000","last_active":"2024-03-09T15:36:22.701752+0000","last_peered":"2024-03-09T15:36:22.701752+0000","last_clean":"2024-03-09T15:36:22.701752+0000","last_became_active":"2024-03-09T14:46:18.766777+0000","last_became_peered":"2024-03-09T14:46:18.766777+0000","last_unstale":"2024-03-09T15:36:22.701752+0000","last_undegraded":"2024-03-09T15:36:22.701752+0000","last_fullsized":"2024-03-09T15:36:22.701752+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.701528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.701528+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.701528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:03:22.022639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.17","version":"258'19","reported_seq":116,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.216647+0000","last_change":"2024-03-09T15:36:13.216647+0000","last_active":"2024-03-09T15:36:13.216647+0000","last_peered":"2024-03-09T15:36:13.216647+0000","last_clean":"2024-03-09T15:36:13.216647+0000","last_became_active":"2024-03-09T14:39:13.279919+0000","last_became_peered":"2024-03-09T14:39:13.279919+0000","last_unstale":"2024-03-09T15:36:13.216647+0000","last_undegraded":"2024-03-09T15:36:13.216647+0000","last_fullsized":"2024-03-09T15:36:13.216647+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'19","last_scrub_stamp":"2024-03-09T15:36:13.216485+0000","last_deep_scrub":"258'19","last_deep_scrub_stamp":"2024-03-09T15:36:13.216485+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.216485+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:22:26.193181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":64,"num_write":32,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.13","version":"261'2","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.296653+0000","last_change":"2024-03-09T15:36:41.296724+0000","last_active":"2024-03-09T15:36:41.296653+0000","last_peered":"2024-03-09T15:36:41.296653+0000","last_clean":"2024-03-09T15:36:41.296653+0000","last_became_active":"2024-03-09T15:09:50.079604+0000","last_became_peered":"2024-03-09T15:09:50.079604+0000","last_unstale":"2024-03-09T15:36:41.296653+0000","last_undegraded":"2024-03-09T15:36:41.296653+0000","last_fullsized":"2024-03-09T15:36:41.296653+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:41.296454+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:41.296454+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.296454+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:36:03.858346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.14","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:09.702163+0000","last_change":"2024-03-09T15:37:09.702224+0000","last_active":"2024-03-09T15:37:09.702163+0000","last_peered":"2024-03-09T15:37:09.702163+0000","last_clean":"2024-03-09T15:37:09.702163+0000","last_became_active":"2024-03-09T14:46:26.894088+0000","last_became_peered":"2024-03-09T14:46:26.894088+0000","last_unstale":"2024-03-09T15:37:09.702163+0000","last_undegraded":"2024-03-09T15:37:09.702163+0000","last_fullsized":"2024-03-09T15:37:09.702163+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:09.701920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:09.701920+0000","last_clean_scrub_stamp":"2024-03-09T15:37:09.701920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:49:58.460178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.15","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.232245+0000","last_change":"2024-03-09T15:36:16.232245+0000","last_active":"2024-03-09T15:36:16.232245+0000","last_peered":"2024-03-09T15:36:16.232245+0000","last_clean":"2024-03-09T15:36:16.232245+0000","last_became_active":"2024-03-09T14:46:18.752413+0000","last_became_peered":"2024-03-09T14:46:18.752413+0000","last_unstale":"2024-03-09T15:36:16.232245+0000","last_undegraded":"2024-03-09T15:36:16.232245+0000","last_fullsized":"2024-03-09T15:36:16.232245+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:16.232005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:16.232005+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.232005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:59:24.622129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.16","version":"261'52","reported_seq":194,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.459199+0000","last_change":"2024-03-09T15:36:16.459199+0000","last_active":"2024-03-09T15:36:16.459199+0000","last_peered":"2024-03-09T15:36:16.459199+0000","last_clean":"2024-03-09T15:36:16.459199+0000","last_became_active":"2024-03-09T14:39:13.262413+0000","last_became_peered":"2024-03-09T14:39:13.262413+0000","last_unstale":"2024-03-09T15:36:16.459199+0000","last_undegraded":"2024-03-09T15:36:16.459199+0000","last_fullsized":"2024-03-09T15:36:16.459199+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'52","last_scrub_stamp":"2024-03-09T15:36:16.458932+0000","last_deep_scrub":"261'52","last_deep_scrub_stamp":"2024-03-09T15:36:16.458932+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.458932+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:28:33.068618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":163,"num_write":78,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.10","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:06.105683+0000","last_change":"2024-03-09T15:37:06.105847+0000","last_active":"2024-03-09T15:37:06.105683+0000","last_peered":"2024-03-09T15:37:06.105683+0000","last_clean":"2024-03-09T15:37:06.105683+0000","last_became_active":"2024-03-09T15:09:50.074274+0000","last_became_peered":"2024-03-09T15:09:50.074274+0000","last_unstale":"2024-03-09T15:37:06.105683+0000","last_undegraded":"2024-03-09T15:37:06.105683+0000","last_fullsized":"2024-03-09T15:37:06.105683+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:06.105130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:06.105130+0000","last_clean_scrub_stamp":"2024-03-09T15:37:06.105130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:58:01.351666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.17","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:08.730877+0000","last_change":"2024-03-09T15:37:08.730877+0000","last_active":"2024-03-09T15:37:08.730877+0000","last_peered":"2024-03-09T15:37:08.730877+0000","last_clean":"2024-03-09T15:37:08.730877+0000","last_became_active":"2024-03-09T14:46:26.893514+0000","last_became_peered":"2024-03-09T14:46:26.893514+0000","last_unstale":"2024-03-09T15:37:08.730877+0000","last_undegraded":"2024-03-09T15:37:08.730877+0000","last_fullsized":"2024-03-09T15:37:08.730877+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:08.730628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:08.730628+0000","last_clean_scrub_stamp":"2024-03-09T15:37:08.730628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:26:35.643642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.16","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.232040+0000","last_change":"2024-03-09T15:36:38.232040+0000","last_active":"2024-03-09T15:36:38.232040+0000","last_peered":"2024-03-09T15:36:38.232040+0000","last_clean":"2024-03-09T15:36:38.232040+0000","last_became_active":"2024-03-09T14:46:18.769274+0000","last_became_peered":"2024-03-09T14:46:18.769274+0000","last_unstale":"2024-03-09T15:36:38.232040+0000","last_undegraded":"2024-03-09T15:36:38.232040+0000","last_fullsized":"2024-03-09T15:36:38.232040+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:38.231785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:38.231785+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.231785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:50:40.645273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.15","version":"0'0","reported_seq":67,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.220202+0000","last_change":"2024-03-09T15:36:24.220202+0000","last_active":"2024-03-09T15:36:24.220202+0000","last_peered":"2024-03-09T15:36:24.220202+0000","last_clean":"2024-03-09T15:36:24.220202+0000","last_became_active":"2024-03-09T14:39:13.262066+0000","last_became_peered":"2024-03-09T14:39:13.262066+0000","last_unstale":"2024-03-09T15:36:24.220202+0000","last_undegraded":"2024-03-09T15:36:24.220202+0000","last_fullsized":"2024-03-09T15:36:24.220202+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:24.219947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:24.219947+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.219947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:56:44.705423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.11","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:59.918041+0000","last_change":"2024-03-09T15:36:59.918095+0000","last_active":"2024-03-09T15:36:59.918041+0000","last_peered":"2024-03-09T15:36:59.918041+0000","last_clean":"2024-03-09T15:36:59.918041+0000","last_became_active":"2024-03-09T15:09:50.085346+0000","last_became_peered":"2024-03-09T15:09:50.085346+0000","last_unstale":"2024-03-09T15:36:59.918041+0000","last_undegraded":"2024-03-09T15:36:59.918041+0000","last_fullsized":"2024-03-09T15:36:59.918041+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:59.917834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:59.917834+0000","last_clean_scrub_stamp":"2024-03-09T15:36:59.917834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:16:55.219020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.16","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:01.614261+0000","last_change":"2024-03-09T15:37:01.614261+0000","last_active":"2024-03-09T15:37:01.614261+0000","last_peered":"2024-03-09T15:37:01.614261+0000","last_clean":"2024-03-09T15:37:01.614261+0000","last_became_active":"2024-03-09T14:46:26.817864+0000","last_became_peered":"2024-03-09T14:46:26.817864+0000","last_unstale":"2024-03-09T15:37:01.614261+0000","last_undegraded":"2024-03-09T15:37:01.614261+0000","last_fullsized":"2024-03-09T15:37:01.614261+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:01.614008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:01.614008+0000","last_clean_scrub_stamp":"2024-03-09T15:37:01.614008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:43:47.987698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.17","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.208514+0000","last_change":"2024-03-09T15:36:31.208514+0000","last_active":"2024-03-09T15:36:31.208514+0000","last_peered":"2024-03-09T15:36:31.208514+0000","last_clean":"2024-03-09T15:36:31.208514+0000","last_became_active":"2024-03-09T14:46:18.776430+0000","last_became_peered":"2024-03-09T14:46:18.776430+0000","last_unstale":"2024-03-09T15:36:31.208514+0000","last_undegraded":"2024-03-09T15:36:31.208514+0000","last_fullsized":"2024-03-09T15:36:31.208514+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:31.208147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:31.208147+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.208147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:52:30.734801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.14","version":"261'68","reported_seq":280,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.178145+0000","last_change":"2024-03-09T15:36:24.178221+0000","last_active":"2024-03-09T15:36:24.178145+0000","last_peered":"2024-03-09T15:36:24.178145+0000","last_clean":"2024-03-09T15:36:24.178145+0000","last_became_active":"2024-03-09T14:39:13.253007+0000","last_became_peered":"2024-03-09T14:39:13.253007+0000","last_unstale":"2024-03-09T15:36:24.178145+0000","last_undegraded":"2024-03-09T15:36:24.178145+0000","last_fullsized":"2024-03-09T15:36:24.178145+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'68","last_scrub_stamp":"2024-03-09T15:36:24.177894+0000","last_deep_scrub":"261'68","last_deep_scrub_stamp":"2024-03-09T15:36:24.177894+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.177894+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:09:13.568231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":411,"num_read_kb":279,"num_write":87,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4473,"num_omap_keys":9,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.16","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:13.663849+0000","last_change":"2024-03-09T15:37:13.663904+0000","last_active":"2024-03-09T15:37:13.663849+0000","last_peered":"2024-03-09T15:37:13.663849+0000","last_clean":"2024-03-09T15:37:13.663849+0000","last_became_active":"2024-03-09T15:09:50.086173+0000","last_became_peered":"2024-03-09T15:09:50.086173+0000","last_unstale":"2024-03-09T15:37:13.663849+0000","last_undegraded":"2024-03-09T15:37:13.663849+0000","last_fullsized":"2024-03-09T15:37:13.663849+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:13.663618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:13.663618+0000","last_clean_scrub_stamp":"2024-03-09T15:37:13.663618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:30:51.350552+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.11","version":"261'8","reported_seq":71,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.163105+0000","last_change":"2024-03-09T15:36:50.163269+0000","last_active":"2024-03-09T15:36:50.163105+0000","last_peered":"2024-03-09T15:36:50.163105+0000","last_clean":"2024-03-09T15:36:50.163105+0000","last_became_active":"2024-03-09T14:46:26.817014+0000","last_became_peered":"2024-03-09T14:46:26.817014+0000","last_unstale":"2024-03-09T15:36:50.163105+0000","last_undegraded":"2024-03-09T15:36:50.163105+0000","last_fullsized":"2024-03-09T15:36:50.163105+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:50.162856+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:50.162856+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.162856+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:04:50.027298+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":23554,"num_write":11,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.10","version":"261'3","reported_seq":63,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.121627+0000","last_change":"2024-03-09T15:36:21.121690+0000","last_active":"2024-03-09T15:36:21.121627+0000","last_peered":"2024-03-09T15:36:21.121627+0000","last_clean":"2024-03-09T15:36:21.121627+0000","last_became_active":"2024-03-09T14:46:18.770936+0000","last_became_peered":"2024-03-09T14:46:18.770936+0000","last_unstale":"2024-03-09T15:36:21.121627+0000","last_undegraded":"2024-03-09T15:36:21.121627+0000","last_fullsized":"2024-03-09T15:36:21.121627+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:21.121408+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:21.121408+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.121408+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:44:45.507890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.13","version":"258'16","reported_seq":104,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.204733+0000","last_change":"2024-03-09T15:36:22.204801+0000","last_active":"2024-03-09T15:36:22.204733+0000","last_peered":"2024-03-09T15:36:22.204733+0000","last_clean":"2024-03-09T15:36:22.204733+0000","last_became_active":"2024-03-09T14:39:13.266228+0000","last_became_peered":"2024-03-09T14:39:13.266228+0000","last_unstale":"2024-03-09T15:36:22.204733+0000","last_undegraded":"2024-03-09T15:36:22.204733+0000","last_fullsized":"2024-03-09T15:36:22.204733+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'16","last_scrub_stamp":"2024-03-09T15:36:22.204502+0000","last_deep_scrub":"258'16","last_deep_scrub_stamp":"2024-03-09T15:36:22.204502+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.204502+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:33:31.523340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":45,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.17","version":"0'0","reported_seq":32,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:11.014947+0000","last_change":"2024-03-09T15:37:11.015000+0000","last_active":"2024-03-09T15:37:11.014947+0000","last_peered":"2024-03-09T15:37:11.014947+0000","last_clean":"2024-03-09T15:37:11.014947+0000","last_became_active":"2024-03-09T15:09:50.075187+0000","last_became_peered":"2024-03-09T15:09:50.075187+0000","last_unstale":"2024-03-09T15:37:11.014947+0000","last_undegraded":"2024-03-09T15:37:11.014947+0000","last_fullsized":"2024-03-09T15:37:11.014947+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:11.014747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:11.014747+0000","last_clean_scrub_stamp":"2024-03-09T15:37:11.014747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:23:32.716146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.10","version":"0'0","reported_seq":52,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.190342+0000","last_change":"2024-03-09T15:36:48.190396+0000","last_active":"2024-03-09T15:36:48.190342+0000","last_peered":"2024-03-09T15:36:48.190342+0000","last_clean":"2024-03-09T15:36:48.190342+0000","last_became_active":"2024-03-09T14:46:26.802476+0000","last_became_peered":"2024-03-09T14:46:26.802476+0000","last_unstale":"2024-03-09T15:36:48.190342+0000","last_undegraded":"2024-03-09T15:36:48.190342+0000","last_fullsized":"2024-03-09T15:36:48.190342+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:48.190138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:48.190138+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.190138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:25.403833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.11","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.548131+0000","last_change":"2024-03-09T15:36:13.548167+0000","last_active":"2024-03-09T15:36:13.548131+0000","last_peered":"2024-03-09T15:36:13.548131+0000","last_clean":"2024-03-09T15:36:13.548131+0000","last_became_active":"2024-03-09T14:46:18.760959+0000","last_became_peered":"2024-03-09T14:46:18.760959+0000","last_unstale":"2024-03-09T15:36:13.548131+0000","last_undegraded":"2024-03-09T15:36:13.548131+0000","last_fullsized":"2024-03-09T15:36:13.548131+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:13.548016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:13.548016+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.548016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:46:22.640051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"65.12","version":"258'7","reported_seq":86,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.252904+0000","last_change":"2024-03-09T15:36:16.252969+0000","last_active":"2024-03-09T15:36:16.252904+0000","last_peered":"2024-03-09T15:36:16.252904+0000","last_clean":"2024-03-09T15:36:16.252904+0000","last_became_active":"2024-03-09T14:39:13.273740+0000","last_became_peered":"2024-03-09T14:39:13.273740+0000","last_unstale":"2024-03-09T15:36:16.252904+0000","last_undegraded":"2024-03-09T15:36:16.252904+0000","last_fullsized":"2024-03-09T15:36:16.252904+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'7","last_scrub_stamp":"2024-03-09T15:36:16.252708+0000","last_deep_scrub":"258'7","last_deep_scrub_stamp":"2024-03-09T15:36:16.252708+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.252708+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:42:28.881579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":26,"num_write":13,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.14","version":"261'2","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:08.964088+0000","last_change":"2024-03-09T15:37:08.964222+0000","last_active":"2024-03-09T15:37:08.964088+0000","last_peered":"2024-03-09T15:37:08.964088+0000","last_clean":"2024-03-09T15:37:08.964088+0000","last_became_active":"2024-03-09T15:09:50.064428+0000","last_became_peered":"2024-03-09T15:09:50.064428+0000","last_unstale":"2024-03-09T15:37:08.964088+0000","last_undegraded":"2024-03-09T15:37:08.964088+0000","last_fullsized":"2024-03-09T15:37:08.964088+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:37:08.963834+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:37:08.963834+0000","last_clean_scrub_stamp":"2024-03-09T15:37:08.963834+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:30:24.868391+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.13","version":"261'3","reported_seq":57,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.462155+0000","last_change":"2024-03-09T15:36:33.462284+0000","last_active":"2024-03-09T15:36:33.462155+0000","last_peered":"2024-03-09T15:36:33.462155+0000","last_clean":"2024-03-09T15:36:33.462155+0000","last_became_active":"2024-03-09T14:46:26.813088+0000","last_became_peered":"2024-03-09T14:46:26.813088+0000","last_unstale":"2024-03-09T15:36:33.462155+0000","last_undegraded":"2024-03-09T15:36:33.462155+0000","last_fullsized":"2024-03-09T15:36:33.462155+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:33.461831+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:33.461831+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.461831+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:23:33.077781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.12","version":"261'6","reported_seq":68,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.313727+0000","last_change":"2024-03-09T15:36:45.313892+0000","last_active":"2024-03-09T15:36:45.313727+0000","last_peered":"2024-03-09T15:36:45.313727+0000","last_clean":"2024-03-09T15:36:45.313727+0000","last_became_active":"2024-03-09T14:46:18.782158+0000","last_became_peered":"2024-03-09T14:46:18.782158+0000","last_unstale":"2024-03-09T15:36:45.313727+0000","last_undegraded":"2024-03-09T15:36:45.313727+0000","last_fullsized":"2024-03-09T15:36:45.313727+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:45.313488+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:45.313488+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.313488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:37.339820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":11264,"num_write":6,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.11","version":"258'36","reported_seq":157,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.155312+0000","last_change":"2024-03-09T15:36:14.155374+0000","last_active":"2024-03-09T15:36:14.155312+0000","last_peered":"2024-03-09T15:36:14.155312+0000","last_clean":"2024-03-09T15:36:14.155312+0000","last_became_active":"2024-03-09T14:39:13.260219+0000","last_became_peered":"2024-03-09T14:39:13.260219+0000","last_unstale":"2024-03-09T15:36:14.155312+0000","last_undegraded":"2024-03-09T15:36:14.155312+0000","last_fullsized":"2024-03-09T15:36:14.155312+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'36","last_scrub_stamp":"2024-03-09T15:36:14.155108+0000","last_deep_scrub":"258'36","last_deep_scrub_stamp":"2024-03-09T15:36:14.155108+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.155108+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:26:18.598712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":118,"num_write":57,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.15","version":"261'2","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:10.665106+0000","last_change":"2024-03-09T15:37:10.665177+0000","last_active":"2024-03-09T15:37:10.665106+0000","last_peered":"2024-03-09T15:37:10.665106+0000","last_clean":"2024-03-09T15:37:10.665106+0000","last_became_active":"2024-03-09T15:09:50.081688+0000","last_became_peered":"2024-03-09T15:09:50.081688+0000","last_unstale":"2024-03-09T15:37:10.665106+0000","last_undegraded":"2024-03-09T15:37:10.665106+0000","last_fullsized":"2024-03-09T15:37:10.665106+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:37:10.664908+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:37:10.664908+0000","last_clean_scrub_stamp":"2024-03-09T15:37:10.664908+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:59:48.186267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.12","version":"261'2","reported_seq":55,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.350486+0000","last_change":"2024-03-09T15:36:47.350548+0000","last_active":"2024-03-09T15:36:47.350486+0000","last_peered":"2024-03-09T15:36:47.350486+0000","last_clean":"2024-03-09T15:36:47.350486+0000","last_became_active":"2024-03-09T14:46:26.806155+0000","last_became_peered":"2024-03-09T14:46:26.806155+0000","last_unstale":"2024-03-09T15:36:47.350486+0000","last_undegraded":"2024-03-09T15:36:47.350486+0000","last_fullsized":"2024-03-09T15:36:47.350486+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:47.350284+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:47.350284+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.350284+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:49:11.128257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.13","version":"261'2","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:53.834708+0000","last_change":"2024-03-09T15:36:53.834774+0000","last_active":"2024-03-09T15:36:53.834708+0000","last_peered":"2024-03-09T15:36:53.834708+0000","last_clean":"2024-03-09T15:36:53.834708+0000","last_became_active":"2024-03-09T14:46:18.782340+0000","last_became_peered":"2024-03-09T14:46:18.782340+0000","last_unstale":"2024-03-09T15:36:53.834708+0000","last_undegraded":"2024-03-09T15:36:53.834708+0000","last_fullsized":"2024-03-09T15:36:53.834708+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:53.834512+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:53.834512+0000","last_clean_scrub_stamp":"2024-03-09T15:36:53.834512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:38:35.688017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":50,"num_write":2,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.10","version":"261'68","reported_seq":266,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.437896+0000","last_change":"2024-03-09T15:36:19.437984+0000","last_active":"2024-03-09T15:36:19.437896+0000","last_peered":"2024-03-09T15:36:19.437896+0000","last_clean":"2024-03-09T15:36:19.437896+0000","last_became_active":"2024-03-09T14:39:13.245107+0000","last_became_peered":"2024-03-09T14:39:13.245107+0000","last_unstale":"2024-03-09T15:36:19.437896+0000","last_undegraded":"2024-03-09T15:36:19.437896+0000","last_fullsized":"2024-03-09T15:36:19.437896+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'68","last_scrub_stamp":"2024-03-09T15:36:19.437698+0000","last_deep_scrub":"261'68","last_deep_scrub_stamp":"2024-03-09T15:36:19.437698+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.437698+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:55:08.019711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":382,"num_read_kb":270,"num_write":93,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1976,"num_omap_keys":4,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.a","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.327061+0000","last_change":"2024-03-09T15:36:42.327115+0000","last_active":"2024-03-09T15:36:42.327061+0000","last_peered":"2024-03-09T15:36:42.327061+0000","last_clean":"2024-03-09T15:36:42.327061+0000","last_became_active":"2024-03-09T15:09:50.079554+0000","last_became_peered":"2024-03-09T15:09:50.079554+0000","last_unstale":"2024-03-09T15:36:42.327061+0000","last_undegraded":"2024-03-09T15:36:42.327061+0000","last_fullsized":"2024-03-09T15:36:42.327061+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:42.326870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:42.326870+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.326870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:01:30.715008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.d","version":"261'5","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.470652+0000","last_change":"2024-03-09T15:36:25.470799+0000","last_active":"2024-03-09T15:36:25.470652+0000","last_peered":"2024-03-09T15:36:25.470652+0000","last_clean":"2024-03-09T15:36:25.470652+0000","last_became_active":"2024-03-09T14:46:26.812859+0000","last_became_peered":"2024-03-09T14:46:26.812859+0000","last_unstale":"2024-03-09T15:36:25.470652+0000","last_undegraded":"2024-03-09T15:36:25.470652+0000","last_fullsized":"2024-03-09T15:36:25.470652+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'5","last_scrub_stamp":"2024-03-09T15:36:25.470415+0000","last_deep_scrub":"261'5","last_deep_scrub_stamp":"2024-03-09T15:36:25.470415+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.470415+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:20:04.740693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":8192,"num_write":10,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.c","version":"261'11","reported_seq":79,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.263874+0000","last_change":"2024-03-09T15:36:17.264004+0000","last_active":"2024-03-09T15:36:17.263874+0000","last_peered":"2024-03-09T15:36:17.263874+0000","last_clean":"2024-03-09T15:36:17.263874+0000","last_became_active":"2024-03-09T14:46:18.766370+0000","last_became_peered":"2024-03-09T14:46:18.766370+0000","last_unstale":"2024-03-09T15:36:17.263874+0000","last_undegraded":"2024-03-09T15:36:17.263874+0000","last_fullsized":"2024-03-09T15:36:17.263874+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'11","last_scrub_stamp":"2024-03-09T15:36:17.263641+0000","last_deep_scrub":"261'11","last_deep_scrub_stamp":"2024-03-09T15:36:17.263641+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.263641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:28:25.038359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":20480,"num_write":16,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.f","version":"258'18","reported_seq":108,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.541046+0000","last_change":"2024-03-09T15:36:12.541106+0000","last_active":"2024-03-09T15:36:12.541046+0000","last_peered":"2024-03-09T15:36:12.541046+0000","last_clean":"2024-03-09T15:36:12.541046+0000","last_became_active":"2024-03-09T14:39:13.268971+0000","last_became_peered":"2024-03-09T14:39:13.268971+0000","last_unstale":"2024-03-09T15:36:12.541046+0000","last_undegraded":"2024-03-09T15:36:12.541046+0000","last_fullsized":"2024-03-09T15:36:12.541046+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'18","last_scrub_stamp":"2024-03-09T15:36:12.540848+0000","last_deep_scrub":"258'18","last_deep_scrub_stamp":"2024-03-09T15:36:12.540848+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.540848+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:51.392572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.b","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.229996+0000","last_change":"2024-03-09T15:36:37.229996+0000","last_active":"2024-03-09T15:36:37.229996+0000","last_peered":"2024-03-09T15:36:37.229996+0000","last_clean":"2024-03-09T15:36:37.229996+0000","last_became_active":"2024-03-09T15:09:50.069508+0000","last_became_peered":"2024-03-09T15:09:50.069508+0000","last_unstale":"2024-03-09T15:36:37.229996+0000","last_undegraded":"2024-03-09T15:36:37.229996+0000","last_fullsized":"2024-03-09T15:36:37.229996+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:37.229744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:37.229744+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.229744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:14:18.843785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.c","version":"261'3","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.457377+0000","last_change":"2024-03-09T15:36:29.457444+0000","last_active":"2024-03-09T15:36:29.457377+0000","last_peered":"2024-03-09T15:36:29.457377+0000","last_clean":"2024-03-09T15:36:29.457377+0000","last_became_active":"2024-03-09T14:46:26.813009+0000","last_became_peered":"2024-03-09T14:46:26.813009+0000","last_unstale":"2024-03-09T15:36:29.457377+0000","last_undegraded":"2024-03-09T15:36:29.457377+0000","last_fullsized":"2024-03-09T15:36:29.457377+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:29.457166+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:29.457166+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.457166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:35:35.350489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2048,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.d","version":"261'3","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.167902+0000","last_change":"2024-03-09T15:36:36.167961+0000","last_active":"2024-03-09T15:36:36.167902+0000","last_peered":"2024-03-09T15:36:36.167902+0000","last_clean":"2024-03-09T15:36:36.167902+0000","last_became_active":"2024-03-09T14:46:18.770732+0000","last_became_peered":"2024-03-09T14:46:18.770732+0000","last_unstale":"2024-03-09T15:36:36.167902+0000","last_undegraded":"2024-03-09T15:36:36.167902+0000","last_fullsized":"2024-03-09T15:36:36.167902+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:36.167630+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:36.167630+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.167630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:24.866080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.e","version":"261'8","reported_seq":88,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.655370+0000","last_change":"2024-03-09T15:36:18.655429+0000","last_active":"2024-03-09T15:36:18.655370+0000","last_peered":"2024-03-09T15:36:18.655370+0000","last_clean":"2024-03-09T15:36:18.655370+0000","last_became_active":"2024-03-09T14:39:13.271672+0000","last_became_peered":"2024-03-09T14:39:13.271672+0000","last_unstale":"2024-03-09T15:36:18.655370+0000","last_undegraded":"2024-03-09T15:36:18.655370+0000","last_fullsized":"2024-03-09T15:36:18.655370+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:18.655167+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:18.655167+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.655167+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:25:51.301220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.8","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.370543+0000","last_change":"2024-03-09T15:36:50.370653+0000","last_active":"2024-03-09T15:36:50.370543+0000","last_peered":"2024-03-09T15:36:50.370543+0000","last_clean":"2024-03-09T15:36:50.370543+0000","last_became_active":"2024-03-09T15:09:50.061662+0000","last_became_peered":"2024-03-09T15:09:50.061662+0000","last_unstale":"2024-03-09T15:36:50.370543+0000","last_undegraded":"2024-03-09T15:36:50.370543+0000","last_fullsized":"2024-03-09T15:36:50.370543+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:50.370325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:50.370325+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.370325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:31:38.698974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"67.f","version":"261'6","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.198675+0000","last_change":"2024-03-09T15:36:51.198827+0000","last_active":"2024-03-09T15:36:51.198675+0000","last_peered":"2024-03-09T15:36:51.198675+0000","last_clean":"2024-03-09T15:36:51.198675+0000","last_became_active":"2024-03-09T14:46:26.899430+0000","last_became_peered":"2024-03-09T14:46:26.899430+0000","last_unstale":"2024-03-09T15:36:51.198675+0000","last_undegraded":"2024-03-09T15:36:51.198675+0000","last_fullsized":"2024-03-09T15:36:51.198675+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:51.198415+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:51.198415+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.198415+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:37:54.991742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5120,"num_write":11,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.e","version":"258'3","reported_seq":65,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.271195+0000","last_change":"2024-03-09T15:36:15.271259+0000","last_active":"2024-03-09T15:36:15.271195+0000","last_peered":"2024-03-09T15:36:15.271195+0000","last_clean":"2024-03-09T15:36:15.271195+0000","last_became_active":"2024-03-09T14:46:18.768694+0000","last_became_peered":"2024-03-09T14:46:18.768694+0000","last_unstale":"2024-03-09T15:36:15.271195+0000","last_undegraded":"2024-03-09T15:36:15.271195+0000","last_fullsized":"2024-03-09T15:36:15.271195+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:15.270937+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:15.270937+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.270937+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:49:15.823574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":3,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.d","version":"261'43","reported_seq":168,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.940181+0000","last_change":"2024-03-09T15:36:15.940345+0000","last_active":"2024-03-09T15:36:15.940181+0000","last_peered":"2024-03-09T15:36:15.940181+0000","last_clean":"2024-03-09T15:36:15.940181+0000","last_became_active":"2024-03-09T14:39:13.257275+0000","last_became_peered":"2024-03-09T14:39:13.257275+0000","last_unstale":"2024-03-09T15:36:15.940181+0000","last_undegraded":"2024-03-09T15:36:15.940181+0000","last_fullsized":"2024-03-09T15:36:15.940181+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:15.939952+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:15.939952+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.939952+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:40:15.055734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":175,"num_read_kb":123,"num_write":70,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.9","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:09.188836+0000","last_change":"2024-03-09T15:37:09.188890+0000","last_active":"2024-03-09T15:37:09.188836+0000","last_peered":"2024-03-09T15:37:09.188836+0000","last_clean":"2024-03-09T15:37:09.188836+0000","last_became_active":"2024-03-09T15:09:50.076702+0000","last_became_peered":"2024-03-09T15:09:50.076702+0000","last_unstale":"2024-03-09T15:37:09.188836+0000","last_undegraded":"2024-03-09T15:37:09.188836+0000","last_fullsized":"2024-03-09T15:37:09.188836+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:09.188634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:09.188634+0000","last_clean_scrub_stamp":"2024-03-09T15:37:09.188634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:40:20.717469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.e","version":"261'6","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.497081+0000","last_change":"2024-03-09T15:36:37.497144+0000","last_active":"2024-03-09T15:36:37.497081+0000","last_peered":"2024-03-09T15:36:37.497081+0000","last_clean":"2024-03-09T15:36:37.497081+0000","last_became_active":"2024-03-09T14:46:26.814709+0000","last_became_peered":"2024-03-09T14:46:26.814709+0000","last_unstale":"2024-03-09T15:36:37.497081+0000","last_undegraded":"2024-03-09T15:36:37.497081+0000","last_fullsized":"2024-03-09T15:36:37.497081+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:37.496875+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:37.496875+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.496875+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:32:45.549662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.f","version":"261'5","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.512325+0000","last_change":"2024-03-09T15:36:13.512372+0000","last_active":"2024-03-09T15:36:13.512325+0000","last_peered":"2024-03-09T15:36:13.512325+0000","last_clean":"2024-03-09T15:36:13.512325+0000","last_became_active":"2024-03-09T14:46:18.766000+0000","last_became_peered":"2024-03-09T14:46:18.766000+0000","last_unstale":"2024-03-09T15:36:13.512325+0000","last_undegraded":"2024-03-09T15:36:13.512325+0000","last_fullsized":"2024-03-09T15:36:13.512325+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'5","last_scrub_stamp":"2024-03-09T15:36:13.512169+0000","last_deep_scrub":"261'5","last_deep_scrub_stamp":"2024-03-09T15:36:13.512169+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.512169+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:07:18.735860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4096,"num_write":10,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.c","version":"0'0","reported_seq":68,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.206724+0000","last_change":"2024-03-09T15:36:26.206778+0000","last_active":"2024-03-09T15:36:26.206724+0000","last_peered":"2024-03-09T15:36:26.206724+0000","last_clean":"2024-03-09T15:36:26.206724+0000","last_became_active":"2024-03-09T14:39:13.270527+0000","last_became_peered":"2024-03-09T14:39:13.270527+0000","last_unstale":"2024-03-09T15:36:26.206724+0000","last_undegraded":"2024-03-09T15:36:26.206724+0000","last_fullsized":"2024-03-09T15:36:26.206724+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:26.206522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:26.206522+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.206522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:18:22.897656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"68.e","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:12.694292+0000","last_change":"2024-03-09T15:37:12.694346+0000","last_active":"2024-03-09T15:37:12.694292+0000","last_peered":"2024-03-09T15:37:12.694292+0000","last_clean":"2024-03-09T15:37:12.694292+0000","last_became_active":"2024-03-09T15:09:50.085548+0000","last_became_peered":"2024-03-09T15:09:50.085548+0000","last_unstale":"2024-03-09T15:37:12.694292+0000","last_undegraded":"2024-03-09T15:37:12.694292+0000","last_fullsized":"2024-03-09T15:37:12.694292+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:12.694011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:12.694011+0000","last_clean_scrub_stamp":"2024-03-09T15:37:12.694011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:14:32.781865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"67.9","version":"0'0","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:57.571078+0000","last_change":"2024-03-09T15:36:57.571134+0000","last_active":"2024-03-09T15:36:57.571078+0000","last_peered":"2024-03-09T15:36:57.571078+0000","last_clean":"2024-03-09T15:36:57.571078+0000","last_became_active":"2024-03-09T14:46:26.801235+0000","last_became_peered":"2024-03-09T14:46:26.801235+0000","last_unstale":"2024-03-09T15:36:57.571078+0000","last_undegraded":"2024-03-09T15:36:57.571078+0000","last_fullsized":"2024-03-09T15:36:57.571078+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:57.570883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:57.570883+0000","last_clean_scrub_stamp":"2024-03-09T15:36:57.570883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:56:17.614656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.8","version":"0'0","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.316818+0000","last_change":"2024-03-09T15:36:23.316871+0000","last_active":"2024-03-09T15:36:23.316818+0000","last_peered":"2024-03-09T15:36:23.316818+0000","last_clean":"2024-03-09T15:36:23.316818+0000","last_became_active":"2024-03-09T14:46:18.774547+0000","last_became_peered":"2024-03-09T14:46:18.774547+0000","last_unstale":"2024-03-09T15:36:23.316818+0000","last_undegraded":"2024-03-09T15:36:23.316818+0000","last_fullsized":"2024-03-09T15:36:23.316818+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:23.316611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:23.316611+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.316611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:09:52.305120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.b","version":"258'10","reported_seq":94,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.250282+0000","last_change":"2024-03-09T15:36:20.250347+0000","last_active":"2024-03-09T15:36:20.250282+0000","last_peered":"2024-03-09T15:36:20.250282+0000","last_clean":"2024-03-09T15:36:20.250282+0000","last_became_active":"2024-03-09T14:39:13.257238+0000","last_became_peered":"2024-03-09T14:39:13.257238+0000","last_unstale":"2024-03-09T15:36:20.250282+0000","last_undegraded":"2024-03-09T15:36:20.250282+0000","last_fullsized":"2024-03-09T15:36:20.250282+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:20.250091+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:20.250091+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.250091+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:55:12.169918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.f","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:05.135984+0000","last_change":"2024-03-09T15:37:05.136040+0000","last_active":"2024-03-09T15:37:05.135984+0000","last_peered":"2024-03-09T15:37:05.135984+0000","last_clean":"2024-03-09T15:37:05.135984+0000","last_became_active":"2024-03-09T15:09:50.070526+0000","last_became_peered":"2024-03-09T15:09:50.070526+0000","last_unstale":"2024-03-09T15:37:05.135984+0000","last_undegraded":"2024-03-09T15:37:05.135984+0000","last_fullsized":"2024-03-09T15:37:05.135984+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:05.135769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:05.135769+0000","last_clean_scrub_stamp":"2024-03-09T15:37:05.135769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:09:12.888354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.8","version":"0'0","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.159602+0000","last_change":"2024-03-09T15:36:25.159656+0000","last_active":"2024-03-09T15:36:25.159602+0000","last_peered":"2024-03-09T15:36:25.159602+0000","last_clean":"2024-03-09T15:36:25.159602+0000","last_became_active":"2024-03-09T14:46:26.893908+0000","last_became_peered":"2024-03-09T14:46:26.893908+0000","last_unstale":"2024-03-09T15:36:25.159602+0000","last_undegraded":"2024-03-09T15:36:25.159602+0000","last_fullsized":"2024-03-09T15:36:25.159602+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.159401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.159401+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.159401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:51:03.092803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.9","version":"261'3","reported_seq":63,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.495189+0000","last_change":"2024-03-09T15:36:11.495211+0000","last_active":"2024-03-09T15:36:11.495189+0000","last_peered":"2024-03-09T15:36:11.495189+0000","last_clean":"2024-03-09T15:36:11.495189+0000","last_became_active":"2024-03-09T14:46:18.761367+0000","last_became_peered":"2024-03-09T14:46:18.761367+0000","last_unstale":"2024-03-09T15:36:11.495189+0000","last_undegraded":"2024-03-09T15:36:11.495189+0000","last_fullsized":"2024-03-09T15:36:11.495189+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:11.495111+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:11.495111+0000","last_clean_scrub_stamp":"2024-03-09T15:36:11.495111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:55:49.995490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.a","version":"258'23","reported_seq":132,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.129434+0000","last_change":"2024-03-09T15:36:26.129498+0000","last_active":"2024-03-09T15:36:26.129434+0000","last_peered":"2024-03-09T15:36:26.129434+0000","last_clean":"2024-03-09T15:36:26.129434+0000","last_became_active":"2024-03-09T14:39:13.252547+0000","last_became_peered":"2024-03-09T14:39:13.252547+0000","last_unstale":"2024-03-09T15:36:26.129434+0000","last_undegraded":"2024-03-09T15:36:26.129434+0000","last_fullsized":"2024-03-09T15:36:26.129434+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'23","last_scrub_stamp":"2024-03-09T15:36:26.129236+0000","last_deep_scrub":"258'23","last_deep_scrub_stamp":"2024-03-09T15:36:26.129236+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.129236+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:37:12.973344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":86,"num_write":38,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.c","version":"261'2","reported_seq":35,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:04.138010+0000","last_change":"2024-03-09T15:37:04.138094+0000","last_active":"2024-03-09T15:37:04.138010+0000","last_peered":"2024-03-09T15:37:04.138010+0000","last_clean":"2024-03-09T15:37:04.138010+0000","last_became_active":"2024-03-09T15:09:50.069324+0000","last_became_peered":"2024-03-09T15:09:50.069324+0000","last_unstale":"2024-03-09T15:37:04.138010+0000","last_undegraded":"2024-03-09T15:37:04.138010+0000","last_fullsized":"2024-03-09T15:37:04.138010+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:37:04.137795+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:37:04.137795+0000","last_clean_scrub_stamp":"2024-03-09T15:37:04.137795+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:45:38.998974+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.b","version":"258'3","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.176799+0000","last_change":"2024-03-09T15:36:49.176973+0000","last_active":"2024-03-09T15:36:49.176799+0000","last_peered":"2024-03-09T15:36:49.176799+0000","last_clean":"2024-03-09T15:36:49.176799+0000","last_became_active":"2024-03-09T14:46:26.804615+0000","last_became_peered":"2024-03-09T14:46:26.804615+0000","last_unstale":"2024-03-09T15:36:49.176799+0000","last_undegraded":"2024-03-09T15:36:49.176799+0000","last_fullsized":"2024-03-09T15:36:49.176799+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:49.176568+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:49.176568+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.176568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:20:16.951902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5120,"num_write":8,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.a","version":"258'3","reported_seq":66,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:11.202430+0000","last_change":"2024-03-09T15:36:11.202430+0000","last_active":"2024-03-09T15:36:11.202430+0000","last_peered":"2024-03-09T15:36:11.202430+0000","last_clean":"2024-03-09T15:36:11.202430+0000","last_became_active":"2024-03-09T14:46:18.768431+0000","last_became_peered":"2024-03-09T14:46:18.768431+0000","last_unstale":"2024-03-09T15:36:11.202430+0000","last_undegraded":"2024-03-09T15:36:11.202430+0000","last_fullsized":"2024-03-09T15:36:11.202430+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:11.202170+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:11.202170+0000","last_clean_scrub_stamp":"2024-03-09T15:36:11.202170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:12:25.295523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"65.9","version":"261'43","reported_seq":166,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.428686+0000","last_change":"2024-03-09T15:36:18.428750+0000","last_active":"2024-03-09T15:36:18.428686+0000","last_peered":"2024-03-09T15:36:18.428686+0000","last_clean":"2024-03-09T15:36:18.428686+0000","last_became_active":"2024-03-09T14:39:13.259975+0000","last_became_peered":"2024-03-09T14:39:13.259975+0000","last_unstale":"2024-03-09T15:36:18.428686+0000","last_undegraded":"2024-03-09T15:36:18.428686+0000","last_fullsized":"2024-03-09T15:36:18.428686+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:18.428488+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:18.428488+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.428488+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:15:47.696457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":169,"num_read_kb":119,"num_write":74,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"68.d","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:12.057933+0000","last_change":"2024-03-09T15:37:12.058058+0000","last_active":"2024-03-09T15:37:12.057933+0000","last_peered":"2024-03-09T15:37:12.057933+0000","last_clean":"2024-03-09T15:37:12.057933+0000","last_became_active":"2024-03-09T15:09:50.084074+0000","last_became_peered":"2024-03-09T15:09:50.084074+0000","last_unstale":"2024-03-09T15:37:12.057933+0000","last_undegraded":"2024-03-09T15:37:12.057933+0000","last_fullsized":"2024-03-09T15:37:12.057933+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:12.057703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:12.057703+0000","last_clean_scrub_stamp":"2024-03-09T15:37:12.057703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:11:02.269165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.a","version":"258'3","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.196067+0000","last_change":"2024-03-09T15:36:21.196067+0000","last_active":"2024-03-09T15:36:21.196067+0000","last_peered":"2024-03-09T15:36:21.196067+0000","last_clean":"2024-03-09T15:36:21.196067+0000","last_became_active":"2024-03-09T14:46:26.825949+0000","last_became_peered":"2024-03-09T14:46:26.825949+0000","last_unstale":"2024-03-09T15:36:21.196067+0000","last_undegraded":"2024-03-09T15:36:21.196067+0000","last_fullsized":"2024-03-09T15:36:21.196067+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:21.195795+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:21.195795+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.195795+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:39:51.756963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.b","version":"261'3","reported_seq":64,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.180486+0000","last_change":"2024-03-09T15:36:29.180546+0000","last_active":"2024-03-09T15:36:29.180486+0000","last_peered":"2024-03-09T15:36:29.180486+0000","last_clean":"2024-03-09T15:36:29.180486+0000","last_became_active":"2024-03-09T14:46:18.784869+0000","last_became_peered":"2024-03-09T14:46:18.784869+0000","last_unstale":"2024-03-09T15:36:29.180486+0000","last_undegraded":"2024-03-09T15:36:29.180486+0000","last_fullsized":"2024-03-09T15:36:29.180486+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'3","last_scrub_stamp":"2024-03-09T15:36:29.180292+0000","last_deep_scrub":"261'3","last_deep_scrub_stamp":"2024-03-09T15:36:29.180292+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.180292+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:09:41.856539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":6144,"num_write":3,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"65.8","version":"258'10","reported_seq":95,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.980744+0000","last_change":"2024-03-09T15:36:16.980807+0000","last_active":"2024-03-09T15:36:16.980744+0000","last_peered":"2024-03-09T15:36:16.980744+0000","last_clean":"2024-03-09T15:36:16.980744+0000","last_became_active":"2024-03-09T14:39:13.264006+0000","last_became_peered":"2024-03-09T14:39:13.264006+0000","last_unstale":"2024-03-09T15:36:16.980744+0000","last_undegraded":"2024-03-09T15:36:16.980744+0000","last_fullsized":"2024-03-09T15:36:16.980744+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:16.980539+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:16.980539+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.980539+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:50:44.529795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.2","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.236049+0000","last_change":"2024-03-09T15:36:33.236049+0000","last_active":"2024-03-09T15:36:33.236049+0000","last_peered":"2024-03-09T15:36:33.236049+0000","last_clean":"2024-03-09T15:36:33.236049+0000","last_became_active":"2024-03-09T15:09:50.065487+0000","last_became_peered":"2024-03-09T15:09:50.065487+0000","last_unstale":"2024-03-09T15:36:33.236049+0000","last_undegraded":"2024-03-09T15:36:33.236049+0000","last_fullsized":"2024-03-09T15:36:33.236049+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:33.235751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:33.235751+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.235751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:45.773762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.5","version":"258'2","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.230666+0000","last_change":"2024-03-09T15:36:35.230666+0000","last_active":"2024-03-09T15:36:35.230666+0000","last_peered":"2024-03-09T15:36:35.230666+0000","last_clean":"2024-03-09T15:36:35.230666+0000","last_became_active":"2024-03-09T14:46:26.812137+0000","last_became_peered":"2024-03-09T14:46:26.812137+0000","last_unstale":"2024-03-09T15:36:35.230666+0000","last_undegraded":"2024-03-09T15:36:35.230666+0000","last_fullsized":"2024-03-09T15:36:35.230666+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'2","last_scrub_stamp":"2024-03-09T15:36:35.230397+0000","last_deep_scrub":"258'2","last_deep_scrub_stamp":"2024-03-09T15:36:35.230397+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.230397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:52:41.341440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4096,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.4","version":"0'0","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.218589+0000","last_change":"2024-03-09T15:36:19.218589+0000","last_active":"2024-03-09T15:36:19.218589+0000","last_peered":"2024-03-09T15:36:19.218589+0000","last_clean":"2024-03-09T15:36:19.218589+0000","last_became_active":"2024-03-09T14:46:18.751608+0000","last_became_peered":"2024-03-09T14:46:18.751608+0000","last_unstale":"2024-03-09T15:36:19.218589+0000","last_undegraded":"2024-03-09T15:36:19.218589+0000","last_fullsized":"2024-03-09T15:36:19.218589+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:19.218339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:19.218339+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.218339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:28:04.515161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.7","version":"261'8","reported_seq":89,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.227007+0000","last_change":"2024-03-09T15:36:14.227039+0000","last_active":"2024-03-09T15:36:14.227007+0000","last_peered":"2024-03-09T15:36:14.227007+0000","last_clean":"2024-03-09T15:36:14.227007+0000","last_became_active":"2024-03-09T14:39:13.273363+0000","last_became_peered":"2024-03-09T14:39:13.273363+0000","last_unstale":"2024-03-09T15:36:14.227007+0000","last_undegraded":"2024-03-09T15:36:14.227007+0000","last_fullsized":"2024-03-09T15:36:14.227007+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'8","last_scrub_stamp":"2024-03-09T15:36:14.226893+0000","last_deep_scrub":"261'8","last_deep_scrub_stamp":"2024-03-09T15:36:14.226893+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.226893+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:21:38.723375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"68.3","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:57.884969+0000","last_change":"2024-03-09T15:36:57.885023+0000","last_active":"2024-03-09T15:36:57.884969+0000","last_peered":"2024-03-09T15:36:57.884969+0000","last_clean":"2024-03-09T15:36:57.884969+0000","last_became_active":"2024-03-09T15:09:50.072488+0000","last_became_peered":"2024-03-09T15:09:50.072488+0000","last_unstale":"2024-03-09T15:36:57.884969+0000","last_undegraded":"2024-03-09T15:36:57.884969+0000","last_fullsized":"2024-03-09T15:36:57.884969+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:57.884771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:57.884771+0000","last_clean_scrub_stamp":"2024-03-09T15:36:57.884771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:47:08.805239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"67.4","version":"258'2","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.256968+0000","last_change":"2024-03-09T15:36:39.257131+0000","last_active":"2024-03-09T15:36:39.256968+0000","last_peered":"2024-03-09T15:36:39.256968+0000","last_clean":"2024-03-09T15:36:39.256968+0000","last_became_active":"2024-03-09T14:46:26.894506+0000","last_became_peered":"2024-03-09T14:46:26.894506+0000","last_unstale":"2024-03-09T15:36:39.256968+0000","last_undegraded":"2024-03-09T15:36:39.256968+0000","last_fullsized":"2024-03-09T15:36:39.256968+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'2","last_scrub_stamp":"2024-03-09T15:36:39.256730+0000","last_deep_scrub":"258'2","last_deep_scrub_stamp":"2024-03-09T15:36:39.256730+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.256730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:48:26.210989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":23,"num_write":2,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.5","version":"261'6","reported_seq":77,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.217768+0000","last_change":"2024-03-09T15:36:37.217842+0000","last_active":"2024-03-09T15:36:37.217768+0000","last_peered":"2024-03-09T15:36:37.217768+0000","last_clean":"2024-03-09T15:36:37.217768+0000","last_became_active":"2024-03-09T14:46:18.770799+0000","last_became_peered":"2024-03-09T14:46:18.770799+0000","last_unstale":"2024-03-09T15:36:37.217768+0000","last_undegraded":"2024-03-09T15:36:37.217768+0000","last_fullsized":"2024-03-09T15:36:37.217768+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:37.217579+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:37.217579+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.217579+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:32:01.418959+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":20482,"num_write":9,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"65.6","version":"261'22","reported_seq":121,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.972029+0000","last_change":"2024-03-09T15:36:14.972165+0000","last_active":"2024-03-09T15:36:14.972029+0000","last_peered":"2024-03-09T15:36:14.972029+0000","last_clean":"2024-03-09T15:36:14.972029+0000","last_became_active":"2024-03-09T14:39:13.266246+0000","last_became_peered":"2024-03-09T14:39:13.266246+0000","last_unstale":"2024-03-09T15:36:14.972029+0000","last_undegraded":"2024-03-09T15:36:14.972029+0000","last_fullsized":"2024-03-09T15:36:14.972029+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'22","last_scrub_stamp":"2024-03-09T15:36:14.971789+0000","last_deep_scrub":"261'22","last_deep_scrub_stamp":"2024-03-09T15:36:14.971789+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.971789+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:50:18.362986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":65,"num_write":41,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.0","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:53.136614+0000","last_change":"2024-03-09T15:36:53.136668+0000","last_active":"2024-03-09T15:36:53.136614+0000","last_peered":"2024-03-09T15:36:53.136614+0000","last_clean":"2024-03-09T15:36:53.136614+0000","last_became_active":"2024-03-09T15:09:50.063350+0000","last_became_peered":"2024-03-09T15:09:50.063350+0000","last_unstale":"2024-03-09T15:36:53.136614+0000","last_undegraded":"2024-03-09T15:36:53.136614+0000","last_fullsized":"2024-03-09T15:36:53.136614+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:53.136413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:53.136413+0000","last_clean_scrub_stamp":"2024-03-09T15:36:53.136413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:11:03.539751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"67.7","version":"0'0","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.151944+0000","last_change":"2024-03-09T15:36:29.151997+0000","last_active":"2024-03-09T15:36:29.151944+0000","last_peered":"2024-03-09T15:36:29.151944+0000","last_clean":"2024-03-09T15:36:29.151944+0000","last_became_active":"2024-03-09T14:46:26.896503+0000","last_became_peered":"2024-03-09T14:46:26.896503+0000","last_unstale":"2024-03-09T15:36:29.151944+0000","last_undegraded":"2024-03-09T15:36:29.151944+0000","last_fullsized":"2024-03-09T15:36:29.151944+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:29.151756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:29.151756+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.151756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:12:40.874461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.6","version":"261'2","reported_seq":63,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.752529+0000","last_change":"2024-03-09T15:36:26.752595+0000","last_active":"2024-03-09T15:36:26.752529+0000","last_peered":"2024-03-09T15:36:26.752529+0000","last_clean":"2024-03-09T15:36:26.752529+0000","last_became_active":"2024-03-09T14:46:18.778572+0000","last_became_peered":"2024-03-09T14:46:18.778572+0000","last_unstale":"2024-03-09T15:36:26.752529+0000","last_undegraded":"2024-03-09T15:36:26.752529+0000","last_fullsized":"2024-03-09T15:36:26.752529+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:26.752326+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:26.752326+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.752326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:06:26.783165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3072,"num_write":2,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"65.5","version":"258'18","reported_seq":116,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.164653+0000","last_change":"2024-03-09T15:36:23.164716+0000","last_active":"2024-03-09T15:36:23.164653+0000","last_peered":"2024-03-09T15:36:23.164653+0000","last_clean":"2024-03-09T15:36:23.164653+0000","last_became_active":"2024-03-09T14:39:13.263833+0000","last_became_peered":"2024-03-09T14:39:13.263833+0000","last_unstale":"2024-03-09T15:36:23.164653+0000","last_undegraded":"2024-03-09T15:36:23.164653+0000","last_fullsized":"2024-03-09T15:36:23.164653+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'18","last_scrub_stamp":"2024-03-09T15:36:23.164449+0000","last_deep_scrub":"258'18","last_deep_scrub_stamp":"2024-03-09T15:36:23.164449+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.164449+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:22:04.204143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":58,"num_write":29,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"68.1","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:02.565044+0000","last_change":"2024-03-09T15:37:02.565097+0000","last_active":"2024-03-09T15:37:02.565044+0000","last_peered":"2024-03-09T15:37:02.565044+0000","last_clean":"2024-03-09T15:37:02.565044+0000","last_became_active":"2024-03-09T15:09:50.067947+0000","last_became_peered":"2024-03-09T15:09:50.067947+0000","last_unstale":"2024-03-09T15:37:02.565044+0000","last_undegraded":"2024-03-09T15:37:02.565044+0000","last_fullsized":"2024-03-09T15:37:02.565044+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:02.564849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:02.564849+0000","last_clean_scrub_stamp":"2024-03-09T15:37:02.564849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:16.040745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"67.6","version":"261'4","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:00.640406+0000","last_change":"2024-03-09T15:37:00.640539+0000","last_active":"2024-03-09T15:37:00.640406+0000","last_peered":"2024-03-09T15:37:00.640406+0000","last_clean":"2024-03-09T15:37:00.640406+0000","last_became_active":"2024-03-09T14:46:26.808384+0000","last_became_peered":"2024-03-09T14:46:26.808384+0000","last_unstale":"2024-03-09T15:37:00.640406+0000","last_undegraded":"2024-03-09T15:37:00.640406+0000","last_fullsized":"2024-03-09T15:37:00.640406+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'4","last_scrub_stamp":"2024-03-09T15:37:00.640182+0000","last_deep_scrub":"261'4","last_deep_scrub_stamp":"2024-03-09T15:37:00.640182+0000","last_clean_scrub_stamp":"2024-03-09T15:37:00.640182+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:28:34.063992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":180,"num_write":4,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.7","version":"261'9","reported_seq":83,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.291656+0000","last_change":"2024-03-09T15:36:14.291833+0000","last_active":"2024-03-09T15:36:14.291656+0000","last_peered":"2024-03-09T15:36:14.291656+0000","last_clean":"2024-03-09T15:36:14.291656+0000","last_became_active":"2024-03-09T14:46:18.762472+0000","last_became_peered":"2024-03-09T14:46:18.762472+0000","last_unstale":"2024-03-09T15:36:14.291656+0000","last_undegraded":"2024-03-09T15:36:14.291656+0000","last_fullsized":"2024-03-09T15:36:14.291656+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-03-09T15:36:14.291396+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-03-09T15:36:14.291396+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.291396+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:45:34.816033+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":30722,"num_write":17,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.4","version":"258'10","reported_seq":96,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.555803+0000","last_change":"2024-03-09T15:36:14.555905+0000","last_active":"2024-03-09T15:36:14.555803+0000","last_peered":"2024-03-09T15:36:14.555803+0000","last_clean":"2024-03-09T15:36:14.555803+0000","last_became_active":"2024-03-09T14:39:13.270238+0000","last_became_peered":"2024-03-09T14:39:13.270238+0000","last_unstale":"2024-03-09T15:36:14.555803+0000","last_undegraded":"2024-03-09T15:36:14.555803+0000","last_fullsized":"2024-03-09T15:36:14.555803+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-09T15:36:14.555648+0000","last_deep_scrub":"258'10","last_deep_scrub_stamp":"2024-03-09T15:36:14.555648+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.555648+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:38:18.000083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"68.6","version":"261'2","reported_seq":35,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.245425+0000","last_change":"2024-03-09T15:36:40.245494+0000","last_active":"2024-03-09T15:36:40.245425+0000","last_peered":"2024-03-09T15:36:40.245425+0000","last_clean":"2024-03-09T15:36:40.245425+0000","last_became_active":"2024-03-09T15:09:50.072180+0000","last_became_peered":"2024-03-09T15:09:50.072180+0000","last_unstale":"2024-03-09T15:36:40.245425+0000","last_undegraded":"2024-03-09T15:36:40.245425+0000","last_fullsized":"2024-03-09T15:36:40.245425+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:40.245236+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:40.245236+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.245236+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:47:53.099498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":188,"num_omap_keys":2,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"67.1","version":"0'0","reported_seq":55,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.542310+0000","last_change":"2024-03-09T15:36:21.542310+0000","last_active":"2024-03-09T15:36:21.542310+0000","last_peered":"2024-03-09T15:36:21.542310+0000","last_clean":"2024-03-09T15:36:21.542310+0000","last_became_active":"2024-03-09T14:46:26.799019+0000","last_became_peered":"2024-03-09T14:46:26.799019+0000","last_unstale":"2024-03-09T15:36:21.542310+0000","last_undegraded":"2024-03-09T15:36:21.542310+0000","last_fullsized":"2024-03-09T15:36:21.542310+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.542044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.542044+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.542044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:24:20.252130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.0","version":"0'0","reported_seq":62,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.229238+0000","last_change":"2024-03-09T15:36:18.229238+0000","last_active":"2024-03-09T15:36:18.229238+0000","last_peered":"2024-03-09T15:36:18.229238+0000","last_clean":"2024-03-09T15:36:18.229238+0000","last_became_active":"2024-03-09T14:46:18.752290+0000","last_became_peered":"2024-03-09T14:46:18.752290+0000","last_unstale":"2024-03-09T15:36:18.229238+0000","last_undegraded":"2024-03-09T15:36:18.229238+0000","last_fullsized":"2024-03-09T15:36:18.229238+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:18.228896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:18.228896+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.228896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:58:17.281185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.3","version":"261'61","reported_seq":224,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.587549+0000","last_change":"2024-03-09T15:36:14.587616+0000","last_active":"2024-03-09T15:36:14.587549+0000","last_peered":"2024-03-09T15:36:14.587549+0000","last_clean":"2024-03-09T15:36:14.587549+0000","last_became_active":"2024-03-09T14:39:13.259125+0000","last_became_peered":"2024-03-09T14:39:13.259125+0000","last_unstale":"2024-03-09T15:36:14.587549+0000","last_undegraded":"2024-03-09T15:36:14.587549+0000","last_fullsized":"2024-03-09T15:36:14.587549+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'61","last_scrub_stamp":"2024-03-09T15:36:14.587351+0000","last_deep_scrub":"261'61","last_deep_scrub_stamp":"2024-03-09T15:36:14.587351+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.587351+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:00:04.188736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":281,"num_read_kb":205,"num_write":98,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"68.7","version":"0'0","reported_seq":33,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:08.000781+0000","last_change":"2024-03-09T15:37:08.000838+0000","last_active":"2024-03-09T15:37:08.000781+0000","last_peered":"2024-03-09T15:37:08.000781+0000","last_clean":"2024-03-09T15:37:08.000781+0000","last_became_active":"2024-03-09T15:09:50.062429+0000","last_became_peered":"2024-03-09T15:09:50.062429+0000","last_unstale":"2024-03-09T15:37:08.000781+0000","last_undegraded":"2024-03-09T15:37:08.000781+0000","last_fullsized":"2024-03-09T15:37:08.000781+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:08.000581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:08.000581+0000","last_clean_scrub_stamp":"2024-03-09T15:37:08.000581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:04:19.242539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.0","version":"261'6","reported_seq":68,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:54.855985+0000","last_change":"2024-03-09T15:36:54.856151+0000","last_active":"2024-03-09T15:36:54.855985+0000","last_peered":"2024-03-09T15:36:54.855985+0000","last_clean":"2024-03-09T15:36:54.855985+0000","last_became_active":"2024-03-09T14:46:26.815311+0000","last_became_peered":"2024-03-09T14:46:26.815311+0000","last_unstale":"2024-03-09T15:36:54.855985+0000","last_undegraded":"2024-03-09T15:36:54.855985+0000","last_fullsized":"2024-03-09T15:36:54.855985+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:54.855723+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:54.855723+0000","last_clean_scrub_stamp":"2024-03-09T15:36:54.855723+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:25:06.455116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":16384,"num_write":11,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1","version":"261'6","reported_seq":73,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.249215+0000","last_change":"2024-03-09T15:36:15.249215+0000","last_active":"2024-03-09T15:36:15.249215+0000","last_peered":"2024-03-09T15:36:15.249215+0000","last_clean":"2024-03-09T15:36:15.249215+0000","last_became_active":"2024-03-09T14:46:18.757694+0000","last_became_peered":"2024-03-09T14:46:18.757694+0000","last_unstale":"2024-03-09T15:36:15.249215+0000","last_undegraded":"2024-03-09T15:36:15.249215+0000","last_fullsized":"2024-03-09T15:36:15.249215+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6","last_scrub_stamp":"2024-03-09T15:36:15.248968+0000","last_deep_scrub":"261'6","last_deep_scrub_stamp":"2024-03-09T15:36:15.248968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.248968+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:50:59.398908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":4098,"num_write":9,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"65.2","version":"258'20","reported_seq":119,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.118799+0000","last_change":"2024-03-09T15:36:13.118868+0000","last_active":"2024-03-09T15:36:13.118799+0000","last_peered":"2024-03-09T15:36:13.118799+0000","last_clean":"2024-03-09T15:36:13.118799+0000","last_became_active":"2024-03-09T14:39:13.264203+0000","last_became_peered":"2024-03-09T14:39:13.264203+0000","last_unstale":"2024-03-09T15:36:13.118799+0000","last_undegraded":"2024-03-09T15:36:13.118799+0000","last_fullsized":"2024-03-09T15:36:13.118799+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'20","last_scrub_stamp":"2024-03-09T15:36:13.118589+0000","last_deep_scrub":"258'20","last_deep_scrub_stamp":"2024-03-09T15:36:13.118589+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.118589+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:29:35.065434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":65,"num_write":31,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"68.4","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:09.986900+0000","last_change":"2024-03-09T15:37:09.986955+0000","last_active":"2024-03-09T15:37:09.986900+0000","last_peered":"2024-03-09T15:37:09.986900+0000","last_clean":"2024-03-09T15:37:09.986900+0000","last_became_active":"2024-03-09T15:09:50.066222+0000","last_became_peered":"2024-03-09T15:09:50.066222+0000","last_unstale":"2024-03-09T15:37:09.986900+0000","last_undegraded":"2024-03-09T15:37:09.986900+0000","last_fullsized":"2024-03-09T15:37:09.986900+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:09.986702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:09.986702+0000","last_clean_scrub_stamp":"2024-03-09T15:37:09.986702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:36:04.063534+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"67.3","version":"0'0","reported_seq":55,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.342569+0000","last_change":"2024-03-09T15:36:49.342621+0000","last_active":"2024-03-09T15:36:49.342569+0000","last_peered":"2024-03-09T15:36:49.342569+0000","last_clean":"2024-03-09T15:36:49.342569+0000","last_became_active":"2024-03-09T14:46:26.826105+0000","last_became_peered":"2024-03-09T14:46:26.826105+0000","last_unstale":"2024-03-09T15:36:49.342569+0000","last_undegraded":"2024-03-09T15:36:49.342569+0000","last_fullsized":"2024-03-09T15:36:49.342569+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:49.342369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:49.342369+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.342369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:03:37.308549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.2","version":"0'0","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.469191+0000","last_change":"2024-03-09T15:36:12.469191+0000","last_active":"2024-03-09T15:36:12.469191+0000","last_peered":"2024-03-09T15:36:12.469191+0000","last_clean":"2024-03-09T15:36:12.469191+0000","last_became_active":"2024-03-09T14:46:18.749091+0000","last_became_peered":"2024-03-09T14:46:18.749091+0000","last_unstale":"2024-03-09T15:36:12.469191+0000","last_undegraded":"2024-03-09T15:36:12.469191+0000","last_fullsized":"2024-03-09T15:36:12.469191+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:12.468992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:12.468992+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.468992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:06:25.628906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"65.1","version":"261'25","reported_seq":134,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.079889+0000","last_change":"2024-03-09T15:36:19.079952+0000","last_active":"2024-03-09T15:36:19.079889+0000","last_peered":"2024-03-09T15:36:19.079889+0000","last_clean":"2024-03-09T15:36:19.079889+0000","last_became_active":"2024-03-09T14:39:13.255589+0000","last_became_peered":"2024-03-09T14:39:13.255589+0000","last_unstale":"2024-03-09T15:36:19.079889+0000","last_undegraded":"2024-03-09T15:36:19.079889+0000","last_fullsized":"2024-03-09T15:36:19.079889+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'25","last_scrub_stamp":"2024-03-09T15:36:19.079692+0000","last_deep_scrub":"261'25","last_deep_scrub_stamp":"2024-03-09T15:36:19.079692+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.079692+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:25:37.071627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":79,"num_write":35,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"68.5","version":"0'0","reported_seq":34,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:10.238864+0000","last_change":"2024-03-09T15:37:10.239023+0000","last_active":"2024-03-09T15:37:10.238864+0000","last_peered":"2024-03-09T15:37:10.238864+0000","last_clean":"2024-03-09T15:37:10.238864+0000","last_became_active":"2024-03-09T15:09:50.083064+0000","last_became_peered":"2024-03-09T15:09:50.083064+0000","last_unstale":"2024-03-09T15:37:10.238864+0000","last_undegraded":"2024-03-09T15:37:10.238864+0000","last_fullsized":"2024-03-09T15:37:10.238864+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:37:10.238633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:37:10.238633+0000","last_clean_scrub_stamp":"2024-03-09T15:37:10.238633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:36:45.349039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"67.2","version":"258'3","reported_seq":60,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:59.604506+0000","last_change":"2024-03-09T15:36:59.604570+0000","last_active":"2024-03-09T15:36:59.604506+0000","last_peered":"2024-03-09T15:36:59.604506+0000","last_clean":"2024-03-09T15:36:59.604506+0000","last_became_active":"2024-03-09T14:46:26.807746+0000","last_became_peered":"2024-03-09T14:46:26.807746+0000","last_unstale":"2024-03-09T15:36:59.604506+0000","last_undegraded":"2024-03-09T15:36:59.604506+0000","last_fullsized":"2024-03-09T15:36:59.604506+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'3","last_scrub_stamp":"2024-03-09T15:36:59.604307+0000","last_deep_scrub":"258'3","last_deep_scrub_stamp":"2024-03-09T15:36:59.604307+0000","last_clean_scrub_stamp":"2024-03-09T15:36:59.604307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:45:47.163987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1024,"num_write":8,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.3","version":"0'0","reported_seq":61,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.030314+0000","last_change":"2024-03-09T15:36:18.030314+0000","last_active":"2024-03-09T15:36:18.030314+0000","last_peered":"2024-03-09T15:36:18.030314+0000","last_clean":"2024-03-09T15:36:18.030314+0000","last_became_active":"2024-03-09T14:46:18.765077+0000","last_became_peered":"2024-03-09T14:46:18.765077+0000","last_unstale":"2024-03-09T15:36:18.030314+0000","last_undegraded":"2024-03-09T15:36:18.030314+0000","last_fullsized":"2024-03-09T15:36:18.030314+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:18.030067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:18.030067+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.030067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:44:34.102574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"65.0","version":"250'25","reported_seq":140,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.180899+0000","last_change":"2024-03-09T15:36:20.180899+0000","last_active":"2024-03-09T15:36:20.180899+0000","last_peered":"2024-03-09T15:36:20.180899+0000","last_clean":"2024-03-09T15:36:20.180899+0000","last_became_active":"2024-03-09T14:39:13.268553+0000","last_became_peered":"2024-03-09T14:39:13.268553+0000","last_unstale":"2024-03-09T15:36:20.180899+0000","last_undegraded":"2024-03-09T15:36:20.180899+0000","last_fullsized":"2024-03-09T15:36:20.180899+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-03-09T15:36:20.180625+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-03-09T15:36:20.180625+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.180625+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:48:22.579836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":97,"num_write":32,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"261'118","reported_seq":685,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.121725+0000","last_change":"2024-03-09T15:36:48.121725+0000","last_active":"2024-03-09T15:36:48.121725+0000","last_peered":"2024-03-09T15:36:48.121725+0000","last_clean":"2024-03-09T15:36:48.121725+0000","last_became_active":"2024-03-09T14:28:32.339785+0000","last_became_peered":"2024-03-09T14:28:32.339785+0000","last_unstale":"2024-03-09T15:36:48.121725+0000","last_undegraded":"2024-03-09T15:36:48.121725+0000","last_fullsized":"2024-03-09T15:36:48.121725+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'118","last_scrub_stamp":"2024-03-09T15:36:48.121408+0000","last_deep_scrub":"261'118","last_deep_scrub_stamp":"2024-03-09T15:36:48.121408+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.121408+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:07:57.417207+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":130,"num_read_kb":5377,"num_write":373,"num_write_kb":16955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"261'146","reported_seq":696,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.743200+0000","last_change":"2024-03-09T15:36:31.743261+0000","last_active":"2024-03-09T15:36:31.743200+0000","last_peered":"2024-03-09T15:36:31.743200+0000","last_clean":"2024-03-09T15:36:31.743200+0000","last_became_active":"2024-03-09T14:28:34.303980+0000","last_became_peered":"2024-03-09T14:28:34.303980+0000","last_unstale":"2024-03-09T15:36:31.743200+0000","last_undegraded":"2024-03-09T15:36:31.743200+0000","last_fullsized":"2024-03-09T15:36:31.743200+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'146","last_scrub_stamp":"2024-03-09T15:36:31.743004+0000","last_deep_scrub":"261'146","last_deep_scrub_stamp":"2024-03-09T15:36:31.743004+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.743004+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:05:36.454535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":546,"num_read_kb":417,"num_write":228,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"261'143","reported_seq":705,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.214056+0000","last_change":"2024-03-09T15:36:45.214056+0000","last_active":"2024-03-09T15:36:45.214056+0000","last_peered":"2024-03-09T15:36:45.214056+0000","last_clean":"2024-03-09T15:36:45.214056+0000","last_became_active":"2024-03-09T14:28:32.306840+0000","last_became_peered":"2024-03-09T14:28:32.306840+0000","last_unstale":"2024-03-09T15:36:45.214056+0000","last_undegraded":"2024-03-09T15:36:45.214056+0000","last_fullsized":"2024-03-09T15:36:45.214056+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'143","last_scrub_stamp":"2024-03-09T15:36:45.213735+0000","last_deep_scrub":"261'143","last_deep_scrub_stamp":"2024-03-09T15:36:45.213735+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.213735+0000","objects_scrubbed":8,"log_size":43,"log_dups_size":100,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:08:41.879574+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3389647,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":156,"num_read_kb":3004,"num_write":523,"num_write_kb":19720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"261'90","reported_seq":665,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.206902+0000","last_change":"2024-03-09T15:36:40.206967+0000","last_active":"2024-03-09T15:36:40.206902+0000","last_peered":"2024-03-09T15:36:40.206902+0000","last_clean":"2024-03-09T15:36:40.206902+0000","last_became_active":"2024-03-09T14:28:34.324053+0000","last_became_peered":"2024-03-09T14:28:34.324053+0000","last_unstale":"2024-03-09T15:36:40.206902+0000","last_undegraded":"2024-03-09T15:36:40.206902+0000","last_fullsized":"2024-03-09T15:36:40.206902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'90","last_scrub_stamp":"2024-03-09T15:36:40.206700+0000","last_deep_scrub":"261'90","last_deep_scrub_stamp":"2024-03-09T15:36:40.206700+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.206700+0000","objects_scrubbed":1,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:36:21.366820+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":372,"num_read_kb":300,"num_write":141,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3d","version":"261'161","reported_seq":731,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:57.183774+0000","last_change":"2024-03-09T15:36:57.183774+0000","last_active":"2024-03-09T15:36:57.183774+0000","last_peered":"2024-03-09T15:36:57.183774+0000","last_clean":"2024-03-09T15:36:57.183774+0000","last_became_active":"2024-03-09T14:28:32.286966+0000","last_became_peered":"2024-03-09T14:28:32.286966+0000","last_unstale":"2024-03-09T15:36:57.183774+0000","last_undegraded":"2024-03-09T15:36:57.183774+0000","last_fullsized":"2024-03-09T15:36:57.183774+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'161","last_scrub_stamp":"2024-03-09T15:36:57.183471+0000","last_deep_scrub":"261'161","last_deep_scrub_stamp":"2024-03-09T15:36:57.183471+0000","last_clean_scrub_stamp":"2024-03-09T15:36:57.183471+0000","objects_scrubbed":3,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:24:18.904914+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":14171,"num_write":610,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"261'157","reported_seq":727,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.378778+0000","last_change":"2024-03-09T15:36:46.378837+0000","last_active":"2024-03-09T15:36:46.378778+0000","last_peered":"2024-03-09T15:36:46.378778+0000","last_clean":"2024-03-09T15:36:46.378778+0000","last_became_active":"2024-03-09T14:28:34.306764+0000","last_became_peered":"2024-03-09T14:28:34.306764+0000","last_unstale":"2024-03-09T15:36:46.378778+0000","last_undegraded":"2024-03-09T15:36:46.378778+0000","last_fullsized":"2024-03-09T15:36:46.378778+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'157","last_scrub_stamp":"2024-03-09T15:36:46.378588+0000","last_deep_scrub":"261'157","last_deep_scrub_stamp":"2024-03-09T15:36:46.378588+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.378588+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:55:21.945660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":481,"num_write":261,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3c","version":"261'196","reported_seq":798,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.832727+0000","last_change":"2024-03-09T15:36:51.832884+0000","last_active":"2024-03-09T15:36:51.832727+0000","last_peered":"2024-03-09T15:36:51.832727+0000","last_clean":"2024-03-09T15:36:51.832727+0000","last_became_active":"2024-03-09T14:28:32.305615+0000","last_became_peered":"2024-03-09T14:28:32.305615+0000","last_unstale":"2024-03-09T15:36:51.832727+0000","last_undegraded":"2024-03-09T15:36:51.832727+0000","last_fullsized":"2024-03-09T15:36:51.832727+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'196","last_scrub_stamp":"2024-03-09T15:36:51.832524+0000","last_deep_scrub":"261'196","last_deep_scrub_stamp":"2024-03-09T15:36:51.832524+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.832524+0000","objects_scrubbed":6,"log_size":96,"log_dups_size":100,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:08:41.077484+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5326894,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":256,"num_read_kb":28937,"num_write":797,"num_write_kb":45835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"261'157","reported_seq":742,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.270196+0000","last_change":"2024-03-09T15:36:41.270258+0000","last_active":"2024-03-09T15:36:41.270196+0000","last_peered":"2024-03-09T15:36:41.270196+0000","last_clean":"2024-03-09T15:36:41.270196+0000","last_became_active":"2024-03-09T14:28:34.316500+0000","last_became_peered":"2024-03-09T14:28:34.316500+0000","last_unstale":"2024-03-09T15:36:41.270196+0000","last_undegraded":"2024-03-09T15:36:41.270196+0000","last_fullsized":"2024-03-09T15:36:41.270196+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'157","last_scrub_stamp":"2024-03-09T15:36:41.269997+0000","last_deep_scrub":"261'157","last_deep_scrub_stamp":"2024-03-09T15:36:41.269997+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.269997+0000","objects_scrubbed":1,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:09:23.890486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":622,"num_read_kb":488,"num_write":257,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3b","version":"261'156","reported_seq":827,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:59.106502+0000","last_change":"2024-03-09T15:36:59.106654+0000","last_active":"2024-03-09T15:36:59.106502+0000","last_peered":"2024-03-09T15:36:59.106502+0000","last_clean":"2024-03-09T15:36:59.106502+0000","last_became_active":"2024-03-09T14:28:32.314923+0000","last_became_peered":"2024-03-09T14:28:32.314923+0000","last_unstale":"2024-03-09T15:36:59.106502+0000","last_undegraded":"2024-03-09T15:36:59.106502+0000","last_fullsized":"2024-03-09T15:36:59.106502+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'156","last_scrub_stamp":"2024-03-09T15:36:59.106291+0000","last_deep_scrub":"261'156","last_deep_scrub_stamp":"2024-03-09T15:36:59.106291+0000","last_clean_scrub_stamp":"2024-03-09T15:36:59.106291+0000","objects_scrubbed":6,"log_size":56,"log_dups_size":100,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:56:41.396083+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1917997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":297,"num_read_kb":17282,"num_write":517,"num_write_kb":14747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3c","version":"261'124","reported_seq":716,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.314515+0000","last_change":"2024-03-09T15:36:35.314515+0000","last_active":"2024-03-09T15:36:35.314515+0000","last_peered":"2024-03-09T15:36:35.314515+0000","last_clean":"2024-03-09T15:36:35.314515+0000","last_became_active":"2024-03-09T14:28:34.314617+0000","last_became_peered":"2024-03-09T14:28:34.314617+0000","last_unstale":"2024-03-09T15:36:35.314515+0000","last_undegraded":"2024-03-09T15:36:35.314515+0000","last_fullsized":"2024-03-09T15:36:35.314515+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'124","last_scrub_stamp":"2024-03-09T15:36:35.314270+0000","last_deep_scrub":"261'124","last_deep_scrub_stamp":"2024-03-09T15:36:35.314270+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.314270+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":100,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:54:11.780427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":493,"num_read_kb":395,"num_write":185,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3a","version":"261'150","reported_seq":693,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:14.166015+0000","last_change":"2024-03-09T15:36:14.166015+0000","last_active":"2024-03-09T15:36:14.166015+0000","last_peered":"2024-03-09T15:36:14.166015+0000","last_clean":"2024-03-09T15:36:14.166015+0000","last_became_active":"2024-03-09T14:29:06.433145+0000","last_became_peered":"2024-03-09T14:29:06.433145+0000","last_unstale":"2024-03-09T15:36:14.166015+0000","last_undegraded":"2024-03-09T15:36:14.166015+0000","last_fullsized":"2024-03-09T15:36:14.166015+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'150","last_scrub_stamp":"2024-03-09T15:36:14.165814+0000","last_deep_scrub":"261'150","last_deep_scrub_stamp":"2024-03-09T15:36:14.165814+0000","last_clean_scrub_stamp":"2024-03-09T15:36:14.165814+0000","objects_scrubbed":3,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:29:26.186154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":6443,"num_write":604,"num_write_kb":33049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"261'275","reported_seq":950,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.257158+0000","last_change":"2024-03-09T15:36:38.257158+0000","last_active":"2024-03-09T15:36:38.257158+0000","last_peered":"2024-03-09T15:36:38.257158+0000","last_clean":"2024-03-09T15:36:38.257158+0000","last_became_active":"2024-03-09T14:28:34.310343+0000","last_became_peered":"2024-03-09T14:28:34.310343+0000","last_unstale":"2024-03-09T15:36:38.257158+0000","last_undegraded":"2024-03-09T15:36:38.257158+0000","last_fullsized":"2024-03-09T15:36:38.257158+0000","mapping_epoch":22,"log_start":"247'200","ondisk_log_start":"247'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'275","last_scrub_stamp":"2024-03-09T15:36:38.256877+0000","last_deep_scrub":"261'275","last_deep_scrub_stamp":"2024-03-09T15:36:38.256877+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.256877+0000","objects_scrubbed":1,"log_size":75,"log_dups_size":200,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:40:32.582105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1205,"num_read_kb":1028,"num_write":418,"num_write_kb":253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25677,"num_omap_keys":101,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"261'137","reported_seq":702,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:58.112254+0000","last_change":"2024-03-09T15:36:58.112328+0000","last_active":"2024-03-09T15:36:58.112254+0000","last_peered":"2024-03-09T15:36:58.112254+0000","last_clean":"2024-03-09T15:36:58.112254+0000","last_became_active":"2024-03-09T14:28:32.314258+0000","last_became_peered":"2024-03-09T14:28:32.314258+0000","last_unstale":"2024-03-09T15:36:58.112254+0000","last_undegraded":"2024-03-09T15:36:58.112254+0000","last_fullsized":"2024-03-09T15:36:58.112254+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'137","last_scrub_stamp":"2024-03-09T15:36:58.112062+0000","last_deep_scrub":"261'137","last_deep_scrub_stamp":"2024-03-09T15:36:58.112062+0000","last_clean_scrub_stamp":"2024-03-09T15:36:58.112062+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":100,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:35:31.896371+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":169,"num_read_kb":17581,"num_write":466,"num_write_kb":22929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3e","version":"261'44","reported_seq":620,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.733770+0000","last_change":"2024-03-09T15:36:29.733770+0000","last_active":"2024-03-09T15:36:29.733770+0000","last_peered":"2024-03-09T15:36:29.733770+0000","last_clean":"2024-03-09T15:36:29.733770+0000","last_became_active":"2024-03-09T14:28:34.322206+0000","last_became_peered":"2024-03-09T14:28:34.322206+0000","last_unstale":"2024-03-09T15:36:29.733770+0000","last_undegraded":"2024-03-09T15:36:29.733770+0000","last_fullsized":"2024-03-09T15:36:29.733770+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:29.733482+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:29.733482+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.733482+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:59:58.948789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":201,"num_read_kb":171,"num_write":60,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"261'155","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:04.787270+0000","last_change":"2024-03-09T15:37:04.787354+0000","last_active":"2024-03-09T15:37:04.787270+0000","last_peered":"2024-03-09T15:37:04.787270+0000","last_clean":"2024-03-09T15:37:04.787270+0000","last_became_active":"2024-03-09T14:28:32.308522+0000","last_became_peered":"2024-03-09T14:28:32.308522+0000","last_unstale":"2024-03-09T15:37:04.787270+0000","last_undegraded":"2024-03-09T15:37:04.787270+0000","last_fullsized":"2024-03-09T15:37:04.787270+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'155","last_scrub_stamp":"2024-03-09T15:37:04.787076+0000","last_deep_scrub":"261'155","last_deep_scrub_stamp":"2024-03-09T15:37:04.787076+0000","last_clean_scrub_stamp":"2024-03-09T15:37:04.787076+0000","objects_scrubbed":5,"log_size":55,"log_dups_size":100,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:38.097065+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525396,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":175,"num_read_kb":14130,"num_write":546,"num_write_kb":36844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"261'52","reported_seq":613,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.219343+0000","last_change":"2024-03-09T15:36:34.219343+0000","last_active":"2024-03-09T15:36:34.219343+0000","last_peered":"2024-03-09T15:36:34.219343+0000","last_clean":"2024-03-09T15:36:34.219343+0000","last_became_active":"2024-03-09T14:28:34.320646+0000","last_became_peered":"2024-03-09T14:28:34.320646+0000","last_unstale":"2024-03-09T15:36:34.219343+0000","last_undegraded":"2024-03-09T15:36:34.219343+0000","last_fullsized":"2024-03-09T15:36:34.219343+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'52","last_scrub_stamp":"2024-03-09T15:36:34.219014+0000","last_deep_scrub":"261'52","last_deep_scrub_stamp":"2024-03-09T15:36:34.219014+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.219014+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:48:47.044377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":174,"num_write":79,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"261'158","reported_seq":752,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:54.501584+0000","last_change":"2024-03-09T15:36:54.501668+0000","last_active":"2024-03-09T15:36:54.501584+0000","last_peered":"2024-03-09T15:36:54.501584+0000","last_clean":"2024-03-09T15:36:54.501584+0000","last_became_active":"2024-03-09T14:28:34.306592+0000","last_became_peered":"2024-03-09T14:28:34.306592+0000","last_unstale":"2024-03-09T15:36:54.501584+0000","last_undegraded":"2024-03-09T15:36:54.501584+0000","last_fullsized":"2024-03-09T15:36:54.501584+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'158","last_scrub_stamp":"2024-03-09T15:36:54.501377+0000","last_deep_scrub":"261'158","last_deep_scrub_stamp":"2024-03-09T15:36:54.501377+0000","last_clean_scrub_stamp":"2024-03-09T15:36:54.501377+0000","objects_scrubbed":1,"log_size":58,"log_dups_size":100,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:42:48.686833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":599,"num_read_kb":473,"num_write":232,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"261'165","reported_seq":727,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.220340+0000","last_change":"2024-03-09T15:36:16.220340+0000","last_active":"2024-03-09T15:36:16.220340+0000","last_peered":"2024-03-09T15:36:16.220340+0000","last_clean":"2024-03-09T15:36:16.220340+0000","last_became_active":"2024-03-09T14:29:06.438523+0000","last_became_peered":"2024-03-09T14:29:06.438523+0000","last_unstale":"2024-03-09T15:36:16.220340+0000","last_undegraded":"2024-03-09T15:36:16.220340+0000","last_fullsized":"2024-03-09T15:36:16.220340+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:16.219968+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:16.219968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.219968+0000","objects_scrubbed":4,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:42:29.990282+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728801,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":293,"num_read_kb":1034,"num_write":597,"num_write_kb":27895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.31","version":"261'88","reported_seq":652,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.145113+0000","last_change":"2024-03-09T15:36:46.145171+0000","last_active":"2024-03-09T15:36:46.145113+0000","last_peered":"2024-03-09T15:36:46.145113+0000","last_clean":"2024-03-09T15:36:46.145113+0000","last_became_active":"2024-03-09T14:28:34.321352+0000","last_became_peered":"2024-03-09T14:28:34.321352+0000","last_unstale":"2024-03-09T15:36:46.145113+0000","last_undegraded":"2024-03-09T15:36:46.145113+0000","last_fullsized":"2024-03-09T15:36:46.145113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'88","last_scrub_stamp":"2024-03-09T15:36:46.144913+0000","last_deep_scrub":"261'88","last_deep_scrub_stamp":"2024-03-09T15:36:46.144913+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.144913+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:37:05.731149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":342,"num_read_kb":270,"num_write":126,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"261'133","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.262628+0000","last_change":"2024-03-09T15:36:43.262702+0000","last_active":"2024-03-09T15:36:43.262628+0000","last_peered":"2024-03-09T15:36:43.262628+0000","last_clean":"2024-03-09T15:36:43.262628+0000","last_became_active":"2024-03-09T14:28:32.334208+0000","last_became_peered":"2024-03-09T14:28:32.334208+0000","last_unstale":"2024-03-09T15:36:43.262628+0000","last_undegraded":"2024-03-09T15:36:43.262628+0000","last_fullsized":"2024-03-09T15:36:43.262628+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'133","last_scrub_stamp":"2024-03-09T15:36:43.262446+0000","last_deep_scrub":"261'133","last_deep_scrub_stamp":"2024-03-09T15:36:43.262446+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.262446+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:21:20.422366+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262185,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":143,"num_read_kb":128,"num_write":408,"num_write_kb":22112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"261'62","reported_seq":618,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.784398+0000","last_change":"2024-03-09T15:36:35.784465+0000","last_active":"2024-03-09T15:36:35.784398+0000","last_peered":"2024-03-09T15:36:35.784398+0000","last_clean":"2024-03-09T15:36:35.784398+0000","last_became_active":"2024-03-09T14:28:34.298428+0000","last_became_peered":"2024-03-09T14:28:34.298428+0000","last_unstale":"2024-03-09T15:36:35.784398+0000","last_undegraded":"2024-03-09T15:36:35.784398+0000","last_fullsized":"2024-03-09T15:36:35.784398+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'62","last_scrub_stamp":"2024-03-09T15:36:35.784200+0000","last_deep_scrub":"261'62","last_deep_scrub_stamp":"2024-03-09T15:36:35.784200+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.784200+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:32:34.722288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":202,"num_write":92,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"261'118","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.242144+0000","last_change":"2024-03-09T15:36:39.242234+0000","last_active":"2024-03-09T15:36:39.242144+0000","last_peered":"2024-03-09T15:36:39.242144+0000","last_clean":"2024-03-09T15:36:39.242144+0000","last_became_active":"2024-03-09T14:28:32.318963+0000","last_became_peered":"2024-03-09T14:28:32.318963+0000","last_unstale":"2024-03-09T15:36:39.242144+0000","last_undegraded":"2024-03-09T15:36:39.242144+0000","last_fullsized":"2024-03-09T15:36:39.242144+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'118","last_scrub_stamp":"2024-03-09T15:36:39.241932+0000","last_deep_scrub":"261'118","last_deep_scrub_stamp":"2024-03-09T15:36:39.241932+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.241932+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:06:46.492058+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170436,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":164,"num_read_kb":1700,"num_write":435,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.33","version":"261'102","reported_seq":673,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.201686+0000","last_change":"2024-03-09T15:36:33.201745+0000","last_active":"2024-03-09T15:36:33.201686+0000","last_peered":"2024-03-09T15:36:33.201686+0000","last_clean":"2024-03-09T15:36:33.201686+0000","last_became_active":"2024-03-09T14:28:34.320331+0000","last_became_peered":"2024-03-09T14:28:34.320331+0000","last_unstale":"2024-03-09T15:36:33.201686+0000","last_undegraded":"2024-03-09T15:36:33.201686+0000","last_fullsized":"2024-03-09T15:36:33.201686+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'102","last_scrub_stamp":"2024-03-09T15:36:33.201501+0000","last_deep_scrub":"261'102","last_deep_scrub_stamp":"2024-03-09T15:36:33.201501+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.201501+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:07:30.758372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":320,"num_write":144,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"261'136","reported_seq":698,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.465189+0000","last_change":"2024-03-09T15:36:51.465274+0000","last_active":"2024-03-09T15:36:51.465189+0000","last_peered":"2024-03-09T15:36:51.465189+0000","last_clean":"2024-03-09T15:36:51.465189+0000","last_became_active":"2024-03-09T14:28:32.337934+0000","last_became_peered":"2024-03-09T14:28:32.337934+0000","last_unstale":"2024-03-09T15:36:51.465189+0000","last_undegraded":"2024-03-09T15:36:51.465189+0000","last_fullsized":"2024-03-09T15:36:51.465189+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'136","last_scrub_stamp":"2024-03-09T15:36:51.465001+0000","last_deep_scrub":"261'136","last_deep_scrub_stamp":"2024-03-09T15:36:51.465001+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.465001+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:36:13.771727+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263250,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":163,"num_read_kb":12763,"num_write":465,"num_write_kb":21372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"261'69","reported_seq":615,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.229898+0000","last_change":"2024-03-09T15:36:31.229983+0000","last_active":"2024-03-09T15:36:31.229898+0000","last_peered":"2024-03-09T15:36:31.229898+0000","last_clean":"2024-03-09T15:36:31.229898+0000","last_became_active":"2024-03-09T14:28:34.310703+0000","last_became_peered":"2024-03-09T14:28:34.310703+0000","last_unstale":"2024-03-09T15:36:31.229898+0000","last_undegraded":"2024-03-09T15:36:31.229898+0000","last_fullsized":"2024-03-09T15:36:31.229898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'69","last_scrub_stamp":"2024-03-09T15:36:31.229701+0000","last_deep_scrub":"261'69","last_deep_scrub_stamp":"2024-03-09T15:36:31.229701+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.229701+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:52:58.555942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":272,"num_read_kb":220,"num_write":109,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"261'162","reported_seq":738,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:56.774266+0000","last_change":"2024-03-09T15:36:56.774342+0000","last_active":"2024-03-09T15:36:56.774266+0000","last_peered":"2024-03-09T15:36:56.774266+0000","last_clean":"2024-03-09T15:36:56.774266+0000","last_became_active":"2024-03-09T14:28:32.336636+0000","last_became_peered":"2024-03-09T14:28:32.336636+0000","last_unstale":"2024-03-09T15:36:56.774266+0000","last_undegraded":"2024-03-09T15:36:56.774266+0000","last_fullsized":"2024-03-09T15:36:56.774266+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'162","last_scrub_stamp":"2024-03-09T15:36:56.774051+0000","last_deep_scrub":"261'162","last_deep_scrub_stamp":"2024-03-09T15:36:56.774051+0000","last_clean_scrub_stamp":"2024-03-09T15:36:56.774051+0000","objects_scrubbed":5,"log_size":62,"log_dups_size":100,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:51:33.272809+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262241,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":231,"num_read_kb":12490,"num_write":578,"num_write_kb":28462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"261'87","reported_seq":647,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.123887+0000","last_change":"2024-03-09T15:36:33.123953+0000","last_active":"2024-03-09T15:36:33.123887+0000","last_peered":"2024-03-09T15:36:33.123887+0000","last_clean":"2024-03-09T15:36:33.123887+0000","last_became_active":"2024-03-09T14:28:34.314665+0000","last_became_peered":"2024-03-09T14:28:34.314665+0000","last_unstale":"2024-03-09T15:36:33.123887+0000","last_undegraded":"2024-03-09T15:36:33.123887+0000","last_fullsized":"2024-03-09T15:36:33.123887+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'87","last_scrub_stamp":"2024-03-09T15:36:33.123670+0000","last_deep_scrub":"261'87","last_deep_scrub_stamp":"2024-03-09T15:36:33.123670+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.123670+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:07:39.698152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":335,"num_read_kb":265,"num_write":124,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"261'122","reported_seq":693,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:01.146225+0000","last_change":"2024-03-09T15:37:01.146373+0000","last_active":"2024-03-09T15:37:01.146225+0000","last_peered":"2024-03-09T15:37:01.146225+0000","last_clean":"2024-03-09T15:37:01.146225+0000","last_became_active":"2024-03-09T14:28:32.298428+0000","last_became_peered":"2024-03-09T14:28:32.298428+0000","last_unstale":"2024-03-09T15:37:01.146225+0000","last_undegraded":"2024-03-09T15:37:01.146225+0000","last_fullsized":"2024-03-09T15:37:01.146225+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'122","last_scrub_stamp":"2024-03-09T15:37:01.145980+0000","last_deep_scrub":"261'122","last_deep_scrub_stamp":"2024-03-09T15:37:01.145980+0000","last_clean_scrub_stamp":"2024-03-09T15:37:01.145980+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":100,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:41.749323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":191,"num_read_kb":31150,"num_write":464,"num_write_kb":37586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"261'76","reported_seq":591,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.170281+0000","last_change":"2024-03-09T15:36:18.170281+0000","last_active":"2024-03-09T15:36:18.170281+0000","last_peered":"2024-03-09T15:36:18.170281+0000","last_clean":"2024-03-09T15:36:18.170281+0000","last_became_active":"2024-03-09T14:28:42.403553+0000","last_became_peered":"2024-03-09T14:28:42.403553+0000","last_unstale":"2024-03-09T15:36:18.170281+0000","last_undegraded":"2024-03-09T15:36:18.170281+0000","last_fullsized":"2024-03-09T15:36:18.170281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'76","last_scrub_stamp":"2024-03-09T15:36:18.170018+0000","last_deep_scrub":"261'76","last_deep_scrub_stamp":"2024-03-09T15:36:18.170018+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.170018+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:55:35.040311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":84,"num_write":184,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"261'98","reported_seq":674,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.480604+0000","last_change":"2024-03-09T15:36:20.480604+0000","last_active":"2024-03-09T15:36:20.480604+0000","last_peered":"2024-03-09T15:36:20.480604+0000","last_clean":"2024-03-09T15:36:20.480604+0000","last_became_active":"2024-03-09T14:29:06.439359+0000","last_became_peered":"2024-03-09T14:29:06.439359+0000","last_unstale":"2024-03-09T15:36:20.480604+0000","last_undegraded":"2024-03-09T15:36:20.480604+0000","last_fullsized":"2024-03-09T15:36:20.480604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'98","last_scrub_stamp":"2024-03-09T15:36:20.480339+0000","last_deep_scrub":"261'98","last_deep_scrub_stamp":"2024-03-09T15:36:20.480339+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.480339+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:32:16.216352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":413,"num_read_kb":333,"num_write":149,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"261'172","reported_seq":803,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.158265+0000","last_change":"2024-03-09T15:36:50.158375+0000","last_active":"2024-03-09T15:36:50.158265+0000","last_peered":"2024-03-09T15:36:50.158265+0000","last_clean":"2024-03-09T15:36:50.158265+0000","last_became_active":"2024-03-09T14:28:32.329457+0000","last_became_peered":"2024-03-09T14:28:32.329457+0000","last_unstale":"2024-03-09T15:36:50.158265+0000","last_undegraded":"2024-03-09T15:36:50.158265+0000","last_fullsized":"2024-03-09T15:36:50.158265+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'172","last_scrub_stamp":"2024-03-09T15:36:50.157939+0000","last_deep_scrub":"261'172","last_deep_scrub_stamp":"2024-03-09T15:36:50.157939+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.157939+0000","objects_scrubbed":7,"log_size":72,"log_dups_size":100,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:28.447473+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506100,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":263,"num_read_kb":141328,"num_write":603,"num_write_kb":37022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.929658+0000","last_change":"2024-03-09T15:36:40.929714+0000","last_active":"2024-03-09T15:36:40.929658+0000","last_peered":"2024-03-09T15:36:40.929658+0000","last_clean":"2024-03-09T15:36:40.929658+0000","last_became_active":"2024-03-09T14:28:36.320603+0000","last_became_peered":"2024-03-09T14:28:36.320603+0000","last_unstale":"2024-03-09T15:36:40.929658+0000","last_undegraded":"2024-03-09T15:36:40.929658+0000","last_fullsized":"2024-03-09T15:36:40.929658+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:40.929451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:40.929451+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.929451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:14:31.906125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"261'233","reported_seq":853,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:53.265716+0000","last_change":"2024-03-09T15:36:53.265789+0000","last_active":"2024-03-09T15:36:53.265716+0000","last_peered":"2024-03-09T15:36:53.265716+0000","last_clean":"2024-03-09T15:36:53.265716+0000","last_became_active":"2024-03-09T14:28:38.326826+0000","last_became_peered":"2024-03-09T14:28:38.326826+0000","last_unstale":"2024-03-09T15:36:53.265716+0000","last_undegraded":"2024-03-09T15:36:53.265716+0000","last_fullsized":"2024-03-09T15:36:53.265716+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'233","last_scrub_stamp":"2024-03-09T15:36:53.265513+0000","last_deep_scrub":"261'233","last_deep_scrub_stamp":"2024-03-09T15:36:53.265513+0000","last_clean_scrub_stamp":"2024-03-09T15:36:53.265513+0000","objects_scrubbed":4,"log_size":33,"log_dups_size":200,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:52:51.677937+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11101,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":365,"num_read_kb":621,"num_write":250,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.268381+0000","last_change":"2024-03-09T15:36:36.268437+0000","last_active":"2024-03-09T15:36:36.268381+0000","last_peered":"2024-03-09T15:36:36.268381+0000","last_clean":"2024-03-09T15:36:36.268381+0000","last_became_active":"2024-03-09T14:28:40.358954+0000","last_became_peered":"2024-03-09T14:28:40.358954+0000","last_unstale":"2024-03-09T15:36:36.268381+0000","last_undegraded":"2024-03-09T15:36:36.268381+0000","last_fullsized":"2024-03-09T15:36:36.268381+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:36.268123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:36.268123+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.268123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:59:04.556053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"261'772","reported_seq":2395,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.694693+0000","last_change":"2024-03-09T15:36:20.694693+0000","last_active":"2024-03-09T15:36:20.694693+0000","last_peered":"2024-03-09T15:36:20.694693+0000","last_clean":"2024-03-09T15:36:20.694693+0000","last_became_active":"2024-03-09T14:28:42.394691+0000","last_became_peered":"2024-03-09T14:28:42.394691+0000","last_unstale":"2024-03-09T15:36:20.694693+0000","last_undegraded":"2024-03-09T15:36:20.694693+0000","last_fullsized":"2024-03-09T15:36:20.694693+0000","mapping_epoch":30,"log_start":"261'700","ondisk_log_start":"261'700","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'772","last_scrub_stamp":"2024-03-09T15:36:20.694425+0000","last_deep_scrub":"261'772","last_deep_scrub_stamp":"2024-03-09T15:36:20.694425+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.694425+0000","objects_scrubbed":3,"log_size":72,"log_dups_size":700,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:30:23.952683+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":612,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2611,"num_read_kb":2601,"num_write":1560,"num_write_kb":569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":376,"num_omap_keys":2,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"261'40","reported_seq":588,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.759957+0000","last_change":"2024-03-09T15:36:32.760017+0000","last_active":"2024-03-09T15:36:32.759957+0000","last_peered":"2024-03-09T15:36:32.759957+0000","last_clean":"2024-03-09T15:36:32.759957+0000","last_became_active":"2024-03-09T14:28:34.324569+0000","last_became_peered":"2024-03-09T14:28:34.324569+0000","last_unstale":"2024-03-09T15:36:32.759957+0000","last_undegraded":"2024-03-09T15:36:32.759957+0000","last_fullsized":"2024-03-09T15:36:32.759957+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'40","last_scrub_stamp":"2024-03-09T15:36:32.759758+0000","last_deep_scrub":"261'40","last_deep_scrub_stamp":"2024-03-09T15:36:32.759758+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.759758+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:33:41.150112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":168,"num_read_kb":136,"num_write":58,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"261'176","reported_seq":756,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:56.238761+0000","last_change":"2024-03-09T15:36:56.238986+0000","last_active":"2024-03-09T15:36:56.238761+0000","last_peered":"2024-03-09T15:36:56.238761+0000","last_clean":"2024-03-09T15:36:56.238761+0000","last_became_active":"2024-03-09T14:28:32.309536+0000","last_became_peered":"2024-03-09T14:28:32.309536+0000","last_unstale":"2024-03-09T15:36:56.238761+0000","last_undegraded":"2024-03-09T15:36:56.238761+0000","last_fullsized":"2024-03-09T15:36:56.238761+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'176","last_scrub_stamp":"2024-03-09T15:36:56.238518+0000","last_deep_scrub":"261'176","last_deep_scrub_stamp":"2024-03-09T15:36:56.238518+0000","last_clean_scrub_stamp":"2024-03-09T15:36:56.238518+0000","objects_scrubbed":7,"log_size":76,"log_dups_size":100,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:49:34.426356+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":29533,"num_write":652,"num_write_kb":30801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:53.738712+0000","last_change":"2024-03-09T15:36:53.738824+0000","last_active":"2024-03-09T15:36:53.738712+0000","last_peered":"2024-03-09T15:36:53.738712+0000","last_clean":"2024-03-09T15:36:53.738712+0000","last_became_active":"2024-03-09T14:28:36.320079+0000","last_became_peered":"2024-03-09T14:28:36.320079+0000","last_unstale":"2024-03-09T15:36:53.738712+0000","last_undegraded":"2024-03-09T15:36:53.738712+0000","last_fullsized":"2024-03-09T15:36:53.738712+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:53.738470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:53.738470+0000","last_clean_scrub_stamp":"2024-03-09T15:36:53.738470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:59:26.919081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"261'90","reported_seq":636,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.122254+0000","last_change":"2024-03-09T15:36:27.122328+0000","last_active":"2024-03-09T15:36:27.122254+0000","last_peered":"2024-03-09T15:36:27.122254+0000","last_clean":"2024-03-09T15:36:27.122254+0000","last_became_active":"2024-03-09T14:28:38.340896+0000","last_became_peered":"2024-03-09T14:28:38.340896+0000","last_unstale":"2024-03-09T15:36:27.122254+0000","last_undegraded":"2024-03-09T15:36:27.122254+0000","last_fullsized":"2024-03-09T15:36:27.122254+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'90","last_scrub_stamp":"2024-03-09T15:36:27.122054+0000","last_deep_scrub":"261'90","last_deep_scrub_stamp":"2024-03-09T15:36:27.122054+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.122054+0000","objects_scrubbed":3,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:23:13.339583+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":101,"num_write":68,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.119178+0000","last_change":"2024-03-09T15:36:23.119232+0000","last_active":"2024-03-09T15:36:23.119178+0000","last_peered":"2024-03-09T15:36:23.119178+0000","last_clean":"2024-03-09T15:36:23.119178+0000","last_became_active":"2024-03-09T14:28:40.370788+0000","last_became_peered":"2024-03-09T14:28:40.370788+0000","last_unstale":"2024-03-09T15:36:23.119178+0000","last_undegraded":"2024-03-09T15:36:23.119178+0000","last_fullsized":"2024-03-09T15:36:23.119178+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:23.118978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:23.118978+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.118978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:04:36.080050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"261'64","reported_seq":576,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.282044+0000","last_change":"2024-03-09T15:36:21.282129+0000","last_active":"2024-03-09T15:36:21.282044+0000","last_peered":"2024-03-09T15:36:21.282044+0000","last_clean":"2024-03-09T15:36:21.282044+0000","last_became_active":"2024-03-09T14:28:42.408738+0000","last_became_peered":"2024-03-09T14:28:42.408738+0000","last_unstale":"2024-03-09T15:36:21.282044+0000","last_undegraded":"2024-03-09T15:36:21.282044+0000","last_fullsized":"2024-03-09T15:36:21.282044+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'64","last_scrub_stamp":"2024-03-09T15:36:21.281839+0000","last_deep_scrub":"261'64","last_deep_scrub_stamp":"2024-03-09T15:36:21.281839+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.281839+0000","objects_scrubbed":5,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:53:36.132862+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":681,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":76,"num_read_kb":75,"num_write":140,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"261'157","reported_seq":755,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.346427+0000","last_change":"2024-03-09T15:36:44.346583+0000","last_active":"2024-03-09T15:36:44.346427+0000","last_peered":"2024-03-09T15:36:44.346427+0000","last_clean":"2024-03-09T15:36:44.346427+0000","last_became_active":"2024-03-09T14:28:34.304892+0000","last_became_peered":"2024-03-09T14:28:34.304892+0000","last_unstale":"2024-03-09T15:36:44.346427+0000","last_undegraded":"2024-03-09T15:36:44.346427+0000","last_fullsized":"2024-03-09T15:36:44.346427+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'157","last_scrub_stamp":"2024-03-09T15:36:44.346198+0000","last_deep_scrub":"261'157","last_deep_scrub_stamp":"2024-03-09T15:36:44.346198+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.346198+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:31:38.007816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":622,"num_read_kb":492,"num_write":239,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"261'164","reported_seq":874,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:02.159926+0000","last_change":"2024-03-09T15:37:02.160136+0000","last_active":"2024-03-09T15:37:02.159926+0000","last_peered":"2024-03-09T15:37:02.159926+0000","last_clean":"2024-03-09T15:37:02.159926+0000","last_became_active":"2024-03-09T14:28:32.318428+0000","last_became_peered":"2024-03-09T14:28:32.318428+0000","last_unstale":"2024-03-09T15:37:02.159926+0000","last_undegraded":"2024-03-09T15:37:02.159926+0000","last_fullsized":"2024-03-09T15:37:02.159926+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'164","last_scrub_stamp":"2024-03-09T15:37:02.159697+0000","last_deep_scrub":"261'164","last_deep_scrub_stamp":"2024-03-09T15:37:02.159697+0000","last_clean_scrub_stamp":"2024-03-09T15:37:02.159697+0000","objects_scrubbed":3,"log_size":64,"log_dups_size":100,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:47:12.429092+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505054,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":544,"num_read_kb":113871,"num_write":619,"num_write_kb":46622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":506,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:55.753285+0000","last_change":"2024-03-09T15:36:55.753338+0000","last_active":"2024-03-09T15:36:55.753285+0000","last_peered":"2024-03-09T15:36:55.753285+0000","last_clean":"2024-03-09T15:36:55.753285+0000","last_became_active":"2024-03-09T14:28:36.320095+0000","last_became_peered":"2024-03-09T14:28:36.320095+0000","last_unstale":"2024-03-09T15:36:55.753285+0000","last_undegraded":"2024-03-09T15:36:55.753285+0000","last_fullsized":"2024-03-09T15:36:55.753285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:55.753090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:55.753090+0000","last_clean_scrub_stamp":"2024-03-09T15:36:55.753090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:31:29.223715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"261'108","reported_seq":668,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.165000+0000","last_change":"2024-03-09T15:36:17.165000+0000","last_active":"2024-03-09T15:36:17.165000+0000","last_peered":"2024-03-09T15:36:17.165000+0000","last_clean":"2024-03-09T15:36:17.165000+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:36:17.165000+0000","last_undegraded":"2024-03-09T15:36:17.165000+0000","last_fullsized":"2024-03-09T15:36:17.165000+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'108","last_scrub_stamp":"2024-03-09T15:36:17.164692+0000","last_deep_scrub":"261'108","last_deep_scrub_stamp":"2024-03-09T15:36:17.164692+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.164692+0000","objects_scrubbed":5,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:41:19.308226+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"261'505","reported_seq":2301,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.252225+0000","last_change":"2024-03-09T15:36:34.252290+0000","last_active":"2024-03-09T15:36:34.252225+0000","last_peered":"2024-03-09T15:36:34.252225+0000","last_clean":"2024-03-09T15:36:34.252225+0000","last_became_active":"2024-03-09T14:28:40.371507+0000","last_became_peered":"2024-03-09T14:28:40.371507+0000","last_unstale":"2024-03-09T15:36:34.252225+0000","last_undegraded":"2024-03-09T15:36:34.252225+0000","last_fullsized":"2024-03-09T15:36:34.252225+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'505","last_scrub_stamp":"2024-03-09T15:36:34.252038+0000","last_deep_scrub":"261'505","last_deep_scrub_stamp":"2024-03-09T15:36:34.252038+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.252038+0000","objects_scrubbed":1,"log_size":105,"log_dups_size":400,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:50:10.320691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"261'138","reported_seq":953,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.158161+0000","last_change":"2024-03-09T15:36:19.158227+0000","last_active":"2024-03-09T15:36:19.158161+0000","last_peered":"2024-03-09T15:36:19.158161+0000","last_clean":"2024-03-09T15:36:19.158161+0000","last_became_active":"2024-03-09T14:28:42.398498+0000","last_became_peered":"2024-03-09T14:28:42.398498+0000","last_unstale":"2024-03-09T15:36:19.158161+0000","last_undegraded":"2024-03-09T15:36:19.158161+0000","last_fullsized":"2024-03-09T15:36:19.158161+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'138","last_scrub_stamp":"2024-03-09T15:36:19.157968+0000","last_deep_scrub":"261'138","last_deep_scrub_stamp":"2024-03-09T15:36:19.157968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.157968+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":100,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:49:21.053160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":498,"num_read_kb":498,"num_write":277,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":234,"num_omap_keys":2,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"261'53","reported_seq":602,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.567436+0000","last_change":"2024-03-09T15:36:43.567493+0000","last_active":"2024-03-09T15:36:43.567436+0000","last_peered":"2024-03-09T15:36:43.567436+0000","last_clean":"2024-03-09T15:36:43.567436+0000","last_became_active":"2024-03-09T14:28:34.330347+0000","last_became_peered":"2024-03-09T14:28:34.330347+0000","last_unstale":"2024-03-09T15:36:43.567436+0000","last_undegraded":"2024-03-09T15:36:43.567436+0000","last_fullsized":"2024-03-09T15:36:43.567436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'53","last_scrub_stamp":"2024-03-09T15:36:43.567238+0000","last_deep_scrub":"261'53","last_deep_scrub_stamp":"2024-03-09T15:36:43.567238+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.567238+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:24:51.206691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":212,"num_read_kb":171,"num_write":77,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"261'169","reported_seq":751,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.275838+0000","last_change":"2024-03-09T15:36:38.275929+0000","last_active":"2024-03-09T15:36:38.275838+0000","last_peered":"2024-03-09T15:36:38.275838+0000","last_clean":"2024-03-09T15:36:38.275838+0000","last_became_active":"2024-03-09T14:28:32.319819+0000","last_became_peered":"2024-03-09T14:28:32.319819+0000","last_unstale":"2024-03-09T15:36:38.275838+0000","last_undegraded":"2024-03-09T15:36:38.275838+0000","last_fullsized":"2024-03-09T15:36:38.275838+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'169","last_scrub_stamp":"2024-03-09T15:36:38.275636+0000","last_deep_scrub":"261'169","last_deep_scrub_stamp":"2024-03-09T15:36:38.275636+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.275636+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":100,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:18:08.037585+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5244508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":231,"num_read_kb":15567,"num_write":579,"num_write_kb":34195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.872479+0000","last_change":"2024-03-09T15:36:38.872534+0000","last_active":"2024-03-09T15:36:38.872479+0000","last_peered":"2024-03-09T15:36:38.872479+0000","last_clean":"2024-03-09T15:36:38.872479+0000","last_became_active":"2024-03-09T14:28:36.310231+0000","last_became_peered":"2024-03-09T14:28:36.310231+0000","last_unstale":"2024-03-09T15:36:38.872479+0000","last_undegraded":"2024-03-09T15:36:38.872479+0000","last_fullsized":"2024-03-09T15:36:38.872479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:38.872296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:38.872296+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.872296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:03:09.877161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"261'174","reported_seq":767,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:03.157279+0000","last_change":"2024-03-09T15:37:03.157372+0000","last_active":"2024-03-09T15:37:03.157279+0000","last_peered":"2024-03-09T15:37:03.157279+0000","last_clean":"2024-03-09T15:37:03.157279+0000","last_became_active":"2024-03-09T14:28:38.598308+0000","last_became_peered":"2024-03-09T14:28:38.598308+0000","last_unstale":"2024-03-09T15:37:03.157279+0000","last_undegraded":"2024-03-09T15:37:03.157279+0000","last_fullsized":"2024-03-09T15:37:03.157279+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'174","last_scrub_stamp":"2024-03-09T15:37:03.157072+0000","last_deep_scrub":"261'174","last_deep_scrub_stamp":"2024-03-09T15:37:03.157072+0000","last_clean_scrub_stamp":"2024-03-09T15:37:03.157072+0000","objects_scrubbed":4,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:09:04.298940+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8724,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":532,"num_write":190,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"261'513","reported_seq":2509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.458248+0000","last_change":"2024-03-09T15:36:32.458248+0000","last_active":"2024-03-09T15:36:32.458248+0000","last_peered":"2024-03-09T15:36:32.458248+0000","last_clean":"2024-03-09T15:36:32.458248+0000","last_became_active":"2024-03-09T14:28:40.387990+0000","last_became_peered":"2024-03-09T14:28:40.387990+0000","last_unstale":"2024-03-09T15:36:32.458248+0000","last_undegraded":"2024-03-09T15:36:32.458248+0000","last_fullsized":"2024-03-09T15:36:32.458248+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'513","last_scrub_stamp":"2024-03-09T15:36:32.457985+0000","last_deep_scrub":"261'513","last_deep_scrub_stamp":"2024-03-09T15:36:32.457985+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.457985+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":500,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:24:58.183827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"261'101","reported_seq":687,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.740079+0000","last_change":"2024-03-09T15:36:28.740141+0000","last_active":"2024-03-09T15:36:28.740079+0000","last_peered":"2024-03-09T15:36:28.740079+0000","last_clean":"2024-03-09T15:36:28.740079+0000","last_became_active":"2024-03-09T14:28:34.322731+0000","last_became_peered":"2024-03-09T14:28:34.322731+0000","last_unstale":"2024-03-09T15:36:28.740079+0000","last_undegraded":"2024-03-09T15:36:28.740079+0000","last_fullsized":"2024-03-09T15:36:28.740079+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'101","last_scrub_stamp":"2024-03-09T15:36:28.739877+0000","last_deep_scrub":"261'101","last_deep_scrub_stamp":"2024-03-09T15:36:28.739877+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.739877+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:43:42.753565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":401,"num_read_kb":330,"num_write":141,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"261'130","reported_seq":688,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:52.150089+0000","last_change":"2024-03-09T15:36:52.150201+0000","last_active":"2024-03-09T15:36:52.150089+0000","last_peered":"2024-03-09T15:36:52.150089+0000","last_clean":"2024-03-09T15:36:52.150089+0000","last_became_active":"2024-03-09T14:28:32.329148+0000","last_became_peered":"2024-03-09T14:28:32.329148+0000","last_unstale":"2024-03-09T15:36:52.150089+0000","last_undegraded":"2024-03-09T15:36:52.150089+0000","last_fullsized":"2024-03-09T15:36:52.150089+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'130","last_scrub_stamp":"2024-03-09T15:36:52.149897+0000","last_deep_scrub":"261'130","last_deep_scrub_stamp":"2024-03-09T15:36:52.149897+0000","last_clean_scrub_stamp":"2024-03-09T15:36:52.149897+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:56:19.005464+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7947490,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":1766,"num_write":513,"num_write_kb":18289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":518,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.203914+0000","last_change":"2024-03-09T15:36:39.204023+0000","last_active":"2024-03-09T15:36:39.203914+0000","last_peered":"2024-03-09T15:36:39.203914+0000","last_clean":"2024-03-09T15:36:39.203914+0000","last_became_active":"2024-03-09T14:28:36.313904+0000","last_became_peered":"2024-03-09T14:28:36.313904+0000","last_unstale":"2024-03-09T15:36:39.203914+0000","last_undegraded":"2024-03-09T15:36:39.203914+0000","last_fullsized":"2024-03-09T15:36:39.203914+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:39.203674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:39.203674+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.203674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:18:37.090196+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"261'276","reported_seq":920,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.746370+0000","last_change":"2024-03-09T15:36:24.746541+0000","last_active":"2024-03-09T15:36:24.746370+0000","last_peered":"2024-03-09T15:36:24.746370+0000","last_clean":"2024-03-09T15:36:24.746370+0000","last_became_active":"2024-03-09T14:28:38.597247+0000","last_became_peered":"2024-03-09T14:28:38.597247+0000","last_unstale":"2024-03-09T15:36:24.746370+0000","last_undegraded":"2024-03-09T15:36:24.746370+0000","last_fullsized":"2024-03-09T15:36:24.746370+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'276","last_scrub_stamp":"2024-03-09T15:36:24.746130+0000","last_deep_scrub":"261'276","last_deep_scrub_stamp":"2024-03-09T15:36:24.746130+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.746130+0000","objects_scrubbed":9,"log_size":76,"log_dups_size":200,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:53:37.971114+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9398,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":421,"num_read_kb":664,"num_write":284,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.394226+0000","last_change":"2024-03-09T15:36:37.394277+0000","last_active":"2024-03-09T15:36:37.394226+0000","last_peered":"2024-03-09T15:36:37.394226+0000","last_clean":"2024-03-09T15:36:37.394226+0000","last_became_active":"2024-03-09T14:28:40.359906+0000","last_became_peered":"2024-03-09T14:28:40.359906+0000","last_unstale":"2024-03-09T15:36:37.394226+0000","last_undegraded":"2024-03-09T15:36:37.394226+0000","last_fullsized":"2024-03-09T15:36:37.394226+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:37.394037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:37.394037+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.394037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:42:28.463933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"261'522","reported_seq":1370,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.664829+0000","last_change":"2024-03-09T15:36:16.664829+0000","last_active":"2024-03-09T15:36:16.664829+0000","last_peered":"2024-03-09T15:36:16.664829+0000","last_clean":"2024-03-09T15:36:16.664829+0000","last_became_active":"2024-03-09T14:28:42.402739+0000","last_became_peered":"2024-03-09T14:28:42.402739+0000","last_unstale":"2024-03-09T15:36:16.664829+0000","last_undegraded":"2024-03-09T15:36:16.664829+0000","last_fullsized":"2024-03-09T15:36:16.664829+0000","mapping_epoch":30,"log_start":"261'500","ondisk_log_start":"261'500","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'522","last_scrub_stamp":"2024-03-09T15:36:16.664560+0000","last_deep_scrub":"261'522","last_deep_scrub_stamp":"2024-03-09T15:36:16.664560+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.664560+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":500,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:05:16.194957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1336,"num_read_kb":1311,"num_write":1083,"num_write_kb":376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":381,"num_omap_keys":2,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"261'92","reported_seq":671,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:53.451238+0000","last_change":"2024-03-09T15:36:53.451297+0000","last_active":"2024-03-09T15:36:53.451238+0000","last_peered":"2024-03-09T15:36:53.451238+0000","last_clean":"2024-03-09T15:36:53.451238+0000","last_became_active":"2024-03-09T14:28:34.300795+0000","last_became_peered":"2024-03-09T14:28:34.300795+0000","last_unstale":"2024-03-09T15:36:53.451238+0000","last_undegraded":"2024-03-09T15:36:53.451238+0000","last_fullsized":"2024-03-09T15:36:53.451238+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'92","last_scrub_stamp":"2024-03-09T15:36:53.451044+0000","last_deep_scrub":"261'92","last_deep_scrub_stamp":"2024-03-09T15:36:53.451044+0000","last_clean_scrub_stamp":"2024-03-09T15:36:53.451044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:08:06.800010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":361,"num_read_kb":288,"num_write":150,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"261'148","reported_seq":730,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:01.166928+0000","last_change":"2024-03-09T15:37:01.167023+0000","last_active":"2024-03-09T15:37:01.166928+0000","last_peered":"2024-03-09T15:37:01.166928+0000","last_clean":"2024-03-09T15:37:01.166928+0000","last_became_active":"2024-03-09T14:28:32.318111+0000","last_became_peered":"2024-03-09T14:28:32.318111+0000","last_unstale":"2024-03-09T15:37:01.166928+0000","last_undegraded":"2024-03-09T15:37:01.166928+0000","last_fullsized":"2024-03-09T15:37:01.166928+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'148","last_scrub_stamp":"2024-03-09T15:37:01.166734+0000","last_deep_scrub":"261'148","last_deep_scrub_stamp":"2024-03-09T15:37:01.166734+0000","last_clean_scrub_stamp":"2024-03-09T15:37:01.166734+0000","objects_scrubbed":6,"log_size":48,"log_dups_size":100,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:17:04.846737+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242921,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":6230,"num_write":597,"num_write_kb":19779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.197656+0000","last_change":"2024-03-09T15:36:27.197765+0000","last_active":"2024-03-09T15:36:27.197656+0000","last_peered":"2024-03-09T15:36:27.197656+0000","last_clean":"2024-03-09T15:36:27.197656+0000","last_became_active":"2024-03-09T14:28:36.301227+0000","last_became_peered":"2024-03-09T14:28:36.301227+0000","last_unstale":"2024-03-09T15:36:27.197656+0000","last_undegraded":"2024-03-09T15:36:27.197656+0000","last_fullsized":"2024-03-09T15:36:27.197656+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:27.197420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:27.197420+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.197420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:56:31.881994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"261'177","reported_seq":763,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.577608+0000","last_change":"2024-03-09T15:36:40.577792+0000","last_active":"2024-03-09T15:36:40.577608+0000","last_peered":"2024-03-09T15:36:40.577608+0000","last_clean":"2024-03-09T15:36:40.577608+0000","last_became_active":"2024-03-09T14:28:38.346837+0000","last_became_peered":"2024-03-09T14:28:38.346837+0000","last_unstale":"2024-03-09T15:36:40.577608+0000","last_undegraded":"2024-03-09T15:36:40.577608+0000","last_fullsized":"2024-03-09T15:36:40.577608+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'177","last_scrub_stamp":"2024-03-09T15:36:40.577391+0000","last_deep_scrub":"261'177","last_deep_scrub_stamp":"2024-03-09T15:36:40.577391+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.577391+0000","objects_scrubbed":4,"log_size":77,"log_dups_size":100,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:47:13.834091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16267,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":276,"num_read_kb":528,"num_write":188,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.485270+0000","last_change":"2024-03-09T15:36:30.485329+0000","last_active":"2024-03-09T15:36:30.485270+0000","last_peered":"2024-03-09T15:36:30.485270+0000","last_clean":"2024-03-09T15:36:30.485270+0000","last_became_active":"2024-03-09T14:28:40.389175+0000","last_became_peered":"2024-03-09T14:28:40.389175+0000","last_unstale":"2024-03-09T15:36:30.485270+0000","last_undegraded":"2024-03-09T15:36:30.485270+0000","last_fullsized":"2024-03-09T15:36:30.485270+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:30.485071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:30.485071+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.485071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:54:36.706597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"261'59","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.297133+0000","last_change":"2024-03-09T15:36:34.297200+0000","last_active":"2024-03-09T15:36:34.297133+0000","last_peered":"2024-03-09T15:36:34.297133+0000","last_clean":"2024-03-09T15:36:34.297133+0000","last_became_active":"2024-03-09T14:28:42.401214+0000","last_became_peered":"2024-03-09T14:28:42.401214+0000","last_unstale":"2024-03-09T15:36:34.297133+0000","last_undegraded":"2024-03-09T15:36:34.297133+0000","last_fullsized":"2024-03-09T15:36:34.297133+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'59","last_scrub_stamp":"2024-03-09T15:36:34.296927+0000","last_deep_scrub":"261'59","last_deep_scrub_stamp":"2024-03-09T15:36:34.296927+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.296927+0000","objects_scrubbed":1,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:18:34.284628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":47,"num_write":140,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"261'78","reported_seq":643,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.249814+0000","last_change":"2024-03-09T15:36:33.249877+0000","last_active":"2024-03-09T15:36:33.249814+0000","last_peered":"2024-03-09T15:36:33.249814+0000","last_clean":"2024-03-09T15:36:33.249814+0000","last_became_active":"2024-03-09T14:28:34.316812+0000","last_became_peered":"2024-03-09T14:28:34.316812+0000","last_unstale":"2024-03-09T15:36:33.249814+0000","last_undegraded":"2024-03-09T15:36:33.249814+0000","last_fullsized":"2024-03-09T15:36:33.249814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'78","last_scrub_stamp":"2024-03-09T15:36:33.249613+0000","last_deep_scrub":"261'78","last_deep_scrub_stamp":"2024-03-09T15:36:33.249613+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.249613+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:38:51.977272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":320,"num_read_kb":256,"num_write":116,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"261'111","reported_seq":683,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.172728+0000","last_change":"2024-03-09T15:36:27.172803+0000","last_active":"2024-03-09T15:36:27.172728+0000","last_peered":"2024-03-09T15:36:27.172728+0000","last_clean":"2024-03-09T15:36:27.172728+0000","last_became_active":"2024-03-09T14:29:06.446333+0000","last_became_peered":"2024-03-09T14:29:06.446333+0000","last_unstale":"2024-03-09T15:36:27.172728+0000","last_undegraded":"2024-03-09T15:36:27.172728+0000","last_fullsized":"2024-03-09T15:36:27.172728+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'111","last_scrub_stamp":"2024-03-09T15:36:27.172540+0000","last_deep_scrub":"261'111","last_deep_scrub_stamp":"2024-03-09T15:36:27.172540+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.172540+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:12:44.004446+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":147,"num_read_kb":1025,"num_write":402,"num_write_kb":11167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":515,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.252418+0000","last_change":"2024-03-09T15:36:25.252530+0000","last_active":"2024-03-09T15:36:25.252418+0000","last_peered":"2024-03-09T15:36:25.252418+0000","last_clean":"2024-03-09T15:36:25.252418+0000","last_became_active":"2024-03-09T14:28:36.303695+0000","last_became_peered":"2024-03-09T14:28:36.303695+0000","last_unstale":"2024-03-09T15:36:25.252418+0000","last_undegraded":"2024-03-09T15:36:25.252418+0000","last_fullsized":"2024-03-09T15:36:25.252418+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.252128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.252128+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.252128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:47:06.758741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"261'227","reported_seq":844,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.149330+0000","last_change":"2024-03-09T15:36:51.149432+0000","last_active":"2024-03-09T15:36:51.149330+0000","last_peered":"2024-03-09T15:36:51.149330+0000","last_clean":"2024-03-09T15:36:51.149330+0000","last_became_active":"2024-03-09T14:28:38.339300+0000","last_became_peered":"2024-03-09T14:28:38.339300+0000","last_unstale":"2024-03-09T15:36:51.149330+0000","last_undegraded":"2024-03-09T15:36:51.149330+0000","last_fullsized":"2024-03-09T15:36:51.149330+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'227","last_scrub_stamp":"2024-03-09T15:36:51.149113+0000","last_deep_scrub":"261'227","last_deep_scrub_stamp":"2024-03-09T15:36:51.149113+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.149113+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":200,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:50:51.145329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8477,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":353,"num_read_kb":624,"num_write":244,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"261'501","reported_seq":2519,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.232198+0000","last_change":"2024-03-09T15:36:34.232325+0000","last_active":"2024-03-09T15:36:34.232198+0000","last_peered":"2024-03-09T15:36:34.232198+0000","last_clean":"2024-03-09T15:36:34.232198+0000","last_became_active":"2024-03-09T14:28:40.367119+0000","last_became_peered":"2024-03-09T14:28:40.367119+0000","last_unstale":"2024-03-09T15:36:34.232198+0000","last_undegraded":"2024-03-09T15:36:34.232198+0000","last_fullsized":"2024-03-09T15:36:34.232198+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'501","last_scrub_stamp":"2024-03-09T15:36:34.231965+0000","last_deep_scrub":"261'501","last_deep_scrub_stamp":"2024-03-09T15:36:34.231965+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.231965+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":400,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:18:21.657518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"261'82","reported_seq":585,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.245845+0000","last_change":"2024-03-09T15:36:28.245919+0000","last_active":"2024-03-09T15:36:28.245845+0000","last_peered":"2024-03-09T15:36:28.245845+0000","last_clean":"2024-03-09T15:36:28.245845+0000","last_became_active":"2024-03-09T14:28:42.396389+0000","last_became_peered":"2024-03-09T14:28:42.396389+0000","last_unstale":"2024-03-09T15:36:28.245845+0000","last_undegraded":"2024-03-09T15:36:28.245845+0000","last_fullsized":"2024-03-09T15:36:28.245845+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'82","last_scrub_stamp":"2024-03-09T15:36:28.245628+0000","last_deep_scrub":"261'82","last_deep_scrub_stamp":"2024-03-09T15:36:28.245628+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.245628+0000","objects_scrubbed":1,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:26:51.961711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":65,"num_write":200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"261'44","reported_seq":591,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.354458+0000","last_change":"2024-03-09T15:36:46.354523+0000","last_active":"2024-03-09T15:36:46.354458+0000","last_peered":"2024-03-09T15:36:46.354458+0000","last_clean":"2024-03-09T15:36:46.354458+0000","last_became_active":"2024-03-09T14:28:34.297080+0000","last_became_peered":"2024-03-09T14:28:34.297080+0000","last_unstale":"2024-03-09T15:36:46.354458+0000","last_undegraded":"2024-03-09T15:36:46.354458+0000","last_fullsized":"2024-03-09T15:36:46.354458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:46.354264+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:46.354264+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.354264+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:33:23.149348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":189,"num_read_kb":155,"num_write":63,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"261'152","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.200259+0000","last_change":"2024-03-09T15:36:20.200259+0000","last_active":"2024-03-09T15:36:20.200259+0000","last_peered":"2024-03-09T15:36:20.200259+0000","last_clean":"2024-03-09T15:36:20.200259+0000","last_became_active":"2024-03-09T14:29:06.445573+0000","last_became_peered":"2024-03-09T14:29:06.445573+0000","last_unstale":"2024-03-09T15:36:20.200259+0000","last_undegraded":"2024-03-09T15:36:20.200259+0000","last_fullsized":"2024-03-09T15:36:20.200259+0000","mapping_epoch":33,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'152","last_scrub_stamp":"2024-03-09T15:36:20.199970+0000","last_deep_scrub":"261'152","last_deep_scrub_stamp":"2024-03-09T15:36:20.199970+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.199970+0000","objects_scrubbed":5,"log_size":52,"log_dups_size":100,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:59:48.356486+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":204,"num_read_kb":157,"num_write":458,"num_write_kb":32761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.186224+0000","last_change":"2024-03-09T15:36:19.186332+0000","last_active":"2024-03-09T15:36:19.186224+0000","last_peered":"2024-03-09T15:36:19.186224+0000","last_clean":"2024-03-09T15:36:19.186224+0000","last_became_active":"2024-03-09T14:28:36.322638+0000","last_became_peered":"2024-03-09T14:28:36.322638+0000","last_unstale":"2024-03-09T15:36:19.186224+0000","last_undegraded":"2024-03-09T15:36:19.186224+0000","last_fullsized":"2024-03-09T15:36:19.186224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:19.185963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:19.185963+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.185963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:52:57.100559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"261'323","reported_seq":975,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:55.211712+0000","last_change":"2024-03-09T15:36:55.211901+0000","last_active":"2024-03-09T15:36:55.211712+0000","last_peered":"2024-03-09T15:36:55.211712+0000","last_clean":"2024-03-09T15:36:55.211712+0000","last_became_active":"2024-03-09T14:28:38.328897+0000","last_became_peered":"2024-03-09T14:28:38.328897+0000","last_unstale":"2024-03-09T15:36:55.211712+0000","last_undegraded":"2024-03-09T15:36:55.211712+0000","last_fullsized":"2024-03-09T15:36:55.211712+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'323","last_scrub_stamp":"2024-03-09T15:36:55.211462+0000","last_deep_scrub":"261'323","last_deep_scrub_stamp":"2024-03-09T15:36:55.211462+0000","last_clean_scrub_stamp":"2024-03-09T15:36:55.211462+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":300,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:45:05.244251+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25167,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":543,"num_read_kb":1162,"num_write":388,"num_write_kb":277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:12.192323+0000","last_change":"2024-03-09T15:36:12.192350+0000","last_active":"2024-03-09T15:36:12.192323+0000","last_peered":"2024-03-09T15:36:12.192323+0000","last_clean":"2024-03-09T15:36:12.192323+0000","last_became_active":"2024-03-09T14:28:40.383758+0000","last_became_peered":"2024-03-09T14:28:40.383758+0000","last_unstale":"2024-03-09T15:36:12.192323+0000","last_undegraded":"2024-03-09T15:36:12.192323+0000","last_fullsized":"2024-03-09T15:36:12.192323+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:12.192210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:12.192210+0000","last_clean_scrub_stamp":"2024-03-09T15:36:12.192210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:18:21.362249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"261'56","reported_seq":576,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.525835+0000","last_change":"2024-03-09T15:36:19.525925+0000","last_active":"2024-03-09T15:36:19.525835+0000","last_peered":"2024-03-09T15:36:19.525835+0000","last_clean":"2024-03-09T15:36:19.525835+0000","last_became_active":"2024-03-09T14:28:42.401880+0000","last_became_peered":"2024-03-09T14:28:42.401880+0000","last_unstale":"2024-03-09T15:36:19.525835+0000","last_undegraded":"2024-03-09T15:36:19.525835+0000","last_fullsized":"2024-03-09T15:36:19.525835+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:19.525622+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:19.525622+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.525622+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:58:32.566894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":69,"num_write":120,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"261'81","reported_seq":650,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.131661+0000","last_change":"2024-03-09T15:36:31.131726+0000","last_active":"2024-03-09T15:36:31.131661+0000","last_peered":"2024-03-09T15:36:31.131661+0000","last_clean":"2024-03-09T15:36:31.131661+0000","last_became_active":"2024-03-09T14:28:34.315432+0000","last_became_peered":"2024-03-09T14:28:34.315432+0000","last_unstale":"2024-03-09T15:36:31.131661+0000","last_undegraded":"2024-03-09T15:36:31.131661+0000","last_fullsized":"2024-03-09T15:36:31.131661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'81","last_scrub_stamp":"2024-03-09T15:36:31.131473+0000","last_deep_scrub":"261'81","last_deep_scrub_stamp":"2024-03-09T15:36:31.131473+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.131473+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:42:11.454205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":315,"num_read_kb":251,"num_write":126,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":466,"num_omap_keys":2,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"261'163","reported_seq":710,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:02.770414+0000","last_change":"2024-03-09T15:37:02.770536+0000","last_active":"2024-03-09T15:37:02.770414+0000","last_peered":"2024-03-09T15:37:02.770414+0000","last_clean":"2024-03-09T15:37:02.770414+0000","last_became_active":"2024-03-09T14:28:32.299897+0000","last_became_peered":"2024-03-09T14:28:32.299897+0000","last_unstale":"2024-03-09T15:37:02.770414+0000","last_undegraded":"2024-03-09T15:37:02.770414+0000","last_fullsized":"2024-03-09T15:37:02.770414+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'163","last_scrub_stamp":"2024-03-09T15:37:02.770220+0000","last_deep_scrub":"261'163","last_deep_scrub_stamp":"2024-03-09T15:37:02.770220+0000","last_clean_scrub_stamp":"2024-03-09T15:37:02.770220+0000","objects_scrubbed":6,"log_size":63,"log_dups_size":100,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:31:20.313548+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5506226,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":160,"num_read_kb":225,"num_write":553,"num_write_kb":26448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":515,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.168556+0000","last_change":"2024-03-09T15:36:21.168616+0000","last_active":"2024-03-09T15:36:21.168556+0000","last_peered":"2024-03-09T15:36:21.168556+0000","last_clean":"2024-03-09T15:36:21.168556+0000","last_became_active":"2024-03-09T14:28:36.320029+0000","last_became_peered":"2024-03-09T14:28:36.320029+0000","last_unstale":"2024-03-09T15:36:21.168556+0000","last_undegraded":"2024-03-09T15:36:21.168556+0000","last_fullsized":"2024-03-09T15:36:21.168556+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.168322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.168322+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.168322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:54:01.071807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"261'213","reported_seq":814,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.154985+0000","last_change":"2024-03-09T15:36:28.155065+0000","last_active":"2024-03-09T15:36:28.154985+0000","last_peered":"2024-03-09T15:36:28.154985+0000","last_clean":"2024-03-09T15:36:28.154985+0000","last_became_active":"2024-03-09T14:28:38.338658+0000","last_became_peered":"2024-03-09T14:28:38.338658+0000","last_unstale":"2024-03-09T15:36:28.154985+0000","last_undegraded":"2024-03-09T15:36:28.154985+0000","last_fullsized":"2024-03-09T15:36:28.154985+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'213","last_scrub_stamp":"2024-03-09T15:36:28.154798+0000","last_deep_scrub":"261'213","last_deep_scrub_stamp":"2024-03-09T15:36:28.154798+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.154798+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":200,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:48:17.369213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12662,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":352,"num_read_kb":613,"num_write":292,"num_write_kb":188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13964,"num_omap_keys":34,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.231293+0000","last_change":"2024-03-09T15:36:30.231399+0000","last_active":"2024-03-09T15:36:30.231293+0000","last_peered":"2024-03-09T15:36:30.231293+0000","last_clean":"2024-03-09T15:36:30.231293+0000","last_became_active":"2024-03-09T14:28:40.378973+0000","last_became_peered":"2024-03-09T14:28:40.378973+0000","last_unstale":"2024-03-09T15:36:30.231293+0000","last_undegraded":"2024-03-09T15:36:30.231293+0000","last_fullsized":"2024-03-09T15:36:30.231293+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:30.231002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:30.231002+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.231002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:08:44.672009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"261'44","reported_seq":547,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.449669+0000","last_change":"2024-03-09T15:36:23.449669+0000","last_active":"2024-03-09T15:36:23.449669+0000","last_peered":"2024-03-09T15:36:23.449669+0000","last_clean":"2024-03-09T15:36:23.449669+0000","last_became_active":"2024-03-09T14:28:42.397533+0000","last_became_peered":"2024-03-09T14:28:42.397533+0000","last_unstale":"2024-03-09T15:36:23.449669+0000","last_undegraded":"2024-03-09T15:36:23.449669+0000","last_fullsized":"2024-03-09T15:36:23.449669+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'44","last_scrub_stamp":"2024-03-09T15:36:23.449413+0000","last_deep_scrub":"261'44","last_deep_scrub_stamp":"2024-03-09T15:36:23.449413+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.449413+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:46:36.909599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":91,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"261'56","reported_seq":625,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.166995+0000","last_change":"2024-03-09T15:36:34.167057+0000","last_active":"2024-03-09T15:36:34.166995+0000","last_peered":"2024-03-09T15:36:34.166995+0000","last_clean":"2024-03-09T15:36:34.166995+0000","last_became_active":"2024-03-09T14:28:34.313475+0000","last_became_peered":"2024-03-09T14:28:34.313475+0000","last_unstale":"2024-03-09T15:36:34.166995+0000","last_undegraded":"2024-03-09T15:36:34.166995+0000","last_fullsized":"2024-03-09T15:36:34.166995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:34.166804+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:34.166804+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.166804+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:15:44.814256+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":198,"num_write":78,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"261'136","reported_seq":697,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.314272+0000","last_change":"2024-03-09T15:36:37.314511+0000","last_active":"2024-03-09T15:36:37.314272+0000","last_peered":"2024-03-09T15:36:37.314272+0000","last_clean":"2024-03-09T15:36:37.314272+0000","last_became_active":"2024-03-09T14:28:32.316406+0000","last_became_peered":"2024-03-09T14:28:32.316406+0000","last_unstale":"2024-03-09T15:36:37.314272+0000","last_undegraded":"2024-03-09T15:36:37.314272+0000","last_fullsized":"2024-03-09T15:36:37.314272+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'136","last_scrub_stamp":"2024-03-09T15:36:37.314063+0000","last_deep_scrub":"261'136","last_deep_scrub_stamp":"2024-03-09T15:36:37.314063+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.314063+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:50:27.588191+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655835,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":162,"num_read_kb":15736,"num_write":466,"num_write_kb":13578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.511704+0000","last_change":"2024-03-09T15:36:27.511852+0000","last_active":"2024-03-09T15:36:27.511704+0000","last_peered":"2024-03-09T15:36:27.511704+0000","last_clean":"2024-03-09T15:36:27.511704+0000","last_became_active":"2024-03-09T14:28:36.319210+0000","last_became_peered":"2024-03-09T14:28:36.319210+0000","last_unstale":"2024-03-09T15:36:27.511704+0000","last_undegraded":"2024-03-09T15:36:27.511704+0000","last_fullsized":"2024-03-09T15:36:27.511704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:27.511458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:27.511458+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.511458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:16:07.969688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"261'223","reported_seq":837,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.196282+0000","last_change":"2024-03-09T15:36:15.196282+0000","last_active":"2024-03-09T15:36:15.196282+0000","last_peered":"2024-03-09T15:36:15.196282+0000","last_clean":"2024-03-09T15:36:15.196282+0000","last_became_active":"2024-03-09T14:28:38.350390+0000","last_became_peered":"2024-03-09T14:28:38.350390+0000","last_unstale":"2024-03-09T15:36:15.196282+0000","last_undegraded":"2024-03-09T15:36:15.196282+0000","last_fullsized":"2024-03-09T15:36:15.196282+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'223","last_scrub_stamp":"2024-03-09T15:36:15.195858+0000","last_deep_scrub":"261'223","last_deep_scrub_stamp":"2024-03-09T15:36:15.195858+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.195858+0000","objects_scrubbed":7,"log_size":23,"log_dups_size":200,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:47:25.524775+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9540,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":338,"num_read_kb":580,"num_write":230,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"261'474","reported_seq":2368,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.200778+0000","last_change":"2024-03-09T15:36:13.200827+0000","last_active":"2024-03-09T15:36:13.200778+0000","last_peered":"2024-03-09T15:36:13.200778+0000","last_clean":"2024-03-09T15:36:13.200778+0000","last_became_active":"2024-03-09T14:28:40.373360+0000","last_became_peered":"2024-03-09T14:28:40.373360+0000","last_unstale":"2024-03-09T15:36:13.200778+0000","last_undegraded":"2024-03-09T15:36:13.200778+0000","last_fullsized":"2024-03-09T15:36:13.200778+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'474","last_scrub_stamp":"2024-03-09T15:36:13.200687+0000","last_deep_scrub":"261'474","last_deep_scrub_stamp":"2024-03-09T15:36:13.200687+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.200687+0000","objects_scrubbed":1,"log_size":74,"log_dups_size":400,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:36:38.570841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"261'108","reported_seq":664,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.587396+0000","last_change":"2024-03-09T15:36:16.587396+0000","last_active":"2024-03-09T15:36:16.587396+0000","last_peered":"2024-03-09T15:36:16.587396+0000","last_clean":"2024-03-09T15:36:16.587396+0000","last_became_active":"2024-03-09T14:28:42.408233+0000","last_became_peered":"2024-03-09T14:28:42.408233+0000","last_unstale":"2024-03-09T15:36:16.587396+0000","last_undegraded":"2024-03-09T15:36:16.587396+0000","last_fullsized":"2024-03-09T15:36:16.587396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'108","last_scrub_stamp":"2024-03-09T15:36:16.587096+0000","last_deep_scrub":"261'108","last_deep_scrub_stamp":"2024-03-09T15:36:16.587096+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.587096+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:31:09.055955+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":274,"num_read_kb":227,"num_write":264,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"261'72","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.342615+0000","last_change":"2024-03-09T15:36:44.342675+0000","last_active":"2024-03-09T15:36:44.342615+0000","last_peered":"2024-03-09T15:36:44.342615+0000","last_clean":"2024-03-09T15:36:44.342615+0000","last_became_active":"2024-03-09T14:28:34.310061+0000","last_became_peered":"2024-03-09T14:28:34.310061+0000","last_unstale":"2024-03-09T15:36:44.342615+0000","last_undegraded":"2024-03-09T15:36:44.342615+0000","last_fullsized":"2024-03-09T15:36:44.342615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-09T15:36:44.342422+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-09T15:36:44.342422+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.342422+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:25:32.650543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":294,"num_read_kb":233,"num_write":114,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"261'174","reported_seq":790,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.437240+0000","last_change":"2024-03-09T15:36:48.437332+0000","last_active":"2024-03-09T15:36:48.437240+0000","last_peered":"2024-03-09T15:36:48.437240+0000","last_clean":"2024-03-09T15:36:48.437240+0000","last_became_active":"2024-03-09T14:28:32.313566+0000","last_became_peered":"2024-03-09T14:28:32.313566+0000","last_unstale":"2024-03-09T15:36:48.437240+0000","last_undegraded":"2024-03-09T15:36:48.437240+0000","last_fullsized":"2024-03-09T15:36:48.437240+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'174","last_scrub_stamp":"2024-03-09T15:36:48.437063+0000","last_deep_scrub":"261'174","last_deep_scrub_stamp":"2024-03-09T15:36:48.437063+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.437063+0000","objects_scrubbed":8,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:03:36.289305+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":245,"num_read_kb":84880,"num_write":567,"num_write_kb":19718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"261'10","reported_seq":1472,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.904683+0000","last_change":"2024-03-09T15:36:42.904753+0000","last_active":"2024-03-09T15:36:42.904683+0000","last_peered":"2024-03-09T15:36:42.904683+0000","last_clean":"2024-03-09T15:36:42.904683+0000","last_became_active":"2024-03-09T14:28:36.325820+0000","last_became_peered":"2024-03-09T14:28:36.325820+0000","last_unstale":"2024-03-09T15:36:42.904683+0000","last_undegraded":"2024-03-09T15:36:42.904683+0000","last_fullsized":"2024-03-09T15:36:42.904683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'10","last_scrub_stamp":"2024-03-09T15:36:42.904497+0000","last_deep_scrub":"261'10","last_deep_scrub_stamp":"2024-03-09T15:36:42.904497+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.904497+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:01:23.989269+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1195,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1118,"num_read_kb":1421,"num_write":14,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"261'91","reported_seq":630,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.533379+0000","last_change":"2024-03-09T15:36:38.533460+0000","last_active":"2024-03-09T15:36:38.533379+0000","last_peered":"2024-03-09T15:36:38.533379+0000","last_clean":"2024-03-09T15:36:38.533379+0000","last_became_active":"2024-03-09T14:28:38.595478+0000","last_became_peered":"2024-03-09T14:28:38.595478+0000","last_unstale":"2024-03-09T15:36:38.533379+0000","last_undegraded":"2024-03-09T15:36:38.533379+0000","last_fullsized":"2024-03-09T15:36:38.533379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'91","last_scrub_stamp":"2024-03-09T15:36:38.533177+0000","last_deep_scrub":"261'91","last_deep_scrub_stamp":"2024-03-09T15:36:38.533177+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.533177+0000","objects_scrubbed":5,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:59:32.253642+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":105,"num_read_kb":100,"num_write":70,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"261'489","reported_seq":2304,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.167232+0000","last_change":"2024-03-09T15:36:26.167298+0000","last_active":"2024-03-09T15:36:26.167232+0000","last_peered":"2024-03-09T15:36:26.167232+0000","last_clean":"2024-03-09T15:36:26.167232+0000","last_became_active":"2024-03-09T14:28:40.374408+0000","last_became_peered":"2024-03-09T14:28:40.374408+0000","last_unstale":"2024-03-09T15:36:26.167232+0000","last_undegraded":"2024-03-09T15:36:26.167232+0000","last_fullsized":"2024-03-09T15:36:26.167232+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'489","last_scrub_stamp":"2024-03-09T15:36:26.166904+0000","last_deep_scrub":"261'489","last_deep_scrub_stamp":"2024-03-09T15:36:26.166904+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.166904+0000","objects_scrubbed":1,"log_size":89,"log_dups_size":400,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:49:54.207124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"261'94","reported_seq":607,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:16.145423+0000","last_change":"2024-03-09T15:36:16.145423+0000","last_active":"2024-03-09T15:36:16.145423+0000","last_peered":"2024-03-09T15:36:16.145423+0000","last_clean":"2024-03-09T15:36:16.145423+0000","last_became_active":"2024-03-09T14:28:42.411523+0000","last_became_peered":"2024-03-09T14:28:42.411523+0000","last_unstale":"2024-03-09T15:36:16.145423+0000","last_undegraded":"2024-03-09T15:36:16.145423+0000","last_fullsized":"2024-03-09T15:36:16.145423+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'94","last_scrub_stamp":"2024-03-09T15:36:16.145166+0000","last_deep_scrub":"261'94","last_deep_scrub_stamp":"2024-03-09T15:36:16.145166+0000","last_clean_scrub_stamp":"2024-03-09T15:36:16.145166+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:56:22.170750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":72,"num_write":221,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"261'42","reported_seq":605,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.506243+0000","last_change":"2024-03-09T15:36:28.506314+0000","last_active":"2024-03-09T15:36:28.506243+0000","last_peered":"2024-03-09T15:36:28.506243+0000","last_clean":"2024-03-09T15:36:28.506243+0000","last_became_active":"2024-03-09T14:28:34.317226+0000","last_became_peered":"2024-03-09T14:28:34.317226+0000","last_unstale":"2024-03-09T15:36:28.506243+0000","last_undegraded":"2024-03-09T15:36:28.506243+0000","last_fullsized":"2024-03-09T15:36:28.506243+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'42","last_scrub_stamp":"2024-03-09T15:36:28.505993+0000","last_deep_scrub":"261'42","last_deep_scrub_stamp":"2024-03-09T15:36:28.505993+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.505993+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:14:19.130657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":188,"num_read_kb":158,"num_write":62,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"261'119","reported_seq":664,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:59.715451+0000","last_change":"2024-03-09T15:36:59.715624+0000","last_active":"2024-03-09T15:36:59.715451+0000","last_peered":"2024-03-09T15:36:59.715451+0000","last_clean":"2024-03-09T15:36:59.715451+0000","last_became_active":"2024-03-09T14:28:32.295179+0000","last_became_peered":"2024-03-09T14:28:32.295179+0000","last_unstale":"2024-03-09T15:36:59.715451+0000","last_undegraded":"2024-03-09T15:36:59.715451+0000","last_fullsized":"2024-03-09T15:36:59.715451+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'119","last_scrub_stamp":"2024-03-09T15:36:59.715201+0000","last_deep_scrub":"261'119","last_deep_scrub_stamp":"2024-03-09T15:36:59.715201+0000","last_clean_scrub_stamp":"2024-03-09T15:36:59.715201+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":100,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:06:38.255376+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262159,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":143,"num_read_kb":124,"num_write":390,"num_write_kb":5920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.379645+0000","last_change":"2024-03-09T15:36:45.379697+0000","last_active":"2024-03-09T15:36:45.379645+0000","last_peered":"2024-03-09T15:36:45.379645+0000","last_clean":"2024-03-09T15:36:45.379645+0000","last_became_active":"2024-03-09T14:28:36.319102+0000","last_became_peered":"2024-03-09T14:28:36.319102+0000","last_unstale":"2024-03-09T15:36:45.379645+0000","last_undegraded":"2024-03-09T15:36:45.379645+0000","last_fullsized":"2024-03-09T15:36:45.379645+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:45.379458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:45.379458+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.379458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:10:12.455227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"261'321","reported_seq":958,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.774275+0000","last_change":"2024-03-09T15:36:27.774372+0000","last_active":"2024-03-09T15:36:27.774275+0000","last_peered":"2024-03-09T15:36:27.774275+0000","last_clean":"2024-03-09T15:36:27.774275+0000","last_became_active":"2024-03-09T14:28:38.337254+0000","last_became_peered":"2024-03-09T14:28:38.337254+0000","last_unstale":"2024-03-09T15:36:27.774275+0000","last_undegraded":"2024-03-09T15:36:27.774275+0000","last_fullsized":"2024-03-09T15:36:27.774275+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'321","last_scrub_stamp":"2024-03-09T15:36:27.774086+0000","last_deep_scrub":"261'321","last_deep_scrub_stamp":"2024-03-09T15:36:27.774086+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.774086+0000","objects_scrubbed":9,"log_size":21,"log_dups_size":300,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:34:50.482182+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13936,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":673,"num_read_kb":965,"num_write":788,"num_write_kb":629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":141964,"num_omap_keys":356,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.219910+0000","last_change":"2024-03-09T15:36:30.219963+0000","last_active":"2024-03-09T15:36:30.219910+0000","last_peered":"2024-03-09T15:36:30.219910+0000","last_clean":"2024-03-09T15:36:30.219910+0000","last_became_active":"2024-03-09T14:28:40.377714+0000","last_became_peered":"2024-03-09T14:28:40.377714+0000","last_unstale":"2024-03-09T15:36:30.219910+0000","last_undegraded":"2024-03-09T15:36:30.219910+0000","last_fullsized":"2024-03-09T15:36:30.219910+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:30.219708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:30.219708+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.219708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:30:50.699663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"261'69","reported_seq":578,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.545264+0000","last_change":"2024-03-09T15:36:20.545264+0000","last_active":"2024-03-09T15:36:20.545264+0000","last_peered":"2024-03-09T15:36:20.545264+0000","last_clean":"2024-03-09T15:36:20.545264+0000","last_became_active":"2024-03-09T14:28:42.401775+0000","last_became_peered":"2024-03-09T14:28:42.401775+0000","last_unstale":"2024-03-09T15:36:20.545264+0000","last_undegraded":"2024-03-09T15:36:20.545264+0000","last_fullsized":"2024-03-09T15:36:20.545264+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'69","last_scrub_stamp":"2024-03-09T15:36:20.545001+0000","last_deep_scrub":"261'69","last_deep_scrub_stamp":"2024-03-09T15:36:20.545001+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.545001+0000","objects_scrubbed":1,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:08:40.010383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":77,"num_write":153,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"261'174","reported_seq":834,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.140904+0000","last_change":"2024-03-09T15:36:27.140963+0000","last_active":"2024-03-09T15:36:27.140904+0000","last_peered":"2024-03-09T15:36:27.140904+0000","last_clean":"2024-03-09T15:36:27.140904+0000","last_became_active":"2024-03-09T14:28:34.326919+0000","last_became_peered":"2024-03-09T14:28:34.326919+0000","last_unstale":"2024-03-09T15:36:27.140904+0000","last_undegraded":"2024-03-09T15:36:27.140904+0000","last_fullsized":"2024-03-09T15:36:27.140904+0000","mapping_epoch":22,"log_start":"34'100","ondisk_log_start":"34'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'174","last_scrub_stamp":"2024-03-09T15:36:27.140715+0000","last_deep_scrub":"261'174","last_deep_scrub_stamp":"2024-03-09T15:36:27.140715+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.140715+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":100,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:06:14.462396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":914,"num_read_kb":819,"num_write":300,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"261'119","reported_seq":693,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:55.122472+0000","last_change":"2024-03-09T15:36:55.122551+0000","last_active":"2024-03-09T15:36:55.122472+0000","last_peered":"2024-03-09T15:36:55.122472+0000","last_clean":"2024-03-09T15:36:55.122472+0000","last_became_active":"2024-03-09T14:28:12.262430+0000","last_became_peered":"2024-03-09T14:28:12.262430+0000","last_unstale":"2024-03-09T15:36:55.122472+0000","last_undegraded":"2024-03-09T15:36:55.122472+0000","last_fullsized":"2024-03-09T15:36:55.122472+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'119","last_scrub_stamp":"2024-03-09T15:36:55.122277+0000","last_deep_scrub":"261'119","last_deep_scrub_stamp":"2024-03-09T15:36:55.122277+0000","last_clean_scrub_stamp":"2024-03-09T15:36:55.122277+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":100,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:38:23.689863+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":532,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.969844+0000","last_change":"2024-03-09T15:36:44.969844+0000","last_active":"2024-03-09T15:36:44.969844+0000","last_peered":"2024-03-09T15:36:44.969844+0000","last_clean":"2024-03-09T15:36:44.969844+0000","last_became_active":"2024-03-09T14:28:19.232517+0000","last_became_peered":"2024-03-09T14:28:19.232517+0000","last_unstale":"2024-03-09T15:36:44.969844+0000","last_undegraded":"2024-03-09T15:36:44.969844+0000","last_fullsized":"2024-03-09T15:36:44.969844+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:44.969581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:44.969581+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.969581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:43:52.001009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"261'134","reported_seq":703,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:03.797620+0000","last_change":"2024-03-09T15:37:03.797620+0000","last_active":"2024-03-09T15:37:03.797620+0000","last_peered":"2024-03-09T15:37:03.797620+0000","last_clean":"2024-03-09T15:37:03.797620+0000","last_became_active":"2024-03-09T14:28:32.300468+0000","last_became_peered":"2024-03-09T14:28:32.300468+0000","last_unstale":"2024-03-09T15:37:03.797620+0000","last_undegraded":"2024-03-09T15:37:03.797620+0000","last_fullsized":"2024-03-09T15:37:03.797620+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'134","last_scrub_stamp":"2024-03-09T15:37:03.797338+0000","last_deep_scrub":"261'134","last_deep_scrub_stamp":"2024-03-09T15:37:03.797338+0000","last_clean_scrub_stamp":"2024-03-09T15:37:03.797338+0000","objects_scrubbed":4,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:05:05.538076+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311644,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":173,"num_read_kb":6450,"num_write":457,"num_write_kb":21101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":520,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.163783+0000","last_change":"2024-03-09T15:36:41.163783+0000","last_active":"2024-03-09T15:36:41.163783+0000","last_peered":"2024-03-09T15:36:41.163783+0000","last_clean":"2024-03-09T15:36:41.163783+0000","last_became_active":"2024-03-09T14:28:36.318172+0000","last_became_peered":"2024-03-09T14:28:36.318172+0000","last_unstale":"2024-03-09T15:36:41.163783+0000","last_undegraded":"2024-03-09T15:36:41.163783+0000","last_fullsized":"2024-03-09T15:36:41.163783+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:41.163438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:41.163438+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.163438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:55:32.987190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"261'502","reported_seq":1226,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.737856+0000","last_change":"2024-03-09T15:36:25.737931+0000","last_active":"2024-03-09T15:36:25.737856+0000","last_peered":"2024-03-09T15:36:25.737856+0000","last_clean":"2024-03-09T15:36:25.737856+0000","last_became_active":"2024-03-09T14:28:38.348941+0000","last_became_peered":"2024-03-09T14:28:38.348941+0000","last_unstale":"2024-03-09T15:36:25.737856+0000","last_undegraded":"2024-03-09T15:36:25.737856+0000","last_fullsized":"2024-03-09T15:36:25.737856+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'502","last_scrub_stamp":"2024-03-09T15:36:25.737665+0000","last_deep_scrub":"261'502","last_deep_scrub_stamp":"2024-03-09T15:36:25.737665+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.737665+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":400,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:22:57.116693+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37400,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":987,"num_read_kb":1910,"num_write":934,"num_write_kb":767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":90684,"num_omap_keys":220,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.239905+0000","last_change":"2024-03-09T15:36:37.239957+0000","last_active":"2024-03-09T15:36:37.239905+0000","last_peered":"2024-03-09T15:36:37.239905+0000","last_clean":"2024-03-09T15:36:37.239905+0000","last_became_active":"2024-03-09T14:28:40.362781+0000","last_became_peered":"2024-03-09T14:28:40.362781+0000","last_unstale":"2024-03-09T15:36:37.239905+0000","last_undegraded":"2024-03-09T15:36:37.239905+0000","last_fullsized":"2024-03-09T15:36:37.239905+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:37.239705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:37.239705+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.239705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:28.768337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"261'121","reported_seq":829,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.279007+0000","last_change":"2024-03-09T15:36:44.279073+0000","last_active":"2024-03-09T15:36:44.279007+0000","last_peered":"2024-03-09T15:36:44.279007+0000","last_clean":"2024-03-09T15:36:44.279007+0000","last_became_active":"2024-03-09T14:28:42.398626+0000","last_became_peered":"2024-03-09T14:28:42.398626+0000","last_unstale":"2024-03-09T15:36:44.279007+0000","last_undegraded":"2024-03-09T15:36:44.279007+0000","last_fullsized":"2024-03-09T15:36:44.279007+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'121","last_scrub_stamp":"2024-03-09T15:36:44.278808+0000","last_deep_scrub":"261'121","last_deep_scrub_stamp":"2024-03-09T15:36:44.278808+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.278808+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":100,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:44:17.595367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":356,"num_read_kb":343,"num_write":215,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":66,"num_omap_keys":1,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"261'90","reported_seq":679,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.266824+0000","last_change":"2024-03-09T15:36:50.266889+0000","last_active":"2024-03-09T15:36:50.266824+0000","last_peered":"2024-03-09T15:36:50.266824+0000","last_clean":"2024-03-09T15:36:50.266824+0000","last_became_active":"2024-03-09T14:28:34.297472+0000","last_became_peered":"2024-03-09T14:28:34.297472+0000","last_unstale":"2024-03-09T15:36:50.266824+0000","last_undegraded":"2024-03-09T15:36:50.266824+0000","last_fullsized":"2024-03-09T15:36:50.266824+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'90","last_scrub_stamp":"2024-03-09T15:36:50.266621+0000","last_deep_scrub":"261'90","last_deep_scrub_stamp":"2024-03-09T15:36:50.266621+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.266621+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:34:30.319896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":356,"num_read_kb":292,"num_write":141,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":535,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:57.107045+0000","last_change":"2024-03-09T15:36:57.107099+0000","last_active":"2024-03-09T15:36:57.107045+0000","last_peered":"2024-03-09T15:36:57.107045+0000","last_clean":"2024-03-09T15:36:57.107045+0000","last_became_active":"2024-03-09T14:28:19.241110+0000","last_became_peered":"2024-03-09T14:28:19.241110+0000","last_unstale":"2024-03-09T15:36:57.107045+0000","last_undegraded":"2024-03-09T15:36:57.107045+0000","last_fullsized":"2024-03-09T15:36:57.107045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:57.106845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:57.106845+0000","last_clean_scrub_stamp":"2024-03-09T15:36:57.106845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:15:58.886485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"261'139","reported_seq":721,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:54.246715+0000","last_change":"2024-03-09T15:36:54.246715+0000","last_active":"2024-03-09T15:36:54.246715+0000","last_peered":"2024-03-09T15:36:54.246715+0000","last_clean":"2024-03-09T15:36:54.246715+0000","last_became_active":"2024-03-09T14:28:32.312536+0000","last_became_peered":"2024-03-09T14:28:32.312536+0000","last_unstale":"2024-03-09T15:36:54.246715+0000","last_undegraded":"2024-03-09T15:36:54.246715+0000","last_fullsized":"2024-03-09T15:36:54.246715+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'139","last_scrub_stamp":"2024-03-09T15:36:54.246441+0000","last_deep_scrub":"261'139","last_deep_scrub_stamp":"2024-03-09T15:36:54.246441+0000","last_clean_scrub_stamp":"2024-03-09T15:36:54.246441+0000","objects_scrubbed":1,"log_size":39,"log_dups_size":100,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:59:59.656356+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":219,"num_read_kb":12489,"num_write":525,"num_write_kb":17953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.821324+0000","last_change":"2024-03-09T15:36:36.821377+0000","last_active":"2024-03-09T15:36:36.821324+0000","last_peered":"2024-03-09T15:36:36.821324+0000","last_clean":"2024-03-09T15:36:36.821324+0000","last_became_active":"2024-03-09T14:28:36.306912+0000","last_became_peered":"2024-03-09T14:28:36.306912+0000","last_unstale":"2024-03-09T15:36:36.821324+0000","last_undegraded":"2024-03-09T15:36:36.821324+0000","last_fullsized":"2024-03-09T15:36:36.821324+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:36.821128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:36.821128+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.821128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:16:15.465239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"261'854","reported_seq":2541,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.493336+0000","last_change":"2024-03-09T15:36:31.493336+0000","last_active":"2024-03-09T15:36:31.493336+0000","last_peered":"2024-03-09T15:36:31.493336+0000","last_clean":"2024-03-09T15:36:31.493336+0000","last_became_active":"2024-03-09T14:28:38.328738+0000","last_became_peered":"2024-03-09T14:28:38.328738+0000","last_unstale":"2024-03-09T15:36:31.493336+0000","last_undegraded":"2024-03-09T15:36:31.493336+0000","last_fullsized":"2024-03-09T15:36:31.493336+0000","mapping_epoch":26,"log_start":"261'800","ondisk_log_start":"261'800","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'854","last_scrub_stamp":"2024-03-09T15:36:31.493032+0000","last_deep_scrub":"261'854","last_deep_scrub_stamp":"2024-03-09T15:36:31.493032+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.493032+0000","objects_scrubbed":11,"log_size":54,"log_dups_size":800,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:44:58.935541+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34683,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1323,"num_read_kb":2148,"num_write":662,"num_write_kb":462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5354,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":511,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.192845+0000","last_change":"2024-03-09T15:36:32.192899+0000","last_active":"2024-03-09T15:36:32.192845+0000","last_peered":"2024-03-09T15:36:32.192845+0000","last_clean":"2024-03-09T15:36:32.192845+0000","last_became_active":"2024-03-09T14:28:40.386451+0000","last_became_peered":"2024-03-09T14:28:40.386451+0000","last_unstale":"2024-03-09T15:36:32.192845+0000","last_undegraded":"2024-03-09T15:36:32.192845+0000","last_fullsized":"2024-03-09T15:36:32.192845+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:32.192627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:32.192627+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.192627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:09:46.269539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"261'129","reported_seq":670,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.567045+0000","last_change":"2024-03-09T15:36:18.567214+0000","last_active":"2024-03-09T15:36:18.567045+0000","last_peered":"2024-03-09T15:36:18.567045+0000","last_clean":"2024-03-09T15:36:18.567045+0000","last_became_active":"2024-03-09T14:28:42.403013+0000","last_became_peered":"2024-03-09T14:28:42.403013+0000","last_unstale":"2024-03-09T15:36:18.567045+0000","last_undegraded":"2024-03-09T15:36:18.567045+0000","last_fullsized":"2024-03-09T15:36:18.567045+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'129","last_scrub_stamp":"2024-03-09T15:36:18.566788+0000","last_deep_scrub":"261'129","last_deep_scrub_stamp":"2024-03-09T15:36:18.566788+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.566788+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":100,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:45:17.069116+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1248,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":261,"num_read_kb":241,"num_write":300,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"261'48","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.363144+0000","last_change":"2024-03-09T15:36:38.363206+0000","last_active":"2024-03-09T15:36:38.363144+0000","last_peered":"2024-03-09T15:36:38.363144+0000","last_clean":"2024-03-09T15:36:38.363144+0000","last_became_active":"2024-03-09T14:28:34.296733+0000","last_became_peered":"2024-03-09T14:28:34.296733+0000","last_unstale":"2024-03-09T15:36:38.363144+0000","last_undegraded":"2024-03-09T15:36:38.363144+0000","last_fullsized":"2024-03-09T15:36:38.363144+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:38.362948+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:38.362948+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.362948+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:32:01.096416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":200,"num_read_kb":166,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":532,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.234318+0000","last_change":"2024-03-09T15:36:28.234318+0000","last_active":"2024-03-09T15:36:28.234318+0000","last_peered":"2024-03-09T15:36:28.234318+0000","last_clean":"2024-03-09T15:36:28.234318+0000","last_became_active":"2024-03-09T14:28:19.436985+0000","last_became_peered":"2024-03-09T14:28:19.436985+0000","last_unstale":"2024-03-09T15:36:28.234318+0000","last_undegraded":"2024-03-09T15:36:28.234318+0000","last_fullsized":"2024-03-09T15:36:28.234318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:28.234071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:28.234071+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.234071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:27:16.042052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"261'159","reported_seq":826,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.221405+0000","last_change":"2024-03-09T15:36:43.221405+0000","last_active":"2024-03-09T15:36:43.221405+0000","last_peered":"2024-03-09T15:36:43.221405+0000","last_clean":"2024-03-09T15:36:43.221405+0000","last_became_active":"2024-03-09T14:28:32.303176+0000","last_became_peered":"2024-03-09T14:28:32.303176+0000","last_unstale":"2024-03-09T15:36:43.221405+0000","last_undegraded":"2024-03-09T15:36:43.221405+0000","last_fullsized":"2024-03-09T15:36:43.221405+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'159","last_scrub_stamp":"2024-03-09T15:36:43.221036+0000","last_deep_scrub":"261'159","last_deep_scrub_stamp":"2024-03-09T15:36:43.221036+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.221036+0000","objects_scrubbed":7,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:18:01.135873+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8554724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":486,"num_read_kb":3616,"num_write":592,"num_write_kb":29269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"261'2","reported_seq":518,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.203415+0000","last_change":"2024-03-09T15:36:23.203478+0000","last_active":"2024-03-09T15:36:23.203415+0000","last_peered":"2024-03-09T15:36:23.203415+0000","last_clean":"2024-03-09T15:36:23.203415+0000","last_became_active":"2024-03-09T14:28:36.305971+0000","last_became_peered":"2024-03-09T14:28:36.305971+0000","last_unstale":"2024-03-09T15:36:23.203415+0000","last_undegraded":"2024-03-09T15:36:23.203415+0000","last_fullsized":"2024-03-09T15:36:23.203415+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:23.203217+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:23.203217+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.203217+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:42:55.897785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"261'239","reported_seq":1616,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:53.186831+0000","last_change":"2024-03-09T15:36:53.186919+0000","last_active":"2024-03-09T15:36:53.186831+0000","last_peered":"2024-03-09T15:36:53.186831+0000","last_clean":"2024-03-09T15:36:53.186831+0000","last_became_active":"2024-03-09T14:28:38.597403+0000","last_became_peered":"2024-03-09T14:28:38.597403+0000","last_unstale":"2024-03-09T15:36:53.186831+0000","last_undegraded":"2024-03-09T15:36:53.186831+0000","last_fullsized":"2024-03-09T15:36:53.186831+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'239","last_scrub_stamp":"2024-03-09T15:36:53.186633+0000","last_deep_scrub":"261'239","last_deep_scrub_stamp":"2024-03-09T15:36:53.186633+0000","last_clean_scrub_stamp":"2024-03-09T15:36:53.186633+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":200,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:34:56.366739+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":226,"num_read_kb":216,"num_write":195,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5400,"num_omap_keys":12,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.247884+0000","last_change":"2024-03-09T15:36:32.247884+0000","last_active":"2024-03-09T15:36:32.247884+0000","last_peered":"2024-03-09T15:36:32.247884+0000","last_clean":"2024-03-09T15:36:32.247884+0000","last_became_active":"2024-03-09T14:28:40.371383+0000","last_became_peered":"2024-03-09T14:28:40.371383+0000","last_unstale":"2024-03-09T15:36:32.247884+0000","last_undegraded":"2024-03-09T15:36:32.247884+0000","last_fullsized":"2024-03-09T15:36:32.247884+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:32.247639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:32.247639+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.247639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:52:44.454327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"261'203","reported_seq":803,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.435372+0000","last_change":"2024-03-09T15:36:24.435440+0000","last_active":"2024-03-09T15:36:24.435372+0000","last_peered":"2024-03-09T15:36:24.435372+0000","last_clean":"2024-03-09T15:36:24.435372+0000","last_became_active":"2024-03-09T14:28:42.395781+0000","last_became_peered":"2024-03-09T14:28:42.395781+0000","last_unstale":"2024-03-09T15:36:24.435372+0000","last_undegraded":"2024-03-09T15:36:24.435372+0000","last_fullsized":"2024-03-09T15:36:24.435372+0000","mapping_epoch":30,"log_start":"247'100","ondisk_log_start":"247'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'203","last_scrub_stamp":"2024-03-09T15:36:24.435182+0000","last_deep_scrub":"261'203","last_deep_scrub_stamp":"2024-03-09T15:36:24.435182+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.435182+0000","objects_scrubbed":2,"log_size":103,"log_dups_size":100,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:28.404974+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":435,"num_read_kb":456,"num_write":425,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":851,"num_omap_keys":5,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"261'74","reported_seq":628,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.454571+0000","last_change":"2024-03-09T15:36:32.454571+0000","last_active":"2024-03-09T15:36:32.454571+0000","last_peered":"2024-03-09T15:36:32.454571+0000","last_clean":"2024-03-09T15:36:32.454571+0000","last_became_active":"2024-03-09T14:28:34.313302+0000","last_became_peered":"2024-03-09T14:28:34.313302+0000","last_unstale":"2024-03-09T15:36:32.454571+0000","last_undegraded":"2024-03-09T15:36:32.454571+0000","last_fullsized":"2024-03-09T15:36:32.454571+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'74","last_scrub_stamp":"2024-03-09T15:36:32.454104+0000","last_deep_scrub":"261'74","last_deep_scrub_stamp":"2024-03-09T15:36:32.454104+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.454104+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:47:12.348471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":225,"num_write":104,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":532,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.939088+0000","last_change":"2024-03-09T15:36:45.939088+0000","last_active":"2024-03-09T15:36:45.939088+0000","last_peered":"2024-03-09T15:36:45.939088+0000","last_clean":"2024-03-09T15:36:45.939088+0000","last_became_active":"2024-03-09T14:28:19.237937+0000","last_became_peered":"2024-03-09T14:28:19.237937+0000","last_unstale":"2024-03-09T15:36:45.939088+0000","last_undegraded":"2024-03-09T15:36:45.939088+0000","last_fullsized":"2024-03-09T15:36:45.939088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:45.938842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:45.938842+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.938842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:04:17.332833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"261'130","reported_seq":715,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.409498+0000","last_change":"2024-03-09T15:36:47.409498+0000","last_active":"2024-03-09T15:36:47.409498+0000","last_peered":"2024-03-09T15:36:47.409498+0000","last_clean":"2024-03-09T15:36:47.409498+0000","last_became_active":"2024-03-09T14:28:32.303963+0000","last_became_peered":"2024-03-09T14:28:32.303963+0000","last_unstale":"2024-03-09T15:36:47.409498+0000","last_undegraded":"2024-03-09T15:36:47.409498+0000","last_fullsized":"2024-03-09T15:36:47.409498+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'130","last_scrub_stamp":"2024-03-09T15:36:47.409169+0000","last_deep_scrub":"261'130","last_deep_scrub_stamp":"2024-03-09T15:36:47.409169+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.409169+0000","objects_scrubbed":3,"log_size":30,"log_dups_size":100,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:17:16.816469+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6898850,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":175,"num_read_kb":15115,"num_write":430,"num_write_kb":32085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.253103+0000","last_change":"2024-03-09T15:36:49.253156+0000","last_active":"2024-03-09T15:36:49.253103+0000","last_peered":"2024-03-09T15:36:49.253103+0000","last_clean":"2024-03-09T15:36:49.253103+0000","last_became_active":"2024-03-09T14:28:36.317586+0000","last_became_peered":"2024-03-09T14:28:36.317586+0000","last_unstale":"2024-03-09T15:36:49.253103+0000","last_undegraded":"2024-03-09T15:36:49.253103+0000","last_fullsized":"2024-03-09T15:36:49.253103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:49.252862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:49.252862+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.252862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:35:36.249291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"261'136","reported_seq":698,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:57.805050+0000","last_change":"2024-03-09T15:36:57.805127+0000","last_active":"2024-03-09T15:36:57.805050+0000","last_peered":"2024-03-09T15:36:57.805050+0000","last_clean":"2024-03-09T15:36:57.805050+0000","last_became_active":"2024-03-09T14:28:38.334463+0000","last_became_peered":"2024-03-09T14:28:38.334463+0000","last_unstale":"2024-03-09T15:36:57.805050+0000","last_undegraded":"2024-03-09T15:36:57.805050+0000","last_fullsized":"2024-03-09T15:36:57.805050+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'136","last_scrub_stamp":"2024-03-09T15:36:57.804850+0000","last_deep_scrub":"261'136","last_deep_scrub_stamp":"2024-03-09T15:36:57.804850+0000","last_clean_scrub_stamp":"2024-03-09T15:36:57.804850+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:28:50.226857+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":178,"num_read_kb":172,"num_write":132,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1510,"num_omap_keys":4,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.103290+0000","last_change":"2024-03-09T15:36:20.103341+0000","last_active":"2024-03-09T15:36:20.103290+0000","last_peered":"2024-03-09T15:36:20.103290+0000","last_clean":"2024-03-09T15:36:20.103290+0000","last_became_active":"2024-03-09T14:28:40.370436+0000","last_became_peered":"2024-03-09T14:28:40.370436+0000","last_unstale":"2024-03-09T15:36:20.103290+0000","last_undegraded":"2024-03-09T15:36:20.103290+0000","last_fullsized":"2024-03-09T15:36:20.103290+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:20.103091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:20.103091+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.103091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:29:38.107241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"261'48","reported_seq":573,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:17.682061+0000","last_change":"2024-03-09T15:36:17.682122+0000","last_active":"2024-03-09T15:36:17.682061+0000","last_peered":"2024-03-09T15:36:17.682061+0000","last_clean":"2024-03-09T15:36:17.682061+0000","last_became_active":"2024-03-09T14:28:42.397058+0000","last_became_peered":"2024-03-09T14:28:42.397058+0000","last_unstale":"2024-03-09T15:36:17.682061+0000","last_undegraded":"2024-03-09T15:36:17.682061+0000","last_fullsized":"2024-03-09T15:36:17.682061+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:17.681861+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:17.681861+0000","last_clean_scrub_stamp":"2024-03-09T15:36:17.681861+0000","objects_scrubbed":1,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:38:01.033648+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":100,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"261'43","reported_seq":603,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.426328+0000","last_change":"2024-03-09T15:36:36.426328+0000","last_active":"2024-03-09T15:36:36.426328+0000","last_peered":"2024-03-09T15:36:36.426328+0000","last_clean":"2024-03-09T15:36:36.426328+0000","last_became_active":"2024-03-09T14:28:34.296953+0000","last_became_peered":"2024-03-09T14:28:34.296953+0000","last_unstale":"2024-03-09T15:36:36.426328+0000","last_undegraded":"2024-03-09T15:36:36.426328+0000","last_fullsized":"2024-03-09T15:36:36.426328+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:36.426051+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:36.426051+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.426051+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:43:59.353686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":193,"num_read_kb":162,"num_write":75,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"261'165","reported_seq":809,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.889523+0000","last_change":"2024-03-09T15:36:48.889523+0000","last_active":"2024-03-09T15:36:48.889523+0000","last_peered":"2024-03-09T15:36:48.889523+0000","last_clean":"2024-03-09T15:36:48.889523+0000","last_became_active":"2024-03-09T14:28:32.302970+0000","last_became_peered":"2024-03-09T14:28:32.302970+0000","last_unstale":"2024-03-09T15:36:48.889523+0000","last_undegraded":"2024-03-09T15:36:48.889523+0000","last_fullsized":"2024-03-09T15:36:48.889523+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:48.889098+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:48.889098+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.889098+0000","objects_scrubbed":2,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:53:35.895952+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655822,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":94521,"num_write":632,"num_write_kb":45805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":530,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.321259+0000","last_change":"2024-03-09T15:36:39.321259+0000","last_active":"2024-03-09T15:36:39.321259+0000","last_peered":"2024-03-09T15:36:39.321259+0000","last_clean":"2024-03-09T15:36:39.321259+0000","last_became_active":"2024-03-09T14:28:19.236240+0000","last_became_peered":"2024-03-09T14:28:19.236240+0000","last_unstale":"2024-03-09T15:36:39.321259+0000","last_undegraded":"2024-03-09T15:36:39.321259+0000","last_fullsized":"2024-03-09T15:36:39.321259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:39.321005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:39.321005+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.321005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:47:02.712539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":515,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.328482+0000","last_change":"2024-03-09T15:36:47.328482+0000","last_active":"2024-03-09T15:36:47.328482+0000","last_peered":"2024-03-09T15:36:47.328482+0000","last_clean":"2024-03-09T15:36:47.328482+0000","last_became_active":"2024-03-09T14:28:36.314526+0000","last_became_peered":"2024-03-09T14:28:36.314526+0000","last_unstale":"2024-03-09T15:36:47.328482+0000","last_undegraded":"2024-03-09T15:36:47.328482+0000","last_fullsized":"2024-03-09T15:36:47.328482+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:47.328224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:47.328224+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.328224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:38:07.502391+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"261'393","reported_seq":1085,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.271984+0000","last_change":"2024-03-09T15:36:40.271984+0000","last_active":"2024-03-09T15:36:40.271984+0000","last_peered":"2024-03-09T15:36:40.271984+0000","last_clean":"2024-03-09T15:36:40.271984+0000","last_became_active":"2024-03-09T14:28:38.334976+0000","last_became_peered":"2024-03-09T14:28:38.334976+0000","last_unstale":"2024-03-09T15:36:40.271984+0000","last_undegraded":"2024-03-09T15:36:40.271984+0000","last_fullsized":"2024-03-09T15:36:40.271984+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'393","last_scrub_stamp":"2024-03-09T15:36:40.271702+0000","last_deep_scrub":"261'393","last_deep_scrub_stamp":"2024-03-09T15:36:40.271702+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.271702+0000","objects_scrubbed":9,"log_size":93,"log_dups_size":300,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:45:05.080884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21042,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":640,"num_read_kb":1204,"num_write":456,"num_write_kb":268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2284,"num_omap_keys":4,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":511,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:20.131154+0000","last_change":"2024-03-09T15:36:20.131154+0000","last_active":"2024-03-09T15:36:20.131154+0000","last_peered":"2024-03-09T15:36:20.131154+0000","last_clean":"2024-03-09T15:36:20.131154+0000","last_became_active":"2024-03-09T14:28:40.379108+0000","last_became_peered":"2024-03-09T14:28:40.379108+0000","last_unstale":"2024-03-09T15:36:20.131154+0000","last_undegraded":"2024-03-09T15:36:20.131154+0000","last_fullsized":"2024-03-09T15:36:20.131154+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:20.130903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:20.130903+0000","last_clean_scrub_stamp":"2024-03-09T15:36:20.130903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:41:11.167873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"261'104","reported_seq":639,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.264151+0000","last_change":"2024-03-09T15:36:26.264345+0000","last_active":"2024-03-09T15:36:26.264151+0000","last_peered":"2024-03-09T15:36:26.264151+0000","last_clean":"2024-03-09T15:36:26.264151+0000","last_became_active":"2024-03-09T14:28:42.405277+0000","last_became_peered":"2024-03-09T14:28:42.405277+0000","last_unstale":"2024-03-09T15:36:26.264151+0000","last_undegraded":"2024-03-09T15:36:26.264151+0000","last_fullsized":"2024-03-09T15:36:26.264151+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'104","last_scrub_stamp":"2024-03-09T15:36:26.263887+0000","last_deep_scrub":"261'104","last_deep_scrub_stamp":"2024-03-09T15:36:26.263887+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.263887+0000","objects_scrubbed":3,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:52.786785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":717,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185,"num_read_kb":160,"num_write":261,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"261'43","reported_seq":585,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.293348+0000","last_change":"2024-03-09T15:36:48.293348+0000","last_active":"2024-03-09T15:36:48.293348+0000","last_peered":"2024-03-09T15:36:48.293348+0000","last_clean":"2024-03-09T15:36:48.293348+0000","last_became_active":"2024-03-09T14:28:34.296660+0000","last_became_peered":"2024-03-09T14:28:34.296660+0000","last_unstale":"2024-03-09T15:36:48.293348+0000","last_undegraded":"2024-03-09T15:36:48.293348+0000","last_fullsized":"2024-03-09T15:36:48.293348+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'43","last_scrub_stamp":"2024-03-09T15:36:48.292984+0000","last_deep_scrub":"261'43","last_deep_scrub_stamp":"2024-03-09T15:36:48.292984+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.292984+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:58:19.981031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":129,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"261'157","reported_seq":797,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.182693+0000","last_change":"2024-03-09T15:36:44.182693+0000","last_active":"2024-03-09T15:36:44.182693+0000","last_peered":"2024-03-09T15:36:44.182693+0000","last_clean":"2024-03-09T15:36:44.182693+0000","last_became_active":"2024-03-09T14:28:32.305318+0000","last_became_peered":"2024-03-09T14:28:32.305318+0000","last_unstale":"2024-03-09T15:36:44.182693+0000","last_undegraded":"2024-03-09T15:36:44.182693+0000","last_fullsized":"2024-03-09T15:36:44.182693+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'157","last_scrub_stamp":"2024-03-09T15:36:44.182405+0000","last_deep_scrub":"261'157","last_deep_scrub_stamp":"2024-03-09T15:36:44.182405+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.182405+0000","objects_scrubbed":9,"log_size":57,"log_dups_size":100,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:14:02.068526+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262245,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":357,"num_read_kb":10578,"num_write":456,"num_write_kb":20505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":534,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:56.079981+0000","last_change":"2024-03-09T15:36:56.079981+0000","last_active":"2024-03-09T15:36:56.079981+0000","last_peered":"2024-03-09T15:36:56.079981+0000","last_clean":"2024-03-09T15:36:56.079981+0000","last_became_active":"2024-03-09T14:28:19.240704+0000","last_became_peered":"2024-03-09T14:28:19.240704+0000","last_unstale":"2024-03-09T15:36:56.079981+0000","last_undegraded":"2024-03-09T15:36:56.079981+0000","last_fullsized":"2024-03-09T15:36:56.079981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:56.079740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:56.079740+0000","last_clean_scrub_stamp":"2024-03-09T15:36:56.079740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:46:45.156526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":515,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.191369+0000","last_change":"2024-03-09T15:36:25.191369+0000","last_active":"2024-03-09T15:36:25.191369+0000","last_peered":"2024-03-09T15:36:25.191369+0000","last_clean":"2024-03-09T15:36:25.191369+0000","last_became_active":"2024-03-09T14:28:36.314676+0000","last_became_peered":"2024-03-09T14:28:36.314676+0000","last_unstale":"2024-03-09T15:36:25.191369+0000","last_undegraded":"2024-03-09T15:36:25.191369+0000","last_fullsized":"2024-03-09T15:36:25.191369+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.191100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.191100+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.191100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:12:04.217794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"261'205","reported_seq":796,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.476874+0000","last_change":"2024-03-09T15:36:35.476874+0000","last_active":"2024-03-09T15:36:35.476874+0000","last_peered":"2024-03-09T15:36:35.476874+0000","last_clean":"2024-03-09T15:36:35.476874+0000","last_became_active":"2024-03-09T14:28:38.595028+0000","last_became_peered":"2024-03-09T14:28:38.595028+0000","last_unstale":"2024-03-09T15:36:35.476874+0000","last_undegraded":"2024-03-09T15:36:35.476874+0000","last_fullsized":"2024-03-09T15:36:35.476874+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'205","last_scrub_stamp":"2024-03-09T15:36:35.476600+0000","last_deep_scrub":"261'205","last_deep_scrub_stamp":"2024-03-09T15:36:35.476600+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.476600+0000","objects_scrubbed":8,"log_size":105,"log_dups_size":100,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:50:56.058991+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":269,"num_read_kb":262,"num_write":230,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":330,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"261'501","reported_seq":2307,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.727185+0000","last_change":"2024-03-09T15:36:23.727185+0000","last_active":"2024-03-09T15:36:23.727185+0000","last_peered":"2024-03-09T15:36:23.727185+0000","last_clean":"2024-03-09T15:36:23.727185+0000","last_became_active":"2024-03-09T14:28:40.369165+0000","last_became_peered":"2024-03-09T14:28:40.369165+0000","last_unstale":"2024-03-09T15:36:23.727185+0000","last_undegraded":"2024-03-09T15:36:23.727185+0000","last_fullsized":"2024-03-09T15:36:23.727185+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'501","last_scrub_stamp":"2024-03-09T15:36:23.726932+0000","last_deep_scrub":"261'501","last_deep_scrub_stamp":"2024-03-09T15:36:23.726932+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.726932+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":400,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:05:11.145403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"261'79","reported_seq":577,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:27.487857+0000","last_change":"2024-03-09T15:36:27.487857+0000","last_active":"2024-03-09T15:36:27.487857+0000","last_peered":"2024-03-09T15:36:27.487857+0000","last_clean":"2024-03-09T15:36:27.487857+0000","last_became_active":"2024-03-09T14:28:42.392402+0000","last_became_peered":"2024-03-09T14:28:42.392402+0000","last_unstale":"2024-03-09T15:36:27.487857+0000","last_undegraded":"2024-03-09T15:36:27.487857+0000","last_fullsized":"2024-03-09T15:36:27.487857+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'79","last_scrub_stamp":"2024-03-09T15:36:27.487564+0000","last_deep_scrub":"261'79","last_deep_scrub_stamp":"2024-03-09T15:36:27.487564+0000","last_clean_scrub_stamp":"2024-03-09T15:36:27.487564+0000","objects_scrubbed":3,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:09:13.446303+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":57,"num_write":172,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"261'48","reported_seq":599,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:18.188944+0000","last_change":"2024-03-09T15:36:18.188944+0000","last_active":"2024-03-09T15:36:18.188944+0000","last_peered":"2024-03-09T15:36:18.188944+0000","last_clean":"2024-03-09T15:36:18.188944+0000","last_became_active":"2024-03-09T14:29:06.440556+0000","last_became_peered":"2024-03-09T14:29:06.440556+0000","last_unstale":"2024-03-09T15:36:18.188944+0000","last_undegraded":"2024-03-09T15:36:18.188944+0000","last_fullsized":"2024-03-09T15:36:18.188944+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:18.188663+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:18.188663+0000","last_clean_scrub_stamp":"2024-03-09T15:36:18.188663+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:03:02.670504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":216,"num_read_kb":182,"num_write":66,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"261'150","reported_seq":704,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:06.778364+0000","last_change":"2024-03-09T15:37:06.778452+0000","last_active":"2024-03-09T15:37:06.778364+0000","last_peered":"2024-03-09T15:37:06.778364+0000","last_clean":"2024-03-09T15:37:06.778364+0000","last_became_active":"2024-03-09T14:28:32.318971+0000","last_became_peered":"2024-03-09T14:28:32.318971+0000","last_unstale":"2024-03-09T15:37:06.778364+0000","last_undegraded":"2024-03-09T15:37:06.778364+0000","last_fullsized":"2024-03-09T15:37:06.778364+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'150","last_scrub_stamp":"2024-03-09T15:37:06.778183+0000","last_deep_scrub":"261'150","last_deep_scrub_stamp":"2024-03-09T15:37:06.778183+0000","last_clean_scrub_stamp":"2024-03-09T15:37:06.778183+0000","objects_scrubbed":5,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:15:24.839103+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":151,"num_read_kb":8647,"num_write":476,"num_write_kb":27350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":528,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.293647+0000","last_change":"2024-03-09T15:36:40.293706+0000","last_active":"2024-03-09T15:36:40.293647+0000","last_peered":"2024-03-09T15:36:40.293647+0000","last_clean":"2024-03-09T15:36:40.293647+0000","last_became_active":"2024-03-09T14:28:19.237750+0000","last_became_peered":"2024-03-09T14:28:19.237750+0000","last_unstale":"2024-03-09T15:36:40.293647+0000","last_undegraded":"2024-03-09T15:36:40.293647+0000","last_fullsized":"2024-03-09T15:36:40.293647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:40.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:40.293440+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:43:40.824370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":519,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:55.494308+0000","last_change":"2024-03-09T15:36:55.494308+0000","last_active":"2024-03-09T15:36:55.494308+0000","last_peered":"2024-03-09T15:36:55.494308+0000","last_clean":"2024-03-09T15:36:55.494308+0000","last_became_active":"2024-03-09T14:28:36.327601+0000","last_became_peered":"2024-03-09T14:28:36.327601+0000","last_unstale":"2024-03-09T15:36:55.494308+0000","last_undegraded":"2024-03-09T15:36:55.494308+0000","last_fullsized":"2024-03-09T15:36:55.494308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:55.494042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:55.494042+0000","last_clean_scrub_stamp":"2024-03-09T15:36:55.494042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:15:02.668845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"261'111","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.173180+0000","last_change":"2024-03-09T15:36:26.173180+0000","last_active":"2024-03-09T15:36:26.173180+0000","last_peered":"2024-03-09T15:36:26.173180+0000","last_clean":"2024-03-09T15:36:26.173180+0000","last_became_active":"2024-03-09T14:28:38.347750+0000","last_became_peered":"2024-03-09T14:28:38.347750+0000","last_unstale":"2024-03-09T15:36:26.173180+0000","last_undegraded":"2024-03-09T15:36:26.173180+0000","last_fullsized":"2024-03-09T15:36:26.173180+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'111","last_scrub_stamp":"2024-03-09T15:36:26.172900+0000","last_deep_scrub":"261'111","last_deep_scrub_stamp":"2024-03-09T15:36:26.172900+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.172900+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":100,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:00:14.752406+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"261'514","reported_seq":2496,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.486405+0000","last_change":"2024-03-09T15:36:34.486405+0000","last_active":"2024-03-09T15:36:34.486405+0000","last_peered":"2024-03-09T15:36:34.486405+0000","last_clean":"2024-03-09T15:36:34.486405+0000","last_became_active":"2024-03-09T14:28:40.386859+0000","last_became_peered":"2024-03-09T14:28:40.386859+0000","last_unstale":"2024-03-09T15:36:34.486405+0000","last_undegraded":"2024-03-09T15:36:34.486405+0000","last_fullsized":"2024-03-09T15:36:34.486405+0000","mapping_epoch":28,"log_start":"261'500","ondisk_log_start":"261'500","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'514","last_scrub_stamp":"2024-03-09T15:36:34.486115+0000","last_deep_scrub":"261'514","last_deep_scrub_stamp":"2024-03-09T15:36:34.486115+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.486115+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":500,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:43:00.093432+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"261'108","reported_seq":618,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.271001+0000","last_change":"2024-03-09T15:36:25.271065+0000","last_active":"2024-03-09T15:36:25.271001+0000","last_peered":"2024-03-09T15:36:25.271001+0000","last_clean":"2024-03-09T15:36:25.271001+0000","last_became_active":"2024-03-09T14:28:42.406139+0000","last_became_peered":"2024-03-09T14:28:42.406139+0000","last_unstale":"2024-03-09T15:36:25.271001+0000","last_undegraded":"2024-03-09T15:36:25.271001+0000","last_fullsized":"2024-03-09T15:36:25.271001+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'108","last_scrub_stamp":"2024-03-09T15:36:25.270811+0000","last_deep_scrub":"261'108","last_deep_scrub_stamp":"2024-03-09T15:36:25.270811+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.270811+0000","objects_scrubbed":2,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:05:43.169466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":65,"num_write":235,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"261'217","reported_seq":798,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:52.498953+0000","last_change":"2024-03-09T15:36:52.498953+0000","last_active":"2024-03-09T15:36:52.498953+0000","last_peered":"2024-03-09T15:36:52.498953+0000","last_clean":"2024-03-09T15:36:52.498953+0000","last_became_active":"2024-03-09T14:28:34.299988+0000","last_became_peered":"2024-03-09T14:28:34.299988+0000","last_unstale":"2024-03-09T15:36:52.498953+0000","last_undegraded":"2024-03-09T15:36:52.498953+0000","last_fullsized":"2024-03-09T15:36:52.498953+0000","mapping_epoch":22,"log_start":"258'200","ondisk_log_start":"258'200","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'217","last_scrub_stamp":"2024-03-09T15:36:52.498694+0000","last_deep_scrub":"261'217","last_deep_scrub_stamp":"2024-03-09T15:36:52.498694+0000","last_clean_scrub_stamp":"2024-03-09T15:36:52.498694+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":200,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:46:08.054569+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":843,"num_read_kb":645,"num_write":377,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":268,"num_omap_keys":1,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"261'154","reported_seq":716,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:02.150046+0000","last_change":"2024-03-09T15:37:02.150218+0000","last_active":"2024-03-09T15:37:02.150046+0000","last_peered":"2024-03-09T15:37:02.150046+0000","last_clean":"2024-03-09T15:37:02.150046+0000","last_became_active":"2024-03-09T14:28:32.298977+0000","last_became_peered":"2024-03-09T14:28:32.298977+0000","last_unstale":"2024-03-09T15:37:02.150046+0000","last_undegraded":"2024-03-09T15:37:02.150046+0000","last_fullsized":"2024-03-09T15:37:02.150046+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'154","last_scrub_stamp":"2024-03-09T15:37:02.149792+0000","last_deep_scrub":"261'154","last_deep_scrub_stamp":"2024-03-09T15:37:02.149792+0000","last_clean_scrub_stamp":"2024-03-09T15:37:02.149792+0000","objects_scrubbed":4,"log_size":54,"log_dups_size":100,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:49:58.935130+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242905,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":173,"num_read_kb":5781,"num_write":510,"num_write_kb":37567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.336163+0000","last_change":"2024-03-09T15:36:36.336215+0000","last_active":"2024-03-09T15:36:36.336163+0000","last_peered":"2024-03-09T15:36:36.336163+0000","last_clean":"2024-03-09T15:36:36.336163+0000","last_became_active":"2024-03-09T14:28:19.241700+0000","last_became_peered":"2024-03-09T14:28:19.241700+0000","last_unstale":"2024-03-09T15:36:36.336163+0000","last_undegraded":"2024-03-09T15:36:36.336163+0000","last_fullsized":"2024-03-09T15:36:36.336163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:36.335976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:36.335976+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.335976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:25:25.910931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"261'4","reported_seq":833,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.712424+0000","last_change":"2024-03-09T15:36:51.712560+0000","last_active":"2024-03-09T15:36:51.712424+0000","last_peered":"2024-03-09T15:36:51.712424+0000","last_clean":"2024-03-09T15:36:51.712424+0000","last_became_active":"2024-03-09T14:28:36.317327+0000","last_became_peered":"2024-03-09T14:28:36.317327+0000","last_unstale":"2024-03-09T15:36:51.712424+0000","last_undegraded":"2024-03-09T15:36:51.712424+0000","last_fullsized":"2024-03-09T15:36:51.712424+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'4","last_scrub_stamp":"2024-03-09T15:36:51.712182+0000","last_deep_scrub":"261'4","last_deep_scrub_stamp":"2024-03-09T15:36:51.712182+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.712182+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:06:10.620608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":484,"num_read_kb":482,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"261'478","reported_seq":1227,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.248788+0000","last_change":"2024-03-09T15:36:38.248788+0000","last_active":"2024-03-09T15:36:38.248788+0000","last_peered":"2024-03-09T15:36:38.248788+0000","last_clean":"2024-03-09T15:36:38.248788+0000","last_became_active":"2024-03-09T14:28:38.602908+0000","last_became_peered":"2024-03-09T14:28:38.602908+0000","last_unstale":"2024-03-09T15:36:38.248788+0000","last_undegraded":"2024-03-09T15:36:38.248788+0000","last_fullsized":"2024-03-09T15:36:38.248788+0000","mapping_epoch":26,"log_start":"261'400","ondisk_log_start":"261'400","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'478","last_scrub_stamp":"2024-03-09T15:36:38.248482+0000","last_deep_scrub":"261'478","last_deep_scrub_stamp":"2024-03-09T15:36:38.248482+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.248482+0000","objects_scrubbed":8,"log_size":78,"log_dups_size":400,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:50:57.022028+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29305,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":797,"num_read_kb":1561,"num_write":550,"num_write_kb":344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.482326+0000","last_change":"2024-03-09T15:36:35.482326+0000","last_active":"2024-03-09T15:36:35.482326+0000","last_peered":"2024-03-09T15:36:35.482326+0000","last_clean":"2024-03-09T15:36:35.482326+0000","last_became_active":"2024-03-09T14:28:40.375884+0000","last_became_peered":"2024-03-09T14:28:40.375884+0000","last_unstale":"2024-03-09T15:36:35.482326+0000","last_undegraded":"2024-03-09T15:36:35.482326+0000","last_fullsized":"2024-03-09T15:36:35.482326+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:35.482232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:35.482232+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.482232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:57:53.662951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"261'56","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:13.577103+0000","last_change":"2024-03-09T15:36:13.577103+0000","last_active":"2024-03-09T15:36:13.577103+0000","last_peered":"2024-03-09T15:36:13.577103+0000","last_clean":"2024-03-09T15:36:13.577103+0000","last_became_active":"2024-03-09T14:28:42.405998+0000","last_became_peered":"2024-03-09T14:28:42.405998+0000","last_unstale":"2024-03-09T15:36:13.577103+0000","last_undegraded":"2024-03-09T15:36:13.577103+0000","last_fullsized":"2024-03-09T15:36:13.577103+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:13.576825+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:13.576825+0000","last_clean_scrub_stamp":"2024-03-09T15:36:13.576825+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:29:57.437161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":52,"num_write":123,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1f","version":"261'149","reported_seq":736,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.301934+0000","last_change":"2024-03-09T15:36:24.301934+0000","last_active":"2024-03-09T15:36:24.301934+0000","last_peered":"2024-03-09T15:36:24.301934+0000","last_clean":"2024-03-09T15:36:24.301934+0000","last_became_active":"2024-03-09T14:28:42.408774+0000","last_became_peered":"2024-03-09T14:28:42.408774+0000","last_unstale":"2024-03-09T15:36:24.301934+0000","last_undegraded":"2024-03-09T15:36:24.301934+0000","last_fullsized":"2024-03-09T15:36:24.301934+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'149","last_scrub_stamp":"2024-03-09T15:36:24.301670+0000","last_deep_scrub":"261'149","last_deep_scrub_stamp":"2024-03-09T15:36:24.301670+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.301670+0000","objects_scrubbed":3,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:31:20.109756+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":256,"num_read_kb":233,"num_write":254,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"261'183","reported_seq":753,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.152146+0000","last_change":"2024-03-09T15:36:32.152205+0000","last_active":"2024-03-09T15:36:32.152146+0000","last_peered":"2024-03-09T15:36:32.152146+0000","last_clean":"2024-03-09T15:36:32.152146+0000","last_became_active":"2024-03-09T14:28:34.314893+0000","last_became_peered":"2024-03-09T14:28:34.314893+0000","last_unstale":"2024-03-09T15:36:32.152146+0000","last_undegraded":"2024-03-09T15:36:32.152146+0000","last_fullsized":"2024-03-09T15:36:32.152146+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'183","last_scrub_stamp":"2024-03-09T15:36:32.151949+0000","last_deep_scrub":"261'183","last_deep_scrub_stamp":"2024-03-09T15:36:32.151949+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.151949+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":100,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:40:37.495594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":706,"num_read_kb":592,"num_write":291,"num_write_kb":155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"261'132","reported_seq":686,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.480459+0000","last_change":"2024-03-09T15:36:50.480551+0000","last_active":"2024-03-09T15:36:50.480459+0000","last_peered":"2024-03-09T15:36:50.480459+0000","last_clean":"2024-03-09T15:36:50.480459+0000","last_became_active":"2024-03-09T14:28:32.322364+0000","last_became_peered":"2024-03-09T14:28:32.322364+0000","last_unstale":"2024-03-09T15:36:50.480459+0000","last_undegraded":"2024-03-09T15:36:50.480459+0000","last_fullsized":"2024-03-09T15:36:50.480459+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'132","last_scrub_stamp":"2024-03-09T15:36:50.480252+0000","last_deep_scrub":"261'132","last_deep_scrub_stamp":"2024-03-09T15:36:50.480252+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.480252+0000","objects_scrubbed":3,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:46:23.715589+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":128,"num_write":389,"num_write_kb":27813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":512,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:25.576225+0000","last_change":"2024-03-09T15:36:25.576277+0000","last_active":"2024-03-09T15:36:25.576225+0000","last_peered":"2024-03-09T15:36:25.576225+0000","last_clean":"2024-03-09T15:36:25.576225+0000","last_became_active":"2024-03-09T14:28:36.313984+0000","last_became_peered":"2024-03-09T14:28:36.313984+0000","last_unstale":"2024-03-09T15:36:25.576225+0000","last_undegraded":"2024-03-09T15:36:25.576225+0000","last_fullsized":"2024-03-09T15:36:25.576225+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:25.576027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:25.576027+0000","last_clean_scrub_stamp":"2024-03-09T15:36:25.576027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:16:03.346315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"261'196","reported_seq":788,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:07.771948+0000","last_change":"2024-03-09T15:37:07.772039+0000","last_active":"2024-03-09T15:37:07.771948+0000","last_peered":"2024-03-09T15:37:07.771948+0000","last_clean":"2024-03-09T15:37:07.771948+0000","last_became_active":"2024-03-09T14:28:38.338262+0000","last_became_peered":"2024-03-09T14:28:38.338262+0000","last_unstale":"2024-03-09T15:37:07.771948+0000","last_undegraded":"2024-03-09T15:37:07.771948+0000","last_fullsized":"2024-03-09T15:37:07.771948+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'196","last_scrub_stamp":"2024-03-09T15:37:07.771742+0000","last_deep_scrub":"261'196","last_deep_scrub_stamp":"2024-03-09T15:37:07.771742+0000","last_clean_scrub_stamp":"2024-03-09T15:37:07.771742+0000","objects_scrubbed":5,"log_size":96,"log_dups_size":100,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:34:19.385252+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15994,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":315,"num_read_kb":601,"num_write":220,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":499,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.450333+0000","last_change":"2024-03-09T15:36:31.450390+0000","last_active":"2024-03-09T15:36:31.450333+0000","last_peered":"2024-03-09T15:36:31.450333+0000","last_clean":"2024-03-09T15:36:31.450333+0000","last_became_active":"2024-03-09T14:28:40.389277+0000","last_became_peered":"2024-03-09T14:28:40.389277+0000","last_unstale":"2024-03-09T15:36:31.450333+0000","last_undegraded":"2024-03-09T15:36:31.450333+0000","last_fullsized":"2024-03-09T15:36:31.450333+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:31.450128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:31.450128+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.450128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:52:04.464244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"261'56","reported_seq":561,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.467616+0000","last_change":"2024-03-09T15:36:22.467616+0000","last_active":"2024-03-09T15:36:22.467616+0000","last_peered":"2024-03-09T15:36:22.467616+0000","last_clean":"2024-03-09T15:36:22.467616+0000","last_became_active":"2024-03-09T14:28:42.409229+0000","last_became_peered":"2024-03-09T14:28:42.409229+0000","last_unstale":"2024-03-09T15:36:22.467616+0000","last_undegraded":"2024-03-09T15:36:22.467616+0000","last_fullsized":"2024-03-09T15:36:22.467616+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:22.467355+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:22.467355+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.467355+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:46:09.238574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":828,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":44,"num_write":116,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"261'78","reported_seq":640,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.558934+0000","last_change":"2024-03-09T15:36:26.558992+0000","last_active":"2024-03-09T15:36:26.558934+0000","last_peered":"2024-03-09T15:36:26.558934+0000","last_clean":"2024-03-09T15:36:26.558934+0000","last_became_active":"2024-03-09T14:28:34.320474+0000","last_became_peered":"2024-03-09T14:28:34.320474+0000","last_unstale":"2024-03-09T15:36:26.558934+0000","last_undegraded":"2024-03-09T15:36:26.558934+0000","last_fullsized":"2024-03-09T15:36:26.558934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'78","last_scrub_stamp":"2024-03-09T15:36:26.558741+0000","last_deep_scrub":"261'78","last_deep_scrub_stamp":"2024-03-09T15:36:26.558741+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.558741+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:58:25.865037+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":231,"num_write":128,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"261'163","reported_seq":805,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:00.149520+0000","last_change":"2024-03-09T15:37:00.149647+0000","last_active":"2024-03-09T15:37:00.149520+0000","last_peered":"2024-03-09T15:37:00.149520+0000","last_clean":"2024-03-09T15:37:00.149520+0000","last_became_active":"2024-03-09T14:28:32.315355+0000","last_became_peered":"2024-03-09T14:28:32.315355+0000","last_unstale":"2024-03-09T15:37:00.149520+0000","last_undegraded":"2024-03-09T15:37:00.149520+0000","last_fullsized":"2024-03-09T15:37:00.149520+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'163","last_scrub_stamp":"2024-03-09T15:37:00.149328+0000","last_deep_scrub":"261'163","last_deep_scrub_stamp":"2024-03-09T15:37:00.149328+0000","last_clean_scrub_stamp":"2024-03-09T15:37:00.149328+0000","objects_scrubbed":4,"log_size":63,"log_dups_size":100,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:05:07.700021+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":325,"num_read_kb":85989,"num_write":581,"num_write_kb":44811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.649651+0000","last_change":"2024-03-09T15:36:49.649702+0000","last_active":"2024-03-09T15:36:49.649651+0000","last_peered":"2024-03-09T15:36:49.649651+0000","last_clean":"2024-03-09T15:36:49.649651+0000","last_became_active":"2024-03-09T14:28:36.311487+0000","last_became_peered":"2024-03-09T14:28:36.311487+0000","last_unstale":"2024-03-09T15:36:49.649651+0000","last_undegraded":"2024-03-09T15:36:49.649651+0000","last_fullsized":"2024-03-09T15:36:49.649651+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:49.649459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:49.649459+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.649459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:35:37.031126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"261'42","reported_seq":563,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:05.098952+0000","last_change":"2024-03-09T15:37:05.099025+0000","last_active":"2024-03-09T15:37:05.098952+0000","last_peered":"2024-03-09T15:37:05.098952+0000","last_clean":"2024-03-09T15:37:05.098952+0000","last_became_active":"2024-03-09T14:28:38.332610+0000","last_became_peered":"2024-03-09T14:28:38.332610+0000","last_unstale":"2024-03-09T15:37:05.098952+0000","last_undegraded":"2024-03-09T15:37:05.098952+0000","last_fullsized":"2024-03-09T15:37:05.098952+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'42","last_scrub_stamp":"2024-03-09T15:37:05.098734+0000","last_deep_scrub":"261'42","last_deep_scrub_stamp":"2024-03-09T15:37:05.098734+0000","last_clean_scrub_stamp":"2024-03-09T15:37:05.098734+0000","objects_scrubbed":3,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:23:34.461015+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":41,"num_write":30,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":294,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":508,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.258023+0000","last_change":"2024-03-09T15:36:35.258136+0000","last_active":"2024-03-09T15:36:35.258023+0000","last_peered":"2024-03-09T15:36:35.258023+0000","last_clean":"2024-03-09T15:36:35.258023+0000","last_became_active":"2024-03-09T14:28:40.367795+0000","last_became_peered":"2024-03-09T14:28:40.367795+0000","last_unstale":"2024-03-09T15:36:35.258023+0000","last_undegraded":"2024-03-09T15:36:35.258023+0000","last_fullsized":"2024-03-09T15:36:35.258023+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:35.257744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:35.257744+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.257744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:42:18.958156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1d","version":"261'172","reported_seq":700,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.304337+0000","last_change":"2024-03-09T15:36:22.304413+0000","last_active":"2024-03-09T15:36:22.304337+0000","last_peered":"2024-03-09T15:36:22.304337+0000","last_clean":"2024-03-09T15:36:22.304337+0000","last_became_active":"2024-03-09T14:28:42.408913+0000","last_became_peered":"2024-03-09T14:28:42.408913+0000","last_unstale":"2024-03-09T15:36:22.304337+0000","last_undegraded":"2024-03-09T15:36:22.304337+0000","last_fullsized":"2024-03-09T15:36:22.304337+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'172","last_scrub_stamp":"2024-03-09T15:36:22.304104+0000","last_deep_scrub":"261'172","last_deep_scrub_stamp":"2024-03-09T15:36:22.304104+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.304104+0000","objects_scrubbed":2,"log_size":72,"log_dups_size":100,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:45:45.379355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":208,"num_write":351,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":398,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"261'54","reported_seq":603,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.672674+0000","last_change":"2024-03-09T15:36:47.672732+0000","last_active":"2024-03-09T15:36:47.672674+0000","last_peered":"2024-03-09T15:36:47.672674+0000","last_clean":"2024-03-09T15:36:47.672674+0000","last_became_active":"2024-03-09T14:28:34.330813+0000","last_became_peered":"2024-03-09T14:28:34.330813+0000","last_unstale":"2024-03-09T15:36:47.672674+0000","last_undegraded":"2024-03-09T15:36:47.672674+0000","last_fullsized":"2024-03-09T15:36:47.672674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'54","last_scrub_stamp":"2024-03-09T15:36:47.672474+0000","last_deep_scrub":"261'54","last_deep_scrub_stamp":"2024-03-09T15:36:47.672474+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.672474+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:48:25.059611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":220,"num_read_kb":178,"num_write":76,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"261'133","reported_seq":699,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.908045+0000","last_change":"2024-03-09T15:36:41.908262+0000","last_active":"2024-03-09T15:36:41.908045+0000","last_peered":"2024-03-09T15:36:41.908045+0000","last_clean":"2024-03-09T15:36:41.908045+0000","last_became_active":"2024-03-09T14:28:32.311843+0000","last_became_peered":"2024-03-09T14:28:32.311843+0000","last_unstale":"2024-03-09T15:36:41.908045+0000","last_undegraded":"2024-03-09T15:36:41.908045+0000","last_fullsized":"2024-03-09T15:36:41.908045+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'133","last_scrub_stamp":"2024-03-09T15:36:41.907782+0000","last_deep_scrub":"261'133","last_deep_scrub_stamp":"2024-03-09T15:36:41.907782+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.907782+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":100,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:05:53.440636+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655838,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":157,"num_read_kb":5348,"num_write":475,"num_write_kb":35385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.141589+0000","last_change":"2024-03-09T15:36:43.141654+0000","last_active":"2024-03-09T15:36:43.141589+0000","last_peered":"2024-03-09T15:36:43.141589+0000","last_clean":"2024-03-09T15:36:43.141589+0000","last_became_active":"2024-03-09T14:28:36.323335+0000","last_became_peered":"2024-03-09T14:28:36.323335+0000","last_unstale":"2024-03-09T15:36:43.141589+0000","last_undegraded":"2024-03-09T15:36:43.141589+0000","last_fullsized":"2024-03-09T15:36:43.141589+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:43.141360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:43.141360+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.141360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:29:28.525098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"261'227","reported_seq":834,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.585747+0000","last_change":"2024-03-09T15:36:42.585928+0000","last_active":"2024-03-09T15:36:42.585747+0000","last_peered":"2024-03-09T15:36:42.585747+0000","last_clean":"2024-03-09T15:36:42.585747+0000","last_became_active":"2024-03-09T14:28:38.344951+0000","last_became_peered":"2024-03-09T14:28:38.344951+0000","last_unstale":"2024-03-09T15:36:42.585747+0000","last_undegraded":"2024-03-09T15:36:42.585747+0000","last_fullsized":"2024-03-09T15:36:42.585747+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'227","last_scrub_stamp":"2024-03-09T15:36:42.585503+0000","last_deep_scrub":"261'227","last_deep_scrub_stamp":"2024-03-09T15:36:42.585503+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.585503+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":200,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:09:44.534942+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7791,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":353,"num_read_kb":629,"num_write":244,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.128626+0000","last_change":"2024-03-09T15:36:22.128680+0000","last_active":"2024-03-09T15:36:22.128626+0000","last_peered":"2024-03-09T15:36:22.128626+0000","last_clean":"2024-03-09T15:36:22.128626+0000","last_became_active":"2024-03-09T14:28:40.370313+0000","last_became_peered":"2024-03-09T14:28:40.370313+0000","last_unstale":"2024-03-09T15:36:22.128626+0000","last_undegraded":"2024-03-09T15:36:22.128626+0000","last_fullsized":"2024-03-09T15:36:22.128626+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.128436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.128436+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.128436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:31:27.821903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"261'47","reported_seq":550,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.629026+0000","last_change":"2024-03-09T15:36:15.629026+0000","last_active":"2024-03-09T15:36:15.629026+0000","last_peered":"2024-03-09T15:36:15.629026+0000","last_clean":"2024-03-09T15:36:15.629026+0000","last_became_active":"2024-03-09T14:28:42.404360+0000","last_became_peered":"2024-03-09T14:28:42.404360+0000","last_unstale":"2024-03-09T15:36:15.629026+0000","last_undegraded":"2024-03-09T15:36:15.629026+0000","last_fullsized":"2024-03-09T15:36:15.629026+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'47","last_scrub_stamp":"2024-03-09T15:36:15.628774+0000","last_deep_scrub":"261'47","last_deep_scrub_stamp":"2024-03-09T15:36:15.628774+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.628774+0000","objects_scrubbed":2,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:04:15.376099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":43,"num_write":116,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"261'88","reported_seq":651,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.697326+0000","last_change":"2024-03-09T15:36:50.697389+0000","last_active":"2024-03-09T15:36:50.697326+0000","last_peered":"2024-03-09T15:36:50.697326+0000","last_clean":"2024-03-09T15:36:50.697326+0000","last_became_active":"2024-03-09T14:28:34.305301+0000","last_became_peered":"2024-03-09T14:28:34.305301+0000","last_unstale":"2024-03-09T15:36:50.697326+0000","last_undegraded":"2024-03-09T15:36:50.697326+0000","last_fullsized":"2024-03-09T15:36:50.697326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'88","last_scrub_stamp":"2024-03-09T15:36:50.697133+0000","last_deep_scrub":"261'88","last_deep_scrub_stamp":"2024-03-09T15:36:50.697133+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.697133+0000","objects_scrubbed":1,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:31:45.483994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":317,"num_write":145,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2876,"num_omap_keys":12,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"261'149","reported_seq":715,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:58.170601+0000","last_change":"2024-03-09T15:36:58.170723+0000","last_active":"2024-03-09T15:36:58.170601+0000","last_peered":"2024-03-09T15:36:58.170601+0000","last_clean":"2024-03-09T15:36:58.170601+0000","last_became_active":"2024-03-09T14:28:32.292393+0000","last_became_peered":"2024-03-09T14:28:32.292393+0000","last_unstale":"2024-03-09T15:36:58.170601+0000","last_undegraded":"2024-03-09T15:36:58.170601+0000","last_fullsized":"2024-03-09T15:36:58.170601+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'149","last_scrub_stamp":"2024-03-09T15:36:58.170403+0000","last_deep_scrub":"261'149","last_deep_scrub_stamp":"2024-03-09T15:36:58.170403+0000","last_clean_scrub_stamp":"2024-03-09T15:36:58.170403+0000","objects_scrubbed":6,"log_size":49,"log_dups_size":100,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:06:06.109922+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6900333,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":188,"num_read_kb":14616,"num_write":468,"num_write_kb":23484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":511,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.124875+0000","last_change":"2024-03-09T15:36:21.124931+0000","last_active":"2024-03-09T15:36:21.124875+0000","last_peered":"2024-03-09T15:36:21.124875+0000","last_clean":"2024-03-09T15:36:21.124875+0000","last_became_active":"2024-03-09T14:28:36.328946+0000","last_became_peered":"2024-03-09T14:28:36.328946+0000","last_unstale":"2024-03-09T15:36:21.124875+0000","last_undegraded":"2024-03-09T15:36:21.124875+0000","last_fullsized":"2024-03-09T15:36:21.124875+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.124678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.124678+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.124678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:16:00.704022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"261'172","reported_seq":766,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.110832+0000","last_change":"2024-03-09T15:36:49.110993+0000","last_active":"2024-03-09T15:36:49.110832+0000","last_peered":"2024-03-09T15:36:49.110832+0000","last_clean":"2024-03-09T15:36:49.110832+0000","last_became_active":"2024-03-09T14:28:38.335107+0000","last_became_peered":"2024-03-09T14:28:38.335107+0000","last_unstale":"2024-03-09T15:36:49.110832+0000","last_undegraded":"2024-03-09T15:36:49.110832+0000","last_fullsized":"2024-03-09T15:36:49.110832+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'172","last_scrub_stamp":"2024-03-09T15:36:49.110544+0000","last_deep_scrub":"261'172","last_deep_scrub_stamp":"2024-03-09T15:36:49.110544+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.110544+0000","objects_scrubbed":4,"log_size":72,"log_dups_size":100,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:18:50.261252+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9765,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":273,"num_read_kb":518,"num_write":186,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":503,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.274828+0000","last_change":"2024-03-09T15:36:36.274880+0000","last_active":"2024-03-09T15:36:36.274828+0000","last_peered":"2024-03-09T15:36:36.274828+0000","last_clean":"2024-03-09T15:36:36.274828+0000","last_became_active":"2024-03-09T14:28:40.363778+0000","last_became_peered":"2024-03-09T14:28:40.363778+0000","last_unstale":"2024-03-09T15:36:36.274828+0000","last_undegraded":"2024-03-09T15:36:36.274828+0000","last_fullsized":"2024-03-09T15:36:36.274828+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:36.274642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:36.274642+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.274642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:53:47.883218+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"261'159","reported_seq":743,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.206413+0000","last_change":"2024-03-09T15:36:29.206413+0000","last_active":"2024-03-09T15:36:29.206413+0000","last_peered":"2024-03-09T15:36:29.206413+0000","last_clean":"2024-03-09T15:36:29.206413+0000","last_became_active":"2024-03-09T14:28:34.325031+0000","last_became_peered":"2024-03-09T14:28:34.325031+0000","last_unstale":"2024-03-09T15:36:29.206413+0000","last_undegraded":"2024-03-09T15:36:29.206413+0000","last_fullsized":"2024-03-09T15:36:29.206413+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'159","last_scrub_stamp":"2024-03-09T15:36:29.206170+0000","last_deep_scrub":"261'159","last_deep_scrub_stamp":"2024-03-09T15:36:29.206170+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.206170+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":100,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:05:19.071599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":584,"num_read_kb":461,"num_write":254,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"261'118","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:00.714054+0000","last_change":"2024-03-09T15:37:00.714149+0000","last_active":"2024-03-09T15:37:00.714054+0000","last_peered":"2024-03-09T15:37:00.714054+0000","last_clean":"2024-03-09T15:37:00.714054+0000","last_became_active":"2024-03-09T14:28:32.296179+0000","last_became_peered":"2024-03-09T14:28:32.296179+0000","last_unstale":"2024-03-09T15:37:00.714054+0000","last_undegraded":"2024-03-09T15:37:00.714054+0000","last_fullsized":"2024-03-09T15:37:00.714054+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'118","last_scrub_stamp":"2024-03-09T15:37:00.713835+0000","last_deep_scrub":"261'118","last_deep_scrub_stamp":"2024-03-09T15:37:00.713835+0000","last_clean_scrub_stamp":"2024-03-09T15:37:00.713835+0000","objects_scrubbed":7,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:51:35.012606+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263265,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":136,"num_read_kb":11385,"num_write":371,"num_write_kb":17181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":516,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:51.269733+0000","last_change":"2024-03-09T15:36:51.269792+0000","last_active":"2024-03-09T15:36:51.269733+0000","last_peered":"2024-03-09T15:36:51.269733+0000","last_clean":"2024-03-09T15:36:51.269733+0000","last_became_active":"2024-03-09T14:28:36.319702+0000","last_became_peered":"2024-03-09T14:28:36.319702+0000","last_unstale":"2024-03-09T15:36:51.269733+0000","last_undegraded":"2024-03-09T15:36:51.269733+0000","last_fullsized":"2024-03-09T15:36:51.269733+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:51.269512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:51.269512+0000","last_clean_scrub_stamp":"2024-03-09T15:36:51.269512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:30:38.384018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"261'316","reported_seq":965,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.480382+0000","last_change":"2024-03-09T15:36:33.480469+0000","last_active":"2024-03-09T15:36:33.480382+0000","last_peered":"2024-03-09T15:36:33.480382+0000","last_clean":"2024-03-09T15:36:33.480382+0000","last_became_active":"2024-03-09T14:28:38.332661+0000","last_became_peered":"2024-03-09T14:28:38.332661+0000","last_unstale":"2024-03-09T15:36:33.480382+0000","last_undegraded":"2024-03-09T15:36:33.480382+0000","last_fullsized":"2024-03-09T15:36:33.480382+0000","mapping_epoch":26,"log_start":"261'300","ondisk_log_start":"261'300","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'316","last_scrub_stamp":"2024-03-09T15:36:33.480185+0000","last_deep_scrub":"261'316","last_deep_scrub_stamp":"2024-03-09T15:36:33.480185+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.480185+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":300,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:48:54.047973+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16427,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":565,"num_read_kb":1160,"num_write":446,"num_write_kb":361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22450,"num_omap_keys":42,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:21.679444+0000","last_change":"2024-03-09T15:36:21.679497+0000","last_active":"2024-03-09T15:36:21.679444+0000","last_peered":"2024-03-09T15:36:21.679444+0000","last_clean":"2024-03-09T15:36:21.679444+0000","last_became_active":"2024-03-09T14:28:40.378829+0000","last_became_peered":"2024-03-09T14:28:40.378829+0000","last_unstale":"2024-03-09T15:36:21.679444+0000","last_undegraded":"2024-03-09T15:36:21.679444+0000","last_fullsized":"2024-03-09T15:36:21.679444+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:21.679245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:21.679245+0000","last_clean_scrub_stamp":"2024-03-09T15:36:21.679245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:45:45.724495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"261'126","reported_seq":635,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:26.487636+0000","last_change":"2024-03-09T15:36:26.487767+0000","last_active":"2024-03-09T15:36:26.487636+0000","last_peered":"2024-03-09T15:36:26.487636+0000","last_clean":"2024-03-09T15:36:26.487636+0000","last_became_active":"2024-03-09T14:28:42.392509+0000","last_became_peered":"2024-03-09T14:28:42.392509+0000","last_unstale":"2024-03-09T15:36:26.487636+0000","last_undegraded":"2024-03-09T15:36:26.487636+0000","last_fullsized":"2024-03-09T15:36:26.487636+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'126","last_scrub_stamp":"2024-03-09T15:36:26.487384+0000","last_deep_scrub":"261'126","last_deep_scrub_stamp":"2024-03-09T15:36:26.487384+0000","last_clean_scrub_stamp":"2024-03-09T15:36:26.487384+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":100,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:33:29.950896+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":643,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":173,"num_write":275,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"261'132","reported_seq":694,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.361023+0000","last_change":"2024-03-09T15:36:45.361023+0000","last_active":"2024-03-09T15:36:45.361023+0000","last_peered":"2024-03-09T15:36:45.361023+0000","last_clean":"2024-03-09T15:36:45.361023+0000","last_became_active":"2024-03-09T14:28:34.330686+0000","last_became_peered":"2024-03-09T14:28:34.330686+0000","last_unstale":"2024-03-09T15:36:45.361023+0000","last_undegraded":"2024-03-09T15:36:45.361023+0000","last_fullsized":"2024-03-09T15:36:45.361023+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'132","last_scrub_stamp":"2024-03-09T15:36:45.360646+0000","last_deep_scrub":"261'132","last_deep_scrub_stamp":"2024-03-09T15:36:45.360646+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.360646+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":100,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:37:31.175012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":527,"num_read_kb":451,"num_write":212,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"261'143","reported_seq":691,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.218846+0000","last_change":"2024-03-09T15:36:42.218957+0000","last_active":"2024-03-09T15:36:42.218846+0000","last_peered":"2024-03-09T15:36:42.218846+0000","last_clean":"2024-03-09T15:36:42.218846+0000","last_became_active":"2024-03-09T14:28:32.301087+0000","last_became_peered":"2024-03-09T14:28:32.301087+0000","last_unstale":"2024-03-09T15:36:42.218846+0000","last_undegraded":"2024-03-09T15:36:42.218846+0000","last_fullsized":"2024-03-09T15:36:42.218846+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'143","last_scrub_stamp":"2024-03-09T15:36:42.218633+0000","last_deep_scrub":"261'143","last_deep_scrub_stamp":"2024-03-09T15:36:42.218633+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.218633+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":100,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:38:14.730766+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":137,"num_read_kb":3393,"num_write":460,"num_write_kb":23270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":515,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.130851+0000","last_change":"2024-03-09T15:36:23.130851+0000","last_active":"2024-03-09T15:36:23.130851+0000","last_peered":"2024-03-09T15:36:23.130851+0000","last_clean":"2024-03-09T15:36:23.130851+0000","last_became_active":"2024-03-09T14:28:36.328814+0000","last_became_peered":"2024-03-09T14:28:36.328814+0000","last_unstale":"2024-03-09T15:36:23.130851+0000","last_undegraded":"2024-03-09T15:36:23.130851+0000","last_fullsized":"2024-03-09T15:36:23.130851+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:23.130603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:23.130603+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.130603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:54:38.685950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"261'260","reported_seq":873,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.173698+0000","last_change":"2024-03-09T15:36:47.173796+0000","last_active":"2024-03-09T15:36:47.173698+0000","last_peered":"2024-03-09T15:36:47.173698+0000","last_clean":"2024-03-09T15:36:47.173698+0000","last_became_active":"2024-03-09T14:28:38.331930+0000","last_became_peered":"2024-03-09T14:28:38.331930+0000","last_unstale":"2024-03-09T15:36:47.173698+0000","last_undegraded":"2024-03-09T15:36:47.173698+0000","last_fullsized":"2024-03-09T15:36:47.173698+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'260","last_scrub_stamp":"2024-03-09T15:36:47.173452+0000","last_deep_scrub":"261'260","last_deep_scrub_stamp":"2024-03-09T15:36:47.173452+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.173452+0000","objects_scrubbed":8,"log_size":60,"log_dups_size":200,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:18:24.859389+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21830,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":424,"num_read_kb":798,"num_write":342,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11460,"num_omap_keys":24,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.208618+0000","last_change":"2024-03-09T15:36:47.208675+0000","last_active":"2024-03-09T15:36:47.208618+0000","last_peered":"2024-03-09T15:36:47.208618+0000","last_clean":"2024-03-09T15:36:47.208618+0000","last_became_active":"2024-03-09T14:28:40.366723+0000","last_became_peered":"2024-03-09T14:28:40.366723+0000","last_unstale":"2024-03-09T15:36:47.208618+0000","last_undegraded":"2024-03-09T15:36:47.208618+0000","last_fullsized":"2024-03-09T15:36:47.208618+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:47.208400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:47.208400+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.208400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:22:48.196708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"261'112","reported_seq":627,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.220706+0000","last_change":"2024-03-09T15:36:32.220786+0000","last_active":"2024-03-09T15:36:32.220706+0000","last_peered":"2024-03-09T15:36:32.220706+0000","last_clean":"2024-03-09T15:36:32.220706+0000","last_became_active":"2024-03-09T14:28:42.395375+0000","last_became_peered":"2024-03-09T14:28:42.395375+0000","last_unstale":"2024-03-09T15:36:32.220706+0000","last_undegraded":"2024-03-09T15:36:32.220706+0000","last_fullsized":"2024-03-09T15:36:32.220706+0000","mapping_epoch":30,"log_start":"261'100","ondisk_log_start":"261'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'112","last_scrub_stamp":"2024-03-09T15:36:32.220503+0000","last_deep_scrub":"261'112","last_deep_scrub_stamp":"2024-03-09T15:36:32.220503+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.220503+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":100,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:58:59.546341+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1199,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":139,"num_read_kb":136,"num_write":341,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"261'99","reported_seq":695,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:52.760381+0000","last_change":"2024-03-09T15:36:52.760444+0000","last_active":"2024-03-09T15:36:52.760381+0000","last_peered":"2024-03-09T15:36:52.760381+0000","last_clean":"2024-03-09T15:36:52.760381+0000","last_became_active":"2024-03-09T14:28:34.312297+0000","last_became_peered":"2024-03-09T14:28:34.312297+0000","last_unstale":"2024-03-09T15:36:52.760381+0000","last_undegraded":"2024-03-09T15:36:52.760381+0000","last_fullsized":"2024-03-09T15:36:52.760381+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'99","last_scrub_stamp":"2024-03-09T15:36:52.760182+0000","last_deep_scrub":"261'99","last_deep_scrub_stamp":"2024-03-09T15:36:52.760182+0000","last_clean_scrub_stamp":"2024-03-09T15:36:52.760182+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:34:46.696922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":415,"num_read_kb":347,"num_write":146,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"261'120","reported_seq":697,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.470441+0000","last_change":"2024-03-09T15:36:49.470646+0000","last_active":"2024-03-09T15:36:49.470441+0000","last_peered":"2024-03-09T15:36:49.470441+0000","last_clean":"2024-03-09T15:36:49.470441+0000","last_became_active":"2024-03-09T14:28:32.312937+0000","last_became_peered":"2024-03-09T14:28:32.312937+0000","last_unstale":"2024-03-09T15:36:49.470441+0000","last_undegraded":"2024-03-09T15:36:49.470441+0000","last_fullsized":"2024-03-09T15:36:49.470441+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'120","last_scrub_stamp":"2024-03-09T15:36:49.470219+0000","last_deep_scrub":"261'120","last_deep_scrub_stamp":"2024-03-09T15:36:49.470219+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.470219+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:58:33.393108+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262175,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":159,"num_read_kb":47424,"num_write":430,"num_write_kb":24173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.273419+0000","last_change":"2024-03-09T15:36:41.273535+0000","last_active":"2024-03-09T15:36:41.273419+0000","last_peered":"2024-03-09T15:36:41.273419+0000","last_clean":"2024-03-09T15:36:41.273419+0000","last_became_active":"2024-03-09T14:28:36.316094+0000","last_became_peered":"2024-03-09T14:28:36.316094+0000","last_unstale":"2024-03-09T15:36:41.273419+0000","last_undegraded":"2024-03-09T15:36:41.273419+0000","last_fullsized":"2024-03-09T15:36:41.273419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:41.273162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:41.273162+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.273162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:10:54.703371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"261'195","reported_seq":791,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.519208+0000","last_change":"2024-03-09T15:36:36.519208+0000","last_active":"2024-03-09T15:36:36.519208+0000","last_peered":"2024-03-09T15:36:36.519208+0000","last_clean":"2024-03-09T15:36:36.519208+0000","last_became_active":"2024-03-09T14:28:38.595522+0000","last_became_peered":"2024-03-09T14:28:38.595522+0000","last_unstale":"2024-03-09T15:36:36.519208+0000","last_undegraded":"2024-03-09T15:36:36.519208+0000","last_fullsized":"2024-03-09T15:36:36.519208+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'195","last_scrub_stamp":"2024-03-09T15:36:36.518818+0000","last_deep_scrub":"261'195","last_deep_scrub_stamp":"2024-03-09T15:36:36.518818+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.518818+0000","objects_scrubbed":5,"log_size":95,"log_dups_size":100,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:00:11.684059+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12058,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":311,"num_read_kb":573,"num_write":216,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":509,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.585690+0000","last_change":"2024-03-09T15:36:22.585690+0000","last_active":"2024-03-09T15:36:22.585690+0000","last_peered":"2024-03-09T15:36:22.585690+0000","last_clean":"2024-03-09T15:36:22.585690+0000","last_became_active":"2024-03-09T14:28:40.376549+0000","last_became_peered":"2024-03-09T14:28:40.376549+0000","last_unstale":"2024-03-09T15:36:22.585690+0000","last_undegraded":"2024-03-09T15:36:22.585690+0000","last_fullsized":"2024-03-09T15:36:22.585690+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.585447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.585447+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.585447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:45:35.078062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"261'45","reported_seq":550,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:19.666451+0000","last_change":"2024-03-09T15:36:19.666451+0000","last_active":"2024-03-09T15:36:19.666451+0000","last_peered":"2024-03-09T15:36:19.666451+0000","last_clean":"2024-03-09T15:36:19.666451+0000","last_became_active":"2024-03-09T14:28:42.394780+0000","last_became_peered":"2024-03-09T14:28:42.394780+0000","last_unstale":"2024-03-09T15:36:19.666451+0000","last_undegraded":"2024-03-09T15:36:19.666451+0000","last_fullsized":"2024-03-09T15:36:19.666451+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'45","last_scrub_stamp":"2024-03-09T15:36:19.666178+0000","last_deep_scrub":"261'45","last_deep_scrub_stamp":"2024-03-09T15:36:19.666178+0000","last_clean_scrub_stamp":"2024-03-09T15:36:19.666178+0000","objects_scrubbed":3,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:53:32.256553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":84,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"261'68","reported_seq":624,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:32.190901+0000","last_change":"2024-03-09T15:36:32.190971+0000","last_active":"2024-03-09T15:36:32.190901+0000","last_peered":"2024-03-09T15:36:32.190901+0000","last_clean":"2024-03-09T15:36:32.190901+0000","last_became_active":"2024-03-09T14:28:34.313125+0000","last_became_peered":"2024-03-09T14:28:34.313125+0000","last_unstale":"2024-03-09T15:36:32.190901+0000","last_undegraded":"2024-03-09T15:36:32.190901+0000","last_fullsized":"2024-03-09T15:36:32.190901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'68","last_scrub_stamp":"2024-03-09T15:36:32.190702+0000","last_deep_scrub":"261'68","last_deep_scrub_stamp":"2024-03-09T15:36:32.190702+0000","last_clean_scrub_stamp":"2024-03-09T15:36:32.190702+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:14:08.005357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"261'191","reported_seq":804,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:03.136315+0000","last_change":"2024-03-09T15:37:03.136404+0000","last_active":"2024-03-09T15:37:03.136315+0000","last_peered":"2024-03-09T15:37:03.136315+0000","last_clean":"2024-03-09T15:37:03.136315+0000","last_became_active":"2024-03-09T14:28:32.303835+0000","last_became_peered":"2024-03-09T14:28:32.303835+0000","last_unstale":"2024-03-09T15:37:03.136315+0000","last_undegraded":"2024-03-09T15:37:03.136315+0000","last_fullsized":"2024-03-09T15:37:03.136315+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'191","last_scrub_stamp":"2024-03-09T15:37:03.136127+0000","last_deep_scrub":"261'191","last_deep_scrub_stamp":"2024-03-09T15:37:03.136127+0000","last_clean_scrub_stamp":"2024-03-09T15:37:03.136127+0000","objects_scrubbed":4,"log_size":91,"log_dups_size":100,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:18:30.887965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":371,"num_read_kb":1431,"num_write":999,"num_write_kb":8832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":510,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.488481+0000","last_change":"2024-03-09T15:36:29.488628+0000","last_active":"2024-03-09T15:36:29.488481+0000","last_peered":"2024-03-09T15:36:29.488481+0000","last_clean":"2024-03-09T15:36:29.488481+0000","last_became_active":"2024-03-09T14:28:36.322060+0000","last_became_peered":"2024-03-09T14:28:36.322060+0000","last_unstale":"2024-03-09T15:36:29.488481+0000","last_undegraded":"2024-03-09T15:36:29.488481+0000","last_fullsized":"2024-03-09T15:36:29.488481+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:29.488248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:29.488248+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.488248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:40:34.293730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"261'96","reported_seq":647,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.151474+0000","last_change":"2024-03-09T15:36:24.151474+0000","last_active":"2024-03-09T15:36:24.151474+0000","last_peered":"2024-03-09T15:36:24.151474+0000","last_clean":"2024-03-09T15:36:24.151474+0000","last_became_active":"2024-03-09T14:28:38.595749+0000","last_became_peered":"2024-03-09T14:28:38.595749+0000","last_unstale":"2024-03-09T15:36:24.151474+0000","last_undegraded":"2024-03-09T15:36:24.151474+0000","last_fullsized":"2024-03-09T15:36:24.151474+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'96","last_scrub_stamp":"2024-03-09T15:36:24.151188+0000","last_deep_scrub":"261'96","last_deep_scrub_stamp":"2024-03-09T15:36:24.151188+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.151188+0000","objects_scrubbed":7,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:06:38.216926+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":132,"num_read_kb":127,"num_write":124,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11222,"num_omap_keys":30,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"261'489","reported_seq":2267,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.169035+0000","last_change":"2024-03-09T15:36:28.169035+0000","last_active":"2024-03-09T15:36:28.169035+0000","last_peered":"2024-03-09T15:36:28.169035+0000","last_clean":"2024-03-09T15:36:28.169035+0000","last_became_active":"2024-03-09T14:28:40.376946+0000","last_became_peered":"2024-03-09T14:28:40.376946+0000","last_unstale":"2024-03-09T15:36:28.169035+0000","last_undegraded":"2024-03-09T15:36:28.169035+0000","last_fullsized":"2024-03-09T15:36:28.169035+0000","mapping_epoch":28,"log_start":"258'400","ondisk_log_start":"258'400","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'489","last_scrub_stamp":"2024-03-09T15:36:28.168784+0000","last_deep_scrub":"261'489","last_deep_scrub_stamp":"2024-03-09T15:36:28.168784+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.168784+0000","objects_scrubbed":1,"log_size":89,"log_dups_size":400,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:26:51.629900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.10","version":"261'45","reported_seq":559,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.174006+0000","last_change":"2024-03-09T15:36:31.174074+0000","last_active":"2024-03-09T15:36:31.174006+0000","last_peered":"2024-03-09T15:36:31.174006+0000","last_clean":"2024-03-09T15:36:31.174006+0000","last_became_active":"2024-03-09T14:28:42.397829+0000","last_became_peered":"2024-03-09T14:28:42.397829+0000","last_unstale":"2024-03-09T15:36:31.174006+0000","last_undegraded":"2024-03-09T15:36:31.174006+0000","last_fullsized":"2024-03-09T15:36:31.174006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'45","last_scrub_stamp":"2024-03-09T15:36:31.173791+0000","last_deep_scrub":"261'45","last_deep_scrub_stamp":"2024-03-09T15:36:31.173791+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.173791+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:08:44.989801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":97,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"261'144","reported_seq":684,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.216346+0000","last_change":"2024-03-09T15:36:37.216431+0000","last_active":"2024-03-09T15:36:37.216346+0000","last_peered":"2024-03-09T15:36:37.216346+0000","last_clean":"2024-03-09T15:36:37.216346+0000","last_became_active":"2024-03-09T14:28:34.319891+0000","last_became_peered":"2024-03-09T14:28:34.319891+0000","last_unstale":"2024-03-09T15:36:37.216346+0000","last_undegraded":"2024-03-09T15:36:37.216346+0000","last_fullsized":"2024-03-09T15:36:37.216346+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'144","last_scrub_stamp":"2024-03-09T15:36:37.216142+0000","last_deep_scrub":"261'144","last_deep_scrub_stamp":"2024-03-09T15:36:37.216142+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.216142+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:10:29.932991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":571,"num_read_kb":433,"num_write":263,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"261'165","reported_seq":820,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.905667+0000","last_change":"2024-03-09T15:36:39.905809+0000","last_active":"2024-03-09T15:36:39.905667+0000","last_peered":"2024-03-09T15:36:39.905667+0000","last_clean":"2024-03-09T15:36:39.905667+0000","last_became_active":"2024-03-09T14:28:32.323198+0000","last_became_peered":"2024-03-09T14:28:32.323198+0000","last_unstale":"2024-03-09T15:36:39.905667+0000","last_undegraded":"2024-03-09T15:36:39.905667+0000","last_fullsized":"2024-03-09T15:36:39.905667+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:39.905460+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:39.905460+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.905460+0000","objects_scrubbed":4,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:16:12.321789+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4801550,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":406,"num_read_kb":2095,"num_write":652,"num_write_kb":25158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.154703+0000","last_change":"2024-03-09T15:36:45.154756+0000","last_active":"2024-03-09T15:36:45.154703+0000","last_peered":"2024-03-09T15:36:45.154703+0000","last_clean":"2024-03-09T15:36:45.154703+0000","last_became_active":"2024-03-09T14:28:36.328529+0000","last_became_peered":"2024-03-09T14:28:36.328529+0000","last_unstale":"2024-03-09T15:36:45.154703+0000","last_undegraded":"2024-03-09T15:36:45.154703+0000","last_fullsized":"2024-03-09T15:36:45.154703+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:45.154500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:45.154500+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.154500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:30:04.214748+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"261'231","reported_seq":843,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:52.869145+0000","last_change":"2024-03-09T15:36:52.869321+0000","last_active":"2024-03-09T15:36:52.869145+0000","last_peered":"2024-03-09T15:36:52.869145+0000","last_clean":"2024-03-09T15:36:52.869145+0000","last_became_active":"2024-03-09T14:28:38.334991+0000","last_became_peered":"2024-03-09T14:28:38.334991+0000","last_unstale":"2024-03-09T15:36:52.869145+0000","last_undegraded":"2024-03-09T15:36:52.869145+0000","last_fullsized":"2024-03-09T15:36:52.869145+0000","mapping_epoch":26,"log_start":"261'200","ondisk_log_start":"261'200","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'231","last_scrub_stamp":"2024-03-09T15:36:52.868879+0000","last_deep_scrub":"261'231","last_deep_scrub_stamp":"2024-03-09T15:36:52.868879+0000","last_clean_scrub_stamp":"2024-03-09T15:36:52.868879+0000","objects_scrubbed":7,"log_size":31,"log_dups_size":200,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:38:44.778956+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13314,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":373,"num_read_kb":687,"num_write":264,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.226908+0000","last_change":"2024-03-09T15:36:33.226963+0000","last_active":"2024-03-09T15:36:33.226908+0000","last_peered":"2024-03-09T15:36:33.226908+0000","last_clean":"2024-03-09T15:36:33.226908+0000","last_became_active":"2024-03-09T14:28:40.368039+0000","last_became_peered":"2024-03-09T14:28:40.368039+0000","last_unstale":"2024-03-09T15:36:33.226908+0000","last_undegraded":"2024-03-09T15:36:33.226908+0000","last_fullsized":"2024-03-09T15:36:33.226908+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:33.226717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:33.226717+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.226717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:03:17.480958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"261'79","reported_seq":609,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.203923+0000","last_change":"2024-03-09T15:36:28.203997+0000","last_active":"2024-03-09T15:36:28.203923+0000","last_peered":"2024-03-09T15:36:28.203923+0000","last_clean":"2024-03-09T15:36:28.203923+0000","last_became_active":"2024-03-09T14:28:42.413840+0000","last_became_peered":"2024-03-09T14:28:42.413840+0000","last_unstale":"2024-03-09T15:36:28.203923+0000","last_undegraded":"2024-03-09T15:36:28.203923+0000","last_fullsized":"2024-03-09T15:36:28.203923+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'79","last_scrub_stamp":"2024-03-09T15:36:28.203746+0000","last_deep_scrub":"261'79","last_deep_scrub_stamp":"2024-03-09T15:36:28.203746+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.203746+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:20:00.519578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":129,"num_write":187,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"261'36","reported_seq":582,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.213368+0000","last_change":"2024-03-09T15:36:35.213428+0000","last_active":"2024-03-09T15:36:35.213368+0000","last_peered":"2024-03-09T15:36:35.213368+0000","last_clean":"2024-03-09T15:36:35.213368+0000","last_became_active":"2024-03-09T14:28:34.314194+0000","last_became_peered":"2024-03-09T14:28:34.314194+0000","last_unstale":"2024-03-09T15:36:35.213368+0000","last_undegraded":"2024-03-09T15:36:35.213368+0000","last_fullsized":"2024-03-09T15:36:35.213368+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'36","last_scrub_stamp":"2024-03-09T15:36:35.213167+0000","last_deep_scrub":"261'36","last_deep_scrub_stamp":"2024-03-09T15:36:35.213167+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.213167+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T15:48:30.268110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":120,"num_write":54,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"261'150","reported_seq":812,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:50.865057+0000","last_change":"2024-03-09T15:36:50.865141+0000","last_active":"2024-03-09T15:36:50.865057+0000","last_peered":"2024-03-09T15:36:50.865057+0000","last_clean":"2024-03-09T15:36:50.865057+0000","last_became_active":"2024-03-09T14:28:32.307702+0000","last_became_peered":"2024-03-09T14:28:32.307702+0000","last_unstale":"2024-03-09T15:36:50.865057+0000","last_undegraded":"2024-03-09T15:36:50.865057+0000","last_fullsized":"2024-03-09T15:36:50.865057+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'150","last_scrub_stamp":"2024-03-09T15:36:50.864867+0000","last_deep_scrub":"261'150","last_deep_scrub_stamp":"2024-03-09T15:36:50.864867+0000","last_clean_scrub_stamp":"2024-03-09T15:36:50.864867+0000","objects_scrubbed":3,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:16:58.825247+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":273,"num_read_kb":1758,"num_write":483,"num_write_kb":23078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":512,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.746052+0000","last_change":"2024-03-09T15:36:34.746107+0000","last_active":"2024-03-09T15:36:34.746052+0000","last_peered":"2024-03-09T15:36:34.746052+0000","last_clean":"2024-03-09T15:36:34.746052+0000","last_became_active":"2024-03-09T14:28:36.304241+0000","last_became_peered":"2024-03-09T14:28:36.304241+0000","last_unstale":"2024-03-09T15:36:34.746052+0000","last_undegraded":"2024-03-09T15:36:34.746052+0000","last_fullsized":"2024-03-09T15:36:34.746052+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:34.745840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:34.745840+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.745840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:07:47.752362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"261'182","reported_seq":771,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:38.223598+0000","last_change":"2024-03-09T15:36:38.223787+0000","last_active":"2024-03-09T15:36:38.223598+0000","last_peered":"2024-03-09T15:36:38.223598+0000","last_clean":"2024-03-09T15:36:38.223598+0000","last_became_active":"2024-03-09T14:28:38.338794+0000","last_became_peered":"2024-03-09T14:28:38.338794+0000","last_unstale":"2024-03-09T15:36:38.223598+0000","last_undegraded":"2024-03-09T15:36:38.223598+0000","last_fullsized":"2024-03-09T15:36:38.223598+0000","mapping_epoch":26,"log_start":"258'100","ondisk_log_start":"258'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'182","last_scrub_stamp":"2024-03-09T15:36:38.223354+0000","last_deep_scrub":"261'182","last_deep_scrub_stamp":"2024-03-09T15:36:38.223354+0000","last_clean_scrub_stamp":"2024-03-09T15:36:38.223354+0000","objects_scrubbed":4,"log_size":82,"log_dups_size":100,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:01:53.396420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":278,"num_read_kb":521,"num_write":190,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":502,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.171034+0000","last_change":"2024-03-09T15:36:24.171034+0000","last_active":"2024-03-09T15:36:24.171034+0000","last_peered":"2024-03-09T15:36:24.171034+0000","last_clean":"2024-03-09T15:36:24.171034+0000","last_became_active":"2024-03-09T14:28:40.371820+0000","last_became_peered":"2024-03-09T14:28:40.371820+0000","last_unstale":"2024-03-09T15:36:24.171034+0000","last_undegraded":"2024-03-09T15:36:24.171034+0000","last_fullsized":"2024-03-09T15:36:24.171034+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:24.170785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:24.170785+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.170785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:48:51.479283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.16","version":"261'56","reported_seq":580,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:28.501906+0000","last_change":"2024-03-09T15:36:28.501999+0000","last_active":"2024-03-09T15:36:28.501906+0000","last_peered":"2024-03-09T15:36:28.501906+0000","last_clean":"2024-03-09T15:36:28.501906+0000","last_became_active":"2024-03-09T14:28:42.392326+0000","last_became_peered":"2024-03-09T14:28:42.392326+0000","last_unstale":"2024-03-09T15:36:28.501906+0000","last_undegraded":"2024-03-09T15:36:28.501906+0000","last_fullsized":"2024-03-09T15:36:28.501906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'56","last_scrub_stamp":"2024-03-09T15:36:28.501691+0000","last_deep_scrub":"261'56","last_deep_scrub_stamp":"2024-03-09T15:36:28.501691+0000","last_clean_scrub_stamp":"2024-03-09T15:36:28.501691+0000","objects_scrubbed":3,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:17:33.874116+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":772,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":99,"num_write":117,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"261'70","reported_seq":642,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.589053+0000","last_change":"2024-03-09T15:36:44.589114+0000","last_active":"2024-03-09T15:36:44.589053+0000","last_peered":"2024-03-09T15:36:44.589053+0000","last_clean":"2024-03-09T15:36:44.589053+0000","last_became_active":"2024-03-09T14:28:34.323969+0000","last_became_peered":"2024-03-09T14:28:34.323969+0000","last_unstale":"2024-03-09T15:36:44.589053+0000","last_undegraded":"2024-03-09T15:36:44.589053+0000","last_fullsized":"2024-03-09T15:36:44.589053+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'70","last_scrub_stamp":"2024-03-09T15:36:44.588877+0000","last_deep_scrub":"261'70","last_deep_scrub_stamp":"2024-03-09T15:36:44.588877+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.588877+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:27:16.122474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":256,"num_write":98,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"261'183","reported_seq":784,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:54.782043+0000","last_change":"2024-03-09T15:36:54.782180+0000","last_active":"2024-03-09T15:36:54.782043+0000","last_peered":"2024-03-09T15:36:54.782043+0000","last_clean":"2024-03-09T15:36:54.782043+0000","last_became_active":"2024-03-09T14:28:32.337755+0000","last_became_peered":"2024-03-09T14:28:32.337755+0000","last_unstale":"2024-03-09T15:36:54.782043+0000","last_undegraded":"2024-03-09T15:36:54.782043+0000","last_fullsized":"2024-03-09T15:36:54.782043+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'183","last_scrub_stamp":"2024-03-09T15:36:54.781815+0000","last_deep_scrub":"261'183","last_deep_scrub_stamp":"2024-03-09T15:36:54.781815+0000","last_clean_scrub_stamp":"2024-03-09T15:36:54.781815+0000","objects_scrubbed":7,"log_size":83,"log_dups_size":100,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:51:13.814026+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663535,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":299,"num_read_kb":3602,"num_write":667,"num_write_kb":11566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":514,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.298938+0000","last_change":"2024-03-09T15:36:43.298995+0000","last_active":"2024-03-09T15:36:43.298938+0000","last_peered":"2024-03-09T15:36:43.298938+0000","last_clean":"2024-03-09T15:36:43.298938+0000","last_became_active":"2024-03-09T14:28:36.317569+0000","last_became_peered":"2024-03-09T14:28:36.317569+0000","last_unstale":"2024-03-09T15:36:43.298938+0000","last_undegraded":"2024-03-09T15:36:43.298938+0000","last_fullsized":"2024-03-09T15:36:43.298938+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:43.298743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:43.298743+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.298743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:54:29.675835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"261'154","reported_seq":723,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.559070+0000","last_change":"2024-03-09T15:36:39.559219+0000","last_active":"2024-03-09T15:36:39.559070+0000","last_peered":"2024-03-09T15:36:39.559070+0000","last_clean":"2024-03-09T15:36:39.559070+0000","last_became_active":"2024-03-09T14:28:38.347370+0000","last_became_peered":"2024-03-09T14:28:38.347370+0000","last_unstale":"2024-03-09T15:36:39.559070+0000","last_undegraded":"2024-03-09T15:36:39.559070+0000","last_fullsized":"2024-03-09T15:36:39.559070+0000","mapping_epoch":26,"log_start":"261'100","ondisk_log_start":"261'100","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'154","last_scrub_stamp":"2024-03-09T15:36:39.558842+0000","last_deep_scrub":"261'154","last_deep_scrub_stamp":"2024-03-09T15:36:39.558842+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.558842+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":100,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:06:29.281642+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18257,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":531,"num_write":172,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":505,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:22.105659+0000","last_change":"2024-03-09T15:36:22.105715+0000","last_active":"2024-03-09T15:36:22.105659+0000","last_peered":"2024-03-09T15:36:22.105659+0000","last_clean":"2024-03-09T15:36:22.105659+0000","last_became_active":"2024-03-09T14:28:40.376513+0000","last_became_peered":"2024-03-09T14:28:40.376513+0000","last_unstale":"2024-03-09T15:36:22.105659+0000","last_undegraded":"2024-03-09T15:36:22.105659+0000","last_fullsized":"2024-03-09T15:36:22.105659+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:22.105441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:22.105441+0000","last_clean_scrub_stamp":"2024-03-09T15:36:22.105441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:26:26.970816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"261'96","reported_seq":988,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.177168+0000","last_change":"2024-03-09T15:36:30.177269+0000","last_active":"2024-03-09T15:36:30.177168+0000","last_peered":"2024-03-09T15:36:30.177168+0000","last_clean":"2024-03-09T15:36:30.177168+0000","last_became_active":"2024-03-09T14:28:42.409717+0000","last_became_peered":"2024-03-09T14:28:42.409717+0000","last_unstale":"2024-03-09T15:36:30.177168+0000","last_undegraded":"2024-03-09T15:36:30.177168+0000","last_fullsized":"2024-03-09T15:36:30.177168+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'96","last_scrub_stamp":"2024-03-09T15:36:30.176968+0000","last_deep_scrub":"261'96","last_deep_scrub_stamp":"2024-03-09T15:36:30.176968+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.176968+0000","objects_scrubbed":4,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:07:18.249596+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":500,"num_read_kb":493,"num_write":215,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"261'53","reported_seq":595,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.270125+0000","last_change":"2024-03-09T15:36:40.270186+0000","last_active":"2024-03-09T15:36:40.270125+0000","last_peered":"2024-03-09T15:36:40.270125+0000","last_clean":"2024-03-09T15:36:40.270125+0000","last_became_active":"2024-03-09T14:28:34.313380+0000","last_became_peered":"2024-03-09T14:28:34.313380+0000","last_unstale":"2024-03-09T15:36:40.270125+0000","last_undegraded":"2024-03-09T15:36:40.270125+0000","last_fullsized":"2024-03-09T15:36:40.270125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'53","last_scrub_stamp":"2024-03-09T15:36:40.269935+0000","last_deep_scrub":"261'53","last_deep_scrub_stamp":"2024-03-09T15:36:40.269935+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.269935+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:22:56.243161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":197,"num_read_kb":156,"num_write":74,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"261'164","reported_seq":761,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:59.111059+0000","last_change":"2024-03-09T15:36:59.111059+0000","last_active":"2024-03-09T15:36:59.111059+0000","last_peered":"2024-03-09T15:36:59.111059+0000","last_clean":"2024-03-09T15:36:59.111059+0000","last_became_active":"2024-03-09T14:28:32.292862+0000","last_became_peered":"2024-03-09T14:28:32.292862+0000","last_unstale":"2024-03-09T15:36:59.111059+0000","last_undegraded":"2024-03-09T15:36:59.111059+0000","last_fullsized":"2024-03-09T15:36:59.111059+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'164","last_scrub_stamp":"2024-03-09T15:36:59.110646+0000","last_deep_scrub":"261'164","last_deep_scrub_stamp":"2024-03-09T15:36:59.110646+0000","last_clean_scrub_stamp":"2024-03-09T15:36:59.110646+0000","objects_scrubbed":3,"log_size":64,"log_dups_size":100,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:05:14.848338+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":264,"num_read_kb":35693,"num_write":668,"num_write_kb":24072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"261'2","reported_seq":513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:23.597930+0000","last_change":"2024-03-09T15:36:23.597930+0000","last_active":"2024-03-09T15:36:23.597930+0000","last_peered":"2024-03-09T15:36:23.597930+0000","last_clean":"2024-03-09T15:36:23.597930+0000","last_became_active":"2024-03-09T14:28:36.313405+0000","last_became_peered":"2024-03-09T14:28:36.313405+0000","last_unstale":"2024-03-09T15:36:23.597930+0000","last_undegraded":"2024-03-09T15:36:23.597930+0000","last_fullsized":"2024-03-09T15:36:23.597930+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'2","last_scrub_stamp":"2024-03-09T15:36:23.597678+0000","last_deep_scrub":"261'2","last_deep_scrub_stamp":"2024-03-09T15:36:23.597678+0000","last_clean_scrub_stamp":"2024-03-09T15:36:23.597678+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:11:34.369475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"261'213","reported_seq":819,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.171733+0000","last_change":"2024-03-09T15:36:30.171809+0000","last_active":"2024-03-09T15:36:30.171733+0000","last_peered":"2024-03-09T15:36:30.171733+0000","last_clean":"2024-03-09T15:36:30.171733+0000","last_became_active":"2024-03-09T14:28:38.330571+0000","last_became_peered":"2024-03-09T14:28:38.330571+0000","last_unstale":"2024-03-09T15:36:30.171733+0000","last_undegraded":"2024-03-09T15:36:30.171733+0000","last_fullsized":"2024-03-09T15:36:30.171733+0000","mapping_epoch":26,"log_start":"261'201","ondisk_log_start":"261'201","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'213","last_scrub_stamp":"2024-03-09T15:36:30.171534+0000","last_deep_scrub":"261'213","last_deep_scrub_stamp":"2024-03-09T15:36:30.171534+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.171534+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":201,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:00:50.639816+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11093,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":349,"num_read_kb":659,"num_write":246,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":501,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.213225+0000","last_change":"2024-03-09T15:36:29.213283+0000","last_active":"2024-03-09T15:36:29.213225+0000","last_peered":"2024-03-09T15:36:29.213225+0000","last_clean":"2024-03-09T15:36:29.213225+0000","last_became_active":"2024-03-09T14:28:40.385080+0000","last_became_peered":"2024-03-09T14:28:40.385080+0000","last_unstale":"2024-03-09T15:36:29.213225+0000","last_undegraded":"2024-03-09T15:36:29.213225+0000","last_fullsized":"2024-03-09T15:36:29.213225+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T15:36:29.213014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T15:36:29.213014+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.213014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:01:47.618516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"261'69","reported_seq":573,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:15.140935+0000","last_change":"2024-03-09T15:36:15.140935+0000","last_active":"2024-03-09T15:36:15.140935+0000","last_peered":"2024-03-09T15:36:15.140935+0000","last_clean":"2024-03-09T15:36:15.140935+0000","last_became_active":"2024-03-09T14:28:42.412616+0000","last_became_peered":"2024-03-09T14:28:42.412616+0000","last_unstale":"2024-03-09T15:36:15.140935+0000","last_undegraded":"2024-03-09T15:36:15.140935+0000","last_fullsized":"2024-03-09T15:36:15.140935+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'69","last_scrub_stamp":"2024-03-09T15:36:15.140656+0000","last_deep_scrub":"261'69","last_deep_scrub_stamp":"2024-03-09T15:36:15.140656+0000","last_clean_scrub_stamp":"2024-03-09T15:36:15.140656+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:26:05.406444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":49,"num_write":163,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.27","version":"261'50","reported_seq":606,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:34.426480+0000","last_change":"2024-03-09T15:36:34.426540+0000","last_active":"2024-03-09T15:36:34.426480+0000","last_peered":"2024-03-09T15:36:34.426480+0000","last_clean":"2024-03-09T15:36:34.426480+0000","last_became_active":"2024-03-09T14:28:34.306685+0000","last_became_peered":"2024-03-09T14:28:34.306685+0000","last_unstale":"2024-03-09T15:36:34.426480+0000","last_undegraded":"2024-03-09T15:36:34.426480+0000","last_fullsized":"2024-03-09T15:36:34.426480+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'50","last_scrub_stamp":"2024-03-09T15:36:34.426288+0000","last_deep_scrub":"261'50","last_deep_scrub_stamp":"2024-03-09T15:36:34.426288+0000","last_clean_scrub_stamp":"2024-03-09T15:36:34.426288+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:49:13.124210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":204,"num_read_kb":164,"num_write":70,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"261'146","reported_seq":739,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:40.283538+0000","last_change":"2024-03-09T15:36:40.283538+0000","last_active":"2024-03-09T15:36:40.283538+0000","last_peered":"2024-03-09T15:36:40.283538+0000","last_clean":"2024-03-09T15:36:40.283538+0000","last_became_active":"2024-03-09T14:28:32.333008+0000","last_became_peered":"2024-03-09T14:28:32.333008+0000","last_unstale":"2024-03-09T15:36:40.283538+0000","last_undegraded":"2024-03-09T15:36:40.283538+0000","last_fullsized":"2024-03-09T15:36:40.283538+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'146","last_scrub_stamp":"2024-03-09T15:36:40.283252+0000","last_deep_scrub":"261'146","last_deep_scrub_stamp":"2024-03-09T15:36:40.283252+0000","last_clean_scrub_stamp":"2024-03-09T15:36:40.283252+0000","objects_scrubbed":6,"log_size":46,"log_dups_size":100,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:58:17.359879+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655838,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":237,"num_read_kb":10390,"num_write":554,"num_write_kb":33266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.26","version":"261'181","reported_seq":781,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.520994+0000","last_change":"2024-03-09T15:36:30.521058+0000","last_active":"2024-03-09T15:36:30.520994+0000","last_peered":"2024-03-09T15:36:30.520994+0000","last_clean":"2024-03-09T15:36:30.520994+0000","last_became_active":"2024-03-09T14:28:34.314007+0000","last_became_peered":"2024-03-09T14:28:34.314007+0000","last_unstale":"2024-03-09T15:36:30.520994+0000","last_undegraded":"2024-03-09T15:36:30.520994+0000","last_fullsized":"2024-03-09T15:36:30.520994+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'181","last_scrub_stamp":"2024-03-09T15:36:30.520807+0000","last_deep_scrub":"261'181","last_deep_scrub_stamp":"2024-03-09T15:36:30.520807+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.520807+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":100,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:13:34.179786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":729,"num_read_kb":586,"num_write":272,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"261'155","reported_seq":714,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:29.244629+0000","last_change":"2024-03-09T15:36:29.244629+0000","last_active":"2024-03-09T15:36:29.244629+0000","last_peered":"2024-03-09T15:36:29.244629+0000","last_clean":"2024-03-09T15:36:29.244629+0000","last_became_active":"2024-03-09T14:28:32.286609+0000","last_became_peered":"2024-03-09T14:28:32.286609+0000","last_unstale":"2024-03-09T15:36:29.244629+0000","last_undegraded":"2024-03-09T15:36:29.244629+0000","last_fullsized":"2024-03-09T15:36:29.244629+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'155","last_scrub_stamp":"2024-03-09T15:36:29.244332+0000","last_deep_scrub":"261'155","last_deep_scrub_stamp":"2024-03-09T15:36:29.244332+0000","last_clean_scrub_stamp":"2024-03-09T15:36:29.244332+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":100,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:53:34.232406+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":810211,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":147,"num_read_kb":1907,"num_write":492,"num_write_kb":32192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"261'94","reported_seq":667,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.704648+0000","last_change":"2024-03-09T15:36:30.704712+0000","last_active":"2024-03-09T15:36:30.704648+0000","last_peered":"2024-03-09T15:36:30.704648+0000","last_clean":"2024-03-09T15:36:30.704648+0000","last_became_active":"2024-03-09T14:28:34.323541+0000","last_became_peered":"2024-03-09T14:28:34.323541+0000","last_unstale":"2024-03-09T15:36:30.704648+0000","last_undegraded":"2024-03-09T15:36:30.704648+0000","last_fullsized":"2024-03-09T15:36:30.704648+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'94","last_scrub_stamp":"2024-03-09T15:36:30.704449+0000","last_deep_scrub":"261'94","last_deep_scrub_stamp":"2024-03-09T15:36:30.704449+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.704449+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:34:11.739435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":384,"num_read_kb":306,"num_write":146,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":230,"num_omap_keys":1,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"261'173","reported_seq":779,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:47.935747+0000","last_change":"2024-03-09T15:36:47.935747+0000","last_active":"2024-03-09T15:36:47.935747+0000","last_peered":"2024-03-09T15:36:47.935747+0000","last_clean":"2024-03-09T15:36:47.935747+0000","last_became_active":"2024-03-09T14:28:32.295699+0000","last_became_peered":"2024-03-09T14:28:32.295699+0000","last_unstale":"2024-03-09T15:36:47.935747+0000","last_undegraded":"2024-03-09T15:36:47.935747+0000","last_fullsized":"2024-03-09T15:36:47.935747+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'173","last_scrub_stamp":"2024-03-09T15:36:47.935356+0000","last_deep_scrub":"261'173","last_deep_scrub_stamp":"2024-03-09T15:36:47.935356+0000","last_clean_scrub_stamp":"2024-03-09T15:36:47.935356+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":100,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:15:12.568313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":297,"num_read_kb":14189,"num_write":716,"num_write_kb":33521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"261'101","reported_seq":676,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:24.552706+0000","last_change":"2024-03-09T15:36:24.552770+0000","last_active":"2024-03-09T15:36:24.552706+0000","last_peered":"2024-03-09T15:36:24.552706+0000","last_clean":"2024-03-09T15:36:24.552706+0000","last_became_active":"2024-03-09T14:28:34.321647+0000","last_became_peered":"2024-03-09T14:28:34.321647+0000","last_unstale":"2024-03-09T15:36:24.552706+0000","last_undegraded":"2024-03-09T15:36:24.552706+0000","last_fullsized":"2024-03-09T15:36:24.552706+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'101","last_scrub_stamp":"2024-03-09T15:36:24.552512+0000","last_deep_scrub":"261'101","last_deep_scrub_stamp":"2024-03-09T15:36:24.552512+0000","last_clean_scrub_stamp":"2024-03-09T15:36:24.552512+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:59:27.278086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":412,"num_read_kb":329,"num_write":161,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"261'137","reported_seq":698,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:00.109163+0000","last_change":"2024-03-09T15:37:00.109387+0000","last_active":"2024-03-09T15:37:00.109163+0000","last_peered":"2024-03-09T15:37:00.109163+0000","last_clean":"2024-03-09T15:37:00.109163+0000","last_became_active":"2024-03-09T14:28:32.297866+0000","last_became_peered":"2024-03-09T14:28:32.297866+0000","last_unstale":"2024-03-09T15:37:00.109163+0000","last_undegraded":"2024-03-09T15:37:00.109163+0000","last_fullsized":"2024-03-09T15:37:00.109163+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'137","last_scrub_stamp":"2024-03-09T15:37:00.108913+0000","last_deep_scrub":"261'137","last_deep_scrub_stamp":"2024-03-09T15:37:00.108913+0000","last_clean_scrub_stamp":"2024-03-09T15:37:00.108913+0000","objects_scrubbed":9,"log_size":37,"log_dups_size":100,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:00:30.085008+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":163,"num_read_kb":3467,"num_write":495,"num_write_kb":23815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"261'52","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:45.607074+0000","last_change":"2024-03-09T15:36:45.607269+0000","last_active":"2024-03-09T15:36:45.607074+0000","last_peered":"2024-03-09T15:36:45.607074+0000","last_clean":"2024-03-09T15:36:45.607074+0000","last_became_active":"2024-03-09T14:28:34.330511+0000","last_became_peered":"2024-03-09T14:28:34.330511+0000","last_unstale":"2024-03-09T15:36:45.607074+0000","last_undegraded":"2024-03-09T15:36:45.607074+0000","last_fullsized":"2024-03-09T15:36:45.607074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'52","last_scrub_stamp":"2024-03-09T15:36:45.606838+0000","last_deep_scrub":"261'52","last_deep_scrub_stamp":"2024-03-09T15:36:45.606838+0000","last_clean_scrub_stamp":"2024-03-09T15:36:45.606838+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T18:14:12.247135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":225,"num_read_kb":185,"num_write":82,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"261'120","reported_seq":678,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:58.760534+0000","last_change":"2024-03-09T15:36:58.760602+0000","last_active":"2024-03-09T15:36:58.760534+0000","last_peered":"2024-03-09T15:36:58.760534+0000","last_clean":"2024-03-09T15:36:58.760534+0000","last_became_active":"2024-03-09T14:28:32.326148+0000","last_became_peered":"2024-03-09T14:28:32.326148+0000","last_unstale":"2024-03-09T15:36:58.760534+0000","last_undegraded":"2024-03-09T15:36:58.760534+0000","last_fullsized":"2024-03-09T15:36:58.760534+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'120","last_scrub_stamp":"2024-03-09T15:36:58.760335+0000","last_deep_scrub":"261'120","last_deep_scrub_stamp":"2024-03-09T15:36:58.760335+0000","last_clean_scrub_stamp":"2024-03-09T15:36:58.760335+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":100,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:25:57.931006+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":173,"num_read_kb":5614,"num_write":451,"num_write_kb":21169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"261'134","reported_seq":695,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:39.224849+0000","last_change":"2024-03-09T15:36:39.224849+0000","last_active":"2024-03-09T15:36:39.224849+0000","last_peered":"2024-03-09T15:36:39.224849+0000","last_clean":"2024-03-09T15:36:39.224849+0000","last_became_active":"2024-03-09T14:28:34.322225+0000","last_became_peered":"2024-03-09T14:28:34.322225+0000","last_unstale":"2024-03-09T15:36:39.224849+0000","last_undegraded":"2024-03-09T15:36:39.224849+0000","last_fullsized":"2024-03-09T15:36:39.224849+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'134","last_scrub_stamp":"2024-03-09T15:36:39.224588+0000","last_deep_scrub":"261'134","last_deep_scrub_stamp":"2024-03-09T15:36:39.224588+0000","last_clean_scrub_stamp":"2024-03-09T15:36:39.224588+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":100,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:04:21.612595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":498,"num_read_kb":399,"num_write":194,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"261'177","reported_seq":752,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.267800+0000","last_change":"2024-03-09T15:36:41.267910+0000","last_active":"2024-03-09T15:36:41.267800+0000","last_peered":"2024-03-09T15:36:41.267800+0000","last_clean":"2024-03-09T15:36:41.267800+0000","last_became_active":"2024-03-09T14:28:32.299839+0000","last_became_peered":"2024-03-09T14:28:32.299839+0000","last_unstale":"2024-03-09T15:36:41.267800+0000","last_undegraded":"2024-03-09T15:36:41.267800+0000","last_fullsized":"2024-03-09T15:36:41.267800+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'177","last_scrub_stamp":"2024-03-09T15:36:41.267607+0000","last_deep_scrub":"261'177","last_deep_scrub_stamp":"2024-03-09T15:36:41.267607+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.267607+0000","objects_scrubbed":6,"log_size":77,"log_dups_size":100,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:54:41.435261+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242922,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":18833,"num_write":606,"num_write_kb":39169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"261'74","reported_seq":618,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.623071+0000","last_change":"2024-03-09T15:36:46.623071+0000","last_active":"2024-03-09T15:36:46.623071+0000","last_peered":"2024-03-09T15:36:46.623071+0000","last_clean":"2024-03-09T15:36:46.623071+0000","last_became_active":"2024-03-09T14:28:34.315302+0000","last_became_peered":"2024-03-09T14:28:34.315302+0000","last_unstale":"2024-03-09T15:36:46.623071+0000","last_undegraded":"2024-03-09T15:36:46.623071+0000","last_fullsized":"2024-03-09T15:36:46.623071+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'74","last_scrub_stamp":"2024-03-09T15:36:46.622798+0000","last_deep_scrub":"261'74","last_deep_scrub_stamp":"2024-03-09T15:36:46.622798+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.622798+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:10:27.130072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":290,"num_read_kb":224,"num_write":116,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"261'156","reported_seq":719,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.267202+0000","last_change":"2024-03-09T15:36:42.267361+0000","last_active":"2024-03-09T15:36:42.267202+0000","last_peered":"2024-03-09T15:36:42.267202+0000","last_clean":"2024-03-09T15:36:42.267202+0000","last_became_active":"2024-03-09T14:28:32.334117+0000","last_became_peered":"2024-03-09T14:28:32.334117+0000","last_unstale":"2024-03-09T15:36:42.267202+0000","last_undegraded":"2024-03-09T15:36:42.267202+0000","last_fullsized":"2024-03-09T15:36:42.267202+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'156","last_scrub_stamp":"2024-03-09T15:36:42.266976+0000","last_deep_scrub":"261'156","last_deep_scrub_stamp":"2024-03-09T15:36:42.266976+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.266976+0000","objects_scrubbed":7,"log_size":56,"log_dups_size":100,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:04:14.319419+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":182,"num_read_kb":192,"num_write":563,"num_write_kb":14566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"261'48","reported_seq":616,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:31.189173+0000","last_change":"2024-03-09T15:36:31.189173+0000","last_active":"2024-03-09T15:36:31.189173+0000","last_peered":"2024-03-09T15:36:31.189173+0000","last_clean":"2024-03-09T15:36:31.189173+0000","last_became_active":"2024-03-09T14:28:34.323202+0000","last_became_peered":"2024-03-09T14:28:34.323202+0000","last_unstale":"2024-03-09T15:36:31.189173+0000","last_undegraded":"2024-03-09T15:36:31.189173+0000","last_fullsized":"2024-03-09T15:36:31.189173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'48","last_scrub_stamp":"2024-03-09T15:36:31.188911+0000","last_deep_scrub":"261'48","last_deep_scrub_stamp":"2024-03-09T15:36:31.188911+0000","last_clean_scrub_stamp":"2024-03-09T15:36:31.188911+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:36:34.360701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":219,"num_read_kb":183,"num_write":70,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"261'144","reported_seq":701,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:01.743658+0000","last_change":"2024-03-09T15:37:01.743756+0000","last_active":"2024-03-09T15:37:01.743658+0000","last_peered":"2024-03-09T15:37:01.743658+0000","last_clean":"2024-03-09T15:37:01.743658+0000","last_became_active":"2024-03-09T14:28:32.299737+0000","last_became_peered":"2024-03-09T14:28:32.299737+0000","last_unstale":"2024-03-09T15:37:01.743658+0000","last_undegraded":"2024-03-09T15:37:01.743658+0000","last_fullsized":"2024-03-09T15:37:01.743658+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'144","last_scrub_stamp":"2024-03-09T15:37:01.743474+0000","last_deep_scrub":"261'144","last_deep_scrub_stamp":"2024-03-09T15:37:01.743474+0000","last_clean_scrub_stamp":"2024-03-09T15:37:01.743474+0000","objects_scrubbed":6,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:12:52.005193+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":168,"num_read_kb":12613,"num_write":521,"num_write_kb":31618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"261'94","reported_seq":654,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.306693+0000","last_change":"2024-03-09T15:36:42.306758+0000","last_active":"2024-03-09T15:36:42.306693+0000","last_peered":"2024-03-09T15:36:42.306693+0000","last_clean":"2024-03-09T15:36:42.306693+0000","last_became_active":"2024-03-09T14:28:34.315451+0000","last_became_peered":"2024-03-09T14:28:34.315451+0000","last_unstale":"2024-03-09T15:36:42.306693+0000","last_undegraded":"2024-03-09T15:36:42.306693+0000","last_fullsized":"2024-03-09T15:36:42.306693+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'94","last_scrub_stamp":"2024-03-09T15:36:42.306481+0000","last_deep_scrub":"261'94","last_deep_scrub_stamp":"2024-03-09T15:36:42.306481+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.306481+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:34:40.259583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":374,"num_read_kb":293,"num_write":145,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"261'150","reported_seq":725,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:49.844582+0000","last_change":"2024-03-09T15:36:49.844668+0000","last_active":"2024-03-09T15:36:49.844582+0000","last_peered":"2024-03-09T15:36:49.844582+0000","last_clean":"2024-03-09T15:36:49.844582+0000","last_became_active":"2024-03-09T14:28:32.304540+0000","last_became_peered":"2024-03-09T14:28:32.304540+0000","last_unstale":"2024-03-09T15:36:49.844582+0000","last_undegraded":"2024-03-09T15:36:49.844582+0000","last_fullsized":"2024-03-09T15:36:49.844582+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'150","last_scrub_stamp":"2024-03-09T15:36:49.844389+0000","last_deep_scrub":"261'150","last_deep_scrub_stamp":"2024-03-09T15:36:49.844389+0000","last_clean_scrub_stamp":"2024-03-09T15:36:49.844389+0000","objects_scrubbed":6,"log_size":50,"log_dups_size":100,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:30:01.179531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1655899,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":200,"num_read_kb":1806,"num_write":509,"num_write_kb":18066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2e","version":"261'40","reported_seq":610,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:36.169829+0000","last_change":"2024-03-09T15:36:36.169895+0000","last_active":"2024-03-09T15:36:36.169829+0000","last_peered":"2024-03-09T15:36:36.169829+0000","last_clean":"2024-03-09T15:36:36.169829+0000","last_became_active":"2024-03-09T14:28:34.312399+0000","last_became_peered":"2024-03-09T14:28:34.312399+0000","last_unstale":"2024-03-09T15:36:36.169829+0000","last_undegraded":"2024-03-09T15:36:36.169829+0000","last_fullsized":"2024-03-09T15:36:36.169829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'40","last_scrub_stamp":"2024-03-09T15:36:36.169631+0000","last_deep_scrub":"261'40","last_deep_scrub_stamp":"2024-03-09T15:36:36.169631+0000","last_clean_scrub_stamp":"2024-03-09T15:36:36.169631+0000","objects_scrubbed":1,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:31:09.147800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":182,"num_read_kb":156,"num_write":57,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"261'151","reported_seq":703,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:05.826947+0000","last_change":"2024-03-09T15:37:05.827117+0000","last_active":"2024-03-09T15:37:05.826947+0000","last_peered":"2024-03-09T15:37:05.826947+0000","last_clean":"2024-03-09T15:37:05.826947+0000","last_became_active":"2024-03-09T14:28:32.314596+0000","last_became_peered":"2024-03-09T14:28:32.314596+0000","last_unstale":"2024-03-09T15:37:05.826947+0000","last_undegraded":"2024-03-09T15:37:05.826947+0000","last_fullsized":"2024-03-09T15:37:05.826947+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'151","last_scrub_stamp":"2024-03-09T15:37:05.826718+0000","last_deep_scrub":"261'151","last_deep_scrub_stamp":"2024-03-09T15:37:05.826718+0000","last_clean_scrub_stamp":"2024-03-09T15:37:05.826718+0000","objects_scrubbed":6,"log_size":51,"log_dups_size":100,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:11:13.341909+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263241,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":148,"num_read_kb":12793,"num_write":461,"num_write_kb":35350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"261'64","reported_seq":635,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:35.186824+0000","last_change":"2024-03-09T15:36:35.186888+0000","last_active":"2024-03-09T15:36:35.186824+0000","last_peered":"2024-03-09T15:36:35.186824+0000","last_clean":"2024-03-09T15:36:35.186824+0000","last_became_active":"2024-03-09T14:28:34.325461+0000","last_became_peered":"2024-03-09T14:28:34.325461+0000","last_unstale":"2024-03-09T15:36:35.186824+0000","last_undegraded":"2024-03-09T15:36:35.186824+0000","last_fullsized":"2024-03-09T15:36:35.186824+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'64","last_scrub_stamp":"2024-03-09T15:36:35.186622+0000","last_deep_scrub":"261'64","last_deep_scrub_stamp":"2024-03-09T15:36:35.186622+0000","last_clean_scrub_stamp":"2024-03-09T15:36:35.186622+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:12:15.358732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":264,"num_read_kb":214,"num_write":89,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"261'165","reported_seq":750,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.949321+0000","last_change":"2024-03-09T15:36:43.949499+0000","last_active":"2024-03-09T15:36:43.949321+0000","last_peered":"2024-03-09T15:36:43.949321+0000","last_clean":"2024-03-09T15:36:43.949321+0000","last_became_active":"2024-03-09T14:28:32.311750+0000","last_became_peered":"2024-03-09T14:28:32.311750+0000","last_unstale":"2024-03-09T15:36:43.949321+0000","last_undegraded":"2024-03-09T15:36:43.949321+0000","last_fullsized":"2024-03-09T15:36:43.949321+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'165","last_scrub_stamp":"2024-03-09T15:36:43.949104+0000","last_deep_scrub":"261'165","last_deep_scrub_stamp":"2024-03-09T15:36:43.949104+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.949104+0000","objects_scrubbed":6,"log_size":65,"log_dups_size":100,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T16:52:33.164001+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":213,"num_read_kb":14896,"num_write":533,"num_write_kb":41774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"261'162","reported_seq":783,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.302576+0000","last_change":"2024-03-09T15:36:42.302635+0000","last_active":"2024-03-09T15:36:42.302576+0000","last_peered":"2024-03-09T15:36:42.302576+0000","last_clean":"2024-03-09T15:36:42.302576+0000","last_became_active":"2024-03-09T14:28:34.312212+0000","last_became_peered":"2024-03-09T14:28:34.312212+0000","last_unstale":"2024-03-09T15:36:42.302576+0000","last_undegraded":"2024-03-09T15:36:42.302576+0000","last_fullsized":"2024-03-09T15:36:42.302576+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'162","last_scrub_stamp":"2024-03-09T15:36:42.302384+0000","last_deep_scrub":"261'162","last_deep_scrub_stamp":"2024-03-09T15:36:42.302384+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.302384+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":100,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T20:11:17.535660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":664,"num_read_kb":532,"num_write":243,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2b","version":"261'153","reported_seq":734,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:37:04.104837+0000","last_change":"2024-03-09T15:37:04.104909+0000","last_active":"2024-03-09T15:37:04.104837+0000","last_peered":"2024-03-09T15:37:04.104837+0000","last_clean":"2024-03-09T15:37:04.104837+0000","last_became_active":"2024-03-09T14:28:32.309562+0000","last_became_peered":"2024-03-09T14:28:32.309562+0000","last_unstale":"2024-03-09T15:37:04.104837+0000","last_undegraded":"2024-03-09T15:37:04.104837+0000","last_fullsized":"2024-03-09T15:37:04.104837+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'153","last_scrub_stamp":"2024-03-09T15:37:04.104636+0000","last_deep_scrub":"261'153","last_deep_scrub_stamp":"2024-03-09T15:37:04.104636+0000","last_clean_scrub_stamp":"2024-03-09T15:37:04.104636+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":100,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:30:43.113561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":246,"num_read_kb":5348,"num_write":573,"num_write_kb":20887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"261'118","reported_seq":720,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:42.176542+0000","last_change":"2024-03-09T15:36:42.176738+0000","last_active":"2024-03-09T15:36:42.176542+0000","last_peered":"2024-03-09T15:36:42.176542+0000","last_clean":"2024-03-09T15:36:42.176542+0000","last_became_active":"2024-03-09T14:28:34.321424+0000","last_became_peered":"2024-03-09T14:28:34.321424+0000","last_unstale":"2024-03-09T15:36:42.176542+0000","last_undegraded":"2024-03-09T15:36:42.176542+0000","last_fullsized":"2024-03-09T15:36:42.176542+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'118","last_scrub_stamp":"2024-03-09T15:36:42.176313+0000","last_deep_scrub":"261'118","last_deep_scrub_stamp":"2024-03-09T15:36:42.176313+0000","last_clean_scrub_stamp":"2024-03-09T15:36:42.176313+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":100,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:02:02.079925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":513,"num_read_kb":409,"num_write":169,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3064,"num_omap_keys":18,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"261'167","reported_seq":744,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:41.264592+0000","last_change":"2024-03-09T15:36:41.264697+0000","last_active":"2024-03-09T15:36:41.264592+0000","last_peered":"2024-03-09T15:36:41.264592+0000","last_clean":"2024-03-09T15:36:41.264592+0000","last_became_active":"2024-03-09T14:28:32.333838+0000","last_became_peered":"2024-03-09T14:28:32.333838+0000","last_unstale":"2024-03-09T15:36:41.264592+0000","last_undegraded":"2024-03-09T15:36:41.264592+0000","last_fullsized":"2024-03-09T15:36:41.264592+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'167","last_scrub_stamp":"2024-03-09T15:36:41.264378+0000","last_deep_scrub":"261'167","last_deep_scrub_stamp":"2024-03-09T15:36:41.264378+0000","last_clean_scrub_stamp":"2024-03-09T15:36:41.264378+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":100,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:23:18.360963+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":238,"num_read_kb":3315,"num_write":673,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2a","version":"261'72","reported_seq":645,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:52.300833+0000","last_change":"2024-03-09T15:36:52.300899+0000","last_active":"2024-03-09T15:36:52.300833+0000","last_peered":"2024-03-09T15:36:52.300833+0000","last_clean":"2024-03-09T15:36:52.300833+0000","last_became_active":"2024-03-09T14:28:34.297042+0000","last_became_peered":"2024-03-09T14:28:34.297042+0000","last_unstale":"2024-03-09T15:36:52.300833+0000","last_undegraded":"2024-03-09T15:36:52.300833+0000","last_fullsized":"2024-03-09T15:36:52.300833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'72","last_scrub_stamp":"2024-03-09T15:36:52.300623+0000","last_deep_scrub":"261'72","last_deep_scrub_stamp":"2024-03-09T15:36:52.300623+0000","last_clean_scrub_stamp":"2024-03-09T15:36:52.300623+0000","objects_scrubbed":1,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:15:26.404006+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":242,"num_write":99,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":473,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"261'126","reported_seq":708,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:54.140373+0000","last_change":"2024-03-09T15:36:54.140525+0000","last_active":"2024-03-09T15:36:54.140373+0000","last_peered":"2024-03-09T15:36:54.140373+0000","last_clean":"2024-03-09T15:36:54.140373+0000","last_became_active":"2024-03-09T14:28:32.317797+0000","last_became_peered":"2024-03-09T14:28:32.317797+0000","last_unstale":"2024-03-09T15:36:54.140373+0000","last_undegraded":"2024-03-09T15:36:54.140373+0000","last_fullsized":"2024-03-09T15:36:54.140373+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'126","last_scrub_stamp":"2024-03-09T15:36:54.140156+0000","last_deep_scrub":"261'126","last_deep_scrub_stamp":"2024-03-09T15:36:54.140156+0000","last_clean_scrub_stamp":"2024-03-09T15:36:54.140156+0000","objects_scrubbed":4,"log_size":26,"log_dups_size":100,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:25:48.296017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263201,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":187,"num_read_kb":4139,"num_write":475,"num_write_kb":19145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"261'113","reported_seq":677,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:48.688184+0000","last_change":"2024-03-09T15:36:48.688250+0000","last_active":"2024-03-09T15:36:48.688184+0000","last_peered":"2024-03-09T15:36:48.688184+0000","last_clean":"2024-03-09T15:36:48.688184+0000","last_became_active":"2024-03-09T14:28:34.314073+0000","last_became_peered":"2024-03-09T14:28:34.314073+0000","last_unstale":"2024-03-09T15:36:48.688184+0000","last_undegraded":"2024-03-09T15:36:48.688184+0000","last_fullsized":"2024-03-09T15:36:48.688184+0000","mapping_epoch":22,"log_start":"261'100","ondisk_log_start":"261'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'113","last_scrub_stamp":"2024-03-09T15:36:48.687973+0000","last_deep_scrub":"261'113","last_deep_scrub_stamp":"2024-03-09T15:36:48.687973+0000","last_clean_scrub_stamp":"2024-03-09T15:36:48.687973+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":100,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:23:16.042642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":445,"num_read_kb":351,"num_write":167,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"261'140","reported_seq":726,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:30.283437+0000","last_change":"2024-03-09T15:36:30.283533+0000","last_active":"2024-03-09T15:36:30.283437+0000","last_peered":"2024-03-09T15:36:30.283437+0000","last_clean":"2024-03-09T15:36:30.283437+0000","last_became_active":"2024-03-09T14:28:32.334187+0000","last_became_peered":"2024-03-09T14:28:32.334187+0000","last_unstale":"2024-03-09T15:36:30.283437+0000","last_undegraded":"2024-03-09T15:36:30.283437+0000","last_fullsized":"2024-03-09T15:36:30.283437+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'140","last_scrub_stamp":"2024-03-09T15:36:30.283226+0000","last_deep_scrub":"261'140","last_deep_scrub_stamp":"2024-03-09T15:36:30.283226+0000","last_clean_scrub_stamp":"2024-03-09T15:36:30.283226+0000","objects_scrubbed":6,"log_size":40,"log_dups_size":100,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T17:10:30.041775+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2181199,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":224,"num_read_kb":1842,"num_write":561,"num_write_kb":7884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"261'4028","reported_seq":4588,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:44.151978+0000","last_change":"2024-03-09T15:36:44.152037+0000","last_active":"2024-03-09T15:36:44.151978+0000","last_peered":"2024-03-09T15:36:44.151978+0000","last_clean":"2024-03-09T15:36:44.151978+0000","last_became_active":"2024-03-09T14:28:34.322421+0000","last_became_peered":"2024-03-09T14:28:34.322421+0000","last_unstale":"2024-03-09T15:36:44.151978+0000","last_undegraded":"2024-03-09T15:36:44.151978+0000","last_fullsized":"2024-03-09T15:36:44.151978+0000","mapping_epoch":22,"log_start":"261'4000","ondisk_log_start":"261'4000","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'4028","last_scrub_stamp":"2024-03-09T15:36:44.151779+0000","last_deep_scrub":"261'4028","last_deep_scrub_stamp":"2024-03-09T15:36:44.151779+0000","last_clean_scrub_stamp":"2024-03-09T15:36:44.151779+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T23:57:46.039852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14144,"num_read_kb":11461,"num_write":6038,"num_write_kb":3015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"261'136","reported_seq":715,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:37.873937+0000","last_change":"2024-03-09T15:36:37.874066+0000","last_active":"2024-03-09T15:36:37.873937+0000","last_peered":"2024-03-09T15:36:37.873937+0000","last_clean":"2024-03-09T15:36:37.873937+0000","last_became_active":"2024-03-09T14:28:32.324166+0000","last_became_peered":"2024-03-09T14:28:32.324166+0000","last_unstale":"2024-03-09T15:36:37.873937+0000","last_undegraded":"2024-03-09T15:36:37.873937+0000","last_fullsized":"2024-03-09T15:36:37.873937+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'136","last_scrub_stamp":"2024-03-09T15:36:37.873720+0000","last_deep_scrub":"261'136","last_deep_scrub_stamp":"2024-03-09T15:36:37.873720+0000","last_clean_scrub_stamp":"2024-03-09T15:36:37.873720+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":100,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:39:18.821060+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269861,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":9428,"num_write":515,"num_write_kb":30713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"261'144","reported_seq":732,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.957343+0000","last_change":"2024-03-09T15:36:46.957426+0000","last_active":"2024-03-09T15:36:46.957343+0000","last_peered":"2024-03-09T15:36:46.957343+0000","last_clean":"2024-03-09T15:36:46.957343+0000","last_became_active":"2024-03-09T14:28:32.287937+0000","last_became_peered":"2024-03-09T14:28:32.287937+0000","last_unstale":"2024-03-09T15:36:46.957343+0000","last_undegraded":"2024-03-09T15:36:46.957343+0000","last_fullsized":"2024-03-09T15:36:46.957343+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'144","last_scrub_stamp":"2024-03-09T15:36:46.957162+0000","last_deep_scrub":"261'144","last_deep_scrub_stamp":"2024-03-09T15:36:46.957162+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.957162+0000","objects_scrubbed":3,"log_size":44,"log_dups_size":100,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T21:17:49.579394+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5243043,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":214,"num_read_kb":48289,"num_write":459,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"261'161","reported_seq":736,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:43.317920+0000","last_change":"2024-03-09T15:36:43.318022+0000","last_active":"2024-03-09T15:36:43.317920+0000","last_peered":"2024-03-09T15:36:43.317920+0000","last_clean":"2024-03-09T15:36:43.317920+0000","last_became_active":"2024-03-09T14:28:34.316689+0000","last_became_peered":"2024-03-09T14:28:34.316689+0000","last_unstale":"2024-03-09T15:36:43.317920+0000","last_undegraded":"2024-03-09T15:36:43.317920+0000","last_fullsized":"2024-03-09T15:36:43.317920+0000","mapping_epoch":22,"log_start":"258'100","ondisk_log_start":"258'100","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'161","last_scrub_stamp":"2024-03-09T15:36:43.317727+0000","last_deep_scrub":"261'161","last_deep_scrub_stamp":"2024-03-09T15:36:43.317727+0000","last_clean_scrub_stamp":"2024-03-09T15:36:43.317727+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":100,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T22:34:41.055202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":634,"num_read_kb":490,"num_write":268,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"261'163","reported_seq":751,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:46.213002+0000","last_change":"2024-03-09T15:36:46.213092+0000","last_active":"2024-03-09T15:36:46.213002+0000","last_peered":"2024-03-09T15:36:46.213002+0000","last_clean":"2024-03-09T15:36:46.213002+0000","last_became_active":"2024-03-09T14:28:32.313132+0000","last_became_peered":"2024-03-09T14:28:32.313132+0000","last_unstale":"2024-03-09T15:36:46.213002+0000","last_undegraded":"2024-03-09T15:36:46.213002+0000","last_fullsized":"2024-03-09T15:36:46.213002+0000","mapping_epoch":20,"log_start":"261'100","ondisk_log_start":"261'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'163","last_scrub_stamp":"2024-03-09T15:36:46.212812+0000","last_deep_scrub":"261'163","last_deep_scrub_stamp":"2024-03-09T15:36:46.212812+0000","last_clean_scrub_stamp":"2024-03-09T15:36:46.212812+0000","objects_scrubbed":6,"log_size":63,"log_dups_size":100,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T19:51:57.319048+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262227,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":255,"num_read_kb":5401,"num_write":635,"num_write_kb":32648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"261'41","reported_seq":596,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-09T15:36:33.761668+0000","last_change":"2024-03-09T15:36:33.761734+0000","last_active":"2024-03-09T15:36:33.761668+0000","last_peered":"2024-03-09T15:36:33.761668+0000","last_clean":"2024-03-09T15:36:33.761668+0000","last_became_active":"2024-03-09T14:28:34.303701+0000","last_became_peered":"2024-03-09T14:28:34.303701+0000","last_unstale":"2024-03-09T15:36:33.761668+0000","last_undegraded":"2024-03-09T15:36:33.761668+0000","last_fullsized":"2024-03-09T15:36:33.761668+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'41","last_scrub_stamp":"2024-03-09T15:36:33.761432+0000","last_deep_scrub":"261'41","last_deep_scrub_stamp":"2024-03-09T15:36:33.761432+0000","last_clean_scrub_stamp":"2024-03-09T15:36:33.761432+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:47:50.469481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":166,"num_read_kb":138,"num_write":64,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":65,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3686,"num_read_kb":2614,"num_write":1236,"num_write_kb":514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6449,"num_omap_keys":13,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":13176,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":8518,"num_read_kb":8294,"num_write":8580,"num_write_kb":2546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2306,"num_omap_keys":14,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":26352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1984,"ondisk_log_size":1984,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":64,"acting":64,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":377960,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":12181,"num_read_kb":21090,"num_write":9023,"num_write_kb":5457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":306916,"num_omap_keys":744,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1857,"ondisk_log_size":1857,"up":64,"acting":64,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1719,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1602,"num_read_kb":1903,"num_write":22,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":38590,"num_read_kb":31236,"num_write":15314,"num_write_kb":7493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":33054,"num_omap_keys":137,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3953,"ondisk_log_size":3953,"up":128,"acting":128,"num_store_stats":8},{"poolid":68,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1128,"num_omap_keys":12,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":135300586,"num_objects":320,"num_object_clones":0,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":14057,"num_read_kb":1121153,"num_write":34295,"num_write_kb":1701153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":271892480,"data_stored":270601172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3168,"ondisk_log_size":3168,"up":128,"acting":128,"num_store_stats":8},{"poolid":67,"num_pg":32,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":140,"num_read_kb":192739,"num_write":156,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":66,"num_pg":32,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":138,"num_read_kb":192649,"num_write":156,"num_write_kb":134273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575688,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293320,"kb_used_data":45568,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":94078520,"statfs":{"total":96636764160,"available":96336404480,"internally_reserved":0,"allocated":46661632,"data_stored":45012346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":253688266},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575683,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233960,"kb_used_data":22304,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":94137880,"statfs":{"total":96636764160,"available":96397189120,"internally_reserved":0,"allocated":22839296,"data_stored":21261512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575684,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132532,"kb_used_data":61164,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94239308,"statfs":{"total":96636764160,"available":96501051392,"internally_reserved":0,"allocated":62631936,"data_stored":60939198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575688,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100148,"kb_used_data":55084,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94271692,"statfs":{"total":96636764160,"available":96534212608,"internally_reserved":0,"allocated":56406016,"data_stored":54802077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575683,"num_pgs":118,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252776,"kb_used_data":41120,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":94119064,"statfs":{"total":96636764160,"available":96377921536,"internally_reserved":0,"allocated":42106880,"data_stored":40381748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":216725959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575685,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91428,"kb_used_data":38044,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94280412,"statfs":{"total":96636764160,"available":96543141888,"internally_reserved":0,"allocated":38957056,"data_stored":37358501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575684,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131376,"kb_used_data":41960,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94240464,"statfs":{"total":96636764160,"available":96502235136,"internally_reserved":0,"allocated":42967040,"data_stored":41342836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575686,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244644,"kb_used_data":55196,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":94127196,"statfs":{"total":96636764160,"available":96386248704,"internally_reserved":0,"allocated":56520704,"data_stored":54866964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40763392,"data_stored":40607042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23621632,"data_stored":23479247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30126080,"data_stored":29981692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31137792,"data_stored":30907070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":42364928,"data_stored":42183762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":48488448,"data_stored":48318049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":19247104,"data_stored":19128319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36143104,"data_stored":35995991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":3901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":2727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":3405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":65,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10485760,"data_stored":10485760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":1048576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15728640,"data_stored":15728640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":1048576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5242880,"data_stored":5242880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10485760,"data_stored":10485760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":68,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T15:37:36.425 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-09T15:37:36.781 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-09T15:37:36.781 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-09T15:37:36.782 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-09T15:37:36.782 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:36.873 INFO:tasks.ceph.osd.0:Stopped 2024-03-09T15:37:36.873 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-09T15:37:36.874 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:36.962 INFO:tasks.ceph.osd.1:Stopped 2024-03-09T15:37:36.963 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-09T15:37:36.963 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:40.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-09T15:37:43.065 INFO:tasks.ceph.osd.2:Stopped 2024-03-09T15:37:43.066 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-09T15:37:43.066 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:43.144 INFO:tasks.ceph.osd.3:Stopped 2024-03-09T15:37:43.145 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-09T15:37:43.145 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:43.234 INFO:tasks.ceph.osd.4:Stopped 2024-03-09T15:37:43.234 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-09T15:37:43.235 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:45.952 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-03-09T15:37:45.953 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-09T15:37:49.336 INFO:tasks.ceph.osd.5:Stopped 2024-03-09T15:37:49.337 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-09T15:37:49.337 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:49.420 INFO:tasks.ceph.osd.6:Stopped 2024-03-09T15:37:49.420 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-09T15:37:49.420 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:51.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-09T15:37:51.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-03-09T15:37:55.522 INFO:tasks.ceph.osd.7:Stopped 2024-03-09T15:37:55.522 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-09T15:37:55.523 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-09T15:37:55.523 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:55.554 INFO:tasks.ceph.mgr.y:Stopped 2024-03-09T15:37:55.555 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-09T15:37:55.555 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:55.591 INFO:tasks.ceph.mgr.x:Stopped 2024-03-09T15:37:55.592 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-09T15:37:55.592 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-09T15:37:55.592 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:55.617 INFO:tasks.ceph.mon.a:Stopped 2024-03-09T15:37:55.618 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-09T15:37:55.618 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:55.677 INFO:tasks.ceph.mon.c:Stopped 2024-03-09T15:37:55.677 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-09T15:37:55.677 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T15:37:55.705 INFO:tasks.ceph.mon.b:Stopped 2024-03-09T15:37:55.706 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-09T15:37:55.706 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-03-09T15:37:55.736 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-09T15:37:55.736 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-09T15:37:56.060 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-09T15:37:56.061 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-09T15:37:56.275 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-09T15:37:56.275 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-09T15:37:56.402 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-09T15:37:56.403 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-09T15:37:56.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-09T15:37:56.531 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-09T15:37:56.531 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-09T15:37:56.860 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-09T15:37:56.860 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-09T15:37:57.005 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-09T15:37:57.006 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-09T15:37:57.160 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-09T15:37:57.160 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-09T15:37:57.291 INFO:tasks.ceph:Archiving mon data... 2024-03-09T15:37:57.292 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479/data/mon.a.tgz 2024-03-09T15:37:57.292 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-09T15:37:57.298 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.FvKoh5C4rZ 2024-03-09T15:37:57.299 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.FvKoh5C4rZ 2024-03-09T15:37:57.472 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.FvKoh5C4rZ 2024-03-09T15:37:57.537 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.FvKoh5C4rZ is 482KB 2024-03-09T15:37:57.598 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.FvKoh5C4rZ 2024-03-09T15:37:57.604 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479/data/mon.c.tgz 2024-03-09T15:37:57.604 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-09T15:37:57.654 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.FX8taLycUV 2024-03-09T15:37:57.655 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.FX8taLycUV 2024-03-09T15:37:57.835 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.FX8taLycUV 2024-03-09T15:37:57.901 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.FX8taLycUV is 502KB 2024-03-09T15:37:57.962 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.FX8taLycUV 2024-03-09T15:37:57.968 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479/data/mon.b.tgz 2024-03-09T15:37:57.968 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-03-09T15:37:57.974 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.I0bP8oJjAO 2024-03-09T15:37:57.975 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.I0bP8oJjAO 2024-03-09T15:37:58.150 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.I0bP8oJjAO 2024-03-09T15:37:58.211 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.I0bP8oJjAO is 502KB 2024-03-09T15:37:58.276 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.I0bP8oJjAO 2024-03-09T15:37:58.281 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-09T15:37:58.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-09T15:37:58.284 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-09T15:37:58.402 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_PossiblyLost calloc calloc allocate_dtv 2024-03-09T15:37:58.410 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-09T15:37:58.411 INFO:tasks.ceph:Archiving crash dumps... 2024-03-09T15:37:58.412 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/crash to /home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479/remote/smithi001/crash 2024-03-09T15:37:58.413 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-09T15:37:58.428 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/crash to /home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479/remote/smithi053/crash 2024-03-09T15:37:58.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-09T15:37:58.464 INFO:tasks.ceph:Compressing logs... 2024-03-09T15:37:58.464 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-09T15:37:58.473 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-09T15:37:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109674.log 2024-03-09T15:37:58.497 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/rgw.ceph.client.0.log 2024-03-09T15:37:58.497 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51773.log 2024-03-09T15:37:58.497 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109674.log.gz 2024-03-09T15:37:58.497 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/rgw.ceph.client.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71864.log 2024-03-09T15:37:58.498 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73634.log 2024-03-09T15:37:58.498 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51773.log.gz 2024-03-09T15:37:58.499 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71864.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-09T15:37:58.499 INFO:teuthology.orchestra.run.smithi001.stderr: 82.1% -- replaced with /var/log/ceph/ceph-client.admin.71864.log.gz 2024-03-09T15:37:58.499 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67591.log 2024-03-09T15:37:58.500 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73634.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106677.log 2024-03-09T15:37:58.501 INFO:teuthology.orchestra.run.smithi001.stderr: 81.2% -- replaced with /var/log/ceph/ceph-client.admin.73634.log.gz 2024-03-09T15:37:58.501 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54748.log 2024-03-09T15:37:58.501 INFO:teuthology.orchestra.run.smithi001.stderr: 48.9% -- replaced with /var/log/ceph/ceph-client.admin.67591.log.gz 2024-03-09T15:37:58.501 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.106677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121506.log 2024-03-09T15:37:58.501 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106677.log.gz 2024-03-09T15:37:58.501 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.127415.log 2024-03-09T15:37:58.501 INFO:teuthology.orchestra.run.smithi001.stderr: 19.9% -- replaced with /var/log/ceph/ceph-client.admin.54748.log.gz 2024-03-09T15:37:58.511 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.121506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30717.log 2024-03-09T15:37:58.511 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.127415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.125645.log 2024-03-09T15:37:58.512 INFO:teuthology.orchestra.run.smithi001.stderr: 80.5% -- replaced with /var/log/ceph/ceph-client.0.127415.log.gz 2024-03-09T15:37:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-09T15:37:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30717.log.gz 2024-03-09T15:37:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.125645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116635.log 2024-03-09T15:37:58.512 INFO:teuthology.orchestra.run.smithi001.stderr: 80.7% -- replaced with /var/log/ceph/ceph-client.0.125645.log.gz 2024-03-09T15:37:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42233.log 2024-03-09T15:37:58.512 INFO:teuthology.orchestra.run.smithi001.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.121506.log.gz 2024-03-09T15:37:58.512 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.116635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74814.log 2024-03-09T15:37:58.512 INFO:teuthology.orchestra.run.smithi001.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.116635.log.gz 2024-03-09T15:37:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54207.log 2024-03-09T15:37:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42233.log: /var/log/ceph/ceph-client.admin.74814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114188.log 2024-03-09T15:37:58.513 INFO:teuthology.orchestra.run.smithi001.stderr: 81.0% -- replaced with /var/log/ceph/ceph-client.admin.42233.log.gz 2024-03-09T15:37:58.513 INFO:teuthology.orchestra.run.smithi001.stderr: 83.0% -- replaced with /var/log/ceph/ceph-client.admin.74814.log.gz 2024-03-09T15:37:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45233.log 2024-03-09T15:37:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.114188.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41053.log 2024-03-09T15:37:58.513 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104907.log 2024-03-09T15:37:58.514 INFO:teuthology.orchestra.run.smithi001.stderr: 80.9% -- replaced with /var/log/ceph/ceph-client.admin.45233.log.gz 2024-03-09T15:37:58.514 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31238.log 2024-03-09T15:37:58.514 INFO:teuthology.orchestra.run.smithi001.stderr: 80.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.0.124465.log 2024-03-09T15:37:58.514 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41053.log.gz 2024-03-09T15:37:58.514 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.104907.log: /var/log/ceph/ceph-client.admin.31238.log: 82.5% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31238.log.gz 2024-03-09T15:37:58.514 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104907.log.gz 2024-03-09T15:37:58.514 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61828.log 2024-03-09T15:37:58.514 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49388.log 2024-03-09T15:37:58.515 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.124465.log: /var/log/ceph/ceph-client.admin.61828.log: 80.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111469.log 2024-03-09T15:37:58.515 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.0.124465.log.gz 2024-03-09T15:37:58.515 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114188.log.gz 2024-03-09T15:37:58.515 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49388.log.gz 2024-03-09T15:37:58.515 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133923.log 2024-03-09T15:37:58.515 INFO:teuthology.orchestra.run.smithi001.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61828.log.gz 2024-03-09T15:37:58.515 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.111469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98416.log 2024-03-09T15:37:58.515 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111469.log.gz 2024-03-09T15:37:58.516 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.130955.log 2024-03-09T15:37:58.516 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133923.log.gz 2024-03-09T15:37:58.516 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.98416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50003.log 2024-03-09T15:37:58.516 INFO:teuthology.orchestra.run.smithi001.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.98416.log.gz 2024-03-09T15:37:58.516 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.130955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30823.log 2024-03-09T15:37:58.516 INFO:teuthology.orchestra.run.smithi001.stderr: 80.9% -- replaced with /var/log/ceph/ceph-client.0.130955.log.gz 2024-03-09T15:37:58.523 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133819.log 2024-03-09T15:37:58.524 INFO:teuthology.orchestra.run.smithi001.stderr: 81.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87205.log 2024-03-09T15:37:58.524 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30823.log: -- replaced with /var/log/ceph/ceph-client.admin.50003.log.gz 2024-03-09T15:37:58.524 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30823.log.gz 2024-03-09T15:37:58.524 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134210.log 2024-03-09T15:37:58.524 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133819.log.gz 2024-03-09T15:37:58.524 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.122695.log 2024-03-09T15:37:58.524 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134027.log 2024-03-09T15:37:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.134210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134210.log.gz/var/log/ceph/ceph-client.0.122695.log: 2024-03-09T15:37:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134184.log 2024-03-09T15:37:58.525 INFO:teuthology.orchestra.run.smithi001.stderr: 84.4% -- replaced with /var/log/ceph/ceph-client.0.122695.log.gz 2024-03-09T15:37:58.525 INFO:teuthology.orchestra.run.smithi001.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.87205.log.gz 2024-03-09T15:37:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.134027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84255.log 2024-03-09T15:37:58.525 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134027.log.gz 2024-03-09T15:37:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76584.log 2024-03-09T15:37:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.134184.log: /var/log/ceph/ceph-client.admin.84255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134184.log.gz 2024-03-09T15:37:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.129185.log 2024-03-09T15:37:58.526 INFO:teuthology.orchestra.run.smithi001.stderr: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.84255.log.gz 2024-03-09T15:37:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76584.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-09T15:37:58.526 INFO:teuthology.orchestra.run.smithi001.stderr: 82.6% -- replaced with /var/log/ceph/ceph-client.admin.76584.log.gz 2024-03-09T15:37:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.129185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70094.log 2024-03-09T15:37:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.2.log: 80.8% -- replaced with /var/log/ceph/ceph-client.0.129185.log.gz 2024-03-09T15:37:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120307.log 2024-03-09T15:37:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111519.log 2024-03-09T15:37:58.527 INFO:teuthology.orchestra.run.smithi001.stderr: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.70094.log.gz 2024-03-09T15:37:58.527 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62477.log 2024-03-09T15:37:58.527 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.120307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106087.log 2024-03-09T15:37:58.527 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.111519.log: /var/log/ceph/ceph-client.admin.62477.log: 88.0% 92.3% -- replaced with /var/log/ceph/ceph-client.admin.120307.log.gz 2024-03-09T15:37:58.527 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.111519.log.gz 2024-03-09T15:37:58.527 INFO:teuthology.orchestra.run.smithi001.stderr: 84.5% -- replaced with /var/log/ceph/ceph-client.admin.62477.log.gz 2024-03-09T15:37:58.527 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.128595.log 2024-03-09T15:37:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-09T15:37:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.106087.log: 81.4% -- replaced with /var/log/ceph/ceph-client.admin.106087.log.gzgzip -5 --verbose --/var/log/ceph/ceph-client.0.128595.log: /var/log/ceph/ceph-client.admin.101956.log 2024-03-09T15:37:58.528 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-09T15:37:58.528 INFO:teuthology.orchestra.run.smithi001.stderr: 84.5% -- replaced with /var/log/ceph/ceph-client.0.128595.log.gz 2024-03-09T15:37:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mon.a.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54207.log.gz 2024-03-09T15:37:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84845.log 2024-03-09T15:37:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.101956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117813.log 2024-03-09T15:37:58.528 INFO:teuthology.orchestra.run.smithi001.stderr: 83.1% -- replaced with /var/log/ceph/ceph-client.admin.101956.log.gz 2024-03-09T15:37:58.546 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25413.log 2024-03-09T15:37:58.546 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-09T15:37:58.546 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-09T15:37:58.546 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.25413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25413.log.gz 2024-03-09T15:37:58.547 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25389.log 2024-03-09T15:37:58.547 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-09T15:37:58.547 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.25389.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.25389.log.gz 2024-03-09T15:37:58.547 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.25485.log 2024-03-09T15:37:58.547 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25437.log 2024-03-09T15:37:58.547 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.25485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25485.log.gz 2024-03-09T15:37:58.547 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25341.log 2024-03-09T15:37:58.547 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.25437.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-09T15:37:58.548 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25437.log.gz 2024-03-09T15:37:58.548 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.25341.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-09T15:37:58.548 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25341.log.gz 2024-03-09T15:37:58.548 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25365.log 2024-03-09T15:37:58.548 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-09T15:37:58.548 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.25365.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25365.log.gz 2024-03-09T15:37:58.548 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.25461.log 2024-03-09T15:37:58.548 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-09T15:37:58.549 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.25461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25461.log.gz 2024-03-09T15:37:58.549 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-09T15:37:58.549 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mon.b.log: 90.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-09T15:37:58.553 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25317.log 2024-03-09T15:37:58.553 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.25317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25317.log.gz 2024-03-09T15:37:58.553 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-09T15:37:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84845.log: 82.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31029.log 2024-03-09T15:37:58.560 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84845.log.gz 2024-03-09T15:37:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.126235.log 2024-03-09T15:37:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.117813.log: /var/log/ceph/ceph-client.admin.31029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116046.log 2024-03-09T15:37:58.560 INFO:teuthology.orchestra.run.smithi001.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.117813.log.gz 2024-03-09T15:37:58.561 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31029.log.gz 2024-03-09T15:37:58.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.126235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92516.log 2024-03-09T15:37:58.561 INFO:teuthology.orchestra.run.smithi001.stderr: 84.5% -- replaced with /var/log/ceph/ceph-client.0.126235.log.gz 2024-03-09T15:37:58.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.116046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133689.log 2024-03-09T15:37:58.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.92516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71274.log 2024-03-09T15:37:58.561 INFO:teuthology.orchestra.run.smithi001.stderr: 82.9% -- replaced with /var/log/ceph/ceph-client.admin.92516.log.gz/var/log/ceph/ceph-client.admin.133689.log: 2024-03-09T15:37:58.561 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133689.log.gz 2024-03-09T15:37:58.561 INFO:teuthology.orchestra.run.smithi001.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.116046.log.gz 2024-03-09T15:37:58.562 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48773.log 2024-03-09T15:37:58.562 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.71274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85435.log 2024-03-09T15:37:58.562 INFO:teuthology.orchestra.run.smithi001.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.71274.log.gz 2024-03-09T15:37:58.562 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87795.log 2024-03-09T15:37:58.562 INFO:teuthology.orchestra.run.smithi001.stderr: 82.1% -- replaced with /var/log/ceph/ceph-client.admin.48773.log.gz 2024-03-09T15:37:58.562 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.85435.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.85435.log.gz 2024-03-09T15:37:58.562 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133663.log 2024-03-09T15:37:58.563 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82485.log 2024-03-09T15:37:58.563 INFO:teuthology.orchestra.run.smithi001.stderr: 82.7% -- replaced with /var/log/ceph/ceph-client.admin.87795.log.gz 2024-03-09T15:37:58.563 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110289.log 2024-03-09T15:37:58.563 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133663.log.gz/var/log/ceph/ceph-client.admin.82485.log: 2024-03-09T15:37:58.563 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44643.log 2024-03-09T15:37:58.563 INFO:teuthology.orchestra.run.smithi001.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.82485.log.gz 2024-03-09T15:37:58.563 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110289.log: 82.6% -- replaced with /var/log/ceph/ceph-client.admin.110289.log.gz 2024-03-09T15:37:58.563 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88976.log 2024-03-09T15:37:58.563 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30815.log 2024-03-09T15:37:58.564 INFO:teuthology.orchestra.run.smithi001.stderr: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.44643.log.gz 2024-03-09T15:37:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111494.log 2024-03-09T15:37:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30822.log 2024-03-09T15:37:58.564 INFO:teuthology.orchestra.run.smithi001.stderr: 85.4% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88976.log.gz 2024-03-09T15:37:58.564 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30815.log.gz 2024-03-09T15:37:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.111494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111494.log.gz 2024-03-09T15:37:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118540.log 2024-03-09T15:37:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30822.log.gz 2024-03-09T15:37:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100186.log 2024-03-09T15:37:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100776.log 2024-03-09T15:37:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.118540.log: /var/log/ceph/ceph-client.admin.100186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33161.log 2024-03-09T15:37:58.565 INFO:teuthology.orchestra.run.smithi001.stderr: 91.8% 82.7% -- replaced with /var/log/ceph/ceph-client.admin.100186.log.gz 2024-03-09T15:37:58.565 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.118540.log.gz 2024-03-09T15:37:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.100776.log: 82.7% -- replaced with /var/log/ceph/ceph-client.admin.100776.log.gz 2024-03-09T15:37:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134053.log 2024-03-09T15:37:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42823.log 2024-03-09T15:37:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.33161.log: 43.2%/var/log/ceph/ceph-client.admin.134053.log: -- replaced with /var/log/ceph/ceph-client.0.33161.log.gz 2024-03-09T15:37:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134053.log.gz -5 2024-03-09T15:37:58.566 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58613.log 2024-03-09T15:37:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42823.log: 80.7% -- replaced with /var/log/ceph/ceph-client.admin.42823.log.gz 2024-03-09T15:37:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134079.log 2024-03-09T15:37:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51183.log 2024-03-09T15:37:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.134079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109699.log 2024-03-09T15:37:58.566 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134079.log.gz 2024-03-09T15:37:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51183.log: 81.1% -- replaced with /var/log/ceph/ceph-client.admin.51183.log.gz 2024-03-09T15:37:58.567 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.58613.log.gz 2024-03-09T15:37:58.567 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.81304.log 2024-03-09T15:37:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.35509.log 2024-03-09T15:37:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34335.log 2024-03-09T15:37:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81304.log: 82.9% -- replaced with /var/log/ceph/ceph-client.admin.109699.log.gz 2024-03-09T15:37:58.567 INFO:teuthology.orchestra.run.smithi001.stderr: 82.7% -- replaced with /var/log/ceph/ceph-client.admin.81304.log.gz 2024-03-09T15:37:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.35509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53690.log 2024-03-09T15:37:58.567 INFO:teuthology.orchestra.run.smithi001.stderr: 43.9% -- replaced with /var/log/ceph/ceph-client.0.35509.log.gz 2024-03-09T15:37:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.34335.log: 43.5% -- replaced with /var/log/ceph/ceph-client.0.34335.log.gz 2024-03-09T15:37:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31030.log 2024-03-09T15:37:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47593.log 2024-03-09T15:37:58.568 INFO:teuthology.orchestra.run.smithi001.stderr: 58.4% -- replaced with /var/log/ceph/ceph-client.admin.53690.log.gz 2024-03-09T15:37:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91336.log 2024-03-09T15:37:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31030.log.gz 2024-03-09T15:37:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31226.log 2024-03-09T15:37:58.568 INFO:teuthology.orchestra.run.smithi001.stderr: 80.4% -- replaced with /var/log/ceph/ceph-client.admin.47593.log.gz 2024-03-09T15:37:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.91336.log: 82.8% -- replaced with /var/log/ceph/ceph-client.admin.91336.log.gz 2024-03-09T15:37:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52413.log 2024-03-09T15:37:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31226.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68912.log 2024-03-09T15:37:58.569 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31226.log.gz 2024-03-09T15:37:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.132274.log 2024-03-09T15:37:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52413.log: /var/log/ceph/ceph-client.admin.68912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114777.log 2024-03-09T15:37:58.569 INFO:teuthology.orchestra.run.smithi001.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.52413.log.gz 2024-03-09T15:37:58.569 INFO:teuthology.orchestra.run.smithi001.stderr: 83.4% -- replaced with /var/log/ceph/ceph-client.admin.68912.log.gz 2024-03-09T15:37:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.132274.log: 95.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-09T15:37:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90746.log 2024-03-09T15:37:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.114777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104317.log 2024-03-09T15:37:58.570 INFO:teuthology.orchestra.run.smithi001.stderr: 92.3%gzip/var/log/ceph/ceph-client.admin.90746.log: -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-09T15:37:58.570 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.114777.log.gz 2024-03-09T15:37:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.104317.log: 83.4% -- replaced with /var/log/ceph/ceph-client.admin.90746.log.gz 2024-03-09T15:37:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50593.log 2024-03-09T15:37:58.571 INFO:teuthology.orchestra.run.smithi001.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.104317.log.gz 2024-03-09T15:37:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56039.log 2024-03-09T15:37:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53974.log 2024-03-09T15:37:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56039.log: 80.9% -- replaced with /var/log/ceph/ceph-client.admin.50593.log.gz 2024-03-09T15:37:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93696.log 2024-03-09T15:37:58.571 INFO:teuthology.orchestra.run.smithi001.stderr: 78.2% -- replaced with /var/log/ceph/ceph-client.admin.56039.log.gz/var/log/ceph/ceph-client.admin.53974.log: 2024-03-09T15:37:58.571 INFO:teuthology.orchestra.run.smithi001.stderr: 49.1% -- replaced with /var/log/ceph/ceph-client.admin.53974.log.gz 2024-03-09T15:37:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91926.log 2024-03-09T15:37:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.93696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134105.log 2024-03-09T15:37:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.91926.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.93696.log.gz 2024-03-09T15:37:58.572 INFO:teuthology.orchestra.run.smithi001.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.91926.log.gz 2024-03-09T15:37:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31003.log 2024-03-09T15:37:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.134105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134105.log.gz 2024-03-09T15:37:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49413.log 2024-03-09T15:37:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78354.log 2024-03-09T15:37:58.572 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31003.log.gz 2024-03-09T15:37:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49413.log: 80.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77174.log 2024-03-09T15:37:58.573 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49413.log.gz 2024-03-09T15:37:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78354.log: 82.8% -- replaced with /var/log/ceph/ceph-client.admin.78354.log.gz 2024-03-09T15:37:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74224.log 2024-03-09T15:37:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43463.log 2024-03-09T15:37:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74224.log: 86.1% 87.3% -- replaced with /var/log/ceph/ceph-client.0.132274.log.gz 2024-03-09T15:37:58.573 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77174.log.gz 2024-03-09T15:37:58.573 INFO:teuthology.orchestra.run.smithi001.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.74224.log.gz 2024-03-09T15:37:58.574 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133637.log 2024-03-09T15:37:58.574 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25328.log 2024-03-09T15:37:58.574 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43463.log: /var/log/ceph/ceph-client.admin.133637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103727.log 2024-03-09T15:37:58.574 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133637.log.gz 2024-03-09T15:37:58.574 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25328.log: 81.6% -- replaced with /var/log/ceph/ceph-client.admin.43463.log.gz 2024-03-09T15:37:58.574 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25328.log.gz 2024-03-09T15:37:58.574 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32524.log 2024-03-09T15:37:58.574 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.103727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56682.log 2024-03-09T15:37:58.575 INFO:teuthology.orchestra.run.smithi001.stderr: 82.2% -- replaced with /var/log/ceph/ceph-client.admin.103727.log.gz/var/log/ceph/ceph-client.admin.32524.log: 2024-03-09T15:37:58.575 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133871.log 2024-03-09T15:37:58.575 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32524.log.gz 2024-03-09T15:37:58.582 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39233.log 2024-03-09T15:37:58.582 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32548.log 2024-03-09T15:37:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133871.log.gz 2024-03-09T15:37:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65127.log 2024-03-09T15:37:58.583 INFO:teuthology.orchestra.run.smithi001.stderr: 81.0% -- replaced with /var/log/ceph/ceph-client.admin.39233.log.gz 2024-03-09T15:37:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32548.log.gz 2024-03-09T15:37:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36677.log 2024-03-09T15:37:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49363.log 2024-03-09T15:37:58.583 INFO:teuthology.orchestra.run.smithi001.stderr: 81.0%/var/log/ceph/ceph-client.0.36677.log: -- replaced with /var/log/ceph/ceph-client.admin.65127.log.gz 2024-03-09T15:37:58.583 INFO:teuthology.orchestra.run.smithi001.stderr: 43.9% -- replaced with /var/log/ceph/ceph-client.0.36677.log.gz 2024-03-09T15:37:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31471.log 2024-03-09T15:37:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49363.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ops-log-ceph-client.admin.log 2024-03-09T15:37:58.584 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49363.log.gz 2024-03-09T15:37:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31471.log.gz 2024-03-09T15:37:58.584 INFO:teuthology.orchestra.run.smithi001.stderr: 93.4% -- replaced with /var/log/ceph/ceph-client.admin.56682.log.gz 2024-03-09T15:37:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31016.log 2024-03-09T15:37:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-09T15:37:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ops-log-ceph-client.admin.log: /var/log/ceph/ceph-client.admin.31016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67626.log 2024-03-09T15:37:58.585 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31016.log.gz 2024-03-09T15:37:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.1.log: 94.2% -- replaced with /var/log/ceph/ops-log-ceph-client.admin.log.gz 2024-03-09T15:37:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93106.log 2024-03-09T15:37:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63354.log 2024-03-09T15:37:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.67626.log: /var/log/ceph/ceph-client.admin.93106.log: 82.4% -- replaced with /var/log/ceph/ceph-client.admin.93106.log.gz 2024-03-09T15:37:58.585 INFO:teuthology.orchestra.run.smithi001.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.0.67626.log.gz 2024-03-09T15:37:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31010.log 2024-03-09T15:37:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31419.log 2024-03-09T15:37:58.586 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63354.log: /var/log/ceph/ceph-client.admin.31010.log: 0.0% 81.3% -- replaced with /var/log/ceph/ceph-client.admin.63354.log.gz 2024-03-09T15:37:58.586 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31010.log.gz 2024-03-09T15:37:58.586 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.125055.log 2024-03-09T15:37:58.586 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80714.log 2024-03-09T15:37:58.586 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31419.log.gz 2024-03-09T15:37:58.586 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.125055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95466.log 2024-03-09T15:37:58.586 INFO:teuthology.orchestra.run.smithi001.stderr: 84.2% -- replaced with /var/log/ceph/ceph-client.0.125055.log.gz 2024-03-09T15:37:58.586 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80714.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.80714.log.gz 2024-03-09T15:37:58.590 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30743.log 2024-03-09T15:37:58.590 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75994.log 2024-03-09T15:37:58.591 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.95466.log: /var/log/ceph/ceph-client.admin.30743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30743.log.gz 2024-03-09T15:37:58.591 INFO:teuthology.orchestra.run.smithi001.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.95466.log.gz 2024-03-09T15:37:58.591 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.37261.log 2024-03-09T15:37:58.591 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134158.log 2024-03-09T15:37:58.591 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75994.log: /var/log/ceph/ceph-client.0.37261.log: 43.5% 81.8% -- replaced with /var/log/ceph/ceph-client.admin.75994.log.gz 2024-03-09T15:37:58.591 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.0.37261.log.gz 2024-03-09T15:37:58.591 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.113596.log 2024-03-09T15:37:58.592 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43438.log 2024-03-09T15:37:58.592 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.134158.log: /var/log/ceph/ceph-client.0.113596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134158.log.gz 2024-03-09T15:37:58.592 INFO:teuthology.orchestra.run.smithi001.stderr: 84.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134001.log 2024-03-09T15:37:58.592 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.0.113596.log.gz 2024-03-09T15:37:58.592 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43438.log.gz 2024-03-09T15:37:58.592 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54142.log 2024-03-09T15:37:58.592 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77764.log 2024-03-09T15:37:58.592 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.134001.log: /var/log/ceph/ceph-client.admin.54142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134001.log.gz 2024-03-09T15:37:58.592 INFO:teuthology.orchestra.run.smithi001.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.54142.log.gz 2024-03-09T15:37:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41643.log 2024-03-09T15:37:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77764.log: 82.7% -- replaced with /var/log/ceph/ceph-client.admin.77764.log.gz 2024-03-09T15:37:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48183.log 2024-03-09T15:37:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30821.log 2024-03-09T15:37:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41643.log: /var/log/ceph/ceph-client.admin.48183.log: 80.9% -- replaced with /var/log/ceph/ceph-client.admin.41643.log.gz 2024-03-09T15:37:58.596 INFO:teuthology.orchestra.run.smithi001.stderr: 80.6% -- replaced with /var/log/ceph/ceph-client.admin.48183.log.gz 2024-03-09T15:37:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119718.log 2024-03-09T15:37:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78944.log 2024-03-09T15:37:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30821.log: /var/log/ceph/ceph-client.admin.119718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30821.log.gz 2024-03-09T15:37:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.126825.log 2024-03-09T15:37:58.596 INFO:teuthology.orchestra.run.smithi001.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.119718.log.gz 2024-03-09T15:37:58.596 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78944.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.78944.log.gz 2024-03-09T15:37:58.597 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57325.log 2024-03-09T15:37:58.597 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94286.log 2024-03-09T15:37:58.597 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57325.log: /var/log/ceph/ceph-client.0.126825.log: 81.1% -- replaced with /var/log/ceph/ceph-client.0.126825.log.gz 2024-03-09T15:37:58.599 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30819.log 2024-03-09T15:37:58.599 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.94286.log: 82.9% -- replaced with /var/log/ceph/ceph-client.admin.94286.log.gz 2024-03-09T15:37:58.599 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101366.log 2024-03-09T15:37:58.599 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30819.log.gz 2024-03-09T15:37:58.599 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109649.log 2024-03-09T15:37:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.101366.log: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.57325.log.gz 2024-03-09T15:37:58.600 INFO:teuthology.orchestra.run.smithi001.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.101366.log.gz 2024-03-09T15:37:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-09T15:37:58.601 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46413.log 2024-03-09T15:37:58.601 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109649.log: /var/log/ceph/ceph.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109649.log.gz 2024-03-09T15:37:58.603 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83665.log 2024-03-09T15:37:58.603 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46413.log: 82.0% -- replaced with /var/log/ceph/ceph-client.admin.46413.log.gz 2024-03-09T15:37:58.603 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81894.log 2024-03-09T15:37:58.604 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83665.log: 82.3% -- replaced with /var/log/ceph/ceph-client.admin.83665.log.gz 2024-03-09T15:37:58.604 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55396.log 2024-03-09T15:37:58.605 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.81894.log: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.81894.log.gz 2024-03-09T15:37:58.606 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122095.log 2024-03-09T15:37:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55396.log: 93.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-09T15:37:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59899.log 2024-03-09T15:37:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.122095.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.122095.log.gz 2024-03-09T15:37:58.622 INFO:teuthology.orchestra.run.smithi001.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.55396.log.gz 2024-03-09T15:37:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133611.log 2024-03-09T15:37:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31239.log 2024-03-09T15:37:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59899.log: /var/log/ceph/ceph-client.admin.133611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133611.log.gz 2024-03-09T15:37:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.131678.log 2024-03-09T15:37:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31239.log.gz 2024-03-09T15:37:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107289.log 2024-03-09T15:37:58.634 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.131678.log: 80.9% -- replaced with /var/log/ceph/ceph-client.0.131678.log.gz 2024-03-09T15:37:58.635 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99006.log 2024-03-09T15:37:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.107289.log: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.107289.log.gz 2024-03-09T15:37:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30814.log 2024-03-09T15:37:58.637 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.99006.log: 82.7% -- replaced with /var/log/ceph/ceph-client.admin.99006.log.gz 2024-03-09T15:37:58.637 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41003.log 2024-03-09T15:37:58.638 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30814.log.gz 2024-03-09T15:37:58.638 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108469.log 2024-03-09T15:37:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41003.log.gz 2024-03-09T15:37:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41028.log 2024-03-09T15:37:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.108469.log: 82.6% -- replaced with /var/log/ceph/ceph-client.admin.108469.log.gz 2024-03-09T15:37:58.641 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133975.log 2024-03-09T15:37:58.642 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41028.log.gz 2024-03-09T15:37:58.642 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-09T15:37:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133975.log.gz 2024-03-09T15:37:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53903.log 2024-03-09T15:37:58.650 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mon.c.log: 95.2% -- replaced with /var/log/ceph/ceph-client.admin.59899.log.gz 2024-03-09T15:37:58.656 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45823.log 2024-03-09T15:37:58.657 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53903.log: 54.7% -- replaced with /var/log/ceph/ceph-client.admin.53903.log.gz 2024-03-09T15:37:58.657 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32500.log 2024-03-09T15:37:58.658 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45823.log: 80.5% -- replaced with /var/log/ceph/ceph-client.admin.45823.log.gz 2024-03-09T15:37:58.659 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106699.log 2024-03-09T15:37:58.659 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32500.log.gz 2024-03-09T15:37:58.660 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.129775.log 2024-03-09T15:37:58.661 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.106699.log: 82.5% -- replaced with /var/log/ceph/ceph-client.admin.106699.log.gz 2024-03-09T15:37:58.661 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.66954.log 2024-03-09T15:37:58.662 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.129775.log: 84.1% -- replaced with /var/log/ceph/ceph-client.0.129775.log.gz 2024-03-09T15:37:58.662 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133793.log 2024-03-09T15:37:58.663 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.66954.log: 84.3% -- replaced with /var/log/ceph/ceph-client.0.66954.log.gz 2024-03-09T15:37:58.663 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.123875.log 2024-03-09T15:37:58.664 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133793.log.gz 2024-03-09T15:37:58.664 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44053.log 2024-03-09T15:37:58.665 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.123875.log: 84.5% -- replaced with /var/log/ceph/ceph-client.0.123875.log.gz 2024-03-09T15:37:58.666 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30667.log 2024-03-09T15:37:58.667 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44053.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.44053.log.gz 2024-03-09T15:37:58.667 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47003.log 2024-03-09T15:37:58.668 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30667.log.gz 2024-03-09T15:37:58.668 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73044.log 2024-03-09T15:37:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47003.log: 82.8% -- replaced with /var/log/ceph/ceph-client.admin.47003.log.gz 2024-03-09T15:37:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72454.log 2024-03-09T15:37:58.670 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73044.log: 82.1% -- replaced with /var/log/ceph/ceph-client.admin.73044.log.gz 2024-03-09T15:37:58.670 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134132.log 2024-03-09T15:37:58.671 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.72454.log: 82.4% -- replaced with /var/log/ceph/ceph-client.admin.72454.log.gz 2024-03-09T15:37:58.672 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31224.log 2024-03-09T15:37:58.672 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.134132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134132.log.gz 2024-03-09T15:37:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86615.log 2024-03-09T15:37:58.674 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31224.log.gz 2024-03-09T15:37:58.674 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57968.log 2024-03-09T15:37:58.675 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86615.log: 82.6% -- replaced with /var/log/ceph/ceph-client.admin.86615.log.gz 2024-03-09T15:37:58.675 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ops-log-ceph-client.0.log 2024-03-09T15:37:58.681 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57968.log: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.57968.log.gz 2024-03-09T15:37:58.681 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31237.log 2024-03-09T15:37:58.696 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ops-log-ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96646.log 2024-03-09T15:37:58.697 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31237.log.gz 2024-03-09T15:37:58.707 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97826.log 2024-03-09T15:37:58.708 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.96646.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.96646.log.gz 2024-03-09T15:37:58.724 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31031.log 2024-03-09T15:37:58.725 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.97826.log: 82.5% -- replaced with /var/log/ceph/ceph-client.admin.97826.log.gz 2024-03-09T15:37:58.734 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-09T15:37:58.734 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31031.log.gz 2024-03-09T15:37:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31019.log 2024-03-09T15:37:58.764 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.audit.log: 94.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-09T15:37:58.764 INFO:teuthology.orchestra.run.smithi001.stderr: 94.2% -- replaced with /var/log/ceph/ops-log-ceph-client.0.log.gz 2024-03-09T15:37:58.764 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25601.log 2024-03-09T15:37:58.764 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31019.log.gz 2024-03-09T15:37:58.764 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30691.log 2024-03-09T15:37:58.764 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25601.log.gz 2024-03-09T15:37:58.764 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109059.log 2024-03-09T15:37:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30691.log.gz 2024-03-09T15:37:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97236.log 2024-03-09T15:37:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109059.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.109059.log.gz 2024-03-09T15:37:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133741.log 2024-03-09T15:37:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.97236.log: 83.2% -- replaced with /var/log/ceph/ceph-client.admin.97236.log.gz 2024-03-09T15:37:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.123285.log 2024-03-09T15:37:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133741.log.gz 2024-03-09T15:37:58.767 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68216.log 2024-03-09T15:37:58.767 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.123285.log: 80.7% -- replaced with /var/log/ceph/ceph-client.0.123285.log.gz 2024-03-09T15:37:58.767 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105497.log 2024-03-09T15:37:58.767 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.68216.log: 80.3% -- replaced with /var/log/ceph/ceph-client.0.68216.log.gz 2024-03-09T15:37:58.768 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33745.log 2024-03-09T15:37:58.768 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.105497.log: 82.3% -- replaced with /var/log/ceph/ceph-client.admin.105497.log.gz 2024-03-09T15:37:58.769 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31445.log 2024-03-09T15:37:58.769 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.33745.log: 81.5% -- replaced with /var/log/ceph/ceph-client.0.33745.log.gz 2024-03-09T15:37:58.769 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119129.log 2024-03-09T15:37:58.770 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31445.log.gz 2024-03-09T15:37:58.770 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133845.log 2024-03-09T15:37:58.771 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.119129.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.119129.log.gz 2024-03-09T15:37:58.771 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30804.log 2024-03-09T15:37:58.772 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133845.log.gz 2024-03-09T15:37:58.772 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69504.log 2024-03-09T15:37:58.773 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30804.log.gz 2024-03-09T15:37:58.773 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54184.log 2024-03-09T15:37:58.774 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.69504.log: 82.5% -- replaced with /var/log/ceph/ceph-client.admin.69504.log.gz 2024-03-09T15:37:58.774 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88385.log 2024-03-09T15:37:58.775 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54184.log.gz 2024-03-09T15:37:58.775 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103137.log 2024-03-09T15:37:58.776 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.88385.log: 81.4% -- replaced with /var/log/ceph/ceph-client.admin.88385.log.gz 2024-03-09T15:37:58.776 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.130365.log 2024-03-09T15:37:58.777 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.103137.log: 82.2% -- replaced with /var/log/ceph/ceph-client.admin.103137.log.gz 2024-03-09T15:37:58.777 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/ceph.client.0.log 2024-03-09T15:37:58.778 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.130365.log: 84.6% -- replaced with /var/log/ceph/ceph-client.0.130365.log.gz 2024-03-09T15:37:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51823.log 2024-03-09T15:37:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/valgrind/ceph.client.0.log: 69.1% -- replaced with /var/log/ceph/valgrind/ceph.client.0.log.gz 2024-03-09T15:37:58.780 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20508.log 2024-03-09T15:37:58.781 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51823.log: 80.7% -- replaced with /var/log/ceph/ceph-client.admin.51823.log.gz 2024-03-09T15:37:58.781 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59256.log 2024-03-09T15:37:58.782 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.tmp-client.admin.20508.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20508.log.gz 2024-03-09T15:37:58.782 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38643.log 2024-03-09T15:37:58.784 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59256.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59256.log.gz 2024-03-09T15:37:58.784 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110879.log 2024-03-09T15:37:58.785 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38643.log: 83.1% -- replaced with /var/log/ceph/ceph-client.admin.38643.log.gz 2024-03-09T15:37:58.786 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25475.log 2024-03-09T15:37:58.786 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110879.log: 82.6% -- replaced with /var/log/ceph/ceph-client.admin.110879.log.gz 2024-03-09T15:37:58.787 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30582.log 2024-03-09T15:37:58.787 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.25475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25475.log.gz 2024-03-09T15:37:58.788 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43413.log 2024-03-09T15:37:58.788 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30582.log.gz 2024-03-09T15:37:58.789 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96056.log 2024-03-09T15:37:58.790 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43413.log.gz 2024-03-09T15:37:58.790 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79534.log 2024-03-09T15:37:58.791 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.96056.log: 82.5% -- replaced with /var/log/ceph/ceph-client.admin.96056.log.gz 2024-03-09T15:37:58.791 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133897.log 2024-03-09T15:37:58.791 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79534.log: 82.4% -- replaced with /var/log/ceph/ceph-client.admin.79534.log.gz 2024-03-09T15:37:58.793 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31218.log 2024-03-09T15:37:58.793 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133897.log.gz 2024-03-09T15:37:58.793 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30817.log 2024-03-09T15:37:58.794 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31218.log.gz 2024-03-09T15:37:58.795 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32476.log 2024-03-09T15:37:58.795 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30817.log.gz 2024-03-09T15:37:58.795 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70684.log 2024-03-09T15:37:58.796 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32476.log.gz 2024-03-09T15:37:58.796 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133585.log 2024-03-09T15:37:58.797 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.70684.log: 80.7% -- replaced with /var/log/ceph/ceph-client.admin.70684.log.gz 2024-03-09T15:37:58.797 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54232.log 2024-03-09T15:37:58.798 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133585.log.gz 2024-03-09T15:37:58.798 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54773.log 2024-03-09T15:37:58.799 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54232.log: 56.9% -- replaced with /var/log/ceph/ceph-client.admin.54232.log.gz 2024-03-09T15:37:58.799 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36093.log 2024-03-09T15:37:58.800 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54773.log: 20.3% -- replaced with /var/log/ceph/ceph-client.admin.54773.log.gz 2024-03-09T15:37:58.800 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40413.log 2024-03-09T15:37:58.801 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.36093.log: 43.9% -- replaced with /var/log/ceph/ceph-client.0.36093.log.gz 2024-03-09T15:37:58.801 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64537.log 2024-03-09T15:37:58.802 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40413.log: 82.1% -- replaced with /var/log/ceph/ceph-client.admin.40413.log.gz 2024-03-09T15:37:58.802 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54807.log 2024-03-09T15:37:58.804 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64537.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64537.log.gz 2024-03-09T15:37:58.804 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99596.log 2024-03-09T15:37:58.805 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54807.log: 84.3% -- replaced with /var/log/ceph/ceph-client.admin.54807.log.gz 2024-03-09T15:37:58.805 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30769.log 2024-03-09T15:37:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.99596.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.99596.log.gz 2024-03-09T15:37:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65721.log 2024-03-09T15:37:58.807 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30769.log.gz 2024-03-09T15:37:58.807 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39823.log 2024-03-09T15:37:58.808 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.65721.log: 82.6% -- replaced with /var/log/ceph/ceph-client.admin.65721.log.gz 2024-03-09T15:37:58.809 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75404.log 2024-03-09T15:37:58.809 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39823.log: 82.5% -- replaced with /var/log/ceph/ceph-client.admin.39823.log.gz 2024-03-09T15:37:58.810 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53003.log 2024-03-09T15:37:58.810 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.75404.log: 82.6% -- replaced with /var/log/ceph/ceph-client.admin.75404.log.gz 2024-03-09T15:37:58.811 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86025.log 2024-03-09T15:37:58.812 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53003.log: 82.5% -- replaced with /var/log/ceph/ceph-client.admin.53003.log.gz 2024-03-09T15:37:58.812 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30642.log 2024-03-09T15:37:58.813 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.86025.log: 82.4% -- replaced with /var/log/ceph/ceph-client.admin.86025.log.gz 2024-03-09T15:37:58.813 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134234.log 2024-03-09T15:37:58.815 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30642.log.gz 2024-03-09T15:37:58.815 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83075.log 2024-03-09T15:37:58.815 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.134234.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134234.log.gz 2024-03-09T15:37:58.816 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133949.log 2024-03-09T15:37:58.816 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83075.log: 82.6% -- replaced with /var/log/ceph/ceph-client.admin.83075.log.gz 2024-03-09T15:37:58.816 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34919.log 2024-03-09T15:37:58.817 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133949.log.gz 2024-03-09T15:37:58.817 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112699.log 2024-03-09T15:37:58.818 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.34919.log: 80.7% -- replaced with /var/log/ceph/ceph-client.0.34919.log.gz 2024-03-09T15:37:58.818 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107879.log 2024-03-09T15:37:58.819 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.112699.log: 82.3% -- replaced with /var/log/ceph/ceph-client.admin.112699.log.gz 2024-03-09T15:37:58.820 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133767.log 2024-03-09T15:37:58.820 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.107879.log: 80.0% -- replaced with /var/log/ceph/ceph-client.admin.107879.log.gz 2024-03-09T15:37:58.821 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31012.log 2024-03-09T15:37:58.821 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133767.log.gz 2024-03-09T15:37:58.822 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102547.log 2024-03-09T15:37:58.822 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31012.log.gz 2024-03-09T15:37:58.822 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.128005.log 2024-03-09T15:37:58.823 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.102547.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.102547.log.gz 2024-03-09T15:37:58.824 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63943.log 2024-03-09T15:37:58.825 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.128005.log: 84.5% -- replaced with /var/log/ceph/ceph-client.0.128005.log.gz 2024-03-09T15:37:58.825 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66312.log 2024-03-09T15:37:58.826 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63943.log: 84.4% -- replaced with /var/log/ceph/ceph-client.admin.63943.log.gz 2024-03-09T15:37:58.826 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31227.log 2024-03-09T15:37:58.827 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66312.log: 25.1% -- replaced with /var/log/ceph/ceph-client.admin.66312.log.gz 2024-03-09T15:37:58.827 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60542.log 2024-03-09T15:37:58.828 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31227.log.gz 2024-03-09T15:37:58.828 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51798.log 2024-03-09T15:37:58.830 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60542.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.60542.log.gz 2024-03-09T15:37:58.831 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94876.log 2024-03-09T15:37:58.831 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51798.log.gz 2024-03-09T15:37:58.832 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112109.log 2024-03-09T15:37:58.833 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.94876.log: 82.8% -- replaced with /var/log/ceph/ceph-client.admin.94876.log.gz 2024-03-09T15:37:58.833 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.32571.log 2024-03-09T15:37:58.834 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.112109.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.112109.log.gz 2024-03-09T15:37:58.834 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31211.log 2024-03-09T15:37:58.836 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.32571.log: 95.9% -- replaced with /var/log/ceph/ceph-client.0.32571.log.gz 2024-03-09T15:37:58.836 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90156.log 2024-03-09T15:37:58.837 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31211.log.gz 2024-03-09T15:37:58.837 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89566.log 2024-03-09T15:37:58.838 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90156.log: 82.3% -- replaced with /var/log/ceph/ceph-client.admin.90156.log.gz 2024-03-09T15:37:58.838 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80124.log 2024-03-09T15:37:58.839 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89566.log: 82.7% -- replaced with /var/log/ceph/ceph-client.admin.89566.log.gz 2024-03-09T15:37:58.839 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133715.log 2024-03-09T15:37:58.840 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.80124.log: 82.6% -- replaced with /var/log/ceph/ceph-client.admin.80124.log.gz 2024-03-09T15:37:58.840 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.66364.log 2024-03-09T15:37:58.841 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.133715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133715.log.gz 2024-03-09T15:37:58.841 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61185.log 2024-03-09T15:37:58.842 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.66364.log: 84.4% -- replaced with /var/log/ceph/ceph-client.0.66364.log.gz 2024-03-09T15:37:58.842 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54255.log 2024-03-09T15:37:58.844 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61185.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.61185.log.gz 2024-03-09T15:37:58.845 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117224.log 2024-03-09T15:37:58.846 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54255.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.54255.log.gz 2024-03-09T15:37:58.847 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.117224.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.117224.log.gz 2024-03-09T15:37:59.012 INFO:teuthology.orchestra.run.smithi053.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-09T15:37:59.864 INFO:teuthology.orchestra.run.smithi001.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-09T15:37:59.923 INFO:teuthology.orchestra.run.smithi053.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-09T15:38:00.759 INFO:teuthology.orchestra.run.smithi001.stderr: 92.7% -- replaced with /var/log/ceph/rgw.ceph.client.0.log.gz 2024-03-09T15:38:01.165 INFO:teuthology.orchestra.run.smithi001.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-09T15:38:02.930 INFO:teuthology.orchestra.run.smithi001.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-09T15:38:03.724 INFO:teuthology.orchestra.run.smithi053.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-09T15:38:04.375 INFO:teuthology.orchestra.run.smithi053.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-09T15:38:04.799 INFO:teuthology.orchestra.run.smithi001.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-09T15:38:06.861 INFO:teuthology.orchestra.run.smithi053.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-09T15:38:07.170 INFO:teuthology.orchestra.run.smithi001.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-09T15:38:07.646 INFO:teuthology.orchestra.run.smithi053.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-09T15:38:07.647 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-09T15:38:07.647 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m9.134s 2024-03-09T15:38:07.647 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m29.357s 2024-03-09T15:38:07.647 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.998s 2024-03-09T15:38:14.271 INFO:teuthology.orchestra.run.smithi001.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-09T15:38:14.273 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-09T15:38:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:real 0m15.797s 2024-03-09T15:38:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:user 0m40.313s 2024-03-09T15:38:14.274 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m1.387s 2024-03-09T15:38:14.274 INFO:tasks.ceph:Archiving logs... 2024-03-09T15:38:14.274 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479/remote/smithi001/log 2024-03-09T15:38:14.275 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-09T15:38:18.885 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479/remote/smithi053/log 2024-03-09T15:38:18.885 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-09T15:38:22.454 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_PossiblyLost calloc calloc allocate_dtv 2024-03-09T15:38:22.467 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-09T15:38:22.478 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c00d9c609fb42a2a25f1c196bed064ca8bb02413/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_PossiblyLost calloc calloc allocate_dtv 2024-03-09T15:38:22.481 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-09T15:38:22.481 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-09T15:38:22.497 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-09T15:38:22.561 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-09T15:38:22.572 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-09T15:38:22.572 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-09T15:38:22.576 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-09T15:38:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: remote refid st t when poll reach delay offset jitter 2024-03-09T15:38:22.590 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================== 2024-03-09T15:38:22.590 INFO:teuthology.orchestra.run.smithi001.stdout:*hv01.front.sepi 44.4.53.2 2 u 68 128 377 0.204 -2.601 0.322 2024-03-09T15:38:22.590 INFO:teuthology.orchestra.run.smithi001.stdout:+hv02.front.sepi 63.231.80.2 3 u 1 128 377 0.164 +0.801 0.617 2024-03-09T15:38:22.590 INFO:teuthology.orchestra.run.smithi001.stdout:+hv03.front.sepi 74.6.168.72 3 u 79 128 377 0.193 -1.907 0.815 2024-03-09T15:38:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-09T15:38:22.593 INFO:teuthology.orchestra.run.smithi053.stdout: remote refid st t when poll reach delay offset jitter 2024-03-09T15:38:22.594 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================== 2024-03-09T15:38:22.594 INFO:teuthology.orchestra.run.smithi053.stdout:*hv01.front.sepi 44.4.53.2 2 u 21 128 377 0.098 -2.579 0.333 2024-03-09T15:38:22.594 INFO:teuthology.orchestra.run.smithi053.stdout:+hv02.front.sepi 63.231.80.2 3 u 79 128 377 0.102 -0.361 0.616 2024-03-09T15:38:22.594 INFO:teuthology.orchestra.run.smithi053.stdout:+hv03.front.sepi 74.6.168.72 3 u 16 128 377 0.094 -1.743 0.560 2024-03-09T15:38:22.594 INFO:teuthology.orchestra.run.smithi053.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-09T15:38:22.595 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-09T15:38:22.613 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-09T15:38:22.614 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-09T15:38:22.649 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-09T15:38:22.693 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-09T15:38:22.740 INFO:teuthology.task.internal:Duration was 4840.183642 seconds 2024-03-09T15:38:22.740 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-09T15:38:22.785 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-09T15:38:22.785 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-09T15:38:22.788 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-09T15:38:22.829 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-09T15:38:22.830 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-03-09T15:38:22.830 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-09T15:38:22.878 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-03-09T15:38:22.878 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-09T15:38:22.890 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-09T15:38:22.890 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-09T15:38:22.922 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-09T15:38:22.957 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-09T15:38:22.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-09T15:38:22.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-09T15:38:23.054 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-09T15:38:23.067 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-09T15:38:23.068 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-09T15:38:23.081 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-09T15:38:23.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-09T15:38:23.111 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:38:23.126 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:38:23.134 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-03-09T15:38:23.153 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-03-09T15:38:23.180 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:38:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T15:38:23.196 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:38:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T15:38:23.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-09T15:38:23.245 INFO:teuthology.task.internal:Transferring archived files... 2024-03-09T15:38:23.246 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479/remote/smithi001 2024-03-09T15:38:23.246 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-09T15:38:23.272 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479/remote/smithi053 2024-03-09T15:38:23.273 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-09T15:38:23.302 INFO:teuthology.task.internal:Removing archive directory... 2024-03-09T15:38:23.302 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-09T15:38:23.309 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-09T15:38:23.344 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-09T15:38:23.359 INFO:teuthology.task.internal:Not uploading archives. 2024-03-09T15:38:23.359 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-09T15:38:23.385 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-09T15:38:23.386 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-09T15:38:23.388 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-09T15:38:23.392 INFO:teuthology.orchestra.run.smithi001.stdout: 658625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 9 15:38 /home/ubuntu/cephtest 2024-03-09T15:38:23.394 INFO:teuthology.orchestra.run.smithi053.stdout: 658672 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 9 15:38 /home/ubuntu/cephtest 2024-03-09T15:38:23.396 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-09T15:38:23.430 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-09T15:38:23.471 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_PossiblyLost calloc calloc allocate_dtv 2024-03-09T15:38:23.471 INFO:teuthology.nuke:Checking targets against current locks 2024-03-09T15:38:23.503 DEBUG:teuthology.nuke:shortname: smithi001 2024-03-09T15:38:23.504 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T15:38:23.526 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-09 14:09:16.574079', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDXuFOquNPh30bZgAJqTouKGG7p0NNxx6xXxJjUuU/VardxqUiKgFx1dYvcLV+s2NboGkt+PBwES7RqxIXJxcpQ='} 2024-03-09T15:38:23.530 DEBUG:teuthology.nuke:shortname: smithi053 2024-03-09T15:38:23.530 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T15:38:23.553 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2024-03-09_13:26:09-rgw-wip-rgw-account-v3-distro-default-smithi/7588479', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-09 14:09:16.576045', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKaPOccuN9FVN66Lr7uOPJ7Q4JA1CmmSB1iQvohXt3rGZ6uBS7/6xY3JYjapv/HuJSah4Lhzo+bkGObImlln7ic='} 2024-03-09T15:38:23.584 INFO:teuthology.orchestra.console.smithi001:Power off 2024-03-09T15:38:23.585 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T15:38:23.610 DEBUG:teuthology.orchestra.console.smithi001:power off output: Chassis Power Control: Down/Off 2024-03-09T15:38:23.611 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T15:38:23.634 INFO:teuthology.orchestra.console.smithi053:Power off 2024-03-09T15:38:23.635 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T15:38:23.654 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-09T15:38:23.672 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-03-09T15:38:23.672 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T15:38:23.698 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-09T15:38:27.656 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T15:38:27.699 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T15:38:27.773 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-09T15:38:27.812 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-09T15:38:31.774 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T15:38:31.813 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T15:38:31.889 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-09T15:38:31.923 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-09T15:38:35.894 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T15:38:35.924 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T15:38:36.006 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is off 2024-03-09T15:38:36.040 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-09T15:38:36.108 INFO:teuthology.orchestra.console.smithi001:Power off completed 2024-03-09T15:38:36.142 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-03-09T15:38:36.248 INFO:teuthology.run:Summary data: description: rgw/verify/{0-install accounts$/{none} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} duration: 4840.183641910553 failure_reason: 'Command failed (s3 tests against rgw) on smithi001 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m ''not fails_on_rgw and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3''"' flavor: default owner: scheduled_cbodley@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=43cae2bbeecb47dd8b4ca966794a07c5 status: fail success: false 2024-03-09T15:38:36.249 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-09T15:38:36.330 INFO:teuthology.run:FAIL